Community inspection management method and system based on multi-modal fusion

Abstract

The invention discloses a community inspection management method and system based on multi-modal fusion, and the method comprises the following steps: carrying out the feature extraction of monitoring point modal data based on multi-modal fusion, obtaining monitoring point feature data, obtaining a clustering result based on the monitoring point feature data through a clustering algorithm, and constructing an edge calculation node; constructing an edge anomaly screening model based on the inspection anomaly result according to the historical monitoring data; constructing an anomaly analysis model by using the edge anomaly screening model and the edge tie line; dynamically adjusting the anomaly analysis model by using an incremental clustering algorithm based on the state change of the monitoring point; and outputting a community inspection strategy according to the anomaly analysis model and the current monitoring data. The method has the beneficial effects that the accuracy of inspection planning is improved while the inspection analysis efficiency is improved.

Description

Technical Field

[0001] This application relates to the technical field of community management, and in particular, to a community patrol management method and system based on multimodal fusion. Background Art

[0002] With the continuous expansion of the scale of various regions such as communities, industrial parks, and cities, as well as the increasing demands for security management, operation and maintenance, etc., monitoring systems play an increasingly crucial role. Traditional monitoring methods often rely on single-modal data and have many deficiencies in aspects such as abnormal analysis and patrol strategy formulation, making it difficult to meet the requirements of current complex and efficient monitoring application scenarios.

[0003] In traditional monitoring technologies, most only focus on single-modal data, such as only collecting video image data or simple sensor numerical data, etc., lacking the effective utilization of multimodal data (such as the integration of multiple types of data like video, audio, environmental sensor data, etc.). When formulating community patrol strategies, there is often a lack of combination with real-time and comprehensive monitoring data. Usually, fixed periodic patrol routes and task arrangements are adopted, without fully considering the potential abnormal risks reflected by the current actual monitoring data and important information such as the regional correlation and abnormal occurrence probability provided by the abnormal analysis model.

[0004] In related technologies, the judgment of whether there is an abnormality at a monitoring point is only based on the information conveyed by the monitoring point itself, treating each monitoring point in isolation, thus causing errors in abnormal analysis when the information conveyed by a single monitoring point is not comprehensive enough, affecting the accuracy of patrol route planning.

[0005] The patent "Intelligent Patrol Method, System and Terminal Device Based on Data Acquisition and Multimodal AI", publication number: CN119203042A, publication date: December 27, 2024, specifically discloses performing data consistency judgment and analysis on each voxel chunk in the voxel space to obtain various multimodal data groups of the patrol target within a preset time; obtaining the modal features of each multimodal data group, processing the modal features of each multimodal data group according to a preset mapping and weight mechanism to obtain a multimodal fusion feature vector fusion model of the patrol target; analyzing the patrol target according to the multimodal fusion feature vector fusion model and the multimodal fusion feature vector reference model to obtain the working state of the patrol target. This solution uniformly fuses and processes all data, has high requirements for a single processing unit, large computational pressure, and low efficiency.

[0006] Patent "An Intelligent Patrol Management Method and System for Surveillance Videos Based on Artificial Intelligence", Publication Number: CN119131702A, Publication Date: December 13, 2024. Specifically, it discloses collecting historical video surveillance data, establishing a sample set, preprocessing the historical video surveillance data in the sample set, and based on the preprocessed historical video surveillance data, extracting abnormal behavior features and spatio-temporal features in the historical video surveillance data through machine learning methods. At the same time, based on the spatio-temporal features extracted from the historical video surveillance data, the reminder frequency of the surveillance camera is set. And by setting a video surveillance event, when the administrator selects a frame of video surveillance data in the corresponding time period, the selected video frame is identified and a patrol record is generated based on machine learning methods and data matching methods. Finally, by setting a patrol evaluation management standard and evaluating and optimizing the patrol records generated by the administrator. Similarly, this solution integrates and processes all data, which has high requirements for a single processing unit, large computing pressure, and low efficiency. Summary of the Invention

[0007] In view of the problems of low efficiency and one-sided analysis existing in community patrol management in the prior art, the present application provides a community patrol management method and system based on multi-modal fusion. By extracting feature data of monitoring points through multi-modal fusion to construct edge computing nodes for distributed computing, sharing the computing pressure and improving the computing efficiency. At the same time, using edge liaison lines to reflect the correlation between edge computing nodes to comprehensively consider the possibility of anomalies, improving the accuracy of analysis, and dynamically adjusting with the incremental clustering algorithm according to the change of the monitoring point state to improve the adaptability of the anomaly analysis model, improving the accuracy of patrol planning while improving the patrol analysis efficiency.

[0008] To achieve the above technical objectives, a technical solution provided by the present application is a community patrol management method based on multi-modal fusion, including the following steps: S1: Obtain the modal data of the monitoring points, extract the feature data of the monitoring points through multi-modal fusion to obtain the monitoring point feature data, use the clustering algorithm to obtain the clustering result based on the monitoring point feature data, and construct edge computing nodes according to the clustering result; S2: Obtain the historical monitoring data corresponding to each edge computing node, and construct an edge anomaly screening model based on the historical monitoring data and the patrol anomaly results; S3: Construct an edge liaison line based on the correlation of the monitoring area, and construct an anomaly analysis model with the edge anomaly screening model and the edge liaison line; S4: Real-time obtain the status of the monitoring points, and dynamically adjust the anomaly analysis model using the incremental clustering algorithm based on the change of the monitoring point status; S5: Obtain the current monitoring data, and output the community patrol strategy according to the anomaly analysis model and the current monitoring data.

[0009] Further, S1 further includes: obtaining monitoring point image data, sensor data, and text description data, performing feature extraction on the monitoring point image data, sensor data, and text description data to obtain an extracted feature set; using the principal component analysis method combined with feature splicing to perform feature fusion on the extracted feature set to obtain monitoring point feature data; using the DBSCAN clustering algorithm to cluster the monitoring point feature data to obtain a clustering result; and matching edge computing nodes according to the clustering result.

[0010] Further, the performing feature extraction on the monitoring point image data, sensor data, and text description data to obtain an extracted feature set includes: matching the image segmentation resolution according to the image complexity of the monitoring point image data, segmenting the image based on the image segmentation resolution to obtain a segmented image set, and using a convolutional neural network to perform feature extraction on the segmented image set to obtain image features; performing feature extraction on the sensor data according to the time series and sensing correlation to obtain sensing features; using semantic recognition to perform feature extraction on the text description data to obtain semantic features; and obtaining the extracted feature set with the image features, sensing features, and semantic features.

[0011] Further, the matching the image segmentation resolution according to the image complexity of the monitoring point image data, segmenting the image based on the image segmentation resolution to obtain a segmented image set includes: calculating the information entropy of the monitoring point image data, and obtaining the image complexity according to the information entropy; matching the image segmentation resolution corresponding to the image complexity based on the preset complexity and segmentation correlation; and segmenting the image based on the image segmentation resolution to obtain a segmented image set.

[0012] Further, the using a convolutional neural network to perform feature extraction on the segmented image set to obtain image features includes: constructing parallel feature recognition streams based on the convolutional neural network structure according to the segmentation resolution, and constructing a multi-stream parallel convolutional neural network with the feature recognition streams and the resolution judgment layer; using the segmented image set as the input of the multi-stream parallel convolutional neural network, and outputting image features.

[0013] Further, the performing feature extraction on the sensor data according to the time series and sensing correlation to obtain sensing features includes: performing trend feature and periodic feature extraction according to the trend change and periodic change of the sensor data under the time series; using correlation analysis to obtain the sensing correlation features between sensors, and obtaining the sensing features with the sensing correlation features, trend features, and periodic features.

[0014] Further, the step S3 further includes: constructing the monitoring area correlation based on the geographical location correlation of the monitoring points and the functional correlation of the monitoring areas; obtaining the association between the monitoring points according to the monitoring area correlation, and constructing the edge connection lines based on the association between the monitoring points; constructing the anomaly analysis model with the edge anomaly screening model and the edge connection lines.

[0015] Further, the step S4 further includes: obtaining the status of the monitoring points in real time, and when the status of the monitoring points changes, obtaining the monitoring points whose status has changed; using the density-based incremental clustering algorithm to update the clustering results according to the monitoring points whose status has changed; dynamically updating the coverage of the edge connection lines and the edge computing nodes with the updated clustering results and the preset node load threshold.

[0016] Further, the step S5 further includes: obtaining the current monitoring data, and obtaining the abnormal monitoring location and abnormal factors based on the current monitoring data according to the anomaly analysis model; constructing the community patrol route according to the path and urgency based on the community road data, the abnormal monitoring location, and the abnormal factors.

[0017] Another technical solution provided by this application is a community patrol management system based on multimodal fusion, which is used to implement the method as described above, and includes: a data processing unit, which is used to extract features from the modal data of the monitoring points based on multimodal fusion to obtain the monitoring point feature data; an edge computing node, which is constructed according to the clustering results of the monitoring point feature data, and is used to construct an edge anomaly screening model according to the historical monitoring data; an overall analysis unit, which is used to construct the edge connection lines according to the monitoring area correlation, and output the anomaly analysis result based on the output value of the edge anomaly screening model and the edge connection lines; a patrol planning unit, which is used to output the community patrol strategy according to the anomaly analysis result.

[0018] The beneficial effects of this application are as follows: 1. The monitoring point feature data is obtained by fusing the modal data of the monitoring points, the edge computing nodes are constructed based on the clustering results of the monitoring point feature data, and then the edge anomaly screening models for each edge computing node are constructed according to the historical monitoring data of each edge computing node. By deploying the edge computing nodes and constructing the models, the computing pressure of a single computing unit is reduced, the anomaly screening efficiency is improved, and the edge connection lines between the edge computing nodes are constructed using the monitoring area correlation, so as to re-evaluate the anomaly possibility based on the overall correlation, reduce the computing data during the overall evaluation, improve the efficiency and ensure the accuracy of the anomaly analysis. Furthermore, the community patrol strategy is output according to the anomaly analysis result, improving the patrol efficiency.

[0019] 2. For simple images, due to the use of a lower segmentation resolution, unnecessary noise and interference information introduced by over-segmentation are avoided, enabling the convolutional neural network to focus on extracting key and representative features in the image. For complex images, a higher segmentation resolution ensures that rich details in the image are not lost during the segmentation process. As a result, the convolutional neural network does not need to perform repeated and unnecessary operations on excessive and relatively single-content image blocks, saving computing resources and time costs. At the same time, images of different complexities are processed according to their respective matching segmentation resolutions, enabling the entire image feature extraction process to make full use of computing resources for parallel processing and further improving the feature extraction efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 The flowchart shows the community patrol management method based on multi-modal fusion of this application. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0021] To make the objectives, technical solutions, and advantages of this application clearer, the following further elaborates on this application in conjunction with the drawings and embodiments. It should be understood that the specific embodiments described herein are only the best embodiments of this application, which are only used to explain this application and do not limit the protection scope of this application. All other embodiments obtained by those of ordinary skill in the art without creative efforts fall within the protection scope of this application.

[0022] As Figure 1 shown, as the first embodiment of this application, the community patrol management method based on multi-modal fusion includes the following steps: S1: Obtain the modal data of the monitoring points, extract the feature data of the monitoring points based on multi-modal fusion, obtain the clustering result based on the feature data of the monitoring points using the clustering algorithm, and construct edge computing nodes according to the clustering result; S2: Obtain the historical monitoring data corresponding to each edge computing node, and construct an edge anomaly screening model based on the historical monitoring data according to the patrol anomaly results; S3: Construct edge connection lines based on the relevance of the monitoring areas, and construct an anomaly analysis model with the edge anomaly screening model and the edge connection lines; S4: Obtain the status of the monitoring points in real time, and dynamically adjust the anomaly analysis model using the incremental clustering algorithm based on the changes in the status of the monitoring points; S5: Obtain the current monitoring data, and output the community patrol strategy according to the anomaly analysis model and the current monitoring data.

[0023] In this embodiment, based on the modal data fusion of the monitoring points, the feature data of the monitoring points is obtained. An edge computing node is constructed based on the clustering result of the feature data of the monitoring points. Then, an edge anomaly screening model for each edge computing node is constructed according to the historical monitoring data of each edge computing node. By deploying the edge computing nodes and constructing the model, the computing pressure of a single computing unit is reduced, the anomaly screening efficiency is improved, and the edge connection lines between the edge computing nodes are constructed by using the relevance of the monitoring areas, so as to re-evaluate the possibility of anomalies based on the overall relevance, reduce the computing data during the overall evaluation, improve the efficiency while ensuring the accuracy of anomaly analysis, and then output a community inspection strategy according to the anomaly analysis result to improve the inspection efficiency.

[0024] Among them, step S1 further includes: Obtain the monitoring point image data, sensor data, and text description data, perform feature extraction on the monitoring point image data, sensor data, and text description data, and obtain an extraction feature set; Use the principal component analysis method combined with feature splicing to perform feature fusion on the extraction feature set to obtain the feature data of the monitoring points; Use the DBSCAN clustering algorithm to cluster the feature data of the monitoring points to obtain a clustering result; Match the edge computing nodes according to the clustering result.

[0025] Collect the image data, sensor data, and text description data of the monitoring area. Among them, the image data at least includes the spatial layout data and the facility distribution data, the sensor data at least includes the environmental data and the personnel flow data, and the text description data at least includes the regional function annotation data and the special requirement data.

[0026] Perform data preprocessing on the collected image data, sensor data, and text description data, and perform feature extraction on the image data, sensor data, and text description data respectively. At this time, performing feature extraction on the monitoring point image data, sensor data, and text description data to obtain an extraction feature set includes: Match the image segmentation resolution according to the image complexity of the monitoring point image data, segment the image based on the image segmentation resolution to obtain a segmented image set, and use a convolutional neural network to perform feature extraction on the segmented image set to obtain image features; Perform feature extraction on the sensor data according to the time series and the sensing correlation to obtain sensing features; Use semantic recognition to perform feature extraction on the text description data to obtain semantic features; Obtain an extraction feature set with the image features, sensing features, and semantic features.

[0027] In this embodiment, by matching the corresponding image segmentation resolution according to the image complexity, refined processing of images with different complexities is achieved. For simple images, due to the adoption of a lower segmentation resolution, unnecessary noise and interference information introduced by over-segmentation are avoided, enabling the convolutional neural network to focus on extracting key and representative features in the image; for complex images, a higher segmentation resolution ensures that rich details in the image are not lost during the segmentation process. Thus, the convolutional neural network does not need to perform repeated and unnecessary operations on excessive and relatively single-content image blocks, saving computational resources and time costs. At the same time, images with different complexities are processed according to their respective matched segmentation resolutions, enabling the entire image feature extraction process to make full use of computational resources for parallel processing and further improving the feature extraction efficiency.

[0028] Specifically, match the image segmentation resolution according to the image complexity of the monitoring point image data, and segment the image based on the image segmentation resolution to obtain a set of segmented images, including: Calculate the information entropy of the monitoring point image data, and obtain the image complexity according to the information entropy; Match the image segmentation resolution corresponding to the image complexity based on the correlation between complexity and segmentation; Segment the image based on the image segmentation resolution to obtain a set of segmented images.

[0029] For each monitoring point image, calculate the information entropy by traversing the image pixels to construct a grayscale histogram, and use the information entropy as a reference value for the image complexity. The higher the information entropy of the image, the higher its image complexity is considered. Construct the correlation between complexity and segmentation according to the evaluation result of the actual scene complexity and the importance relationship of the scene. Match the image segmentation resolution corresponding to the image complexity based on the correlation between complexity and segmentation, and segment the image based on the image segmentation resolution. Thus, for images with higher complexity, select a higher segmentation resolution for segmentation to retain more detailed information, and for images with lower complexity, select a lower segmentation resolution for segmentation to reduce the amount of calculation and improve the processing speed.

[0030] Use a convolutional neural network to extract features from the set of segmented images to obtain image features, including: Based on the convolutional neural network structure, construct parallel feature recognition streams according to the segmentation resolution, and build a multi-stream parallel convolutional neural network with the feature recognition streams and the resolution judgment layer; Use the set of segmented images as the input of the multi-stream parallel convolutional neural network and output the image features.

[0031] Construct corresponding feature recognition streams for different segmentation resolutions. For example, set the first complexity threshold and the second complexity threshold according to the image complexity. When the image complexity is less than the first complexity threshold, the image is classified as a low-complexity image. When the image complexity is greater than or equal to the first complexity threshold and less than or equal to the second complexity threshold, the image is classified as a medium-complexity image. When the image complexity is greater than the second complexity threshold, the image is classified as a high-complexity image. Correspondingly, the low-complexity image matches the low segmentation resolution, the medium-complexity image matches the medium segmentation resolution, and the high-complexity image matches the high segmentation resolution. At this time, the feature recognition stream includes a low-resolution feature recognition stream corresponding to the low segmentation resolution, a medium-resolution feature recognition stream corresponding to the medium segmentation resolution, and a high-resolution feature recognition stream corresponding to the high segmentation resolution. The feature recognition stream at least includes a convolutional layer, a pooling layer, and an activation function.

[0032] The resolution judgment layer is located at the front end of the entire network. According to the resolution of the input image patch, it guides the data flow to the corresponding feature recognition stream, ensuring that the image patches with different segmentation resolutions are accurately assigned to the appropriate feature recognition stream, and avoiding problems such as poor feature extraction effect or waste of computing resources caused by the mismatch between the resolution and the feature recognition stream. In this embodiment, the feature recognition streams with different resolutions are arranged in parallel and independently extract features from the image patches with the corresponding resolutions. The parallel architecture is used to achieve efficient and accurate feature extraction for diverse image data, improving the adaptability of the entire network to complex image scenarios and the adaptability of community applications.

[0033] Extract features from sensor data according to the time series and sensing correlation. The obtained sensing features include: extracting trend features and periodic features according to the trend change and periodic change of sensor data in the time series; Obtain the sensing correlation features between sensors through correlation analysis, and obtain the sensing features with the sensing correlation features, trend features, and periodic features.

[0034] Use methods such as linear regression analysis to calculate the change slope of sensor data over time. For example, for a temperature sensor, if the slope suddenly increases, it may indicate an abnormal temperature rise in the area. At the same time, use correlation analysis such as calculating the Pearson correlation coefficient to obtain the correlation features between different sensors to demonstrate the correlation relationship between sensor data.

[0035] Apply tools such as lexical analysis and syntactic analysis in natural language processing (NLP) to process the organized text data, realize the processing of text description data, and then extract semantic feature associations based on topic associations and entity associations to obtain semantic features.

[0036] Furthermore, the principal component analysis method is used in combination with feature splicing to perform feature fusion on the extracted feature set, and the monitoring point feature data obtained includes: The principal component analysis method is used to perform dimensionality reduction processing on the extracted feature set; Based on feature splicing, the extracted features after dimensionality reduction processing are fused to obtain the monitoring point feature data.

[0037] Since the number of different modal features extracted is large and there is a certain amount of redundant information, the principal component analysis (PCA) is used to perform dimensionality reduction processing on each modal feature respectively to reduce the dimension of each modal feature, remove noise and redundancy, and improve the subsequent fusion efficiency. After the PCA dimensionality reduction of each modal feature is completed, the dimensionality-reduced image feature vector, sensor feature vector, and text feature vector are spliced in chronological order to form a new feature vector, and the new feature vector is used as the feature data of the monitoring point.

[0038] Furthermore, the similarity between the monitoring point feature data is calculated, the DBSCAN clustering algorithm is used to cluster the monitoring point feature data to obtain the clustering result, the edge computing nodes are matched according to the clustering result, and the edge computing node positions are obtained according to the installation position requirements of the edge computing nodes, the minimum distance from the clustering center, and the maximum coverage position of the clustering cluster. The installation position requirements of the edge computing nodes include network environment requirements, physical environment requirements, etc. Thus, distributed computing is realized by using edge computing, the computing pressure of a single computing unit is reduced, the monitoring point data belonging to the cluster can be processed nearby, the data transmission delay is reduced, and the overall data processing efficiency and real-time performance are improved.

[0039] In this embodiment, step S2 further includes: For each edge computing node, historical monitoring data is obtained, and the historical monitoring data is divided into normal data and abnormal data based on the inspection abnormal results; An edge anomaly screening model is trained using a machine learning algorithm based on the normal data and the abnormal data.

[0040] The monitoring data within the historical time period is extracted from the storage system associated with the edge computing node. The historical monitoring data with abnormal marks is used as the positive sample, and the normal historical monitoring data is used as the negative sample. An edge anomaly screening model for distinguishing abnormal and normal situations is trained using a machine learning algorithm.

[0041] Step S3 further includes: Construct the monitoring area correlation based on the geographical location correlation of the monitoring points and the functional correlation of the monitoring areas; Obtain the correlation between the monitoring points according to the monitoring area correlation, and construct the edge connection line based on the correlation between the monitoring points; construct the anomaly analysis model based on the edge anomaly screening model and the edge connection line.

[0042] Construct the monitoring area correlation by monitoring the geographical location correlation of monitoring points and the functional correlation of monitoring areas, associate and integrate the scattered monitoring points according to the actual spatial distribution and the functional characteristics of the areas where they are located, grasp the internal connections between different areas from a macroscopic perspective, avoid the abnormal analysis errors caused by isolated monitoring points, and assist in judgment through the remaining associated monitoring points, so as to improve the accuracy of abnormal judgment on the basis of improving the efficiency of distributed computing.

[0043] Step S4 further includes: Obtain the status of monitoring points in real time. When the status of a monitoring point changes, obtain the monitoring point whose status has changed. Use the density-based incremental clustering algorithm to update the clustering result according to the monitoring points whose status has changed. Dynamically update the coverage ranges of the edge liaison lines and the edge computing nodes based on the updated clustering result and the preset node load threshold.

[0044] In the monitoring system, the status of monitoring points may change over time, and these changes may be caused by various reasons such as environmental factors, equipment failures, and human interventions. Thus, when the number of monitoring points increases or decreases, the incremental clustering algorithm is used for rapid clustering to update the clustering result. The incremental clustering algorithm can quickly cluster the newly added or changed data without recalculating the entire data set. Then, according to the result adjusted by the incremental clustering, the coverage range of the edge computing nodes and the monitoring points they are responsible for are dynamically updated. For example, if the number of monitoring points in a certain cluster increases (such as after new monitoring points are added) and exceeds the processing capacity range of the current corresponding edge computing node, that is, the preset node load threshold, such as when indicators such as CPU usage rate and memory occupancy reach a certain upper limit, then consider dividing some of the monitoring points in this cluster into the responsible areas of other edge computing nodes with lower loads, or adding a new edge computing node to share the processing pressure; conversely, if the number of monitoring points in a certain cluster decreases (due to demolition or relocation, etc.), and the data volume of the remaining monitoring points makes the resource utilization rate of the current edge computing node too low (such as lower than a certain set lower limit value), then the coverage range of the edge computing node is shrunk, and some of the monitoring points in other clusters it is responsible for are reallocated to ensure that the resources of each edge computing node can be reasonably utilized, and at the same time, the entire edge computing architecture always matches the actual monitoring layout of the community to ensure the efficient processing of data and regional adaptability.

[0045] In this embodiment, step S5 further includes: Obtain the current monitoring data, and based on the current monitoring data, obtain the abnormal monitoring location and abnormal factors according to the abnormal analysis model; construct the community patrol route according to the path and urgency based on the community road data, abnormal monitoring location, and abnormal factors.

[0046] Build the matching relationship between abnormal factors and urgency according to the potential impact degree of abnormal factors on community safety. When the abnormal analysis model outputs abnormal factors, generate the corresponding community patrol route according to the urgency matched by the abnormal factors and the convenience of the abnormal monitoring location, so as to ensure that the abnormal situations with higher urgency can be preferentially investigated while improving the patrol efficiency.

[0047] Among them, use the shortest path algorithm to calculate the shortest path from the patrol starting point to the abnormal monitoring location with the highest urgency, and add this path to the patrol route. Then, calculate the next patrol route in turn according to the urgency order. When there are two abnormal monitoring locations with the same urgency order, plan the route according to the shortest patrol route between them to improve the patrol efficiency.

[0048] In this embodiment, use the abnormal analysis model to obtain the abnormal monitoring location and abnormal factors according to the current monitoring data, and combine the community road data and reasonable shortest path and urgency considerations to construct a community patrol route with higher efficiency, higher urgency and higher priority, so as to improve the response speed and processing efficiency of community safety management and reduce the safety impact of abnormal situations on the community.

[0049] As the second embodiment of this application, a community patrol management system based on multimodal fusion is connected to the community monitoring unit and includes: A data processing unit for extracting features from the monitoring point modal data based on multimodal fusion to obtain monitoring point feature data; an edge computing node constructed according to the clustering result of the monitoring point feature data for constructing an edge abnormal screening model according to historical monitoring data; An overall analysis unit for constructing an edge connection line according to the monitoring area correlation, and outputting an abnormal analysis result based on the output value of the edge abnormal screening model and the edge connection line; A patrol planning unit for outputting a community patrol strategy according to the abnormal analysis result.

[0050] In this embodiment, the data processing unit is connected to the community monitoring unit, the position of the edge computing node is obtained by clustering the monitoring point feature data output by the data processing unit, and the overall analysis unit obtains the overall abnormal analysis result according to the edge abnormal result output by the edge abnormal screening model in the edge computing node and the edge connection line, so as to consider the overall influence relationship while reducing the computing pressure of a single computing unit through distributed computing and ensure the accuracy of abnormal analysis.

[0051] Among them, the edge computing node pre-stores an incremental clustering algorithm and a preset node load threshold. When there is a change in the status of the monitoring point, it is determined whether the edge computing node corresponding to the current clustering cluster can meet the monitoring point addition requirement according to the incremental clustering algorithm and the preset node load threshold. If not, the newly added monitoring point is allocated according to the monitoring area correlation in the edge connection line, and the correlation between this monitoring area and the other monitoring areas, that is, the edge connection line, is updated to adapt to the change of the monitoring layout and ensure the accuracy of the community patrol plan.

[0052] The above specific implementation manners are the preferred implementation manners of the community patrol management method and system based on multi-modal fusion of the present application, and do not limit the specific implementation scope of the present application. The scope of the present application includes but is not limited to this specific implementation manner. All equivalent changes made according to the shape and structure of the present application are within the protection scope of the present application.

Claims

1. A community inspection management method based on multimodal fusion, characterized by: The steps include: S1: Acquire the modal data of the monitoring point, extract the features of the modal data of the monitoring point based on multimodal fusion to obtain the feature data of the monitoring point, use the clustering algorithm to obtain the clustering results based on the feature data of the monitoring point, and build the edge computing node according to the clustering results; S2: Obtain the historical monitoring data corresponding to each edge computing node, and build an edge anomaly screening model based on the inspection anomaly results according to the historical monitoring data; S3: Construct edge contact lines based on the correlation of the monitoring area, and construct an anomaly analysis model based on the edge anomaly screening model and the edge contact lines; S4: Obtain the status of monitoring points in real time, and dynamically adjust the abnormal analysis model using the incremental clustering algorithm based on the changes in the status of monitoring points; S5: Obtain current monitoring data and output community inspection strategies based on the anomaly analysis model and current monitoring data.

2. The community inspection management method based on multimodal fusion according to claim 1, characterized in that: The S1 further comprises: Acquire monitoring point image data, sensor data, and text description data, perform feature extraction on the monitoring point image data, sensor data, and text description data, and obtain an extracted feature set; The principal component analysis method is combined with feature splicing to fuse the extracted feature set and obtain the feature data of the monitoring point; Use the DBSCAN clustering algorithm to cluster the monitoring point feature data and obtain the clustering results; Match edge computing nodes based on clustering results.

3. The community inspection management method based on multimodal fusion as claimed in claim 2, characterized in that: The feature extraction of the monitoring point image data, sensor data and text description data to obtain the extracted feature set includes: Match the image segmentation resolution according to the image complexity of the monitoring point image data, segment the image based on the image segmentation resolution to obtain a segmented image set, and use a convolutional neural network to extract features from the segmented image set to obtain image features; Extract features from sensor data based on time series and sensing correlation to obtain sensing features; Use semantic recognition to extract features from text description data and obtain semantic features; The extracted feature set is obtained by using image features, sensor features and semantic features.

4. The community inspection management method based on multimodal fusion as claimed in claim 3 is characterized by: The step of matching the image segmentation resolution according to the image complexity of the monitoring point image data and segmenting the image based on the image segmentation resolution to obtain the segmented image set comprises: Calculate the information entropy of the monitoring point image data and obtain the image complexity based on the information entropy; Matching an image segmentation resolution corresponding to the complexity of the image based on a preset complexity and segmentation correlation; The image is segmented based on the image segmentation resolution to obtain a segmented image set.

5. The community inspection management method based on multimodal fusion as claimed in claim 4 is characterized by: The method of extracting features from the segmented image set using a convolutional neural network to obtain image features includes: Based on the convolutional neural network structure, a parallel feature recognition stream is constructed according to the segmentation resolution, and a multi-stream parallel convolutional neural network is constructed with the feature recognition stream and the resolution judgment layer; The segmented image set is used as the input of the multi-stream parallel convolutional neural network, which outputs image features.

6. The community inspection management method based on multimodal fusion as claimed in claim 3, characterized in that: The extracting features of the sensor data according to the time series and the sensing correlation to obtain the sensing features includes: Perform trend feature and periodic feature extraction based on the trend change and periodic change of sensor data in time series; Correlation analysis is used to obtain sensing association features between sensors, and sensing features are obtained using sensing association features, trend features, and periodic features.

7. The community inspection management method based on multimodal fusion according to claim 1, characterized in that: The S3 further includes: Construct monitoring area correlation based on the geographical location correlation of monitoring points and the functional correlation of monitoring areas; Obtain the correlation between monitoring points according to the correlation of monitoring areas, and construct edge contact lines based on the correlation between monitoring points; The anomaly analysis model is constructed using the edge anomaly screening model and edge contact lines.

8. The community inspection management method based on multimodal fusion according to claim 1, characterized in that: The S4 further comprises: Get the status of monitoring points in real time. When the status of monitoring points changes, get the monitoring points whose status has changed. The density-based incremental clustering algorithm is used to update the clustering results according to the monitoring points whose status has changed; The edge contact lines and edge computing node coverage are dynamically updated with the updated clustering results and the preset node load threshold.

9. The community inspection management method based on multimodal fusion according to claim 1, characterized in that: The S5 further includes: Obtain current monitoring data, and obtain abnormal monitoring locations and abnormal factors based on the current monitoring data according to the abnormal analysis model; Based on community road data, abnormal monitoring locations, abnormal factors, and path and urgency, community inspection routes are constructed.

10. A community inspection management system based on multimodal fusion, used to implement the method according to any one of claims 1 to 9, characterized in that: include: A data processing unit, used for extracting features from the monitoring point modal data based on multimodal fusion to obtain monitoring point feature data; Edge computing nodes are constructed based on the clustering results of the monitoring point feature data and are used to build edge anomaly screening models based on historical monitoring data; The overall analysis unit is used to construct edge connection lines according to the correlation of the monitoring area, and output anomaly analysis results based on the edge anomaly screening model output value and the edge connection line output; The inspection planning unit is used to output community inspection strategies based on the abnormal analysis results.

Images