Intelligent processing method for public security command based on multi-modal fusion deep learning

Abstract

The application discloses a public safety command intelligent processing method based on multi-modal fusion deep learning, belongs to the technical field of public safety command, and aims to solve the problems of unclear field perception, delayed abnormal early warning and insufficient joint linkage of the existing public safety command. The metadata model supporting mode adaptation and relationship customization is constructed, multi-modal and billion-level data convergence fusion is realized, cross-domain and cross-layer streaming media processing is cooperated with video content analysis and mining, the smoothness of media transmission is ensured, the accuracy of abnormal event time section and space section positioning in monitoring video is improved, and the identification accuracy under complex conditions is ensured. Based on the multi-modal information fusion of expert knowledge model and deep learning model in the decision-making stage, the research and judgment accuracy is greatly improved, and the practical ability of the professional research and judgment model is supported. The application is suitable for the public safety command intelligent processing method based on multi-modal fusion deep learning.

Description

Technical Field

[0001] This invention belongs to the field of public safety command technology, specifically relating to an intelligent processing method for public safety command based on multimodal fusion deep learning. Background Technology

[0002] Command systems are crucial platforms for governments to implement security control, emergency response, and resource allocation. With social development and changes in the public safety landscape, the need for precise command and rapid response from public security agencies and other organizations in maintaining social order and dealing with emergencies is becoming increasingly urgent.

[0003] Traditional command systems suffer from problems such as unclear on-site perception, delayed anomaly warnings, insufficient joint coordination, and a lack of professional analysis tools. They can no longer meet the current overall requirements for precise command, scientific use of police force, and rapid response, making it urgent to develop and build new intelligent command systems. Summary of the Invention

[0004] The purpose of this invention is to provide an intelligent processing method for public safety command based on multimodal fusion deep learning, which solves the problems of unclear on-site perception, delayed anomaly warning, and insufficient joint action in existing public safety command systems.

[0005] The technical solution adopted in this invention is as follows:

[0006] A public safety command intelligent processing method based on multimodal fusion deep learning is characterized by the following steps:

[0007] (1) For multimodal datasets, construct a metadata model that supports modality adaptation and customizable relationships to extract and aggregate various types of raw heterogeneous data;

[0008] (2) Construct a dataset quality assessment model for large-scale full-sample data sources and perform automated quality assessment of datasets;

[0009] (3) Construct a spatiotemporal data consistency analysis model to identify and determine the identity of cross-modal target entities and form the true probability distribution of target entities;

[0010] (4) A unified data representation model P=(O,R,G) based on knowledge graphs is used to form data fusion elements based on statistical machine learning, and multimodal data fusion is achieved through dynamic association and aggregation of data mapping.

[0011] (5) Construct a unified metadata management system to provide a unified data hierarchical classification management system for different types of data sources, and realize unified data storage and services;

[0012] (6) Cross-domain and cross-layer streaming media processing: Based on the free energy principle in human visual perception, modal separation and modal fusion are performed according to the characteristics of streaming media components. The smoothness of media transmission is ensured through a global visual perception streaming media coding model, a timestamp-based buffer mechanism optimization, and a variable-length ring buffer design.

[0013] (7) Video content analysis and mining: Based on the variational autoencoder, more meaningful features are extracted than shallow artificial features. The degree of abnormality of test samples is scored by sparse reconstruction of implicit spatial information, thereby improving the accuracy of locating abnormal events in time and space segments in surveillance videos.

[0014] (8) The system adopts a combination of auto-learning and deep reinforcement learning to select the most discriminative features by considering the order and weight of training samples, and combines the temporal and spatial information dual-stream structure to perform pedestrian re-identification, thereby improving the recognition accuracy under different lighting conditions; and realizes abnormal event detection and target association retrieval in complex command scenarios.

[0015] (9) Multimodal data is analyzed by a feature information fusion device based on support, confidence, lift and credibility indices, and a mapping from feature multimodal information to training multimodal information is constructed, so that the salient features of multimodal information can be interacted and verified.

[0016] (10) A deep learning model is constructed by continuously training a deep neural network to achieve iterative optimization of the network model and parameters; and multimodal information fusion of expert knowledge model and deep learning model in the decision-making stage is achieved based on dynamic weight allocation factor.

[0017] (11) Through the decision information fusion device, generate a practical business model, provide the optimal decision-making scheme for practical application, and complete public safety command.

[0018] Furthermore, in step (1), the specific steps for extracting and aggregating various types of original heterogeneous data are as follows:

[0019] Step a: Extract metadata from the original heterogeneous data;

[0020] Step b: Define the relationships between the extracted metadata and form the corresponding metadata model;

[0021] Step c: Store the resulting metadata model in the database in a graphical format.

[0022] Furthermore, in step (6), the streaming media component characteristics include static and dynamic information components, audio and video components, and high and low frequency components.

[0023] Furthermore, in step (11), the practical business model includes feature profiling, situational awareness, anomaly warning, predictive analysis, trajectory analysis, investigation and control, and prevention and control layers.

[0024] In summary, due to the adoption of the above technical solution, the beneficial effects of the present invention are:

[0025] 1. In this invention, by constructing a metadata model that supports modal adaptation and customizable relationships, multimodal, billion-level data aggregation and fusion is achieved. Based on customizable relationships, unified hierarchical and classification management of data is realized. Through a dataset quality assessment model, automated quality assessment of datasets is supported. Through a spatiotemporal data consistency analysis model and a unified data expression model based on knowledge graphs, the correlation of data is improved. Through the relationships formed in the process of data generation, processing and fusion, circulation and circulation to final destruction, a hierarchical and classification management system for data is constructed, realizing unified data storage and service, and solving the problem of weak data management.

[0026] 2. In this invention, in response to the challenges of cross-domain video sharing and scheduling in complex network scenarios, based on the principle of free energy in human visual perception, modal separation and modal fusion are performed according to the characteristics of streaming media components. Through a global visual perception streaming media encoding model, optimization of the timestamp-based buffer mechanism, and design of a variable-length circular buffer, the smoothness of media transmission is ensured, the video compression ratio is improved by about 20%, and the average stuttering rate is reduced by more than 15%.

[0027] 3. In this invention, a variational autoencoder is used to extract features that are more meaningful than shallow artificial features. The anomaly degree of the test sample is scored by sparse reconstruction using latent spatial information, thereby improving the accuracy of locating abnormal events in time and space segments in surveillance videos. The invention also employs a combination of auto-learning and deep reinforcement learning to consider the order and weight of training samples to select the most discriminative features. Furthermore, a dual-stream structure of temporal and spatial information is used for pedestrian re-identification, thereby improving the recognition accuracy under different lighting conditions and enabling abnormal event detection and target association retrieval in complex command scenarios.

[0028] 4. This invention starts from multiple business scenarios and changes the single-sensor or single-modal data analysis method. It starts from the analysis of multimodal data and achieves multimodal information fusion of expert knowledge models and deep learning models in the decision-making stage by fusing feature information and decision information. This is based on dynamic weight allocation factors, which greatly improves the accuracy of judgment and supports the practical capabilities of hundreds of professional judgment models in seven categories, including situational awareness, anomaly early warning and intelligent prevention and control circles. Attached Figure Description

[0029] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly described below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort, wherein:

[0030] Figure 1 This is a flowchart illustrating the multimodal data aggregation and fusion process of the present invention.

[0031] Figure 2 This is a flowchart illustrating the practical application of the deep learning model of this invention. Detailed Implementation

[0032] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.

[0033] Therefore, the following detailed description of the embodiments of the invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the invention without inventive effort are within the scope of protection of the invention.

[0034] It should be noted that the labels and letters in the following figures represent similar items, therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.

[0035] In the description of this invention, it should be noted that the terms "center," "upper," "lower," "left," "right," "vertical," "horizontal," "inner," and "outer," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings, or the orientation or positional relationship commonly used when the product of this invention is in use. They are only used for the purpose of simplifying the description of this invention and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on this invention. In addition, the terms "first," "second," and "third," etc., are only used to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0036] Furthermore, terms such as "horizontal" and "vertical" do not imply that components must be absolutely horizontal or suspended, but rather that they can be slightly tilted. For example, "horizontal" simply means that its direction is more horizontal than "vertical," not that the structure must be completely horizontal, but can be slightly tilted.

[0037] In the description of this invention, it should also be noted that, unless otherwise explicitly specified and limited, the terms "set," "install," "connect," and "link" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art can understand the specific meaning of the above terms in this invention based on the specific circumstances.

[0038] A public safety command intelligent processing method based on multimodal fusion deep learning is characterized by the following steps:

[0039] (1) For multimodal datasets, construct a metadata model that supports modality adaptation and customizable relationships to extract and aggregate various types of raw heterogeneous data;

[0040] (2) Construct a dataset quality assessment model for large-scale full-sample data sources and perform automated quality assessment of datasets;

[0041] (3) Construct a spatiotemporal data consistency analysis model to identify and determine the identity of cross-modal target entities and form the true probability distribution of target entities;

[0042] (4) A unified data representation model P=(O,R,G) based on knowledge graphs is used to form data fusion elements based on statistical machine learning, and multimodal data fusion is achieved through dynamic association and aggregation of data mapping.

[0043] (5) Construct a unified metadata management system to provide a unified data hierarchical classification management system for different types of data sources, and realize unified data storage and services;

[0044] (6) Cross-domain and cross-layer streaming media processing: Based on the free energy principle in human visual perception, modal separation and modal fusion are performed according to the characteristics of streaming media components. The smoothness of media transmission is ensured through a global visual perception streaming media coding model, a timestamp-based buffer mechanism optimization, and a variable-length ring buffer design.

[0045] (7) Video content analysis and mining: Based on the variational autoencoder, more meaningful features are extracted than shallow artificial features. The degree of abnormality of test samples is scored by sparse reconstruction of implicit spatial information, thereby improving the accuracy of locating abnormal events in time and space segments in surveillance videos.

[0046] (8) The system adopts a combination of auto-learning and deep reinforcement learning to select the most discriminative features by considering the order and weight of training samples, and combines the temporal and spatial information dual-stream structure to perform pedestrian re-identification, thereby improving the recognition accuracy under different lighting conditions; and realizes abnormal event detection and target association retrieval in complex command scenarios.

[0047] (9) Multimodal data is analyzed by a feature information fusion device based on support, confidence, lift and credibility indices, and a mapping from feature multimodal information to training multimodal information is constructed, so that the salient features of multimodal information can be interacted and verified.

[0048] (10) A deep learning model is constructed by continuously training a deep neural network to achieve iterative optimization of the network model and parameters; and multimodal information fusion of expert knowledge model and deep learning model in the decision-making stage is achieved based on dynamic weight allocation factor.

[0049] (11) Through the decision information fusion device, generate a practical business model, provide the optimal decision-making scheme for practical application, and complete public safety command.

[0050] Furthermore, in step (1), the specific steps for extracting and aggregating various types of original heterogeneous data are as follows:

[0051] Step a: Extract metadata from the original heterogeneous data;

[0052] Step b: Define the relationships between the extracted metadata and form the corresponding metadata model;

[0053] Step c: Store the resulting metadata model in the database in a graphical format.

[0054] Furthermore, in step (6), the streaming media component characteristics include static and dynamic information components, audio and video components, and high and low frequency components.

[0055] Furthermore, in step (11), the practical business model includes feature profiling, situational awareness, anomaly warning, predictive analysis, trajectory analysis, investigation and control, and prevention and control layers.

[0056] In its implementation, this invention achieves the convergence and fusion of multimodal, billion-level data by constructing a metadata model that supports modal adaptation and customizable relationships. Based on customizable relationships, it realizes unified hierarchical and classification management of data. Through a dataset quality assessment model, it supports automated quality assessment of datasets. Through a spatiotemporal data consistency analysis model and a unified data representation model based on knowledge graphs, it improves the relevance of data. By constructing a hierarchical and classification management system for data through the relationships formed during data generation, processing, fusion, circulation, and eventual disappearance, it achieves unified data storage and services, thus solving the problem of weak data management.

[0057] To address the challenges of cross-domain video sharing and scheduling in complex network scenarios, this paper proposes a method based on the principle of free energy in human visual perception. It performs modal separation and modal fusion according to the characteristics of streaming media components. Through a global visual perception streaming media coding model, optimization of the timestamp-based buffer mechanism, and design of a variable-length circular buffer, the smoothness of media transmission is ensured. The video compression ratio is improved by about 20%, and the average stuttering rate is reduced by more than 15%.

[0058] Based on variational autoencoders, more meaningful features than shallow artificial features are extracted. The degree of anomaly of test samples is scored by sparse reconstruction using latent spatial information, which improves the accuracy of locating abnormal events in time and space segments in surveillance videos. The system adopts a combination of auto-learning and deep reinforcement learning to select the most discriminative features by considering the order and weight of training samples. Pedestrian re-identification is performed by combining a dual-stream structure of temporal and spatial information, which improves the recognition accuracy under different lighting conditions and enables abnormal event detection and target association retrieval in complex command scenarios.

[0059] Starting from multiple business scenarios, it changes the single-sensor or single-modal data analysis method and starts from the analysis perspective of multimodal data. By fusing feature information and decision information, it realizes the fusion of multimodal information of expert knowledge models and deep learning models in the decision-making stage based on dynamic weight allocation factors, which greatly improves the accuracy of judgment and supports the practical capabilities of hundreds of professional judgment models in seven categories, including situational awareness, anomaly early warning and intelligent prevention and control circles.

[0060] Example 1

[0061] A public safety command intelligent processing method based on multimodal fusion deep learning is characterized by the following steps:

[0062] (1) For multimodal datasets, construct a metadata model that supports modality adaptation and customizable relationships to extract and aggregate various types of raw heterogeneous data;

[0063] (2) Construct a dataset quality assessment model for large-scale full-sample data sources and perform automated quality assessment of datasets;

[0064] (3) Construct a spatiotemporal data consistency analysis model to identify and determine the identity of cross-modal target entities and form the true probability distribution of target entities;

[0065] (4) A unified data representation model P=(O,R,G) based on knowledge graphs is used to form data fusion elements based on statistical machine learning, and multimodal data fusion is achieved through dynamic association and aggregation of data mapping.

[0066] (5) Construct a unified metadata management system to provide a unified data hierarchical classification management system for different types of data sources, and realize unified data storage and services;

[0067] (6) Cross-domain and cross-layer streaming media processing: Based on the free energy principle in human visual perception, modal separation and modal fusion are performed according to the characteristics of streaming media components. The smoothness of media transmission is ensured through a global visual perception streaming media coding model, a timestamp-based buffer mechanism optimization, and a variable-length ring buffer design.

[0068] (7) Video content analysis and mining: Based on the variational autoencoder, more meaningful features are extracted than shallow artificial features. The degree of abnormality of test samples is scored by sparse reconstruction of implicit spatial information, thereby improving the accuracy of locating abnormal events in time and space segments in surveillance videos.

[0069] (8) The system adopts a combination of auto-learning and deep reinforcement learning to select the most discriminative features by considering the order and weight of training samples, and combines the temporal and spatial information dual-stream structure to perform pedestrian re-identification, thereby improving the recognition accuracy under different lighting conditions; and realizes abnormal event detection and target association retrieval in complex command scenarios.

[0070] (9) Multimodal data is analyzed by a feature information fusion device based on support, confidence, lift and credibility indices, and a mapping from feature multimodal information to training multimodal information is constructed, so that the salient features of multimodal information can be interacted and verified.

[0071] (10) A deep learning model is constructed by continuously training a deep neural network to achieve iterative optimization of the network model and parameters; and multimodal information fusion of expert knowledge model and deep learning model in the decision-making stage is achieved based on dynamic weight allocation factor.

[0072] (11) Through the decision information fusion device, generate a practical business model, provide the optimal decision-making scheme for practical application, and complete public safety command.

[0073] Example 2

[0074] Based on Example 1, the specific steps for extracting and aggregating various types of original heterogeneous data in step (1) are as follows:

[0075] Step a: Extract metadata from the original heterogeneous data;

[0076] Step b: Define the relationships between the extracted metadata and form the corresponding metadata model;

[0077] Step c: Store the resulting metadata model in the database in a graphical format.

[0078] Example 3

[0079] Based on the above embodiments, in step (6), the streaming media component features include dynamic and static information components, audio and video components, and high and low frequency components.

[0080] Example 4

[0081] Based on the above embodiments, in step (11), the practical business model includes feature profiling, situational awareness, anomaly warning, predictive analysis, trajectory analysis, investigation and control, and prevention and control layers.

[0082] The above description constitutes an embodiment of the present invention. The foregoing descriptions are preferred embodiments of the present invention. Unless there is a clear contradiction or a prerequisite for a particular preferred embodiment, the preferred embodiments can be arbitrarily combined and used. The embodiments and specific parameters described are merely for clearly illustrating the verification process of the invention and are not intended to limit the scope of patent protection of the present invention. The scope of patent protection of the present invention is still determined by its claims. Similarly, any equivalent structural changes made based on the description and drawings of the present invention should also be included within the scope of protection of the present invention.

Claims

1. A public safety command intelligent processing method based on multimodal fusion deep learning, characterized in that, Includes the following steps: (1) For multimodal datasets, construct a metadata model that supports modality adaptation and customizable relationships to extract and aggregate various types of original heterogeneous data; (2) Construct a dataset quality assessment model for large-scale full-sample data sources and perform automated quality assessment of the datasets; (3) Construct a spatiotemporal data consistency analysis model to identify and determine the identity of cross-modal target entities and form the true probability distribution of the target entities; (4) Based on the knowledge graph, a unified data representation model P = (O, R, G) is formed to create data fusion elements based on statistical machine learning. Multimodal data fusion is achieved through dynamic association and aggregation of data mapping. (5) Construct a unified metadata management system to provide a unified data hierarchical classification management system for different types of data sources, and realize unified data storage and services; (6) Cross-domain and cross-layer streaming media processing: Based on the free energy principle in human visual perception, modal separation and modal fusion are performed according to the characteristics of streaming media components. The smoothness of media transmission is ensured by a global visual perception streaming media coding model, a timestamp-based buffer mechanism optimization, and a variable-length ring buffer design. The characteristics of streaming media components include dynamic and static information components, audio and video components, and high and low frequency components. (7) Video content analysis and mining: Based on the variational autoencoder, more meaningful features are extracted than shallow artificial features. The degree of anomaly of the test sample is scored by sparse reconstruction of implicit spatial information, thereby improving the accuracy of locating abnormal events in time and space segments in the surveillance video. (8) The system adopts self-step learning and deep reinforcement learning to consider the order and weight of training samples to select the features with the highest discriminative rate, and combines the temporal and spatial information dual-stream structure to perform pedestrian re-identification, thereby improving the recognition accuracy under different lighting conditions. To enable abnormal event detection and target association retrieval in complex command scenarios; (9) Multimodal data is analyzed by a feature information fusion device based on support, confidence, lift and credibility indices to construct a mapping from feature multimodal information to training multimodal information, so that the salient features of multimodal information can be interacted and verified. (10) A deep learning model is constructed by continuously training deep neural networks to achieve iterative optimization of network models and parameters; Multimodal information fusion between expert knowledge models and deep learning models in the decision-making stage is achieved based on dynamic weight allocation factors; (11) Through the decision information fusion device, generate a practical business model, provide the best practical decision-making scheme, and complete the command of public security; the practical business model includes feature profiling, situational awareness, abnormal early warning, predictive analysis, trajectory analysis, investigation and control, and prevention and control layers.

2. The intelligent processing method for public safety command based on multimodal fusion deep learning as described in claim 1, characterized in that, In step (1), the specific steps for extracting and aggregating various types of original heterogeneous data are as follows: Step a: Extract metadata from the original heterogeneous data; Step b: Define the relationships between the extracted metadata and form the corresponding metadata model; Step c: Store the resulting metadata model in the database in a graphical format.

Images