Integrated monitoring system and method based on multimodal information fusion

Abstract

The present invention discloses an integrated monitoring system and method based on multimodal information fusion, which relates to the field of intelligent transportation technology, including: obtaining an information collection integrated machine to collect multimodal information flow for monitoring a traffic area; performing parallel preprocessing on the multimodal information flow for monitoring a traffic area to obtain a multimodal information flow for monitoring a standard traffic area, performing feature extraction in sequence to obtain a multimodal traffic area feature set, performing abnormal feature extraction to obtain an abnormal multimodal traffic feature set; performing mapping, extraction and fusion on the multimodal traffic area feature set to generate an abnormal multimodal traffic feature matrix, performing strategy analysis to obtain traffic coordination control strategy parameters, and coordinating operation and maintenance management of the target traffic area. The present invention solves the technical problems of incomplete information, slow response speed and low anomaly detection accuracy in the prior art when performing traffic monitoring, and achieves the technical effect of improving the efficiency of traffic abnormality monitoring and management.

Description

Technical Field

[0001] The present invention relates to the field of intelligent transportation technology, and in particular to an integrated monitoring system and method based on multimodal information fusion. Background Art

[0002] With the acceleration of urbanization and the rapid growth of transportation demand, the application of intelligent traffic management systems in modern urban transportation has become more and more common. However, with the complexity of the transportation network and the sharp increase in the number of vehicles, traditional traffic monitoring and management methods are unable to cope with the increasingly complex traffic conditions. Traditional traffic monitoring systems mainly rely on single-modal data sources, such as video monitoring or GPS positioning. These methods have problems such as single information, limited coverage, and slow response speed, which makes it difficult to meet the high standards of modern traffic management for real-time, accuracy, and comprehensiveness. Summary of the invention

[0003] The present application provides an integrated monitoring system and method based on multimodal information fusion, which is used to solve the technical problems of incomplete information, slow response speed and low anomaly detection accuracy in the existing technology when performing traffic monitoring.

[0004] In view of the above problems, the present application provides an integrated monitoring system and method based on multimodal information fusion.

[0005] In a first aspect of the present application, an integrated monitoring system based on multimodal information fusion is provided, the system comprising:

[0006] An information acquisition module, wherein the information acquisition module acquires an information acquisition integrated machine, wherein the information acquisition integrated machine integrates a camera, a GPS locator and an intercom communication module, and synchronously acquires a multimodal information flow for monitoring a traffic area through the information acquisition integrated machine; a standard information acquisition module, wherein the standard information acquisition module respectively performs preprocessing step analysis on the multimodal information flow for monitoring a traffic area, obtains a multimodal data preprocessing program, and performs parallel preprocessing on the multimodal information flow for monitoring a traffic area based on the multimodal data preprocessing program to obtain a standard multimodal information flow for monitoring a traffic area; a feature extraction module, wherein the feature extraction module performs preprocessing analysis on the standard traffic area Monitor the multimodal information flow and extract features in sequence to obtain a multimodal traffic area feature set, perform abnormal feature extraction based on the multimodal traffic area feature set, and obtain an abnormal multimodal traffic feature set; a mapping extraction fusion module, the mapping extraction fusion module performs mapping extraction fusion on the multimodal traffic area feature set based on the abnormal multimodal traffic feature set to generate an abnormal multimodal traffic feature matrix; a coordinated operation and maintenance control module, the coordinated operation and maintenance control module performs strategy analysis on the abnormal multimodal traffic feature matrix to obtain traffic coordination control strategy parameters, and performs coordinated operation and maintenance control on the target traffic area based on the traffic coordination control strategy parameters.

[0007] The second aspect of the present application provides an integrated monitoring method based on multimodal information fusion, the method comprising:

[0008] An information collection integrated machine is obtained, wherein the information collection integrated machine integrates a camera, a GPS locator and an intercom communication module, and a multimodal information flow for monitoring a traffic area is synchronously collected through the information collection integrated machine; preprocessing steps are respectively analyzed for the multimodal information flow for monitoring the traffic area to obtain a multimodal data preprocessing program, and the multimodal information flow for monitoring the traffic area is parallelly preprocessed based on the multimodal data preprocessing program to obtain a standard traffic area monitoring multimodal information flow; feature extraction is performed on the standard traffic area monitoring multimodal information flow in sequence to obtain a multimodal traffic area feature set, and abnormal feature extraction is performed based on the multimodal traffic area feature set to obtain an abnormal multimodal traffic feature set; mapping, extraction and fusion are performed on the multimodal traffic area feature set based on the abnormal multimodal traffic feature set to generate an abnormal multimodal traffic feature matrix; strategy analysis is performed on the abnormal multimodal traffic feature matrix to obtain traffic coordination control strategy parameters, and coordinated operation and maintenance management of the target traffic area is performed based on the traffic coordination control strategy parameters.

[0009] One or more technical solutions provided in this application have at least the following technical effects or advantages:

[0010] The present application obtains an all-in-one information collection machine, which integrates a camera, a GPS locator and an intercom communication module, and synchronously collects and obtains a multimodal information flow for monitoring a traffic area through the all-in-one information collection machine; performs preprocessing step analysis on the multimodal information flow for monitoring a traffic area respectively to obtain a multimodal data preprocessing program, and performs parallel preprocessing on the multimodal information flow for monitoring a traffic area based on the multimodal data preprocessing program to obtain a multimodal information flow for monitoring a standard traffic area; performs feature extraction on the multimodal information flow for monitoring a standard traffic area in turn to obtain a multimodal traffic area feature set, and performs abnormal feature extraction based on the multimodal traffic area feature set to obtain an abnormal multimodal traffic feature set; maps, extracts and fuses the multimodal traffic area feature set based on the abnormal multimodal traffic feature set to generate an abnormal multimodal traffic feature matrix; performs strategy analysis on the abnormal multimodal traffic feature matrix to obtain traffic coordination control strategy parameters, and coordinates operation and maintenance management of the target traffic area based on the traffic coordination control strategy parameters. The present invention solves the technical problems of incomplete information, slow response speed and low anomaly detection accuracy in the prior art when conducting traffic monitoring. The multimodal information flow of the traffic area is synchronously collected by an information collection all-in-one machine, pre-processed by a standard information acquisition module, the traffic characteristics are extracted by a feature extraction module, the mapping extraction fusion module generates an abnormal feature matrix, and finally the operation and maintenance control module is coordinated to perform strategy analysis and optimization, so as to achieve the technical effect of improving the efficiency of traffic anomaly monitoring and management. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0012] Figure 1 A schematic diagram of the structure of an integrated monitoring system based on multimodal information fusion provided in an embodiment of the present application;

[0013] Figure 2 A schematic flow chart of an integrated monitoring method based on multimodal information fusion provided in an embodiment of the present application.

[0014] Explanation of the reference numerals: information collection module 11 , standard information acquisition module 12 , feature extraction module 13 , mapping extraction and fusion module 14 , coordination and operation and maintenance control module 15 . DETAILED DESCRIPTION

[0015] The present application provides an integrated monitoring system and method based on multimodal information fusion, aiming to solve the technical problems of incomplete information, slow response speed and low anomaly detection accuracy in the prior art when conducting traffic monitoring. The multimodal information flow of the traffic area is synchronously collected by an information collection all-in-one machine, pre-processed by a standard information acquisition module, traffic features extracted by a feature extraction module, an anomaly feature matrix generated by a mapping extraction and fusion module, and finally coordinated with the operation and maintenance control module to perform strategy analysis and optimization, so as to achieve the technical effect of improving the efficiency of traffic anomaly detection and management.

[0016] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0017] It should be noted that any variations of the terms "include" and "have" are intended to cover non-exclusive inclusions. For example, a process, method, system, product or server that includes a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or modules that are not explicitly listed or inherent to these processes, methods, products or devices.

[0018] Embodiment 1, as Figure 1 As shown, the embodiment of the present application provides an integrated monitoring system based on multimodal information fusion, the system comprising:

[0019] The information collection module 11 obtains an information collection integrated machine, which integrates a camera, a GPS locator and an intercom communication module. The multimodal information flow of traffic area monitoring is obtained through synchronous collection by the information collection integrated machine.

[0020] In an embodiment of the present application, the information acquisition module is first connected to the information acquisition integrated machine, which is pre-deployed and includes a camera, a GPS locator and an intercom communication module. Among them, the camera is responsible for acquiring visual information of the traffic area in real time, such as vehicles, pedestrians and traffic signs. The camera uses high temperature resistant, waterproof and dustproof hardware to ensure that it can still work stably under adverse weather conditions such as rain, snow and high temperature. The GPS locator provides the precise geographic location of the acquisition device in real time by receiving satellite signals, and this location information is recorded synchronously with the image data. In addition, the intercom communication module is responsible for transmitting the collected multimodal information stream to the data center or command center in real time.

[0021] Through the all-in-one information collection machine, the information collection module synchronously acquires and transmits multimodal information streams for traffic area monitoring including images, location information and communication data.

[0022] The standard information acquisition module 12 performs preprocessing step analysis on the multimodal information flow of traffic area monitoring respectively to obtain a multimodal data preprocessing program, and performs parallel preprocessing on the multimodal information flow of traffic area monitoring based on the multimodal data preprocessing program to obtain a standard traffic area monitoring multimodal information flow.

[0023] In an embodiment of the present application, the standard information acquisition module first performs a preprocessing step analysis on the multimodal information stream monitored in the traffic area, and identifies different types of data such as video streams, GPS positioning data, and voice data. For each data type, a special preprocessing scheme is designed using existing technology to generate a multimodal data preprocessing program. Specifically, in terms of image processing, a video compression algorithm such as H.264 or H.265 is used to compress the video stream captured by the camera to reduce the data volume and improve transmission efficiency. Then, denoising techniques such as median filtering or Gaussian filtering are used to eliminate noise in the video and enhance the clarity of the image. Subsequently, the contrast of the image is enhanced by contrast stretching or histogram equalization technology to highlight the details, and finally the Canny edge detection algorithm is used to detect the edge of the image to identify and mark the outlines of objects in the image.

[0024] In terms of positioning data processing, the GPS positioning data is first verified using anomaly detection algorithms, such as the mean-based outlier detection method, to remove outliers and ensure the accuracy of the positioning data. Then, Kalman filtering or weighted average filtering technology is applied to smooth the data to reduce the impact of noise and enhance the reliability of the data.

[0025] In terms of speech processing, common speech coding standards such as G.711 or AAC are used to encode and compress speech data to reduce the data volume. The speech data is then filtered through digital filters such as low-pass filters or band-pass filters to remove background noise while retaining sufficient sound quality for subsequent analysis.

[0026] After generating the multimodal data preprocessing program, the standard information acquisition module performs parallel preprocessing on the multimodal information streams for traffic area monitoring. Parallel preprocessing means that the corresponding processing operations are performed simultaneously on different data channels, thereby improving the overall processing efficiency. The video stream is first compressed, and then denoised, contrast enhanced, and edge detected in sequence; the GPS positioning data is first checked for outliers in the parallel channel, and then filtered and smoothed; the voice data is encoded and compressed, and then immediately enters the filtering channel for noise processing. Through these preprocessing steps, all data streams are standardized and integrated into a consistent format to obtain a standard traffic area monitoring multimodal information stream.

[0027] The feature extraction module 13 performs feature extraction on the multimodal information flow monitored in the standard traffic area in sequence to obtain a multimodal traffic area feature set, and performs abnormal feature extraction based on the multimodal traffic area feature set to obtain an abnormal multimodal traffic feature set.

[0028] In an embodiment of the present application, first, the feature extraction module divides the standard traffic area monitoring multimodal information stream into three parts: image information stream, voice information stream and position information stream through data diversion. For the image information stream, the YOLO algorithm is used to perform target detection and bounding box marking to obtain a traffic area detection target set. Then, behavioral features are extracted from these detection targets to generate a target multi-dimensional behavioral feature vector set. At the same time, feature extraction is performed on the voice information stream and the position information stream to obtain traffic event feature information and traffic location feature sets, respectively. Finally, based on these extracted features, a multimodal traffic area feature set is generated.

[0029] Next, based on the multimodal traffic area feature set, abnormal feature extraction is performed. First, the abnormal state of the detection target is defined, and multimodal data crawling and abnormal clustering identification are performed based on this rule to generate a multimodal abnormal clustering dataset. Based on this dataset, abnormal feature classification training is performed to build a multimodal target abnormal feature classifier. Finally, the classifier is used to identify abnormal features of the multimodal traffic area feature set to obtain an abnormal multimodal traffic feature set.

[0030] Furthermore, in the system provided in the embodiment of the application, the feature extraction module 13 is also used for:

[0031] The standard traffic area monitoring multimodal information stream is subjected to data diversion to obtain a traffic area image information stream, a traffic area voice information stream and a traffic area location information stream; the traffic area image information stream is subjected to target detection and bounding box marking by using the YOLO algorithm to obtain a traffic area detection target set; the traffic area detection target set is subjected to behavioral feature extraction and identification in turn to obtain a target multidimensional behavioral feature vector set; feature extraction is performed on the traffic area voice information stream and the traffic area location information stream respectively to obtain traffic event feature information and a traffic location feature set; based on the target multidimensional behavior feature vector set, the traffic event feature information and the traffic location feature set, the multimodal traffic area feature set is obtained.

[0032] In the embodiment of the present application, firstly, data diversion is performed on the standard traffic area monitoring multimodal information stream to separate the traffic area image information stream, the traffic area voice information stream and the traffic area location information stream.

[0033] Next, the YOLO algorithm is used to detect targets and mark bounding boxes on the traffic area image information stream. YOLO is a deep learning-based target detection technology that divides the image into multiple grids, each of which is responsible for detecting the presence of a target and predicting the bounding box position and category of the target. In this process, the YOLO model is trained on a large number of annotated images through a convolutional neural network to learn how to quickly and accurately identify targets in images and generate bounding boxes. Through continuous training, it can eventually quickly identify multiple targets in a single image. Using the YOLO algorithm, a corresponding bounding box mark is generated for each detected target, including information such as the target's location, size, and category, to form a set of traffic area detection targets.

[0034] After that, behavioral features are extracted for each target in the traffic area detection target set. Specifically, visual features such as shape, size and color of each target are extracted. Shape features are realized through image contour analysis technology, such as shape feature extraction method based on Fourier descriptor; size features are obtained by calculating the size of the bounding box; color features are extracted using the color histogram method. In addition, the position, direction and posture information of the target in the image are extracted. The position is directly represented by the coordinates of the center point of the bounding box; the direction is estimated by the motion trajectory of the target or the direction of the long side of the bounding box; the posture information is obtained through a posture estimation algorithm, such as OpenPose. The extracted features, including shape, size, color, position, direction and posture are combined into a multi-dimensional behavioral feature vector to comprehensively describe the behavioral characteristics of the target and form a set of multi-dimensional behavioral feature vectors of the target.

[0035] At the same time, feature extraction is performed on the traffic area voice information stream and the traffic area location information stream respectively. The feature extraction of the traffic area voice information stream is performed through speech feature extraction technology, such as Mel frequency cepstral coefficient, to obtain traffic event feature information. The trajectory analysis of the traffic area location information stream is performed through Kalman filtering and clustering-based trajectory pattern recognition to obtain the traffic location feature set.

[0036] Finally, the target multi-dimensional behavior feature vector set, traffic event feature information and traffic location feature set are integrated, and the features are compressed through dimensionality reduction algorithms such as principal component analysis to generate a multimodal traffic area feature set.

[0037] Furthermore, in the system provided in the embodiment of the application, the feature extraction module 13 is also used for:

[0038] The traffic area voice information stream is converted into a traffic area text information stream, and the traffic area text information stream is subjected to word segmentation and keyword recognition according to a traffic event text library to obtain traffic area keyword information; context sentiment analysis is performed on the traffic area text information stream based on the traffic area keyword information to obtain traffic area sentiment feature information; information fusion and event classification are performed on the traffic area keyword information and the traffic area sentiment feature information to obtain the traffic event feature information; associated position extraction and position trajectory analysis are performed on the traffic area position information stream to obtain the traffic position feature set.

[0039] In the embodiment of the present application, the traffic area voice information flow is first converted into a text information flow by automatic speech recognition technology. Common ASR technologies include models based on CTC or Transformer architectures of deep learning, such as Google's Speech-to-Text API. Through these models, voice information in the traffic area, such as sirens, driver conversations, etc., is transcribed into a traffic area text information flow. Next, the converted traffic area text information flow is segmented, and the text is segmented by using tools such as Jieba segmentation to segment the text and divide continuous sentences into independent words. Then, keyword recognition is performed based on algorithms such as TF-IDF or TextRank to extract keywords such as "accident", "collision", "congestion" and other keywords that reflect traffic anomalies. At the same time, emergency words such as "help", "emergency", "alarm" and other emergency words for priority processing, as well as words such as "speed limit", "no passage", "construction" and other auxiliary drivers and traffic managers to understand road conditions information, are identified. Through this step, traffic area keyword information is generated.

[0040] Next, we conduct contextual sentiment analysis on the text information flow of the traffic area based on the traffic area keyword information. During the analysis, we use a pre-trained sentiment analysis model. The sentiment analysis model is trained using a large-scale text dataset with sentiment labels. These datasets contain text records labeled with positive, negative, or neutral sentiment labels. In the field of transportation, data includes traffic accident reports, complaint records, public feedback, etc., and these texts are annotated to reflect positive, negative, or neutral sentiment. Through the analysis of the sentiment analysis model, the sentiment tendency of the context in which the keywords are located is judged, and the sentiment feature information of the traffic area is obtained. For example, when an "accident" is identified, the context is used to determine whether the event actually occurred or is just a potential risk reminder. Through this step, the sentiment feature information of the traffic area is generated.

[0041] Next, the traffic area keyword information is fused with the sentiment feature information, and event classification is performed. This fusion process generates a comprehensive feature vector by splicing the keyword and sentiment feature vectors. Subsequently, the event classification model is used to classify the fused features into events, and the event types include "accident alarm", "traffic congestion", "road condition change", etc. After this step, traffic event feature information is generated. Among them, the event classification model is pre-trained. The support vector machine is trained using historical traffic event records. These records are manually classified into different event types, such as accidents, congestion, road condition changes, etc., and are equipped with corresponding description texts and related information. Through training, the event classification model can generate traffic event feature information, that is, accurately classify events according to keywords and sentiment information.

[0042] At the same time, the GPS data of the traffic area is processed to extract location information such as latitude and longitude, speed, and direction. The Kalman filter technology is used to smooth these data, eliminate noise, and ensure the accuracy of the location information. Then, trajectory analysis technology, such as the DBSCAN clustering algorithm, is used to analyze the vehicle's movement trajectory to extract features such as the vehicle's stop point, driving path, and speed change. Through this step, a traffic location feature set is generated, covering the vehicle's movement pattern and geographic location information.

[0043] Finally, the traffic area keyword information, sentiment feature information and traffic location feature set are integrated into a unified multimodal traffic area feature set through feature-level fusion technology.

[0044] Furthermore, in the system provided in the embodiment of the application, the feature extraction module 13 is also used for:

[0045] An abnormal state is defined for each detection target in the traffic area detection target set to obtain a detection target abnormal state judgment rule; multimodal data crawling and abnormal clustering identification are performed according to the detection target abnormal state judgment rule to obtain a multimodal abnormal clustering data set; abnormal feature classification training is performed based on the multimodal abnormal clustering data set to construct a multimodal target abnormal feature classifier; abnormal feature identification is performed on the multimodal traffic area feature set based on the multimodal target abnormal feature classifier to obtain the abnormal multimodal traffic feature set.

[0046] In an embodiment of the present application, first, an abnormal state is defined for each target in the traffic area detection target set, and by combining the existing business logic and common abnormal behaviors and events in the traffic field, the detection target abnormal state judgment rules are set. For example, by comparing the real-time speed data of the vehicle with the road speed limit sign data, when the speed exceeds the speed limit value, it is judged as speeding behavior; by comparing the vehicle's driving direction with road signs, such as traffic markings, it is judged whether there is a reverse behavior, for example, by analyzing the deviation between the vehicle's driving direction and the legal direction of the road through GPS data; by combining the state of the traffic light with the vehicle's moving trajectory, it is judged whether the vehicle crosses the stop line during the red light period; using trajectory analysis technology, such as Kalman filtering or Bessel curve fitting, monitor the vehicle's driving path, and when it is detected that the vehicle has obviously deviated from the normal driving path, the behavior is identified as trajectory deviation. By setting these abnormal state judgment criteria, a set of detection target abnormal state judgment rules is generated.

[0047] Next, we start crawling data related to the detection target from multimodal information streams, including images, voice, GPS location information, etc. Based on the abnormal state judgment rules set in the previous step, we screen out targets that meet the abnormal criteria and mark them as potential abnormal events. In this process, multimodal data processing technology is used to synchronously obtain information streams such as images, audio, and location. Subsequently, clustering algorithms are applied to identify abnormal data. Cluster analysis is performed on the crawled data by using DBSCAN or K-means clustering algorithms. For example, by analyzing the speed data of a large number of vehicles, speeding vehicles are clustered into one type of abnormal event; by analyzing the trajectory data, vehicles with trajectory deviations are clustered into another type of abnormal event. Through this process, a multimodal abnormal clustering dataset is finally generated, which includes targets marked as abnormal and their related information.

[0048] After obtaining the multimodal anomaly clustering dataset, the feature classification training phase begins. In order to accurately identify various abnormal behaviors, the clustering results are used to perform feature classification training that combines supervised learning and unsupervised learning. Common machine learning algorithms, such as random forests, support vector machines, or decision trees, are used to train classification models at this stage. These models are trained with a large amount of historical data to learn how to identify different types of abnormal behaviors. For example, for speeding behavior, based on a large amount of historical vehicle speed data compared to the road speed limit, the training model learns how to identify speeding vehicles. Similarly, the recognition model for reverse traffic is trained by analyzing the deviation between the vehicle's driving direction and the expected direction. In this way, the recognition ability of the model is continuously optimized, enabling it to more accurately classify various abnormal behaviors. After training, a multimodal target abnormal feature classifier is generated.

[0049] Finally, the trained multimodal target abnormal feature classifier is used to identify abnormal features of the multimodal traffic area feature set obtained in real time. The feature set includes various features in the multimodal information flow, such as the target multidimensional behavior feature vector set, traffic event feature information, and traffic location feature set. The classifier is used to classify and analyze these features to detect whether there is abnormal behavior. For example, when the trajectory of the target shows an abnormal deviation, the target is marked as a trajectory anomaly; when the speed of the vehicle exceeds the road speed limit, it is marked as a speeding anomaly. The classifier quickly identifies multiple types of abnormal behaviors by analyzing multimodal data in real time. Through this process, an abnormal multimodal traffic feature set is generated, which contains all detected abnormal behaviors, events, and trajectory features.

[0050] The mapping extraction fusion module 14 performs mapping extraction fusion on the multimodal traffic area feature set based on the abnormal multimodal traffic feature set to generate an abnormal multimodal traffic feature matrix.

[0051] In an embodiment of the present application, when the mapping extraction fusion module performs mapping extraction fusion on the multimodal traffic area feature set based on the abnormal multimodal traffic feature set, firstly, based on the abnormal multimodal traffic feature set, the abnormal behavior multidimensional feature set, abnormal event feature information and abnormal location feature set are extracted. Next, these three types of features are mapped, extracted and fused. In this process, by analyzing their correlation in space and time, they are mapped and matched to generate an abnormal multimodal matching feature set and a discrete abnormal multimodal feature set. On this basis, based on the discrete abnormal multimodal feature set, the multimodal traffic area feature set is further associated with extraction and cross-validation, and which features play a key role in abnormal behavior are determined through verification, thereby obtaining an abnormal multimodal effective feature set.

[0052] Finally, the abnormal multimodal matching feature set and the abnormal multimodal effective feature set are aligned, fused and matrix transformed. These multimodal features are standardized and combined into a matrix to generate the final abnormal multimodal traffic feature matrix.

[0053] Furthermore, in the system provided in the embodiment of the application, the mapping extraction fusion module 14 is also used for:

[0054] According to the abnormal multimodal traffic feature set, determine the abnormal behavior multidimensional feature set, abnormal event feature information and abnormal location feature set; map and match the abnormal behavior multidimensional feature set, abnormal event feature information and abnormal location feature set to obtain an abnormal multimodal matching feature set and a discrete abnormal multimodal feature set; based on the discrete abnormal multimodal feature set, perform association extraction and cross-validation on the multimodal traffic area feature set to determine the abnormal multimodal valid feature set; align, fuse and matrix-convert the abnormal multimodal matching feature set and the abnormal multimodal valid feature set to generate the abnormal multimodal traffic feature matrix.

[0055] In an embodiment of the present application, firstly, based on the abnormal multimodal traffic feature set, an abnormal behavior multidimensional feature set, abnormal event feature information and abnormal location feature set are extracted. To extract these features, a multimodal feature extraction technique is used. Specifically, the abnormal behavior multidimensional feature set extracts the behavior characteristics of the vehicle from the image data through YOLO, such as speed, acceleration, direction change, motion trajectory, etc.; the abnormal event feature information extracts event-related information from the voice or text data through the speech recognition system and natural language processing technology; the abnormal location feature set uses GPS positioning data, and the vehicle's trajectory is smoothed by Kalman filtering to extract the vehicle's latitude and longitude, speed and direction and other information.

[0056] Next, the extracted features are mapped and matched. The purpose of mapping matching is to match the data from different modalities in time and space. First, the data of different modalities are synchronized in time using the timestamp alignment algorithm. For example, the vehicle detected in the image is time-matched with the corresponding location information in the GPS. Then, the spatial mapping technology is used to spatially match the data under different modalities through the geographic location and driving trajectory of the vehicle. For example, the vehicle outline in the image is associated with the location coordinates in the GPS data. Through these steps, two results are generated: an abnormal multimodal matching feature set and a discrete abnormal multimodal feature set. The abnormal multimodal matching feature set refers to a feature set that has abnormalities in all three modalities of image, GPS positioning and voice and can match each other. That is, only when the data of the three modalities show abnormalities and are correlated with each other, they are included in the set. The discrete abnormal multimodal feature set includes features that do not have matching abnormalities in all modalities. These features can be features with abnormal matching between only two modalities, or abnormal features in a single modality. For example, if only the GPS positioning modality data has an anomaly, while the image and voice modalities do not detect an anomaly, the anomaly will be classified as a discrete anomaly feature.

[0057] Then, based on the discrete abnormal multimodal feature set, the entire multimodal traffic area feature set is subjected to correlation extraction and cross-validation. Specifically, first, based on the discrete abnormal multimodal feature set, time-correlated features are extracted through temporal correlation extraction to form an abnormal correlation multimodal feature set. Then, these features are subjected to modal credibility evaluation to determine the confidence factor of each feature. Subsequently, based on these confidence factors, the feature set is cross-validated to generate an abnormal multimodal feature credibility coefficient set. Finally, the credibility coefficient set is screened through the preset credibility benchmark to determine the final abnormal multimodal valid feature set.

[0058] Finally, the abnormal multimodal matching feature set and the abnormal multimodal effective feature set are aligned, fused and transformed into a matrix. Specifically, in the alignment and fusion process, the features of different modes are integrated through temporal and spatial alignment. The vehicle features detected in the image, such as contour, color, and shape, are synchronized with the GPS location information at the corresponding time point through temporal alignment, and the timestamp is used to ensure that the data occurs at the same time. Subsequently, the spatial alignment algorithm is used to match these vehicle features with the geographic location information provided by GPS to ensure that the spatial position of the vehicle is consistent in the image data and the location information. After completing the temporal and spatial alignment, the matrix construction stage is entered. Using feature mapping technology, the features extracted under multimodality, such as the vehicle contour, color, shape in the image and the position and speed in the GPS data, are converted into a matrix form. Each row in the matrix represents a target or event, and each column represents the specific features of the target. For example, the features in the image occupy the first few columns, the GPS location features occupy the next few columns, and the last few columns include data such as event type. Through this feature mapping and matrix construction, the data of all modes are integrated into a matrix, and each matrix element represents a specific abnormal feature, such as the appearance, location or abnormal event type of the vehicle. Through this process, an abnormal multimodal traffic feature matrix is ​​generated, which aligns the features under different modes through a standard row-column structure to ensure that all abnormal features can be uniformly represented in the same matrix.

[0059] Furthermore, in the system provided in the embodiment of the application, the mapping extraction fusion module 14 is also used for:

[0060] Based on the discrete abnormal multimodal feature set, the multimodal traffic area feature set is subjected to time series correlation extraction to obtain an abnormal correlation multimodal feature set; a modal credibility assessment is performed on the abnormal correlation multimodal feature set to determine modal feature confidence factor information; based on the modal feature confidence factor information, the abnormal correlation multimodal feature set is cross-validated to obtain an abnormal multimodal feature credibility coefficient set; abnormal feature screening is performed on the abnormal multimodal feature credibility coefficient set according to a preset credibility coefficient benchmark to determine the abnormal multimodal effective feature set.

[0061] In an embodiment of the present application, firstly, the features under different modes are synchronized and associated in time through time series association analysis. Specifically, the abnormal features in the discrete abnormal multimodal feature set are first identified, for example, the abnormalities detected in the GPS mode alone, and then the features related to the abnormal features in other modes such as images or voices in the same time period are found through time series association analysis. Through this time series association extraction, the discrete abnormal features are combined with other modal features that may be related to generate an abnormal association multimodal feature set.

[0062] Next, the modal credibility of each modal feature in the abnormal correlation multimodal feature set is evaluated. For image features, the clarity of the image is first evaluated by the Laplace variance method to obtain the image clarity score. The specific calculation process is to first calculate the Laplace transform of the image to obtain the variance value, and then compare the variance value with the preset maximum benchmark variance, standardize it into a score, and obtain the image clarity score. At the same time, the illumination uniformity is evaluated by analyzing the standard deviation of the brightness histogram to obtain the illumination uniformity score. When performing specific calculations, the brightness standard deviation of all pixels in the image is first calculated, and then compared with the set maximum brightness standard deviation, and then the wallpaper obtained by the above calculation is subtracted from 1 to obtain the illumination uniformity score. In addition, by calculating the area ratio of the visible target object in the image, whether the image is blocked is detected to obtain the occlusion detection score. Finally, the confidence factor of the image feature is obtained by weighted calculation of the three scores of clarity, illumination uniformity and occlusion detection, where the weights of clarity, illumination uniformity and occlusion detection are pre-set. For GPS features, the signal strength is first evaluated by analyzing the signal-to-noise ratio to obtain a signal strength score. Specifically, the signal-to-noise ratio received by GPS is first measured and normalized with the maximum SNR. For example, if the SNR is 30 and the maximum SNR is 50, the score is 0.6. Then, after Kalman filtering, the error between the current position and the predicted position is calculated to obtain an error score. Specifically, the position error after Kalman filtering is calculated, and then the score is standardized. The error score is obtained by subtracting the ratio of the position error and the maximum error benchmark from 1. For example, if the error is 5 and the maximum error benchmark is 20, the score is 0.75. Finally, the continuity of the GPS position is evaluated by position drift detection to obtain a position drift score. For example, if the position drift is 10 and the maximum benchmark is 50, the score is 0.8. The specific calculation process is to obtain the position drift score by subtracting the ratio of the position drift to the maximum benchmark from 1. Finally, the confidence factor of the GPS feature is obtained by weighted calculation of the signal strength, error and position drift score. The weights of the signal strength, error and position drift score are pre-set. When evaluating the credibility of speech features, the signal-to-noise ratio score, the effective speech detection score, and the background noise score are performed separately. When performing the signal-to-noise ratio score, the signal-to-noise ratio of the speech signal is calculated. The clearer the signal, the higher the score. For example, if the SNR is 20 and the maximum SNR is 40, the score is 0.5. When performing the effective speech detection score, the effective speech segment in the speech is detected by the VAD algorithm, and after removing the background noise, the clear speech segment is identified. For example, if 80% of the speech is detected as a valid segment, the score is 0.8. When performing the background noise score, the background noise level is evaluated. The less noise, the higher the score. For example, if the background noise level is 30%, the score is 0.7.Finally, the signal-to-noise ratio score, effective speech detection score and background noise score are weighted to obtain the confidence factor of the speech feature, where the signal-to-noise ratio score, effective speech detection score and background noise score have the same weight.

[0063] After obtaining the confidence factors of the three modalities of image, GPS and voice, they are used as the confidence factor information of modal features. Then, the abnormal associated multimodal feature set is cross-validated based on the modal feature confidence factor information. Specifically, during the cross-validation process, weighted validation is performed according to the modal feature confidence factor information to ensure the consistency of the performance of each modal feature in the whole. For example, check whether the vehicle behavior characteristics in the image match the driving path of the GPS and the related events in the voice. Through the K-fold cross-validation method, a cross-validated credibility coefficient is assigned to each feature. For example, for a specific time point, the confidence factors of the image, GPS and voice features are 0.74, 0.73 and 0.68 respectively. The comprehensive credibility coefficient obtained by cross-validation weighted calculation is 0.4×0.74+0.4×0.73+0.2×0.68, which is equal to 0.726, where 0.4, 0.4 and 0.2 are the weights of the confidence factors of the image, GPS and voice features, respectively, which are pre-set by technical experts. Through this process, the abnormal multimodal feature credibility coefficient set is obtained.

[0064] Finally, the cross-validated abnormal multimodal feature credibility coefficient set is screened according to the preset credibility benchmark. If the comprehensive credibility coefficient of a feature is higher than the benchmark value, the feature is retained in the abnormal multimodal valid feature set, and the abnormal multimodal valid feature set is obtained through this screening process.

[0065] The coordinated operation and maintenance control module 15 performs a strategy analysis on the abnormal multimodal traffic feature matrix, obtains traffic coordination control strategy parameters, and performs coordinated operation and maintenance control on the target traffic area based on the traffic coordination control strategy parameters.

[0066] In an embodiment of the present application, the coordinated operation and maintenance control module first extracts multiple traffic operation and maintenance strategy parameters related to the current traffic conditions through correlation analysis of the abnormal feature matrix based on the predefined strategies in the traffic abnormality operation and maintenance strategy library. The effect simulation evaluation is performed on the multiple extracted traffic operation and maintenance strategy parameters, and the operation and maintenance effect of each strategy parameter is evaluated by simulating factors such as traffic flow and event impact. Based on these operation and maintenance effects, multiple strategy parameters are selected and cross-expanded. By combining and expanding different strategies, a new traffic operation and maintenance strategy parameter optimization population is generated. Finally, iterative optimization is performed on the optimized population, and the optimal strategy combination is found through algorithm optimization. After multiple iterations, the optimal traffic coordination control strategy parameters are obtained.

[0067] Finally, the target traffic area is coordinated, operated and maintained according to the traffic coordination control strategy parameters.

[0068] Furthermore, in the system provided in the application embodiment, the coordination operation and maintenance control module 15 is also used for:

[0069] Construct a traffic abnormality operation and maintenance strategy library, perform correlation strategy analysis on the abnormal multimodal traffic feature matrix based on the traffic abnormality operation and maintenance strategy library, and obtain multiple traffic operation and maintenance strategy parameters; perform effect simulation evaluation on the multiple traffic operation and maintenance strategy parameters to determine the operation and maintenance effects of multiple strategy parameters; select and cross-expand the multiple traffic operation and maintenance strategy parameters based on the operation and maintenance effects of the multiple strategy parameters to obtain a traffic operation and maintenance strategy parameter optimization population, perform iterative optimization on the traffic operation and maintenance strategy parameter optimization population, and obtain the traffic coordination control strategy parameters.

[0070] In an embodiment of the present application, when constructing a traffic anomaly operation and maintenance strategy library, common traffic problems and corresponding solutions are determined by collecting and analyzing historical traffic data, such as accident records, traffic flow changes, congestion events, etc. Next, based on these historical data and the advice of traffic management experts, technical experts manually define rules for dealing with different traffic anomalies. For example, for congestion, the rule is defined as "If the vehicle flow on a certain section of road exceeds the set threshold, the green light duration is automatically extended and the vehicle is diverted to the backup road." These rules cover solutions such as signal light adjustment, vehicle diversion, and emergency event priority handling. Through this process, a traffic anomaly handling strategy library is formed.

[0071] Next, the abnormal multimodal traffic feature matrix is ​​subjected to association strategy analysis based on the traffic abnormality operation and maintenance strategy library. Specifically, when an abnormal traffic event occurs, relevant traffic abnormality features are extracted from the abnormal multimodal traffic feature matrix. By using association rule mining technology, such as the Apriori algorithm, the characteristics of traffic events are analyzed to find frequent patterns related to common events in historical data. For example, if a sudden drop in vehicle speed during a certain period of time is detected accompanied by a voice alarm of a traffic accident, the diversion and accident handling strategies in the strategy library are matched to generate a series of traffic operation and maintenance strategy parameters, including the adjustment time of traffic lights, vehicle diversion paths, and priority passage for rescue vehicles.

[0072] Next, we simulated and evaluated the effects of multiple traffic operation and maintenance strategy parameters. We built traffic simulation scenarios using traffic simulation tools such as SUMO and VISSIM, input different traffic events, flow data, and signal control strategies, and simulated these operation and maintenance parameters. Through simulation, we evaluated the performance of each strategy in different traffic scenarios. For example, we simulated whether the signal adjustment plan for a certain section of road could effectively reduce congestion, or the impact of vehicle diversion on surrounding traffic. Through multiple simulation runs and Monte Carlo random scenario tests, we generated a detailed report on the operation and maintenance effect of each strategy parameter, including indicators such as the time required to restore traffic flow and the speed of accident handling.

[0073] Based on the simulation evaluation results, the policy parameters with good performance are selected and cross-expanded. Different strategies are combined to generate new solutions through genetic algorithms. For example, traffic light adjustment and vehicle diversion strategies are combined to form a more effective solution. This process generates a traffic operation and maintenance policy parameter optimization population, which contains a variety of expanded policy combinations. These policy combinations are tested multiple times through cross-validation to ensure their effectiveness in different traffic scenarios. Finally, particle swarm optimization is used for iterative optimization. Each policy combination is regarded as a particle, and the effect of each policy is scored through the fitness function. The fitness function measures the performance of each policy combination on different indicators, such as reducing traffic congestion time and improving accident handling efficiency. Through multiple rounds of iterations, these policy combinations are continuously optimized, and poorly performing policies are eliminated, and finally the optimal traffic coordination control policy parameters are generated.

[0074] The fitness function is as follows:

[0075]

[0076] Among them, F is the value of the fitness function, which represents the comprehensive performance score of the strategy; is the target time for traffic flow recovery, usually the average traffic flow time without abnormalities; T is the actual traffic flow recovery time. is the current traffic congestion index; C is the maximum tolerable traffic congestion index; is the speed at which the rescue vehicle passes; S is the expected speed at which the rescue vehicle passes; is the actual vehicle throughput: V is the theoretical maximum vehicle throughput; , , and It is a weight parameter used to balance the impact of different indicators and is pre-set by technical experts. When calculating the traffic congestion index, it is obtained by subtracting the ratio of the actual average speed of vehicles on the road to the designed free flow speed of the road or the ideal speed in the absence of congestion.

[0077] In the embodiments of the present application, in summary, the embodiments of the present application have at least the following technical effects:

[0078] The present application obtains an all-in-one information collection machine, which integrates a camera, a GPS locator and an intercom communication module, and synchronously collects and obtains a multimodal information flow for monitoring a traffic area through the all-in-one information collection machine; performs preprocessing step analysis on the multimodal information flow for monitoring a traffic area respectively to obtain a multimodal data preprocessing program, and performs parallel preprocessing on the multimodal information flow for monitoring a traffic area based on the multimodal data preprocessing program to obtain a multimodal information flow for monitoring a standard traffic area; performs feature extraction on the multimodal information flow for monitoring a standard traffic area in turn to obtain a multimodal traffic area feature set, and performs abnormal feature extraction based on the multimodal traffic area feature set to obtain an abnormal multimodal traffic feature set; maps, extracts and fuses the multimodal traffic area feature set based on the abnormal multimodal traffic feature set to generate an abnormal multimodal traffic feature matrix; performs strategy analysis on the abnormal multimodal traffic feature matrix to obtain traffic coordination control strategy parameters, and coordinates operation and maintenance management of the target traffic area based on the traffic coordination control strategy parameters. The present invention solves the technical problems of incomplete information, slow response speed and low anomaly detection accuracy in the prior art when conducting traffic monitoring. The multimodal information flow of the traffic area is synchronously collected by an information collection all-in-one machine, pre-processed by a standard information acquisition module, the traffic characteristics are extracted by a feature extraction module, the mapping extraction fusion module generates an abnormal feature matrix, and finally the operation and maintenance control module is coordinated to perform strategy analysis and optimization, so as to achieve the technical effect of improving the efficiency of traffic anomaly monitoring and management.

[0079] Embodiment 2 is based on the same inventive concept as the integrated monitoring system based on multimodal information fusion in the above embodiment. Figure 2 As shown, the embodiment of the present application provides an integrated monitoring method based on multimodal information fusion, the method comprising:

[0080] An information collection integrated machine is obtained, wherein the information collection integrated machine integrates a camera, a GPS locator and an intercom communication module, and a multimodal information flow for monitoring a traffic area is synchronously collected through the information collection integrated machine; preprocessing steps are respectively analyzed for the multimodal information flow for monitoring the traffic area to obtain a multimodal data preprocessing program, and the multimodal information flow for monitoring the traffic area is parallelly preprocessed based on the multimodal data preprocessing program to obtain a standard traffic area monitoring multimodal information flow; feature extraction is performed on the standard traffic area monitoring multimodal information flow in sequence to obtain a multimodal traffic area feature set, and abnormal feature extraction is performed based on the multimodal traffic area feature set to obtain an abnormal multimodal traffic feature set; mapping, extraction and fusion are performed on the multimodal traffic area feature set based on the abnormal multimodal traffic feature set to generate an abnormal multimodal traffic feature matrix; strategy analysis is performed on the abnormal multimodal traffic feature matrix to obtain traffic coordination control strategy parameters, and coordinated operation and maintenance management of the target traffic area is performed based on the traffic coordination control strategy parameters.

[0081] Furthermore, the method for obtaining a multimodal traffic area feature set further includes:

[0082] The standard traffic area monitoring multimodal information stream is subjected to data diversion to obtain a traffic area image information stream, a traffic area voice information stream and a traffic area location information stream; the traffic area image information stream is subjected to target detection and bounding box marking by using the YOLO algorithm to obtain a traffic area detection target set; the traffic area detection target set is subjected to behavioral feature extraction and identification in turn to obtain a target multidimensional behavioral feature vector set; feature extraction is performed on the traffic area voice information stream and the traffic area location information stream respectively to obtain traffic event feature information and a traffic location feature set; based on the target multidimensional behavior feature vector set, the traffic event feature information and the traffic location feature set, the multimodal traffic area feature set is obtained.

[0083] Furthermore, the method for obtaining the traffic event feature information and the traffic location feature set further includes:

[0084] The traffic area voice information stream is converted into a traffic area text information stream, and the traffic area text information stream is subjected to word segmentation and keyword recognition according to a traffic event text library to obtain traffic area keyword information; context sentiment analysis is performed on the traffic area text information stream based on the traffic area keyword information to obtain traffic area sentiment feature information; information fusion and event classification are performed on the traffic area keyword information and the traffic area sentiment feature information to obtain the traffic event feature information; associated position extraction and position trajectory analysis are performed on the traffic area position information stream to obtain the traffic position feature set.

[0085] Furthermore, the method for obtaining the abnormal multimodal traffic feature set further includes:

[0086] An abnormal state is defined for each detection target in the traffic area detection target set to obtain a detection target abnormal state judgment rule; multimodal data crawling and abnormal clustering identification are performed according to the detection target abnormal state judgment rule to obtain a multimodal abnormal clustering data set; abnormal feature classification training is performed based on the multimodal abnormal clustering data set to construct a multimodal target abnormal feature classifier; abnormal feature identification is performed on the multimodal traffic area feature set based on the multimodal target abnormal feature classifier to obtain the abnormal multimodal traffic feature set.

[0087] Furthermore, the method for generating an abnormal multimodal traffic feature matrix further includes:

[0088] According to the abnormal multimodal traffic feature set, determine the abnormal behavior multidimensional feature set, abnormal event feature information and abnormal location feature set; map and match the abnormal behavior multidimensional feature set, abnormal event feature information and abnormal location feature set to obtain an abnormal multimodal matching feature set and a discrete abnormal multimodal feature set; based on the discrete abnormal multimodal feature set, perform association extraction and cross-validation on the multimodal traffic area feature set to determine the abnormal multimodal valid feature set; align, fuse and matrix-convert the abnormal multimodal matching feature set and the abnormal multimodal valid feature set to generate the abnormal multimodal traffic feature matrix.

[0089] Furthermore, the method for determining an abnormal multimodal valid feature set further includes:

[0090] Based on the discrete abnormal multimodal feature set, the multimodal traffic area feature set is subjected to time series correlation extraction to obtain an abnormal correlation multimodal feature set; a modal credibility assessment is performed on the abnormal correlation multimodal feature set to determine modal feature confidence factor information; based on the modal feature confidence factor information, the abnormal correlation multimodal feature set is cross-validated to obtain an abnormal multimodal feature credibility coefficient set; abnormal feature screening is performed on the abnormal multimodal feature credibility coefficient set according to a preset credibility coefficient benchmark to determine the abnormal multimodal effective feature set.

[0091] Furthermore, the method for obtaining the traffic coordination control strategy parameters further includes:

[0092] Construct a traffic abnormality operation and maintenance strategy library, perform correlation strategy analysis on the abnormal multimodal traffic feature matrix based on the traffic abnormality operation and maintenance strategy library, and obtain multiple traffic operation and maintenance strategy parameters; perform effect simulation evaluation on the multiple traffic operation and maintenance strategy parameters to determine the operation and maintenance effects of multiple strategy parameters; select and cross-expand the multiple traffic operation and maintenance strategy parameters based on the operation and maintenance effects of the multiple strategy parameters to obtain a traffic operation and maintenance strategy parameter optimization population, perform iterative optimization on the traffic operation and maintenance strategy parameter optimization population, and obtain the traffic coordination control strategy parameters.

[0093] It should be noted that the above-mentioned sequence of the embodiments of the present application is only for description and does not represent the advantages and disadvantages of the embodiments. And the above-mentioned specific embodiments of this specification are described. The processes depicted in the accompanying drawings do not necessarily require the specific order and continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0094] The above description is only a preferred embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.

[0095] This specification and drawings are merely exemplary illustrations of the present application and are deemed to cover any and all modifications, variations, combinations or equivalents within the scope of the present application. Obviously, a person skilled in the art may make various modifications and variations to the present application without departing from the scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the present application and its equivalents, the present application intends to include these modifications and variations.

Claims

1. An integrated monitoring system based on multimodal information fusion, characterized in that: The system comprises: An information collection module, wherein the information collection module obtains an information collection integrated machine, wherein the information collection integrated machine integrates a camera, a GPS locator and an intercom communication module, and synchronously collects and obtains a multi-modal information flow for traffic area monitoring through the information collection integrated machine; A standard information acquisition module, wherein the standard information acquisition module performs preprocessing step analysis on the traffic area monitoring multimodal information flow respectively to obtain a multimodal data preprocessing program, and performs parallel preprocessing on the traffic area monitoring multimodal information flow based on the multimodal data preprocessing program to obtain a standard traffic area monitoring multimodal information flow; A feature extraction module, wherein the feature extraction module sequentially extracts features from the multimodal information flow of the standard traffic area monitoring to obtain a multimodal traffic area feature set, and performs abnormal feature extraction based on the multimodal traffic area feature set to obtain an abnormal multimodal traffic feature set; A mapping extraction fusion module, wherein the mapping extraction fusion module performs mapping extraction fusion on the multimodal traffic area feature set based on the abnormal multimodal traffic feature set to generate an abnormal multimodal traffic feature matrix; A coordinated operation and maintenance control module, wherein the coordinated operation and maintenance control module performs a strategy analysis on the abnormal multimodal traffic feature matrix, obtains traffic coordination control strategy parameters, and performs coordinated operation and maintenance control on the target traffic area based on the traffic coordination control strategy parameters; Wherein, the step of obtaining a multimodal traffic area feature set includes: Performing data splitting on the standard traffic area monitoring multimodal information stream to obtain a traffic area image information stream, a traffic area voice information stream, and a traffic area location information stream; Using the YOLO algorithm to perform target detection and bounding box marking on the traffic area image information stream, to obtain a traffic area detection target set; Extracting and marking the behavior characteristics of the traffic area detection target set in turn to obtain a target multi-dimensional behavior characteristic vector set; Extracting features from the traffic area voice information stream and the traffic area location information stream respectively to obtain traffic event feature information and a traffic location feature set; Based on the target multi-dimensional behavior feature vector set, the traffic event feature information and the traffic location feature set, obtaining the multi-modal traffic area feature set; The step of obtaining traffic event feature information and traffic location feature set includes: Converting the traffic area voice information stream into a traffic area text information stream, performing word segmentation and keyword recognition on the traffic area text information stream according to a traffic event text library, and obtaining traffic area keyword information; Based on the traffic area keyword information, contextual sentiment analysis is performed on the traffic area text information flow to obtain traffic area sentiment feature information; The traffic area keyword information and the traffic area sentiment feature information are subjected to information fusion and event classification to obtain the traffic event feature information; Performing associated position extraction and position trajectory analysis on the traffic area position information flow to obtain the traffic position feature set; The abnormal multimodal traffic feature set is obtained, including: Defining an abnormal state of each detection target in the traffic area detection target set to obtain a detection target abnormal state judgment rule; Multimodal data crawling and abnormal clustering identification are performed through the detection target abnormal state judgment rule to obtain a multimodal abnormal clustering data set; Based on the multimodal abnormal clustering data set, abnormal feature classification training is performed respectively to construct a multimodal target abnormal feature classifier; Based on the multimodal target abnormal feature classifier, abnormal features of the multimodal traffic area feature set are respectively identified to obtain the abnormal multimodal traffic feature set.

2. The integrated monitoring system based on multimodal information fusion according to claim 1, characterized in that: The generating of the abnormal multimodal traffic feature matrix includes: Determine an abnormal behavior multidimensional feature set, abnormal event feature information, and abnormal location feature set based on the abnormal multimodal traffic feature set; Mapping and matching the abnormal behavior multidimensional feature set, abnormal event feature information and abnormal location feature set to obtain an abnormal multimodal matching feature set and a discrete abnormal multimodal feature set; Based on the discrete abnormal multimodal feature set, the multimodal traffic area feature set is subjected to association extraction and cross-validation to determine an abnormal multimodal effective feature set; The abnormal multimodal matching feature set and the abnormal multimodal effective feature set are aligned, fused and matrix-converted to generate the abnormal multimodal traffic feature matrix.

3. The integrated monitoring system based on multimodal information fusion as claimed in claim 2, characterized in that: The step of determining an abnormal multimodal effective feature set includes: Based on the discrete abnormal multimodal feature set, the multimodal traffic area feature set is subjected to time series correlation extraction to obtain an abnormal correlation multimodal feature set; Performing modal credibility assessment on the abnormal associated multimodal feature set to determine modal feature confidence factor information; Cross-validating the abnormal associated multimodal feature set based on the modal feature confidence factor information to obtain an abnormal multimodal feature credibility coefficient set; The abnormal multimodal feature credibility coefficient set is screened for abnormal features by using a preset credibility coefficient benchmark to determine the abnormal multimodal valid feature set.

4. The integrated monitoring system based on multimodal information fusion according to claim 1, characterized in that: The obtaining of traffic coordination control strategy parameters includes: Constructing a traffic abnormality operation and maintenance strategy library, and performing a correlation strategy analysis on the abnormal multimodal traffic feature matrix based on the traffic abnormality operation and maintenance strategy library to obtain a plurality of traffic operation and maintenance strategy parameters; Performing effect simulation evaluation on the multiple traffic operation and maintenance strategy parameters to determine the operation and maintenance effects of the multiple strategy parameters; Based on the operation and maintenance effects of the multiple strategy parameters, the multiple traffic operation and maintenance strategy parameters are selected and cross-expanded to obtain an optimized population of traffic operation and maintenance strategy parameters, and iterative optimization is performed on the optimized population of traffic operation and maintenance strategy parameters to obtain the traffic coordination control strategy parameters.

5. An integrated monitoring method based on multimodal information fusion, characterized in that: The method is performed by an integrated monitoring system based on multimodal information fusion according to any one of claims 1 to 4, comprising: Obtaining an information collection integrated machine, wherein the information collection integrated machine integrates a camera, a GPS locator and an intercom communication module, and synchronously collects and obtains a multi-modal information flow for traffic area monitoring through the information collection integrated machine; Performing preprocessing step analysis on the traffic area monitoring multimodal information flow respectively to obtain a multimodal data preprocessing program, and performing parallel preprocessing on the traffic area monitoring multimodal information flow based on the multimodal data preprocessing program to obtain a standard traffic area monitoring multimodal information flow; Performing feature extraction on the multimodal information flow monitored in the standard traffic area in sequence to obtain a multimodal traffic area feature set, and performing abnormal feature extraction based on the multimodal traffic area feature set to obtain an abnormal multimodal traffic feature set; Based on the abnormal multimodal traffic feature set, mapping, extracting and fusing the multimodal traffic area feature set is performed to generate an abnormal multimodal traffic feature matrix; A strategy analysis is performed on the abnormal multimodal traffic feature matrix to obtain traffic coordination control strategy parameters, and coordinated operation and maintenance management of the target traffic area is performed based on the traffic coordination control strategy parameters.

Images