Ecological environment commanding and dispatching system and method based on audio and video fusion

Abstract

The invention discloses an ecological environment commanding and dispatching system and method based on audio and video fusion, and relates to the technical field of ecological environment dispatching. The commanding and dispatching platform is in communication connection with a data acquisition module, a data preprocessing module, an event detection and identification module, a severity evaluation module, a commanding and dispatching module and a resource management module, and all the modules are in electric signal connection; the data acquisition module is used for collecting real-time audio and video data from various sensors and monitoring equipment which are deployed in advance. According to the invention, through real-time monitoring and automatic detection technologies, abnormal events are identified at the first time, and an early warning mechanism is started immediately, so that compared with traditional manual inspection, the time for finding the abnormal events can be shortened, in addition, the automation characteristic of the system reduces delay caused by human factors, and the system is more convenient to use. Therefore, resources and personnel can be quickly mobilized in emergency, and the influence of events can be effectively controlled and relieved.

Description

Technical Field

[0001] The present invention relates to the technical field of ecological environment dispatching, and in particular to an ecological environment command and dispatching system and method based on audio and video fusion. Background Art

[0002] As global environmental problems become increasingly severe, ecological environmental protection has become a focus of attention from all sectors of society. In order to effectively respond to challenges such as environmental pollution and ecological destruction, it is necessary to establish an efficient and intelligent command and dispatch system to achieve comprehensive monitoring, rapid response and scientific decision-making of the ecological environment. Audio and video fusion technology is a technology that integrates and processes multiple media information such as audio and video. With the continuous advancement of IP technology, soft switching technology, video encoding and decoding technology, audio and video fusion technology has become highly integrated, real-time and interactive. These technical characteristics have made audio and video fusion technology widely used in ecological environmental command and dispatch systems.

[0003] For example, the Chinese patent publication number: CN111950842A is a blockchain-based ecological environment comprehensive scheduling method and system, and the specific steps are as follows: S1: According to the type of ecological environment that needs to be controlled, information collection platforms of different types of ecological environments are set up, and each information collection platform sets up an information processing scheduling room in a different collection area; S2: Each regional chain storage module is provided with multiple regional blocks, and the information processing scheduling room sends and stores the ecological environment information in a regional block in the corresponding regional chain storage module; S3: A texture editing module and a database are provided in the comprehensive scheduling room; S4: The comprehensive scheduling room sends the scheduling information to the information processing scheduling room for ecological environment management scheduling or remote monitoring.

[0004] In the prior art, although the dispatching system has early warning and dispatching functions, it lacks effective severity analysis measures for sudden ecological and environmental events, such as forest fires, water pollution, invasion of invasive species, etc., and it is difficult to give early warnings to sudden ecological and environmental events of different states to a corresponding degree, which makes it easy to waste dispatching resources and fail to respond quickly when conducting command and dispatch. The dispatching system has certain limitations. Therefore, how to analyze ecological and environmental events and judge their severity in order to achieve rapid response and effectively dispatch resources is a problem we need to solve. To this end, an ecological and environmental command and dispatch system and method based on audio and video fusion is proposed. Summary of the invention

[0005] The purpose of the present invention is to provide an ecological environment command and dispatch system and method based on audio and video fusion to solve the problems raised in the above background technology.

[0006] In order to solve the above technical problems, the technical solution adopted by the present invention is:

[0007] In a first aspect, an ecological environment command and dispatch system based on audio and video fusion includes a command and dispatch platform, wherein the command and dispatch platform is communicatively connected to a data acquisition module, a data preprocessing module, an event detection and identification module, a severity assessment module, a command and dispatch module, and a resource management module, wherein electrical signals are connected between the modules;

[0008] The data acquisition module is used to collect real-time audio and video data from various pre-deployed sensors and monitoring equipment. The data comes from environmental monitoring stations, drones, cameras, and sound sensors, and obtains video images and sound information on the scene to provide basic data for subsequent analysis and dispatch. The collected audio and video data is transmitted to the command and dispatch platform, and advanced communication technologies and network architectures such as 4G / 5G, satellite communications, and optical fibers are used to ensure the stability and efficiency of data transmission.

[0009] The data preprocessing module allows users to process and analyze the collected audio and video data, including video encoding and decoding, audio noise reduction, and image enhancement to ensure the clarity and consistency of the data, store and back up the audio and video data, ensure the security and traceability of the data, improve the processing efficiency and accuracy of the data, and prepare for further analysis;

[0010] The event detection and identification module combines image recognition and machine learning technologies to analyze the processed audio and video data, identify abnormal events in the ecological environment, provide a preliminary judgment of the event, and help formulate corresponding response strategies and priorities;

[0011] The severity assessment module analyzes and assesses the severity of abnormal events in the identified ecological environment in combination with historical data based on the impact scope, potential hazards and urgency of the abnormal events, sets an early warning level for the severity of the abnormal events, and classifies the abnormal events. Through quantitative indicators and classification standards, a scientific basis is provided for the allocation of scheduling resources and the priority of emergency response.

[0012] The command and dispatch module is used to formulate and issue corresponding dispatch instructions and allocate dispatch resources according to the results of the event detection and identification module and the severity assessment module, combined with the determined warning level of the severity of the abnormal event, and realize real-time communication and command with on-site personnel through audio and video communication means, such as video conferencing, voice broadcasting, etc. At the same time, it supports multi-department and multi-regional coordinated dispatch to ensure the timeliness and effectiveness of emergency response;

[0013] The resource management module uniformly manages and optimizes the allocation of various resources in the dispatching system, including the dispatching and allocation of human, material and financial resources, as well as the dynamic monitoring and evaluation of resources, to ensure that the required resources can be quickly mobilized when an emergency occurs, providing strong support for emergency response.

[0014] A further improvement of the technical solution of the present invention is that in the data acquisition module, the process of collecting real-time audio and video data is:

[0015] Environmental monitoring stations, drones, cameras and sound sensors are deployed in advance at each monitoring point in the area to be monitored to ensure that all areas that need to be monitored are covered. Environmental monitoring stations provide environmental monitoring data and are equipped with cameras and sound sensors to capture on-site video images and sound information. Drones use high-definition cameras and sound sensors to conduct aerial inspections of specific areas and transmit on-site audio and video data in real time. The high maneuverability and flexibility of drones enable them to quickly reach areas that are difficult for people to reach or dangerous. Fixed or mobile cameras are used to monitor specific areas or targets and provide continuous video streaming data. Cameras are installed in key locations, such as ecological protection areas and near pollution sources. Sound sensors are used to capture and record sound information in the environment and provide sound dimension data for ecological and environmental analysis.

[0016] Sensors and monitoring devices monitor environmental parameters in real time and obtain audio and video data, including video images and sound information, covering the entire area where the incident occurred. The collected audio and video data are encoded and compressed to adapt to different network environments and reduce the demand for transmission bandwidth;

[0017] Utilize advanced communication technology and network architecture (such as 4G / 5G, satellite communication, optical fiber, etc.) to transmit encoded and compressed data from the field to the command and dispatch platform, ensuring the efficiency and reliability of data transmission, and maintaining the real-time nature of data even over long distances or in complex environments;

[0018] The command and dispatch platform receives data from various sensors and monitoring equipment, integrates them into a unified data format, stores them on the cloud platform, and synchronizes the time of data collected from different sources to maintain the consistency of audio and video information for subsequent processing and analysis.

[0019] A further improvement of the technical solution of the present invention is that in the data preprocessing module, the process of storing and backing up the audio and video data is as follows:

[0020] Retrieve the original audio and video data stream from the command and dispatch platform, perform encoding and decoding on the video, decode the collected video data from the compressed format (such as H.264, HEVC, etc.) into the original video frame for subsequent processing, and after the video processing is completed, re-encode the video data into the corresponding compression format according to storage or transmission requirements to reduce data volume and optimize transmission efficiency;

[0021] Enhance the video frames, such as brightness adjustment, contrast enhancement, color correction, etc., to improve the clarity and visual effects of the video, and divide the video into different shot segments based on the inter-frame difference method to facilitate subsequent key frame extraction and feature analysis, and extract key frames from each shot segment. The key frames reflect the main content of the shot;

[0022] Applying noise reduction algorithms to reduce or eliminate noise components in audio signals improves audio clarity and audibility, helping to more clearly capture sound information in the environment;

[0023] The processed and analyzed audio and video data are cleaned and formatted. Data cleaning removes outliers and incomplete data records to prevent them from affecting the accuracy of the analysis results. Data formatting converts data from different sources into a unified format for easy integration and analysis. The processed data is stored in a data warehouse for subsequent analysis. At the same time, the data is backed up regularly and redundant storage is set up to ensure data security and recoverability to prevent data loss or damage.

[0024] A further improvement of the technical solution of the present invention is that in the event detection and identification module, the identification process of abnormal events in the ecological environment is:

[0025] Receive audio and video data from the data preprocessing module, and extract sample data for training and testing from the processed audio and video data, ensuring that the sample data covers various possible event types, such as forest fires, water pollution, invasive species invasion, drought, etc., annotate the sample data, and mark the characteristic areas of different abnormal event types;

[0026] Use computer vision technology to identify images in video frames and extract image features from video frames, including color, texture, shape, and motion trajectory. Perform feature extraction on audio data to obtain sound features, including frequency, intensity, and duration of sound. Use OpenCV and other libraries to extract static features such as color, texture, and shape in video frames. Use optical flow or background subtraction technology to extract dynamic features such as motion trajectory. Use audio processing libraries (such as librosa) to analyze audio signals and extract features such as frequency, intensity, and duration.

[0027] A large amount of annotated ecological environmental abnormal event data is used to train an abnormal event recognition model, and the extracted features are analyzed to identify the difference between the normal environmental background and abnormal events. The abnormal event recognition model is obtained based on a convolutional neural network model.

[0028] Based on the abnormal event recognition model and the associated data of image features, the image is detected for abnormality, the abnormal image recognition index is obtained through analysis, the abnormal image is detected, the processed sound feature data is input into the abnormal event recognition model, the abnormal sound recognition index is obtained, and the abnormal sound is detected;

[0029] The detection results of abnormal images and abnormal sounds are integrated to analyze and identify abnormal events in the ecological environment, classify the detected abnormal events, determine the specific type of abnormal events, and locate them in combination with the geographic information of the video frames to determine the location where the abnormal events occurred.

[0030] A further improvement of the technical solution of the present invention is that the expression of the abnormal image recognition index is:

[0031]

[0032] Among them, AP is the abnormal image recognition index, n is the number of frames in the video, and C i is the color feature value of the i-th frame image, T i is the texture feature value of the i-th frame image, F i is the shape feature value of the i-th frame image, M i is the motion feature value of the i-th frame image, e is the base of the natural logarithm, the AP value tends to 0, indicating that the image features are very close to the normal environment and there is no abnormality, and the AP value tends to 1, indicating that the image features are significantly different from the normal environment and the degree of abnormality is high;

[0033] The expression of the abnormal sound recognition index is:

[0034]

[0035] Among them, AS is the abnormal sound recognition index, s is the number of sound samples, L j is the frequency characteristic value of the jth sound sample, B is the reference frequency value, representing the typical frequency characteristic value under normal environmental conditions, S is the standard deviation, indicating the distribution range of the frequency characteristic value, and H j is the intensity characteristic value of the jth sound sample, A j is the duration feature value of the jth sound sample, It is a limit function, which means that as the number of samples increases, the value of AS tends to be stable. When the value of AS tends to 0, it means that the sound characteristics are very close to the normal environment and there is no abnormality. When the value of AS tends to 1, it means that the sound characteristics are significantly different from the normal environment and the degree of abnormality is high.

[0036] A further improvement of the technical solution of the present invention is that in the severity assessment module, the process of grading abnormal events is as follows:

[0037] Collect relevant historical data on abnormal ecological and environmental events, including the impact scope, potential hazards, emergency response measures and their effects of previous events, and obtain the latest data on identified abnormal ecological and environmental events and relevant data on current weather factors in real time, and analyze abnormal events from three dimensions: impact scope, potential hazards and urgency;

[0038] Combined with the data of historical ecological and environmental abnormal events, the current abnormal events are compared and analyzed with the data of historical ecological and environmental abnormal events. The geographic information system (GIS) technology is used to analyze the impact range of the abnormal events, quantify the size of the affected area, assess the ecological importance of the affected area, assess the impact of the abnormal events on the ecosystem, estimate the direct and indirect economic losses caused by the abnormal events, and assess the development speed of the abnormal events to determine whether they show a rapid deterioration trend, so as to determine the urgency of controlling the abnormal events;

[0039] Set quantitative indicators, quantify the scores of each assessment dimension such as impact scope, potential harm and urgency, calculate the severity assessment index, and analyze the trend of abnormal events based on the scoring results of the quantitative indicators;

[0040] Based on the abnormal image recognition index and the abnormal sound recognition index, the current abnormal event type is determined, and combined with the severity assessment index, the abnormal assessment index is obtained to analyze the severity of the current abnormal event;

[0041] Combining current weather factors and abnormal assessment indexes, we can obtain abnormal warning coefficients, predict the expected duration of abnormal events, and further determine the long-term interference and impact of abnormal events on the ecological environment;

[0042] Based on the historical ecological environment abnormal event data and the abnormal warning coefficient, the warning levels of the severity of abnormal events are analyzed and divided into low warning level, medium warning level and high warning level, and the corresponding warning threshold is determined for each warning level.

[0043] A further improvement of the technical solution of the present invention is that the expression of the severity assessment index is:

[0044]

[0045] Among them, SA is the severity assessment index, I g is the quantitative score of the g-th dimension, g is the score of the impact range, the score of potential harm or the score of urgency, and w g is the weight of the g-th dimension, which is allocated according to the importance of each dimension, and J g is the benchmark value of the g-th dimension, indicating the score of the normal state of the dimension, which is used to compare the actual score with the score of the normal state, σ g is the standard deviation of the g-th dimension, indicating the expected fluctuation range of the score, and is used to calculate the degree to which the score deviates from the baseline value. The value range of SA is between 0 and 1;

[0046] The expression of the abnormal evaluation index is:

[0047]

[0048] Among them, AWC is the abnormal assessment index, AP k is the abnormal image recognition index of the kth image recognition event, AS k is the abnormal sound recognition index of the kth sound recognition event, α k is the weight of the kth event, indicating the relative importance of the image and sound recognition index in the early warning coefficient, β k is the adjustment parameter of the kth event, which is used to control the sensitivity of the exponential function. SA is the severity assessment index. γ is the baseline value of severity assessment, which indicates the normal severity level. δ is the adjustment parameter of severity assessment, which is used to control the sensitivity of the exponential function. The result of AWC is quantified between 0 and 1. The closer the value is to 1, the more severe the warning is.

[0049] A further improvement of the technical solution of the present invention is that the abnormal warning coefficient is obtained based on the current weather factors and the abnormal evaluation index, and its expression is:

[0050]

[0051] Among them, AIC is the abnormal warning coefficient, AWC is u is the quantitative value of the u-th abnormal warning coefficient, reflecting the severity of warnings of different events, W u is the weight factor related to weather, considering the impact of different weather conditions on the degree of interference of abnormal events, T c is the temperature index of the current weather, T b is the weather reference value, indicating the ideal weather state with the least impact on the ecological environment. r is the range of weather indicators, reflecting the span from ideal to extreme conditions, p is the number of abnormal events, It is a limiting function, indicating that as time goes on infinitely, the interference coefficient tends to be stable, and the value range of AIC is between 0 and 1;

[0052] A plurality of the warning levels correspond to a plurality of the warning thresholds, wherein the warning thresholds include an upper threshold and a lower threshold;

[0053] The multiple warning levels and the multiple warning thresholds satisfy the following relationship:

[0054] Low alert level 0 <AIC≤AIC zy ; Small impact, may require monitoring and minor intervention;

[0055] Medium warning level AIC zy <AIC≤AIC gy ; The impact is obvious and certain emergency measures need to be taken;

[0056] High warning level AIC>AIC gy ; The impact is serious and requires immediate emergency response measures;

[0057] Among them, AIC is the abnormal warning coefficient, AIC zy is the lower threshold corresponding to the medium warning level and the upper threshold corresponding to the low warning level, AIC gy It is the lower threshold corresponding to the high warning level and the upper threshold corresponding to the medium warning level.

[0058] A further improvement of the technical solution of the present invention is that in the command and dispatch module, the process of dispatching resource allocation is:

[0059] Receive the type and location information of the abnormal event from the event detection and identification module, and receive the abnormal warning coefficient and warning level of the abnormal event from the severity assessment module;

[0060] Formulate emergency response strategies and dispatch instructions based on the type and severity of abnormal events, and determine the resources to be mobilized, including manpower, materials, equipment, etc.;

[0061] According to the emergency response plan, dispatch the required resources, determine the resource allocation plan, including the source, quantity and arrival time of the resources, and issue dispatch instructions to relevant departments and personnel through the command and dispatch platform;

[0062] Use audio and video communication methods to communicate with on-site personnel in real time, use video conferencing and voice broadcasting tools for remote command and coordination, and use GIS systems and Internet of Things technology to monitor resource scheduling progress and on-site conditions in real time to ensure that scheduling instructions are effectively executed and adjust emergency response plans and resource scheduling according to actual conditions;

[0063] Record all important information and data during the emergency response process and provide feedback to relevant personnel to report on the progress of abnormal event handling and resource usage.

[0064] In a second aspect, an ecological environment command and dispatch method based on audio and video fusion is implemented based on an ecological environment command and dispatch system based on audio and video fusion, and includes the following steps:

[0065] Step 1: Use pre-deployed environmental monitoring stations, drones, cameras, and sound sensor equipment to collect audio and video data and pre-process the collected data;

[0066] Step 2: Use image recognition and machine learning techniques to analyze the preprocessed audio and video data to identify abnormal events in the ecological environment;

[0067] Step 3: Combine historical ecological and environmental abnormal event data to analyze the impact scope, potential hazards and urgency of abnormal events, conduct quantitative assessment of current events, set the warning level of abnormal events, and determine the corresponding warning threshold;

[0068] Step 4: Based on the results of the event detection and identification module and the severity assessment module, formulate an emergency response plan and dispatch instructions, and communicate with on-site personnel in real time through audio and video communication means to optimize the allocation and dispatch of resources.

[0069] Due to the adoption of the above technical solution, the present invention has the following technical advances compared with the prior art:

[0070] 1. The present invention provides an ecological environment command and dispatch system and method based on audio and video fusion. Through real-time monitoring and automatic detection technology, abnormal events can be identified at the first time and the early warning mechanism can be immediately activated. Compared with traditional manual inspections, the time for discovering abnormal events can be reduced, thereby gaining precious time for taking emergency measures. In addition, the automation characteristics of the system reduce delays caused by human factors, ensuring that resources and personnel can be quickly mobilized in emergency situations to effectively control and alleviate the impact of events.

[0071] 2. The present invention provides an ecological environment command and dispatch system and method based on audio and video fusion. By integrating multiple sensors and monitoring equipment, a large amount of environmental data is collected, and historical data and real-time analysis are combined to provide comprehensive and accurate information support. In addition, the severity of abnormal events is determined by quantitatively evaluating the impact scope, potential hazards and urgency of the events, so as to assist in formulating response strategies, realize rapid integration and deployment of resources, and further improve the overall efficiency of emergency response. BRIEF DESCRIPTION OF THE DRAWINGS

[0072] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present invention. For ordinary technicians in this field, other drawings can also be obtained based on these drawings.

[0073] Figure 1 It is a module structure diagram of the present invention;

[0074] Figure 2 This is a flow chart for identifying abnormal events in the ecological environment of the present invention;

[0075] Figure 3 A flow chart for classifying abnormal events according to the present invention;

[0076] Figure 4 The figure is a flow chart of the method of the present invention. DETAILED DESCRIPTION

[0077] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0078] Embodiment 1, as Figure 1 , Figure 2 As shown, the present invention provides an ecological environment command and dispatch system based on audio and video fusion, including a command and dispatch platform, which is communicatively connected with a data acquisition module, a data preprocessing module, an event detection and identification module, a severity assessment module, a command and dispatch module, and a resource management module, wherein the modules are connected by electrical signals;

[0079] The data acquisition module is used to collect real-time audio and video data from various pre-deployed sensors and monitoring equipment. The data comes from environmental monitoring stations, drones, cameras, and sound sensors to obtain on-site video images and sound information, provide basic data for subsequent analysis and dispatch, and transmit the collected audio and video data to the command and dispatch platform. Advanced communication technology and network architecture, such as 4G / 5G, satellite communication, optical fiber, etc., are used to ensure the stability and efficiency of data transmission. Environmental monitoring stations, drones, cameras and sound sensors are pre-deployed at various monitoring points in the area to be monitored to ensure that all areas to be monitored can be covered. The environmental monitoring station provides environmental monitoring data and is equipped with cameras and sound sensors to capture on-site video images and sound information. The drone uses the high-definition camera and sound sensor it carries to conduct aerial inspections of specific areas and transmit on-site audio and video data in real time. The high maneuverability and flexibility of the drone enable it to quickly reach areas that are difficult for people to reach or dangerous. Fixed or mobile cameras are used to monitor specific areas domain or target, providing continuous video streaming data. Cameras are installed in key locations, such as ecological protection areas and near pollution sources. Sound sensors are used to capture and record sound information in the environment, providing sound dimension data for ecological environmental analysis. Sensors and various monitoring devices monitor environmental parameters in real time and obtain audio and video data. The data includes video images and sound information, covering the entire area where the incident occurred. The collected audio and video data are encoded and compressed to adapt to different network environments and reduce the demand for transmission bandwidth. Advanced communication technology and network architecture (such as 4G / 5G, satellite communication, optical fiber, etc.) are used to transmit the encoded and compressed data from the scene to the command and dispatch platform to ensure the efficiency and reliability of data transmission, and maintain the real-time nature of data even in long distances or complex environments. The command and dispatch platform receives data from various sensors and monitoring devices, integrates them into a unified data format, stores them on the cloud platform, and synchronizes the time of data collected from different sources to maintain the consistency of audio and video information for subsequent processing and analysis;

[0080] In the data preprocessing module, users process and analyze the collected audio and video data, including video encoding and decoding, audio noise reduction, and image enhancement to ensure the clarity and consistency of the data, and store and back up the audio and video data to ensure the security and traceability of the data, improve the processing efficiency and accuracy of the data, and prepare for further analysis. The original audio and video data stream is retrieved from the command and dispatch platform, and the video is encoded and decoded. The collected video data is decoded from the compressed format (such as H.264, HEVC, etc.) into the original video frame for subsequent processing. After the video processing is completed, according to the storage or transmission requirements, the video data is re-encoded into the corresponding compression format to reduce the data volume and optimize the transmission efficiency. The video frame is enhanced, such as brightness adjustment, contrast enhancement, color correction, etc., to improve the video clarity and Visual effects, and based on the frame difference method, the video is divided into different shot segments, which is convenient for subsequent key frame extraction and feature analysis. Key frames are extracted from each shot segment. The key frames reflect the main content of the shot. The noise reduction algorithm is applied to reduce or eliminate the noise components in the audio signal, improve the clarity and audibility of the audio, and help to capture the sound information in the environment more clearly. The processed and analyzed audio and video data are cleaned and formatted. Data cleaning removes outliers and incomplete data records to prevent them from affecting the accuracy of the analysis results. Data formatting converts data from different sources into a unified format for easy integration and analysis. The processed data is stored in a data warehouse for subsequent analysis. At the same time, the data is backed up regularly and redundant storage is set to ensure the security and recoverability of the data to prevent data loss or damage;

[0081] The event detection and recognition module combines image recognition and machine learning technology to analyze the processed audio and video data, identify abnormal events in the ecological environment, provide preliminary judgments on the events, and help formulate corresponding response strategies and priorities. It receives audio and video data from the data preprocessing module, and extracts sample data for training and testing from the processed audio and video data to ensure that the sample data covers various possible event types, such as forest fires, water pollution, invasive species invasion, drought, etc. It annotates the sample data and marks the characteristic areas of different abnormal event types. It uses computer vision technology to recognize images in video frames and extract image features from video frames, including color, texture, shape, and motion trajectory. It extracts features from audio data and obtains sound features, including frequency, intensity, and duration of sound. It uses OpenCV and other libraries to extract static features such as color, texture, and shape in video frames. It uses optical flow or background subtraction technology to extract dynamic features such as motion trajectory. It uses audio processing libraries (such as li brosa) analyzes audio signals, extracts features such as frequency, intensity, and duration, uses a large amount of annotated ecological environment abnormal event data, trains an abnormal event recognition model, and analyzes the extracted features to identify the difference between the normal environment background and the abnormal event, wherein the abnormal event recognition model is obtained based on a convolutional neural network model, and based on the association data of the abnormal event recognition model and image features, performs abnormal detection on the image, analyzes to obtain an abnormal image recognition index, detects abnormal images, inputs the processed sound feature data into the abnormal event recognition model, obtains an abnormal sound recognition index, detects abnormal sounds, integrates the detection results of abnormal images and abnormal sounds, analyzes and identifies abnormal events in the ecological environment, and classifies the detected abnormal events, determines the specific type of the abnormal event, and locates the location where the abnormal event occurs in combination with the geographic information of the video frame;

[0082] Furthermore, the expression of the abnormal image recognition index is:

[0083]

[0084] Among them, AP is the abnormal image recognition index, n is the number of frames in the video, and C i is the color feature value of the i-th frame image, T i is the texture feature value of the i-th frame image, F i is the shape feature value of the i-th frame image, M i is the motion feature value of the i-th frame image, e is the base of the natural logarithm, the AP value tends to 0, indicating that the image features are very close to the normal environment and there is no abnormality, and the AP value tends to 1, indicating that the image features are significantly different from the normal environment and the degree of abnormality is high. i and T i When the difference from the normal environment characteristic value increases, the exponential term decreases, resulting in an increase in AP, indicating an increase in the degree of abnormality. i and M i When it is larger, it will increase the AP value, indicating a more likely anomaly;

[0085] The expression of abnormal sound recognition index is:

[0086]

[0087] Among them, AS is the abnormal sound recognition index, s is the number of sound samples, L j is the frequency characteristic value of the jth sound sample, B is the reference frequency value, representing the typical frequency characteristic value under normal environmental conditions, S is the standard deviation, indicating the distribution range of the frequency characteristic value, and H j is the intensity characteristic value of the jth sound sample, A j is the duration feature value of the jth sound sample, is a limit function, indicating that as the number of samples increases, the value of AS tends to be stable. When the value of AS tends to 0, it means that the sound characteristics are very close to the normal environment and there is no abnormality. When the value of AS tends to 1, it means that the sound characteristics are significantly different from the normal environment and the degree of abnormality is high. j As the difference from B increases, the exponential term decreases, resulting in an increase in AS, indicating an increase in the degree of abnormality. j and A j When it is larger, it increases the value of AS, indicating a more likely abnormality;

[0088] The severity assessment module analyzes and evaluates the severity of abnormal events in the identified ecological environment based on the impact scope, potential hazards and urgency of the abnormal events in combination with historical data, sets the warning level of the severity of the abnormal events, and classifies the abnormal events. Through quantitative indicators and classification standards, it provides a scientific basis for the allocation of scheduling resources and the priority of emergency response;

[0089] The command and dispatch module is used to formulate and issue corresponding dispatch instructions and allocate dispatch resources according to the results of the event detection and identification module and the severity assessment module, combined with the warning level of the severity of the abnormal event. It can also realize real-time communication and command with on-site personnel through audio and video communication means, such as video conferencing and voice broadcasting. At the same time, it supports coordinated dispatch of multiple departments and regions to ensure the timeliness and effectiveness of emergency response.

[0090] The resource management module uniformly manages and optimizes the allocation of various resources in the dispatching system, including the dispatching and allocation of human, material and financial resources, as well as the dynamic monitoring and evaluation of resources, to ensure that the required resources can be quickly mobilized when an emergency occurs, providing strong support for emergency response.

[0091] Embodiment 2, as Figure 3 As shown, based on Example 1, the present invention provides a technical solution: Preferably, in the severity assessment module, the process of grading abnormal events is:

[0092] Collect relevant historical ecological and environmental abnormal event data, including the impact scope, potential hazards, emergency response measures and their effects of previous events, and obtain the latest data of identified ecological and environmental abnormal events and relevant data of current weather factors in real time. Analyze abnormal events from three dimensions: impact scope, potential hazards and urgency of events. Combined with historical ecological and environmental abnormal event data, compare and analyze current abnormal events with historical ecological and environmental abnormal event data. Use Geographic Information System (GIS) IS) technology is used to analyze the impact scope of abnormal events, quantify the size of the affected area, evaluate the ecological importance of the affected area, evaluate the impact of abnormal events on the ecosystem, estimate the direct and indirect economic losses caused by abnormal events, and evaluate the development speed of abnormal events to determine whether they show a rapid deterioration trend, so as to determine the urgency of controlling abnormal events, set quantitative indicators, quantify the scores of each evaluation dimension of the impact scope, potential hazards and urgency, calculate the severity assessment index, and analyze the trend of abnormal events based on the scoring results of quantitative indicators. The current abnormal event type is determined based on the abnormal image recognition index and the abnormal sound recognition index, and the abnormal assessment index is obtained in combination with the severity assessment index. The severity of the current abnormal event is analyzed, and the abnormal warning coefficient is obtained in combination with the current weather factors and the abnormal assessment index. The expected duration of the abnormal event is predicted, and the long-term interference and impact of the abnormal event on the ecological environment are further determined. According to the historical ecological environment abnormal event data and the abnormal warning coefficient, the warning level of the severity of the abnormal event is analyzed and divided into low warning level, medium warning level and high warning level, and the corresponding warning threshold is determined for each warning level;

[0093] Furthermore, the expression of the severity assessment index is:

[0094]

[0095] Among them, SA is the severity assessment index, I g is the quantitative score of the g-th dimension, g is the score of the impact range, the score of potential harm or the score of urgency, and w g is the weight of the g-th dimension, which is allocated according to the importance of each dimension, and J g is the benchmark value of the g-th dimension, indicating the score of the normal state of the dimension, which is used to compare the actual score with the score of the normal state, σ g is the standard deviation of the g-th dimension, indicating the expected fluctuation range of the score, and is used to calculate the degree to which the score deviates from the baseline value. The value range of SA is between 0 and 1. When the scores of all dimensions are close to the baseline value, SA approaches 0, indicating that the event is minor. When the score of any dimension deviates significantly from the baseline value, SA approaches 1, indicating that the event is serious.

[0096] The expression of the abnormal evaluation index is:

[0097]

[0098] Among them, AWC is the abnormal assessment index, AP k is the abnormal image recognition index of the kth image recognition event, AS k is the abnormal sound recognition index of the kth sound recognition event, α k is the weight of the kth event, indicating the relative importance of the image and sound recognition index in the early warning coefficient, β k is the adjustment parameter of the kth event, which is used to control the sensitivity of the exponential function. SA is the severity assessment index. γ is the baseline value of severity assessment, which indicates the normal severity level. δ is the adjustment parameter of severity assessment, which is used to control the sensitivity of the exponential function. The result of AWC is quantified between 0 and 1. The closer the value is to 1, the more serious the warning is. When the values ​​of AP and AS are close to each other, and the value of SA is close to the baseline value γ, AWC approaches 0, indicating a low warning level. When the values ​​of AP or AS deviate significantly from each other, or the value of SA is significantly higher than γ, AWC approaches 1, indicating a high warning level.

[0099] Furthermore, the abnormal warning coefficient is obtained based on the current weather factors and the abnormal assessment index, and its expression is:

[0100]

[0101] Among them, AIC is the abnormal warning coefficient, AWC is u is the quantitative value of the u-th abnormal warning coefficient, reflecting the severity of warnings of different events, W u is the weight factor related to weather, considering the impact of different weather conditions on the degree of interference of abnormal events, T c is the temperature index of the current weather, T b is the weather reference value, indicating the ideal weather state with the least impact on the ecological environment. r is the range of weather indicators, reflecting the span from ideal to extreme conditions, p is the number of abnormal events, It is a limit function, indicating that as time goes by, the interference coefficient tends to be stable. The value range of AIC is between 0 and 1. u and W u The value of is low, and the current weather is close to the benchmark value T b When AIC approaches 0, it means the interference is low. u and W u The value of is high, or the current weather is far from the reference value T b When , AIC approaches 1, indicating a high degree of interference;

[0102] Multiple warning levels correspond to multiple warning thresholds, where the warning thresholds include an upper threshold and a lower threshold;

[0103] Multiple warning levels and multiple warning thresholds satisfy the following relationship:

[0104] Low alert level 0 <AIC≤AIC zy ; Small impact, may require monitoring and minor intervention;

[0105] Medium warning level AIC zy <AIC≤AIC gy ; The impact is obvious and certain emergency measures need to be taken;

[0106] High warning level AIC>AIC gy ; The impact is serious and requires immediate emergency response measures;

[0107] Among them, AIC is the abnormal warning coefficient, AIC zy is the lower threshold corresponding to the medium warning level and the upper threshold corresponding to the low warning level, AIC gy The lower threshold corresponding to the high warning level and the upper threshold corresponding to the medium warning level;

[0108] In the command and dispatch module, the process of dispatch resource allocation is as follows:

[0109] Receive the type and location information of abnormal events from the event detection and identification module, and receive the abnormal warning coefficient and warning level of abnormal events from the severity assessment module. According to the type of abnormal events and the severity assessment results, formulate emergency response strategies and dispatch instructions, determine the resources to be mobilized, including manpower, materials, equipment, etc., dispatch the required resources according to the emergency response plan, determine the resource allocation plan, including the source, quantity and arrival time of the resources, and issue dispatch instructions to relevant departments and personnel through the command and dispatch platform. Use audio and video communication means to communicate with on-site personnel in real time, and use video conferencing and voice broadcasting tools for remote command and coordination. Use GIS system and Internet of Things technology to monitor the progress of resource dispatch and on-site conditions in real time to ensure that the dispatch instructions are effectively executed, adjust the emergency response plan and resource dispatch according to actual conditions, record all important information and data in the emergency response process, and feedback to relevant personnel to report the progress of abnormal event handling and resource usage.

[0110] Embodiment 3, as Figure 4 As shown, on the basis of Embodiment 1-2, the present invention further provides an ecological environment command and dispatch method based on audio and video fusion, which is implemented based on an ecological environment command and dispatch system based on audio and video fusion, and includes the following steps:

[0111] Step 1: Use pre-deployed environmental monitoring stations, drones, cameras, and sound sensor equipment to collect audio and video data and pre-process the collected data;

[0112] Step 2: Use image recognition and machine learning techniques to analyze the preprocessed audio and video data to identify abnormal events in the ecological environment;

[0113] Step 3: Combine historical ecological and environmental abnormal event data to analyze the impact scope, potential hazards and urgency of abnormal events, conduct quantitative assessment of current events, set the warning level of abnormal events, and determine the corresponding warning threshold;

[0114] Step 4: Based on the results of the event detection and identification module and the severity assessment module, formulate an emergency response plan and dispatch instructions, and communicate with on-site personnel in real time through audio and video communication means to optimize the allocation and dispatch of resources.

[0115] The above are only specific implementations of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. An ecological environment command and dispatch system based on audio and video fusion, including a command and dispatch platform, characterized in that: The command and dispatch platform is communicatively connected with a data acquisition module, a data preprocessing module, an event detection and identification module, a severity assessment module, a command and dispatch module, and a resource management module, wherein electrical signals are connected between the modules; The data acquisition module is used to collect real-time audio and video data from various pre-deployed sensors and monitoring equipment, and transmit the collected audio and video data to the command and dispatch platform; The data preprocessing module allows the user to process and analyze the collected audio and video data, and store and back up the audio and video data; The event detection and recognition module combines image recognition and machine learning technology to analyze the processed audio and video data and identify abnormal events in the ecological environment; The severity assessment module analyzes and assesses the severity of the abnormal events in the identified ecological environment in combination with historical data based on the impact scope, potential hazards and urgency of the abnormal events, sets an early warning level for the severity of the abnormal events, and classifies the abnormal events; The command and dispatch module is used to formulate and issue corresponding dispatch instructions and allocate dispatch resources according to the results of the event detection and identification module and the severity assessment module, combined with the determined warning level of the severity of the abnormal event; The resource management module uniformly manages and optimizes the configuration of various resources in the scheduling system.

2. The ecological environment command and dispatch system based on audio and video fusion according to claim 1 is characterized by: In the data acquisition module, the process of collecting real-time audio and video data is as follows: Pre-deploy environmental monitoring stations, drones, cameras and sound sensors at each monitoring point in the area to be monitored; Sensors and monitoring devices monitor environmental parameters in real time, obtain audio and video data, including video images and sound information, and encode and compress the collected audio and video data; Utilize communication technology and network architecture to transmit encoded and compressed data from the field to the command and dispatch platform; The command and dispatch platform receives data from various sensors and monitoring equipment, integrates them into a unified data format, stores them on the cloud platform, and synchronizes the time of data collected from different sources.

3. The ecological environment command and dispatch system based on audio and video fusion according to claim 2 is characterized by: In the data preprocessing module, the process of storing and backing up the audio and video data is as follows: Retrieve the original audio and video data stream from the command and dispatch platform, perform encoding and decoding on the video, decode the collected video data from the compressed format into the original video frame, and after the video processing is completed, re-encode the video data into the corresponding compression format according to the storage or transmission requirements; Enhance the video frames and divide the video into different shot segments based on the frame difference method. Extract key frames from each shot segment. The key frames reflect the main content of the shot. Apply noise reduction algorithms to reduce or eliminate noise components in audio signals; The processed and analyzed audio and video data is cleaned and formatted, and the processed data is stored in the data warehouse, and the data is backed up regularly.

4. The ecological environment command and dispatch system based on audio and video fusion according to claim 3 is characterized by: In the event detection and identification module, the identification process of abnormal events in the ecological environment is as follows: Receive audio and video data from the data preprocessing module, extract sample data for training and testing from the processed audio and video data, annotate the sample data, and mark the characteristic areas of different abnormal event types; Use computer vision technology to identify images in video frames and extract image features from video frames, including color, texture, shape, and motion trajectory. Perform feature extraction on audio data to obtain sound features, including frequency, intensity, and duration of the sound. Using the labeled ecological environment abnormal event data, the abnormal event recognition model is trained, and the extracted features are analyzed to identify the difference between the normal environmental background and the abnormal events, wherein the abnormal event recognition model is obtained based on the convolutional neural network model; Based on the abnormal event recognition model and the associated data of image features, the image is detected for abnormality, the abnormal image recognition index is obtained through analysis, the abnormal image is detected, the processed sound feature data is input into the abnormal event recognition model, the abnormal sound recognition index is obtained, and the abnormal sound is detected; The detection results of abnormal images and abnormal sounds are integrated to analyze and identify abnormal events in the ecological environment, classify the detected abnormal events, determine the specific type of abnormal events, and locate them in combination with the geographic information of the video frames to determine the location where the abnormal events occurred.

5. The ecological environment command and dispatch system based on audio and video fusion according to claim 4 is characterized by: The expression of the abnormal image recognition index is: Among them, AP is the abnormal image recognition index, n is the number of frames in the video, and C i is the color feature value of the i-th frame image, T i is the texture feature value of the i-th frame image, F i is the shape feature value of the i-th frame image, M i is the motion feature value of the i-th frame image, and e is the base of the natural logarithm; The expression of the abnormal sound recognition index is: Among them, AS is the abnormal sound recognition index, s is the number of sound samples, L j is the frequency characteristic value of the jth sound sample, B is the reference frequency value, representing the typical frequency characteristic value under normal environmental conditions, S is the standard deviation, indicating the distribution range of the frequency characteristic value, and H j is the intensity characteristic value of the jth sound sample, A j is the duration feature value of the jth sound sample.

6. The ecological environment command and dispatch system based on audio and video fusion according to claim 5 is characterized by: In the severity assessment module, the process of grading abnormal events is as follows: Collect relevant historical data on abnormal ecological and environmental events, including the impact scope, potential hazards, emergency response measures and their effects of previous events, and obtain the latest data on identified abnormal ecological and environmental events and relevant data on current weather factors in real time, and analyze abnormal events from three dimensions: impact scope, potential hazards and urgency; Combined with the data of historical ecological and environmental abnormal events, the current abnormal events are compared and analyzed with the data of historical ecological and environmental abnormal events. The impact range of abnormal events is analyzed using geographic information system technology, the size of the affected area is quantified, the ecological importance of the affected area is assessed, the impact of abnormal events on the ecosystem is assessed, the direct and indirect economic losses caused by abnormal events are estimated, and the development speed of abnormal events is assessed to determine whether they show a rapid deterioration trend, so as to determine the urgency of controlling abnormal events; Set quantitative indicators, quantify the scores of each assessment dimension such as impact scope, potential harm and urgency, calculate the severity assessment index, and analyze the trend of abnormal events based on the scoring results of the quantitative indicators; Based on the abnormal image recognition index and the abnormal sound recognition index, the current abnormal event type is determined, and combined with the severity assessment index, the abnormal assessment index is obtained to analyze the severity of the current abnormal event; Combining current weather factors and abnormal assessment indexes, we can obtain abnormal warning coefficients, predict the expected duration of abnormal events, and further determine the long-term interference and impact of abnormal events on the ecological environment; Based on the historical ecological environment abnormal event data and the abnormal warning coefficient, the warning levels of the severity of abnormal events are analyzed and divided into low warning level, medium warning level and high warning level, and the corresponding warning threshold is determined for each warning level.

7. The ecological environment command and dispatch system based on audio and video fusion according to claim 6 is characterized by: The expression of the severity assessment index is: Among them, SA is the severity assessment index, I g is the quantitative score of the g-th dimension, g is the score of the impact range, the score of potential harm or the score of urgency, and w g is the weight of the g-th dimension, which is allocated according to the importance of each dimension, and J g is the reference value of the g-th dimension, σ g is the standard deviation of the g-th dimension; The expression of the abnormal evaluation index is: Among them, AWC is the abnormal assessment index, AP k is the abnormal image recognition index of the kth image recognition event, AS k is the abnormal sound recognition index of the kth sound recognition event, α k is the weight of the kth event, β k is the adjustment parameter of the kth event, SA is the severity assessment index, γ is the baseline value of severity assessment, indicating the normal severity level, and δ is the adjustment parameter of severity assessment.

8. The ecological environment command and dispatch system based on audio and video fusion according to claim 7 is characterized by: The abnormal warning coefficient is obtained based on the current weather factors and the abnormal evaluation index, and its expression is: Among them, AIC is the abnormal warning coefficient, AWC is u is the quantitative value of the u-th abnormal warning coefficient, reflecting the severity of warnings of different events, W u is the weight factor related to weather, T c is the temperature index of the current weather, T b is the weather reference value, T r is the range of weather index changes, p is the number of abnormal events; A plurality of the warning levels correspond to a plurality of the warning thresholds, wherein the warning thresholds include an upper threshold and a lower threshold; The multiple warning levels and the multiple warning thresholds satisfy the following relationship: Low alert level 0 <AIC≤AIC zy ; Medium warning level AIC zy <AIC≤AIC gy ; High warning level AIC>AIC gy ; Among them, AIC is the abnormal warning coefficient, AIC zy is the lower threshold corresponding to the medium warning level and the upper threshold corresponding to the low warning level, AIC gy It is the lower threshold corresponding to the high warning level and the upper threshold corresponding to the medium warning level.

9. The ecological environment command and dispatch system based on audio and video fusion according to claim 8 is characterized by: In the command and dispatch module, the process of dispatching resource allocation is as follows: Receive the type and location information of the abnormal event from the event detection and identification module, and receive the abnormal warning coefficient and warning level of the abnormal event from the severity assessment module; Develop emergency response strategies and dispatch instructions based on the type and severity of abnormal events and determine the resources to be mobilized; According to the emergency response plan, dispatch the required resources, determine the resource allocation plan, including the source, quantity and arrival time of the resources, and issue dispatch instructions to relevant departments and personnel through the command and dispatch platform; Use audio and video communication methods to communicate with on-site personnel in real time, use video conferencing and voice broadcasting tools for remote command and coordination, use GIS systems and Internet of Things technology to monitor resource scheduling progress and on-site conditions in real time, and adjust emergency response plans and resource scheduling according to actual conditions; Record all important information and data during the emergency response process and provide feedback to relevant personnel to report on the progress of abnormal event handling and resource usage.

10. An ecological environment command and dispatch method based on audio and video fusion, implemented based on an ecological environment command and dispatch system based on audio and video fusion according to any one of claims 1 to 9, characterized in that: The following steps are involved: Step 1: Use pre-deployed environmental monitoring stations, drones, cameras, and sound sensor equipment to collect audio and video data and pre-process the collected data; Step 2: Use image recognition and machine learning techniques to analyze the preprocessed audio and video data to identify abnormal events in the ecological environment; Step 3: Combine historical ecological and environmental abnormal event data to analyze the impact scope, potential hazards and urgency of abnormal events, conduct quantitative assessment of current events, set the warning level of abnormal events, and determine the corresponding warning threshold; Step 4: Based on the results of the event detection and identification module and the severity assessment module, formulate an emergency response plan and dispatch instructions, and communicate with on-site personnel in real time through audio and video communication means to optimize the allocation and dispatch of resources.

Images