An ecological environment command and dispatch system and method based on audio and video fusion

Abstract

The present invention discloses an ecological environment command and dispatch system and method based on audio and video fusion, which relates to the field of ecological environment dispatch technology, including a command and dispatch platform, wherein the command and dispatch platform is communicatively connected with a data acquisition module, a data preprocessing module, an event detection and identification module, a severity assessment module, a command and dispatch module, and a resource management module, wherein the modules are connected by electrical signals; the data acquisition module is used to collect real-time audio and video data from various pre-deployed sensors and monitoring equipment. The present invention uses real-time monitoring and automatic detection technology to identify abnormal events at the first time and immediately activate the early warning mechanism. Compared with traditional manual inspections, it can reduce the time to discover abnormal events. In addition, the automation characteristics of the system reduce delays caused by human factors, ensuring that resources and personnel can be quickly mobilized in an emergency to effectively control and alleviate the impact of the event.

Description

Technical Field

[0001] The present invention relates to the technical field of ecological environment scheduling, and in particular to an ecological environment command and scheduling system and method based on audio and video fusion. Background Art

[0002] As global environmental problems become increasingly severe, ecological and environmental protection has become a focus of attention from all walks of life. In order to effectively respond to challenges such as environmental pollution and ecological destruction, it is necessary to establish an efficient and intelligent command and dispatch system to achieve comprehensive monitoring, rapid response and scientific decision-making of the ecological environment. Audio and video fusion technology is a technology that integrates and processes multiple media information such as audio and video. With the continuous advancement of IP technology, soft switching technology, video encoding and decoding technology, audio and video fusion technology has become highly integrated, real-time and interactive. These technical characteristics have made audio and video fusion technology widely used in ecological and environmental command and dispatch systems.

[0003] For example, the Chinese patent publication number: CN111950842A is a blockchain-based ecological environment comprehensive scheduling method and system, and the specific steps are as follows: S1: according to the type of ecological environment that needs to be controlled, different types of ecological environment information collection platforms are set up, and each information collection platform sets up an information processing scheduling room in a different collection area; S2: each regional chain storage module is provided with multiple regional blocks, and the information processing scheduling room sends the ecological environment information to a regional block in the corresponding regional chain storage module for storage; S3: a texture editing module and a database are provided in the comprehensive scheduling room; S4: the comprehensive scheduling room sends the scheduling information to the information processing scheduling room for ecological environment management scheduling or remote monitoring.

[0004] In the existing technology, although the dispatching system has early warning and dispatching functions, it lacks effective severity analysis measures for sudden ecological and environmental events, such as forest fires, water pollution, and invasive species invasions. It is difficult to provide corresponding degrees of early warning for sudden ecological and environmental events in different states, which easily leads to waste of dispatching resources and inability to respond quickly when conducting command and dispatch. The dispatching system has certain limitations. Therefore, how to analyze ecological and environmental events and judge their severity in order to achieve rapid response and effective dispatch of resources is the problem we need to solve. To this end, an ecological and environmental command and dispatch system and method based on audio and video fusion is proposed. Summary of the Invention

[0005] The purpose of the present invention is to provide an ecological environment command and dispatch system and method based on audio and video fusion to solve the problems raised in the above background technology.

[0006] In order to solve the above technical problems, the technical solution adopted by the present invention is:

[0007] In a first aspect, an ecological environment command and dispatch system based on audio and video fusion includes a command and dispatch platform, wherein the command and dispatch platform is communicatively connected to a data acquisition module, a data preprocessing module, an event detection and identification module, a severity assessment module, a command and dispatch module, and a resource management module, wherein the modules are electrically connected;

[0008] The data acquisition module is used to collect real-time audio and video data from various pre-deployed sensors and monitoring equipment. The data comes from environmental monitoring stations, drones, cameras, and sound sensors, obtaining on-site video and sound information to provide basic data for subsequent analysis and dispatch. The collected audio and video data is then transmitted to the command and dispatch platform, using advanced communication technologies and network architectures such as 4G / 5G, satellite communications, and optical fiber to ensure the stability and efficiency of data transmission.

[0009] The data preprocessing module allows users to process and analyze the collected audio and video data, including video encoding and decoding, audio noise reduction, and image enhancement to ensure data clarity and consistency. It also stores and backs up audio and video data to ensure data security and traceability, improve data processing efficiency and accuracy, and prepare for further analysis.

[0010] The event detection and identification module combines image recognition and machine learning technologies to analyze the processed audio and video data, identify abnormal events in the ecological environment, provide preliminary judgments on the events, and help formulate corresponding response strategies and priorities;

[0011] The severity assessment module analyzes and assesses the severity of identified abnormal events in the ecological environment based on historical data, the scope of impact, potential hazards, and urgency of the abnormal events, sets warning levels for the severity of the abnormal events, and classifies the abnormal events. Through quantitative indicators and classification standards, it provides a scientific basis for the allocation of scheduling resources and the priority of emergency response.

[0012] The command and dispatch module is used to formulate and issue corresponding dispatch instructions based on the results of the event detection and identification module and the severity assessment module, combined with the determined warning level of the abnormal event severity, and allocate dispatch resources. It uses audio and video communication methods such as video conferencing and voice broadcasting to achieve real-time communication and command with on-site personnel. At the same time, it supports multi-department and multi-regional coordinated dispatch to ensure the timeliness and effectiveness of emergency response;

[0013] The resource management module uniformly manages and optimizes the allocation of various resources in the scheduling system, including the scheduling and allocation of human, material and financial resources, as well as the dynamic monitoring and evaluation of resources, to ensure that the required resources can be quickly mobilized when an emergency occurs, providing strong support for emergency response.

[0014] A further improvement of the technical solution of the present invention is that: in the data acquisition module, the process of collecting real-time audio and video data is:

[0015] Environmental monitoring stations, drones, cameras, and sound sensors are pre-deployed at each monitoring point in the area to be monitored to ensure coverage of all areas that need to be monitored. The environmental monitoring stations provide environmental monitoring data and are equipped with cameras and sound sensors to capture on-site video and sound information. Drones use their high-definition cameras and sound sensors to conduct aerial inspections of specific areas and transmit on-site audio and video data in real time. The high maneuverability and flexibility of drones enable them to quickly reach areas that are difficult for humans to reach or dangerous. Fixed or mobile cameras are used to monitor specific areas or targets and provide continuous video streaming data. Cameras are installed in key locations, such as ecological protection areas and near pollution sources. Sound sensors are used to capture and record sound information in the environment, providing sound-dimensional data for ecological and environmental analysis.

[0016] Sensors and monitoring equipment monitor environmental parameters in real time and acquire audio and video data, including video and sound information, covering the entire area where the incident occurred. The collected audio and video data is encoded and compressed to adapt to different network environments and reduce transmission bandwidth requirements.

[0017] Utilize advanced communication technologies and network architectures (such as 4G / 5G, satellite communications, and optical fiber) to transmit encoded and compressed data from the field to the command and dispatch platform, ensuring efficient and reliable data transmission and maintaining real-time data even over long distances or in complex environments.

[0018] The command and dispatch platform receives data from various sensors and monitoring equipment, integrates it into a unified data format, stores it on the cloud platform, and synchronizes the time of data collected from different sources to maintain the consistency of audio and video information for subsequent processing and analysis.

[0019] A further improvement of the technical solution of the present invention is that: in the data preprocessing module, the process of storing and backing up the audio and video data is as follows:

[0020] Retrieve the original audio and video data stream from the command and dispatch platform, perform video encoding and decoding, and decode the collected video data from the compressed format (such as H.264, HEVC, etc.) into the original video frames for subsequent processing. After the video processing is completed, re-encode the video data into the corresponding compression format according to storage or transmission requirements to reduce data volume and optimize transmission efficiency;

[0021] Perform enhancement processing on video frames, such as brightness adjustment, contrast enhancement, and color correction, to improve video clarity and visual effects. Based on the inter-frame difference method, the video is segmented into different shot segments to facilitate subsequent key frame extraction and feature analysis. Key frames are extracted from each shot segment, and the key frames reflect the main content of the shot.

[0022] Applying noise reduction algorithms to reduce or eliminate noise components in audio signals, improving audio clarity and audibility, and helping to more clearly capture sound information in the environment;

[0023] The processed and analyzed audio and video data is cleaned and formatted. Data cleaning removes outliers and incomplete data records to prevent them from affecting the accuracy of the analysis results. Data formatting converts data from different sources into a unified format for easy integration and analysis. The processed data is stored in a data warehouse for subsequent analysis. At the same time, the data is backed up regularly and redundant storage is set up to ensure data security and recoverability to prevent data loss or damage.

[0024] A further improvement of the technical solution of the present invention is that in the event detection and identification module, the identification process of abnormal events in the ecological environment is:

[0025] Receive audio and video data from the data preprocessing module and extract sample data for training and testing from the processed audio and video data, ensuring that the sample data covers various possible event types, such as forest fires, water pollution, invasive species invasions, droughts, etc. The sample data is annotated to mark the characteristic areas of different abnormal event types;

[0026] Use computer vision technology to identify images in video frames and extract image features from video frames, including color, texture, shape, and motion trajectory. Perform feature extraction on audio data to obtain sound characteristics, including frequency, intensity, and duration. Use libraries such as OpenCV to extract static features such as color, texture, and shape from video frames. Use optical flow or background subtraction technology to extract dynamic features such as motion trajectory. Use audio processing libraries (such as librosa) to analyze audio signals and extract features such as frequency, intensity, and duration.

[0027] Using a large amount of labeled ecological and environmental abnormal event data, we trained an abnormal event recognition model and analyzed the extracted features to identify the differences between normal environmental backgrounds and abnormal events. The abnormal event recognition model was obtained based on a convolutional neural network model.

[0028] Based on the abnormal event recognition model and the associated data of image features, the image is detected for abnormality, and the abnormal image recognition index is obtained through analysis to detect abnormal images. The processed sound feature data is input into the abnormal event recognition model to obtain the abnormal sound recognition index to detect abnormal sounds.

[0029] The detection results of abnormal images and abnormal sounds are integrated to analyze and identify abnormal events in the ecological environment, classify the detected abnormal events, determine the specific type of abnormal events, and locate them in combination with the geographic information of the video frames to determine the location where the abnormal events occurred.

[0030] A further improvement of the technical solution of the present invention is that the expression of the abnormal image recognition index is:

[0031]

[0032] Among them, AP is the abnormal image recognition index, n is the number of frames in the video, C i is the color feature value of the i-th frame image, T i is the texture feature value of the i-th frame image, F i is the shape feature value of the i-th frame image, M i is the motion feature value of the i-th frame image, e is the base of the natural logarithm, and the AP value tends to 0, indicating that the image features are very close to the normal environment and there is no abnormality. The AP value tends to 1, indicating that the image features are significantly different from the normal environment and the degree of abnormality is high.

[0033] The expression of the abnormal sound recognition index is:

[0034]

[0035] Among them, AS is the abnormal sound recognition index, s is the number of sound samples, L j is the frequency characteristic value of the j-th sound sample, B is the reference frequency value, representing the typical frequency characteristic value under normal environmental conditions, S is the standard deviation, indicating the distribution range of the frequency characteristic value, and H j is the intensity characteristic value of the j-th sound sample, A j is the duration characteristic value of the j-th sound sample, It is a limit function, indicating that as the number of samples increases, the value of AS tends to be stable. When the value of AS tends to 0, it means that the sound characteristics are very close to the normal environment and there is no abnormality. When the value of AS tends to 1, it means that the sound characteristics are significantly different from the normal environment and the degree of abnormality is high.

[0036] A further improvement of the technical solution of the present invention is that: in the severity assessment module, the process of grading abnormal events is:

[0037] Collect relevant historical data on abnormal ecological and environmental events, including the impact scope, potential hazards, emergency response measures and their effectiveness of previous events, and obtain the latest data on identified abnormal ecological and environmental events and relevant data on current weather factors in real time, and analyze abnormal events from three dimensions: impact scope, potential hazards and urgency;

[0038] Combined with historical data on abnormal ecological and environmental events, current abnormal events are compared and analyzed with historical data on abnormal ecological and environmental events. Geographic Information System (GIS) technology is used to analyze the impact range of abnormal events, quantify the size of the affected area, assess the ecological importance of the affected area, evaluate the impact of abnormal events on the ecosystem, estimate the direct and indirect economic losses caused by abnormal events, and evaluate the development speed of abnormal events to determine whether they show a rapid deterioration trend, so as to determine the urgency of controlling abnormal events;

[0039] Set quantitative indicators to quantify the scores of each assessment dimension, including the scope of impact, potential harm, and urgency, calculate the severity assessment index, and analyze the trend of abnormal events based on the scoring results of the quantitative indicators;

[0040] Based on the abnormal image recognition index and the abnormal sound recognition index, the current abnormal event type is determined. Combined with the severity assessment index, the abnormal assessment index is obtained to analyze the severity of the current abnormal event.

[0041] Combining current weather factors and anomaly assessment indexes, we can obtain anomaly warning coefficients, predict the expected duration of abnormal events, and further determine the long-term interference and impact of abnormal events on the ecological environment.

[0042] Based on historical ecological and environmental abnormal event data and abnormal warning coefficients, the warning levels of the severity of abnormal events are analyzed and divided into low warning level, medium warning level and high warning level, and the corresponding warning threshold is determined for each warning level.

[0043] A further improvement of the technical solution of the present invention is that the expression of the severity assessment index is:

[0044]

[0045] Among them, SA is the severity assessment index, I g is the quantitative score of the g-th dimension, g is the score of the impact range, the score of potential harm or the score of urgency, and w g is the weight of the g-th dimension, which is allocated according to the importance of each dimension, and J g is the benchmark value of the g-th dimension, indicating the score of the normal state of the dimension, which is used to compare the actual score with the score of the normal state, σ g is the standard deviation of the g-th dimension, which indicates the expected fluctuation range of the score and is used to calculate the degree to which the score deviates from the baseline value. The value range of SA is between 0 and 1;

[0046] The expression of the abnormality assessment index is:

[0047]

[0048] Among them, AWC is the abnormal assessment index, AP k is the abnormal image recognition index of the kth image recognition event, AS k is the abnormal sound recognition index of the kth sound recognition event, α k is the weight of the kth event, indicating the relative importance of image and sound recognition index in the early warning coefficient, β k is the adjustment parameter of the kth event, which is used to control the sensitivity of the exponential function. SA is the severity assessment index. γ is the baseline value of severity assessment, which indicates the normal severity level. δ is the adjustment parameter of severity assessment, which is used to control the sensitivity of the exponential function. The result of AWC is quantified between 0 and 1. The closer the value is to 1, the more severe the warning is.

[0049] A further improvement of the technical solution of the present invention is that the abnormal warning coefficient is obtained based on the current weather factors and the abnormal evaluation index, and its expression is:

[0050]

[0051] Among them, AIC is the abnormal warning coefficient, AWC u is the quantitative value of the u-th abnormal warning coefficient, reflecting the severity of warnings of different events, W u is a weight factor related to weather, considering the impact of different weather conditions on the degree of interference of abnormal events, T c is the temperature index of the current weather, T b is the weather reference value, which indicates the ideal weather state with the least impact on the ecological environment. r is the range of weather indicators, reflecting the span from ideal to extreme conditions, p is the number of abnormal events, It is a limiting function, indicating that as time goes on infinitely, the interference coefficient tends to be stable, and the value range of AIC is between 0 and 1;

[0052] The plurality of warning levels correspond to the plurality of warning thresholds, wherein the warning thresholds include an upper threshold and a lower threshold;

[0053] The multiple warning levels and the multiple warning thresholds satisfy the following relationship:

[0054] Low alert level 0 <AIC≤AIC zy ; The impact is minor and may require monitoring and mild intervention;

[0055] Medium warning level AIC zy <AIC≤AIC gy ; The impact is obvious and certain emergency measures need to be taken;

[0056] High warning level AIC>AIC gy ; The impact is serious and requires immediate emergency response measures;

[0057] Among them, AIC is the abnormal warning coefficient, AIC zy is the lower threshold corresponding to the medium warning level and the upper threshold corresponding to the low warning level, AIC gy The lower threshold corresponding to the high warning level and the upper threshold corresponding to the medium warning level.

[0058] A further improvement of the technical solution of the present invention is that: in the command and dispatch module, the process of dispatching resource allocation is:

[0059] Receive the type and location information of the abnormal event from the event detection and identification module, and receive the abnormal warning coefficient and warning level of the abnormal event from the severity assessment module;

[0060] Develop emergency response strategies and dispatch instructions based on the type and severity of abnormal events, and determine the resources to be mobilized, including manpower, supplies, equipment, etc.;

[0061] According to the emergency response plan, dispatch the required resources, determine the resource allocation plan, including the source, quantity and arrival time of the resources, and issue dispatch instructions to relevant departments and personnel through the command and dispatch platform;

[0062] Utilize audio and video communication methods to communicate with on-site personnel in real time, and use video conferencing and voice broadcasting tools for remote command and coordination. Utilize GIS systems and IoT technologies to monitor resource dispatch progress and on-site conditions in real time to ensure that dispatch instructions are effectively executed and adjust emergency response plans and resource dispatch based on actual conditions.

[0063] Record all important information and data during the emergency response process and provide feedback to relevant personnel to report the progress of abnormal event handling and resource usage.

[0064] In a second aspect, an ecological environment command and dispatch method based on audio and video fusion is implemented based on an ecological environment command and dispatch system based on audio and video fusion, comprising the following steps:

[0065] Step 1: Use pre-deployed environmental monitoring stations, drones, cameras, and sound sensor equipment to collect audio and video data and pre-process the collected data;

[0066] Step 2: Use image recognition and machine learning techniques to analyze the pre-processed audio and video data to identify abnormal events in the ecological environment;

[0067] Step 3: Combine historical data on abnormal ecological and environmental events to analyze the impact, potential hazards, and urgency of abnormal events, conduct a quantitative assessment of the current event, set an early warning level for the abnormal event, and determine the corresponding early warning threshold;

[0068] Step 4: Based on the results of the event detection and identification module and the severity assessment module, formulate an emergency response plan and dispatch instructions, and communicate with on-site personnel in real time through audio and video communication means to optimize the allocation and dispatch of resources.

[0069] Due to the adoption of the above technical solution, the present invention has the following technical advancements compared to the prior art:

[0070] 1. The present invention provides an ecological environment command and dispatch system and method based on audio and video fusion. Through real-time monitoring and automatic detection technology, abnormal events can be identified in the first time and the early warning mechanism can be immediately activated. Compared with traditional manual inspections, the time to discover abnormal events can be reduced, thereby gaining valuable time for taking emergency measures. In addition, the automation characteristics of the system reduce delays caused by human factors, ensuring that resources and personnel can be quickly mobilized in emergency situations to effectively control and mitigate the impact of events.

[0071] 2. The present invention provides an ecological environment command and dispatch system and method based on audio and video fusion. By integrating multiple sensors and monitoring equipment, a large amount of environmental data is collected, and combined with historical data and real-time analysis, comprehensive and accurate information support is provided. In addition, by quantitatively evaluating the impact range, potential hazards and urgency of the event, the severity of the abnormal event is determined to assist in formulating response strategies, realize rapid integration and deployment of resources, and further improve the overall efficiency of emergency response. BRIEF DESCRIPTION OF THE DRAWINGS

[0072] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments described in the present invention. For ordinary technicians in this field, other drawings can also be obtained based on these drawings.

[0073] Figure 1 It is a module structure diagram of the present invention;

[0074] Figure 2 This is a flow chart for identifying abnormal events in the ecological environment of the present invention;

[0075] Figure 3 A flow chart for classifying abnormal events according to the present invention;

[0076] Figure 4 Flow chart of the method of the present invention. DETAILED DESCRIPTION

[0077] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0078] Example 1, as Figure 1 、 Figure 2 As shown, the present invention provides an ecological environment command and dispatch system based on audio and video fusion, including a command and dispatch platform, which is communicatively connected to a data acquisition module, a data preprocessing module, an event detection and identification module, a severity assessment module, a command and dispatch module, and a resource management module, wherein electrical signals are connected between the modules;

[0079] The data acquisition module is used to collect real-time audio and video data from various pre-deployed sensors and monitoring equipment. The data comes from environmental monitoring stations, drones, cameras, and sound sensors to obtain on-site video images and sound information, providing basic data for subsequent analysis and scheduling, and transmitting the collected audio and video data to the command and dispatch platform. Advanced communication technologies and network architectures, such as 4G / 5G, satellite communications, and optical fibers, are used to ensure the stability and efficiency of data transmission. Environmental monitoring stations, drones, cameras, and sound sensors are pre-deployed at various monitoring points in the area to be monitored to ensure that all areas to be monitored are covered. Environmental monitoring stations provide environmental monitoring data and are equipped with cameras and sound sensors to capture on-site video images and sound information. Drones use their high-definition cameras and sound sensors to conduct aerial inspections of specific areas and transmit on-site audio and video data in real time. The high maneuverability and flexibility of drones enable them to quickly reach areas that are difficult for people to reach or dangerous. Fixed or mobile cameras are used to monitor specific areas. Domain or target, providing continuous video streaming data. Cameras are installed in key locations, such as ecological protection areas and near pollution sources. Sound sensors are used to capture and record sound information in the environment, providing sound dimension data for ecological and environmental analysis. Sensors and various monitoring devices monitor environmental parameters in real time and obtain audio and video data. The data includes video images and sound information, covering the entire area where the incident occurred. The collected audio and video data is encoded and compressed to adapt to different network environments and reduce the demand for transmission bandwidth. Advanced communication technologies and network architectures (such as 4G / 5G, satellite communications, optical fibers, etc.) are used to transmit the encoded and compressed data from the scene to the command and dispatch platform to ensure the efficiency and reliability of data transmission, and maintain real-time data even in long distances or complex environments. The command and dispatch platform receives data from various sensors and monitoring devices, integrates them into a unified data format, stores them on the cloud platform, and synchronizes the data collected from different sources to maintain the consistency of audio and video information for subsequent processing and analysis;

[0080] In the data preprocessing module, users process and analyze the collected audio and video data, including video encoding and decoding, audio noise reduction, and image enhancement to ensure the clarity and consistency of the data, and store and back up the audio and video data to ensure the security and traceability of the data, improve the processing efficiency and accuracy of the data, and prepare for further analysis. The original audio and video data stream is retrieved from the command and dispatch platform, and the video is encoded and decoded. The collected video data is decoded from the compressed format (such as H.264, HEVC, etc.) into the original video frame for subsequent processing. After the video processing is completed, the video data is re-encoded into the corresponding compression format according to storage or transmission requirements to reduce the data volume and optimize the transmission efficiency. The video frame is enhanced, such as brightness adjustment, contrast enhancement, color correction, etc., to improve the clarity and Visual effects, and based on the inter-frame difference method, the video is divided into different shot segments to facilitate subsequent key frame extraction and feature analysis. Key frames are extracted from each shot segment. The key frames reflect the main content of the shot. The noise reduction algorithm is applied to reduce or eliminate the noise components in the audio signal, improve the clarity and audibility of the audio, and help to capture the sound information in the environment more clearly. The processed and analyzed audio and video data are cleaned and formatted. Data cleaning removes outliers and incomplete data records to prevent them from affecting the accuracy of the analysis results. Data formatting converts data from different sources into a unified format for easy integration and analysis. The processed data is stored in a data warehouse for subsequent analysis. At the same time, the data is backed up regularly and redundant storage is set to ensure data security and recoverability to prevent data loss or damage;

[0081] The event detection and recognition module combines image recognition and machine learning technology to analyze the processed audio and video data, identify abnormal events in the ecological environment, provide preliminary judgments on the events, and help formulate corresponding response strategies and priorities. It receives audio and video data from the data preprocessing module, and extracts sample data for training and testing from the processed audio and video data to ensure that the sample data covers various possible event types, such as forest fires, water pollution, invasive species invasion, drought, etc. It labels the sample data and marks the characteristic areas of different abnormal event types. It uses computer vision technology to recognize images in video frames and extract image features from video frames, including color, texture, shape, and motion trajectory. It performs feature extraction on audio data to obtain sound features, including frequency, intensity, and duration of sound. It uses OpenCV and other libraries to extract static features such as color, texture, and shape in video frames, and extracts dynamic features such as motion trajectory through optical flow or background subtraction technology. It uses audio processing libraries (such as li brosa) analyze audio signals, extract features such as frequency, intensity, and duration, use a large amount of labeled ecological environment abnormal event data, train an abnormal event recognition model, and analyze the extracted features to identify the difference between the normal environment background and abnormal events, wherein the abnormal event recognition model is obtained based on a convolutional neural network model, and based on the association data of the abnormal event recognition model and image features, perform abnormality detection on the image, analyze and obtain an abnormal image recognition index, detect abnormal images, input the processed sound feature data into the abnormal event recognition model, obtain an abnormal sound recognition index, detect abnormal sounds, integrate the detection results of abnormal images and abnormal sounds, analyze and identify abnormal events in the ecological environment, classify the detected abnormal events, determine the specific type of abnormal events, and locate them in combination with the geographic information of the video frames to determine the location where the abnormal events occurred;

[0082] Furthermore, the expression of the abnormal image recognition index is:

[0083]

[0084] Among them, AP is the abnormal image recognition index, n is the number of frames in the video, C i is the color feature value of the i-th frame image, T i is the texture feature value of the i-th frame image, F i is the shape feature value of the i-th frame image, M i is the motion feature value of the i-th frame image, e is the base of the natural logarithm, the AP value tends to 0, indicating that the image feature is very close to the normal environment and there is no abnormality, and the AP value tends to 1, indicating that the image feature is significantly different from the normal environment and the degree of abnormality is high. i and T i When the difference from the normal environment characteristic value increases, the exponential term Decreases, resulting in an increase in AP, indicating an increase in the degree of abnormality. i and M i When it is larger, it increases the AP value, indicating a more likely abnormality;

[0085] The expression of abnormal sound recognition index is:

[0086]

[0087] Among them, AS is the abnormal sound recognition index, s is the number of sound samples, L j is the frequency characteristic value of the j-th sound sample, B is the reference frequency value, representing the typical frequency characteristic value under normal environmental conditions, S is the standard deviation, indicating the distribution range of the frequency characteristic value, and H j is the intensity characteristic value of the j-th sound sample, A j is the duration characteristic value of the j-th sound sample, It is a limit function, indicating that as the number of samples increases, the value of AS tends to be stable. When the value of AS tends to 0, it means that the sound characteristics are very close to the normal environment and there is no abnormality. When the value of AS tends to 1, it means that the sound characteristics are significantly different from the normal environment and the degree of abnormality is high. j As the difference from B increases, the exponential term decreases, resulting in an increase in AS, indicating an increase in the degree of abnormality. j and A j When it is larger, it increases the value of AS, indicating a more likely abnormality;

[0088] The severity assessment module analyzes and evaluates the severity of identified abnormal events in the ecological environment based on historical data, the scope of impact, potential hazards, and urgency of the abnormal events. It also sets warning levels for the severity of abnormal events and classifies abnormal events. Through quantitative indicators and classification standards, it provides a scientific basis for the allocation of scheduling resources and the priority of emergency response.

[0089] The command and dispatch module is used to formulate and issue corresponding dispatch instructions and allocate dispatch resources based on the results of the event detection and identification module and the severity assessment module, combined with the determined warning level of the abnormal event severity. It also enables real-time communication and command with on-site personnel through audio and video communication means such as video conferencing and voice broadcasting. At the same time, it supports coordinated dispatch across multiple departments and regions to ensure the timeliness and effectiveness of emergency response.

[0090] The resource management module uniformly manages and optimizes the allocation of various resources in the scheduling system, including the scheduling and allocation of human, material and financial resources, as well as the dynamic monitoring and evaluation of resources, to ensure that the required resources can be quickly mobilized when an emergency occurs, providing strong support for emergency response.

[0091] Example 2, as Figure 3 As shown, based on Example 1, the present invention provides a technical solution: Preferably, in the severity assessment module, the process of grading abnormal events is:

[0092] Collect relevant historical ecological and environmental abnormal event data, including the impact range, potential hazards, emergency response measures and their effects of previous events, and obtain the latest data of identified ecological and environmental abnormal events and relevant data of current weather factors in real time. Analyze abnormal events from three dimensions: impact range, potential hazards and urgency of the event. Combined with the historical ecological and environmental abnormal event data, compare and analyze the current abnormal event with the historical ecological and environmental abnormal event data. Use Geographic Information System (GIS) to analyze the abnormal events. IS) technology is used to analyze the impact scope of abnormal events, quantify the size of the affected area, assess the ecological importance of the affected area, evaluate the impact of abnormal events on the ecosystem, estimate the direct and indirect economic losses caused by abnormal events, and evaluate the development speed of abnormal events to determine whether they are showing a rapid deterioration trend. This is to determine the urgency of controlling abnormal events, set quantitative indicators, quantify the scores of each assessment dimension, including the impact scope, potential hazards, and urgency, calculate the severity assessment index, and analyze the trend of abnormal events based on the scoring results of the quantitative indicators. The current abnormal event type is determined based on the abnormal image recognition index and the abnormal sound recognition index. Combined with the severity assessment index, the abnormal assessment index is obtained to analyze the severity of the current abnormal event. The abnormal warning coefficient is combined with current weather factors and the abnormal assessment index to predict the expected duration of the abnormal event. The long-term interference and impact of the abnormal event on the ecological environment are further determined. Based on historical ecological and environmental abnormal event data and the abnormal warning coefficient, the warning level of the severity of the abnormal event is analyzed and divided into low warning level, medium warning level, and high warning level, and the corresponding warning threshold is determined for each warning level;

[0093] Furthermore, the expression of severity assessment index is:

[0094]

[0095] Among them, SA is the severity assessment index, I g is the quantitative score of the g-th dimension, g is the score of the impact range, the score of potential harm or the score of urgency, and w g is the weight of the g-th dimension, which is allocated according to the importance of each dimension, and J g is the benchmark value of the g-th dimension, indicating the score of the normal state of the dimension, which is used to compare the actual score with the score of the normal state, σ g is the standard deviation of the g-th dimension, indicating the expected fluctuation range of the score. It is used to calculate the degree to which the score deviates from the baseline value. The value range of SA is between 0 and 1. When the scores of all dimensions are close to the baseline value, SA approaches 0, indicating that the event is mild. When the score of any dimension deviates significantly from the baseline value, SA approaches 1, indicating that the event is severe.

[0096] The expression of the abnormal evaluation index is:

[0097]

[0098] Among them, AWC is the abnormal assessment index, AP k is the abnormal image recognition index of the kth image recognition event, AS k is the abnormal sound recognition index of the kth sound recognition event, α k is the weight of the kth event, indicating the relative importance of image and sound recognition index in the early warning coefficient, β k is the adjustment parameter of the kth event, which is used to control the sensitivity of the exponential function. SA is the severity assessment index. γ is the baseline value of severity assessment, which represents the normal severity level. δ is the adjustment parameter of severity assessment, which is used to control the sensitivity of the exponential function. The result of AWC is quantified between 0 and 1. The closer the value is to 1, the more severe the warning is. When the values ​​of AP and AS are close to each other, and the value of SA is close to the baseline value γ, AWC approaches 0, indicating a low warning level. When the values ​​of AP or AS deviate significantly from each other, or the value of SA is significantly higher than γ, AWC approaches 1, indicating a high warning level.

[0099] Furthermore, the abnormal warning coefficient is obtained based on the current weather factors and the abnormal assessment index, and its expression is:

[0100]

[0101] Among them, AIC is the abnormal warning coefficient, AWC u is the quantitative value of the u-th abnormal warning coefficient, reflecting the severity of warnings of different events, W u is a weight factor related to weather, considering the impact of different weather conditions on the degree of interference of abnormal events, T c is the temperature index of the current weather, T b is the weather reference value, which indicates the ideal weather state with the least impact on the ecological environment. r is the range of weather indicators, reflecting the span from ideal to extreme conditions, p is the number of abnormal events, It is a limit function, indicating that as time goes by, the interference coefficient tends to be stable. The value range of AIC is between 0 and 1. When AWC u and W u The value of is low, and the current weather is close to the benchmark value T b When AIC approaches 0, it means the interference level is low. u and W u The value of is high, or the current weather is far from the reference value T b When , AIC approaches 1, indicating a high degree of interference;

[0102] Multiple warning levels correspond to multiple warning thresholds, where the warning thresholds include an upper threshold and a lower threshold;

[0103] Multiple warning levels and multiple warning thresholds satisfy the following relationship:

[0104] Low alert level 0 <AIC≤AIC zy ; The impact is minor and may require monitoring and mild intervention;

[0105] Medium warning level AIC zy <AIC≤AIC gy ; The impact is obvious and certain emergency measures need to be taken;

[0106] High warning level AIC>AIC gy ; The impact is serious and requires immediate emergency response measures;

[0107] Among them, AIC is the abnormal warning coefficient, AIC zy is the lower threshold corresponding to the medium warning level and the upper threshold corresponding to the low warning level, AIC gy The lower threshold corresponding to the high warning level and the upper threshold corresponding to the medium warning level;

[0108] In the command and dispatch module, the process of allocating dispatch resources is as follows:

[0109] Receive the type and location information of abnormal events from the event detection and identification module, and receive the abnormal warning coefficient and warning level of abnormal events from the severity assessment module. According to the type of abnormal event and the severity assessment results, formulate emergency response strategies and dispatch instructions, determine the resources to be mobilized, including manpower, materials, equipment, etc., dispatch the required resources according to the emergency response plan, determine the resource allocation plan, including the source, quantity and arrival time of the resources, and issue dispatch instructions to relevant departments and personnel through the command and dispatch platform. Use audio and video communication means to communicate with on-site personnel in real time, and use video conferencing and voice broadcasting tools for remote command and coordination. Use GIS system and Internet of Things technology to monitor resource dispatch progress and on-site conditions in real time to ensure that dispatch instructions are effectively executed, adjust the emergency response plan and resource dispatch according to actual conditions, record all important information and data during the emergency response process, and feedback to relevant personnel to report the progress of abnormal event handling and resource usage.

[0110] Example 3, as Figure 4 As shown, based on Examples 1-2, the present invention further provides an ecological environment command and dispatch method based on audio and video fusion, which is implemented based on an ecological environment command and dispatch system based on audio and video fusion, and includes the following steps:

[0111] Step 1: Use pre-deployed environmental monitoring stations, drones, cameras, and sound sensor equipment to collect audio and video data and pre-process the collected data;

[0112] Step 2: Use image recognition and machine learning techniques to analyze the pre-processed audio and video data to identify abnormal events in the ecological environment;

[0113] Step 3: Combine historical data on abnormal ecological and environmental events to analyze the impact, potential hazards, and urgency of abnormal events, conduct a quantitative assessment of the current event, set an early warning level for the abnormal event, and determine the corresponding early warning threshold;

[0114] Step 4: Based on the results of the event detection and identification module and the severity assessment module, formulate an emergency response plan and dispatch instructions, and communicate with on-site personnel in real time through audio and video communication means to optimize the allocation and dispatch of resources.

[0115] The above are only specific embodiments of the present application, but the scope of protection of this application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of this application. Therefore, the scope of protection of this application should be based on the scope of protection of the claims.

Claims

1. An ecological environment command and dispatch system based on audio and video fusion, including a command and dispatch platform, characterized by: The command and dispatch platform is communicatively connected to a data acquisition module, a data pre-processing module, an event detection and identification module, a severity assessment module, a command and dispatch module, and a resource management module, wherein electrical signal connections are made between the modules; The data acquisition module is used to collect real-time audio and video data from various pre-deployed sensors and monitoring equipment, and transmit the collected audio and video data to the command and dispatch platform; The data pre-processing module allows users to process and analyze the collected audio and video data, and store and back up the audio and video data; The event detection and recognition module combines image recognition and machine learning technology to analyze the processed audio and video data and identify abnormal events in the ecological environment. The identification process of abnormal events in the ecological environment is as follows: Receive audio and video data from the data preprocessing module, extract sample data for training and testing from the processed audio and video data, annotate the sample data, and mark the characteristic areas of different abnormal event types; Use computer vision technology to identify images in video frames and extract image features from video frames, including color, texture, shape, and motion trajectory. Perform feature extraction on audio data to obtain sound features, including frequency, intensity, and duration. Using labeled ecological and environmental abnormal event data, we train an abnormal event recognition model and analyze the extracted features to identify the difference between normal environmental backgrounds and abnormal events. The abnormal event recognition model is based on a convolutional neural network model. Based on the abnormal event recognition model and the associated data of image features, the image is detected for abnormality, and the abnormal image recognition index is obtained through analysis to detect abnormal images. The processed sound feature data is input into the abnormal event recognition model to obtain the abnormal sound recognition index to detect abnormal sounds. The system integrates the detection results of abnormal images and sounds, analyzes and identifies abnormal events in the ecological environment, classifies the detected abnormal events, determines the specific type of abnormal events, and locates the location of the abnormal events based on the geographic information of the video frames; The severity assessment module analyzes and assesses the severity of identified abnormal events in the ecological environment based on historical data, the scope of impact, potential hazards, and urgency of the abnormal events, sets an early warning level for the severity of the abnormal events, and classifies the abnormal events. The process of classifying abnormal events is as follows: Collect relevant historical data on abnormal ecological and environmental events, including the impact scope, potential hazards, emergency response measures and their effectiveness of previous events, and obtain the latest data on identified abnormal ecological and environmental events and relevant data on current weather factors in real time, and analyze abnormal events from three dimensions: impact scope, potential hazards and urgency; Combined with historical data on abnormal ecological and environmental events, conduct a comparative analysis of current abnormal events with historical data on abnormal ecological and environmental events, use geographic information system technology to analyze the impact range of abnormal events, quantify the size of the affected area, assess the ecological importance of the affected area, assess the impact of abnormal events on the ecosystem, estimate the direct and indirect economic losses caused by abnormal events, and assess the development speed of abnormal events to determine whether they show a rapid deterioration trend, so as to determine the urgency of controlling abnormal events; Set quantitative indicators to quantify the scores of each assessment dimension, including the scope of impact, potential harm, and urgency, calculate the severity assessment index, and analyze the trend of abnormal events based on the scoring results of the quantitative indicators; Based on the abnormal image recognition index and the abnormal sound recognition index, the current abnormal event type is determined. Combined with the severity assessment index, the abnormal assessment index is obtained to analyze the severity of the current abnormal event. Combining current weather factors and anomaly assessment indexes, we can obtain anomaly warning coefficients, predict the expected duration of abnormal events, and further determine the long-term interference and impact of abnormal events on the ecological environment. Based on historical ecological and environmental abnormal event data and abnormal warning coefficients, analyze and classify the warning levels of abnormal event severity into low warning level, medium warning level, and high warning level, and determine the corresponding warning threshold for each warning level; The command and dispatch module is used to formulate and issue corresponding dispatch instructions and allocate dispatch resources based on the results of the event detection and identification module and the severity assessment module, combined with the warning level of the severity of the determined abnormal event; The resource management module uniformly manages and optimizes the configuration of various resources in the scheduling system; The expression of the abnormal image recognition index is: Among them, AP is the abnormal image recognition index, n is the number of frames in the video, C i is the color feature value of the i-th frame image, T i is the texture feature value of the i-th frame image, F i is the shape feature value of the i-th frame image, M i is the motion feature value of the i-th frame image, and e is the base of the natural logarithm; The expression of the abnormal sound recognition index is: Among them, AS is the abnormal sound recognition index, s is the number of sound samples, L j is the frequency characteristic value of the j-th sound sample, B is the reference frequency value, representing the typical frequency characteristic value under normal environmental conditions, S is the standard deviation, indicating the distribution range of the frequency characteristic value, and H j is the intensity characteristic value of the j-th sound sample, A j is the duration feature value of the j-th sound sample; The expression of the severity assessment index is: Among them, SA is the severity assessment index, I g is the quantitative score of the g-th dimension, g is the score of the impact range, the score of potential harm or the score of urgency, and w g is the weight of the g-th dimension, which is allocated according to the importance of each dimension, and J g is the reference value of the g-th dimension, σ g is the standard deviation of the g-th dimension; The expression of the abnormality assessment index is: Among them, AWC is the abnormal assessment index, AP k is the abnormal image recognition index of the kth image recognition event, AS k is the abnormal sound recognition index of the kth sound recognition event, α k is the weight of the k-th event, β k is the adjustment parameter of the kth event, SA is the severity assessment index, γ is the baseline value of severity assessment, indicating the normal severity level, and δ is the adjustment parameter of severity assessment; The abnormal warning coefficient is obtained based on the current weather factors and the abnormal assessment index, and its expression is: Among them, AIC is the abnormal warning coefficient, AWC u is the quantitative value of the u-th abnormal warning coefficient, reflecting the severity of warnings of different events, W u is the weight factor related to weather, T c is the temperature index of the current weather, T b is the weather reference value, T r is the range of variation of weather indicators, and p is the number of abnormal events; The plurality of warning levels correspond to the plurality of warning thresholds, wherein the warning thresholds include an upper threshold and a lower threshold; The multiple warning levels and the multiple warning thresholds satisfy the following relationship: Low alert level 0 <AIC≤AIC zy ; Medium warning level AIC zy <AIC≤AIC gy ; High warning level AIC>AIC gy ; Among them, AIC is the abnormal warning coefficient, AIC zy is the lower threshold corresponding to the medium warning level and the upper threshold corresponding to the low warning level, AIC gy The lower threshold corresponding to the high warning level and the upper threshold corresponding to the medium warning level; Through real-time monitoring and automatic detection technology, abnormal events can be identified at the first time and the early warning mechanism can be activated immediately, reducing the time to discover abnormal events. The automated characteristics of the system reduce delays caused by human factors, ensuring that resources and personnel can be quickly mobilized in emergency situations to effectively control and mitigate the impact of events.

2. The ecological environment command and dispatch system based on audio and video fusion according to claim 1 is characterized by: In the data acquisition module, the process of collecting real-time audio and video data is as follows: Pre-deploy environmental monitoring stations, drones, cameras, and sound sensors at each monitoring point in the area to be monitored; Sensors and monitoring equipment monitor environmental parameters in real time, acquire audio and video data, including video images and sound information, and encode and compress the collected audio and video data; Utilize communication technology and network architecture to transmit encoded and compressed data from the field to the command and dispatch platform; The command and dispatch platform receives data from various sensors and monitoring equipment, integrates them into a unified data format, stores them on the cloud platform, and synchronizes the time of data collected from different sources.

3. The ecological environment command and dispatch system based on audio and video fusion according to claim 2 is characterized by: In the data preprocessing module, the process of storing and backing up audio and video data is as follows: Retrieve the original audio and video data stream from the command and dispatch platform, perform video encoding and decoding, decode the collected video data from the compressed format into the original video frame, and after the video processing is completed, re-encode the video data into the corresponding compression format according to storage or transmission requirements; Enhance the video frames and segment the video into different shot segments based on the inter-frame difference method. Extract key frames from each shot segment, which reflect the main content of the shot. Applying noise reduction algorithms to reduce or eliminate noise components in audio signals; The processed and analyzed audio and video data is cleaned and formatted, and the processed data is stored in the data warehouse, and the data is backed up regularly.

4. The ecological environment command and dispatch system based on audio and video fusion according to claim 1 is characterized by: In the command and dispatch module, the process of dispatching resource allocation is as follows: Receive the type and location information of the abnormal event from the event detection and identification module, and receive the abnormal warning coefficient and warning level of the abnormal event from the severity assessment module; Develop emergency response strategies and dispatch instructions based on the type and severity of abnormal events, and determine the resources to be mobilized; According to the emergency response plan, dispatch the required resources, determine the resource allocation plan, including the source, quantity and arrival time of the resources, and issue dispatch instructions to relevant departments and personnel through the command and dispatch platform; Utilize audio and video communication methods to communicate with on-site personnel in real time, use video conferencing and voice broadcasting tools for remote command and coordination, utilize GIS systems and Internet of Things technologies to monitor resource scheduling progress and on-site conditions in real time, and adjust emergency response plans and resource scheduling based on actual conditions; Record all important information and data during the emergency response process and provide feedback to relevant personnel to report the progress of abnormal event handling and resource usage.

5. An ecological environment command and dispatch method based on audio and video fusion, implemented based on the ecological environment command and dispatch system based on audio and video fusion according to any one of claims 1 to 4, characterized in that: The following steps are involved: Step 1: Use pre-deployed environmental monitoring stations, drones, cameras, and sound sensor equipment to collect audio and video data and pre-process the collected data; Step 2: Use image recognition and machine learning techniques to analyze the pre-processed audio and video data to identify abnormal events in the ecological environment; Step 3: Combine historical data on abnormal ecological and environmental events to analyze the impact, potential hazards, and urgency of abnormal events, conduct a quantitative assessment of the current event, set an early warning level for the abnormal event, and determine the corresponding early warning threshold; Step 4: Based on the results of the event detection and identification module and the severity assessment module, formulate an emergency response plan and dispatch instructions, and communicate with on-site personnel in real time through audio and video communication means to optimize the allocation and dispatch of resources.

Images