5G audio and video monitoring quality optimization method and system based on deep learning

Abstract

The invention discloses a 5G audio and video monitoring quality optimization method and system based on deep learning, and the method comprises the steps: collecting multi-modal data through a high-sensitivity microphone array, a high-resolution camera and an environment sensor, and improving the audio quality through spatial filtering, beam forming and adaptive noise reduction; and enhancing the video quality by adopting a high-order function partial derivative and a complex variable function. In the data preprocessing stage, deep learning denoising and PCA compression are adopted, and the transmission efficiency is optimized; the bandwidth is predicted through the LSTM / RNN, the flow is adjusted, and efficient data transmission is ensured. And the CNN and RNN are combined in anomaly detection, so that the accuracy is improved, and real-time feedback is realized. The system adopts distributed storage and a big data analysis tool, and supports deep learning model optimization. According to the invention, the audio and video monitoring quality is significantly improved in the aspects of data acquisition, processing, transmission and storage, and the system is suitable for efficient monitoring in a 5G environment.

Description

Technical Field

[0001] The present invention relates to the field of audio and video monitoring technology, and in particular to a 5G audio and video monitoring quality optimization method and system based on deep learning. Background Art

[0002] With the rapid development and popularization of 5G technology, audio and video surveillance systems have been widely used in security, transportation, public safety and other fields. The high bandwidth and low latency characteristics of 5G networks provide strong support for the real-time transmission and processing of audio and video data. However, existing audio and video surveillance systems still have many problems in data quality and transmission efficiency.

[0003] In the prior art, the audio and video monitoring system has defects in the following aspects:

[0004] 1. Data collection and processing: Traditional audio and video collection equipment is easily affected by noise and light in complex environments, resulting in low quality of collected data. Existing preprocessing methods cannot fully remove noise and enhance images, affecting subsequent data analysis and transmission effects.

[0005] 2. Data transmission: Although 5G networks provide the advantages of high bandwidth and low latency, in actual applications, due to factors such as large data volume and long transmission distance, audio and video data are prone to delays and packet loss during transmission, resulting in unsatisfactory monitoring effects.

[0006] 3. Intelligent analysis and processing: Most existing audio and video surveillance systems use traditional image processing and sound processing algorithms, lacking intelligent and automated deep learning technology, making it difficult to accurately identify and analyze complex scenes and events.

[0007] 4. System integration and optimization: The existing system lacks full-process optimization of data collection, preprocessing, transmission and analysis, resulting in low overall system efficiency and difficulty in meeting efficient and real-time monitoring needs. Summary of the invention

[0008] The present invention provides a 5G audio and video monitoring quality optimization method and system based on deep learning, so as to solve the problem of how to achieve real-time optimization of audio and video monitoring quality in a 5G environment through multimodal data synchronization, efficient edge computing, adaptive flow control, optimized data transmission and deep learning analysis, and improve the accuracy, real-time and intelligence level of the monitoring system.

[0009] In order to solve the above problems, the present invention provides a 5G audio and video monitoring quality optimization method based on deep learning.

[0010] Acquire audio, video, and environmental sensor data; Preprocess and analyze audio, video, and environmental data through deep learning models; Optimize the audio and video data according to the preprocessing and analysis results; The optimized audio and video data is transmitted to the monitoring center via the 5G network.

[0011] As a preferred solution, the audio data is collected by a high-sensitivity microphone array and processed by spatial filtering and beamforming technology.

[0012] As a preferred solution, the processing formula of the spatial filtering and beamforming is:

[0013] Where y(t) is the synthesized audio signal, is the audio signal collected by the i-th microphone at time t, is the delay of the iiith microphone, is the frequency of the ith microphone and N is the number of microphones.

[0014] As a preferred solution, the video data is collected by a high-resolution camera, and image enhancement processing is performed using multi-frame image fusion and complex function.

[0015] As a preferred solution, the multi-frame image fusion formula is:

[0016] Among them, L is the fused image, is the i-th exposure image, is the weight function and N is the number of images.

[0017] As a preferred solution, the environmental data is collected by environmental sensors, and Kalman filtering and unscented Kalman filtering are used for data fusion and filtering.

[0018] As a preferred solution, the processing formula of the unscented Kalman filter is:

[0019] in, is the observational prediction, is the covariance between states and observations, is the observation covariance.

[0020] As a preferred solution, audio, video and environmental data are synchronously processed through efficient timestamp synchronization and cubic spline interpolation technology.

[0021] As a preferred solution, the optimization processing includes data denoising and data compression, which are processed using wavelet transform and principal component analysis respectively.

[0022] As a preferred solution, a 5G audio and video monitoring quality optimization system based on deep learning includes: Data acquisition module, used to collect audio, video and environmental data; Data preprocessing module, used for denoising and compressing the collected data; Deep learning analysis module, used to analyze and optimize the preprocessed data;

[0023] The 5G communication module is used to transmit the optimized data to the monitoring center through the 5G network. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 It is a flow chart of a 5G audio and video monitoring quality optimization method and system based on deep learning provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0025] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0026] In the following, only some exemplary embodiments are briefly described. As those skilled in the art will appreciate, the described embodiments may be modified in various ways without departing from the spirit or scope of the present application. Therefore, the drawings and descriptions are considered to be exemplary and non-restrictive in nature.

[0027] Reference Figure 1 , a flowchart of a 5G audio and video monitoring quality optimization method based on deep learning according to an embodiment of the present application is shown. The 5G audio and video monitoring quality optimization method based on deep learning may at least include steps S100-S500:

[0028] S100, collect audio, video and environmental data in real time.

[0029] S200, denoising, compressing and preprocessing the collected data to improve data quality and transmission efficiency.

[0030] S300. Analyze the preprocessed data using a deep learning model to identify abnormal situations and extract key features.

[0031] S400 efficiently transmits processed data to the cloud through the 5G network and stores and manages the data.

[0032] S500, real-time feedback of abnormal detection results, and dynamic optimization and adjustment of the system to improve system performance and stability.

[0033] A 5G audio and video monitoring quality optimization system based on deep learning provided in an embodiment of the present invention is mainly used to solve the problem of how to achieve real-time optimization of audio and video monitoring quality in a 5G environment through multimodal data synchronization, efficient edge computing, adaptive traffic control, optimized data transmission and deep learning analysis, and improve the accuracy, real-time and intelligence level of the monitoring system.

[0034] The present invention discloses a 5G audio and video monitoring quality optimization method and system based on deep learning, which involves multiple technical fields such as data acquisition, data preprocessing, intelligent flow control, anomaly detection, data transmission and storage management. The method includes multimodal data acquisition: multimodal data acquisition is performed through a high-sensitivity microphone array, a high-resolution camera and an environmental sensor, audio quality is improved by spatial filtering, beamforming and adaptive noise reduction technology, and video quality is enhanced by partial derivatives of high-order functions and complex functions. Data preprocessing: data preprocessing is performed using filtering algorithms and deep learning denoising technology, data compression is performed through principal component analysis (PCA) to improve transmission efficiency, and a lightweight deep learning model is deployed on the edge device for preliminary analysis. Intelligent flow control: bandwidth prediction is performed through LSTM or RNN, flow adjustment is performed based on calculus and functional analysis, and efficient data transmission is achieved using a 5G communication module. Anomaly detection and key feature extraction: anomaly detection is performed on multimodal data by combining convolutional neural network (CNN) and recurrent neural network (RNN), detection accuracy is improved by using deep learning models and rule engines, and real-time feedback and intelligent response are achieved through 5G networks. Data storage and management: Distributed storage systems and big data analysis tools are used to efficiently store and manage large-scale data, supporting the optimization of deep learning models and system improvements. The present invention has significant advantages in data collection quality, processing efficiency, transmission delay, detection accuracy and storage management, solves many deficiencies of the existing technology, and has high innovation and practicality.

[0035] S100, real-time collection of audio, video and environmental data The system collects multi-modal data from the user. Step S100 includes steps S110-S150:

[0036] S110, audio acquisition device, equipped with a high-sensitivity microphone array, capable of collecting high-quality audio signals.

[0037] Spatial filtering and beamforming:

[0038] in,

[0039] is the synthesized audio signal,

[0040] It is microphones at a time The collected audio signal,

[0041] It is The delay of each microphone,

[0042] It is The frequency of the microphone,

[0043] is the number of microphones.

[0044] S120, Adaptive Noise Reduction: Noise reduction using Generalized Least Squares (GLS):

[0045] in,

[0046] are the estimated filter parameters,

[0047] is the observation matrix,

[0048] is the covariance matrix,

[0049] is the observation vector.

[0050] S130, video acquisition equipment, equipped with a high-resolution camera, supports video acquisition with a resolution of 4K and above, ensuring the clarity and details of video data.

[0051] Multi-frame image fusion enhancement: Image enhancement using higher-order function partial derivatives:

[0052] in,

[0053] is the fused image,

[0054] It is exposure images,

[0055] is the weight function,

[0056] is the number of images.

[0057] Low light enhancement using complex functions:

[0058] in,

[0059] is the reflectance image,

[0060] is the input image,

[0061] is the illumination image,

[0062] is the pixel position represented by a complex number.

[0063] S140, environmental sensor, integrates multiple sensors (such as temperature, humidity, light, motion sensors, etc.), can collect environmental data and provide comprehensive monitoring information.

[0064] Kalman Filter: Fusion and filtering of environmental sensor data:

[0065] in,

[0066] is the current state estimate,

[0067] is the observed value at the current moment,

[0068] is the observation matrix,

[0069] is the Kalman gain,

[0070] is the prior error covariance,

[0071] is the observation noise covariance matrix.

[0072] S150, Unscented Kalman Filter (UKF):

[0073] in,

[0074] is the observational prediction,

[0075] is the covariance between states and observations,

[0076] is the observation covariance.

[0077] Here are a few examples: Example 1: Home security monitoring system

[0078] Audio acquisition equipment. In the home security monitoring system, a high-sensitivity microphone array is installed in each room to collect all audio signals in the room in real time. The system processes the collected audio through spatial filtering and beamforming technology to ensure that the sound is clear and discernible.

[0079] Adaptive noise reduction. The system uses the generalized least squares method (GLS) to reduce the noise of the collected audio data. By removing background noise, such as air conditioning and external traffic noise, it ensures that abnormal sounds, such as glass breaking or alarms, can be clearly heard during monitoring.

[0080] Video acquisition equipment. High-resolution cameras (supporting 4K and above resolution) in the home security monitoring system are installed in key locations to collect video data in real time. The system enhances the video image through multi-frame image fusion and complex function to ensure clear video images with rich details.

[0081] Environmental sensors. Environmental sensors in home security monitoring systems (such as temperature and humidity sensors) collect indoor environmental data in real time. The system uses Kalman filtering and unscented Kalman filtering to fuse and filter environmental data to improve data accuracy and help identify abnormal environmental changes, such as fire or flooding.

[0082] Example 2: Traffic Monitoring System

[0083] Audio acquisition equipment. In the traffic monitoring system, a high-sensitivity microphone array installed at the main traffic intersection collects traffic noise and vehicle horn sounds in real time. The system processes audio signals through spatial filtering and beamforming technology to ensure that abnormal sounds, such as accident sounds or sirens, can be clearly identified.

[0084] Adaptive noise reduction. The system uses the generalized least squares method (GLS) to reduce the noise of collected traffic noise data. By removing background noise, such as the continuous noise of vehicles driving, it ensures that abnormal sounds, such as emergency braking or collision sounds, can be clearly heard during monitoring.

[0085] Video acquisition equipment. The high-resolution cameras in the traffic monitoring system are installed at major traffic intersections to collect video data of traffic flow in real time. The system enhances the video images through multi-frame image fusion and complex function to ensure clear video images and facilitate the identification of traffic violations or accidents.

[0086] Environmental sensors. Environmental sensors in the traffic monitoring system (such as light sensors, rain sensors, etc.) collect environmental data in real time. The system fuses and filters environmental data through Kalman filtering and unscented Kalman filtering to improve data accuracy and help identify abnormal environmental changes, such as bad weather or road waterlogging.

[0087] Example 3: Smart City Monitoring System

[0088] Audio acquisition equipment. In the smart city monitoring system, high-sensitivity microphone arrays distributed in every corner of the city collect all audio signals in the urban environment in real time. The system processes audio signals through spatial filtering and beamforming technology to ensure that sounds are clearly identifiable, such as street noise or the sound of emergencies.

[0089] Adaptive noise reduction. The system uses the generalized least squares method (GLS) to reduce the noise of the collected urban environment audio data. By removing background noise, such as constant traffic noise, it ensures that abnormal sounds, such as violent incidents or fire alarms, can be clearly heard during monitoring.

[0090] Video acquisition equipment. High-resolution cameras in smart city monitoring systems are installed in key areas of the city, such as squares and main streets, to collect video data in real time. The system enhances video images through multi-frame image fusion and complex functions to ensure clear video images and facilitate the identification of emergencies or abnormal behaviors.

[0091] Environmental sensors. Environmental sensors in smart city monitoring systems (such as air quality sensors, temperature and humidity sensors, etc.) collect environmental data in real time. The system fuses and filters environmental data through Kalman filtering and unscented Kalman filtering to improve data accuracy and help identify abnormal environmental changes, such as air pollution or extreme weather.

[0092] The above examples demonstrate the actual operation process of multimodal data collection in different application scenarios of our invention patent. These processes not only ensure the high quality and reliability of data, but also improve the overall monitoring efficiency and accuracy of the system.

[0093] In one embodiment, S200 performs denoising, compression and preprocessing on the collected data to improve data quality and transmission efficiency. Step S200 includes steps S210-S290: S210, data denoising module The data denoising module is responsible for processing the collected audio and video data to remove noise and ensure the clarity of the data.

[0094] S220. First, audio data is collected through a high-sensitivity microphone array. The collected audio data enters the denoising module, and a filtering algorithm (such as low-pass filtering and mean filtering) is used to pre-process the audio data to remove background noise. The pre-processed audio data is further processed by a machine learning denoising algorithm, and a deep learning model identifies and removes complex noise. After processing, clear audio data is obtained.

[0095] Audio data acquisition: High-sensitivity microphone arrays are used to capture high-quality audio signals, ensuring the details and clarity of sound data.

[0096] Preprocessing: Use traditional filtering algorithms such as low-pass filtering and mean filtering to remove background noise, such as wind noise, vehicle noise, etc.

[0097] Deep learning denoising: Through a well-trained deep learning model, complex noises such as crowd noise and mechanical noise are further removed to ensure the high fidelity of audio data.

[0098] S230, video data denoising. Video data is collected by a high-resolution camera that supports 4K and above resolution. The collected video data enters the denoising module, and first uses basic image processing algorithms (such as median filtering and Gaussian filtering) to remove noise in the image. The pre-processed video data is further processed by a deep learning model to identify and remove complex image noise. Finally, clear video data is obtained.

[0099] Video data acquisition: Use high-resolution cameras that support 4K and above resolutions to ensure high clarity and detail of image data.

[0100] Basic image processing: Use basic image processing algorithms such as median filtering and Gaussian filtering to remove noise from images, such as random noise generated by camera sensors.

[0101] Deep learning denoising: Through a well-trained deep learning model, it can identify and remove complex image noise, such as uneven lighting and motion blur, to ensure the high quality of video data.

[0102] S240, data compression module: The data compression module is responsible for compressing the audio and video data after the denoising process to reduce the data volume and improve the transmission efficiency.

[0103] S250, audio data compression. Clear audio data enters the compression module and is compressed using an efficient audio compression algorithm (such as AAC). The compression algorithm significantly reduces the amount of data while maintaining audio quality by removing redundant information.

[0104] Compression algorithm selection: Use efficient audio compression algorithms such as AAC (Advanced Audio Coding) to remove redundant information while maintaining audio quality.

[0105] Compression process: Audio data is encoded and compressed through compression algorithms to reduce data volume and optimize transmission efficiency.

[0106] S260, video data compression. Clear video data enters the compression module and is compressed using efficient video codec technology (such as H.265). H.265 technology uses a more efficient encoding method to reduce the amount of video data and optimize transmission efficiency while ensuring video quality.

[0107] Compression algorithm selection: Use high-efficiency video coding technologies such as H.265 (HEVC, high-efficiency video coding) to reduce the amount of data.

[0108] Compression process: Video data is encoded and compressed through compression algorithms, which significantly reduces the amount of data while maintaining video quality and optimizes transmission efficiency.

[0109] S270, preliminary analysis module. The preliminary analysis module uses edge computing equipment to perform preliminary analysis on the compressed data and provide real-time feedback.

[0110] S280, motion detection. Video data enters the preliminary analysis module and performs motion detection through a pre-trained convolutional neural network (CNN) model. The CNN model analyzes video frames in real time, identifies moving objects in the video, and marks moving areas.

[0111] CNN model: Use a pre-trained convolutional neural network model to analyze video frames in real time and identify moving targets.

[0112] Mark motion area: Mark the detected moving targets and generate motion area information for subsequent processing and analysis.

[0113] S290, sound recognition. The audio data enters the preliminary analysis module and is recognized by the pre-trained recurrent neural network (RNN) model. The RNN model analyzes the audio signal in real time, identifies specific sound events (such as alarm sounds, breaking sounds, etc.), and provides corresponding feedback.

[0114] RNN model: Use a pre-trained recurrent neural network model to analyze audio signals in real time and identify specific sound events.

[0115] Feedback mechanism: Provide feedback on detected sound events, such as triggering alarms, sending notifications, etc., to ensure the system's real-time response and processing of abnormal sounds.

[0116] Specific examples are as follows: Example 1: Home security monitoring system Data denoising module

[0117] Audio data denoising: In the home security monitoring system, the audio data collected by the microphone array enters the denoising module, which first uses low-pass filtering and mean filtering to pre-process the audio data to remove background noise, such as air conditioning and road noise. Then, a machine learning denoising algorithm is used to further identify and remove complex noise, such as a baby crying or a pet barking. Finally, clear audio data is obtained.

[0118] Video data denoising: Video data collected by a high-resolution camera enters the denoising module, which first removes image noise through median filtering and Gaussian filtering to improve image quality. Then, the deep learning model further processes the video data to identify and remove complex image noise, such as noise or motion blur in low-light environments. After processing, clear video data is obtained.

[0119] Data Compression Module

[0120] Audio data compression: Clear audio data enters the compression module and is compressed using the AAC audio compression algorithm. While maintaining audio quality, it significantly reduces data volume and improves transmission efficiency by removing redundant information.

[0121] Video data compression: Clear video data enters the compression module and is compressed using H.265 video codec technology. H.265 uses a more efficient encoding method to reduce the amount of video data and optimize transmission efficiency while ensuring video quality.

[0122] Preliminary analysis module

[0123] Motion detection: The compressed video data enters the preliminary analysis module and performs motion detection through a pre-trained convolutional neural network (CNN) model. The CNN model analyzes video frames in real time, identifies moving targets in the video, such as people entering the monitoring area, and marks the moving area.

[0124] Sound recognition: The compressed audio data enters the preliminary analysis module and is used for sound recognition through a pre-trained recurrent neural network (RNN) model. The RNN model analyzes the audio signal in real time, identifies specific sound events, such as glass breaking or alarms, and provides corresponding feedback.

[0125] Example 2: Traffic Monitoring System Data denoising module

[0126] Audio data denoising: In the traffic monitoring system, the traffic noise data collected by the microphone array enters the denoising module, and uses low-pass filtering and mean filtering to remove background noise, such as vehicle driving noise and wind noise. Then, the machine learning denoising algorithm is used to further identify and remove complex noise, such as emergency braking or collision sounds. Finally, clear audio data is obtained.

[0127] Video data denoising: The video data collected by the traffic monitoring camera enters the denoising module, which first removes image noise through median filtering and Gaussian filtering to improve image quality. Then, the deep learning model further processes the video data to identify and remove complex image noise, such as noise caused by insufficient light at night or blur on rainy days. After processing, clear video data is obtained.

[0128] Data Compression Module

[0129] Audio data compression: Clear audio data enters the compression module and is compressed using the AAC audio compression algorithm. While maintaining audio quality, it significantly reduces data volume and improves transmission efficiency by removing redundant information.

[0130] Video data compression: Clear video data enters the compression module and is compressed using H.265 video codec technology. H.265 uses a more efficient encoding method to reduce the amount of video data and optimize transmission efficiency while ensuring video quality.

[0131] Preliminary analysis module

[0132] Motion detection: The compressed video data enters the preliminary analysis module and performs motion detection through a pre-trained convolutional neural network (CNN) model. The CNN model analyzes video frames in real time, identifies moving targets in the video, such as pedestrians crossing the road, and marks the moving areas.

[0133] Sound recognition: The compressed audio data enters the preliminary analysis module and is recognized by a pre-trained recurrent neural network (RNN) model. The RNN model analyzes the audio signal in real time, identifies specific sound events, such as vehicle collisions or emergency braking sounds, and provides corresponding feedback.

[0134] Example 3: Smart City Monitoring System Data denoising module

[0135] Audio data denoising: In the smart city monitoring system, the ambient noise data collected by the microphone array enters the denoising module, using low-pass filtering and mean filtering to remove background noise, such as continuous traffic noise and crowd noise. Then, the machine learning denoising algorithm is used to further identify and remove complex noise, such as shouting or sirens in violent incidents. Finally, clear audio data is obtained.

[0136] Video data denoising: The video data collected by the city surveillance cameras enters the denoising module, which first removes image noise through median filtering and Gaussian filtering to improve image quality. Then, the deep learning model further processes the video data to identify and remove complex image noise, such as noise in low light conditions or weather effects. After processing, clear video data is obtained.

[0137] Data Compression Module

[0138] Audio data compression: Clear audio data enters the compression module and is compressed using the AAC audio compression algorithm. While maintaining audio quality, it significantly reduces data volume and improves transmission efficiency by removing redundant information.

[0139] Video data compression: Clear video data enters the compression module and is compressed using H.265 video codec technology. H.265 uses a more efficient encoding method to reduce the amount of video data and optimize transmission efficiency while ensuring video quality.

[0140] Preliminary analysis module

[0141] Motion detection: The compressed video data enters the preliminary analysis module and performs motion detection through a pre-trained convolutional neural network (CNN) model. The CNN model analyzes video frames in real time, identifies moving targets in the video, such as abnormal crowds, and marks moving areas.

[0142] Sound recognition: The compressed audio data enters the preliminary analysis module and is used for sound recognition through a pre-trained recurrent neural network (RNN) model. The RNN model analyzes the audio signal in real time, identifies specific sound events, such as shouting or sirens in violent incidents, and provides corresponding feedback.

[0143] In one embodiment, S300 uses a deep learning model to analyze the preprocessed data, identify abnormal situations and extract key features. Step S300 includes steps S310-S380:

[0144] S310, data input. The clear and compressed audio and video data processed by the data preprocessing unit are input into the deep learning model. These data include audio signals, video frames and related environmental data.

[0145] Data source: Audio signals, video frames and environmental sensor data are preprocessed, denoised and compressed to ensure data clarity and transmission efficiency.

[0146] Data format: The input data is formatted into a form acceptable to the deep learning model, ready for feature extraction and anomaly identification.

[0147] S320, model loading and preparation. The system loads pre-trained deep learning models, including convolutional neural networks (CNNs) and recurrent neural networks (RNNs). CNNs are mainly used for processing image and video data, while RNNs are used for processing audio data.

[0148] Model selection: Select a deep learning model suitable for the task, CNN for video data analysis, and RNN for audio data processing.

[0149] Model loading: Load pre-trained model weights and configuration files to ensure that the model can run efficiently in the current environment.

[0150] S330, video data analysis. Feature extraction: Clear compressed video data is input into the CNN model. CNN extracts key features in video frames, such as object outlines, motion trajectories, etc., through multiple layers of convolution, pooling, and fully connected layers.

[0151] Feature extraction: Extract key features in video frames, such as object contours, motion trajectories, etc., through multi-layer convolution, pooling, and fully connected layers.

[0152] Abnormal identification: The extracted features are sent to the classification layer through the fully connected layer. The classification layer identifies abnormal situations in the video, such as intrusion, fighting, etc. The identified abnormal situations are marked and corresponding alarm information is generated.

[0153] Keyframe extraction: The system extracts keyframes based on the results of motion detection and feature extraction in the video data. These keyframes contain important visual information for further analysis and storage.

[0154] S340, Audio Data Analysis. Feature Extraction: Clear compressed audio data is input into the RNN model. RNN processes audio signals through multi-layer recurrent units (such as LSTM or GRU) to extract key features in the audio, such as frequency, pitch, rhythm, etc.

[0155] Feature extraction: RNN processes audio signals through multiple layers of recurrent units (such as LSTM or GRU) to extract key features in the audio, such as frequency, pitch, rhythm, etc.

[0156] Abnormal identification: The extracted audio features are sent to the classification layer through the fully connected layer. The classification layer identifies abnormal situations in the audio, such as glass breaking, alarm sounds, etc. The identified abnormal sounds are marked and corresponding alarm information is generated.

[0157] S350, Environmental Data Analysis: Through Kalman filtering and unscented Kalman filtering (UKF), the data collected by environmental sensors are fused and filtered to remove noise and improve data accuracy. Based on the processed environmental data, abnormal environmental changes such as sharp temperature rise and humidity changes can be identified.

[0158] Data fusion: Kalman filtering and unscented Kalman filtering (UKF) are used to fuse and filter environmental data to remove noise.

[0159] Abnormality recognition: Based on the processed environmental data, abnormal environmental changes such as sharp temperature rise and humidity changes are identified, and corresponding alarm information is generated.

[0160] S360, Comprehensive analysis of abnormal situations: Comprehensively analyze the abnormal identification results of video, audio and environmental data. Use fusion algorithms and rule engines to perform correlation analysis on multimodal data to ensure accurate identification of abnormal situations and reduce false alarm rates.

[0161] Data integration: Integrate the anomaly recognition results of video, audio and environmental data to form a comprehensive anomaly situation.

[0162] Correlation analysis: Utilize fusion algorithms and rule engines to conduct correlation analysis on multimodal data to ensure accurate identification of abnormal situations and reduce false alarm rates.

[0163] S370, key feature extraction: The system extracts features from video and audio data, as well as the analysis results of environmental data, to form structured key feature information. This key feature information includes the type, occurrence time, and location of the abnormal event.

[0164] Feature integration: Integrate the features extracted from video, audio, and environmental data to form structured information.

[0165] Information marking: Mark key characteristic information, including the type, time of occurrence, location, etc. of the abnormal event, to ensure the integrity and availability of the information.

[0166] S380, the system generates real-time alarms and feedback information based on the identified abnormal conditions and extracted key features. Through the 5G network, these alarm information are transmitted to the monitoring center or user terminal in real time to achieve rapid response and processing.

[0167] Alert generation: Based on identified anomalies, real-time alert information is generated to ensure rapid response.

[0168] Information transmission: Through the 5G network, alarm information is transmitted to the monitoring center or user terminal in real time to ensure the timeliness and reliability of information transmission.

[0169] Quick response: After the monitoring center or user terminal receives the alarm information, it responds and processes it quickly to ensure the performance and stability of the system.

[0170] In one embodiment, S400 efficiently transmits the processed data to the cloud via the 5G network, and stores and manages the data. Step S400 includes steps S410-S490:

[0171] S410, Adaptive Traffic Control Unit: Real-time monitoring module, equipped with high-precision bandwidth monitoring equipment, real-time monitoring of network bandwidth and data flow, and collection of relevant data.

[0172] Real-time monitoring module: bandwidth monitoring

[0173] in, It's time The total bandwidth, It is The bandwidth of the stream, is the total number of flows. High-precision bandwidth monitoring is achieved through time domain and frequency domain analysis:

[0174] in, is the bandwidth spectrum, is the autocorrelation function, is the frequency, It's a delay.

[0175] S420, traffic prediction module: uses machine learning algorithms (such as LSTM or RNN) to predict network bandwidth changes and adjust transmission strategies in advance.

[0176] Traffic prediction based on multiple regression analysis and LSTM, multiple regression model:

[0177] in, It's time The predicted bandwidth is is the intercept, is the regression coefficient, It is predictor variables, is the error term, is the number of predictor variables.

[0178] The formula of LSTM unit is:

[0179] in, is the input gate, It's the Gate of Forgetfulness. is the output gate, is a candidate memory cell, is a memory unit, is the hidden state, is the input, and are weight and bias respectively.

[0180] S430, traffic adjustment module: dynamically adjust data traffic and optimize transmission strategy based on prediction results and real-time data.

[0181] Flow regulation based on calculus and functional analysis:

[0182] in, is the control signal, e(t) is the error signal, e(t) = (t) B(t) , is the proportional gain, is the integral gain, is the differential gain.

[0183] Optimization based on topology and abstract algebra: The coverage and compactness properties of topology are used to ensure the stability and robustness of the flow control strategy: T(X)={U⊆X|U is open}

[0184] Where T(X) is the topology on the topological space X, U is a subset of X, and U is an open set. Combining group theory and ring theory in abstract algebra, the computational efficiency of the flow control algorithm is optimized:

[0185] in, is the generating group, is the generated set, is an integer exponent.

[0186] S440, transmission scheduling unit: bandwidth management module, using SDN technology, dynamically allocates and manages bandwidth resources to ensure that key data is transmitted first. Transmission optimization module, using the optimized transmission solution based on the QUIC (Quick UDP Internet Connections) protocol to improve the reliability and efficiency of data transmission.

[0187] S450, 5G communication module: 5G network interface. Integrates high-performance 5G communication module, supports NSA (Non-Standalone) and SA (Standalone) modes, and achieves efficient data transmission. For the optimization of transmission efficiency in 5G communication, the transmission rate can be expressed by the following formula :

[0188] in, is the transmission rate, is the channel bandwidth, is the transmit power, is the transmit antenna gain, is the receiving antenna gain, is the wavelength, is the distance between the transmitting antenna and the receiving antenna, is the noise power spectral density.

[0189] Channel capacity optimization formula: According to Shannon's theorem, channel capacity It can be expressed as:

[0190] in, is the signal power, is the noise power.

[0191] S460, Transport protocol module: Use the optimized transport protocol stack, combining the advantages of TCP and UDP to improve transmission efficiency and reliability. Transport protocol module: Transport protocol based on the optimized transport protocol stack. The optimized transport protocol stack combines the advantages of TCP and UDP. We can use the following mathematical model for optimization:

[0192] Flow control model: TCP flow control model: if no loss

[0193] if loss detected

[0194] in, is the congestion window size at time t, is the congestion window size at time t-1.

[0195] Transmission efficiency optimization: By combining the low latency characteristics of UDP and the reliable transmission characteristics of TCP, the transmission protocol stack is optimized:

[0196] in, is the transmission efficiency, and are the transmission rates of TCP and UDP respectively, and are the delays of TCP and UDP respectively, and is the weight coefficient.

[0197] S470, Optimization algorithm and its process: Transmission protocol optimization

[0198] Transmission rate optimization: Use segment optimization algorithm to dynamically adjust the transmission ratio of TCP and UDP according to the real-time network status: and

[0199] By solving the above partial derivative equation, we can determine the optimal and value.

[0200] Channel allocation optimization: Use a channel allocation strategy based on game theory to optimize the allocation of channel resources between multiple devices:

[0201] in, Yes Equipment The transmission rate, Yes Equipment transmission cost.

[0202] Algorithm logic: Optimizing transmission efficiency and reliability through multivariate regression and high-order calculus:

[0203] in, is the loss function, is the actual transmission status, is the predicted transmission status.

[0204] S480, overall optimization process: 5G network interface optimization

[0205] Dynamic Power Control: Using calculus methods to optimize transmit power in real time and received power :

[0206] Optimize transmission rate by fine-tuning power .

[0207] Channel bandwidth optimization: By optimizing the channel bandwidth Increase the transfer rate:

[0208] S490, transmission protocol optimization

[0209] Protocol stack optimization: Combining TCP's flow control mechanism and UDP's low latency characteristics, using a hybrid transport protocol:

[0210] Dynamic parameter adjustment: Real-time monitoring of network status and dynamic adjustment and value to ensure transmission efficiency and reliability.

[0211] Optimization algorithm implementation: Build mathematical models to describe the transmission characteristics of TCP and UDP. Use advanced calculus and multivariate regression analysis to optimize transmission parameters. Adjust transmission strategies in real time to improve overall transmission performance.

[0212] S500: Feedback abnormal detection results in real time, and dynamically optimize and adjust the system to improve system performance and stability. Step S500 includes steps S510-S590: S500, real-time feedback of abnormal detection results. After detecting an abnormal situation, the system immediately generates corresponding abnormal detection results and alarm information. Through the 5G network, the detection results are transmitted to the monitoring center or user terminal in real time to ensure real-time response.

[0213] S510: Real-time transmission of anomaly detection results. Data packaging: The system packages anomaly detection results, related video and audio clips, environmental data and other information. Encryption processing: To ensure data security, the packaged data is encrypted. Network transmission: The encrypted data is transmitted to the monitoring center or user terminal in real time through the 5G network. The high bandwidth and low latency characteristics of the 5G network are used to ensure the timeliness and reliability of data transmission.

[0214] S520: The monitoring center receives and analyzes the data. Data reception: The receiving device of the monitoring center receives the transmitted abnormal detection result data packet in real time. Data decryption: Decrypt the received encrypted data to restore the original data content. Data analysis: Analyze the data packet to extract key information such as abnormal detection results, video and audio clips, and environmental data.

[0215] S530: Confirmation and handling of abnormal situations. Manual confirmation: The operator of the monitoring center manually confirms the received abnormal detection results to ensure the accuracy of the abnormal situation. Automatic processing: Combined with the predefined automated response strategy, the system automatically handles some common abnormal situations, such as triggering alarms, sending notifications, adjusting camera angles, etc.

[0216] S540: Dynamic optimization suggestion generation. Data analysis: The system further analyzes the anomaly detection results and monitoring data to find out the potential causes of the anomaly. Optimization suggestions: Based on the analysis results, the system generates a series of optimization suggestions, such as adjusting camera parameters, optimizing network transmission strategies, and upgrading deep learning models.

[0217] S550: Dynamic adjustment and optimization of the system. Automatic optimization: The system automatically performs some optimization adjustments based on the generated optimization suggestions, such as adjusting the camera focal length, updating model parameters, etc. Manual optimization: For optimization suggestions that require manual intervention, the relevant technicians are notified to perform adjustments and optimization operations.

[0218] S560: Performance and stability monitoring. Real-time monitoring: The system monitors the optimized operating status in real time to ensure the effectiveness of the optimization measures. Data recording: Record the system's performance and stability data for subsequent analysis and improvement.

[0219] S570: Feedback on optimization results. Results evaluation: The system evaluates the results of optimization measures and analyzes their impact on performance and stability. Feedback report: Generates feedback reports on optimization results and provides them to the monitoring center and technical team to ensure continuous improvement of optimization measures.

[0220] S580: Storage and recording of optimization results. Result storage: Store the optimization measures and effect evaluation results in the system database as a reference for subsequent optimization and improvement. History records: Save the history records of all optimization operations for easy tracing and analysis.

[0221] S590: System self-learning and improvement. Self-learning: The system conducts self-learning by analyzing historical optimization data and effect evaluation results, and continuously improves the anomaly detection and processing algorithms. Continuous optimization: The system continuously performs dynamic optimization based on the learning results to improve overall performance and stability.

[0222] In general, the method of the embodiment of the present application has at least the following effects:

[0223] (1) Improve data clarity and accuracy: In the preprocessing stage of audio and video data, the present invention uses a combination of filtering algorithms (such as low-pass filtering and mean filtering) and deep learning denoising algorithms (such as convolutional neural networks (CNNs)) to remove complex noise, thereby obtaining clear audio and video data. Compared with traditional simple filtering processing, the present invention effectively improves the denoising effect through the application of deep learning models, making audio and video data clearer and more accurate.

[0224] (2) Improve data compression efficiency: Through efficient audio and video compression algorithms (such as AAC and H.265), the amount of data is significantly reduced while ensuring the quality of audio and video, and the data transmission efficiency is optimized. Compared with traditional compression methods, the present invention adopts more advanced encoding and decoding technology to achieve more efficient data compression and significantly reduce the bandwidth requirements during data transmission.

[0225] (3) Enhanced abnormality identification and feature extraction capabilities: The present invention uses pre-trained deep learning models (such as CNN and RNN) to analyze pre-processed audio and video data, which can accurately identify abnormal situations and extract key feature information. Compared with traditional rule-based or simple model-based recognition methods, deep learning models have stronger learning and generalization capabilities, and can more accurately identify abnormal situations in complex environments, such as intrusions, fights, alarm sounds, etc., improving the intelligence level of the system.

[0226] (4) Achieve efficient data transmission and management: Through the 5G network, the processed data is efficiently transmitted to the cloud, where it is stored and managed. The high bandwidth and low latency characteristics of the 5G network ensure fast data transmission. At the same time, the cloud server uses distributed storage and big data management platforms to achieve efficient management and analysis of data, supporting users to quickly query and make decisions on monitoring data.

[0227] (5) Improve system response speed and stability: The system further improves system performance and stability by providing real-time feedback on anomaly detection results and dynamically optimizing and adjusting the system. By integrating the anomaly recognition results of multimodal data and using fusion algorithms and rule engines for correlation analysis, the system ensures accurate recognition of anomalies, reduces false alarm rates, and improves system response speed and stability.

[0228] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example" or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. Moreover, the described specific features, structures, materials or characteristics may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine different embodiments or examples described in this specification and the features of different embodiments or examples, unless they are contradictory.

[0229] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include at least one of the features. In the description of this application, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.

[0230] Any process or method description in the flow chart or otherwise described herein can be understood to represent a module, fragment or portion of a code including one or more executable instructions for implementing the steps of a specific logical function or process. And the scope of the preferred embodiment of the present application includes other implementations, in which the functions may not be performed in the order shown or discussed, including in a substantially simultaneous manner or in a reverse order according to the functions involved.

[0231] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, which can be embodied in any computer-readable medium for use by an instruction execution system, apparatus or device (such as a computer-based system, a system including a processor or other system that can fetch instructions from an instruction execution system, apparatus or device and execute instructions), or used in combination with these instruction execution systems, apparatuses or devices.

[0232] It should be understood that the various parts of the present application can be implemented with hardware, software, firmware or a combination thereof. In the above embodiments, multiple steps or methods can be implemented with software or firmware stored in a memory and executed by a suitable instruction execution system. All or part of the steps of the above embodiment method can be completed by instructing the relevant hardware through a program, which can be stored in a computer-readable storage medium, and when the program is executed, it includes one of the steps of the method embodiment or a combination thereof.

[0233] In addition, each functional unit in each embodiment of the present application can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into one module. The above-mentioned integrated module can be implemented in the form of hardware or in the form of a software functional module. If the above-mentioned integrated module is implemented in the form of a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. The storage medium can be a read-only memory, a disk or an optical disk, etc.

[0234] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of various changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A 5G audio and video monitoring quality optimization method based on deep learning, characterized by: Acquire audio, video, and environmental sensor data; Preprocess and analyze audio, video, and environmental data through deep learning models; Optimize the audio and video data according to the preprocessing and analysis results; The optimized audio and video data is transmitted to the monitoring center via the 5G network.

2. The method according to claim 1, characterized in that The audio data is collected by a high-sensitivity microphone array and processed by spatial filtering and beamforming technology.

3. The method according to claim 2, characterized in that The processing formula for spatial filtering and beamforming is: Where y(t) is the synthesized audio signal, is the audio signal collected by the ith microphone at time t, is the delay of the iiith microphone, is the frequency of the ith microphone and N is the number of microphones.

4. The method according to claim 1, characterized in that The video data is collected by a high-resolution camera and image enhancement is performed using multi-frame image fusion and complex function.

5. The method according to claim 4, characterized in that The multi-frame image fusion formula is: Among them, L is the fused image, is the i-th exposure image, is the weight function and N is the number of images.

6. The method according to claim 1, characterized in that The environmental data is collected through environmental sensors, and Kalman filtering and unscented Kalman filtering are used to perform data fusion and filtering processing.

7. The method according to claim 6, characterized in that The processing formula of the unscented Kalman filter is: in, is the observational prediction, is the covariance between states and observations, is the observation covariance.

8. The method according to claim 1, characterized in that Audio, video and environmental data are synchronized through efficient timestamp synchronization and cubic spline interpolation technology.

9. The method according to claim 1, characterized in that: The optimization process includes data denoising and data compression, which are processed using wavelet transform and principal component analysis respectively.

10. A 5G audio and video monitoring quality optimization system based on deep learning, characterized in that: include: Data acquisition module, used to collect audio, video and environmental data; Data preprocessing module, used for denoising and compressing the collected data; Deep learning analysis module, used to analyze and optimize the preprocessed data; The 5G communication module is used to transmit the optimized data to the monitoring center through the 5G network.

Images