Monitoring video quality diagnosis method based on image processing

Abstract

The invention provides a monitoring video quality diagnosis method based on image processing, and the method comprises the steps: classifying a working condition environment according to the sound level, temperature value and vibration intensity of equipment under normal operation; audio, temperature and vibration sensors are placed on the site of various working condition monitoring cameras so as to synchronously collect multi-modal data of working condition environments; carrying out data enhancement and preprocessing on the multi-modal data and then normalizing the multi-modal data into an input vector; and training a multi-modal deep learning model by using the input vector, and deploying the multi-modal deep learning model to monitor and diagnose the monitoring video quality in real time. The method has the beneficial effect that the capability of identifying and diagnosing the fault picture of the monitoring video is improved.

Description

Technical Field

[0001] The invention belongs to the technical field of monitoring videos, and in particular relates to a monitoring video quality diagnosis method based on image processing. Background Art

[0002] In industrial production, visual recognition technology has long been applied to various process links in production, such as quality inspection and classification, production process monitoring, defect prediction, etc. Each frame of image captured by these monitoring devices may contain key information. Due to working environment problems, the same monitoring device may face different noise interference or other faults in different environments. Therefore, it is necessary to collect, analyze and diagnose various fault images of monitoring videos. Traditional monitoring quality assessment generally requires steps such as data collection, feature extraction, and fault classification. It mainly relies on image processing technology for image feature extraction, but it often faces the challenges of low detection rate and false alarms. Alternatively, deep neural network models can be used to provide more accurate quality assessments, but they require large video data sets, and training these networks requires high-performance server hardware, which is costly and time-consuming. Summary of the invention

[0003] In view of this, the present invention aims to propose a monitoring video quality diagnosis method based on image processing to improve the recognition and diagnosis capabilities of monitoring video fault images.

[0004] To achieve the above object, the technical solution of the present invention is achieved as follows: Surveillance video quality diagnosis method based on image processing.

[0005] Further, the following steps are included: T1. Classification of working conditions: Classify the working conditions according to the sound level, temperature value and vibration intensity under normal operation of the equipment; T2. Data collection: Place audio, temperature, and vibration sensors at various working condition monitoring cameras to synchronously collect multi-modal data of the working environment; T3, data enhancement and preprocessing: perform data enhancement and preprocessing on multimodal data and normalize them into input vectors; T4. Training and deployment: Use input vectors to train multimodal deep learning models, and deploy multimodal deep learning models to monitor and diagnose surveillance video quality in real time.

[0006] Furthermore, in T1, the operating environment is divided into normal operating conditions, abnormal operating conditions, and faulty operating conditions. Normal operating conditions indicate that the sound level, temperature value, and vibration intensity are all within the normal threshold range. Abnormal operating conditions indicate that one or two of the sound level, temperature value, and vibration intensity exceed the safety threshold. Faulty operating conditions indicate that the sound level, temperature value, and vibration intensity all exceed the safety threshold.

[0007] Furthermore, in T3, data enhancement and time synchronization are performed on the multimodal data.

[0008] Furthermore, the T3 multimodal deep learning model includes an input layer, a fusion layer, and a fully connected layer. The input layer includes a video branch, an audio branch, a temperature branch, and a vibration branch, which are used to extract features of video data, audio data, temperature data, and vibration data, respectively. The fusion layer performs weighted fusion of each feature data and outputs it to the fully connected layer to predict the diagnosis result.

[0009] Furthermore, the temporal attention mechanism and the spatial attention mechanism are used to strengthen the images of key frames and key areas in images in continuous video data. The temporal attention mechanism is also used to identify key sound events in continuous audio data, and key time points in continuous temperature data and vibration data.

[0010] Furthermore, temporal attention processing of video data includes: ; In the formula, is the video feature after time attention weighting, Q, K, V are the query, key and value matrices of the time attention mechanism respectively, is the dimension of key K; Spatial attention processing video data includes: ; In the formula, is the video feature processed by the spatial attention mechanism, The original video features The result after reshaping along the spatial dimension, is the spatial attention weight matrix, is the spatial attention bias vector; Spatial attention processing of audio data includes: ; In the formula, , , are the query, key, and value matrices of the temporal attention mechanism, Yes The dimension of It is the audio feature after time attention weighting; Spatial attention processes temperature data including: ; In the formula, , , are the query, key, and value matrices of the temporal attention mechanism, Yes The dimension of is the temperature feature after time attention weighting; Spatial attention processing of vibration data includes: ; In the formula, , , are the query, key, and value matrices of the temporal attention mechanism, Yes The dimension of It is the temperature feature after time attention weighting.

[0011] Furthermore, the fusion formula of the fusion layer includes: ; In the formula, To fuse the eigenvalues, α, β, and γ are the weighting coefficients of video, audio, and temperature-vibration coupling features, respectively.

[0012] Furthermore, an electronic device includes a processor and a memory that is communicatively connected to the processor and is used to store instructions executable by the processor, and the processor is used to execute the monitoring video quality diagnosis method based on image processing.

[0013] Furthermore, a server includes at least one processor and a memory communicatively connected to the processor, wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the processor so that the at least one processor executes the monitoring video quality diagnosis method based on image processing.

[0014] Furthermore, a computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the monitoring video quality diagnosis method based on image processing is implemented.

[0015] Compared with the prior art, the monitoring video quality diagnosis method based on image processing described in the present invention has the following beneficial effects: (1) The multimodal data fusion described in the present invention integrates video, audio, temperature, and vibration data to provide a more comprehensive equipment status analysis, improve diagnostic accuracy and system robustness, and avoid the impact of single data source failure on monitoring results; (2) The temporal and spatial attention mechanism described in the present invention focuses on the key moments and areas of video, audio, temperature and vibration data through the attention mechanism, improves the feature extraction capability, accurately captures abnormal or fault information, and improves detection efficiency and accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The accompanying drawings constituting a part of the present invention are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the accompanying drawings: Figure 1 A schematic diagram of a video quality diagnosis method according to an embodiment of the present invention; Figure 2 This is a schematic diagram of the structure of the multimodal deep learning model described in an embodiment of the present invention. DETAILED DESCRIPTION

[0017] It should be noted that, in the absence of conflict, the embodiments of the present invention and the features in the embodiments may be combined with each other.

[0018] The present invention will be described in detail below with reference to the accompanying drawings and in conjunction with embodiments.

[0019] Surveillance video quality diagnosis methods based on image processing, such as Figure 1 The following steps are shown: T1. Classification of working conditions: Classify the working conditions according to the sound level, temperature value and vibration intensity under normal operation of the equipment; T2. Data collection: Place audio, temperature, and vibration sensors at various working condition monitoring cameras to synchronously collect multi-modal data of the working environment; T3, data enhancement and preprocessing: perform data enhancement and preprocessing on multimodal data and normalize them into input vectors; T4, Training and Deployment: Use input vectors to train multimodal deep learning models (preferably neural network models), and deploy multimodal deep learning models to monitor and diagnose surveillance video quality in real time. Use audio sensors to measure the sound characteristics of the device during operation, such as decibel level, spectral characteristics, etc. Use temperature sensors to monitor temperature changes on the device surface and in the environment. Use vibration sensors to capture the vibration frequency and amplitude of the device during operation.

[0020] Specifically, in T1, the operating environment is divided into normal operating conditions, abnormal operating conditions, and fault operating conditions. Normal operating conditions indicate that the sound level, temperature value, and vibration intensity are all within the normal threshold range. Abnormal operating conditions indicate that one or two of the sound level, temperature value, and vibration intensity exceed the safety threshold. Fault operating conditions indicate that the sound level, temperature value, and vibration intensity all exceed the safety threshold.

[0021] Specifically, in T3, data enhancement and time synchronization are performed on multimodal data, including but not limited to using spectral subtraction to estimate the power spectral density of audio noise, and using a method based on the statistical distribution of local pixel intensities to estimate the video noise level to attenuate audio and video noise.

[0022] Specifically, Figure 2The T3 multimodal deep learning model shown includes an input layer, a fusion layer, and a fully connected layer. The input layer includes a video branch, an audio branch, a temperature branch, and a vibration branch, which are used to extract the features of video data, audio data, temperature data, and vibration data, respectively. The fusion layer performs weighted fusion of each feature data and outputs it to the fully connected layer to predict the diagnosis result. The temporal attention mechanism and the spatial attention mechanism are used to strengthen the images of key frames in continuous video data and the key areas in the images. At the same time, the temporal attention mechanism is used to identify key sound events in continuous audio data, and key time points of continuous temperature data and vibration data. For example, when the video screen shakes or the device vibrates abnormally, the model can quickly capture the key frames or key areas, thereby effectively diagnosing the video quality.

[0023] Temporal attention processing of video data includes: ; In the formula, is the video feature after time attention weighting, Q, K, V are the query, key and value matrices of the time attention mechanism respectively, is the dimension of key K; Spatial attention processing video data includes: ; In the formula, is the video feature processed by the spatial attention mechanism, The original video features The result after reshaping along the spatial dimension, is the spatial attention weight matrix, is the spatial attention bias vector; Spatial attention processing of audio data includes: ; In the formula, , , are the query, key, and value matrices of the temporal attention mechanism, Yes The dimension of It is the audio feature after time attention weighting; Spatial attention processes temperature data including: ; In the formula, , , are the query, key, and value matrices of the temporal attention mechanism, Yes The dimension of is the temperature feature after time attention weighting; Spatial attention processing of vibration data includes: ; In the formula, , , are the query, key, and value matrices of the temporal attention mechanism, Yes The dimension of It is the temperature feature after time attention weighting.

[0024] Furthermore, in many working conditions, there is a coupling relationship between temperature data and vibration data. Therefore, the fusion layer needs to consider the fusion characteristics after the temperature-vibration coupling effect. The fusion formula of the fusion layer includes: ; In the formula, is the fusion feature value, α, β, and γ are the weighted coefficients of video, audio, and temperature-vibration coupling features respectively. The fusion result data is processed by the fully connected layer to output the video diagnosis result, for example: A quality problem was detected in a certain video. Overall evaluation: Video quality score: 75 / 100, medium quality. Detection results: Video shaking: abnormal.

[0025] Audio Noise: Normal.

[0026] Temperature fluctuations: Normal.

[0027] Vibration data: abnormal, the vibration amplitude exceeds the threshold.

[0028] Those of ordinary skill in the art will appreciate that the units and method steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in terms of function in the above description. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0029] In the several embodiments provided in the present application, it should be understood that the disclosed methods and systems can be implemented in other ways. For example, the division of the units described above is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. The above-mentioned units may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the embodiments of the present invention.

[0030] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein by equivalents. These modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present invention, and they should all be included in the scope of the claims and specification of the present invention.

[0031] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A surveillance video quality diagnosis method based on image processing, characterized in that: The following steps are involved: T1. Classification of working conditions: Classify the working conditions according to the sound level, temperature value and vibration intensity under normal operation of the equipment; T2. Data collection: Place audio, temperature, and vibration sensors at various working condition monitoring cameras to synchronously collect multi-modal data of the working environment; T3, data enhancement and preprocessing: perform data enhancement and preprocessing on multimodal data and normalize them into input vectors; T4. Training and deployment: Use input vectors to train multimodal deep learning models, and deploy multimodal deep learning models to monitor and diagnose surveillance video quality in real time.

2. The monitoring video quality diagnosis method based on image processing according to claim 1 is characterized in that: In T1, the operating environment is divided into normal operating conditions, abnormal operating conditions, and fault operating conditions. Normal operating conditions indicate that the sound level, temperature value, and vibration intensity are all within the normal threshold range. Abnormal operating conditions indicate that one or two of the sound level, temperature value, and vibration intensity exceed the safety threshold. Fault operating conditions indicate that the sound level, temperature value, and vibration intensity all exceed the safety threshold.

3. The monitoring video quality diagnosis method based on image processing according to claim 1 is characterized in that: In T3, data augmentation and temporal synchronization are performed on multimodal data.

4. The monitoring video quality diagnosis method based on image processing according to claim 1 is characterized in that: The T3 multimodal deep learning model includes an input layer, a fusion layer, and a fully connected layer. The input layer includes a video branch, an audio branch, a temperature branch, and a vibration branch, which are used to extract features of video data, audio data, temperature data, and vibration data, respectively. The fusion layer performs weighted fusion of each feature data and outputs it to the fully connected layer to predict the diagnosis result.

5. The monitoring video quality diagnosis method based on image processing according to claim 4 is characterized in that: The temporal attention mechanism and the spatial attention mechanism are used to strengthen the images of key frames and key areas in continuous video data. The temporal attention mechanism is also used to identify key sound events in continuous audio data, and key time points in continuous temperature data and vibration data.

6. The monitoring video quality diagnosis method based on image processing according to claim 5 is characterized in that: Temporal attention processing of video data includes: ； In the formula, is the video feature after time attention weighting, Q, K, V are the query, key and value matrices of the time attention mechanism respectively, is the dimension of key K; Spatial attention processing video data includes: ； In the formula, is the video feature processed by the spatial attention mechanism, The original video features The result after reshaping along the spatial dimension, is the spatial attention weight matrix, is the spatial attention bias vector; Spatial attention processing of audio data includes: ； In the formula, , , are the query, key, and value matrices of the temporal attention mechanism, Yes The dimension of It is the audio feature after time attention weighting; Spatial attention processes temperature data including: ； In the formula, , , are the query, key, and value matrices of the temporal attention mechanism, Yes The dimension of is the temperature feature after time attention weighting; Spatial attention processing of vibration data includes: ； In the formula, , , are the query, key, and value matrices of the temporal attention mechanism, Yes The dimension of It is the temperature feature after time attention weighting.

7. The monitoring video quality diagnosis method based on image processing according to claim 6 is characterized in that: The fusion formula of the fusion layer includes: ； In the formula, To fuse the eigenvalues, α, β, and γ are the weighting coefficients of video, audio, and temperature-vibration coupling features, respectively.

8. An electronic device, comprising a processor and a memory connected to the processor for storing instructions executable by the processor, characterized in that: The processor is used to execute the monitoring video quality diagnosis method based on image processing as described in any one of claims 1 to 7.

9. A server, characterized in that: It includes at least one processor and a memory communicatively connected to the processor, wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the processor so that the at least one processor executes the monitoring video quality diagnosis method based on image processing as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the monitoring video quality diagnosis method based on image processing according to any one of claims 1 to 7 is implemented.

Images