Power plant high-risk environmental data supervision system based on multimodal data analysis

Abstract

The present invention relates to the field of data processing technology, and in particular to a power plant high-risk environment data monitoring system based on multimodal data analysis. The system comprises a memory, a processor, and a computer program stored in the memory and running on the processor. When the processor executes the computer program, the following steps are implemented: losslessly compressing a monitoring video to obtain a predicted frame and a video bit rate; if it is detected that the video bit rate needs to be adjusted, obtaining information complexity based on residual information and motion vectors of each predicted frame, and obtaining an image quality index based on the gradient amplitude of pixel points on the image of each predicted frame; obtaining a compression loss volume of each predicted frame based on network quality, information complexity, and image quality index, obtaining a corresponding adaptive quantization parameter based on the compression loss volume of each predicted frame, and then adjusting the video bit rate. The monitoring video is transmitted based on the adjusted video bit rate, thereby improving the transmission quality of the monitoring video.

Description

Technical Field

[0001] The present invention relates to the technical field of data processing, and in particular to a power plant high-risk environment data supervision system based on multimodal data analysis. Background Art

[0002] Electrical operations in high-risk environments in the power industry pose significant safety risks. Existing error prevention technologies often rely on a single data source (such as electronic operation tickets or basic monitoring). These technologies lack the ability to deeply integrate and conduct real-time analysis of multimodal data, including video, audio, and images, making it difficult to accurately predict risks in complex environments. The power plant high-risk environment data monitoring system, based on multimodal data analysis, collects multimodal data such as video, audio, QR codes, images, and text. Leveraging technologies such as artificial intelligence and the Internet of Things, it enables real-time safety monitoring of power plant operations in high-risk environments, effectively preventing safety incidents caused by improper operation or environmental complexity. However, power plant environments may be subject to interference from various electromagnetic waves, resulting in significant delays or video freezes when real-time monitoring videos of on-site operations are transmitted to the monitoring end. This can affect the monitoring end's ability to safely monitor and provide guidance on the on-site operations. Therefore, the bitrate of the transmitted video needs to be adjusted.

[0003] Traditionally, the quantization parameter in the H.264 encoder is configured to perform a certain degree of lossy compression on the video, thereby reducing the video bitrate and achieving smooth video transmission. For every increase of 6 in the quantization parameter, the bitrate is approximately halved. A larger quantization parameter results in poorer image quality, a lower bitrate, and improved smoothness. Conversely, a smaller quantization parameter results in better image quality, a higher bitrate, and poorer smoothness. Therefore, the quantization parameter should be appropriately set and then used to adjust the bitrate to find the optimal balance between image quality and smoothness, ensuring optimal video transmission quality.

[0004] However, the quantization parameters in traditional methods are often based on manual configuration. When the quantization parameters are fixed, the video bit rate is determined solely by the complexity of the content and cannot be adjusted in a timely manner based on fluctuations in the network bandwidth in the power plant environment, which in turn affects the transmission quality of the video during transmission.

[0005] Therefore, how to obtain adaptive quantization parameters to adjust the bit rate of transmitted video and improve the transmission quality of video has become an urgent problem to be solved. Summary of the Invention

[0006] In view of this, an embodiment of the present invention provides a power plant high-risk environment data monitoring system based on multimodal data analysis to solve the problem of how to obtain adaptive quantization parameters to adjust the bit rate of transmitted video and improve the transmission quality of video.

[0007] An embodiment of the present invention provides a power plant high-risk environmental data monitoring system based on multimodal data analysis, comprising a memory, a processor, and a computer program stored in the memory and running on the processor, wherein the processor implements the following steps when executing the computer program:

[0008] Obtaining a monitoring video of the power plant at the current moment, performing H.264 encoding compression processing on the monitoring video to obtain at least one predicted frame and a video bit rate of the monitoring video, wherein the predicted frame includes a P frame and a B frame;

[0009] Adjusting and detecting the video bit rate according to the network quality of the power plant; if it is detected that the video bit rate needs to be adjusted, obtaining, for any predicted frame, information complexity of the predicted frame based on residual information and a motion vector of the predicted frame; performing image edge detection on the predicted frame to obtain at least one edge; and obtaining an image quality index of the predicted frame based on a gradient magnitude of a pixel point on each edge;

[0010] According to the network quality of the power plant, as well as the information complexity and image quality index of any predicted frame, the compression loss volume of any predicted frame is obtained. According to the compression loss volume of each predicted frame, the adaptive quantization parameter of each video frame when the monitoring video is compressed by H.264 encoding is obtained. According to the adaptive quantization parameter of each video frame, the video bit rate is adjusted to obtain the adjusted video bit rate, and the monitoring video is transmitted according to the adjusted video bit rate.

[0011] Preferably, the adjusting and detecting the video bit rate according to the network quality of the power plant includes:

[0012] Obtaining the network available bandwidth and packet loss rate of the power plant, calculating a difference between a constant 1 and the packet loss rate, and calculating the product of the difference, the network available bandwidth, and a preset network condition fault tolerance parameter to obtain the effective bandwidth of the power plant;

[0013] If the effective bandwidth is less than the video bit rate, it is determined that the video bit rate needs to be adjusted; if the effective bandwidth is greater than or equal to the video bit rate, it is determined that the video bit rate does not need to be adjusted.

[0014] Preferably, obtaining the information complexity of any predicted frame according to the residual information and the motion vector of any predicted frame includes:

[0015] Calculating the absolute value of each residual pixel in the residual matrix of any predicted frame, and linearly normalizing the average of the absolute values ​​of all residual pixels to obtain the mean amplitude value of the residual information of any predicted frame;

[0016] Calculating the amplitude of each motion vector in any one of the predicted frames, and linearly normalizing the average of the amplitudes of all the motion vectors to obtain the mean amplitude of the motion vectors in any one of the predicted frames;

[0017] Accumulating the residual information amplitude mean and the motion vector amplitude mean to obtain the information feature value of any predicted frame;

[0018] If any of the predicted frames is a P frame, weighting the information feature value of any of the predicted frames according to a preset weight of the P frame to obtain the information complexity of the any of the predicted frames;

[0019] If any of the predicted frames belongs to a B frame, the information feature value of any of the predicted frames is weighted according to a preset weight of the B frame to obtain the information complexity of the any of the predicted frames.

[0020] Preferably, obtaining the image quality index of any predicted frame according to the gradient amplitude of the pixel points on each edge includes:

[0021] For any edge, calculating the average of the gradient amplitudes of all pixels on the edge to obtain the edge gradient amplitude of the edge;

[0022] Calculating an average of edge gradient amplitudes of all edges in any one predicted frame to obtain an average edge gradient amplitude, obtaining a maximum value and a minimum value among all edge gradient amplitudes, calculating a difference between the maximum value and the minimum value to obtain a gradient amplitude distribution characteristic value, calculating a sum of a preset constant and the gradient amplitude distribution characteristic value to obtain a sum result, calculating a ratio between the average edge gradient amplitude and the sum result to obtain an edge display consistency index of the predicted frame;

[0023] The product of the average edge gradient amplitude and the edge display consistency index is calculated to obtain the image quality index of any predicted frame.

[0024] Preferably, obtaining the compression loss volume of any predicted frame according to the network quality of the power plant and the information complexity and image quality index of any predicted frame includes:

[0025] Obtaining the network available bandwidth and packet loss rate of the power plant, calculating the difference between a constant 1 and the packet loss rate, calculating the product of the difference, the network available bandwidth, and a preset network condition fault tolerance parameter to obtain the effective bandwidth of the power plant, calculating the difference between the video bit rate and the effective bandwidth, and obtaining an estimated theoretical loss volume of the surveillance video;

[0026] Obtaining the information complexity and image quality index of all predicted frames, and linearly normalizing the information complexity and image quality index of all predicted frames to obtain corresponding information complexity normalized values ​​and image quality index normalized values;

[0027] Calculating a constant 1 minus a normalized value of the image quality index of any predicted frame to obtain a first difference, adding the first difference to the normalized value of the information complexity of any predicted frame to obtain a feature value of the any predicted frame, accumulating the feature values ​​of all predicted frames to obtain an accumulation result, and using the reciprocal of the accumulation result as an overall loss coefficient for all predicted frames;

[0028] Calculating a constant 1 minus the normalized information complexity value of any predicted frame to obtain a second difference, adding the second difference to the normalized image quality index value of any predicted frame to obtain an addition result, and calculating the product of the addition result and the overall loss coefficient to obtain a loss volume ratio coefficient of any predicted frame;

[0029] The product of the loss volume ratio coefficient of any predicted frame and the theoretical loss volume estimation value is calculated to obtain the compression loss volume of any predicted frame.

[0030] Preferably, obtaining the adaptive quantization parameter of each video frame when performing H.264 encoding compression on the surveillance video according to the compression loss volume of each predicted frame includes:

[0031] For any predicted frame, the true volume of the predicted frame is obtained, the ratio between the compression loss volume of the predicted frame and the true volume is calculated, and the product of the ratio and the preset quantization parameter mapping factor is used as the adaptive quantization parameter of the predicted frame.

[0032] Compared with the prior art, the embodiments of the present invention have the following beneficial effects:

[0033] The present invention obtains a monitoring video of a power plant at a current moment, performs H.264 encoding compression processing on the monitoring video to obtain at least one predicted frame and a video bit rate of the monitoring video, wherein the predicted frame includes a P frame and a B frame; the video bit rate is adjusted and detected according to the network quality of the power plant; if it is detected that the video bit rate needs to be adjusted, for any predicted frame, the information complexity of the predicted frame is obtained according to the residual information and motion vector of the predicted frame; image edge detection is performed on the predicted frame to obtain at least one edge; the image quality index of the predicted frame is obtained according to the gradient amplitude of the pixel points on each edge; the compression loss volume of the predicted frame is obtained according to the network quality of the power plant and the information complexity and image quality index of the predicted frame; the adaptive quantization parameter of each video frame when the monitoring video is compressed by H.264 encoding is obtained according to the compression loss volume of each predicted frame; the video bit rate is adjusted according to the adaptive quantization parameter of each video frame to obtain the adjusted video bit rate; and the monitoring video is transmitted according to the adjusted video bit rate. Among them, the information complexity and image display quality (image quality index) contained in each predicted frame are analyzed to obtain the compression loss volume of each predicted frame. Then, the adaptive quantization parameter of each predicted frame is obtained based on the compression loss volume. Then, the video bit rate of the monitoring video is adjusted according to the adaptive quantization parameter to balance the image quality and smoothness of the monitoring video, improve the transmission quality of the monitoring video, and ensure the accuracy and real-time performance of the monitoring end's safety monitoring and operation guidance of the power plant's on-site operations. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0035] Figure 1 This is a flowchart of a method for supervising high-risk environmental data in a power plant based on multimodal data analysis, provided in Example 1 of the present invention. DETAILED DESCRIPTION

[0036] The embodiments of the present disclosure are described in detail below, and examples of the embodiments are shown in the accompanying drawings. The embodiments described below with reference to the accompanying drawings are exemplary and intended to be used to explain the present disclosure, but should not be understood as limiting the present disclosure.

[0037] It should be noted that the terms "first," "second," and the like in the specification of the present disclosure and the accompanying drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or precedence. It should be understood that the terms used in this manner are interchangeable where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure.

[0038] In order to illustrate the technical solution of the present invention, specific embodiments are provided below.

[0039] The specific scenario targeted by the present invention is: during the on-site operation of a power plant, it is necessary to collect a variety of data in real time to monitor its safety behavior during the operation. However, due to the influence of environmental factors, the real-time monitoring video collected at the power plant site may have problems such as freezes and large delays during transmission. Therefore, in the traditional way, the quantization parameters in the H.264 encoder are configured to perform lossy compression on the monitoring video, thereby reducing the bit rate of the video and achieving the purpose of smooth transmission of the monitoring video. However, the quantization parameters in the traditional way are often based on manual configuration and cannot be adjusted in time based on the fluctuation of the network bandwidth in the environment of the power plant, which in turn affects the transmission quality of the video during the transmission process. Therefore, the present invention obtains the adaptive quantization parameters in the H.264 encoder to adjust the bit rate of the transmitted video and improve the transmission quality of the video.

[0040] An embodiment of the present invention provides a power plant high-risk environment data supervision system based on multimodal data analysis, comprising a processor and a memory, wherein the processor executes a computer program stored in the memory to implement a power plant high-risk environment data supervision method based on multimodal data analysis, such as Figure 1 As shown, the method comprises the following steps:

[0041] Step S101 : obtaining a monitoring video of the power plant at the current moment, performing H.264 encoding compression processing on the monitoring video, and obtaining at least one predicted frame and a video bit rate of the monitoring video, wherein the predicted frame includes a P frame and a B frame.

[0042] Electrical operations in high-risk environments in the power industry pose significant safety risks. Therefore, real-time safety monitoring of power plant operations in high-risk environments is required, and surveillance video must be transmitted to the monitoring terminal. This allows personnel to monitor and provide operational guidance for on-site operations, effectively preventing safety incidents caused by improper operation or environmental complexity. However, in actual applications, power plant environments may be subject to interference from various electromagnetic waves, resulting in significant delays or video freezes when real-time surveillance video of on-site power plant operations is transmitted to the monitoring terminal. This can affect the monitoring terminal's ability to monitor and provide operational guidance during on-site operations, necessitating adjustments to the bit rate of the transmitted video.

[0043] Traditionally, H.264 compression technology is used to encode and compress videos. This algorithm achieves a certain degree of lossy compression on the video by configuring the quantization parameter (QP) and other methods, thereby reducing the video bit rate and achieving the purpose of smooth video transmission: for every increase of 6 in the quantization parameter, the bit rate is approximately halved. The larger the quantization parameter, the lower the bit rate; conversely, the smaller the quantization parameter, the higher the bit rate. However, the quantization parameters in traditional methods are often manually configured. When the quantization parameters are fixed, the video bit rate is determined only by the complexity of the content and cannot be adjusted in time based on fluctuations in the network bandwidth in the power plant environment, which in turn affects the transmission quality of the video during the transmission process.

[0044] Therefore, in an embodiment of the present invention, the monitoring video of the power plant at the current moment is first obtained, and then the monitoring video is losslessly compressed using the H.264 compression technology, that is, the initial quantization parameter (QP) is set to 0, to obtain the video bit rate of the monitoring video and three types of frames, namely I frames, P frames and B frames. Among them, the I frame is called a key frame, which is a completely encoded independent frame that does not depend on other frames and is used to provide a reference picture for the monitoring video; the P frame and the B frame are called predicted frames, and the P frame is also called a forward predicted frame, which is used to record the difference between the current frame and its previous frame (I or P frame) and only stores the changed part; the B frame is also called a bidirectional predicted frame, which is used to record the difference between the current frame and the frames before and after it and only stores the changed part. The P frame and the B frame are mainly used to store residual information and the motion vector of the object. The video bitrate is used to determine whether the current surveillance video requires compressed transmission. Furthermore, since I-frames are fully encoded, independent frames that do not rely on other frames and serve as the baseline for surveillance video, they are typically not compressed. Therefore, in this embodiment of the present invention, only predicted frames (P-frames and B-frames) are analyzed to adjust their initial quantization parameters, thereby adjusting the video bitrate of the surveillance video. H.264 compression technology is currently available and will not be further described here.

[0045] In step S102, the video bit rate is adjusted and detected according to the network quality of the power plant. If it is detected that the video bit rate needs to be adjusted, for any predicted frame, the information complexity of the predicted frame is obtained based on the residual information and motion vector of the predicted frame, and image edge detection is performed on the predicted frame to obtain at least one edge. The image quality index of the predicted frame is obtained based on the gradient amplitude of the pixel points on each edge.

[0046] When transmitting surveillance videos, the network bandwidth fluctuates due to interference from various electromagnetic waves in the power plant's environment. To ensure smooth transmission of surveillance videos, the video bit rate of the surveillance videos needs to be adjusted according to changes in network bandwidth. Therefore, in an embodiment of the present invention, the network quality of the power plant at the current moment is first tested to determine whether the surveillance videos at the current moment can be transmitted smoothly in a lossless state. Specifically:

[0047] The TCP BBR detection algorithm is used to measure the network available bandwidth of the power plant at the current moment, and the packet loss rate is obtained based on ACK (positive feedback) and NACK (negative feedback). The TCP BBR detection algorithm, ACK (positive feedback) and NACK (negative feedback) are existing technologies and will not be described here. The difference between the constant 1 and the packet loss rate is calculated, and the product of the difference, the network available bandwidth and the preset network condition fault tolerance parameter is calculated to obtain the effective bandwidth of the power plant, which is recorded as R, that is,

[0048]

[0049] Wherein, R represents the effective bandwidth of the power plant, D represents the available network bandwidth, L represents the packet loss rate, and K represents the preset network condition fault tolerance parameter, which represents the fault tolerance parameter estimated for the current network condition. According to experimental statistics, K is set to 0.8 in the embodiment of the present invention. There is no restriction here, and the implementer can set it according to the specific scenario.

[0050] It should be noted that the larger the network available bandwidth at the current moment, the smaller the packet loss rate, and the larger the effective bandwidth. That is, when the network at the current moment is used to transmit surveillance video, the smoother the surveillance video picture will be.

[0051] If the effective bandwidth is greater than or equal to the video bitrate of the surveillance video, the network quality is good at the current moment and the surveillance video can be transmitted with high quality. There is no need to compress and transmit the surveillance video, that is, the video bitrate of the surveillance video does not need to be adjusted. If the effective bandwidth is less than the video bitrate of the surveillance video, the network quality is poor at the current moment. If the surveillance video is not compressed and transmitted, insufficient network bandwidth and network instability may cause packet loss, leading to delayed surveillance video transmission or playback interruptions, which in turn affects the monitoring end's ability to monitor the safety of the power plant's on-site operations in real time and provide operational guidance. Therefore, it is necessary to increase the initial quantization parameter (QP) of each predicted frame (P frame and B frame) of the surveillance video to achieve compressed transmission of the surveillance video, thereby reducing the video bitrate of the surveillance video. This ensures that the surveillance video can still be transmitted stably and smoothly even with poor network bandwidth, thereby improving the transmission quality of the surveillance video.

[0052] Since each predicted frame records residual information and motion vectors, the magnitude of the residual information and motion vectors directly reflects the dynamic characteristics of the content in the surveillance video. The larger the residual information and motion vectors, the stronger the dynamic characteristics of the content in the surveillance video and the more complex the content, that is, the richer the information contained in the predicted frame. If it is significantly compressed, that is, the initial quantization parameter is significantly increased, important information will be lost. Therefore, the increase in the initial quantization parameter should be reduced to ensure the content quality of the surveillance video. Conversely, the smaller the residual information and motion vectors, the weaker the dynamic characteristics of the content in the surveillance video and the simpler the content, that is, the more non-important information is contained in the predicted frame. At this time, it can be significantly compressed, that is, the initial quantization parameter is significantly increased to reduce the volume occupied by non-important information, thereby reducing the video bit rate of the surveillance video. Therefore, in an embodiment of the present invention, taking the i-th predicted frame as an example, by analyzing the residual information and motion vector of the i-th predicted frame, the information complexity of the i-th predicted frame is obtained, and then the initial quantization parameter of the i-th predicted frame is adjusted according to the information complexity to reduce the video bit rate of the surveillance video and achieve smooth transmission of the surveillance video. The specific method of obtaining the information complexity of the i-th prediction frame is:

[0053] Calculate the absolute value of each residual pixel in the residual matrix of the i-th prediction frame, linearly normalize the average of the absolute values ​​of all residual pixels, and obtain the mean amplitude of the residual information of the i-th prediction frame;

[0054] Calculate the amplitude of each motion vector in the i-th predicted frame, and linearly normalize the average of the amplitudes of all motion vectors to obtain the mean amplitude of the motion vector of the i-th predicted frame, wherein the linear normalization is a prior art and will not be described in detail here;

[0055] Accumulating the residual information amplitude mean and the motion vector amplitude mean to obtain the information feature value of the i-th predicted frame;

[0056] If the i-th predicted frame belongs to a P frame, the information feature value of the i-th predicted frame is weighted according to the preset weight of the P frame to obtain the information complexity of the i-th predicted frame;

[0057] If the i-th predicted frame belongs to a B frame, the information feature value of the i-th predicted frame is weighted according to the preset weight of the B frame to obtain the information complexity of the i-th predicted frame.

[0058] In one embodiment, the information complexity of the i-th prediction frame is calculated as follows:

[0059]

[0060] in, represents the information complexity of the i-th prediction frame, represents the mean amplitude of the residual information of the i-th prediction frame, represents the mean amplitude of the motion vector of the i-th predicted frame, Indicates the preset weight, represents the linear normalization function.

[0061] It should be noted that the larger the mean value of the residual information amplitude and the mean value of the motion vector amplitude are, the more complex the content of the i-th prediction frame is, that is, the richer the information contained in the i-th prediction frame, and thus the higher the information complexity of the i-th prediction frame. When transmitting the surveillance video, the increase in the initial quantization parameter should be reduced to ensure that the loss of important information in the i-th prediction frame is less, thereby reducing the video bit rate of the surveillance video while ensuring the content quality of the surveillance video. Considering that the B frame is a bidirectional prediction frame, which contains information of bidirectional frames, compared with the P frame, lossy compression of the B frame may result in more information loss, so the preset weight of the B frame is set to 0.7, and the preset weight of the P frame is set to 0.3, that is, if the i-th prediction frame belongs to the P frame, then ; If the i-th predicted frame belongs to a B frame, then There is no restriction here, and implementers can set it according to specific scenarios.

[0062] Since higher-quality images contain more details and color gradients, this additional information will be discarded as redundant information during compression. The human eye is less sensitive to such details, that is, the difference in perception after image quality loss is small. Conversely, for lower-quality images, the image itself has already lost a lot of information (such as reduced color bands and blockiness), which will directly destroy structural information (such as sharp edges or large blocks of color) after compression, resulting in obvious visual degradation. Therefore, if the initial quantization parameters of the predicted frame are adjusted only according to the information complexity of the predicted frame, then when the encoded and compressed surveillance video is transmitted to the monitoring end for decoding, it may cause details or content in the surveillance video to be lost, that is, the image quality of the surveillance video is poor, which in turn affects the monitoring end's safety monitoring and operation guidance of the on-site operation process.

[0063] Therefore, in an embodiment of the present invention, an image of the i-th prediction frame is obtained, and the image is grayscaled using OpenCV to obtain a grayscale image. Then, the Sobel operator is used to perform edge detection on the grayscale image to obtain an edge detection image. Furthermore, skeletonization and branch node methods are used to obtain each edge in the edge detection image. The clarity of each edge is reflected according to the gradient amplitude of the pixel points on each edge to obtain an image quality index of the i-th prediction frame, which is used to evaluate the image quality of the i-th prediction frame, that is, the image quality. Then, the initial quantization parameter of the i-th prediction frame is adjusted according to the image quality index to ensure the image quality of the surveillance video while reducing the video bit rate of the surveillance video: the larger the gradient amplitude of the pixel points on the edge, the clearer the edge, the better the image quality of the i-th prediction frame, and the better the image quality. At this time, the initial quantization parameter should be greatly increased so that the loss volume after compression is larger. Conversely, the smaller the gradient amplitude, the worse the image quality, and the increase in the initial quantization parameter should be reduced so that the loss volume after compression is smaller. Among them, using OpenCV to grayscale the image, using the Sobel operator to perform edge detection on the grayscale image, skeletonization and branch node method are existing technologies and will not be described here in detail.

[0064] The specific process of obtaining the image quality index of the i-th predicted frame is:

[0065] For any edge, calculating the average of the gradient amplitudes of all pixels on the edge to obtain the edge gradient amplitude of the edge;

[0066] Calculating an average of the edge gradient amplitudes of all edges in the i-th predicted frame to obtain an average edge gradient amplitude, obtaining a maximum value and a minimum value among all the edge gradient amplitudes, calculating a difference between the maximum value and the minimum value to obtain a gradient amplitude distribution characteristic value, calculating a sum of a preset constant and the gradient amplitude distribution characteristic value to obtain a sum result, calculating a ratio between the average edge gradient amplitude and the sum result to obtain an edge display consistency index of the predicted frame;

[0067] The product of the average edge gradient amplitude and the edge display consistency index is calculated to obtain the image quality index of the i-th predicted frame.

[0068] In one embodiment, the image quality index of the i-th predicted frame is calculated as follows:

[0069]

[0070] in, represents the image quality index of the i-th predicted frame, represents the average edge gradient amplitude, Represents the maximum value of all edge gradient magnitudes, Represents the minimum value of all edge gradient magnitudes, Indicates a preset constant.

[0071] It should be noted that c is used to prevent the denominator from being zero. In the embodiment of the present invention, c=0.01 is set. There is no restriction here, and the implementer can set it according to the specific scenario. It is the characteristic value of the gradient amplitude distribution. The smaller it is, the more concentrated the display effect of all edges in the image of the i-th prediction frame is, and the better the image quality is. The larger it is, the better the image quality of the i-th prediction frame is, the smaller the difference in perception after image quality loss is, and the greater the degree of compression of the i-th prediction frame is, that is, the initial quantization parameter of the i-th prediction frame should be greatly increased, so that the loss volume after compression is larger, so as to reduce the video bit rate of the monitoring video. The larger it is, the clearer each edge in the image of the i-th prediction frame is, and the larger it is, the better the image quality of the i-th prediction frame is, the smaller the difference in perception after image quality loss is, and the greater the degree of compression of the i-th prediction frame is, that is, the initial quantization parameter of the i-th prediction frame should be greatly increased, so that the loss volume after compression is larger, so as to reduce the video bit rate of the monitoring video.

[0072] At this point, the information complexity and image quality index of the i-th predicted frame have been obtained, completing the analysis of the i-th predicted frame. Similarly, the information complexity and image quality of each predicted frame of the monitoring video are obtained. Step S103: Based on the network quality of the power plant and the information complexity and image quality index of each predicted frame, the compression loss volume of each predicted frame is obtained. Based on the compression loss volume of each predicted frame, the adaptive quantization parameter of each video frame when the monitoring video is compressed using H.264 encoding is obtained. The video bit rate is adjusted based on the adaptive quantization parameter of each video frame to obtain the adjusted video bit rate, and the monitoring video is transmitted based on the adjusted video bit rate. In the process of compressing the surveillance video, since the volume loss (image quality loss after compression) shows a certain positive correlation with the quantization parameter (QP), and the quantization parameter (QP) shows a certain negative correlation with the video bit rate after the surveillance video is compressed, that is, the larger the quantization parameter (QP), the smaller the volume of the surveillance video after compression, the greater the volume loss, the more obvious the image quality loss, and the lower the video bit rate. Therefore, in an embodiment of the present invention, after obtaining the information complexity and image quality index of the i-th prediction frame through step S102, the loss volume of the i-th prediction frame after compression is further estimated based on the information complexity and image quality index of the i-th prediction frame to obtain the compression loss volume, which is used to obtain the adaptive quantization parameter of the i-th prediction frame according to the compression loss volume, and then adjust the bit rate of the surveillance video. The specific method of obtaining the compression loss volume of the i-th predicted frame is: Obtain the network available bandwidth and packet loss rate of the power plant, calculate the difference between the constant 1 and the packet loss rate, calculate the product of the difference, the network available bandwidth and the preset network condition fault tolerance parameter to obtain the effective bandwidth of the power plant, calculate the difference between the video bit rate and the effective bandwidth, and obtain the theoretical loss volume estimate of the monitoring video, recorded as S, that is, where S represents the theoretical loss volume estimate of the monitoring video, M represents the video bit rate of the monitoring video (the monitoring video is losslessly compressed, that is, the video bit rate when the initial quantization parameter is 0), D represents the network available bandwidth, L represents the packet loss rate, and K represents the preset network condition fault tolerance parameter (K=0.8 is set in the embodiment of the present invention); Obtain the information complexity and image quality index of all predicted frames, linearly normalize the information complexity and image quality index of all predicted frames respectively, and obtain the corresponding information complexity normalized value and image quality index normalized value, where linear normalization is a prior art and will not be repeated here;

[0073] Calculate a constant 1 minus the normalized value of the image quality index of the i-th prediction frame to obtain a first difference, add the first difference to the normalized value of the information complexity of the i-th prediction frame to obtain the eigenvalue of the i-th prediction frame, accumulate the eigenvalues ​​of all prediction frames to obtain an accumulated result, and use the inverse of the accumulated result as the overall loss coefficient of all prediction frames; Calculate a constant 1 minus the normalized value of the information complexity of the i-th prediction frame to obtain a second difference, add the second difference to the normalized value of the image quality index of the i-th prediction frame to obtain an added result, calculate the product between the added result and the overall loss coefficient to obtain the loss volume ratio coefficient of the i-th prediction frame; Calculate the product between the loss volume ratio coefficient of the i-th prediction frame and the theoretical loss volume estimate to obtain the compression loss volume of the i-th prediction frame. In one embodiment, the calculation formula for the compression loss volume of the i-th prediction frame is:

[0074]

[0075] in, represents the compression loss volume of the i-th prediction frame, S represents the estimated value of the theoretical loss volume of the surveillance video, represents the information complexity of the i-th prediction frame, represents the image quality index of the i-th predicted frame, m represents the number of predicted frames of the surveillance video, that is, the total number of P frames and B frames, represents the information complexity of the j-th prediction frame, represents the image quality index of the j-th predicted frame, represents the linear normalization function.

[0076] It should be noted that in order to reduce the video bit rate of the surveillance video while ensuring the image quality of the surveillance video, the smaller it is, the less important information is contained in the i-th prediction frame. In order to reduce the volume occupied by non-important information, it needs to be significantly compressed at this time, that is, the larger it is; the larger it is, the better the image quality of the i-th prediction frame, and the difference in appearance after compression is small, and the structural information of the i-th prediction frame will not be destroyed. Therefore, it needs to be significantly compressed to reduce the video bit rate of the surveillance video, that is, the larger it is. Furthermore, the adaptive quantization parameter of the i-th prediction frame is obtained according to the compression loss volume of the i-th prediction frame. Specifically: Use the FFprobe command to extract the true volume of the i-th prediction frame, that is, the volume when the initial quantization parameter of the i-th prediction frame is 0, calculate the ratio between the compression loss volume of the i-th prediction frame and the true volume, and use the product of the ratio and the preset quantization parameter mapping factor as the adaptive quantization parameter of the i-th prediction frame, wherein the FFprobe command is a prior art and will not be repeated here. In one embodiment, the calculation formula of the adaptive quantization parameter of the i-th prediction frame is:

[0077]

[0078] in, represents the adaptive quantization parameter of the i-th predicted frame, represents the compression loss volume of the i-th predicted frame, represents the true volume of the i-th predicted frame, and A represents the preset quantization parameter mapping factor.

[0079] It should be noted that A is used to characterize the conversion coefficient between the adaptive quantization parameter and the compression loss volume. According to experimental statistics, A=1.2 is set. There is no restriction here, and implementers can set it according to specific scenarios. The larger the value, the higher the proportion of compression loss in the i-th prediction frame. The larger it is, that is, the greater the increase in the initial quantization parameter of the i-th prediction frame, the lower the image quality of the i-th prediction frame, the higher the smoothness of the surveillance video, and the lower the video bit rate of the surveillance video.

[0080] Similarly, the adaptive quantization parameters of each predicted frame of the surveillance video are obtained, and the initial quantization parameters of each I frame of the surveillance video are used as its corresponding adaptive quantization parameters. Furthermore, based on the adaptive quantization parameters of each video frame of the surveillance video, the surveillance video is encoded and compressed using H.264 compression technology to obtain the compressed surveillance video. At the same time, the FFprobe command is used to obtain the video bit rate of the compressed surveillance video, and the compressed surveillance video is transmitted. In summary, the present invention obtains the monitoring video of the power plant at the current moment, performs H.264 encoding compression processing on the monitoring video, and obtains at least one predicted frame and the video bit rate of the monitoring video, wherein the predicted frame includes a P frame and a B frame; the video bit rate is adjusted and detected according to the network quality of the power plant; if it is detected that the video bit rate needs to be adjusted, for any predicted frame, the information complexity of the any predicted frame is obtained according to the residual information and motion vector of the any predicted frame, and image edge detection is performed on the any predicted frame to obtain at least one edge; the image quality index of the any predicted frame is obtained according to the gradient amplitude of the pixel points on each edge; the compression loss volume of the any predicted frame is obtained according to the network quality of the power plant and the information complexity and image quality index of the any predicted frame; the adaptive quantization parameter of each video frame when the monitoring video is compressed by H.264 encoding is obtained according to the compression loss volume of each predicted frame; the video bit rate is adjusted according to the adaptive quantization parameter of each video frame to obtain the adjusted video bit rate, and the monitoring video is transmitted according to the adjusted video bit rate. Among them, the information complexity and image display quality (image quality index) contained in each predicted frame are analyzed to obtain the compression loss volume of each predicted frame, and then the adaptive quantization parameter of each predicted frame is obtained according to the compression loss volume, and then the video bit rate of the monitoring video is adjusted according to the adaptive quantization parameter to balance the image quality and smoothness of the monitoring video, improve the transmission quality of the monitoring video, and ensure the accuracy and real-time performance of the monitoring terminal's safety monitoring and operation guidance of the power plant's on-site operations. The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it; although the present invention has been described in detail with reference to the above embodiments, ordinary technicians in this field should understand that it is still possible to modify the technical solutions described in the above embodiments, or to replace some of the technical features therein; and these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included in the protection scope of the present invention.

Claims

1. A power plant high-risk environmental data monitoring system based on multimodal data analysis, characterized by: The system comprises a memory, a processor, and a computer program stored in the memory and running on the processor, wherein the processor implements the following steps when executing the computer program: Obtaining a monitoring video of the power plant at the current moment, performing H.264 encoding compression processing on the monitoring video to obtain at least one predicted frame and a video bit rate of the monitoring video, wherein the predicted frame includes a P frame and a B frame; Adjusting and detecting the video bit rate according to the network quality of the power plant; if it is detected that the video bit rate needs to be adjusted, obtaining, for any predicted frame, information complexity of the predicted frame based on residual information and a motion vector of the predicted frame; performing image edge detection on the predicted frame to obtain at least one edge; and obtaining an image quality index of the predicted frame based on a gradient magnitude of a pixel point on each edge; Obtaining, based on the network quality of the power plant and the information complexity and image quality index of any predicted frame, a compression loss volume of any predicted frame; obtaining, based on the compression loss volume of each predicted frame, an adaptive quantization parameter for each video frame when performing H.264 encoding compression on the monitoring video; adjusting the video bit rate based on the adaptive quantization parameter of each video frame to obtain an adjusted video bit rate; and transmitting the monitoring video based on the adjusted video bit rate; The method of obtaining, based on the compression loss volume of each predicted frame, an adaptive quantization parameter for each video frame when performing H.264 encoding compression on the surveillance video, includes: For any predicted frame, the true volume of the predicted frame is obtained, the ratio between the compression loss volume of the predicted frame and the true volume is calculated, and the product of the ratio and the preset quantization parameter mapping factor is used as the adaptive quantization parameter of the predicted frame.

2. The power plant high-risk environment data supervision system based on multimodal data analysis according to claim 1 is characterized in that: The adjusting and detecting the video bit rate according to the network quality of the power plant includes: Obtaining the network available bandwidth and packet loss rate of the power plant, calculating a difference between a constant 1 and the packet loss rate, and calculating the product of the difference, the network available bandwidth, and a preset network condition fault tolerance parameter to obtain the effective bandwidth of the power plant; If the effective bandwidth is less than the video bit rate, it is determined that the video bit rate needs to be adjusted; if the effective bandwidth is greater than or equal to the video bit rate, it is determined that the video bit rate does not need to be adjusted.

3. The power plant high-risk environment data supervision system based on multimodal data analysis according to claim 1 is characterized in that: The obtaining, according to the residual information and the motion vector of the any predicted frame, the information complexity of the any predicted frame includes: Calculating the absolute value of each residual pixel in the residual matrix of any predicted frame, and linearly normalizing the average of the absolute values ​​of all residual pixels to obtain the mean amplitude value of the residual information of any predicted frame; Calculating the amplitude of each motion vector in any one of the predicted frames, and linearly normalizing the average of the amplitudes of all the motion vectors to obtain the mean amplitude of the motion vectors in any one of the predicted frames; Accumulating the residual information amplitude mean and the motion vector amplitude mean to obtain the information feature value of any predicted frame; If any of the predicted frames is a P frame, weighting the information feature value of any of the predicted frames according to a preset weight of the P frame to obtain the information complexity of the any of the predicted frames; If any of the predicted frames belongs to a B frame, the information feature value of any of the predicted frames is weighted according to a preset weight of the B frame to obtain the information complexity of the any of the predicted frames.

4. The power plant high-risk environment data supervision system based on multimodal data analysis according to claim 1 is characterized in that: Obtaining the image quality index of any predicted frame according to the gradient magnitude of the pixel points on each edge includes: For any edge, calculating the average of the gradient amplitudes of all pixels on the edge to obtain the edge gradient amplitude of the edge; Calculating an average of edge gradient amplitudes of all edges in any one predicted frame to obtain an average edge gradient amplitude, obtaining a maximum value and a minimum value among all edge gradient amplitudes, calculating a difference between the maximum value and the minimum value to obtain a gradient amplitude distribution characteristic value, calculating a sum of a preset constant and the gradient amplitude distribution characteristic value to obtain a sum result, calculating a ratio between the average edge gradient amplitude and the sum result to obtain an edge display consistency index of the predicted frame; The product of the average edge gradient amplitude and the edge display consistency index is calculated to obtain the image quality index of any predicted frame.

5. The power plant high-risk environment data supervision system based on multimodal data analysis according to claim 1 is characterized in that: The obtaining, based on the network quality of the power plant and the information complexity and image quality index of any predicted frame, of the compression loss volume of any predicted frame includes: Obtaining the network available bandwidth and packet loss rate of the power plant, calculating the difference between a constant 1 and the packet loss rate, calculating the product of the difference, the network available bandwidth, and a preset network condition fault tolerance parameter to obtain the effective bandwidth of the power plant, calculating the difference between the video bit rate and the effective bandwidth, and obtaining an estimated theoretical loss volume of the surveillance video; Obtaining the information complexity and image quality index of all predicted frames, and linearly normalizing the information complexity and image quality index of all predicted frames to obtain corresponding information complexity normalized values ​​and image quality index normalized values; Calculating a constant 1 minus a normalized value of the image quality index of any predicted frame to obtain a first difference, adding the first difference to the normalized value of the information complexity of any predicted frame to obtain a feature value of the any predicted frame, accumulating the feature values ​​of all predicted frames to obtain an accumulation result, and using the reciprocal of the accumulation result as an overall loss coefficient for all predicted frames; Calculating a constant 1 minus the normalized information complexity value of any predicted frame to obtain a second difference, adding the second difference to the normalized image quality index value of any predicted frame to obtain an addition result, and calculating the product of the addition result and the overall loss coefficient to obtain a loss volume ratio coefficient of any predicted frame; The product of the loss volume ratio coefficient of any predicted frame and the theoretical loss volume estimation value is calculated to obtain the compression loss volume of any predicted frame.

Images