A video power grid frequency estimation method and system based on illumination area detection and fusion

Abstract

This invention proposes a video power grid frequency estimation method and system based on illumination region detection and fusion. The invention constructs a power grid frequency estimation region dataset and uses this dataset to train an illumination region detection model, resulting in an optimized illumination region detection network. Feature frames to be estimated from the video are extracted. These feature frames are then input into the optimized illumination region detection network for detection, yielding multiple estimation regions. The average pixel intensity within each estimation region is calculated, resulting in multiple illumination sequences. Each illumination sequence is preprocessed to obtain a preprocessed illumination sequence. A short-time Fourier transform is performed on the preprocessed illumination sequences, and quadratic interpolation is used to estimate the power grid frequency signal for each region. Finally, the multiple power grid frequency signals from each region are fused to obtain the final power grid frequency estimation result. This invention improves the efficiency of illumination region detection, the accuracy of video power grid frequency estimation, removes noisy regions from the image, and expands the applicability of video power grid frequency estimation.

Description

Technical Field

[0001] This invention belongs to the field of digital signal processing, and specifically relates to a video power grid frequency estimation method and system based on illumination area detection and fusion. Background Technology

[0002] Electric Network Frequency (ENF) is the transmission frequency of alternating current (AC). In my country, the nominal value of the power grid frequency is 50Hz. The power grid frequency is not constant; it is typically monitored and continuously adjusted in real time by the local power control department. Therefore, the power grid frequency fluctuates around its nominal value. Because the actual direction and intensity of the power grid frequency adjustments change with real-world conditions, and such changes are unlikely to repeat in actual production and daily life, the power grid frequency is unique within a certain period. Furthermore, the power grid frequency possesses characteristics such as real-time performance, continuity, and consistency within the same power grid. Therefore, the power grid frequency is widely used in fields such as audio and video forensics and analysis, timestamp verification, and multimedia synchronization.

[0003] Currently, audio-based methods for estimating power grid frequency are relatively mature, while video-based methods still have many shortcomings. Video power grid frequency signals are primarily obtained by capturing brightness changes caused by light source flickering in the power grid environment. Since the frequency of light source changes is twice the power grid frequency, the nominal value of the power grid frequency recorded in the video is 100Hz. However, the sampling frequency of video recording is approximately 30 frames per second. According to the Nyquist sampling theorem, frequency aliasing occurs at 100Hz, making it impossible to extract the correct power grid frequency signal. Therefore, for video power grid frequency estimation, estimation should be performed at a frequency shift of 10Hz. However, current video power grid frequency estimation still faces challenges such as significant interference from the image content and poor estimation results in complex scenes, such as excessively strong or weak lighting, cluttered images, and slight camera shake. Therefore, detecting illuminated areas within the image content, using the detected portions for power grid frequency estimation, and then performing multi-region fusion can address these problems to some extent.

[0004] Given the two-dimensional nature of video images, the embedding of power grid frequency signals within the video is non-uniformly distributed. Regions near light sources exhibit higher embedding strength of the power grid frequency signal. Therefore, we propose a deep learning network architecture for detecting illuminated regions. This network architecture can quickly and easily identify regions in video images with less interference and stronger power grid frequency signals through object detection networks and orientation awareness mechanisms. Estimating the power grid frequency within these regions and fusing the estimated signals effectively addresses the problems of significant interference from unwanted signals and difficulty in effective estimation in complex scenes, thus expanding the applicability of video power grid frequency estimation to a certain extent. Summary of the Invention

[0005] The purpose of this invention is to provide a video power grid frequency estimation method and system for illumination region detection and fusion, which can be used for video power grid frequency estimation in complex scenarios.

[0006] The technical solution of this invention is a video power grid frequency estimation method based on illumination region detection and fusion, specifically including the following steps:

[0007] Step 1: Construct a power grid frequency estimation region dataset, and use the self-built dataset to train the illumination region detection model to obtain the optimized illumination region detection network;

[0008] Step 2: Extract feature frames from the video to be estimated;

[0009] Step 3: Input the feature frames into the optimized illumination region detection network for detection to obtain multiple estimated regions;

[0010] Step 4: Calculate the average pixel intensity within each estimated region to obtain multiple illumination sequences containing power grid frequency information in the video;

[0011] Step 5: Preprocess each illumination sequence to obtain a preprocessed illumination sequence to eliminate the effects of residual background and motion;

[0012] Step 6: Perform a short-time Fourier transform on the preprocessed illumination sequence and use quadratic interpolation to estimate the power grid frequency signal for each region;

[0013] Step 7: Fusion of multiple power grid frequency signals from each region to obtain the final video power grid frequency estimation result.

[0014] Preferably, the method for constructing the power grid frequency estimation regional dataset in step 1 is as follows:

[0015] Extracting P from a video containing power grid frequency signals N 3 images, each with a size of P. X ×P Y ;

[0016] The extracted image is scaled to change its size.

[0017] The scaled image is labeled, with labels divided into two categories: light sources and stable reflective objects;

[0018] Mark all regions in the image that contain these two types of objects, and save them using the following format:

[0019] (L,X center ,Y Center ,W,H)

[0020] Where L is the label for this area, 0 represents a light source, and 1 represents a stable reflecting object. X center Y Center The x and y coordinates represent the center point of the region, and W and H represent the width and height of the region.

[0021] Step 1 describes training the illumination region detection model using a self-built dataset, as follows:

[0022] The self-built dataset is constructed as follows:

[0023] A lighting region detection network is constructed. Each sample in the dataset is input into the lighting region detection network to obtain preliminary prediction results. A loss function model is constructed by combining the real results. The optimized lighting region detection network is obtained through training.

[0024] The illumination area detection network includes a first part network, a second part network, a third part network, a fourth part network, and a fifth part network;

[0025] The first part of the network is a convolutional feature extraction network for extracting image features. It consists of 13 3×3 convolutional layers and 4 pooling layers. At the same time, a ReLU activation function layer is added after each convolutional layer.

[0026] This network extracts feature maps from the input image;

[0027] The second part of the network is a direction-aware feature extraction network, which consists of an attention weight extraction mechanism network and a spatial context-aware network;

[0028] The attention weight extraction mechanism network first inputs the feature map into two 3×3 convolutional layers, and then adds a ReLU activation function layer after each convolutional layer.

[0029] Then, it is fed into a 1×1 convolutional layer to obtain the output W, and then the output W is decomposed into four attention weight matrices Wi. left W right Wup W down ;

[0030] The spatial context-aware network first inputs the feature map into a 1×1 convolutional layer, and then denotes the pixel value at (i,j) in the output feature map as h. i,j The context-aware information can then be calculated using the following formula:

[0031] h ij =max{α right h i,j-1 +h ij ,0}

[0032] This operation executes n w Next, n w α is the feature map width. right The directional weight matrix is ​​initialized as an identity matrix and optimized through learning. This formula provides the calculation formula for the right-side contextual information features; the same applies to other directions. Through context-aware calculation, context-aware feature maps for the four directions (up, down, left, and right) can be obtained, which, together with the four attention weight matrices W obtained in the first part, yield the desired four attention weight matrices. left W right W up W down Multiply and merge the results, then pass them through a 1×1 convolutional layer to obtain the first context-aware feature map. Repeat the above context-aware operation once to obtain an image feature map containing context information.

[0033] The third part of the network is a region candidate network. First, the image feature map containing contextual information is input into the region candidate network. After passing through a 3×3 convolutional layer and a ReLU activation function layer, it is input into two parallel 1×1 convolutional layers. One convolutional output is used for softmax classification, and the other convolutional output is used for boundary regression. Finally, the output is the region containing the class to be detected.

[0034] The fourth part of the network is an interest region pooling network. The region containing the class to be detected is cropped in the image feature map and the max pooling method is used to make the size uniform.

[0035] The fifth part of the network is an interest domain classification and regression network. The input is the uniform-sized output of the interest pooling network, which is fed into two fully connected layers. After each fully connected layer, a ReLU activation function layer is added. Then, it is fed into two parallel fully connected layers. One of them outputs the illumination region, and the other fully connected layer is connected to a softmax function for classification, which outputs the category of the region. Finally, the detected illumination region and category are output.

[0036] The loss function of the network is constructed as follows:

[0037]

[0038] Where, N cls N represents the amount of data fed into the network in each round. reg λ1 and λ2 represent the number of preselected boxes used in the region candidate network, respectively, and p represents the hyperparameters. i This represents the classification probability of the i-th preselected box. The probability of classifying a sample as positive or negative is given when the overlap between the preselected box and the labeled true value is greater than ε. M Or, if the overlap between the preselected box and the true value is the largest, it is considered a positive sample. When the overlap between the preselected box and the actual value of the annotation is less than ε MIN , is a negative sample Where ε is an adjustable control parameter for positive and negative samples. When a sample is neither a positive nor a negative sample, that sample is not used in training.

[0039] t i Parameterize the coordinates of the prediction box. The parameterized coordinates of the real object bounding box can be calculated using the following formula.

[0040]

[0041]

[0042]

[0043]

[0044] Where x, y, h, w are the coordinates of the center point, width, and height of the box, respectively. a , y, h, and w represent the x-coordinates of the center points of the predicted bounding box, the pre-selected bounding box, and the actual bounding box, respectively. L represents the x-coordinates of the center points of the predicted bounding box, the pre-selected bounding box, and the actual bounding box, respectively. reg The following formula can be used to calculate

[0045]

[0046] Where R is the Smooth L1 function.

[0047] L cls The logarithmic loss for both positive and negative classes can be calculated using the following formula.

[0048]

[0049] Where, p i This represents the classification probability of the i-th preselected box. The probability of classifying a sample as positive or negative.

[0050] L cl The classification cross-entropy loss can be calculated using the following formula.

[0051]

[0052] Among them, l ij Let l be the probability that sample i belongs to class j. ij =1 indicates that it belongs to, l ij =0 indicates that it does not belong to, This represents the predicted probability that sample i belongs to class j.

[0053] Furthermore, step 2 involves extracting the feature frames of the video to be estimated, using the following specific method:

[0054] Read the video file frame by frame, denoted as C, with a frame rate of f. s The total number of frames in the video is N, and each frame is F. n Where n∈{1,2,3…N}, and the number of rows in each frame is n. r The number of columns is n c The feature frame can then be calculated using the following formula:

[0055]

[0056] Among them, F r This represents the extracted feature frames.

[0057] Furthermore, in step 3, the feature frames are input into the optimized illumination region detection network for detection to obtain the estimated region. The specific method is as follows:

[0058] Extract the feature frame F r Input the trained illumination region detection network from step 1 to obtain feature frame F. r The rectangular area Ω with high grid frequency signal quality and low noise interference s , where s∈{1,2,3…P}, and P is the total number of regions detected by the network;

[0059] For any rectangular region Ω l It can be represented by the following four-dimensional array:

[0060] (x 1s ,y 1s ,x 2s ,y 2s )

[0061] Where, x 1s ,y 1s Represents the rectangular region Ω s The x and y coordinates of the top-left vertex, x 2s ,y2s Represents the rectangular region Ω s The x and y coordinates of the bottom right vertex;

[0062] Furthermore, step 4 calculates the average pixel intensity within each estimated region to obtain the illumination sequence containing power grid frequency information in the video. The specific method is as follows:

[0063] The original video C is processed according to the rectangular region Ω s Each frame is segmented, and the segmented results are converted to grayscale to obtain several sub-videos C. s ;

[0064] Read each sub-video C s The light intensity of its video pixels can be expressed as F. s (r,c,n), where r is the row coordinate of the pixel, c is the column coordinate of the pixel, and n is the intensity at the nth frame;

[0065] The illumination sequence for each sub-video can then be calculated using the following formula:

[0066]

[0067] Among them, Y s Representative sub-video C s Illumination sequence, Y s [n] represents the average illumination intensity of the video frame at frame n.

[0068] Furthermore, step 5 involves preprocessing each illumination sequence, as detailed below:

[0069] Y s [n] Eliminate the influence of background and motion on power grid frequency estimation, as detailed below:

[0070]

[0071] Among them, M s [n] represents sub-video C s Lighting sequence after removing background and motion effects.

[0072] Furthermore, step 6 involves performing a short-time Fourier transform on the preprocessed illumination sequence, specifically as follows:

[0073] The preprocessed illumination sequence M s [n] is subjected to a short-time Fourier transform to obtain the time-frequency domain sequence X of the illumination sequence. s (l,m), where l represents the l-th frame of the Short Time Fourier Transform (STFT), m represents the m-th frequency point of the STFT, and the number of STFT frequency points is M;

[0074] Step 6 involves estimating the power grid frequency signal for each region using quadratic interpolation. The specific method is as follows:

[0075] Obtain the time-frequency domain sequence X s After (l,m), the ENF value of each frame of STFT is estimated using quadratic interpolation.

[0076] For the estimation of the l-th frame of the STFT, first find X. s (l,m) represents the coordinates of the maximum frequency point in the l-th frame. max Then find the coordinates m of the frequency points to the left and right of the maximum frequency point. l m r The frequency compensation value is calculated using the following formula.

[0077]

[0078] Where Δ is the frequency compensation value, Re(·) represents taking the real part of the complex number, X s In (l,m), l represents the l-th frame of the Short Time Fourier Transform (STFT), and m represents the m-th frequency point of the STFT transformation.

[0079] After obtaining the compensation value, the ENF value of this frame is calculated as follows:

[0080]

[0081] Among them, E s [l] indicates that the sub-video C s The estimated ENF signal value of the l-th frame, Δ is the frequency compensation value, m max For X s (l,m) represents the coordinates of the maximum frequency point in the l-th frame, where M is the number of STFT frequency points, and f s This represents the video frame rate. Repeating the above operation for each frame yields several power grid frequency signal sequences E. s [l];

[0082] Furthermore, step 7 involves fusing multiple regional power grid frequency signals. The specific method is as follows: for multiple sub-video C... s Estimated power grid frequency E s [l], the video power grid frequency estimation result is obtained by fusing the following formula.

[0083]

[0084] Where P is the total number of regions detected by the illumination region detection network mentioned above, and x 1s ,y 1s ,x 2s ,y 2s Let E be the top-left and bottom-right coordinates of the target detection region mentioned earlier.s [l] is for sub-video C s The estimated power grid frequency;

[0085] This invention also provides a video power grid frequency estimation system based on illumination region detection and fusion, comprising:

[0086] The illumination region detection model training module is used to construct a power grid frequency estimation region dataset. The self-built dataset is used to train the illumination region detection model to obtain an optimized illumination region detection network.

[0087] The feature frame extraction module is used to extract feature frames from the video to be estimated.

[0088] The estimated region detection module is used to pass the feature frame through the optimized illumination region detection network to obtain the estimated region;

[0089] The illumination sequence calculation module is used to calculate the average pixel intensity in each estimated region to obtain the illumination sequence containing power grid frequency information in the video;

[0090] The preprocessing module is used to preprocess each illumination sequence to obtain a preprocessed illumination sequence;

[0091] The power grid frequency signal estimation module is used to perform short-time Fourier transform on the preprocessed illumination sequence and estimate the power grid frequency signal for each region using quadratic interpolation.

[0092] The signal fusion module is used to fuse the power grid frequency signals extracted from each region to obtain the video power grid frequency estimation result.

[0093] The advantages of this invention are that,

[0094] An improved illumination region detection network is proposed to segment the image. On the one hand, it can improve the efficiency of illumination region detection, and on the other hand, it removes noisy regions in the image, thereby improving the accuracy and efficiency of video power grid frequency estimation to a certain extent.

[0095] Even in complex scenarios, such as excessively high or low light intensity, cluttered backgrounds, or slight camera shake, this estimation method still achieves high estimation accuracy, expanding the applicability of video power grid frequency estimation.

[0096] The quadratic interpolation estimation method increases the resolution of power grid frequency estimation to some extent and improves its accuracy. Attached Figure Description

[0097] Figure 1 : Flowchart of the method according to an embodiment of the present invention;

[0098] Figure 2: A schematic diagram of the illumination area detection network structure according to an embodiment of the present invention;

[0099] Figure 3 : Schematic diagram of target detection results in feature frames under low illumination and complex background scenarios according to embodiments of the present invention;

[0100] Figure 4 : A schematic diagram illustrating the effect of camera sampling on power grid frequency spectrum shifting in an embodiment of the present invention;

[0101] Figure 5 This is a schematic diagram comparing the power grid frequency signal estimated after multi-region fusion with the power grid frequency signal estimated from the global image and the reference signal in an embodiment of the present invention. Detailed Implementation

[0102] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0103] In specific implementation, the method proposed in the technical solution of this invention can be automatically executed by those skilled in the art using computer software technology. System devices for implementing the method, such as computer-readable storage media storing the corresponding computer program of the technical solution of this invention and computer equipment including the computer program running the corresponding computer program, should also be within the protection scope of this invention.

[0104] The following is combined Figures 1 to 5 The present invention describes embodiments of the present invention. The first embodiment includes the following steps:

[0105] A video power grid frequency estimation method based on illumination region detection and fusion specifically includes the following steps:

[0106] Step 1: Construct a power grid frequency estimation region dataset, and use the self-built dataset to train the illumination region detection model to obtain the optimized illumination region detection network;

[0107] Step 2: Extract feature frames from the video to be estimated;

[0108] Step 3: Input the feature frames into the optimized illumination region detection network for detection to obtain multiple estimated regions;

[0109] Step 4: Calculate the average pixel intensity within each estimated region to obtain multiple illumination sequences containing power grid frequency information in the video;

[0110] Step 5: Preprocess each illumination sequence to obtain a preprocessed illumination sequence to eliminate the effects of residual background and motion;

[0111] Step 6: Perform a short-time Fourier transform on the preprocessed illumination sequence and use quadratic interpolation to estimate the power grid frequency signal for each region;

[0112] Step 7: Fusion of multiple power grid frequency signals from each region to obtain the final video power grid frequency estimation result.

[0113] Preferably, the method for constructing the power grid frequency estimation regional dataset in step 1 is as follows:

[0114] Extracting P from a video containing power grid frequency signals N = 1000 images, each with a size of P X ×P Y =1080×1920;

[0115] The extracted image is scaled to change its size.

[0116] The scaled image is labeled, with labels divided into two categories: light sources and stable reflective objects;

[0117] Mark all regions in the image that contain these two types of objects, and save them using the following format:

[0118] (L,X center ,Y Center ,W,H)

[0119] Where L is the label for this area, 0 represents a light source, and 1 represents a stable reflecting object. X center Y Center The x and y coordinates represent the center point of the region, and W and H represent the width and height of the region.

[0120] Step 1 describes training the illumination region detection model using a self-built dataset, as follows:

[0121] The self-built dataset is constructed as follows:

[0122] A lighting region detection network is constructed. Each sample in the dataset is input into the lighting region detection network to obtain preliminary prediction results. A loss function model is constructed by combining the real results. The optimized lighting region detection network is obtained through training.

[0123] like Figure 2 As shown, the illumination area detection network includes a first part network, a second part network, a third part network, a fourth part network, and a fifth part network;

[0124] The first part of the network is a convolutional feature extraction network for extracting image features. It consists of 13 3×3 convolutional layers and 4 pooling layers. At the same time, a ReLU activation function layer is added after each convolutional layer.

[0125] This network extracts feature maps from the input image;

[0126] The second part of the network is a direction-aware feature extraction network, which consists of an attention weight extraction mechanism network and a spatial context-aware network;

[0127] The attention weight extraction mechanism network first inputs the feature map into two 3×3 convolutional layers, and then adds a ReLU activation function layer after each convolutional layer.

[0128] Then, it is fed into a 1×1 convolutional layer to obtain the output W, and then the output W is decomposed into four attention weight matrices Wi. left W rig h t W up W down ;

[0129] The spatial context-aware network first inputs the feature map into a 1×1 convolutional layer, and then denotes the pixel value at (i,j) in the output feature map as h. i,j The context-aware information can then be calculated using the following formula:

[0130] h ij =max{α right h i,j-1 +h ij ,0}

[0131] This operation executes n w Next, n w α is the feature map width. right The directional weight matrix is ​​initialized as an identity matrix and optimized through learning. This formula provides the calculation formula for the right-side contextual information features; the same applies to other directions. Through context-aware calculation, context-aware feature maps for the four directions (up, down, left, and right) can be obtained, which, together with the four attention weight matrices W obtained in the first part, yield the desired four attention weight matrices. left W rigjt W up W down Multiply and merge the results, then pass them through a 1×1 convolutional layer to obtain the first context-aware feature map. Repeat the above context-aware operation once to obtain an image feature map containing context information.

[0132] The third part of the network is a region candidate network. First, the image feature map containing contextual information is input into the region candidate network. After passing through a 3×3 convolutional layer and a ReLU activation function layer, it is input into two parallel 1×1 convolutional layers. One convolutional output is used for softmax classification, and the other convolutional output is used for boundary regression. Finally, the output is the region containing the class to be detected.

[0133] The fourth part of the network is an interest region pooling network. The region containing the class to be detected is cropped in the image feature map and the max pooling method is used to make the size uniform.

[0134] The fifth part of the network is an interest domain classification and regression network. The input is the uniform-sized output of the interest pooling network, which is fed into two fully connected layers. After each fully connected layer, a ReLU activation function layer is added. Then, it is fed into two parallel fully connected layers. One of them outputs the illumination region, and the other fully connected layer is connected to a softmax function for classification, which outputs the category of the region. Finally, the detected illumination region and category are output.

[0135] The loss function of the network is constructed as follows:

[0136]

[0137] Where, N cls N represents the amount of data fed into the network in each round. reg λ1 and λ2 represent the number of preselected boxes used in the region candidate network, respectively, and p represents the hyperparameters. i This represents the classification probability of the i-th preselected box. The probability of classifying a sample as positive or negative is given when the overlap between the preselected box and the labeled true value is greater than ε. M A positive sample is defined as 70% or when the overlap between the preselected box and the true value is at its maximum. When the overlap between the preselected box and the actual value of the annotation is less than ε MIN =30%, which is a negative sample Where ε M ε MIN These are adjustable positive and negative sample control parameters. A sample that is neither positive nor negative will not participate in training.

[0138] t i Parameterize the coordinates of the prediction box. The parameterized coordinates of the real object bounding box can be calculated using the following formula.

[0139]

[0140]

[0141]

[0142]

[0143] Where x, y, h, w are the coordinates of the center point, width, and height of the box, respectively. a , y, h, and w represent the x-coordinates of the center points of the predicted bounding box, the pre-selected bounding box, and the actual bounding box, respectively. L represents the x-coordinates of the center points of the predicted bounding box, the pre-selected bounding box, and the actual bounding box, respectively. reg The following formula can be used to calculate

[0144]

[0145] Where R is the Smooth L1 function.

[0146] L cls The logarithmic loss for both positive and negative classes can be calculated using the following formula.

[0147]

[0148] Where, p i This represents the classification probability of the i-th preselected box. The probability of classifying a sample as positive or negative.

[0149] L cl The classification cross-entropy loss can be calculated using the following formula.

[0150]

[0151] Among them, l ij Let l be the probability that sample i belongs to class j. ij =1 indicates that it belongs to, l ij =0 indicates that it does not belong to, This represents the predicted probability that sample i belongs to class j.

[0152] Furthermore, step 2 involves extracting the feature frames of the video to be estimated, using the following specific method:

[0153] Read the video file frame by frame, denoted as C, with a frame rate of f. s The total number of frames in the video is N, and each frame is F. n Where n∈{1,2,3…N}, and the number of rows in each frame is n. r The number of columns is n c The feature frame can then be calculated using the following formula:

[0154]

[0155] Among them, F r This represents the extracted feature frames.

[0156] Furthermore, in step 3, the feature frames are input into the optimized illumination region detection network for detection to obtain the estimated region. The specific method is as follows:

[0157] Extract the feature frame F r Input the trained illumination region detection network from step 1 to obtain feature frame F. r The rectangular area Ω with high grid frequency signal quality and low noise interference s , where s∈{1,2,3…P}, and P is the total number of regions detected by the network;

[0158] For any rectangular region Ω l It can be represented by the following four-dimensional array:

[0159] (x 1s ,y 1s ,x 2s ,y 2s )

[0160] Where, x 1s ,y 1s Represents the rectangular region Ω s The x and y coordinates of the top-left vertex, x 2s ,y 2s Represents the rectangular region Ω s The x and y coordinates of the bottom right vertex; Figure 3 The results of feature frame target detection are shown in scenarios with low aurora intensity and complex backgrounds.

[0161] Furthermore, step 4 calculates the average pixel intensity within each estimated region to obtain the illumination sequence containing power grid frequency information in the video. The specific method is as follows:

[0162] The original video C is processed according to the rectangular region Ω s Each frame is segmented, and the segmented results are converted to grayscale to obtain several sub-videos C. s ;

[0163] Read each sub-video C s The light intensity of its video pixels can be expressed as F. s (r,c,n), where r is the row coordinate of the pixel, c is the column coordinate of the pixel, and n is the intensity at the nth frame;

[0164] The illumination sequence for each sub-video can then be calculated using the following formula:

[0165]

[0166] Among them, Y s Representative sub-video C s Illumination sequence, Ys [n] represents the average illumination intensity of the video frame at frame n.

[0167] Furthermore, step 5 involves preprocessing each illumination sequence, as detailed below:

[0168] Y s [n] Eliminate the influence of background and motion on power grid frequency estimation, as detailed below:

[0169]

[0170] Among them, M s [n] represents sub-video C s Lighting sequence after removing background and motion effects.

[0171] Furthermore, step 6 involves performing a short-time Fourier transform on the preprocessed illumination sequence, specifically as follows:

[0172] The preprocessed illumination sequence M s [n] is subjected to a short-time Fourier transform to obtain the time-frequency domain sequence X of the illumination sequence. s (l,m), where l represents the l-th frame of the Short Time Fourier Transform (STFT), m represents the m-th frequency point of the STFT, and the number of STFT frequency points is M;

[0173] For video power grid frequency estimation Figure 4 This demonstrates the effect of camera sampling on the frequency spectrum shifting of the power grid. Generally, the camera's sampling frame rate is 30 frames per second. We will use a common frame rate of 29.96 fps as an example. Figure 4 As shown, the original frequency of the power grid is

[0174] f ENF =k*100

[0175] Where k = 1, 2, 3... represents the harmonics of the power grid frequency. Since the sampling frequency is the same as the frame rate of 29.96Hz,

[0176] Therefore, the sampling frequency is less than twice the power grid frequency. Thus, due to the Nyquist sampling theorem, the video power grid frequency...

[0177] The frequency will undergo a frequency shift phenomenon, and the shifted frequency will be...

[0178] f alias =k*100±m*29.96

[0179] Where m can be a positive integer representing the number of sampling points.

[0180] Step 6 involves estimating the power grid frequency signal for each region using quadratic interpolation. The specific method is as follows:

[0181] Obtain the time-frequency domain sequence X s After (l,m), the ENF value of each frame of STFT is estimated using quadratic interpolation.

[0182] Therefore, the time-frequency domain sequence X is obtained. s After (l,m), the ENF value of each frame of the STFT is estimated using quadratic interpolation. For the estimation of the l-th frame of the STFT, X is first found. s (l,m) represents the coordinates of the maximum frequency point in the l-th frame. max Then find the coordinates m of the frequency points to the left and right of the maximum frequency point. l m r The frequency compensation value is calculated using the following formula.

[0183]

[0184] Where Δ is the frequency compensation value, and Re(·) represents taking the real part of the complex number. After obtaining the compensation value, the ENF value of the frame can be calculated using the following formula.

[0185]

[0186] Among them, E s [l] indicates that the sub-video C s The estimated ENF signal value of the l-th frame, Δ is the frequency compensation value, m max For X s (l,m) represents the coordinates of the maximum frequency point in the l-th frame, where M is the number of STFT frequency points, and f s This represents the video frame rate. Repeating the above operation for each frame yields several power grid frequency signal sequences E. s [l];

[0187] Furthermore, step 7 involves fusing multiple regional power grid frequency signals. The specific method is as follows: for multiple sub-video C... s Estimated power grid frequency E s [l], the video power grid frequency estimation result is obtained by fusing the following formula.

[0188]

[0189] Where P is the total number of regions detected by the illumination region detection network mentioned above, and x 1s ,y 1s ,x 2s ,y 2s Let E be the top-left and bottom-right coordinates of the target detection region mentioned earlier. s [l] is for sub-video C s The estimated power grid frequency;

[0190] Figure 5The comparison extraction results are shown in a specific case study. The solid line represents the power grid frequency signal estimated from the selected area, the short dashed line represents the result estimated using the global image, and the long dashed line represents the reference signal. To clearly display the image, the reference signal and the overall image estimation results were processed by subtracting 0.05 Hz from the reference signal frequency value at each moment and adding 0.05 Hz to the frequency value of the overall image estimation result. To quantitatively describe the estimation results, the Pearson correlation coefficient was used as the evaluation metric.

[0191] The present invention also provides a specific implementation of the system as a video power grid frequency estimation system for illumination region detection and fusion, comprising:

[0192] The illumination region detection model training module is used to construct a power grid frequency estimation region dataset. The self-built dataset is used to train the illumination region detection model to obtain an optimized illumination region detection network.

[0193] The feature frame extraction module is used to extract feature frames from the video to be estimated.

[0194] The estimated region detection module is used to pass the feature frame through the optimized illumination region detection network to obtain the estimated region;

[0195] The illumination sequence calculation module is used to calculate the average pixel intensity in each estimated region to obtain the illumination sequence containing power grid frequency information in the video;

[0196] The preprocessing module is used to preprocess each illumination sequence to obtain a preprocessed illumination sequence;

[0197] The power grid frequency signal estimation module is used to perform short-time Fourier transform on the preprocessed illumination sequence and estimate the power grid frequency signal for each region using quadratic interpolation.

[0198] The signal fusion module is used to fuse the power grid frequency signals extracted from each region to obtain the video power grid frequency estimation result.

[0199] The aforementioned illumination region detection model training module, feature frame extraction module, estimated region detection module, illumination sequence calculation module, preprocessing module, power grid frequency signal estimation module, and signal fusion module are all deployed on the server.

[0200] It should be understood that any parts not described in detail in this specification belong to the prior art.

[0201] It should be understood that the above description of the embodiments is quite detailed, but it should not be considered as a limitation on the scope of protection of the present invention. Those skilled in the art can make substitutions or modifications under the guidance of the present invention without departing from the scope of protection of the claims of the present invention, and all such substitutions or modifications fall within the scope of protection of the present invention. The scope of protection of the present invention should be determined by the appended claims.

Claims

1. A video power grid frequency estimation method based on illumination region detection and fusion, characterized in that, Includes the following steps: Step 1: Construct a power grid frequency estimation region dataset, and use the self-built dataset to train the illumination region detection model to obtain the optimized illumination region detection network; Step 2: Extract feature frames from the video to be estimated; Step 3: Input the feature frames into the optimized illumination region detection network for detection to obtain multiple estimated regions; Step 4: Calculate the average pixel intensity within each estimated region to obtain multiple illumination sequences containing power grid frequency information in the video; Step 5: Preprocess each illumination sequence to obtain a preprocessed illumination sequence to eliminate the effects of residual background and motion; Step 6: Perform a short-time Fourier transform on the preprocessed illumination sequence and use quadratic interpolation to estimate the power grid frequency signal for each region; Step 7: Fusion of multiple power grid frequency signals from each region to obtain the final video power grid frequency estimation result; Step 1, which involves training the illumination region detection model using a self-built dataset, is detailed below: A lighting region detection network is constructed. Each sample in the dataset is input into the lighting region detection network to obtain preliminary prediction results. A loss function model is constructed by combining the real results. The optimized lighting region detection network is obtained through training. The illumination area detection network includes a first part network, a second part network, a third part network, a fourth part network, and a fifth part network; The first part of the network is a convolutional feature extraction network for extracting image features, consisting of 13... It consists of four convolutional layers and four pooling layers, with a ReLU activation function layer added after each convolutional layer; The first part of the network extracts feature maps from the input image; The second part of the network is a direction-aware feature extraction network, which consists of an attention weight extraction mechanism network and a spatial context-aware network; The attention weight extraction mechanism network first inputs the feature map into two layers. The convolutional layers are added, and a ReLU activation function layer is added after each convolutional layer; Then input it into a The output result is obtained in the convolutional layer. Then output the results Decomposed into four attention weight matrices , , , ; The spatial context-aware network first inputs a feature map into a... In the convolutional layer, the output feature map will be... The pixel value at that location is denoted as Perform the following context-aware operations: Calculate context-aware information using the following formula: Among them, the operation of calculating context-aware information is executed. Second-rate, The width of the feature map. The directional weight matrix is ​​initialized as an identity matrix and optimized through learning. This formula provides the calculation formula for the right-direction contextual information features; the same applies to other directions. Through context-aware calculation, context-aware feature maps for the four directions (up, down, left, and right) can be obtained, which, together with the four attention weight matrices obtained in the first part, yield the four attention weight matrices. , , , Multiply, combine, and then pass through one The first context-aware feature map can be obtained by using convolutional layers; Repeat the above context-aware operation once to obtain an image feature map containing context information; The third part of the network is a region candidate network. First, the image feature map containing contextual information is input into the region candidate network, and then... The convolutional layer and a ReLU activation function layer are then input into two parallel layers. In the convolutional layers, one convolutional output is used for softmax classification, and the other convolutional output is used for boundary regression. The final output is the region containing the class to be detected. The fourth part of the network is an interest region pooling network. The region containing the class to be detected is cropped in the image feature map and the max pooling method is used to make the size uniform. The fifth part of the network is an interest domain classification and regression network. The input is the uniformly sized output of the interest pooling network, which is fed into two fully connected layers. After each fully connected layer, a ReLU activation function layer is added. Then, the input is fed into two parallel fully connected layers. One of them outputs the illumination region, and the other fully connected layer is connected to a softmax function for classification, outputting the category of the region. Finally, the detected illumination region and category are output.

2. The video power grid frequency estimation method according to claim 1, characterized in that: Step 1 involves constructing a regional dataset for power grid frequency estimation, and the specific method is as follows: Extracting from videos containing power grid frequency signals Images, each with a size of [size missing]. ; The extracted image is scaled to change its size. ; The scaled image is labeled, with labels divided into two categories: light sources and stable reflective objects; Mark all regions in the image that contain these two types of objects, and save them using the following format: in Label this area, where 0 represents a light source and 1 represents a stable reflecting object; , The x and y coordinates represent the center point of the region. , This represents the width and height of the area.

3. The video power grid frequency estimation method according to claim 2, characterized in that: The loss function of the illumination area detection network is as follows: in, This represents the amount of data sent to the network in each round. This represents the number of preselected boxes used in the regional candidate network. Indicates hyperparameters, Representing the The classification probability of each preselected box. To classify a sample as positive or negative, when the overlap between the preselected box and the labeled true value is greater than... Or, if the overlap between the preselected box and the true value is the largest, it is considered a positive sample. When the overlap between the preselected box and the actual value of the annotation is less than ; , is a negative sample ;in These are adjustable positive and negative sample control parameters; when a sample is neither a positive nor a negative sample, it is not used for training. Parameterize the coordinates of the prediction box. The parameterized coordinates of the real object bounding box can be calculated using the following formula. in, The center point coordinates, width, and height of the prediction box are given. These represent the x-coordinates of the center point of the preselected box and the x-coordinates of the center point of the actual object box, respectively. These represent the ordinates of the center point of the preselection box and the center point of the actual object box, respectively. These represent the width and height of the actual object's bounding box, respectively. The following formula can be used to calculate Where R is the Smooth L1 function; The logarithmic loss for both positive and negative classes can be calculated using the following formula. in, Representing the The classification probability of each preselected box. Assign positive or negative probabilities to the samples; The classification cross-entropy loss can be calculated using the following formula. in, For the sample Belongs to the The probability of a class Indicates belonging to, This indicates that it does not belong to [the group / organization]. Indicates sample Belongs to the Class prediction probability.

4. The video power grid frequency estimation method according to claim 3, characterized in that: Step 2 extracts the feature frames of the video to be estimated. The specific method is as follows: Read the video file frame by frame, and denote the video as... The video frame rate is The total number of video frames is Each frame is ,in The number of lines per frame is The number of columns is The feature frame can then be calculated using the following formula: in, This represents the extracted feature frames.

5. The video power grid frequency estimation method according to claim 4, characterized in that: Step 3 involves inputting the feature frames into the optimized illumination region detection network for detection, thereby obtaining the estimated region. The specific method is as follows: Extracted feature frames Input the trained illumination region detection network from step 1 to obtain feature frames. rectangular area of ​​the power grid frequency signal ,in , This represents the total number of regions detected by the network. For any rectangular region It can be represented by the following four-dimensional array: in, Represents a rectangular area The x and y coordinates of the top left vertex, Represents a rectangular area The x and y coordinates of the bottom right vertex.

6. The video power grid frequency estimation method according to claim 5, characterized in that: Step 4 calculates the average pixel intensity within each estimated region to obtain the illumination sequence containing power grid frequency information in the video. The specific method is as follows: For the original video According to the rectangular area Each frame is segmented, and the segmented results are converted to grayscale to obtain several sub-videos. ; Read each sub-video The light intensity of its video pixels can be expressed as ,in The row coordinates of the pixel are The column coordinates of the pixels. For the first Intensity at frame rate; The illumination sequence for each sub-video can then be calculated using the following formula: in, Representative sub-video Illumination sequence, Indicates the first The average illumination intensity of a video frame.

7. The video power grid frequency estimation method according to claim 6, characterized in that: Step 5 involves preprocessing each illumination sequence, as detailed below: Will To eliminate the influence of background and motion on power grid frequency estimation, the following measures are taken: in, Sub-video Lighting sequence after removing background and motion effects.

8. The video power grid frequency estimation method according to claim 7, characterized in that: Step 6 involves performing a short-time Fourier transform on the preprocessed illumination sequence. The specific method is as follows: The preprocessed lighting sequence Perform a short-time Fourier transform to obtain the time-frequency domain sequence of the illumination sequence. ,in The first digit of the Short-Time Fourier Transform (STFT) is... frame, The STFT transformation is represented by the first The number of STFT frequency points is [number]. ; Step 6 involves estimating the power grid frequency signal for each region using quadratic interpolation. The specific method is as follows: Obtain the time-frequency domain sequence Then, the ENF value of each frame of the STFT is estimated using quadratic interpolation. For the STFT Frame estimation, first find No. Maximum frequency coordinates in the frame Then find the coordinates of the frequency points on the left and right sides of the maximum frequency point. , The frequency compensation value is calculated using the following formula. in, This is the frequency compensation value. To take the real part of a complex number, middle The first digit of the Short-Time Fourier Transform (STFT) is... frame, The STFT transformation is represented by the first One frequency point; After obtaining the compensation value, the ENF value of this frame is calculated as follows: in, Indicates sub-video The estimated first The ENF signal value of the frame. This is the frequency compensation value. for No. The coordinates of the maximum frequency point in the frame. The number of STFT frequency points The video frame rate is used as the reference; repeating the above operation for each frame yields several power grid frequency signal sequences. .

9. The video power grid frequency estimation method according to claim 8, characterized in that: Step 7 involves fusing multiple regional power grid frequency signals. The specific method is as follows: For multiple sub-videos Estimated power grid frequency The video power grid frequency estimation results are obtained by fusing the data using the following formula. in, This represents the total number of regions detected by the illumination area detection network. Let the upper left and lower right coordinates of the target detection area be denoted as . For children's videos The estimated value of the power grid frequency.

10. A video power grid frequency estimation system based on illumination region detection and fusion, wherein the system is implemented according to the video power grid frequency estimation method based on illumination region detection and fusion as described in any one of claims 1 to 9, characterized in that, include: The illumination region detection model training module is used to construct a power grid frequency estimation region dataset. The self-built dataset is used to train the illumination region detection model to obtain an optimized illumination region detection network. The feature frame extraction module is used to extract feature frames from the video to be estimated. The estimated region detection module is used to pass the feature frame through the optimized illumination region detection network to obtain the estimated region; The illumination sequence calculation module is used to calculate the average pixel intensity in each estimated region to obtain the illumination sequence containing power grid frequency information in the video; The preprocessing module is used to preprocess each illumination sequence to obtain a preprocessed illumination sequence; The power grid frequency signal estimation module is used to perform short-time Fourier transform on the preprocessed illumination sequence and estimate the power grid frequency signal for each region using quadratic interpolation. The signal fusion module is used to fuse the power grid frequency signals extracted from each region to obtain the video power grid frequency estimation result.

Images