Method, System and Device for Tracing Homologous Monitoring Videos Based on Pattern Noise

Abstract

The present invention relates to the technical field of surveillance video homology verification, and specifically to a method, system, and device for tracing surveillance videos of the same origin based on pattern noise. The method comprises: screening based on an objective function, obtaining a reference motion-free still image to be restored, a reference motion-free still image to be restored, a motion-free still image to be restored and tested, and a motion-free still image to be restored and tested, according to the screening results, and then superimposing to obtain a restored reference key frame and a restored key frame to be tested; obtaining a key frame matching degree based on a static noise matching degree and a motion noise matching degree, using the structural similarity index value between the restored reference key frame and the corresponding restored key frame to be tested as a weight, and performing a weighted average of the key frame matching degrees to obtain a similarity; when the similarity is greater than a second preset value, the reference video and the video to be tested are from the same surveillance device. The present invention can make the video homology verification results based on pattern noise more accurate.

Description

Technical Field

[0001] The present invention relates to the technical field of surveillance video homology verification, and in particular to a method, system and device for tracing homologous surveillance videos based on pattern noise. Background Art

[0002] With rapid advancements in computer technology, networking, and image transmission and processing, video surveillance has become an integral part of every industry, from road traffic and shopping malls to train stations and even families. These surveillance devices are capable of capturing detailed video footage, which is crucial for analyzing transaction behavior and activity patterns. This is especially true in frontline video investigations, where surveillance video often serves as crucial evidence.

[0003] However, the easy editability of digital images and videos poses significant challenges to forensic evidence collection. To address this challenge, surveillance video homology verification is crucial. This verification aims to confirm whether video clips originate from the same surveillance device, providing solid technical support for case investigation and trial, significantly improving the reliability and authenticity of evidence collection. Notably, with the continuous advancement of video manipulation technology, traditional methods that rely on background similarity to determine video homology are becoming inadequate. Against this backdrop, pattern noise, a stable and unique image sensor noise, has become a new focus of video homology verification. Pattern noise arises from process limitations in imaging sensor manufacturing. It is not only stable across every surveillance device and the media it captures, but even for devices of the same brand and model, the videos captured by their imaging sensors are unique due to variations in pattern noise. Therefore, pattern noise can be considered a device "fingerprint," enabling accurate identification of the capturing device and effective verification of video homology.

[0004] When further exploring PRNU (Photo Response Non-Uniformity) noise within pattern noise, it's important to consider a key factor: core components in surveillance equipment, such as imaging sensors, experience physical wear and tear due to long-term use. Over time, the photosensitive elements and circuit components inevitably age. This aging phenomenon causes a certain degree of degradation or distortion in the PRNU noise carried by the video signal during transmission. This change not only complicates the characteristics of the PRNU noise but also potentially weakens its ability to serve as a unique "fingerprint" for the device, potentially impacting the authenticity of video homology verification results based on pattern noise. Summary of the Invention

[0005] In order to solve the technical problem that the existing video homology verification results based on pattern noise are not accurate enough, the purpose of the present invention is to provide a method for tracing homology surveillance videos based on pattern noise. The technical solution adopted is as follows:

[0006] Extracting key frames from the reference video and the video to be tested respectively to obtain reference key frames and key frames to be tested, and obtaining reference de-motioned still images and reference de-motioned still images, the de-motioned still images to be tested, and the de-motioned still images to be tested respectively based on the reference key frames and the key frames to be tested;

[0007] Calculating the still noise matching degree of corresponding sub-blocks between the reference motion-free still image and the motion-free still image to be tested, and the motion noise matching degree of corresponding sub-blocks between the reference motion-free still image and the motion-free still image to be tested;

[0008] constructing an objective function based on the static noise matching degree, the motion noise matching degree, a first preset value, and a preset iteration step size for iterative screening, and obtaining a reference motion-removed static image to be restored, a reference motion-removed static image to be restored, a motion-removed static image to be restored and tested, and a motion-removed static image to be restored and tested according to the screening result;

[0009] Superimposing the reference motion-free still image to be restored and the reference motion-free still image to be restored to form a restored reference key frame, and superimposing the motion-free still image to be restored and tested and the motion-free still image to be restored and tested to form a restored key frame to be tested;

[0010] Based on the static noise matching degree and the motion noise matching degree, the key frame matching degree between the restored reference key frame and the corresponding restored key frame to be tested is obtained. The structural similarity index value between the restored reference key frame and the corresponding restored key frame to be tested is used as a weight, and the key frame matching degree is weighted averaged to obtain the similarity between the reference video and the video to be tested. When the similarity is greater than a second preset value, the reference video and the video to be tested come from the same monitoring device.

[0011] Furthermore, the process of acquiring the reference motion-free still image and the reference motion-free still image includes:

[0012] Separating the static area and the moving area in the reference key frame;

[0013] Denoising the reference key frame to obtain a denoised reference key frame, and subtracting pixel values at corresponding positions of the reference key frame and the denoised reference key frame to obtain a reference noise image;

[0014] The static area and the motion area are matched with the reference noise image respectively to obtain the reference motion-free static image and the reference motion-free static image.

[0015] Furthermore, the process of obtaining the stationary noise matching degree includes:

[0016] Dividing the reference motion-free still image and the to-be-tested motion-free still image into at least two sub-blocks respectively;

[0017] respectively obtaining pixel values of each pattern noise in each sub-block in the reference motion-free still image and the motion-free still image to be tested;

[0018] Acquire a reference stability factor of each sub-block in the reference motion-free still image and a stability factor to be checked of each sub-block in the motion-free still image to be checked;

[0019] Calculating the squares of the differences between the pixel values corresponding to the pattern noise in the i-th sub-block of the reference de-motioned still image and the i-th sub-block of the to-be-tested de-motioned still image, and calculating the sum of the squares of the differences between the pixel values, where the value of i ranges from 1 to the number of the sub-blocks;

[0020] subtracting the reference stability factor corresponding to the i-th sub-block from the stability factor to be checked, and then dividing the result by the stability factor to be checked to obtain a first ratio, and calculating an absolute value of the first ratio;

[0021] The sum of the squares is multiplied by the absolute value and the negation thereof is taken as an input value of an exponential function with base e, and an output value of the exponential function is used as the static noise matching degree of the i-th sub-block corresponding to the reference de-motioned still image and the de-motioned still image to be tested;

[0022] Repeat the static noise matching degree of the i-th sub-block to obtain the static noise matching degree of each sub-block corresponding to the reference de-motioned still image and the de-motioned still image to be tested.

[0023] Furthermore, the process of obtaining the reference stability factor includes:

[0024] Calculating noise information of noise pixels at each position in each sub-block of the reference motion-free still image, and screening out the noise information greater than the first preset value as pattern noise of the reference motion-free still image;

[0025] Obtaining noise information of a noise pixel at an x-th position in an i-th sub-block of the reference de-motioned still image, where the i-th sub-block contains a minimum value and a maximum value of the pattern noise in the reference key frame, wherein the value of x ranges from 1 to the number of noise pixels at the x-th position in the i-th sub-block;

[0026] A reference stability factor component is obtained by dividing the noise information by the difference between the maximum value and the minimum value after subtracting the minimum value, and the sum of the reference stability factor components is the reference stability factor of the i-th sub-block;

[0027] Repeat the reference stability factor of the i-th sub-block to obtain the reference stability factor of each sub-block in the reference motion-removed still image.

[0028] Furthermore, the noise information acquisition process includes:

[0029] Obtaining a pixel value of a noise pixel at an x-th position in an i-th sub-block in each of the reference motion-removed still images and a pixel mean value of the noise pixel at the x-th position in the i-th sub-block;

[0030] The average value of the absolute value of each pixel value minus the pixel mean is taken as the input value of the exponential function with base e, and the output value of the exponential function is used as the noise information of the noise pixel point at the xth position in the i-th sub-block in the reference de-motion still image.

[0031] Furthermore, the method further includes: before calculating the motion noise matching degree, obtaining the pattern noise of the image to be tested for removing still motion, wherein the process of obtaining the pattern noise of the image to be tested for removing still motion includes:

[0032] Calculating a minimum value of a first standard deviation of a noise variation degree of an i-th sub-block in two adjacent frames of the motion-free still image to be inspected, and calculating a second standard deviation of a noise variation degree of an i-th sub-block in two adjacent frames of the motion-free still image to be inspected, and dividing the second standard deviation by the minimum value to obtain a motion-relative noise variation degree, wherein the value of i ranges from 1 to the number of the sub-blocks;

[0033] The degree of motion relative noise change is divided by the first preset value and then multiplied by the noise information of the noise pixel at the xth position in the i-th sub-block of the motion image to be tested for de-stilling as the pattern noise of the motion image to be tested for de-stilling, where the value of x ranges from 1 to the number of noise pixels at the position in the i-th sub-block.

[0034] Furthermore, the objective function includes:

[0035] The value of the preset iteration step minus the first preset value is used as the input value of an exponential function with base e, and the output value of the exponential function is multiplied by the absolute value of the noise matching degree to obtain the objective function, where the absolute value of the noise matching degree is the absolute value of the motion noise matching degree minus the absolute value of the stationary noise matching degree.

[0036] Furthermore, the process of obtaining the key frame matching degree includes:

[0037] respectively calculating a sum of the stationary noise matching degree and the motion noise matching degree corresponding to each of the sub-blocks;

[0038] An average value of the added values is calculated as the key frame matching degree.

[0039] An embodiment of the present invention further provides a device for tracing homologous surveillance videos based on pattern noise, the device comprising:

[0040] an acquisition module, configured to extract key frames from the reference video and the video to be tested to obtain reference key frames and key frames to be tested, and to obtain reference motion-removed still images and reference motion-removed still images, the motion-removed still images to be tested, and the motion-removed still images to be tested, based on the reference key frames and the key frames to be tested;

[0041] a calculation module, configured to calculate a still noise matching degree of corresponding sub-blocks between the reference motion-removed still image and the motion-removed still image to be tested, and a motion noise matching degree of corresponding sub-blocks between the reference motion-removed still image and the motion-removed still image to be tested;

[0042] a screening module, configured to construct an objective function based on the static noise matching degree, the motion noise matching degree, a first preset value, and a preset iteration step size for iterative screening, and obtain, according to the screening results, a reference motion-removed static image to be restored, a reference motion-removed static image to be restored, a motion-removed static image to be restored and tested, and a motion-removed static image to be restored and tested;

[0043] a superposition module, configured to superimpose the reference motion-removed still image to be restored and the reference motion-removed still image to be restored to form a restored reference key frame, and to superimpose the motion-removed still image to be restored and ...

[0044] The verification module is used to obtain the key frame matching degree between the restored reference key frame and the corresponding restored key frame to be verified based on the static noise matching degree and the motion noise matching degree, and to perform weighted averaging of the key frame matching degrees using the structural similarity index value between the restored reference key frame and the corresponding restored key frame to be verified as a weight to obtain the similarity between the reference video and the video to be verified. When the similarity is greater than a second preset value, the reference video and the video to be verified come from the same monitoring device.

[0045] An embodiment of the present invention further provides a homologous surveillance video tracing system based on pattern noise, the system comprising the above-mentioned device.

[0046] The present invention has the following beneficial effects:

[0047] First, keyframes are extracted from the reference video and the video to be tested, respectively, to obtain reference keyframes and keyframes to be tested. Reference motion-free still images and reference motion-free still images, as well as motion-free still images and motion-free still images to be tested, are then obtained based on the reference keyframes and keyframes to be tested. PRNU noise degradation or distortion caused by aging monitoring equipment has a smaller impact on static areas within the image and a greater impact on motion areas. Therefore, static and motion areas are extracted from both the reference keyframes and the keyframes to be tested.

[0048] Next, the stationary noise matching degree of corresponding sub-blocks between the reference motion-free still image and the to-be-tested motion-free still image, as well as the motion noise matching degree of corresponding sub-blocks between the reference motion-free still image and the to-be-tested motion-free still image, are calculated. The stationary noise matching degree is the degree of noise matching between the reference keyframe and the to-be-tested keyframe in the stationary region, while the motion noise matching degree is the degree of noise matching between the reference keyframe and the to-be-tested keyframe in the moving region.

[0049] Then, an objective function is constructed based on the static noise matching degree, the motion noise matching degree, a first preset value, and a preset iteration step size for iterative screening. A reference de-motioned static image to be restored, a reference de-motioned static image to be restored, a de-motioned static image to be restored and tested, and a de-motioned static image to be restored and tested are obtained according to the screening results. The screening is performed to obtain the de-motioned static image to be restored and tested and the reference de-motioned static image to be restored, which have been free of interference noise. For ease of distinction, the reference de-motioned static image and the de-motioned static image to be tested, which correspond to the de-motioned static image to be restored and tested and the reference de-motioned static image to be restored, are referred to as the reference de-motioned static image to be restored and the de-motioned static image to be tested.

[0050] Furthermore, the reference de-motioned still image to be restored and the reference de-motioned still image to be restored are superimposed to form a restored reference key frame, and the de-motioned still image to be restored and tested are superimposed to form a restored key frame to be tested. The superposition is to restore the original reference key frame and the key frame to be tested.

[0051] Finally, based on the static noise matching degree and the motion noise matching degree, the key frame matching degree between the restored reference key frame and the corresponding restored key frame to be verified is calculated. The structural similarity index value between the restored reference key frame and the corresponding restored key frame to be verified is used as a weight, and the key frame matching degrees are weighted averaged to obtain the similarity between the reference video and the video to be verified. When the similarity is greater than a second preset value, the reference video and the video to be verified are from the same monitoring device. The higher the key frame matching degree, the higher the degree of pattern noise overlap between the reference key frame and the key frame to be verified, and the greater the possibility that the reference key frame and the key frame to be verified are of the same origin. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] In order to more clearly illustrate the technical solutions and advantages of the embodiments of the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the prior art descriptions. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0053] Figure 1 This is a flow chart of a method for tracing homologous surveillance videos based on pattern noise provided by the first embodiment of the present invention;

[0054] Figure 2 Flowchart of the process of obtaining the reference motion-free still image and the reference motion-free still image provided by the second embodiment of the present invention;

[0055] Figure 3 A flowchart of a process for obtaining the stationary noise matching degree provided in the third embodiment of the present invention;

[0056] Figure 4 A flowchart of a process for obtaining the reference stability factor provided in the fourth embodiment of the present invention;

[0057] Figure 5 A flowchart of a process for acquiring noise information provided in a fifth embodiment of the present invention;

[0058] Figure 6A flowchart of a process for obtaining pattern noise of a still motion image to be inspected provided by a sixth embodiment of the present invention;

[0059] Figure 7 A flowchart of the process of obtaining the key frame matching degree provided by the seventh embodiment of the present invention;

[0060] Figure 8 This is a schematic diagram of a homologous surveillance video tracing device based on pattern noise provided by the eighth embodiment of the present invention. DETAILED DESCRIPTION

[0061] To further illustrate the technical means and effects adopted by the present invention to achieve the predetermined purpose of the invention, the following, in conjunction with the accompanying drawings and preferred embodiments, describes in detail the specific implementation, structure, features, and effects of a method, system, and device for tracing homologous surveillance videos based on pattern noise proposed by the present invention. In the following description, different "one embodiment" or "another embodiment" do not necessarily refer to the same embodiment. In addition, specific features, structures, or characteristics of one or more embodiments may be combined in any suitable form.

[0062] Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs.

[0063] The specific scheme of the method for tracing homologous surveillance videos based on pattern noise provided by the present invention is described in detail below with reference to the accompanying drawings.

[0064] See also Figure 1 , which shows a flowchart of a method for tracing homologous surveillance videos based on pattern noise provided by an embodiment of the present invention, the method comprising:

[0065] S101. Extract key frames from the reference video and the video to be tested respectively to obtain reference key frames and key frames to be tested, and obtain reference de-motioned still images and reference de-motioned still images, de-motioned still images to be tested and de-motioned still images to be tested respectively based on the reference key frames and the key frames to be tested.

[0066] A video shot by a known monitoring device is obtained as a reference video, and a video that needs to be tested for homology is collected as a video to be tested. Key frames are extracted from the reference video and the video to be tested to obtain the reference key frame sequence and the key frame sequence to be tested corresponding to the video.

[0067] The method for extracting key frames is as follows: I frames are extracted from the video using the Ffmpeg program, and the Structural Similarity Index (SSIM) is used to calculate the inter-frame performance difference between two I frames. When the SSIM value is less than 0.7, it is considered that the frame difference is large, and the two I frames are recorded as key frames.

[0068] Video files contain a large amount of image information. During the video compression and decoding process, key frames are selected from I frames based on the performance differences between frames. The pattern noise differences between key frames are used to perform video homology verification, which greatly reduces the computational complexity of frame-by-frame analysis during the verification process.

[0069] The aging of surveillance equipment components significantly affects PRNU noise in video, and this effect varies significantly across different regions. Specifically, static areas—i.e., portions of the video where objects remain stationary or lack dynamic objects—typically exhibit relatively constant PRNU noise and are less affected by equipment aging. In contrast, dynamic areas, due to factors such as moving objects and fluctuating lighting, are prone to introducing additional interference noise, which can interfere with stable PRNU noise detection.

[0070] Therefore, when performing video homology verification, to ensure accuracy and reliability, the PRNU noise characteristics of the static area can be used to correct or compensate for the moving area. This aims to reduce noise interference caused by non-PRNU factors in the moving area, thereby improving the accuracy and consistency of the overall noise analysis and providing a more reliable basis for video homology judgment. Therefore, it is necessary to first divide the static area and the moving area in the key frame, divide the reference de-motioned static image and the reference de-motioned moving image from the reference key frame, and divide the de-motioned static image to be tested and the de-motioned moving image to be tested from the key frame to be tested.

[0071] The acquisition process of the reference motion-free still image and the reference motion-free still image will be described in detail in the second embodiment and will not be repeated here.

[0072] It should be noted that the acquisition process of the to-be-tested motion-removed still image and the to-be-tested motion-removed still image is similar to the acquisition process of the reference motion-removed still image and the reference motion-removed still image, and will not be repeated here.

[0073] S102. Calculate the still noise matching degree of corresponding sub-blocks between the reference motion-removed still image and the motion-removed still image to be tested, and the motion noise matching degree of corresponding sub-blocks between the reference motion-removed still image and the motion-removed still image to be tested.

[0074] The process of obtaining the stationary noise matching degree will be described in detail in the third embodiment and will not be repeated here.

[0075] It should be noted that the process for acquiring the motion noise matching degree is similar to the process for acquiring the static noise matching degree, both requiring the acquisition of pattern noise. The only difference is that the process for acquiring the pattern noise of the to-be-tested de-static motion image and the reference de-static motion image requires modification. The process for acquiring the pattern noise of the to-be-tested de-static motion image will be described in detail in the sixth embodiment and will not be repeated here. The process for acquiring the pattern noise of the to-be-tested de-static motion image is similar to the process for acquiring the pattern noise of the reference de-static motion image.

[0076] It should be noted that pattern noise components mainly include fixed pattern noise (FPN) and photosensitive non-uniform noise. Among them, FPN is an additive noise that can be eliminated by subtracting a dark frame and cannot be used as a device fingerprint for homology judgment. PRNU is mainly caused by non-uniform pixels in the sensor. This is because the manufacturing process leads to inconsistencies in the sensor's photosensitive elements. Different video frames shot by the same device contain the same PRNU pattern noise, and the noise is mainly concentrated in the high-frequency band.

[0077] S103. Based on the static noise matching degree, the motion noise matching degree, the first preset value and the preset iteration step size, an objective function is constructed for iterative screening, and according to the screening results, a reference de-motioned static image to be restored, a reference de-motioned static image to be restored, a de-motioned static image to be restored and tested, and a de-motioned static image to be restored and tested are obtained.

[0078] Specifically, the objective function includes:

[0079] The value of the preset iteration step minus the first preset value is used as the input value of an exponential function with base e, and the output value of the exponential function is multiplied by the absolute value of the noise matching degree to obtain the objective function, where the absolute value of the noise matching degree is the absolute value of the motion noise matching degree minus the absolute value of the stationary noise matching degree.

[0080] The objective function can be expressed as follows:

[0081] ;

[0082] Among them, the represents the motion noise matching degree of the i-th sub-block corresponding to the reference de-stilled motion image and the de-stilled motion image to be tested, represents the static noise matching degree of the i-th sub-block corresponding to the reference de-motion still image and the de-motion still image to be tested, wherein the value of i ranges from 1 to the number of the sub-blocks. represents the first preset value, the represents the preset iteration step, preferably 0.05, Denotes the objective function.

[0083] When the screening result is stable, that is, the value remains unchanged, stop the iteration.

[0084] What is obtained through screening is the reference de-stilled motion image and the de-stilled motion image to be tested, which have corrected the noise deviation caused by changes, movements and equipment degradation. In other words, the de-stilled motion image to be tested and the reference de-stilled motion image to be restored, which have removed the interference noise.

[0085] To facilitate corresponding understanding, the reference de-motioned still image corresponding to the de-motioned still motion image to be restored and verified and the reference de-motioned still motion image to be restored and the de-motioned still image to be verified are referred to as the reference de-motioned still image to be restored and the de-motioned still image to be restored and verified.

[0086] S104. Superimposing the reference de-motioned still image to be restored and the reference de-motioned still image to be restored to form a restored reference key frame, and superimposing the de-motioned still image to be restored and tested and the de-motioned still image to be restored and tested to form a restored key frame to be tested.

[0087] The superposition process is a prior art and will not be described in detail herein. The superposition is to restore the original reference key frame and the key frame to be checked.

[0088] S105. Based on the static noise matching degree and the motion noise matching degree, the key frame matching degree between the restored reference key frame and the corresponding restored key frame to be tested is obtained, and the structural similarity index value between the restored reference key frame and the corresponding restored key frame to be tested is used as a weight to perform weighted averaging on the key frame matching degrees to obtain the similarity between the reference video and the video to be tested. When the similarity is greater than a second preset value, the reference video and the video to be tested come from the same monitoring device.

[0089] The process of obtaining the key frame matching degree will be described in detail in the seventh embodiment and will not be repeated here.

[0090] Each pair of key frames, ie, the restored reference key frame and the corresponding restored key frame to be checked, has a corresponding key frame matching degree.

[0091] The second preset value can be set independently, and is preferably 0.85.

[0092] Figure 2 The flowchart of the process of obtaining the reference motion-free still image and the reference motion-free still image provided in the second embodiment of the present invention includes:

[0093] S201. Separate the static area and the moving area in the reference key frame.

[0094] Separating the static area and the motion area from the reference key frame is an existing technology, such as using a Gaussian mixture model for separation.

[0095] S202. Denoise the reference key frame to obtain a denoised reference key frame, and subtract pixel values at corresponding positions between the reference key frame and the denoised reference key frame to obtain a reference noise image.

[0096] Here, a Wiener filter based on wavelet transform can be used for denoising. Wavelet transform is a signal processing technique that decomposes an image into low-frequency and high-frequency components at multiple scales. Since the pattern noise of an image is mainly concentrated in the high-frequency segment, a Wiener filter is used to filter it to remove the high-frequency noise portion, retaining the effective information of the image and obtaining a denoised reference keyframe. The pixel values at the corresponding positions of the reference keyframe and the denoised reference keyframe are subtracted to obtain a reference noise image. The reference noise image represents the difference between the reference keyframe and the denoised reference keyframe.

[0097] S203. Match the still area and the motion area with the reference noise image respectively to obtain the reference motion-free still image and the reference motion-free still image.

[0098] Matching can use key point matching algorithm to perform image matching, which is a well-known algorithm.

[0099] The reference motion-free still image is a noise image containing only the still area, and the reference motion-free still image is a noise image containing only the motion area.

[0100] Figure 3 This is a flowchart of a process for obtaining the static noise matching degree provided in the third embodiment of the present invention. The process for obtaining the static noise matching degree includes:

[0101] S301. Divide the reference de-motioned still image and the to-be-verified de-motioned still image into at least two sub-blocks respectively.

[0102] The sizes of the sub-blocks may be set to be uniform.

[0103] S302. Obtain pixel values of each pattern noise in each sub-block in the reference motion-removed still image and the motion-removed still image to be tested respectively.

[0104] S303. Obtain a reference stability factor of each sub-block in the reference motion-free still image and a stability factor to be checked of each sub-block in the motion-free still image to be checked.

[0105] The process of obtaining the reference stability factor will be described in detail in the fourth embodiment and will not be repeated here.

[0106] The process of obtaining the stability factor to be tested is the same as the process of obtaining the reference stability factor.

[0107] S304. Calculate the square of the difference between the pixel values corresponding to the pattern noise in the i-th sub-block of the reference de-motioned still image and the i-th sub-block of the de-motioned still image to be tested, and calculate the sum of the squares of the differences between the pixel values, where the value of i ranges from 1 to the number of sub-blocks.

[0108] The sum of the squares can be expressed as follows:

[0109] ;

[0110] Among them, the represents the pixel value of the vth pattern noise in the i-th sub-block of the reference de-motioned still image, represents the pixel value of the vth pattern noise in the i-th sub-block of the motion-removed still image to be inspected, and nv represents the number of pattern noises in the i-th sub-block.

[0111] It is used to measure the difference in pixel values between the noise pattern in the same position in the i-th sub-block of the reference de-motioned still image and the de-motioned still image to be tested. The more obvious the difference, the lower the still noise matching degree.

[0112] S305. Subtract the reference stability factor corresponding to the i-th sub-block from the stability factor to be checked, and then divide the result by the stability factor to be checked to obtain a first ratio, and calculate the absolute value of the first ratio.

[0113] The absolute value of the first ratio can be expressed as follows:

[0114] ;

[0115] Among them, the represents the reference stability factor corresponding to the i-th sub-block, represents the stability factor to be tested corresponding to the i-th sub-block, represents the absolute value function.

[0116] It indicates the influence of the stability of the sub-block on the matching degree of the stationary noise. The closer the values of the stability factors of the sub-blocks are, the smaller the variation of the pattern noise in the same sub-block in the time series is, and the better the matching effect of the sub-blocks is.

[0117] S306. Multiply the sum of the squares by the absolute value and take the opposite number as the input value of an exponential function with base e, and the output value of the exponential function is used as the static noise matching degree of the corresponding i-th sub-block between the reference de-motioned still image and the de-motioned still image to be tested.

[0118] The stationary noise matching degree can be expressed as:

[0119] ;

[0120] Among them, the represents the exponential function, the represents the static noise matching degree of the corresponding i-th sub-block between the reference de-motioned still image and the de-motioned still image to be checked.

[0121] S307 . Repeat the static noise matching degree of the i-th sub-block to obtain the static noise matching degree of each sub-block corresponding to the reference de-motioned still image and the de-motioned still image to be checked.

[0122] Figure 4 This is a flowchart of a process for obtaining the reference stability factor provided in a fourth embodiment of the present invention. The process for obtaining the reference stability factor includes:

[0123] S401. Calculate noise information of noise pixels at each position in each sub-block of the reference motion-free still image, and filter out the noise information greater than the first preset value as pattern noise of the reference motion-free still image.

[0124] The process of obtaining the noise information will be described in detail in the fifth embodiment and will not be repeated here.

[0125] The first preset value can be set independently, and is preferably 0.75.

[0126] Pattern noise is caused by inconsistencies in the sensor's photosensitive elements due to the manufacturing process. The same area in multiple images at different times shows consistency. The pattern noise of the monitoring equipment cannot be obtained from only a single image. Therefore, it is necessary to use the noise stability of the same corresponding area in the time series to filter out the pattern noise.

[0127] S402. Obtain the noise information of the x-th position noise pixel in the i-th sub-block in the reference de-motioned still image, where the i-th sub-block contains the minimum and maximum values of the pattern noise in the reference key frame, wherein the value of x ranges from 1 to the number of noise pixels at the position in the i-th sub-block.

[0128] S403. After subtracting the minimum value from the noise information, divide the result by the difference between the maximum value and the minimum value to obtain a reference stability factor component. The sum of the reference stability factor components is the reference stability factor of the i-th sub-block.

[0129] The reference stability factor of the i-th sub-block can be expressed as follows:

[0130] ;

[0131] Among them, the represents the reference stability factor component, the represents the noise information of the x-th noise pixel in the ith sub-block of the reference motion-free still image, Indicates that the i-th sub-block contains the minimum value of the pattern noise in the reference key frame, Indicates that the i-th sub-block contains the maximum value of the pattern noise in the reference key frame, represents the reference stability factor of the i-th sub-block, and nx represents the number of position noise pixels in the i-th sub-block.

[0132] S404. Repeat the reference stability factor of the i-th sub-block to obtain the reference stability factor of each sub-block in the reference motion-free still image.

[0133] Figure 5 This is a flowchart of the noise information acquisition process provided in the fifth embodiment of the present invention. The noise information acquisition process includes:

[0134] S501. Obtain the pixel value of the noise pixel at the xth position in the ith sub-block in each reference motion-removed still image and the pixel mean value of the noise pixel at the xth position in the ith sub-block.

[0135] The pixel value of the noise pixel at the xth position in the kth reference motion-free still image of the i-th sub-block can be expressed as The pixel mean of the x-th position noise pixel in the i-th sub-block can be expressed as To express.

[0136] S502. The average value of the absolute value of each pixel value minus the pixel mean is taken as the input value of an exponential function with base e, and the output value of the exponential function is used as the noise information of the noise pixel point at the xth position in the i-th sub-block in the reference de-motioned still image.

[0137] The noise information can be expressed as follows:

[0138] ;

[0139] Among them, the represents the exponential function, the represents the number of the reference de-motion still images, Represents the noise information of the x-th noise pixel in the ith sub-block of the reference de-motioned still image.

[0140] Furthermore, the method further includes obtaining pattern noise from the motion image to be tested for de-staticing before calculating the motion noise matching degree. Pattern noise is randomly expressed in static regions of an image, while in moving regions, due to influences such as the direction, speed, and illumination of the moving object, the noise distribution exhibits different patterns. By utilizing the randomness of the pattern noise in static regions, the noise in moving regions is filtered and denoised, effectively distinguishing between pattern noise and noise caused by actual motion information, and filtering out interference noise caused by factors such as motion.

[0141] Figure 6 This is a flowchart of a process for obtaining the pattern noise of the image to be tested for removing still motion provided by the sixth embodiment of the present invention. The process for obtaining the pattern noise of the image to be tested for removing still motion includes:

[0142] S601. Calculate the minimum value of the first standard deviation of the noise change degree of the i-th sub-block in the de-motioned still image to be tested between two adjacent frames, and calculate the second standard deviation of the noise change degree of the i-th sub-block in the de-motioned still image to be tested between two adjacent frames, and divide the second standard deviation by the minimum value to obtain the relative noise change degree of motion, wherein the value of i ranges from 1 to the number of sub-blocks.

[0143] The degree of change of the motion relative noise can be expressed as:

[0144] ;

[0145] Among them, the represents the second standard deviation of the noise variation degree of the i-th sub-block in the two adjacent frames of the to-be-tested still motion image, represents the minimum value of the first standard deviation of the noise variation degree of the i-th sub-block in the two adjacent frames of the motion-free still image to be tested, Indicates the degree of change of the motion relative to the noise.

[0146] The noise variation degree refers to the variation degree of noise pixels.

[0147] S602. After the degree of change of the motion relative noise is divided by the first preset value, the value is multiplied by the noise information of the noise pixel at the xth position in the i-th sub-block of the motion image to be tested for de-stilling as the pattern noise of the motion image to be tested for de-stilling, wherein the value of x ranges from 1 to the number of noise pixels at the position in the i-th sub-block.

[0148] The pattern noise can be expressed as follows:

[0149] ;

[0150] Among them, the represents the first preset value, the represents the noise information of the x-th noise pixel in the ith sub-block of the still motion image to be tested, It represents the pattern noise of the image to be tested for still motion.

[0151] The first preset value can be set independently, and is preferably 0.75.

[0152] Figure 7 This is a flowchart of the process of obtaining the key frame matching degree provided by the seventh embodiment of the present invention. The process of obtaining the key frame matching degree includes:

[0153] S701. Calculate the sum of the stationary noise matching degree and the motion noise matching degree corresponding to each sub-block.

[0154] The added value can be expressed as follows:

[0155] ;

[0156] Among them, the represents the motion noise matching degree of the i-th sub-block corresponding to the reference de-stilled motion image and the de-stilled motion image to be tested, represents the static noise matching degree of the corresponding i-th sub-block between the reference de-motioned still image and the de-motioned still image to be checked.

[0157] S702. Calculate the average value of the added values as the key frame matching degree.

[0158] The key frame matching degree can be expressed as:

[0159] );

[0160] Among them, the represents the number of sub-blocks, Indicates the key frame matching degree.

[0161] When the The higher it is, the higher the degree of pattern noise overlap between the restored reference key frame and the corresponding restored key frame to be verified, and the greater the possibility that the restored reference key frame and the corresponding restored key frame to be verified are homologous.

[0162] Figure 8 This is a schematic diagram of a device for tracing homologous surveillance videos based on pattern noise provided by an eighth embodiment of the present invention, the device comprising:

[0163] An acquisition module 801 is configured to extract key frames from a reference video and a video to be tested to obtain reference key frames and key frames to be tested, and to obtain reference motion-removed still images and reference motion-removed still images, and motion-removed still images to be tested and motion-removed still images to be tested, based on the reference key frames and key frames to be tested.

[0164] A calculation module 802 is configured to calculate a still noise matching degree of corresponding sub-blocks between the reference motion-removed still image and the motion-removed still image to be tested, and a motion noise matching degree of corresponding sub-blocks between the reference motion-removed still image and the motion-removed still image to be tested;

[0165] A screening module 803 is configured to construct an objective function based on the static noise matching degree, the motion noise matching degree, a first preset value, and a preset iteration step size for iterative screening, and obtain, according to the screening results, a reference motion-removed static image to be restored, a reference motion-removed static image to be restored, a motion-removed static image to be restored and tested, and a motion-removed static image to be restored and tested;

[0166] The superposition module 804 is configured to superimpose the reference motion-removed still image to be restored and the reference motion-removed still image to be restored to form a restored reference key frame, and to superimpose the motion-removed still image to be restored and ...

[0167] The inspection module 805 is used to obtain the key frame matching degree between the restored reference key frame and the corresponding restored key frame to be inspected based on the static noise matching degree and the motion noise matching degree, and to perform weighted averaging on the key frame matching degrees using the structural similarity index value between the restored reference key frame and the corresponding restored key frame to be inspected as a weight to obtain the similarity between the reference video and the video to be inspected. When the similarity is greater than a second preset value, the reference video and the video to be inspected come from the same monitoring device.

[0168] The technical features and technical effects of the homologous surveillance video tracing device based on pattern noise proposed in the embodiment of the present invention are the same as those of the method proposed in the embodiment of the present invention, and will not be repeated here.

[0169] An embodiment of the present invention further provides a homologous surveillance video tracing system based on pattern noise, the system comprising the above-mentioned device.

[0170] The present invention has the following beneficial effects:

[0171] First, keyframes are extracted from the reference video and the video to be tested, respectively, to obtain reference keyframes and keyframes to be tested. Reference motion-free still images and reference motion-free still images, as well as motion-free still images and motion-free still images to be tested, are then obtained based on the reference keyframes and keyframes to be tested. PRNU noise degradation or distortion caused by aging monitoring equipment has a smaller impact on static areas within the image and a greater impact on motion areas. Therefore, static and motion areas are extracted from both the reference keyframes and the keyframes to be tested.

[0172] Next, the stationary noise matching degree of corresponding sub-blocks between the reference motion-free still image and the to-be-tested motion-free still image, as well as the motion noise matching degree of corresponding sub-blocks between the reference motion-free still image and the to-be-tested motion-free still image, are calculated. The stationary noise matching degree is the degree of noise matching between the reference keyframe and the to-be-tested keyframe in the stationary region, while the motion noise matching degree is the degree of noise matching between the reference keyframe and the to-be-tested keyframe in the moving region.

[0173] Then, an objective function is constructed based on the static noise matching degree, the motion noise matching degree, a first preset value, and a preset iteration step size for iterative screening. A reference de-motioned static image to be restored, a reference de-motioned static image to be restored, a de-motioned static image to be restored and tested, and a de-motioned static image to be restored and tested are obtained according to the screening results. The screening is performed to obtain the de-motioned static image to be restored and tested and the reference de-motioned static image to be restored, which have been free of interference noise. For ease of distinction, the reference de-motioned static image and the de-motioned static image to be tested, which correspond to the de-motioned static image to be restored and tested and the reference de-motioned static image to be restored, are referred to as the reference de-motioned static image to be restored and the de-motioned static image to be tested.

[0174] Furthermore, the reference de-motioned still image to be restored and the reference de-motioned still image to be restored are superimposed to form a restored reference key frame, and the de-motioned still image to be restored and tested are superimposed to form a restored key frame to be tested. The superposition is to restore the original reference key frame and the key frame to be tested.

[0175] Finally, based on the static noise matching degree and the motion noise matching degree, the key frame matching degree between the restored reference key frame and the corresponding restored key frame to be verified is calculated. The structural similarity index value between the restored reference key frame and the corresponding restored key frame to be verified is used as a weight, and the key frame matching degrees are weighted averaged to obtain the similarity between the reference video and the video to be verified. When the similarity is greater than a second preset value, the reference video and the video to be verified are from the same monitoring device. The higher the key frame matching degree, the higher the degree of pattern noise overlap between the reference key frame and the key frame to be verified, and the greater the possibility that the reference key frame and the key frame to be verified are of the same origin.

[0176] It should be noted that the order in which the embodiments of the present invention are described above is for illustrative purposes only and does not necessarily represent the superiority or inferiority of the embodiments. The processes depicted in the accompanying drawings do not necessarily require the specific order or sequential order shown to achieve the desired results. In certain embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0177] The various embodiments in this specification are described in a progressive manner, and the same or similar parts between the various embodiments can be referred to each other. Each embodiment focuses on the differences from other embodiments.

Claims

1. A homologous surveillance video tracing method based on pattern noise, characterized in that: The method comprises: Extracting key frames from the reference video and the video to be tested respectively to obtain reference key frames and key frames to be tested, and obtaining reference de-motioned still images and reference de-motioned still images, the de-motioned still images to be tested, and the de-motioned still images to be tested respectively based on the reference key frames and the key frames to be tested; Calculating the still noise matching degree of corresponding sub-blocks between the reference motion-free still image and the motion-free still image to be tested, and the motion noise matching degree of corresponding sub-blocks between the reference motion-free still image and the motion-free still image to be tested; constructing an objective function based on the static noise matching degree, the motion noise matching degree, a first preset value, and a preset iteration step size for iterative screening, and obtaining a reference motion-removed static image to be restored, a reference motion-removed static image to be restored, a motion-removed static image to be restored and tested, and a motion-removed static image to be restored and tested according to the screening result; Superimposing the reference motion-free still image to be restored and the reference motion-free still image to be restored to form a restored reference key frame, and superimposing the motion-free still image to be restored and tested and the motion-free still image to be restored and tested to form a restored key frame to be tested; Based on the static noise matching degree and the motion noise matching degree, the key frame matching degree between the restored reference key frame and the corresponding restored key frame to be tested is obtained. The structural similarity index value between the restored reference key frame and the corresponding restored key frame to be tested is used as a weight, and the key frame matching degree is weighted averaged to obtain the similarity between the reference video and the video to be tested. When the similarity is greater than a second preset value, the reference video and the video to be tested come from the same monitoring device.

2. The method for tracing homologous surveillance videos based on pattern noise according to claim 1, characterized in that: The process of acquiring the reference motion-free still image and the reference motion-free still image includes: Separating the static area and the moving area in the reference key frame; Denoising the reference key frame to obtain a denoised reference key frame, and subtracting pixel values at corresponding positions of the reference key frame and the denoised reference key frame to obtain a reference noise image; The static area and the motion area are matched with the reference noise image respectively to obtain the reference motion-free static image and the reference motion-free static image.

3. The method for tracing homologous surveillance videos based on pattern noise according to claim 1, characterized in that: The process of obtaining the stationary noise matching degree includes: Dividing the reference motion-free still image and the to-be-tested motion-free still image into at least two sub-blocks respectively; respectively obtaining pixel values of each pattern noise in each sub-block in the reference motion-free still image and the motion-free still image to be tested; Acquire a reference stability factor of each sub-block in the reference motion-free still image and a stability factor to be checked of each sub-block in the motion-free still image to be checked; Calculating the squares of the differences between the pixel values corresponding to the pattern noise in the i-th sub-block of the reference de-motioned still image and the i-th sub-block of the to-be-tested de-motioned still image, and calculating the sum of the squares of the differences between the pixel values, where the value of i ranges from 1 to the number of the sub-blocks; subtracting the reference stability factor corresponding to the i-th sub-block from the stability factor to be checked, and then dividing the result by the stability factor to be checked to obtain a first ratio, and calculating an absolute value of the first ratio; The sum of the squares is multiplied by the absolute value and the negation thereof is taken as an input value of an exponential function with base e, and an output value of the exponential function is used as the static noise matching degree of the i-th sub-block corresponding to the reference de-motioned still image and the de-motioned still image to be tested; Repeat the static noise matching degree of the i-th sub-block to obtain the static noise matching degree of each sub-block corresponding to the reference de-motioned still image and the de-motioned still image to be tested.

4. The method for tracing homologous surveillance videos based on pattern noise according to claim 3, characterized in that: The process of obtaining the reference stability factor includes: Calculating noise information of noise pixels at each position in each sub-block of the reference motion-free still image, and screening out the noise information greater than the first preset value as pattern noise of the reference motion-free still image; Obtaining noise information of a noise pixel at an x-th position in an i-th sub-block of the reference de-motioned still image, where the i-th sub-block contains a minimum value and a maximum value of the pattern noise in the reference key frame, wherein the value of x ranges from 1 to the number of noise pixels at the x-th position in the i-th sub-block; A reference stability factor component is obtained by dividing the noise information by the difference between the maximum value and the minimum value after subtracting the minimum value, and the sum of the reference stability factor components is the reference stability factor of the i-th sub-block; Repeat the reference stability factor of the i-th sub-block to obtain the reference stability factor of each sub-block in the reference motion-removed still image.

5. The method for tracing homologous surveillance videos based on pattern noise according to claim 4, characterized in that: The process of obtaining the noise information includes: Obtaining a pixel value of a noise pixel at an x-th position in an i-th sub-block in each of the reference motion-removed still images and a pixel mean value of the noise pixel at the x-th position in the i-th sub-block; The average value of the absolute value of each pixel value minus the pixel mean is taken as the input value of the exponential function with base e, and the output value of the exponential function is used as the noise information of the noise pixel point at the xth position in the i-th sub-block in the reference de-motion still image.

6. The method for tracing homologous surveillance videos based on pattern noise according to claim 1, characterized in that: The method further includes: before calculating the motion noise matching degree, obtaining the pattern noise of the image to be tested for removing still motion, wherein the process of obtaining the pattern noise of the image to be tested for removing still motion includes: Calculating a minimum value of a first standard deviation of a noise variation degree of an i-th sub-block in two adjacent frames of the motion-free still image to be inspected, and calculating a second standard deviation of a noise variation degree of an i-th sub-block in two adjacent frames of the motion-free still image to be inspected, and dividing the second standard deviation by the minimum value to obtain a motion-relative noise variation degree, wherein the value of i ranges from 1 to the number of the sub-blocks; The degree of motion relative noise change is divided by the first preset value and then multiplied by the noise information of the noise pixel at the xth position in the i-th sub-block of the motion image to be tested for de-stilling as the pattern noise of the motion image to be tested for de-stilling, where the value of x ranges from 1 to the number of noise pixels at the position in the i-th sub-block.

7. The method for tracing homologous surveillance videos based on pattern noise according to claim 1, wherein: The objective function includes: The value of the preset iteration step minus the first preset value is used as the input value of an exponential function with base e, and the output value of the exponential function is multiplied by the absolute value of the noise matching degree to obtain the objective function, where the absolute value of the noise matching degree is the absolute value of the motion noise matching degree minus the absolute value of the stationary noise matching degree.

8. The method for tracing homologous surveillance videos based on pattern noise according to claim 1, wherein: The process of obtaining the key frame matching degree includes: respectively calculating a sum of the stationary noise matching degree and the motion noise matching degree corresponding to each of the sub-blocks; An average value of the added values is calculated as the key frame matching degree.

9. A homologous surveillance video tracing device based on pattern noise, characterized in that: The device comprises: an acquisition module, configured to extract key frames from the reference video and the video to be tested to obtain reference key frames and key frames to be tested, and to obtain reference motion-removed still images and reference motion-removed still images, the motion-removed still images to be tested, and the motion-removed still images to be tested, based on the reference key frames and the key frames to be tested; a calculation module, configured to calculate a still noise matching degree of corresponding sub-blocks between the reference motion-removed still image and the motion-removed still image to be tested, and a motion noise matching degree of corresponding sub-blocks between the reference motion-removed still image and the motion-removed still image to be tested; a screening module, configured to construct an objective function based on the static noise matching degree, the motion noise matching degree, a first preset value, and a preset iteration step size for iterative screening, and obtain, according to the screening results, a reference motion-removed static image to be restored, a reference motion-removed static image to be restored, a motion-removed static image to be restored and tested, and a motion-removed static image to be restored and tested; a superposition module, configured to superimpose the reference motion-removed still image to be restored and the reference motion-removed still image to be restored to form a restored reference key frame, and to superimpose the motion-removed still image to be restored and ... The verification module is used to obtain the key frame matching degree between the restored reference key frame and the corresponding restored key frame to be verified based on the static noise matching degree and the motion noise matching degree, and to perform weighted averaging of the key frame matching degrees using the structural similarity index value between the restored reference key frame and the corresponding restored key frame to be verified as a weight to obtain the similarity between the reference video and the video to be verified. When the similarity is greater than a second preset value, the reference video and the video to be verified come from the same monitoring device.

10. A homologous surveillance video tracing system based on pattern noise, characterized in that: The system comprises the apparatus of claim 9.

Images