Homologous monitoring video tracing method, system and device based on pattern noise

Abstract

The invention relates to the technical field of monitoring video homology inspection, in particular to a homologous monitoring video tracing method, system and device based on pattern noise. Correspondingly obtaining a to-be-restored reference de-static motion image, a to-be-restored reference de-static motion image, a to-be-restored to-be-inspected de-static motion image and a to-be-restored to-be-inspected de-static motion image according to a screening result, and then performing superposition to obtain a restored reference key frame and a restored to-be-inspected key frame; calculating a key frame matching degree based on the static noise matching degree and the motion noise matching degree, taking a structural similarity index value between the restored reference key frame and the corresponding restored to-be-detected key frame as a weight, carrying out weighted average on the key frame matching degree to obtain a similarity, and when the similarity is greater than a second preset value, carrying out weighted average on the key frame matching degree; the reference video and the video to be inspected are from the same monitoring equipment. By adopting the method and the device, the video homology test result based on the pattern noise can be more accurate.

Description

Technical Field

[0001] The present invention relates to the technical field of surveillance video homology verification, and in particular to a method, system and device for tracing homologous surveillance videos based on pattern noise. Background Art

[0002] With the rapid progress of computer technology, network technology and image transmission and processing technology, video surveillance technology has been widely used in many fields such as road traffic, shopping malls, stations and even family life, becoming an indispensable part of all walks of life. These monitoring devices can capture detailed video image data, which is crucial for analyzing business behavior and activity trajectories. Especially in front-line video investigation work, surveillance video often plays the role of key evidence.

[0003] However, the easy editing of digital images and videos has brought severe challenges to the work of collecting evidence. In order to cope with this problem, it is particularly important to conduct homology inspection of surveillance videos, which aims to confirm whether the video clips are from the same surveillance equipment, thereby providing solid technical support for the detection and trial of the case and significantly improving the reliability and authenticity of case evidence. It is worth noting that with the continuous evolution of video tampering technology, the traditional method of judging video homology by relying on background similarity has become inadequate. In this context, pattern noise, as a stable and unique image sensor noise, has become a new focus of video homology inspection. Pattern noise originates from the process limitations of the imaging sensor manufacturing process. It not only exists stably in each surveillance device and the media it shoots, but also even for surveillance devices of the same brand and model, the video shot by its imaging sensor will be unique due to the difference in pattern noise. Therefore, pattern noise can be regarded as the "fingerprint" of the device, which is used to accurately identify the shooting device and realize the effective inspection of video homology.

[0004] When conducting an in-depth study of PRNU (Photo Response Non-Uniformity) noise in pattern noise, an important factor must be considered: core components such as imaging sensors in surveillance equipment will experience physical wear due to long-term use. As time goes by, photosensitive components and circuit components will inevitably age gradually. This aging phenomenon will cause a certain degree of decay or distortion of the PRNU noise carried by the video signal during transmission. This change not only complicates the characteristics of PRNU noise, but may also weaken its recognition as a unique "fingerprint" of the device, thereby potentially affecting the authenticity of the video homology test results based on pattern noise. Summary of the invention

[0005] In order to solve the technical problem that the existing video homology test results based on pattern noise are not accurate enough, the purpose of the present invention is to provide a tracing method for homologous monitoring videos based on pattern noise, and the specific technical solutions are as follows: Extract key frames from the reference video and the video to be tested respectively to obtain the reference key frames and the key frames to be tested, and respectively obtain the reference static-noise-removed images and reference motion-noise-removed images, the to-be-tested static-noise-removed images and the to-be-tested motion-noise-removed images according to the reference key frames and the key frames to be tested; Calculate the static noise matching degree of the corresponding sub-blocks between the reference static-noise-removed image and the to-be-tested static-noise-removed image, and the motion noise matching degree of the corresponding sub-blocks between the reference motion-noise-removed image and the to-be-tested motion-noise-removed image; Construct an objective function based on the static noise matching degree, the motion noise matching degree, a first preset value and a preset iteration step size for iterative screening, and correspondingly obtain the to-be-restored reference static-noise-removed image, the to-be-restored reference motion-noise-removed image, the to-be-restored to-be-tested static-noise-removed image and the to-be-restored to-be-tested motion-noise-removed image according to the screening results; Overlay the to-be-restored reference static-noise-removed image and the to-be-restored reference motion-noise-removed image to form a restored reference key frame, and overlay the to-be-restored to-be-tested static-noise-removed image and the to-be-restored to-be-tested motion-noise-removed image to form a restored to-be-tested key frame; Based on the static noise matching degree and the motion noise matching degree, calculate the key frame matching degree between the restored reference key frame and the corresponding restored to-be-tested key frame, and use the structural similarity index value between the restored reference key frame and the corresponding restored to-be-tested key frame as a weight to perform weighted average on the key frame matching degree to obtain the similarity between the reference video and the to-be-tested video. When the similarity is greater than a second preset value, the reference video and the to-be-tested video come from the same monitoring device.

[0006] Further, the obtaining process of the reference static-noise-removed image and the reference motion-noise-removed image includes: Separate the static area and the motion area in the reference key frame; Denoise the reference key frame to obtain a denoised reference key frame, and subtract the pixel values at the corresponding positions of the reference key frame and the denoised reference key frame to obtain a reference noise image; Match the static area and the motion area with the reference noise image respectively to obtain the reference static-noise-removed image and the reference motion-noise-removed image.

[0007] Further, the obtaining process of the static noise matching degree includes: Divide the reference motionless image and the image to be tested for motionlessness into at least two sub - blocks respectively; Obtain the pixel values of each pattern noise within each sub - block in the reference motionless image and the image to be tested for motionlessness respectively; Obtain the reference stability factor of each sub - block in the reference motionless image and the stability factor to be tested of each sub - block in the image to be tested for motionlessness; Calculate the square of the difference between the pixel values of each corresponding pattern noise in the i - th sub - block of the reference motionless image and the i - th sub - block of the image to be tested for motionlessness, and obtain the sum of the squares of the pixel value differences, where the value of i ranges from 1 to the number of sub - blocks; After subtracting the reference stability factor corresponding to the i - th sub - block from the stability factor to be tested, divide by the stability factor to be tested to obtain a first ratio, and obtain the absolute value of the first ratio; Take the opposite of the product of the sum of squares and the absolute value as the input value of the exponential function with base e, and the output value of the exponential function as the static noise matching degree of the corresponding i - th sub - block between the reference motionless image and the image to be tested for motionlessness; Repeat the static noise matching degree of the i - th sub - block to obtain the static noise matching degrees of each sub - block corresponding between the reference motionless image and the image to be tested for motionlessness.

[0008] Further, the process of obtaining the reference stability factor includes: Calculate the noise information of each noise pixel point at each position in each sub - block of the reference motionless image, and screen out the noise information greater than the first preset value as the pattern noise of the reference motionless image; Obtain the noise information of the noise pixel point at the x - th position in the i - th sub - block of the reference motionless image, where the i - th sub - block contains the minimum value and the maximum value of the pattern noise in the reference key frame, and the value of x ranges from 1 to the number of noise pixel points at the position in the i - th sub - block; After subtracting the minimum value from the noise information, divide by the difference between the maximum value and the minimum value to obtain a reference stability factor component, and the sum of each reference stability factor component is the reference stability factor of the i - th sub - block; Repeat the reference stability factor of the i - th sub - block to obtain the reference stability factors of each sub - block in the reference motionless image.

[0009] Further, the process of obtaining the noise information includes: Obtain the pixel value of the noise pixel at the x-th position in the i-th sub-block of each of the reference motion-removed still images and the pixel mean value of the noise pixel at the x-th position in the i-th sub-block; The average value after taking the absolute value of the difference between each pixel value and the pixel mean value is used as the input value of the exponential function with base e, and the output value of the exponential function is used as the noise information of the noise pixel at the x-th position in the i-th sub-block of the reference motion-removed still image.

[0010] Further, the method further includes: before calculating the motion noise matching degree, obtaining the pattern noise of the to-be-tested motion-removed still image, wherein the process of obtaining the pattern noise of the to-be-tested motion-removed still image includes: Calculate the minimum value of the first standard deviation of the noise change degree of the i-th sub-block in two adjacent frames of the to-be-tested motion-removed still image, and calculate the second standard deviation of the noise change degree of the i-th sub-block in two adjacent frames of the to-be-tested motion-removed still image. The second standard deviation is divided by the minimum value to obtain the relative motion noise change degree, where the value of i ranges from 1 to the number of sub-blocks; After dividing the relative motion noise change degree by the first preset value, multiply it by the noise information of the noise pixel at the x-th position in the i-th sub-block of the to-be-tested motion-removed still image as the pattern noise of the to-be-tested motion-removed still image, where the value of x ranges from 1 to the number of noise pixels at the position in the i-th sub-block.

[0011] Further, the objective function includes: The value obtained by subtracting the first preset value from the preset iteration step is used as the input value of the exponential function with base e, and the output value of the exponential function is multiplied by the absolute value of the noise matching degree to obtain the objective function, where the absolute value of the noise matching degree is the absolute value of the difference between the motion noise matching degree and the still noise matching degree.

[0012] Further, the process of obtaining the key frame matching degree includes: Calculate the sum values of the still noise matching degree and the motion noise matching degree corresponding to each sub-block respectively; Obtain the average value of the sum values as the key frame matching degree.

[0013] The embodiment of the present invention further provides a homologous monitoring video tracing device based on pattern noise, and the device includes: An acquisition module, configured to extract key frames from a reference video and a video to be tested respectively to obtain a reference key frame and a key frame to be tested, and obtain a reference motion-removed still image, a reference still-removed motion image, a to-be-tested motion-removed still image, and a to-be-tested still-removed motion image according to the reference key frame and the key frame to be tested respectively; A calculation module, configured to calculate the still noise matching degree of corresponding sub-blocks between the reference motion-removed still image and the to-be-tested motion-removed still image, and the motion noise matching degree of corresponding sub-blocks between the reference still-removed motion image and the to-be-tested still-removed motion image; A screening module, configured to construct an objective function based on the still noise matching degree, the motion noise matching degree, a first preset value, and a preset iteration step size for iterative screening, and correspondingly obtain a to-be-restored reference motion-removed still image, a to-be-restored reference still-removed motion image, a to-be-restored to-be-tested motion-removed still image, and a to-be-restored to-be-tested still-removed motion image according to the screening result; An overlay module, configured to overlay the to-be-restored reference motion-removed still image and the to-be-restored reference still-removed motion image into a restored reference key frame, and overlay the to-be-restored to-be-tested motion-removed still image and the to-be-restored to-be-tested still-removed motion image into a restored to-be-tested key frame; A verification module, configured to obtain a key frame matching degree between the restored reference key frame and the corresponding restored to-be-tested key frame based on the still noise matching degree and the motion noise matching degree, and perform weighted averaging on the key frame matching degree with the structural similarity index value between the restored reference key frame and the corresponding restored to-be-tested key frame as a weight to obtain the similarity between the reference video and the to-be-tested video. When the similarity is greater than a second preset value, the reference video and the to-be-tested video are from the same monitoring device.

[0014] An embodiment of the present invention further provides a homologous monitoring video tracing system based on pattern noise, and the system includes the above-mentioned device.

[0015] The present invention has the following beneficial effects: First, key frames are extracted from the reference video and the video to be tested respectively to obtain a reference key frame and a key frame to be tested, and a reference motion-removed still image, a reference still-removed motion image, a to-be-tested motion-removed still image, and a to-be-tested still-removed motion image are obtained according to the reference key frame and the key frame to be tested respectively. The influence of the decline or distortion of the PRNU noise caused by the aging of the monitoring device on the still area in the image is small, and the influence on the moving area is large. Therefore, both the reference key frame and the key frame to be tested are extracted for the still area and the moving area.

[0016] Secondly, calculate the still noise matching degree of corresponding sub-blocks between the reference motionless image and the to-be-verified motionless image, and the moving noise matching degree of corresponding sub-blocks between the reference motionless image and the to-be-verified motionless image. The still noise matching degree is the matching degree of the noise between the reference key frame and the to-be-verified key frame in the still region, and the moving noise matching degree is the matching degree of the noise between the reference key frame and the to-be-verified key frame in the moving region.

[0017] Then, construct an objective function based on the still noise matching degree, the moving noise matching degree, a first preset value, and a preset iteration step size for iterative screening. According to the screening results, obtain the to-be-restored reference motionless image, the to-be-restored reference motionless image, the to-be-restored to-be-verified motionless image, and the to-be-restored to-be-verified motionless image. The screening is to obtain the to-be-restored to-be-verified motionless image and the to-be-restored reference motionless image that remove the interference noise. For the sake of easy distinction, the reference motionless image and the to-be-verified motionless image corresponding to the to-be-restored to-be-verified motionless image and the to-be-restored reference motionless image are called the to-be-restored reference motionless image and the to-be-restored to-be-verified motionless image.

[0018] Furthermore, superimpose the to-be-restored reference motionless image and the to-be-restored reference motionless image to form the restored reference key frame, and superimpose the to-be-restored to-be-verified motionless image and the to-be-restored to-be-verified motionless image to form the restored to-be-verified key frame. The superimposition is to restore the original reference key frame and the to-be-verified key frame.

[0019] Finally, based on the still noise matching degree and the moving noise matching degree, calculate the key frame matching degree between the restored reference key frame and the corresponding restored to-be-verified key frame. Use the structural similarity index value between the restored reference key frame and the corresponding restored to-be-verified key frame as the weight, and perform weighted averaging on the key frame matching degree to obtain the similarity between the reference video and the to-be-verified video. When the similarity is greater than a second preset value, the reference video and the to-be-verified video come from the same monitoring device. The higher the key frame matching degree, the higher the degree of coincidence of the pattern noise between the reference key frame and the to-be-verified key frame, and the greater the possibility of homology between the reference key frame and the to-be-verified key frame. Description of the Drawings

[0020] To more clearly illustrate the technical solutions and advantages in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for the description of the embodiments or the prior art. Obviously, the drawings in the following description are only some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can be obtained based on these drawings.

[0021] Figure 1 Flowchart of the method for tracing homologous surveillance videos based on pattern noise provided by the first embodiment of the present invention; Figure 2 Flowchart of the process for obtaining the reference motionless still image and the reference still motion image provided by the second embodiment of the present invention; Figure 3 Flowchart of the process for obtaining the still noise matching degree provided by the third embodiment of the present invention; Figure 4 Flowchart of the process for obtaining the reference stability factor provided by the fourth embodiment of the present invention; Figure 5 Flowchart of the process for obtaining the noise information provided by the fifth embodiment of the present invention; Figure 6 Flowchart of the process for obtaining the pattern noise of the to-be-tested still motion image provided by the sixth embodiment of the present invention; Figure 7 Flowchart of the process for obtaining the key frame matching degree provided by the seventh embodiment of the present invention; Figure 8 Schematic diagram of the device for tracing homologous surveillance videos based on pattern noise provided by the eighth embodiment of the present invention. Detailed implementation manners

[0022] To further elaborate on the technical means and effects adopted by the present invention to achieve the intended invention purpose, the following will, in conjunction with the accompanying drawings and preferred embodiments, detail the specific implementation manners, structures, features, and effects of a method, system, and device for tracing homologous surveillance videos based on pattern noise proposed according to the present invention. In the following description, different "one embodiment" or "another embodiment" do not necessarily refer to the same embodiment. In addition, the specific features, structures, or characteristics in one or more embodiments can be combined in any suitable form.

[0023] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the technical field to which the present invention belongs.

[0024] The following will specifically describe the specific solution of the method for tracing homologous surveillance videos based on pattern noise provided by the present invention in conjunction with the accompanying drawings.

[0025] Please refer to Figure 1 , which shows a flowchart of a method for tracing homologous surveillance videos based on pattern noise provided by an embodiment of the present invention. The method includes: S101. Extract key frames from the reference video and the video to be inspected respectively to obtain the reference key frames and the key frames to be inspected, and obtain the reference motionless still image and the reference still motion image, the key frame to be inspected motionless still image and the key frame to be inspected still motion image according to the reference key frames and the key frames to be inspected respectively.

[0026] Obtain a video captured by a known surveillance device as the reference video, and collect the video to be subjected to homology test as the video to be inspected. Extract key frames from the reference video and the video to be inspected to obtain the reference key frame sequence and the key frame sequence corresponding to the video.

[0027] The method for extracting key frames is as follows: Use the Ffmpeg program to extract I frames from the video, and use SSIM (Structural Similarity Index) to calculate the inter-frame performance difference between two I frames. When the SSIM value is less than 0.7, it is considered that the frame difference is large, and at this time, these two I frames are recorded as key frames.

[0028] Video files contain a large amount of image information. During the video compression and decoding process, key frames are screened out from I frames according to the inter-frame performance difference, and the video homology test is carried out by using the pattern noise difference between key frames, which greatly reduces the computational amount of frame-by-frame analysis during the test.

[0029] The aging of surveillance device components has a significant impact on the PRNU noise in the video, and this impact shows obvious differences in different regions. Specifically, in the static region - that is, the part of the video that is fixed or lacks dynamic objects - its PRNU noise is usually relatively constant and is affected by device aging to a relatively small extent. In contrast, due to factors such as moving objects and light fluctuations in the moving region, it is easy to introduce additional interference noise, which in turn interferes with the stable detection of PRNU noise.

[0030] Therefore, when performing video homology verification, in order to ensure the accuracy and reliability of the verification, the PRNU noise characteristics of the static region can be used to correct or compensate the moving region. This aims to reduce the noise interference caused by non-PRNU factors in the moving region, thereby improving the accuracy and consistency of the overall noise analysis and providing a more reliable basis for video homology judgment. Therefore, it is first necessary to divide the static region and the moving region in the key frame, and divide the reference de-motion static image and the reference de-static moving image from the reference key frame, and divide the to-be-verified de-motion static image and the to-be-verified de-static moving image from the to-be-verified key frame.

[0031] The acquisition processes of the reference de-motion static image and the reference de-static moving image will be described in detail in the second embodiment and will not be elaborated here.

[0032] It should be noted that the acquisition processes of the to-be-verified de-motion static image and the to-be-verified de-static moving image are the same as those of the reference de-motion static image and the reference de-static moving image and will not be elaborated here.

[0033] S102. Calculate the static noise matching degree of the corresponding sub-blocks between the reference de-motion static image and the to-be-verified de-motion static image, and the motion noise matching degree of the corresponding sub-blocks between the reference de-static moving image and the to-be-verified de-static moving image.

[0034] The acquisition process of the static noise matching degree will be described in detail in the third embodiment and will not be elaborated here.

[0035] It should be noted that the acquisition process of the motion noise matching degree is the same as that of the static noise matching degree, and both need to obtain the pattern noise. Only the acquisition process of the pattern noise of the to-be-verified de-static moving image and the reference de-static moving image needs to be corrected, which is different in this regard. The acquisition process of the pattern noise of the to-be-verified de-static moving image will be described in detail in the sixth embodiment and will not be elaborated here. The acquisition process of the pattern noise of the to-be-verified de-static moving image is the same as that of the reference de-static moving image.

[0036] It should be noted that the pattern noise components mainly include fixed pattern noise (FPN, fixed pattern noise) and light-induced non-uniform noise. Among them, FPN is an additive noise that can be eliminated by subtracting a dark frame and cannot be used as a device fingerprint for homology judgment; PRNU is mainly generated by the pixel non-uniformity of the sensor, because the manufacturing process causes inconsistencies in the photosensitive elements of the sensor. The PRNU pattern noise contained in different video frames captured by the same device has the same nature, and the noise is mainly concentrated in the high-frequency band.

[0037] S103. Construct an objective function based on the static noise matching degree, the motion noise matching degree, a first preset value, and a preset iteration step length for iterative screening, and correspondingly obtain a reference de-motion static image to be restored, a reference de-static motion image to be restored, a to-be-tested de-motion static image to be restored, and a to-be-tested de-static motion image to be restored according to the screening results.

[0038] Specifically, the objective function includes: The value obtained by subtracting the first preset value from the preset iteration step length is used as the input value of an exponential function with base e, and the output value of the exponential function is multiplied by the absolute value of the noise matching degree to obtain the objective function, where the absolute value of the noise matching degree is the absolute value of the difference between the motion noise matching degree and the static noise matching degree.

[0039] The objective function can be expressed by the formula: ; where, the represents the motion noise matching degree of the i-th sub-block corresponding between the reference de-static motion image and the to-be-tested de-static motion image, the represents the static noise matching degree of the i-th sub-block corresponding between the reference de-motion static image and the to-be-tested de-motion static image, the value range of i is from 1 to the number of sub-blocks, the represents the first preset value, the represents the preset iteration step length, preferably 0.05, and the represents the objective function.

[0040] When the screening result is stable, i.e., the value remains unchanged, stop the iteration.

[0041] What is obtained by screening is the reference de-static motion image and the to-be-tested de-static motion image that have corrected the noise deviation caused by changes, motion, and equipment degradation. In other words, they are the to-be-restored to-be-tested de-static motion image and the to-be-restored reference de-static motion image that have removed the interference noise.

[0042] For the sake of corresponding understanding, the reference de-motion static image and the to-be-tested de-motion static image corresponding to the to-be-restored to-be-tested de-static motion image and the to-be-restored reference de-static motion image are called the to-be-restored reference de-motion static image and the to-be-restored to-be-tested de-motion static image.

[0043] S104. Superimpose the reference motionless image to be restored and the reference motion image to be restored to obtain the restored reference key frame, and superimpose the test motionless image to be restored and the test motion image to be restored to obtain the restored test key frame.

[0044] The superimposing process is a prior art and will not be elaborated here. The superimposing is to restore the original reference key frame and the test key frame.

[0045] S105. Based on the static noise matching degree and the motion noise matching degree, calculate the key frame matching degree between the restored reference key frame and the corresponding restored test key frame. Use the structural similarity index value between the restored reference key frame and the corresponding restored test key frame as the weight, and perform weighted average on the key frame matching degree to obtain the similarity between the reference video and the test video. When the similarity is greater than the second preset value, the reference video and the test video come from the same monitoring device.

[0046] The process of obtaining the key frame matching degree will be described in detail in the seventh embodiment and will not be elaborated here.

[0047] Each pair of key frames, that is, the restored reference key frame and the corresponding restored test key frame, has a corresponding key frame matching degree.

[0048] The second preset value can be set independently, and is preferably 0.85.

[0049] Figure 2 This is a flowchart of the process for obtaining the reference motionless image and the reference motion image provided by the second embodiment of the present invention. The process for obtaining the reference motionless image and the reference motion image includes: S201. Separate the static region and the motion region in the reference key frame.

[0050] Separating the static region and the motion region from the reference key frame is a prior art, such as using a Gaussian mixture model for separation.

[0051] S202. Denoise the reference key frame to obtain a denoised reference key frame, and subtract the pixel values at the corresponding positions of the reference key frame and the denoised reference key frame to obtain a reference noise image.

[0052] Here, a Wiener filter based on wavelet transform can be used for denoising. Wavelet transform is a signal processing technique that decomposes an image into low-frequency and high-frequency components at multiple scales. Since the pattern noise of the image is mainly concentrated in the high-frequency segment, filtering is performed in combination with the Wiener filter to remove the high-frequency noise part and retain the effective information of the image, obtaining a denoised reference key frame. The pixel values at the corresponding positions of the reference key frame and the denoised reference key frame are subtracted to obtain a reference noise image. The reference noise image represents the difference between the reference key frame and the denoised reference key frame.

[0053] S203. Respectively match the static region and the moving region with the reference noise image to obtain the reference static-noise-removed image and the reference moving-noise-removed image.

[0054] For matching, a key point matching algorithm can be used for image matching, which is a known algorithm.

[0055] The reference static-noise-removed image is a noise image that only contains the static region, and the reference moving-noise-removed image is a noise image that only contains the moving region.

[0056] Figure 3 This is a flowchart of the process for obtaining the static noise matching degree provided by the third embodiment of the present invention. The process for obtaining the static noise matching degree includes: S301. Respectively divide the reference static-noise-removed image and the to-be-tested static-noise-removed image into at least two sub-blocks.

[0057] The size of the sub-blocks can be set to be uniform.

[0058] S302. Respectively obtain the pixel values of each pattern noise in each sub-block of the reference static-noise-removed image and the to-be-tested static-noise-removed image.

[0059] S303. Obtain the reference stability factor of each sub-block in the reference static-noise-removed image and the to-be-tested stability factor of each sub-block in the to-be-tested static-noise-removed image.

[0060] The process for obtaining the reference stability factor will be described in detail in the fourth embodiment and will not be elaborated here.

[0061] The process for obtaining the to-be-tested stability factor is the same as the process for obtaining the reference stability factor.

[0062] S304. Calculate the square of the difference between the pixel values of the i-th sub-block of the reference motion-compensated still image corresponding to each pattern noise and the pixel values of the i-th sub-block of the to-be-tested motion-compensated still image, and obtain the sum of the squares of the pixel value differences, where the value of i ranges from 1 to the number of sub-blocks.

[0063] The sum of the squares can be expressed by the formula: ; where, the represents the pixel value of the v-th pattern noise in the i-th sub-block of the reference motion-compensated still image, and the represents the pixel value of the v-th pattern noise in the i-th sub-block of the to-be-tested motion-compensated still image, and the nv represents the number of pattern noises in the i-th sub-block.

[0064] It is used to measure the pixel value difference between the pattern noises at the same position in the i-th sub-block of the reference motion-compensated still image and the to-be-tested motion-compensated still image. The more obvious the difference, the lower the still noise matching degree.

[0065] S305. After subtracting the reference stability factor corresponding to the i-th sub-block from the to-be-tested stability factor, divide the result by the to-be-tested stability factor to obtain a first ratio, and obtain the absolute value of the first ratio.

[0066] The absolute value of the first ratio can be expressed by the formula: ; where, the represents the reference stability factor corresponding to the i-th sub-block, the represents the to-be-tested stability factor corresponding to the i-th sub-block, and the represents the absolute value function.

[0067] It represents the influence of the stability of the sub-block on the still noise matching degree. The closer the values of the stability factors of the sub-blocks are, the smaller the change degree of the pattern noises in the same sub-block in the time series, and the better the matching effect of the sub-block.

[0068] S306. Multiply the sum of the squares by the absolute value and then take the opposite as the input value of the exponential function with e as the base, and the output value of the exponential function is used as the still noise matching degree of the corresponding i-th sub-block between the reference motion-compensated still image and the to-be-tested motion-compensated still image.

[0069] The still noise matching degree can be expressed as: ; Among them, the represents the exponential function, and the represents the static noise matching degree of the i-th sub-block corresponding between the reference de-motion still image and the to-be-tested de-motion still image.

[0070] S307. Repeat the static noise matching degree of the i-th sub-block to obtain the static noise matching degrees of each sub-block corresponding between the reference de-motion still image and the to-be-tested de-motion still image.

[0071] Figure 4 is a flowchart of the obtaining process of the reference stability factor provided by the fourth embodiment of the present invention. The obtaining process of the reference stability factor includes: S401. Calculate the noise information of each noise pixel point at each position in each sub-block of the reference de-motion still image, and screen out the noise information greater than the first preset value as the pattern noise of the reference de-motion still image.

[0072] The obtaining process of the noise information will be described in detail in the fifth embodiment and will not be elaborated here.

[0073] The first preset value can be set independently, and is preferably 0.75.

[0074] Pattern noise is caused by the inconsistency of the photosensitive elements of the sensor due to the manufacturing process, and shows consistency in the same area of multiple images at different times. The pattern noise of the monitoring device cannot be obtained only through a single image. Therefore, it is necessary to utilize the noise stability of the same corresponding area within the time series to screen out the included pattern noise.

[0075] S402. Obtain the noise information of the noise pixel point at the x-th position in the i-th sub-block of the reference de-motion still image. The i-th sub-block contains the minimum value and the maximum value of the pattern noise in the reference key frame, where the value of x ranges from 1 to the number of the position noise pixel points in the i-th sub-block.

[0076] S403. After subtracting the minimum value from the noise information, divide it by the difference between the maximum value and the minimum value to obtain the reference stability factor component, and the sum of the reference stability factor components is the reference stability factor of the i-th sub-block.

[0077] The reference stability factor of the i-th sub-block can be expressed by the formula: ; Among them, the represents the reference stability factor component, and the represents the noise information of the noise pixel at the x-th position in the i-th sub-block of the reference motion-compensated still image, and the represents the minimum value of the pattern noise contained in the i-th sub-block in the reference key frame, and the represents the maximum value of the pattern noise contained in the i-th sub-block in the reference key frame, and the represents the reference stability factor of the i-th sub-block, and nx represents the number of the position noise pixels in the i-th sub-block.

[0078] S404. Repeat the reference stability factor of the i-th sub-block to obtain the reference stability factors of all the sub-blocks in the reference motion-compensated still image.

[0079] Figure 5 is a flowchart of the process for obtaining the noise information provided in the fifth embodiment of the present invention. The process for obtaining the noise information includes: S501. Obtain the pixel value of the noise pixel at the x-th position in the i-th sub-block of each reference motion-compensated still image and the pixel mean value of the noise pixel at the x-th position in the i-th sub-block.

[0080] The pixel value of the noise pixel at the x-th position in the i-th sub-block in the k-th reference motion-compensated still image can be represented by The pixel mean value of the noise pixel at the x-th position in the i-th sub-block can be represented by to represent.

[0081] S502. Take the average value of the absolute values of the differences between each pixel value and the pixel mean value as the input value of the exponential function with e as the base, and the output value of the exponential function is used as the noise information of the noise pixel at the x-th position in the i-th sub-block of the reference motion-compensated still image.

[0082] The noise information can be represented by the formula: ; where the represents the exponential function, the represents the number of the reference motion-compensated still images, and the represents the noise information of the noise pixel at the x-th position in the i-th sub-block of the reference motion-compensated still image.

[0083] Further, the method further includes: before calculating the motion noise matching degree, obtaining the pattern noise of the to-be-tested motionless motion image. Since the pattern noise is randomly manifested in the stationary region of the image, and due to the influence of factors such as the direction, speed, and illumination of moving objects in the motion region, the noise distribution has different manifestation rules. By using the random manifestation of the pattern noise in the stationary region, filtering and denoising the noise in the motion region, effectively distinguishing the pattern noise from the noise caused by real motion information, and filtering out the interference noise caused by factors such as motion.

[0084] Figure 6 FIG. is a flowchart of the process of obtaining the pattern noise of the to-be-tested motionless motion image provided by the sixth embodiment of the present invention. The process of obtaining the pattern noise of the to-be-tested motionless motion image includes: S601. Calculate the minimum value of the first standard deviation of the noise change degree of the i-th sub-block in two adjacent frames of the to-be-tested motionless motion image, and calculate the second standard deviation of the noise change degree of the i-th sub-block in two adjacent frames of the to-be-tested motionless motion image. Divide the second standard deviation by the minimum value to obtain the relative motion noise change degree, where the value of i ranges from 1 to the number of sub-blocks.

[0085] The relative motion noise change degree can be expressed as: ; where, the represents the second standard deviation of the noise change degree of the i-th sub-block in two adjacent frames of the to-be-tested motionless motion image, the represents the minimum value of the first standard deviation of the noise change degree of the i-th sub-block in two adjacent frames of the to-be-tested motionless motion image, and the represents the relative motion noise change degree.

[0086] The noise change degree refers to the change degree of noise pixel points.

[0087] S602. After dividing the relative motion noise change degree by the first preset value, multiply it by the noise information of the noise pixel point at the x-th position in the i-th sub-block of the to-be-tested motionless motion image as the pattern noise of the to-be-tested motionless motion image, where the value of x ranges from 1 to the number of noise pixel points at the position in the i-th sub-block.

[0088] The pattern noise can be expressed by the formula: ; where, the represents the first preset value, and the Represents the noise information of the noise pixel at the x-th position in the i-th sub-block of the to-be-tested de-stationary motion image, and the Represents the pattern noise of the to-be-tested de-stationary motion image.

[0089] The first preset value can be set independently, and is preferably 0.75.

[0090] Figure 7 Is a flowchart of the obtaining process of the key frame matching degree provided by the seventh embodiment of the present invention. The obtaining process of the key frame matching degree includes: S701. Calculate the sum value of the stationary noise matching degree and the motion noise matching degree corresponding to each sub-block respectively.

[0091] The sum value can be expressed by the formula: ; Wherein, the Represents the motion noise matching degree of the i-th sub-block corresponding between the reference de-stationary motion image and the to-be-tested de-stationary motion image, and the Represents the stationary noise matching degree of the i-th sub-block corresponding between the reference de-motion stationary image and the to-be-tested de-motion stationary image.

[0092] S702. Obtain the average value of the sum value as the key frame matching degree.

[0093] The key frame matching degree can be expressed as: ); Wherein, the Represents the number of sub-blocks, and the Represents the key frame matching degree.

[0094] When the Is higher, it indicates that the pattern noise coincidence degree between the restored reference key frame and the corresponding restored to-be-tested key frame is higher, and then the possibility that the restored reference key frame and the corresponding restored to-be-tested key frame are homologous is greater.

[0095] Figure 8 Is a schematic diagram of a homologous monitoring video tracing device based on pattern noise provided by the eighth embodiment of the present invention. The device includes: An obtaining module 801, configured to extract key frames from a reference video and a to-be-tested video respectively to obtain a reference key frame and a to-be-tested key frame, and respectively obtain a reference de-motion stationary image, a reference de-stationary motion image, a to-be-tested de-motion stationary image, and a to-be-tested de-stationary motion image according to the reference key frame and the to-be-tested key frame; A calculation module 802, configured to calculate the static noise matching degree of corresponding sub-blocks between the reference de-motion static image and the to-be-tested de-motion static image, and the motion noise matching degree of corresponding sub-blocks between the reference de-static motion image and the to-be-tested de-static motion image; A screening module 803, configured to construct an objective function based on the static noise matching degree, the motion noise matching degree, a first preset value, and a preset iteration step size for iterative screening, and correspondingly obtain a to-be-restored reference de-motion static image, a to-be-restored reference de-static motion image, a to-be-restored to-be-tested de-motion static image, and a to-be-restored to-be-tested de-static motion image according to the screening result; An overlay module 804, configured to overlay the to-be-restored reference de-motion static image and the to-be-restored reference de-static motion image into a restored reference key frame, and overlay the to-be-restored to-be-tested de-motion static image and the to-be-restored to-be-tested de-static motion image into a restored to-be-tested key frame; An inspection module 805, configured to obtain a key frame matching degree between the restored reference key frame and the corresponding restored to-be-tested key frame based on the static noise matching degree and the motion noise matching degree, and perform weighted averaging on the key frame matching degree with the structural similarity index value between the restored reference key frame and the corresponding restored to-be-tested key frame as a weight to obtain a similarity between the reference video and the to-be-tested video. When the similarity is greater than a second preset value, the reference video and the to-be-tested video are from the same monitoring device.

[0096] The technical features and technical effects of the same-source monitoring video tracing device based on pattern noise proposed in the embodiments of the present invention are the same as those of the method proposed in the embodiments of the present invention, and will not be elaborated here.

[0097] The embodiments of the present invention further provide a same-source monitoring video tracing system, and the system includes the above-mentioned device.

[0098] The present invention has the following beneficial effects: First, key frames are respectively extracted from a reference video and a to-be-tested video to obtain a reference key frame and a to-be-tested key frame, and a reference de-motion static image and a reference de-static motion image, a to-be-tested de-motion static image, and a to-be-tested de-static motion image are respectively obtained according to the reference key frame and the to-be-tested key frame. The influence of the PRNU noise caused by the aging of the monitoring device on the static area in the image is small, and the influence on the motion area is large. Therefore, both the reference key frame and the to-be-tested key frame are extracted for the static area and the motion area.

[0099] Secondly, calculate the still noise matching degree of corresponding sub-blocks between the reference motion-removed still image and the to-be-tested motion-removed still image, and the motion noise matching degree of corresponding sub-blocks between the reference still-removed motion image and the to-be-tested still-removed motion image. The still noise matching degree is the matching degree of the noise in the still region between the reference key frame and the to-be-tested key frame, and the motion noise matching degree is the matching degree of the noise in the motion region between the reference key frame and the to-be-tested key frame.

[0100] Then, construct an objective function based on the still noise matching degree, the motion noise matching degree, the first preset value, and the preset iteration step size for iterative screening, and correspondingly obtain the to-be-restored reference motion-removed still image, the to-be-restored reference still-removed motion image, the to-be-restored to-be-tested motion-removed still image, and the to-be-restored to-be-tested still-removed motion image according to the screening results. The screening is to obtain the to-be-restored to-be-tested still-removed motion image and the to-be-restored reference still-removed motion image that remove the interference noise. For the convenience of distinction, the reference motion-removed still image and the to-be-tested motion-removed still image corresponding to the to-be-restored to-be-tested still-removed motion image and the to-be-restored reference still-removed motion image are called the to-be-restored reference motion-removed still image and the to-be-restored to-be-tested motion-removed still image.

[0101] Furthermore, superimpose the to-be-restored reference motion-removed still image and the to-be-restored reference still-removed motion image to form the restored reference key frame, and superimpose the to-be-restored to-be-tested motion-removed still image and the to-be-restored to-be-tested still-removed motion image to form the restored to-be-tested key frame. The superimposition is to restore the original reference key frame and the to-be-tested key frame.

[0102] Finally, based on the still noise matching degree and the motion noise matching degree, calculate the key frame matching degree between the restored reference key frame and the corresponding restored to-be-tested key frame, and use the structural similarity index value between the restored reference key frame and the corresponding restored to-be-tested key frame as the weight to perform weighted averaging on the key frame matching degree to obtain the similarity between the reference video and the to-be-tested video. When the similarity is greater than the second preset value, the reference video and the to-be-tested video come from the same monitoring device. The higher the key frame matching degree, the higher the degree of coincidence of the pattern noise between the reference key frame and the to-be-tested key frame, and the greater the possibility of homology between the reference key frame and the to-be-tested key frame.

[0103] It should be noted that: the above sequence of the embodiments of the present invention is only for description and does not represent the advantages and disadvantages of the embodiments. The processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0104] Each embodiment in this specification is described in a progressive manner. For the same or similar parts among the embodiments, reference can be made to each other, and the differences between each embodiment and other embodiments are emphasized respectively.

Claims

1. A method for tracing homologous surveillance videos based on pattern noise, characterized in that The method includes: Extracting key frames from the reference video and the video to be inspected respectively to obtain the reference key frames and the key frames to be inspected, and obtaining the reference motionless still image, the reference still motion image, the to-be-inspected motionless still image, and the to-be-inspected still motion image respectively according to the reference key frames and the to-be-inspected key frames; Calculating the still noise matching degree of corresponding sub-blocks between the reference motionless still image and the to-be-inspected motionless still image, and the motion noise matching degree of corresponding sub-blocks between the reference still motion image and the to-be-inspected still motion image; Constructing an objective function based on the still noise matching degree, the motion noise matching degree, a first preset value, and a preset iteration step length for iterative screening, and correspondingly obtaining the to-be-restored reference motionless still image, the to-be-restored reference still motion image, the to-be-restored to-be-inspected motionless still image, and the to-be-restored to-be-inspected still motion image according to the screening results; Overlaying the to-be-restored reference motionless still image and the to-be-restored reference still motion image to obtain the restored reference key frame, and overlaying the to-be-restored to-be-inspected motionless still image and the to-be-restored to-be-inspected still motion image to obtain the restored to-be-inspected key frame; Calculating the key frame matching degree between the restored reference key frame and the corresponding restored to-be-inspected key frame based on the still noise matching degree and the motion noise matching degree, and taking the structural similarity index value between the restored reference key frame and the corresponding restored to-be-inspected key frame as a weight to perform weighted averaging on the key frame matching degree to obtain the similarity between the reference video and the to-be-inspected video. When the similarity is greater than a second preset value, the reference video and the to-be-inspected video are from the same monitoring device.

2. The method for tracing homologous monitoring videos based on pattern noise according to claim 1, characterized in that, The obtaining process of the reference motionless still image and the reference still motion image includes: Separating the still region and the motion region in the reference key frame; Denosing the reference key frame to obtain the denoised reference key frame, and subtracting the pixel values at the corresponding positions of the reference key frame and the denoised reference key frame to obtain the reference noise image; Matching the still region and the motion region with the reference noise image respectively to obtain the reference motionless still image and the reference still motion image.

3. The method for tracing homologous monitoring videos based on pattern noise according to claim 1, characterized in that The obtaining process of the still noise matching degree includes: Dividing the reference motionless still image and the to-be-inspected motionless still image into at least two sub-blocks respectively; Obtaining the pixel values of each mode noise in each sub-block of the reference motionless still image and the to-be-inspected motionless still image respectively; Obtaining the reference stability factor of each sub-block in the reference motionless still image and the to-be-inspected stability factor of each sub-block in the to-be-inspected motionless still image; Calculating the square of the difference between the pixel values of each corresponding mode noise in the i-th sub-block of the reference motionless still image and the i-th sub-block of the to-be-inspected motionless still image, and obtaining the sum of the squares of the differences of the pixel values, where the value of i ranges from 1 to the number of sub-blocks; After subtracting the reference stability factor corresponding to the i-th sub-block from the to-be-tested stability factor, divide the result by the to-be-tested stability factor to obtain a first ratio, and take the absolute value of the first ratio; Multiply the sum of squares by the absolute value and then take the negative value as the input value of the exponential function with base e, and the output value of the exponential function is used as the static noise matching degree of the i-th sub-block corresponding between the reference motion-removed still image and the to-be-tested motion-removed still image; Repeat the static noise matching degree of the i-th sub-block to obtain the static noise matching degrees of all the sub-blocks corresponding between the reference motion-removed still image and the to-be-tested motion-removed still image.

4. The method for tracing homologous surveillance videos based on pattern noise according to claim 3, wherein, The process of obtaining the reference stability factor includes: Calculate the noise information of the noise pixel points at each position in each sub-block of the reference motion-removed still image, and screen out the noise information greater than the first preset value as the pattern noise of the reference motion-removed still image; Obtain the noise information of the noise pixel point at the x-th position in the i-th sub-block of the reference motion-removed still image, where the i-th sub-block contains the minimum value and the maximum value of the pattern noise in the reference key frame, and the value of x ranges from 1 to the number of the position noise pixel points in the i-th sub-block; After subtracting the minimum value from the noise information, divide the result by the difference between the maximum value and the minimum value to obtain a reference stability factor component, and the sum of all the reference stability factor components is the reference stability factor of the i-th sub-block; Repeat the reference stability factor of the i-th sub-block to obtain the reference stability factors of all the sub-blocks in the reference motion-removed still image.

5. The method for tracing homologous monitoring videos based on pattern noise according to claim 4, wherein The process of obtaining the noise information includes: Obtain the pixel value of the noise pixel point at the x-th position in the i-th sub-block of each reference motion-removed still image and the pixel mean value of the noise pixel point at the x-th position in the i-th sub-block; The average value of the absolute values of the differences between each pixel value and the pixel mean value is used as the input value of the exponential function with base e, and the output value of the exponential function is used as the noise information of the noise pixel point at the x-th position in the i-th sub-block of the reference motion-removed still image.

6. The method for tracing homologous monitoring videos based on pattern noise according to claim 1, wherein, The method further includes: before calculating the motion noise matching degree, obtain the pattern noise of the to-be-tested motion-removed still image, and the process of obtaining the pattern noise of the to-be-tested motion-removed still image includes: Calculate the minimum value of the first standard deviation of the noise change degree of the i-th sub-block in two adjacent frames of the to-be-tested motion-removed still image, and calculate the second standard deviation of the noise change degree of the i-th sub-block in two adjacent frames of the to-be-tested motion-removed still image, and divide the second standard deviation by the minimum value to obtain the relative motion noise change degree, where the value of i ranges from 1 to the number of sub-blocks; After dividing the degree of change in the relative motion noise by the first preset value, multiply the result by the noise information of the noise pixel at the x-th position in the i-th sub-block of the to-be-tested motion-removed still image as the pattern noise of the to-be-tested motion-removed still image, where the value of x ranges from 1 to the number of the position noise pixels in the i-th sub-block.

7. The method for tracing homologous monitoring videos based on pattern noise according to claim 1, wherein, The objective function includes: The value obtained by subtracting the first preset value from the preset iteration step is used as the input value of the exponential function with base e. The output value of the exponential function is multiplied by the absolute value of the noise matching degree to obtain the objective function, and the absolute value of the noise matching degree is the absolute value of the difference between the motion noise matching degree and the static noise matching degree.

8. The method for tracing homologous surveillance videos based on pattern noise according to claim 1, wherein The process of obtaining the key frame matching degree includes: Calculate the sum of the static noise matching degree and the motion noise matching degree corresponding to each sub-block respectively; Obtain the average value of the sum as the key frame matching degree.

9. The same-source monitored video tracing device based on pattern noise, characterized in that, The device includes: An acquisition module, configured to extract key frames from a reference video and a to-be-tested video respectively to obtain a reference key frame and a to-be-tested key frame, and respectively obtain a reference motion-removed still image, a reference motion-removed static image, a to-be-tested motion-removed still image, and a to-be-tested motion-removed static image according to the reference key frame and the to-be-tested key frame; A calculation module, configured to calculate the static noise matching degree of corresponding sub-blocks between the reference motion-removed still image and the to-be-tested motion-removed still image, and the motion noise matching degree of corresponding sub-blocks between the reference motion-removed static image and the to-be-tested motion-removed static image; A screening module, configured to construct an objective function based on the static noise matching degree, the motion noise matching degree, the first preset value, and the preset iteration step for iterative screening, and correspondingly obtain a to-be-restored reference motion-removed still image, a to-be-restored reference motion-removed static image, a to-be-restored to-be-tested motion-removed still image, and a to-be-restored to-be-tested motion-removed static image according to the screening result; An overlay module, configured to overlay the to-be-restored reference motion-removed still image and the to-be-restored reference motion-removed static image into a restored reference key frame, and overlay the to-be-restored to-be-tested motion-removed still image and the to-be-restored to-be-tested motion-removed static image into a restored to-be-tested key frame; A verification module, configured to obtain the key frame matching degree between the restored reference key frame and the corresponding restored to-be-tested key frame based on the static noise matching degree and the motion noise matching degree, and perform weighted averaging on the key frame matching degree with the structural similarity index value between the restored reference key frame and the corresponding restored to-be-tested key frame as the weight to obtain the similarity between the reference video and the to-be-tested video. When the similarity is greater than the second preset value, the reference video and the to-be-tested video are from the same monitoring device.

10. The homologous monitoring video traceability system based on pattern noise is characterized in that The system includes the device as claimed in claim 9.

Images