A high-efficiency matching and retrieval method for visual images based on space-time features

Abstract

The application discloses a kind of based on space-time feature's visual image efficient matching and retrieval method, it is related to computer vision field.The method comprises the following steps: receiving a visual data object to be authenticated;Start a physical space source feature analysis channel to extract a physical space feature from imager, and calculate the source credibility score;Parallelly start a physical time content feature analysis channel to extract the time correlation feature between moving target and its associated phenomenon, and calculate the content authenticity score.When the content authenticity score meets the preset trigger condition, a cross-validation step is performed.By comparing the local physical space features of the suspicious target area and background area identified in the content analysis, a decision result is obtained as to whether the content has been tampered with.The application generates a structured data object based on all analysis results.The application improves the robustness and reliability of authentication through dual verification of source and content and intelligent linkage.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision, in particular to a visual image efficient matching and retrieval method based on space-time features. BACKGROUND

[0002] As an important carrier of information, the authenticity and reliability of the source of digital images and videos are crucial in the fields of judicial evidence, news reporting, copyright protection and national security. Currently, the technical methods for source authentication and content authenticity detection of visual data objects can be mainly divided into the following two categories.

[0003] The first method is based on the inherent traces contained in the digital content itself. One branch of this method is to detect specific imprints introduced during the generation, transmission and storage of digital images. For example, by analyzing the quantization table of JPEG images, the statistical distribution of Hough coefficients or the double compression effect, it can be determined whether the image has been edited. Another branch of this method is to detect the inconsistencies that may be introduced at the pixel level by tampering operations. For example, by analyzing the noise distribution difference of local image regions, the discontinuity of color interpolation patterns or the inconsistency of lighting environments, the possible tampering regions can be located. The limitation of this method is that the digital traces it relies on may be effectively erased, hidden or imitated in the face of advanced editing software, specific processing procedures or artificial intelligence generated content (AIGC) technology, resulting in a decrease or failure of the reliability of the detection method.

[0004] The second method is based on the metadata associated with the visual data object for source authentication. For example, by reading the Exchangeable Image File Format (EXIF) information in the image file, the data such as the model of the shooting device, the shooting time and the geographic location can be obtained. The limitation of this method is that metadata is essentially text information independent of image pixel data and can be easily edited or stripped. This means that metadata cannot establish a stable and tamper-proof association between the visual data object and its physical shooting device, and therefore cannot serve as a high-confidence source authentication basis. SUMMARY

[0005] The present application provides a visual image efficient matching and retrieval method based on space-time features to solve the technical problems of authentication evidence being easily tampered with, detection traces being easily erased, and being unable to effectively deal with advanced composite forgeries in the field of source authentication and content authenticity detection of digital images and videos.

[0006] In view of the above problems, the present application provides a visual image efficient matching and retrieval method based on space-time features, which comprises the following steps:

[0007] receiving a visual data object to be authenticated containing timing information;

[0008] starting a physical space source feature analysis channel for extracting a physical space feature originated from an imaging device from the visual data object, and calculating a source credibility score based on the physical space feature;

[0009] starting a physical time content feature analysis channel in parallel for extracting a time correlation feature between a moving target and its accompanying phenomenon in the visual data object, and calculating a content authenticity score based on the time correlation feature;

[0010] judging whether the content authenticity score meets a condition for starting cross verification according to a preset trigger condition;

[0011] when the trigger condition is met, performing a cross verification step, which specifically includes:

[0012] dividing pixel data of the visual data object into a suspicious target region and a background region other than the suspicious target region based on a pixel mask sequence corresponding to the suspicious target identified in the physical time content feature analysis channel;

[0013] extracting a local physical space feature of the suspicious target region and a physical space feature of the background region respectively;

[0014] obtaining a cross verification decision result by comparing the local physical space feature of the suspicious target region with the physical space feature of the background region;

[0015] generating and outputting a structured data object containing an authentication conclusion based on the source credibility score, the content authenticity score and the cross verification decision result obtained when the cross verification step is performed.

[0016] The technical scheme provided by the present application has at least the following technical effects or advantages:

[0017] By hashing the video aggregated photo response non-uniformity (PRNU) feature for fast retrieval, the present application solves the problem of large-scale, efficient and reliable physical authentication of video sources, and overcomes the limitations of traditional metadata authentication methods and existing PRNU applications that are difficult to scale.

[0018] By quantifying the multi-dimensional spatio-temporal feature correlation between a moving object and its accompanying shadow, the present invention proposes a brand-new content authenticity detection dimension based on high-order physical logic. This can effectively detect advanced fake contents that have successfully imitated the underlying pixel traces, but are difficult to maintain perfect consistency in physical laws, breaking through the recognition bottleneck of traditional detection methods for such fake means.

[0019] The present invention uses the output of content authenticity analysis as a guide signal for source feature analysis. This combination uses the "weak doubts" of content logic analysis to trigger and locate the "focused" physical source review of specific areas, thereby directly converting an uncertain logical inference into a high-confidence decisive evidence about whether the local content is consistent with the background physical source, solving the technical problem that high-hidden implantable fakes are difficult to be finally confirmed. BRIEF DESCRIPTION OF DRAWINGS

[0020] Figure 1 A flowchart of a visual image efficient matching and retrieval method based on spatio-temporal features according to the present invention. DETAILED DESCRIPTION

[0021] To make the objectives, technical solutions and advantages of the present invention clearer, the following will combine embodiments to clearly and completely describe the visual image efficient matching and retrieval method based on spatio-temporal features according to the present invention. It should be understood that the present invention is not limited to the exemplary embodiments for explaining the present invention. Based on the embodiments in the present invention, all other embodiments obtained by those of ordinary skill in the art without making creative efforts fall within the scope of protection of the present invention.

[0022] The complete implementation process of the method according to the present invention starts from receiving a visual data object to be authenticated containing timing information, which in an embodiment can be a digital video file encoded according to a specific standard (e.g. MPEG-4 or H.264). The process goes through a series of data processing and analysis steps designed to have strong causal relationship, and finally outputs a structured data object containing multi-dimensional authentication conclusions. In this linear workflow, the calculation output of any previous step is designed as the necessary input for the decision-making of the subsequent step.

[0023] Please refer to Figure 1 A visual image efficient matching and retrieval method based on spatio-temporal features, the method comprising the following steps:

[0024] receiving a visual data object to be authenticated containing timing information;

[0025] starting a physical space source feature analysis channel for extracting a physical space feature originated from an imaging device from the visual data object, and calculating a source credibility score based on the physical space feature;

[0026] starting a physical time content feature analysis channel in parallel for extracting a time correlation feature between a moving target and its accompanying phenomenon in the visual data object, and calculating a content authenticity score based on the time correlation feature;

[0027] judging whether the content authenticity score meets a condition for starting cross-validation according to a preset trigger condition;

[0028] when the trigger condition is met, performing a cross-validation step, which specifically includes:

[0029] dividing pixel data of the visual data object into a suspicious target region and a background region other than the suspicious target region based on a pixel mask sequence corresponding to the suspicious target identified in the physical time content feature analysis channel;

[0030] extracting a local physical space feature of the suspicious target region and a physical space feature of the background region, respectively;

[0031] obtaining a cross-validation decision result by comparing the local physical space feature of the suspicious target region with the physical space feature of the background region;

[0032] generating and outputting a structured data object containing an authentication conclusion based on the source credibility score, the content authenticity score, and the cross-validation decision result obtained when the cross-validation step is performed.

[0033] First stage: parallel spatiotemporal feature extraction of two channels

[0034] After receiving a visual data object to be authenticated, the system starts two parallel analysis channels that process different physical dimension features: channel A (physical space feature analysis) and channel B (physical time feature analysis). This parallel processing design aims to separate the extraction processes of source fingerprint features and content logic features in execution.

[0035] Channel A: physical space source feature analysis: extract and analyze an inherent feature embedded in the visual data and originated from a physical imaging device to achieve matching of the physical shooting source.

[0036] In a preferred embodiment, the physical space feature is selected as the Photo-Response Non-Uniformity (PRNU) pattern of the imaging sensor (e.g. CCD or CMOS). The PRNU pattern is selected as the feature due to its high stability and individual uniqueness, which is considered as an effective device physical fingerprint in digital image provenance.

[0037] Video aggregated photo-response non-uniformity pattern extraction: As the photo-response non-uniformity signal in a single video frame has low signal-to-noise ratio and is interfered by scene content, compression artifacts and random noise, in order to obtain an aggregated photo-response non-uniformity pattern with higher signal-to-noise ratio and capable of stably representing the entire video shooting process, the following extraction process needs to be performed:

[0038] From the visual data object, a reference frame set containing N frames is selected according to a preset sampling strategy. The sampling strategy can be uniform interval sampling or an adaptive sampling based on scene analysis, for example, frames with more texture information but smaller motion amplitude are preferentially selected. The parameter N is a configurable engineering parameter, which is set to 50 in an embodiment, which can be adjusted in the range of, for example, 30 to 100, to balance the calculation overhead and the signal-to-noise ratio of the final aggregated pattern.

[0039] An image denoising algorithm or model is applied to each frame in the reference frame set to obtain a denoised image.

[0040] In a preferred embodiment, a block matching and three-dimensional collaborative filtering based algorithm or other types of image denoising techniques, such as a wavelet transform based denoising method or a pre-trained denoising convolutional neural network (DnCNN) model, can also be applied to this step. Subsequently, the noise residual of the frame is calculated by pixel-by-pixel subtraction operation. The noise residual theoretically contains the photo-response non-uniformity signal, random noise that is not completely filtered out and other high-frequency components.

[0041] All N noise residuals in the set are pixel-by-pixel weighted averaged to obtain the final aggregated photo-response non-uniformity pattern representing the video. This aggregation process aims to statistically suppress the random noise component through multiple averaging operations.

[0042] In an embodiment, the weights can be equal values; in another implementation, the weights can be allocated according to the energy or texture complexity of each frame residual.

[0043] Hash code generation and retrieval of light response non-uniformity: To achieve the retrieval of physical device source, the high-dimensional light response non-uniformity pattern needs to be converted into a binary feature descriptor, i.e. hash code.

[0044] Firstly, the aggregated light response non-uniformity pattern is spatially divided into M x M non-overlapping rectangular sub-blocks. The sub-block size is a tunable parameter, for example, it can be divided into 64 x 64 sub-blocks. Then, the statistical mean of all pixel values in each sub-block is calculated, thereby generating a 64 x 64 dimensional reduced dimension feature matrix.

[0045] The statistical median of all elements in the 64 x 64 feature matrix is calculated. Each element in the matrix is binarized according to its comparison relationship with the median. All binary bits are spliced according to a predetermined scanning order to form a 4096-bit binary string, which is the light response non-uniformity hash code of the video.

[0046] The generated hash code is matched with a pre-constructed fingerprint database containing known physical device light response non-uniformity hash codes. The matching process is achieved by calculating the Hamming distance between the hash code and each stored hash code in the database. The system will screen out a list of candidate devices with a Hamming distance less than a preset matching threshold.

[0047] Source credibility score: To obtain a quantitative source credibility in the retrieved candidate list, a further verification based on the complete light response non-uniformity pattern can be performed.

[0048] From the fingerprint database, the complete light response non-uniformity reference pattern of each candidate device screened out in the previous step is retrieved without dimension reduction. The peak correlation energy (PCE) between the aggregated light response non-uniformity pattern of the video to be tested and each reference pattern is calculated. PCE is an index for measuring the similarity and correlation peak of two signals. Other statistical similarity measures, such as normalized cross-correlation coefficient, can also be used.

[0049] The maximum value of all calculated PCE values is normalized by a mapping function to fall within a standardized interval (e.g. [0, 1]) as the final source credibility score. The output of the score is the end point of the channel A processing flow and the input of the subsequent decision fusion stage.

[0050] Channel B: Physical temporal content feature analysis: Extract and analyze a dynamic physical correlation feature in the time dimension to determine whether the video content follows the operation rules of the physical world.

[0051] In a preferred embodiment, the physical temporal feature is selected as the temporal synchronicity and geometric consistency between the moving object and its accompanying shadow.

[0052] Temporal-spatial target segmentation: This step separates the analysis object from the background in the video.

[0053] A video object segmentation (VOS) model is used to process the visual data object.

[0054] In a preferred embodiment, a model based on the Transformer architecture, such as Mask2Former or XMem with long-term memory capability, can be used to extract the pixel-level mask sequence of the main moving object frame by frame.

[0055] A shadow detection and segmentation algorithm is used. One implementation is to use a pre-trained deep learning shadow detection model. Another alternative implementation, which is technically characterized by relatively smaller computational overhead, is to use an image physics prior-based method, such as searching and segmenting the pixel region with low brightness (Value) and saturation (Saturation) characteristics in the vicinity of the subject mask in each frame as its corresponding shadow mask sequence by analyzing the HSV (Hue, Saturation, Value) color space.

[0056] Extraction of multi-dimensional temporal-spatial feature sequence: The segmented pixel-level mask target is converted into structured time series data. First, the output subject mask sequence and shadow mask sequence are obtained. Then, for each frame in the above two sequences, a set of multi-dimensional feature vectors that can describe its state is extracted. The vector should at least include:

[0057] Geometric features: centroid coordinates, mask area, minimum circumscribed rectangle aspect ratio, contour perimeter, shape convexity.

[0058] Motion features: instantaneous velocity vector and acceleration vector calculated from the displacement of the centroid between consecutive frames, angular velocity calculated from the change in the direction of the mask principal axis. Finally, the feature vectors of all frames are arranged in chronological order, and two parallel multi-dimensional feature time sequences are finally formed: the subject feature sequence and the shadow feature sequence.

[0059] Logical consistency score: This score quantifies the physical plausibility of the video content by computing the statistical correlation between two time series. First, the subject feature sequence and the shadow feature sequence are obtained. Then, one or more statistical correlation measures are used to compute the correlation between each corresponding dimension in the two sequences. In one embodiment, the Pearson correlation coefficient is used. In another embodiment, the Dynamic Time Warping (DTW) algorithm is used to handle possible non-linear time delay relationships. Finally, all the computed correlation coefficient values are combined into a "physical consistency" feature vector. This vector is input into a machine learning classifier that is pre-trained on a dataset containing both real and manipulated video samples. The classifier can be a Support Vector Machine (SVM), an XGBoost, or a Multi-layer Perceptron (MLP). The classifier outputs a final content authenticity score.

[0060] In one specific embodiment, the machine learning classifier is implemented as follows: its training can use a publicly available video manipulation detection dataset, such as VIDEOSHIELD; the feature vector input to the classifier includes the Pearson correlation coefficients, the Dynamic Time Warping (DTW) distances, and other statistical quantities of multi-dimensional spatio-temporal features, resulting in a 12-dimensional feature vector; the classifier model is a Multi-layer Perceptron (MLP) with a network structure that can be set to 3 layers, each containing 128 neurons.

[0061] Second stage: Cross-validation and decision fusion: This stage uses the output results of the two parallel channels to collaborate and cross-validate, in order to identify specific types of forgeries that may not be detected by a single channel analysis mode.

[0062] Trigger for cross-validation: triggered by a clear, configurable logical condition. In one embodiment, when the content authenticity score output by channel B is lower than a pre-set engineering threshold, the system interprets this event as "doubt about the physical logic of the content", and initiates the subsequent cross-validation process. At the same time, in the analysis of channel B, the moving object that contributes the lowest consistency score is marked as "suspicious target".

[0063] In one embodiment, the determination of the preset engineering threshold value comprises the following steps: first, a verification dataset containing known real samples and fake samples is prepared; then, the content authenticity score of all samples in the dataset is calculated; finally, by analyzing the receiver operating characteristic (ROC) curve, a working point with the highest detection rate at an acceptable false positive rate is selected, and the score corresponding to the point is set as the engineering threshold value. The cross-validation threshold value and other threshold values can also be determined by similar methods.

[0064] Localized spatial feature review based on content suspicion: using the source discrimination ability of channel A to review the content suspicion found by channel B, i.e., verifying whether the "suspicious target" and the "background" of the video have the same physical source fingerprint.

[0065] Division of target and background: using the obtained frame-by-frame mask sequence of the "suspicious target", the pixel data of the reference frame set of the visual data object is divided into two pixel sets corresponding to the "suspicious target region" and the "background region", respectively.

[0066] Localized pattern extraction: the light response non-uniformity pattern aggregation extraction algorithm is repeatedly executed on the above two pixel sets, respectively. The output of this process is two independent, localized patterns: the aggregated light response non-uniformity pattern of the "suspicious target" and the aggregated light response non-uniformity pattern of the "background".

[0067] Cross-validation decision: a conclusion is made about whether there is an implanted tampering in the content by quantitatively comparing the two localized patterns output in the previous step. This step specifically includes:

[0068] The normalized cross-correlation coefficient between the "aggregated light response non-uniformity pattern of the suspicious target" and the "aggregated light response non-uniformity pattern of the background" is calculated to obtain a cross-validation score.

[0069] The cross-validation score is compared with a preset cross-validation threshold value.

[0070] Decision based on comparison results: if the cross-validation score is less than the threshold value, it constitutes a technical evidence showing that there is a difference in the physical source between the suspicious target and the background.

[0071] Third stage: generation of structured data object: all analysis results are summarized into a structured data object with clear structure and traceability.

[0072] Decision logic integration: according to the three key values of source credibility score, content authenticity score and cross-validation score, a preset decision matrix or decision tree is used to generate the final, multi-level authentication conclusion. For example, the decision logic can include the following rules:

[0073] Rule 1: If both the source credibility score and the content authenticity score are not lower than their respective preset threshold values, the conclusion is “source authentication passed, content physical logic self-consistent”.

[0074] Rule 2: If the source credibility score is not lower than its threshold value, the content authenticity score is lower than its threshold value, but the cross-validation score is not lower than its threshold value, the conclusion is “source authentication passed, content exists physical anomaly but no implanted tampering detected”.

[0075] Rule 3: If the source credibility score is not lower than its threshold value, the content authenticity score is lower than its threshold value, and the cross-validation score is lower than its threshold value, the conclusion is “content implanted tampering detected”.

[0076] Data object generation and output: The above conclusion, together with the relevant supporting data and intermediate results (such as the matched candidate device ID, the source credibility score, the content authenticity score, the cross-validation score, and the threshold parameters used, etc.), are integrated into a structured, machine-readable (such as JSON or XML format) file to generate the structured data object and output to the user or downstream system.

[0077] In an alternative embodiment for handling complex scenarios, the method can further include further processing steps. To deal with the problem of ambiguous mask region boundaries, morphological filtering operations (such as image opening operation) can be used to smooth the mask after obtaining the initial mask sequence. In the case of multiple moving targets interacting in a video, the logical consistency score of each independent target can be calculated separately, and all targets with scores lower than a preset attention threshold are collectively defined as a suspicious target set for subsequent cross-validation.

[0078] Those skilled in the art can understand that the idea of the present application has multiple feasible implementation paths and application prospects. For example, in addition to using cross-correlation coefficients, alternative implementations can use statistical hypothesis testing methods to determine whether two light response non-uniformity pattern samples come from the same statistical population.

[0079] Similarly, the analysis and verification idea of the present application can be extended to other physical features. For example, the audio channel in the video can be used as another parallel analysis channel to extract the Electrical Network Frequency (ENF) fluctuation pattern generated by the physical characteristics of the microphone and cross-verify it with the analysis results of the video channel. In addition, the lens distortion pattern generated by a specific optical system or the rolling shutter effect generated by the line-by-line scanning of the image sensor can also be included in the analysis framework of the present application as alternative physical features.

[0080] The preferred embodiments of the application disclosed above are only to facilitate the elucidation of the application. The preferred embodiments do not describe all the details of the application and limit the application to the specific embodiments. Obviously, many modifications and variations can be made in light of the teachings above. The description is chosen and described in order to provide the best illustration of the application and its practical application to those skilled in the art and to enable those skilled in the art to best utilize the application. The application is limited only by the claims and their full scope and equivalents.

Claims

1. A method for efficient visual image matching and retrieval based on spatiotemporal features, characterized in that, The method includes the following steps: Receive a visual data object to be authenticated containing timing information; A physical spatial source feature analysis channel is initiated to extract a physical spatial feature originating from the imaging device from the visual data object, convert the physical spatial feature into a binary hash code, and calculate a source confidence score based on the physical spatial feature, wherein the physical spatial feature is the light response non-uniformity pattern of the imaging sensor. A physical time content feature analysis channel is launched in parallel to extract the temporal correlation features between moving targets and their accompanying phenomena in visual data objects, and a content authenticity score is calculated based on the temporal correlation features. Based on a preset trigger condition, determine whether the content authenticity score meets the conditions for initiating cross-validation; When the triggering condition is met, a cross-validation step is performed, which specifically includes: In the analysis of the physical time content feature analysis channel, a suspicious target was identified; Based on the pixel mask sequence corresponding to the suspicious target identified in the physical time content feature analysis channel, the pixel data of the visual data object is divided into a suspicious target region and a background region other than the suspicious target region. The local physical space features of the suspected target area and the physical space features of the background area are extracted respectively. By comparing the local physical space features of the suspected target area with the physical space features of the background area, a cross-validation decision result is obtained; And a generation step, which generates and outputs a structured data object containing the authentication conclusion based on the source credibility score, the content authenticity score, and the cross-validation decision result obtained when the cross-validation step is performed.

2. The efficient visual image matching and retrieval method based on spatiotemporal features as described in claim 1, characterized in that, The extraction of a physical spatial feature originating from the imaging device specifically includes: Sample a set of reference frames from the visual data object; Denoising is performed on each frame in the reference frame set to calculate its noise residual; The noise residuals of all frames in the reference frame set are aggregated to obtain an aggregated light response non-uniformity pattern representing the visual data object.

3. The method for efficient visual image matching and retrieval based on spatiotemporal features as described in claim 1, characterized in that, The temporal correlation feature is the temporal synchronization and geometric consistency between the moving object and its accompanying shadow; The specific steps of the physical time content feature analysis channel include: First, a video target segmentation algorithm and a shadow detection algorithm are used to segment the frame-by-frame mask sequence of the moving subject and the frame-by-frame mask sequence of the accompanying shadow from the visual data object, respectively. Next, based on the frame-by-frame mask sequence of the moving subject and the frame-by-frame mask sequence of the accompanying shadow, the multidimensional feature time sequence of the subject and the multidimensional feature time sequence of the shadow, which are used to describe their respective states, are extracted respectively. Finally, the content authenticity score is obtained by calculating the statistical correlation between the subject multidimensional feature time series and the shadow multidimensional feature time series.

4. The efficient visual image matching and retrieval method based on spatiotemporal features as described in claim 1, characterized in that, The triggering condition is: the content authenticity score is lower than a preset content authenticity threshold.

5. The method for efficient visual image matching and retrieval based on spatiotemporal features as described in claim 2, characterized in that, The specific steps of comparing the local physical space features in the cross-validation step are as follows: calculating the normalized cross-correlation coefficient between the non-uniformity of the aggregated light response in the suspected target region and the non-uniformity of the aggregated light response in the background region, and comparing the normalized cross-correlation coefficient with a preset cross-validation threshold to obtain the cross-validation decision result.

6. The efficient visual image matching and retrieval method based on spatiotemporal features as described in claim 1, characterized in that, The physical space source feature analysis channel also includes a retrieval step, which specifically includes: The binary hash code is compared with a pre-built database containing fingerprints of known physical devices using Hamming distance to quickly filter out a list of candidate devices.

7. The method for efficient visual image matching and retrieval based on spatiotemporal features as described in claim 3, characterized in that, The step of calculating a content authenticity score is as follows: inputting the calculation result of the statistical correlation into a pre-trained machine learning classifier, and having the classifier output the content authenticity score.

8. The efficient visual image matching and retrieval method based on spatiotemporal features as described in claim 1, characterized in that, The generation step specifically includes: Based on a preset decision matrix, the source credibility score, the content authenticity score, and the cross-validation decision result are mapped to a multi-level authentication conclusion. The authentication conclusion and related supporting data are integrated into a machine-readable file with a predetermined technical format to generate the structured data object.

Images