A method and system for deduplication and compliance verification of confidential evidence images.

Abstract

This invention provides a method and system for deduplication and compliance verification of confidential review evidence images, including S1. Acquisition and preprocessing of confidential review evidence images, S2. Dual fingerprint deduplication processing, S3. Multi-dimensional compliance verification processing, S4. Graded processing of compliance issues, and S5. Encrypted storage processing using the national cryptographic standard SM4. By establishing a dual fingerprint deduplication mechanism combining perceptual hashing and the national cryptographic standard SM3, accurate identification and verification of similar images are achieved, meeting national cryptographic compliance requirements. An automated compliance verification process and deduplication process are linked to achieve fully automated processing of confidential review evidence images. Storage compression is achieved by retaining the original image and replacing it with reference information. Encryption and packaging are performed using the national cryptographic standard SM4 algorithm to ensure the security and validity of confidential review data. This method features high deduplication accuracy, compliance verification tailored to confidential review needs, high processing efficiency, strong security and compliance, low storage cost, and clear data correlation.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to a method and system for deduplication and compliance verification of secret evidence images based on a backbone perceptual hash and the national cryptographic standard SM3 dual fingerprint. Background Technology

[0002] The existing cryptographic application security assessment system has significant limitations in the processing technology of evidence images and has not formed a targeted solution. The main technical pain points are as follows: ① Deduplication technology problem: The mainstream solution uses traditional hash algorithms such as MD5 (Message-Digest Algorithm 5, a traditional hash algorithm used to generate data digests, but it is not sensitive to minor changes in images) and SHA-256, but these algorithms can only identify completely identical image files. In reality, during the confidential assessment process, evidence images are often slightly adjusted due to minor changes in shooting angle, the addition of confidential assessment watermarks, and resolution adaptation to system display requirements. This causes drastic changes in traditional hash values, making it impossible to identify them as similar images and resulting in a large amount of duplicate storage. Some solutions introduce single perceptual hashing (such as pHash, a perceptual hash value generated based on the visual features of an image, which can be used to identify similar images). Although it can identify similar images, it does not combine the national cryptographic algorithms required by the confidential assessment scenario, and cannot guarantee the encryption security of evidence data during the deduplication process. Furthermore, it lacks precise byte-level verification, which can easily misjudge "similar screenshots of interfaces from different cryptographic devices" as duplicate images, thereby affecting the accuracy of the confidential assessment data. ② Compliance Verification Technical Issues: The compliance assessment of existing confidential review evidence images mainly relies on manual review one by one, checking for issues such as whether the images have been tampered with by Photoshop, whether they meet the confidential review specifications, and whether they are clear and identifiable. A few automated solutions can only read EXIF ​​metadata to determine the shooting time and cannot identify metadata tampering. Furthermore, the detection of Photoshop tampering requires separate analysis by specialized software and cannot be integrated into the confidential review operation system for batch processing. The clarity assessment is based solely on pixel threshold division, without considering the core requirement of "identifiable key information" in confidential reviews, which leads to non-compliant images being included in subsequent evaluation stages, thus affecting the validity of the confidential review results.

[0003] It is known that the defects and causes of the deduplication technology in the existing confidential assessment system are as follows: Traditional MD5 hash deduplication relies on complete matching of file bytes, which cannot cope with the slight visual changes in confidential assessment evidence images. The root cause is that it does not associate the visual features of the image and only calculates the file encoding. Single-perception hash deduplication lacks byte-level verification and does not integrate national cryptographic algorithms, which is prone to misjudging similar but different images in confidential assessment scenarios. At the same time, it does not meet the compliance requirements of confidential assessment for data encryption. The root cause is that it has not established a triple adaptation mechanism of "visual features + byte features + national cryptographic security". The deficiencies and causes of compliance verification are as follows: manual review is extremely inefficient, and fatigue-induced misjudgments are prone to occur when faced with numerous evidence images in confidential review projects. The root cause lies in the lack of an automated verification process adapted to confidential review standards. Existing automated solutions only cover a single compliance dimension and do not integrate the core dimensions of "tamper detection, standard matching, and clarity." Furthermore, they are not linked to the deduplication process, resulting in separate verification still required after deduplication, leading to process redundancy. Tamper detection does not take into account the special characteristics of confidential review images and cannot identify targeted tampering such as "replacing device model identifiers or modifying encryption status display." The root cause lies in the lack of a multi-dimensional tamper identification model based on the characteristics of confidential review scenarios. In addition, the existing confidentiality assessment system also has defects in its overall process. The main reasons are: its deduplication and compliance verification are completely separated and need to be operated separately outside the confidentiality assessment system; the processed data is not encrypted and stored using national cryptographic algorithms, which does not meet the assessment specifications; there is no dedicated processing mechanism for confidentiality assessment scenarios for images that fail to pass compliance, which makes it impossible for staff to quickly locate problematic images, affecting the progress of confidentiality assessment projects; and a dedicated integrated process for confidentiality assessment, namely "deduplication-compliance verification-tiered processing-national cryptographic encryption", has not been built.

[0004] Therefore, given the problems existing in the current technology, it is of great importance to provide a deduplication and compliance verification technology for confidential evidence images that has high deduplication accuracy, strong compliance verification capability, and high efficiency in processing confidential evidence. Summary of the Invention

[0005] The purpose of this invention is to overcome the shortcomings of existing technologies and provide a method and system for deduplication and compliance verification of confidential review evidence images. It establishes a dual-fingerprint deduplication mechanism combining perceptual hashing and the national cryptographic standard SM3, achieving accurate identification of similar images and ensuring compliance with national cryptographic standards for confidential review scenarios. Furthermore, by constructing an automated compliance verification process that includes tamper detection, confidential review standard matching, and key information clarity verification, it links with the deduplication process to achieve fully automated processing of confidential review evidence images. Additionally, it achieves storage compression by retaining the original image and replacing it with reference information, establishing a tiered handling mechanism for compliance issues adapted to confidential review scenarios.

[0006] To achieve the above objectives, the present invention adopts the following technical solution:

[0007] A method for deduplicating and verifying the compliance of confidential evidence images, the method comprising the following steps:

[0008] Step S1. Acquisition and Preprocessing of Confidential Assessment Evidence Images: Acquire evidence images from the confidential assessment system, remove abnormal features from the evidence images to form preprocessed confidential assessment evidence images, and save the original confidential assessment evidence images; specifically including:

[0009] S1-1: Obtain evidence image format data from the confidential assessment system, read the EXIF ​​metadata of the images, and store them in the confidential assessment-specific temporary database, associating them with the corresponding confidential assessment project number and assessment node information;

[0010] S1-2: Automatically remove abnormal features from the evidence images to form preprocessed confidential evaluation evidence images, and save the original confidential evaluation evidence image files during the preprocessing process; the preprocessed confidential evaluation evidence images are used for subsequent feature extraction;

[0011] Step S2. Dual fingerprint deduplication:

[0012] S2-1: Obtain the preprocessed secret evaluation evidence image and perform grayscale processing. Set the image size to the target pixel value, calculate the discrete cosine transform of the grayscale image, extract the low-frequency coefficients in the upper left corner of the discrete cosine transform coefficient matrix, and calculate the average value of the extracted low-frequency coefficient matrix. Compare each coefficient with the average value. If the comparison result is greater than the average value, mark it as 1; if it is less than the average value, mark it as 0. Generate a 64-bit binary perceptual hash value pHash.

[0013] S2-2: Obtain the original confidential assessment evidence image, perform SM3 hash value calculation on its byte stream to generate SM3 digest data (e.g., 256-bit SM3 digest data), and use it as the precise fingerprint of the image to verify the integrity of the confidential assessment data, which meets the requirements of the assessment specification for data integrity;

[0014] S2-3: Obtain the pHash value and SM3 summary data, combine them to form deduplication information (the format can be "Secret Assessment Project Number-Assessment Node-pHash Value:SM3 Summary"), and write it into the memory Set collection;

[0015] S2-4: Retrieve the newly imported image, calculate its pHash value, and then calculate the Hamming distance between this pHash value and the existing pHash values ​​of the same item and node in the memory Set; the determination method is as follows:

[0016] If the Hamming distance is less than or equal to the preset distance value d, then the SM3 summary data of the two are compared; if the SM3 summary values ​​of the two are the same, then it is determined to be a duplicate image, the newly imported image is removed, and the first imported confidential assessment evidence image is retained; if the SM3 summary data of the two are different, then it is determined to be a similar image, the newly imported image is removed, and the reference information (the format can be "original image SM3 summary: similar image pHash value - confidential assessment item number") is used for replacement processing, a reference mapping table is established, and the corresponding confidential assessment item is associated;

[0017] If the Hamming distance is greater than the preset distance value d, it is determined to be a different graph, and its deduplication information is written to the memory set, while the original graph is retained;

[0018] Step S3. Multi-dimensional compliance verification processing: Obtain the original image retained after deduplication and perform automated compliance verification processing, including:

[0019] S3-1: Perform PS tampering detection, extract the EXIF ​​metadata of the image, and determine its consistency with the information of the confidential assessment project; if the shooting time shown in the EXIF ​​(Exchangeable Image File Format) metadata is inconsistent with the corresponding confidential assessment project evaluation time, or the shooting device model shown in the EXIF ​​metadata does not match the device model in the evaluation record of the corresponding confidential assessment project, it is determined that the metadata has been tampered with; perform file format compression trace analysis on the image. If the image as a whole meets the format requirements, but the size of the compressed block and the quantization matrix of the key local areas of the image are inconsistent with the whole, it is determined that it has been tampered with;

[0020] S3-2: Perform key information clarity verification. Combine the PSNR (Peak Signal-to-Noise Ratio) value with the gradient value of the key information region in the image. Set a PSNR value ≥ NdB as the basic threshold for initial screening of the overall image quality, where N > 0. After screening the overall image quality, calculate the gradient value of the key information region in the image. If the gradient value of the key information region is < the preset basic threshold, the image is determined to be blurry. Then, determine whether the blurry image meets the requirements for retaining the key information. If the key information of the blurry image can be clearly identified, it is marked as meeting the requirements for retaining the key information. If the key information of the blurry image cannot be identified, it is marked as not meeting the requirements for retaining the key information.

[0021] Step S4. Compliance Issue Classification and Processing: Obtain compliance verification results, classify images, and perform corresponding confidentiality assessment procedures on the classified images; including:

[0022] S4-1: Based on the obtained compliance verification results, images are classified into compliant images, images to be supplemented, and non-compliant images. Among them, compliant images are those that pass the tampering detection, have a confidentiality assessment standard matching degree of ≥90%, or meet the standard for key information clarity; images to be supplemented are those with a confidentiality assessment standard matching degree of 80%~89% or slightly blurry images with identifiable key information; non-compliant images are those confirmed to have been tampered with, have unidentifiable key information, or have a confidentiality assessment standard matching degree of <80%.

[0023] S4-2: Based on the type of the categorized images, perform the following detailed review process:

[0024] If the image is compliant, it will be directly associated with the confidential assessment project number, enter the encrypted storage stage, generate compliance verification record data, and be included in the confidential assessment project file.

[0025] If the image is to be supplemented, the supplementation process will be automatically triggered to generate a time watermark that conforms to the confidentiality assessment specifications, sharpen the image, and then re-perform the compliance verification process in step S3. If the verification is passed, the image will be associated with the confidentiality assessment project as a compliant image and enter encrypted storage; if the verification is not passed, the image will be converted into an unqualified image for processing.

[0026] If the image is deemed unqualified, non-compliant record data for confidential assessment evidence images will be automatically generated, the data will be pushed to the confidential assessment operation system, and the image will be isolated and stored, prohibited from being associated with the confidential assessment project file, pending review.

[0027] The above-mentioned deduplication and compliance verification method further includes: Step S5. SM4 encryption storage processing:

[0028] S5-1: Integrate the reference mapping table corresponding to the compliant images, the reference mapping table of the images to be supplemented, and the compliance verification record data to generate a confidential assessment evidence package;

[0029] S5-2: Read the secret evaluation evidence package and encrypt it using the National Secret Algorithm 4 (a block cipher algorithm issued by the State Cryptography Administration).

[0030] Specifically, step S5 uses the national cryptographic algorithm SM4 to encrypt and package the processed data, ensuring the security and validity of the encrypted assessment data.

[0031] Specifically, the above technical solution includes five core processing flows, enabling full automation of the entire process from importing evidence images for confidential assessment to deduplication, compliance verification, hierarchical processing, and national cryptographic encryption storage. It adopts a dual-fingerprint deduplication method specifically for confidential assessment, deeply integrating it with national cryptographic standards. By combining the visual feature recognition of perceptual hash (pHash) with the byte-level precise verification of national cryptographic standard SM3, and incorporating the confidential assessment project number and assessment node information into the deduplication information, a four-fold adaptation mechanism of "project association, visual similarity, byte uniqueness, and national cryptographic security" is formed. This solves the problems of missed detection by traditional MD5 and misjudgment by single perceptual hash. At the same time, the application of SM3 and SM4 national cryptographic algorithms fully complies with the encryption compliance requirements of confidential assessment scenarios. Regarding the hierarchical processing mechanism adapted to the confidential assessment process, a hierarchical system of "compliant - requiring supplementation - non-compliant" is established. For "images requiring supplementation", a dedicated supplementation program for confidential assessment is automatically triggered to avoid repetitive manual operations. For "non-compliant images", structured confidential assessment-specific record data is generated to visualize the problem and fill the gap in existing technologies that "only judge non-compliance without providing a solution for confidential assessment scenarios". Furthermore, a full-process national cryptographic encryption and permission binding method is adopted. From the SM3 fingerprint generation in the deduplication stage to the SM4 evidence package encryption in the storage stage, national cryptographic algorithms are used, and the key is bound to the identity and permissions of the assessment personnel. At the same time, traceable encrypted logs are generated and included in the confidential assessment project archive, forming a full-process national cryptographic security guarantee of "deduplication-verification-storage", which fully complies with relevant national confidential assessment standards.

[0032] In step S1-1, the evidence image format data includes screenshots of cryptographic device deployment, encrypted data transmission record images, and scanned images of evaluation reports.

[0033] Specifically, screenshots of cryptographic device deployment, encrypted data transmission records, and scanned images of evaluation reports are common evidence image formats in cryptographic evaluation systems, and they can be imported into the cryptographic evaluation system in batches.

[0034] In the above-mentioned steps S1-2, the abnormal features include black edges on the image edges caused by scanning or shooting, or tilting caused by shooting.

[0035] Specifically, the imported images are preprocessed to automatically remove abnormal features, including black edges caused by scanning or shooting, and correction of tilt caused by hand-held shooting. The original image files are preserved during the preprocessing process, and the preprocessed images are used for subsequent feature extraction, which can ensure that the originality of the confidential evidence is not affected.

[0036] In step S2, the deduplication information includes the dense evaluation project number, evaluation node, perceptual hash value (pHash), and SM3 summary; the reference information includes the SM3 summary of the original image, the pHash value of similar images, and the dense evaluation project number.

[0037] Specifically, in step S2, the method of "preserving the original image + replacing with reference information" can reduce the repeated storage of evidence images in confidential review. According to tests, it can save 35% to 75% of storage space for common evidence screenshots in confidential review projects.

[0038] In step S3-2, the key information area of ​​the confidential assessment includes the device model, algorithm parameters, IP address, or report number of the image.

[0039] In step S4-2, the non-compliance record data includes the non-compliance dimension, reason, screenshot annotation of the problem area, and associated confidential review node.

[0040] In step S5-1, the confidential assessment evidence package includes retained original image set data, reference mapping table, compliance verification record data, and non-compliance record data, and each confidential assessment evidence package is configured to be associated with a unique confidential assessment project number and version number.

[0041] In step S5-2 above, during encryption, the encryption evaluation system dynamically generates a key, and the key is configured to be bound to the identity and permissions of the evaluator; the encrypted data is stored in a dedicated server, and an encrypted log is generated; the encrypted log generates a digest using the SM3 algorithm and stores it in the encryption evaluation project file.

[0042] The present invention also provides a system for deduplication and compliance verification of confidential evidence images, which applies the aforementioned method for deduplication and compliance verification of confidential evidence images.

[0043] The beneficial effects of this invention are:

[0044] This invention provides a method for deduplication and compliance verification of confidential assessment evidence images. It establishes a dual-fingerprint deduplication mechanism combining perceptual hashing and the national cryptographic standard SM3, achieving accurate identification and verification of similar images while complying with the national cryptographic standard's encryption compliance requirements for confidential assessment scenarios. This solves the problems of traditional deduplication techniques missing subtle visual changes in confidential assessment evidence images and misjudging by single perceptual hashing. Furthermore, by constructing an automated compliance verification process that includes tamper detection, confidential assessment standard matching, and key information clarity verification, it links with the deduplication process to achieve fully automated processing of confidential assessment evidence images. This addresses the problems of existing compliance verification methods relying on manual intervention, having limited dimensions, and being unable to be integrated into batch processing workflows of confidential assessment systems. Additionally, it achieves storage compression by retaining the original image and replacing it with reference information, establishing a hierarchical processing mechanism for compliance issues adapted to confidential assessment scenarios. Simultaneously, it uses the national cryptographic standard SM4 algorithm to encrypt and package the processed data, ensuring the security and validity of confidential assessment data. It boasts advantages such as high deduplication accuracy, compliance verification tailored to confidential assessment needs, high processing efficiency, strong security and compliance, low storage cost, and clear data correlation. Attached Figure Description

[0045] Figure 1 This is a flowchart of a method for deduplicating and verifying the compliance of confidential evidence images provided in an embodiment of the present invention. Detailed Implementation

[0046] The specific embodiments of the present invention will be further described below with reference to the accompanying drawings.

[0047] like Figure 1 As shown, this embodiment provides a method for deduplication and compliance verification of confidential evidence images. The deduplication and compliance verification method includes the following steps:

[0048] Step S1. Acquisition and preprocessing of confidential evidence images:

[0049] S1-1: Obtain evidence image format data (including screenshots of cryptographic device deployment, encrypted data transmission record images, scanned images of evaluation reports, etc.) from the confidential evaluation system, read the EXIF ​​(Exchangeable Image File Form) metadata of the images, and store them in the confidential evaluation-specific temporary database, associating them with the corresponding confidential evaluation project number and evaluation node information.

[0050] S1-2: Automatically remove abnormal features from the evidence image (such as black edges caused by scanning or shooting, and correct tilt caused by hand-held shooting) to form a pre-processed confidential evaluation evidence image, and save the original confidential evaluation evidence image file during the pre-processing process; the pre-processed confidential evaluation evidence image is used for subsequent feature extraction to ensure that the originality of the confidential evaluation evidence is not affected.

[0051] Step S2. Dual fingerprint deduplication:

[0052] S2-1: Perceptual Hash (pHash) Calculation: Obtain the pre-processed confidential evidence image and perform grayscale processing. Set the image size to the target pixel value (in this embodiment, the target pixel value is set to 32×32 pixels to unify the size and eliminate the impact of resolution, while retaining key visual features such as device interface and parameter text). Calculate the Discrete Cosine Transform (DCT) of the grayscale image. Extract the 8×8 low-frequency coefficients in the upper left corner of the Discrete Cosine Transform (DCT) coefficient matrix (these low-frequency coefficients can reflect the overall visual features of the image and are not affected by local watermarks or small angle changes). Calculate the average value of the extracted 8×8 low-frequency coefficient matrix. Compare each coefficient with the average value. If the comparison result is greater than the average value, mark it as 1; if it is less than the average value, mark it as 0. Generate a 64-bit binary perceptual hash value pHash.

[0053] S2-2: SM3 Precise Fingerprint Calculation: Obtain the original confidential assessment evidence image, perform SM3 hash value calculation on its byte stream to generate 256-bit SM3 digest data, and use it as the precise fingerprint of the image to verify the integrity of the confidential assessment data, which meets the requirements of the assessment specification for data integrity;

[0054] S2-3: Obtain the pHash value and SM3 summary data, combine them to form deduplication information (the format can be "Secret Assessment Project Number-Assessment Node-pHash Value:SM3 Summary"), and write it into the memory Set collection;

[0055] S2-4: Retrieve the newly imported image, calculate its pHash value, and then calculate the Hamming distance between this pHash value and the existing pHash values ​​of the same item and node in the memory Set; the determination method is as follows:

[0056] If the Hamming distance is less than or equal to the preset distance value d=5, then the SM3 summary data of the two images are compared. If the SM3 summary values ​​of the two images are the same, they are determined to be duplicate images, the newly imported images are removed, and the first imported confidential assessment evidence images are retained. If the SM3 summary data of the two images are different, they are determined to be similar images, the newly imported images are removed, and the reference information (the format can be "original image SM3 summary: similar image pHash value - confidential assessment item number") is used for replacement processing, a reference mapping table is established, and the corresponding confidential assessment items are associated.

[0057] If the Hamming distance is greater than the preset distance value d, that is, Hamming distance > 5, then it is determined to be a different graph, its deduplication information is written to the memory Set, and the original graph is retained;

[0058] In this embodiment, the determination method can achieve storage compression. By "retaining the original image + replacing it with reference information", the duplicate storage of evidence images in confidential evaluation is reduced. According to tests, it can save 35% to 75% of storage space for common evidence screenshots in confidential evaluation projects.

[0059] Step S3. Multi-dimensional compliance verification processing: Obtain the original image retained after deduplication, and perform automated compliance verification processing from two core dimensions (i.e., PS tampering detection and key information clarity verification), including:

[0060] S3-1: Perform PS tampering detection, extract the EXIF ​​metadata of the image, and determine its consistency with the information of the confidential evaluation project; if the shooting time shown in the EXIF ​​metadata is inconsistent with the evaluation time of the corresponding confidential evaluation project, or the shooting device model shown in the EXIF ​​metadata does not match the evaluation record device model of the corresponding confidential evaluation project, it is determined that the metadata has been tampered with; this is the first level of detection. In this embodiment, when extracting the EXIF ​​metadata of the image and judging its consistency with the information of the confidential evaluation project, if the EXIF ​​shows the shooting time as "2025-09-01 10:00", but the evaluation time of the corresponding confidential evaluation project is "2025-09-02", or the shooting device model does not match the evaluation record device model, it is determined that the metadata has been tampered with;

[0061] The image is analyzed for file format compression traces. If the image as a whole meets the format requirements, but the size of the compressed blocks and the quantization matrix of the local key areas of the image are inconsistent with the whole, it is determined to be tampered with. This is the second level of detection. In this embodiment, the image format is JPEG, so the image is analyzed for JPEG compression traces. If the image as a whole is in JPEG format, but the size of the compressed blocks and the quantization matrix of the local key areas are inconsistent with the whole (such as traces of recompression after tampering), it is determined to be tampered with.

[0062] S3-2: Perform a critical information clarity check and set the PSNR (Peak Signal-to-Noise Ratio) value. The peak signal-to-noise ratio (PSNR) is combined with the gradient value of the key information region in the confidential assessment. A PSNR value ≥ NdB is set as the basic threshold for initial screening of the overall image quality. In this embodiment, N=32, that is, a PSNR value ≥ 32dB (higher than in general scenarios, ensuring clear details). After screening the overall image quality, the gradient value of the key information region in the confidential assessment (such as device model, algorithm parameters, IP address, report number) is calculated. If the gradient value of the key information region in the confidential assessment is < the preset basic threshold (in this embodiment, it is set according to the type of confidential assessment information: gradient value of device model text region ≥ 25, gradient value of IP address numeric region ≥ 22), then the image is determined to be a blurry image. Then, it is determined whether the blurry image meets the confidential assessment retention requirements. If the key information of the blurry image can be clearly identified, it is marked as meeting the confidential assessment retention requirements; if the key information of the blurry image cannot be identified, it is marked as not meeting the confidential assessment retention requirements (that is, if the blur only affects the non-critical areas of the image and the key information can still be clearly identified, it is marked as meeting the confidential assessment retention requirements; if the blur makes the key information unidentifiable, it is marked as not meeting the confidential assessment retention requirements).

[0063] Step S4. Compliance Issue Classification and Processing: Obtain compliance verification results, classify images, and perform corresponding confidentiality assessment procedures on the classified images; including:

[0064] S4-1: Based on the obtained compliance verification results, images are classified into compliant images, images to be supplemented, and non-compliant images. Among them, compliant images are those that pass the tampering detection, have a confidentiality assessment standard matching degree of ≥90%, or meet the standard for key information clarity; images to be supplemented are those with a confidentiality assessment standard matching degree of 80%~89% or slightly blurry images with identifiable key information; non-compliant images are those confirmed to have been tampered with, have unidentifiable key information, or have a confidentiality assessment standard matching degree of <80%.

[0065] S4-2: Based on the type of the categorized images, perform the following detailed review process:

[0066] If the image is compliant, it will be directly associated with the confidential assessment project number, enter the encrypted storage stage, generate compliance verification record data, and be included in the confidential assessment project file.

[0067] If the image is to be supplemented, the supplementation process will be automatically triggered to generate a time watermark that conforms to the confidentiality assessment specifications, sharpen the image, and then re-perform the compliance verification process in step S3. If the verification is passed, the image will be associated with the confidentiality assessment project as a compliant image and enter encrypted storage; if the verification is not passed, the image will be converted into an unqualified image for processing.

[0068] If the image is deemed unqualified, non-compliance record data for the confidential assessment evidence image will be automatically generated (including non-compliance dimensions, specific reasons, screenshot annotations of the problematic area, and associated confidential assessment nodes, etc.). The data will be pushed to the review interface of the confidential assessment system, and the image will be isolated and stored (marked "Pending Confirmation - Non-compliant"). It will be prohibited from being associated with the confidential assessment project file and will await manual review by the assessment personnel to decide whether to supplement evidence or remove it.

[0069] Step S5. SM4 encryption storage processing:

[0070] S5-1: Integrate the reference mapping table corresponding to compliant images, the reference mapping table of images to be supplemented, and the compliance verification record data to generate a confidential assessment evidence package (including "the original image set + reference mapping table + compliance verification record + non-compliant record").

[0071] S5-2: Read the confidential assessment evidence package and encrypt it using the National Secret Algorithm 4 (SM4, a block cipher algorithm issued by the State Cryptography Administration). The confidential assessment system dynamically generates a key, which is bound to the identity and permissions of the assessment personnel. The encrypted data is stored in the confidential assessment dedicated security server. At the same time, an encrypted log is generated. The log is digested using the SM3 algorithm and written into the confidential assessment project file to ensure data security and traceability, fully meeting the requirements for data storage.

[0072] In this embodiment, the solution includes five core processing flows, enabling full automation of the entire process from importing evidence images for confidential assessment to deduplication, compliance verification, hierarchical processing, and national cryptographic encryption storage. It employs a dual-fingerprint deduplication method specifically for confidential assessment, deeply integrating it with national cryptographic standards. By combining visual feature recognition of perceptual hash (pHash) with byte-level precise verification of national cryptographic standard SM3, and incorporating the confidential assessment project number and assessment node information into the deduplication information, a four-fold adaptation mechanism of "project association, visual similarity, byte uniqueness, and national cryptographic security" is formed. This solves the problems of missed detection by traditional MD5 and misjudgment by single perceptual hash. At the same time, the application of SM3 and SM4 national cryptographic algorithms fully complies with the encryption compliance requirements of confidential assessment scenarios. Regarding the hierarchical processing mechanism adapted to the confidential assessment process, a hierarchical system of "compliant - requiring supplementation - non-compliant" is established. For "images requiring supplementation", a dedicated supplementation program for confidential assessment is automatically triggered to avoid repetitive manual operations. For "non-compliant images", structured confidential assessment-specific record data is generated to visualize the problem and fill the gap in existing technologies that "only judge non-compliance without providing a solution for confidential assessment scenarios". Furthermore, a full-process national cryptographic encryption and permission binding method is adopted. From the SM3 fingerprint generation in the deduplication stage to the SM4 evidence package encryption in the storage stage, national cryptographic algorithms are used, and the key is bound to the identity and permissions of the assessment personnel. At the same time, traceable encrypted logs are generated and included in the confidential assessment project archive, forming a full-process national cryptographic security guarantee of "deduplication-verification-storage", which fully complies with relevant national confidential assessment standards.

[0073] This technical solution has many advantages: ① High deduplication accuracy: For similar images commonly found in confidential evaluation scenarios, such as those with "rotation of 1°~8°, addition of evaluation watermarks, and resolution fine-tuning", the recognition accuracy is ≥99%, which is more than 65% higher than the traditional MD5 deduplication accuracy and 55% lower than the single pHash deduplication false judgment rate. It can also accurately distinguish "similar interface screenshots from different devices". Its deduplication effect is as follows: In confidential evaluation project scenarios, it can accurately identify similar evidence images that have undergone common processing such as slight angle rotation, addition of evaluation watermarks, and resolution fine-tuning. It effectively avoids the limitation of the traditional MD5 algorithm, which can only identify completely identical images. At the same time, it significantly reduces the false judgment of the single perceptual hash algorithm on "similar interface screenshots from different cryptographic devices", ensuring the accuracy of the deduplication process of confidential evaluation evidence images and reducing the problem of evidence redundancy and loss of key information caused by duplication or false judgment.

[0074] ② Compliance verification aligns with confidentiality assessment requirements: Covering three core dimensions—tampering, confidentiality assessment standards, and key information clarity—it achieves higher accuracy in tampering detection and consistency in key information clarity assessment with manual review of confidentiality assessments. This significantly improves the coverage compared to existing single-dimensional verification methods and fully complies with confidentiality assessment standards. Its compliance verification effect is as follows: It can achieve multi-dimensional automated verification for potential issues in confidentiality assessment evidence images, such as tampering, non-compliance with standards, and unclear key information. It can effectively identify targeted tampering behavior, accurately determine whether images meet the characteristics required by confidentiality assessment standards, and accurately distinguish whether key information is clear and identifiable. Simultaneously, it can automatically process images that do not meet the requirements but can be supplemented / repaired, significantly reducing the workload and probability of misjudgment in manual review and ensuring that all evidence images flowing into subsequent assessment stages meet the validity requirements of confidentiality assessments.

[0075] ③ High efficiency in handling confidential assessments: Batch processing of confidential assessment evidence images improves efficiency compared to the "manual review + traditional deduplication" confidential assessment process, significantly shortening the confidential assessment project cycle. The optimization effect of the confidential assessment process is as follows: The multi-step and scattered process of "deduplication - manual review - supplementation and repair - secondary review - storage" in the traditional confidential assessment evidence image processing is integrated into a fully automated integrated processing process. Only a small number of non-compliant images need to be manually reviewed, which significantly reduces the repetitive operations of assessment personnel, improves the overall efficiency of confidential assessment evidence processing, shortens the overall cycle of confidential assessment projects, and optimizes the smoothness of the confidential assessment operation process.

[0076] ④ Strong security and compliance with confidential assessment standards, low storage costs, and clear data association: The entire process utilizes national cryptographic algorithms SM3 and SM4, with keys bound to identity permissions. Encrypted logs are included in the confidential assessment archive, meeting evaluation specifications. In simulated confidential assessment data security testing, no data leakage occurred, satisfying the high security requirements of confidential assessment scenarios. Storage compression is achieved through "reference information substitution," while "deduplicated information and evidence packages" are all associated with the confidential assessment project number, avoiding data confusion. This reduces storage costs compared to existing solutions and facilitates subsequent traceability and review of confidential assessment projects. Its security and storage effects are as follows: Using national cryptographic algorithms for data processing and storage encryption meets the data security requirements of the evaluation specifications, effectively resisting external attacks and data leakage risks, ensuring the security of confidential assessment evidence data, and reducing storage costs.

[0077] Based on the disclosure and teachings of the foregoing specification, those skilled in the art can make changes and modifications to the above embodiments. Therefore, the present invention is not limited to the specific embodiments disclosed and described above, and some modifications and changes to the invention should also fall within the protection scope of the claims of the present invention. Furthermore, although some specific terms are used in this specification, these terms are only for convenience of explanation and do not constitute any limitation on the present invention.

Claims

1. A method for deduplication and compliance verification of secret evidence pictures, characterized in that, The deduplication and compliance verification method comprises: Step S1. Obtain and preprocess the evidence pictures in the secret evaluation system, remove the abnormal features of the evidence pictures, form the preprocessed secret evaluation evidence pictures, and save the original secret evaluation evidence pictures; Step S2. Double-fingerprint deduplication processing: S2-1: Obtain the preprocessed secret evaluation evidence pictures and perform grayscale processing, set the picture size to the target pixel value, calculate the discrete cosine transform of the grayscale processed picture, extract the low frequency coefficient matrix in the upper left corner of the discrete cosine transform coefficient matrix, and calculate the average value of the extracted low frequency coefficient matrix, compare each coefficient with the average value, if the comparison result is greater than the average value, mark it as 1, and less than the average value, mark it as 0, generate a perceptual hash value pHash; S2-2: Obtain the original secret evaluation evidence pictures, calculate the SM3 hash value of the byte stream, generate the SM3 digest data, and use it as the accurate fingerprint of the picture to verify the integrity of the secret evaluation data; S2-3: Obtain the perceptual hash value pHash and the SM3 digest data, combine them to form the deduplication information, and write it into the memory Set collection; S2-4: Obtain the newly imported picture, calculate its pHash value, and then calculate the Hamming distance between the pHash value and the stored pHash value of the same project and node in the memory Set; The determination method is as follows: If the Hamming distance is less than or equal to the preset distance value d, compare the SM3 digest data of the two; If the SM3 digest data of the two is the same, it is determined that the picture is repeated, the newly imported picture is removed, and the first imported secret evaluation evidence picture is retained; If the SM3 digest data of the two is different, it is determined that the picture is similar, the newly imported picture is removed, and the reference information is used for replacement processing, a reference mapping table is established, and the corresponding secret evaluation project is associated; If the Hamming distance is greater than the preset distance value d, it is determined that the picture is different, the deduplication information is written into the memory Set, and the original picture is retained; Step S3. Multi-dimensional compliance verification processing: Obtain the original picture retained after deduplication processing, and perform automatic compliance verification processing; Multi-dimensional specific refers to PS tampering detection and key information clarity verification in two dimensions; Step S4. Compliance problem grading processing: Obtain the compliance verification result, classify the pictures, and execute the corresponding secret evaluation process on the classified pictures.

2. The deduplication and compliance verification method of claim 1, wherein, The step S1 secret evaluation evidence picture acquisition and preprocessing comprises: S1-1: Obtain the evidence picture format data in the secret evaluation system, read the EXIF metadata of the picture, and store it in the secret evaluation exclusive temporary database, associate the corresponding secret evaluation project number and evaluation node information; The EXIF metadata is the Exchangeable Image File Format metadata; S1-2: Automatically remove the abnormal features of the evidence pictures, form the preprocessed secret evaluation evidence pictures, and save the original secret evaluation evidence picture file during preprocessing.

3. The deduplication and compliance verification method of claim 1, wherein, The step S3 multi-dimensional compliance verification processing comprises: S3-1: Perform PS tampering detection, extract the EXIF metadata of the picture, and judge its consistency with the evaluation project information; if the shooting time displayed by the EXIF metadata is inconsistent with the corresponding evaluation project evaluation time, or the shooting device model displayed by the EXIF metadata is inconsistent with the evaluation record device model of the corresponding evaluation project, it is determined that the metadata is tampered; perform file format compression trace analysis on the picture; if the overall picture meets the format requirements, but the compression block size and quantization matrix of the local key area of the picture are inconsistent with the overall picture, it is determined that the picture is tampered; S3-2: Perform key information clarity verification, combine the PSNR value with the gradient value of the evaluation key information area, set the PSNR value ≥ NdB as the basic threshold value for preliminary screening of the overall image quality of the picture, wherein N > 0, and then calculate the gradient value of the evaluation key information area in the picture after screening the overall image quality of the picture; if the gradient value of the evaluation key information area is less than the preset basic threshold value, the picture is determined to be a blurred picture; then determine whether the blurred picture meets the evaluation retention requirements; if the key information of the blurred picture can be clearly identified, it is marked as meeting the evaluation retention requirements; if the key information of the blurred picture cannot be identified, it is marked as not meeting the evaluation retention requirements.

4. The deduplication and compliance verification method of claim 1, wherein, The step S4 compliance problem grading processing includes: S4-1: According to the obtained compliance verification result, the picture is classified into a compliant picture, a to-be-supplemented picture and a non-compliant picture; S4-2: According to the type of the classified picture, the picture is processed according to the following evaluation work process: If the picture is a compliant picture, it is directly associated with the evaluation project number, enters the encrypted storage link, and generates a compliance verification record data, which is included in the evaluation project archive; If the picture is a to-be-supplemented picture, a supplement program is automatically triggered, a time watermark meeting the evaluation specification is generated, the picture is sharpened, and then the picture after processing is subjected to the compliance verification processing of step S3; if the verification is passed, the picture is regarded as a compliant picture associated with the evaluation project and enters the encrypted storage; if the verification is not passed, the picture is converted into a non-compliant picture for processing; If the picture is a non-compliant picture, a non-compliant record data of the evaluation evidence picture is automatically generated, the data is pushed to the evaluation work system, and the picture is stored in isolation, which is prohibited to be associated with the evaluation project archive and waits for review.

5. The deduplication and compliance verification method of claim 1, wherein, In the step S2, the deduplication information includes the evaluation project number, the evaluation node, the perceptual hash value pHash and the SM3 digest; and the reference information includes the SM3 digest of the original picture, the pHash value of the similar picture and the evaluation project number.

6. The deduplication and compliance verification method of claim 3, wherein, In the step S3-2, the evaluation key information area includes the device model, the algorithm parameter, the IP address or the report number of the picture.

7. The deduplication and compliance verification method of claim 4, wherein, The deduplication and compliance verification method further includes a step S5. The national secret SM4 encrypted storage processing: S5-1: Integrate the reference mapping table corresponding to the compliant picture, the reference mapping table of the to-be-supplemented picture and the compliance verification record data to generate an evaluation evidence package; S5-2: Read the evaluation evidence package and encrypt it using the national secret SM4 algorithm.

8. The deduplication and compliance checking method of claim 7, wherein, In the step S5-1, the secret evaluation evidence package includes the reserved original map set data, the reference mapping table, the compliance verification record data and the non-compliance record data, and each secret evaluation evidence package is configured to be associated with a unique secret evaluation project number and version number.

9. The deduplication and compliance checking method of claim 7, wherein, In step S5-2, when encryption is performed, a secret key is dynamically generated by the secret evaluation system, and the secret key is configured to be bound to the identity permission of the evaluator; the encrypted data is stored in a dedicated server, and an encryption log is generated; the encryption log generates a digest through an SM3 algorithm and is stored in a secret evaluation project archive.

Images