PDF document watermark recognition method and system

Abstract

This invention discloses a method for PDF document watermark recognition, comprising obtaining an existing PDF document and extracting text; preprocessing the text to obtain corpus data; segmenting the corpus data and establishing a segmentation model; training the segmentation model; and processing a target PDF document using the trained segmentation model to complete text watermark recognition. The invention also discloses a system for implementing the PDF document watermark recognition method. The method utilizes a method combining natural language processing technology with bitmap recognition, significantly improving recognition accuracy and providing high reliability, accuracy, speed, and security.

Description

Technical Field

[0001] The present invention belongs to the field of computer technology, and in particular relates to a PDF document watermark recognition method and system. Background Art

[0002] With the development of economy and technology and the improvement of people's living standards, the power system has entered the digital age.

[0003] While numerous researchers at home and abroad have conducted extensive research on image, audio, and video protection, relatively little research has been conducted on digital watermarking for PDF documents. Digital watermarking in PDF format has been widely used in various intranets, such as e-government networks and corporate information networks, to protect confidential documents and files.

[0004] At present, the existing methods for identifying watermarks in PDF documents mostly use statistical and image processing techniques. Although the watermark recognition method based on statistics has a faster recognition rate, this type of method mainly focuses on the frequency and location of word occurrence, and generally has problems such as low accuracy and low robustness. Although the recognition method based on image processing is suitable for many non-text watermark scenarios, this type of method has high requirements on the permissions of the read file and has a relatively poor recognition rate. Summary of the Invention

[0005] The purpose of the present invention is to provide a PDF document watermark recognition method that is highly reliable, accurate, fast and safe.

[0006] A second object of the present invention is to provide a system for implementing the PDF document watermark recognition method.

[0007] The PDF document watermark recognition method provided by the present invention comprises the following steps:

[0008] S1. Obtain an existing PDF document and perform text extraction;

[0009] S2. Preprocess the text extracted in step S1 to obtain corpus data;

[0010] S3. Segment the corpus data obtained in step S2 and establish a segmentation model based on the N-gram model;

[0011] S4. Train the word segmentation model constructed in step S3;

[0012] S5. Use the trained word segmentation model to process the target PDF and complete the recognition of text watermarks.

[0013] The step S1 of obtaining an existing PDF document and extracting text specifically includes the following steps:

[0014] Get PDF files with and without watermarks, and extract the text content of the PDF files; at the same time, extract the content and corresponding location of the bitmap elements in the PDF files.

[0015] The pre-processing described in step S2 specifically includes the following steps:

[0016] Delete punctuation marks and special characters from the text content, and process the text to obtain a string set.

[0017] Step S3, in which the corpus data obtained in step S2 is segmented and a segmentation model is established based on the N-gram model, specifically includes the following steps:

[0018] Using the standard dictionary as the word segmentation dictionary, based on the forward matching algorithm and the reverse matching algorithm, the N-gram model is established to convert the string S=(w1,w2,...,w x ) are shortened in sequence and matched with the word segmentation dictionary to obtain several candidate word sequences; different parts in the sequence are found, and the probabilities of different parts are calculated using the Bi-gram algorithm; based on the probability calculation results, the word sequence with the largest probability is selected as the final word segmentation result.

[0019] Step S4, in which the word segmentation model constructed in step S3 is trained, specifically includes the following steps:

[0020] Use the following formula to calculate a sentence (w1,w2,...,w x ) probability P(S):

[0021] P(S)=p(w1)p(w2|w1)...p(w x |w1,w2,...,w x-1 )

[0022] Where p(w1) is the probability of word w1 appearing in the text; p(w2|w1) is the number of times the combination of the previous word w1 and the next word w2 appears in the text; p(w x |w1,w2,...,w x-1 ) is a word sequence (w1,w2,...,w x-1 ) is followed by the word w x The number of times the combination of appears in the text;

[0023] During the calculation, the Markov hypothesis is introduced, the above calculation problem is converted into a 3-gram model, and expressed as follows:

[0024]

[0025] Where p(w i |wi-2 ,w i-1 ) is w i When it appears, it is preceded by (w i-2 ,w i-1 ) probability;

[0026] Establish the KenLM algorithm to solve the formula

[0027] Add the set start symbol and the set end symbol to the beginning and end of each sentence in the text content after word segmentation;

[0028] Initial count: Using the maximum likelihood estimation method, we get Among them, C(w i-2 w i-1 w i ) is a word sequence (w i-2 w i-1 w i ) appears, C(w i-2 w i-1 ) is a word sequence (w i-2 w i-1 ) frequency of occurrence;

[0029] Adjust the initial count: Use the following formula to adjust the initial count:

[0030]

[0031] In the formula is a word sequence (w i-2 w i-1 w i ) of the adjustment count; C(w i-2 w i-1 w i ) is recorded as v represents the word following the word sequence; For the word sequence (w i-2 w i-1 w i ) when the word following it is v;

[0032] Count discount: Use the following formula to calculate the discounted count:

[0033]

[0034] Where D3(k) is the discount value of 3-gram; k is the number of times a 3-gram word sequence appears; t k is the number of 3-grams that appear k times;

[0035] Calculate the pseudo probability: Use the following formula to calculate the pseudo probability:

[0036]

[0037] In the formula is a word sequence (w i-2 ,w i-1 ) followed by w i The pseudo probability of is a word sequence (w i- 2w i-1 w i )'s adjusted count; is a word sequence (w i-2 ,w i-1 ,v)’s adjustment count; v represents any word that may follow the word sequence;

[0038] Calculate the fallback weight: Use the following formula to calculate the fallback weight:

[0039]

[0040] In the formula is a word sequence (w i-2 ,w i-1 )’s fallback weight; is a word sequence (w i-2 ,w i-1 ,v) is counted as the number of times k;

[0041] Interpolation: The following interpolation formula is used to calculate the final conditional probability:

[0042]

[0043] Where p(w i |w i-1 ,w i-2 ) is the final conditional probability.

[0044] Step S5 uses the trained word segmentation model to process the target PDF and complete the text watermark recognition, which specifically includes the following steps:

[0045] Text watermark recognition:

[0046] Use the trained word segmentation model to process the target PDF, and use the sentence fluency calculated by the model as the indicator:

[0047] If the sentence fluency is greater than the set threshold, the subsequent image watermark recognition step is performed;

[0048] If the sentence fluency is less than or equal to the set threshold, the target PDF is determined to contain a text watermark;

[0049] Image watermark recognition:

[0050] Calculate the MD5 digest value of each bitmap in the target PDF and the frequency of occurrence of bitmaps on each page with the same MD5 digest value, and filter out bitmaps with a frequency greater than the set value;

[0051] For the filtered bitmaps, compare the MD5 digest values of the bitmaps with a frequency greater than the set value on each page: if they are the same, the bitmap is determined to be an image watermark; otherwise, the bitmap is determined to be a non-image watermark.

[0052] The present invention also provides a system for implementing the PDF document watermark recognition method, comprising a data acquisition module, a data preprocessing module, a model building module, a training module and a recognition module; the data acquisition module, the data preprocessing module, the model building module, the training module and the recognition module are connected in series in sequence; the data acquisition module is used to acquire existing PDF documents and perform text extraction, and upload the data to the data preprocessing module; the data preprocessing module is used to preprocess the text according to the acquired data to obtain corpus data, and upload the data to the model building module; the model building module is used to segment the corpus data according to the acquired data, and establish a segmentation model based on the N-gram model, and upload the data to the training module; the training module is used to train the segmentation model according to the acquired data, and upload the data to the recognition module; the recognition module is used to process the target PDF according to the acquired data using the trained segmentation model to complete the recognition of the text watermark.

[0053] The PDF document watermark recognition method and system provided by the present invention focus on the influence of watermarks on the text content of PDF documents containing text-type digital watermarks, and determine the fluency of sentences by establishing a language statistical model. Therefore, the present invention can effectively detect digital watermarks embedded in text content. At the same time, for PDF documents containing image-type digital watermarks, the present invention uses the position and frequency of the same image appearing on each page as judgment indicators, and can effectively detect image watermarks embedded in the text. Therefore, the present invention adopts a method combining natural language processing technology with bitmap recognition, which can effectively improve the recognition accuracy, and has high reliability, good precision, fast speed and safety. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 Schematic diagram of the method of the present invention.

[0055] Figure 2 This is a schematic diagram of a text after partial word segmentation in an embodiment of the method of the present invention.

[0056] Figure 3 Schematic diagram of some 1-gram word frequencies in an embodiment of the method of the present invention.

[0057] Figure 4Schematic diagram of the functional modules of the system of the present invention. DETAILED DESCRIPTION

[0058] like Figure 1 The figure shows a flow chart of the method of the present invention: The PDF document watermark recognition method provided by the present invention comprises the following steps:

[0059] S1. Obtain an existing PDF document and extract text; specifically, the steps include:

[0060] Get PDF files with and without watermarks, and extract the text content of the PDF files; at the same time, extract the content and corresponding location of the bitmap elements in the PDF files;

[0061] S2. Preprocess the text extracted in step S1 to obtain corpus data; specifically, the steps include:

[0062] Delete punctuation marks and special characters from the text content, and process the text to obtain a string set;

[0063] S3. Segment the corpus data obtained in step S2 and establish a segmentation model based on the N-gram model; specifically, the steps include:

[0064] Using the standard dictionary as the word segmentation dictionary, based on the forward matching algorithm and the reverse matching algorithm, the N-gram model is established to convert the string S=(w1,w2,...,w x ) are shortened in sequence and matched with the word segmentation dictionary to obtain several (preferably two) candidate word sequences; different parts of the sequence are found and the probabilities of the different parts are calculated using the Bi-gram algorithm (Laplace smoothing is used in the probability calculation); based on the probability calculation results, the word sequence with the highest probability is selected as the final word segmentation result;

[0065] In addition, a segmentation model metric evaluation system is established by matching the correctly segmented test corpus with the results obtained by the N-gram model. This includes calculating the accuracy, recall, and the harmonic mean of the two. If the evaluation metric reaches the specified threshold, the model training phase begins. Otherwise, the size of N is adjusted and the model training is repeated until the metric reaches the threshold.

[0066] S4. Training the word segmentation model constructed in step S3; specifically comprising the following steps:

[0067] Use the following formula to calculate a sentence (w1,w2,...,w x ) probability P(S):

[0068] P(S)=p(w1)p(w2|w1)...p(w x|w1,w2,...,w x-1 )

[0069] Where p(w1) is the probability of word w1 appearing in the text; p(w2|w1) is the number of times the combination of the previous word being w1 and the next word being w2 appears in the text; p(w x |w1,w2,...,w x-1 ) is a word sequence (w1,w2,...,w x-1 ) followed by w x The number of times the combination of appears in the text;

[0070] If the solution is directly obtained, it will be difficult to solve due to the large parameter space and sparse data. Therefore, the Markov hypothesis is introduced in the calculation. Since a large N will weaken the effect of the Markov hypothesis, combined with the actual situation of the text, the above calculation problem is converted into a 3-gram model and expressed as follows:

[0071]

[0072] Where p(w i |w i-2 ,w i-1 ) is w i When it appears, it is preceded by (w i-2 ,w i-1 ) probability;

[0073] Establish the KenLM algorithm to solve the formula

[0074] Add the set start symbol at the beginning and end of each sentence in the text content after word segmentation (for example <s> ) and the set terminator (< / s> );

[0075] Initial count: Using the maximum likelihood estimation method, we get Among them, C(w i-2 w i-1 w i ) is a word sequence (w i-2 w i-1 w i ) appears, C(w i-2 w i-1 ) is a word sequence (w i-2 w i-1 ) frequency of occurrence;

[0076] Adjust the initial count: Since the number of categories of words that can be connected to the front of the 3-gram is different, the word with a large number of possible connections should be given more counts, and the word with a small number of possible connections should be given less counts. Use the following formula to adjust the initial count:

[0077]

[0078] In the formula is a word sequence (w i-2 w i-1 w i ) of the adjustment count; briefly C(w i-2 w i-1 w i ) is recorded as v represents the word following the word sequence; For the word sequence (w i-2 w i-1 w i ) when the word following it is v;

[0079] Count discounting: For 3-grams with high frequency, reducing the count slightly will not significantly affect the final probability estimate; the subtracted counts can be added to those 3-grams that have not appeared to avoid the situation where the counts of the previous 3-grams are 0. However, for 3-grams with low frequency, the counts cannot be reduced. The discounted counts are calculated using the following formula:

[0080]

[0081] Where D3(k) is the discount value of the 3-gram. For 3-grams with low frequency, no discount is applied. For 3-grams with high frequency, the discount value is used for counting discount. k is the number of times a 3-gram word sequence appears. t k is the number of 3-grams that appear k times;

[0082] Calculate pseudo-probability: Reduce the number of occurrences of 3-grams that occur more frequently, then add it to the number of 3-grams that do not occur. Subtract the discount value calculated in step 4 from the numerator. Use the following formula to calculate pseudo-probability:

[0083]

[0084] In the formula is a word sequence (w i-2 ,w i-1 ) followed by w i The pseudo probability of is a word sequence (w i- 2w i-1 w i )'s adjusted count; is a word sequence (w i-2 ,w i-1 ,v)’s adjustment count; v represents any word that may follow the word sequence;

[0085] Calculate the fallback weight: The fallback weight is used to measure the ability of a word to be followed by different words. For different words, the "ability" of adding words after them is different. The fallback weight is calculated using the following formula:

[0086]

[0087] In the formula is a word sequence (w i-2 ,w i-1 )’s fallback weight, i.e., the ability to connect words; is a word sequence (w i-2 ,w i-1 ,v) is counted as the number of times k; is (w i-2 ,w i-1 ,v)’s adjustment count;

[0088] Interpolation: After calculating the pseudo probability and backoff weight corresponding to the 3-gram, the following interpolation formula is used to calculate the final conditional probability:

[0089]

[0090] Where p(w i |w i-1 ,w i-2 ) is the final conditional probability;

[0091] S5. Use the trained word segmentation model to process the target PDF and complete the text watermark recognition. The specific steps include the following:

[0092] In practice, the model can be stored as a language model in ARPA format. ARPA format is the standard storage format for N-gram language models. It is a binary file with a subtitle followed by a table listing all non-zero N-gram probabilities. Each N-gram entry contains, in order: discounted log probability (stored in log format), word sequence, and backoff weight (stored in base 10 log format).

[0093] Text watermark recognition:

[0094] Use the trained word segmentation model to process the target PDF, and use the sentence fluency calculated by the model as the indicator:

[0095] If the sentence fluency is greater than the set threshold, the subsequent image watermark recognition step is performed;

[0096] If the sentence fluency is less than or equal to the set threshold, the target PDF is determined to contain a text watermark;

[0097] Image watermark recognition:

[0098] Calculate the MD5 digest value of each bitmap in the target PDF and the frequency of occurrence of bitmaps on each page with the same MD5 digest value, and filter out bitmaps with a frequency greater than the set value;

[0099] For the filtered bitmaps, compare the MD5 digest values of the bitmaps with a frequency greater than the set value on each page: if they are the same, the bitmap is determined to be an image watermark; otherwise, the bitmap is determined to be a non-image watermark.

[0100] In order to verify the PDF digital watermark recognition effect based on natural language processing technology, the present invention selected PDF files containing digital watermarks and PDF files without digital watermarks downloaded from a certain company's official document rules and regulations collection platform and official document notification platform, totaling 2145 files, to construct an official document text corpus; then the text in the corpus was cleaned, and the cleaned data contained 1123093 sentences. On this basis, text preprocessing was carried out to remove punctuation marks and special characters; the plain text was segmented using the N-gram model to obtain the corpus after segmentation. After segmentation, some sentences are as follows: Figure 2 As shown;

[0101] Add the start symbol at the beginning and end of each sentence in the original training corpus <s> and terminator< / s> , count the training corpus after word segmentation, and calculate the 1-gram, 2-gram, and 3-gram word frequencies in turn. The word frequencies of some 1-grams are as follows Figure 3 As shown;

[0102] The KenLM model is trained using text content without digital watermarks, and the text sentences are scored. The fluency of the sentences on each page is averaged, and the fluency experimental results of each page are shown in Table 1:

[0103] Table 1 Schematic diagram of watermark recognition results using the KenLM model

[0104]

[0105] As shown in Table 1, the KenLM model used in this paper to calculate sentence fluency is highly effective in distinguishing between text in PDF documents with and without digital watermarks. By taking the fluency of each text as the average of the fluency of each page and setting the threshold to 0.1032, the final recognition accuracy reached 98.34%.

[0106] For PDF documents predicted to contain no watermarks using the KenLM model, we continue to use the image watermark recognition method to extract the bitmap of each page and its element information, calculate and compare the MD5 value of the bitmap, and determine whether the PDF document contains a watermark, with an accuracy rate of 97.63%.

[0107] In addition, 896 documents were selected based on the original 2145 documents, including 101 PDF documents without digital watermarks, 467 PDF documents with text-type digital watermarks, and 228 PDF documents with image-type digital watermarks. The recognition accuracy of the present invention is shown in Table 2.

[0108] Table 2 Schematic diagram of the accuracy of identifying watermarks in the newly selected corpus

[0109] Text Type Number of texts Recognition accuracy No digital watermark 101 97.03％ Contains digital watermark (text type) 567 99.11％ With digital watermark (image type) 228 99.60％

[0110] It can be seen from Table 2 that the method of the present invention has a high recognition accuracy.

[0111] like Figure 4 The figure shows a schematic diagram of the functional modules of the system of the present invention: the system provided by the present invention for implementing the PDF document watermark recognition method includes a data acquisition module, a data preprocessing module, a model building module, a training module and a recognition module; the data acquisition module, the data preprocessing module, the model building module, the training module and the recognition module are connected in series in sequence; the data acquisition module is used to acquire existing PDF documents and perform text extraction, and upload the data to the data preprocessing module; the data preprocessing module is used to preprocess the text according to the acquired data to obtain corpus data, and upload the data to the model building module; the model building module is used to segment the corpus data according to the acquired data, and establish a segmentation model based on the N-gram model, and upload the data to the training module; the training module is used to train the segmentation model according to the acquired data, and upload the data to the recognition module; the recognition module is used to process the target PDF according to the acquired data using the trained segmentation model to complete the recognition of the text watermark.

Claims

1. A method for identifying watermarks in a PDF document, comprising the following steps: S1. Obtain an existing PDF document and perform text extraction; S2. Preprocess the text extracted in step S1 to obtain corpus data; S3. Segment the corpus data obtained in step S2 and establish a segmentation model based on the N-gram model; S4. Training the word segmentation model constructed in step S3; specifically comprising the following steps: Use the following formula to calculate a sentence (w1,w2,...,w x ) probability P(S): P(S)=p(w1)p(w2|w1)...p(w x |w1,w2,...,w x-1 ) Where p(w1) is the probability of word w1 appearing in the text; p(w2|w1) is the number of times the combination of the previous word being w1 and the next word being w2 appears in the text; p(w x |w1,w2,...,w x-1 ) is a word sequence (w1,w2,...,w x-1 ) followed by w x The number of times the combination of appears in the text; During the calculation, the Markov hypothesis is introduced, the above calculation problem is converted into a 3-gram model, and expressed as follows: Where p(w i |w i-2 ,w i-1 ) is w i When it appears, it is preceded by (w i-2 ,w i-1 ) probability; Establish the KenLM algorithm to solve the formula Add the set start symbol and the set end symbol to the beginning and end of each sentence in the text content after word segmentation; Initial count: Using the maximum likelihood estimation method, we get in, C(w i-2 w i-1 w i ) is a word sequence (w i-2 w i-1 w i ) appears, C(w i-2 w i-1 ) is a word sequence (w i-2 w i-1 ) frequency of occurrence; Adjust the initial count: Use the following formula to adjust the initial count: In the formula is a word sequence (w i-2 w i-1 w i )’s adjustment count; v represents the word following the word sequence; For the word sequence (w i-2 w i-1 w i ) when the word following it is v; Count discount: Use the following formula to calculate the discounted count: Where D3(k) is the discount value of 3-gram; k is the number of times a 3-gram word sequence appears; t k is the number of 3-grams that appear k times; Calculate the pseudo probability: Use the following formula to calculate the pseudo probability: In the formula is a word sequence (w i-2 ,w i-1 ) followed by w i The pseudo probability of is a word sequence (w i-2 w i- 1w i )'s adjusted count; is a word sequence (w i-2 ,w i-1 ,v)’s adjustment count; v represents any word that may follow the word sequence; Calculate the fallback weight: Use the following formula to calculate the fallback weight: In the formula is a word sequence (w i-2 ,w i-1 )’s fallback weight; is a word sequence (w i-2 ,w i-1 ,v) is counted as the number of times k; Interpolation: The following interpolation formula is used to calculate the final conditional probability: Where p(w i |w i-1 ,w i-2 ) is the final conditional probability; S5. Use the trained word segmentation model to process the target PDF and complete the recognition of text watermarks.

2. The PDF document watermark recognition method according to claim 1, characterized in that The step S1 of obtaining an existing PDF document and extracting text specifically includes the following steps: Get PDF files with and without watermarks, and extract the text content of the PDF files; at the same time, extract the content and corresponding location of the bitmap elements in the PDF files.

3. The PDF document watermark recognition method according to claim 2, characterized in that The pre-processing described in step S2 specifically includes the following steps: Delete punctuation marks and special characters from the text content, and process the text to obtain a string set.

4. The PDF document watermark recognition method according to claim 3, characterized in that Step S3, in which the corpus data obtained in step S2 is segmented and a segmentation model is established based on the N-gram model, specifically includes the following steps: Using the standard dictionary as the word segmentation dictionary, based on the forward matching algorithm and the reverse matching algorithm, the N-gram model is established to convert the string S=(w1,w2,...,w x ) are shortened in sequence and matched with the word segmentation dictionary to obtain several candidate word sequences; different parts in the sequence are found, and the probabilities of different parts are calculated using the Bi-gram algorithm; based on the probability calculation results, the word sequence with the largest probability is selected as the final word segmentation result.

5. The PDF document watermark recognition method according to claim 4, characterized in that Step S5 uses the trained word segmentation model to process the target PDF and complete the text watermark recognition, which specifically includes the following steps: Text watermark recognition: Use the trained word segmentation model to process the target PDF, and use the sentence fluency calculated by the model as the indicator: If the sentence fluency is greater than the set threshold, the subsequent image watermark recognition step is performed; If the sentence fluency is less than or equal to the set threshold, the target PDF is determined to contain a text watermark; Image watermark recognition: Calculate the MD5 digest value of each bitmap in the target PDF and the frequency of occurrence of bitmaps on each page with the same MD5 digest value, and filter out bitmaps with a frequency greater than the set value; For the filtered bitmaps, compare the MD5 digest values of the bitmaps with a frequency greater than the set value on each page: if they are the same, the bitmap is determined to be an image watermark; otherwise, the bitmap is determined to be a non-image watermark.

6. A system for implementing the PDF document watermark recognition method according to any one of claims 1 to 5, characterized in that It includes a data acquisition module, a data preprocessing module, a model building module, a training module and a recognition module; the data acquisition module, the data preprocessing module, the model building module, the training module and the recognition module are connected in series; the data acquisition module is used to obtain existing PDF documents and extract text, and upload the data to the data preprocessing module; The data preprocessing module is used to preprocess the text according to the acquired data, obtain corpus data, and upload the data to the model building module; The model building module is used to segment the corpus data based on the acquired data, build a segmentation model based on the N-gram model, and upload the data to the training module; The training module is used to train the word segmentation model based on the acquired data and upload the data to the recognition module; the recognition module is used to process the target PDF using the trained word segmentation model based on the acquired data to complete the recognition of text watermarks.

Images