File anti-desensitization self-learning recognition system and method based on information entropy

Abstract

The invention discloses a file anti-desensitization self-learning recognition system and method based on information entropy, belongs to the technical field of intersection of natural language processing and content security recognition, and is applied to document screening and risk recognition in a multi-task scene. The implementation method comprises the following steps of: 1, performing character recognition and noise reduction processing on an original file to form a data set; 2, training labeled sample data through small samples, respectively adopting probability distribution of a data sliding window and information entropy to carry out maximum and minimum normalization screening, and further utilizing a fitted linear regression model to form an anti-desensitization word list; 3, screening the anti-desensitization degrees of the sentence segments of the data set by adopting a dictionary tree Trie structure to form an anti-desensitization sentence segment table; 4, marking the chapter-level anti-desensitization degree data set text fragments by using the large model; 5, generating an anti-desensitization report according to the anti-desensitization word and the anti-desensitization degree of the marked anti-desensitization file; compared with the prior art, the anti-desensitization file screening method and device have the advantage that the anti-desensitization file screening accuracy is improved.

Description

Technical Field

[0001] The present invention relates to a file anti-desensitization self-learning identification system and method based on information entropy, belonging to the technical field of the intersection of natural language processing and content security identification, and is applied to document screening and risk identification in multi-task scenarios. Background Art

[0002] In recent years, with the rapid growth of information technology and data volumes, the need for document security and compliance review has become increasingly prominent. In certain industry contexts, such as government agencies, news media, and the military, sensitive information within massive volumes of documents needs to be promptly identified and filtered.

[0003] Automated review methods based on screening pools or keyword matching can improve efficiency to a certain extent, but they struggle to update rules in a timely manner when new violations emerge or regulations change, and are prone to false positives or omissions when processing complex semantics. Large Language Models (LLMs) have made breakthroughs in natural language understanding and reasoning, bringing new possibilities to document review. However, existing solutions still lack the dynamic updating of keyword and rule libraries, the interpretability of rules and reasoning processes, and self-learning mechanisms, making it difficult to form a closed loop of continuous improvement based on feedback from anti-massification data screening.

[0004] Therefore, in multi-tasking scenarios, how to improve the accuracy of anti-massaging file screening has become an urgent problem to be solved. Summary of the Invention

[0005] In response to the technical problem of improving the accuracy of anti-desensitization document screening in multi-scenario tasks, the present invention proposes a document self-learning review system and method for rule derivation and large-model reasoning. By combining keyword filtering, large-model reasoning and screening pool management, a self-learning feedback update model is constructed to adapt to anti-desensitization document screening in multi-task scenarios.

[0006] On the one hand, the present invention discloses a file anti-desensitization self-learning identification method based on information entropy, comprising the following steps:

[0007] Step 1: The original multimodal file is processed through character recognition and noise reduction to form a data set;

[0008] Step 1.1: Divide the original multimodal files into file types;

[0009] Step 1.2: Perform character recognition on the divided files through OCR to generate text data;

[0010] Step 1.3: De-duplication and denoising of text data to obtain a dataset;

[0011] Step 2: Use small sample training to perform maximum value normalization screening on labeled sample data using the data sliding window and information entropy probability distribution, and then use the fitted linear regression model to form an anti-desensitization word list;

[0012] Step 2.1: Screen anti-desensitization sample labels through small sample training and set the anti-desensitization level;

[0013] Step 2.1.1: Use small samples to construct labeled sample data for word training;

[0014] Step 2.1.2: Segment the labeled sample data and remove non-desensitization-resistant words to obtain the desensitization-resistant sample labels;

[0015] Step 2.1.3: Set the anti-desensitization level of the anti-desensitization sample label;

[0016] Step 2.2: Use the data sliding window and information entropy probability distribution to perform maximum value normalization screening on the labeled sample data;

[0017] Step 2.2.1: Obtain the word frequency of labeled sample data using the method shown in formula (1);

[0018]

[0019] Among them, w represents a word and d represents a single document;

[0020] Step 2.2.2: Obtain the inverse document frequency of the labeled sample data using the method shown in formula (2);

[0021]

[0022] Where D represents the set of all documents;

[0023] Step 2.2.3: Use word frequency and inverse document frequency to perform anti-desensitization measurement as shown in formula (3);

[0024] TF-IDF(w,d)=TF(w,d)×IDF(w,D) (3)

[0025] Step 2.2.4: Use the data sliding window to filter the anti-desensitization words and combine the context relevance of the labeled sample data to perform maximum normalization as shown in formula (4);

[0026]

[0027] Among them, C(w) represents the cumulative frequency of word w co-occurring with other words in the data sliding window;

[0028] Step 2.2.5: Use information entropy to obtain the information distribution of anti-desensitization words and combine it with the contextual relevance of the labeled sample data to perform maximum value normalization using the method shown in formula (5);

[0029] H(w)=-∑ v∈上下文(w) P(v|w)·logP(v|w),

[0030]

[0031] in, is the conditional probability of w appearing in the context;

[0032] Step 2.3: Construct a linear regression model and perform minimum mean square error fitting on the model using formula (7);

[0033] S(w)=a·TF-IDF(w)+b·C norm (w)+c·H norm (w)+d (7)

[0034] Among them, a, b, and c represent the regression coefficient of word frequency, the regression coefficient of word co-occurrence rate, and the regression coefficient of information entropy respectively; d represents the fitting deviation;

[0035] Step 2.4: Use the regression model to adjust the anti-desensitization coefficient of word frequency, word co-occurrence rate and information entropy in the manner shown in formula (9) to form an anti-desensitization word list;

[0036]

[0037] W(w)=α·TF-IDF(w)+β·C norm (w)+γ·H norm (w) (9)

[0038] Among them, (α, β, γ) represents the fusion regression coefficient, and W (w) represents the anti-desensitization coefficient;

[0039] Step 3: Use the Trie structure to filter the anti-desensitization degree of the data set segments and form an anti-desensitization segment table;

[0040] Step 3.1: Use the anti-desensitization words as nodes and the number of searches as edges to build a dictionary tree Trie structure;

[0041] Step 3.2: Divide the dataset into segments;

[0042] Step 3.3: Use the Trie structure to obtain the anti-desensitization degree of the dataset segment;

[0043] Step 3.3.1: Use the Trie structure to match the dataset with the anti-desensitization word list to obtain the anti-desensitization words and their positions in the dataset;

[0044] Step 3.3.2: Use formula (10) to obtain the anti-desensitization degree of the dataset segment;

[0045] Score(S)=∑ w∈匹配到的候选词 W(w) (10)

[0046] Step 3.3.3: Set a dynamic threshold T and mark the anti-desensitization words with Score(S) ≥ T;

[0047] Step 3.4: Use the anti-desensitization degree to obtain the anti-desensitization segment table of the dataset segments;

[0048] Step 4: Use the large model to label the text fragments of the chapter-level anti-desensitization dataset formed by fusing sentence anti-desensitization with adjacent sentence anti-desensitization;

[0049] Step 4.1: Construct a fine-grained screening pool of large model prompt words and sentence anti-desensitization E(R i );

[0050] Step 4.2: Use the large model to obtain sentence-level anti-desensitization through anti-desensitization fusion;

[0051] Step 4.2.1: Divide the text fragments of the dataset input to the large model into sentences, and obtain the initial sentence anti-desensitization degree by the method of formula (11);

[0052]

[0053] Among them, M(s) is the anti-desensitization degree of the anti-desensitization sentence table, W(s) is the anti-desensitization degree of the anti-desensitization word table, and f(w,s) represents the frequency of candidate word w appearing in sentence s;

[0054] Step 4.2.2: Use the anti-desensitization fusion method to obtain the sentence anti-desensitization degree;

[0055] Step 4.2.2.1: Set the fine-grained screening pool to exclude the set {R1, R2, ..., R m};

[0056] Step 4.2.2.2: The fine-grained screening pool discharge set, the initial sentence anti-desensitization and the sentence anti-desensitization are fused as shown in formula (12) to form the sentence-level anti-desensitization.

[0057]

[0058] Among them, δ i For rule R iThe confidence factor output by the large model inference when triggered;

[0059] Step 4.3: Obtain the anti-desensitization degree of adjacent sentences in the text fragment of the dataset through similarity;

[0060] Step 4.3.1: Embedding vectors of consecutive sentences in the dataset text fragments i );

[0061] Step 4.3.2: Use the formula (13) to obtain the similarity Rel(s) between adjacent sentences. i ,s j );

[0062]

[0063] Among them, ||Embed(s i )|| represents sentence s i The norm of the embedding vector;

[0064] Step 4.3.3: Use formula (14) to obtain the anti-desensitization degree of adjacent sentences;

[0065] C(s i ,s j )=Rel(s i ,s j )×min{Sos(s i ),Sos(s j )} (14)

[0066] Among them, Sos(s i ) represents sentence s i sentence-level anti-desensitization;

[0067] Step 4.4: Fuse the sentence anti-desensitization score with the adjacent sentence anti-desensitization score to obtain the chapter-level anti-desensitization score as shown in formula (15);

[0068]

[0069] Where λ and η are equilibrium parameters and satisfy λ + η = 1;

[0070] Step 4.5: Set the chapter-level threshold T′ and set W passage The text segments of the dataset ≥T′ are marked as anti-desensitization; the anti-desensitization coefficients described in step 2.4 are interactively updated in a cyclic iterative manner using the anti-desensitization marks;

[0071] Step 5: Generate an anti-desensitization report based on the anti-desensitization words and anti-desensitization degree of the marked anti-desensitization files;

[0072] On the other hand, to achieve the purpose of the present invention, based on the above method, the present invention further proposes a document anti-desensitization self-learning recognition system based on information entropy, including a document preprocessing module, a candidate sensitive word construction module, an initial screening and matching module, and a chapter-level association analysis module;

[0073] The file preprocessing module is used to perform format conversion preprocessing on original files from multiple sources, using OCR, text parsing and other technologies to convert multi-format files such as images, PDFs, Word, and texts into structured text data, which will serve as input to the candidate sensitive word construction module;

[0074] The candidate sensitive word construction module uses a dynamic weight calculation method that combines TF-IDF with word frequency co-occurrence and word information entropy to calculate the comprehensive sensitivity weight of each word based on training samples with sensitivity level labels, forming a table of candidate sensitive words and sensitivity weights; this candidate word table will serve as input to the fast initial screening and matching module;

[0075] The initial screening and matching module uses a dynamic weighted Trie matching algorithm to quickly scan and preliminarily screen structured text data, accurately locate matching candidate sensitive words and their locations in the text, and record the corresponding sensitivity weights; the initial screening results and matching details will serve as input for the detailed rule reasoning and paragraph-level association analysis modules;

[0076] The passage-level association analysis module combines a large model with preset rules to conduct an in-depth review of candidate sensitive texts. It also calculates the sensitivity score of each sentence and the overall sensitivity weight of the entire passage through sentence- and passage-level association analysis, generating a detailed reasoning report and sensitivity judgment results. These serve as input to the manual review feedback update module.

[0077] A computer-readable storage device storing a computer program, characterized in that when the computer program is executed, the method for preventing desensitization and self-learning recognition of files based on information entropy is implemented.

[0078] Compared with the existing technology, it has the following beneficial effects:

[0079] 1. This invention uses a dynamic sensitive word construction and weight update mechanism. It integrates metrics such as TF-IDF, word frequency co-occurrence, and word information entropy to construct a candidate sensitive word list. It then dynamically adjusts the weight of each word based on training samples with sensitivity levels. This method automatically captures newly appearing sensitive terms in text and continuously optimizes the candidate word list based on feedback, significantly improving the real-time detection of sensitive information and the system's self-learning capabilities.

[0080] 2. This invention uses a dynamic weighted Trie matching algorithm to quickly perform initial screening and matching on raw structured text, achieving precise location of sensitive words with low time complexity. This matching algorithm not only significantly improves initial screening efficiency but also effectively reduces the risk of false positives and false negatives, providing precise candidate sensitive information fragments for subsequent detailed rule reasoning.

[0081] 3. This invention combines deep rule-based reasoning with a large model and analysis of both paragraph-level and inter-sentence associations. It assigns different sensitivity weights to different rule violations using a pre-set rule evaluation index. It then uses semantic associations to comprehensively score the sensitivity of sentences and paragraphs, achieving a detailed assessment of the overall sensitivity of the text. This multi-layered, comprehensive review effectively improves review accuracy and system interpretability, further reducing the risk of misjudgment. BRIEF DESCRIPTION OF THE DRAWINGS

[0082] Figure 1 It is a schematic flow chart of the method of the present invention; DETAILED DESCRIPTION

[0083] In order to better illustrate the purpose and advantages of the present invention, the following is a further description of the invention in conjunction with the accompanying drawings and examples. It should be noted that the implementation of the present invention is not limited to the following embodiments, and any form of modification or change made to the present invention will fall within the scope of protection of the present invention.

[0084] This embodiment uses scanned paper contracts in PDF format as the anti-desensitization object. First, character recognition and noise reduction processing are performed on the contracts uploaded by users to be reviewed to form a data set. Then, based on a small number of labeled contract samples, the data sliding window and the probability distribution of information entropy are used to perform maximum value normalization screening, and the fitted linear regression model is used to form an anti-desensitization word list. Subsequently, the dictionary tree Trie structure is used to screen the anti-desensitization degree of the data set segments to form an anti-desensitization sentence segment table. After that, key information is extracted from the internal system documents of the enterprise to construct a fine-grained screening pool of large model prompt words and sentence anti-desensitization degree E(R i ), use the large model to fuse the sentence anti-desensitization degree with the adjacent sentence anti-desensitization degree to form the chapter-level anti-desensitization degree to mark the text fragments of the anti-desensitization dataset, and finally form an anti-desensitization report.

[0085] Example

[0086] like Figure 1 As shown, the document self-learning review method for rule derivation and large model reasoning in this embodiment is specifically implemented as follows:

[0087] Step 1: The original multimodal file is processed through character recognition and noise reduction to form a data set;

[0088] Step 1.1: Divide the original multimodal files into file types;

[0089] Step 1.2: Perform character recognition on the divided files through OCR to generate text data;

[0090] Step 1.3: De-duplication and denoising of text data to obtain a dataset;

[0091] In the embodiment, the present invention uploads a scanned paper contract in PDF format as the document to be reviewed, divides the contract into text types, performs character recognition on it through OCR, and extracts text data; then deduplicates and denoises the text data, such as removing date numbers, watermark descriptions, and recurring template descriptions such as "This contract is in duplicate" in headers and footers, to form a pure contract data set as the basic data for subsequent anti-desensitization.

[0092] Step 2: Use small sample training to perform maximum value normalization screening on labeled sample data using the data sliding window and information entropy probability distribution, and then use the fitted linear regression model to form an anti-desensitization word list;

[0093] Step 2.1: Screen anti-desensitization sample labels through small sample training and set the anti-desensitization level;

[0094] Step 2.1.1: Use small samples to construct labeled sample data for word training;

[0095] Step 2.1.2: Segment the labeled sample data and remove non-desensitization-resistant words to obtain the desensitization-resistant sample labels;

[0096] Step 2.1.3: Set the anti-desensitization level of the anti-desensitization sample label;

[0097] Step 2.2: Use the data sliding window and information entropy probability distribution to perform maximum value normalization screening on the labeled sample data;

[0098] Step 2.2.1: Obtain the word frequency of labeled sample data using the method shown in formula (1);

[0099]

[0100] Among them, w represents a word and d represents a single document;

[0101] Step 2.2.2: Obtain the inverse document frequency of the labeled sample data using the method shown in formula (2);

[0102]

[0103] Where D represents the set of all documents;

[0104] Step 2.2.3: Use word frequency and inverse document frequency to perform anti-desensitization measurement as shown in formula (3);

[0105] TF-IDF(w,d)=TF(w,d)×IDF(w,D) (3)

[0106] Step 2.2.4: Use the data sliding window to filter the anti-desensitization words and combine the context relevance of the labeled sample data to perform maximum normalization as shown in formula (4);

[0107]

[0108] Among them, C(w) represents the cumulative frequency of word w co-occurring with other words in the data sliding window;

[0109] Step 2.2.5: Use information entropy to obtain the information distribution of anti-desensitization words and combine it with the contextual relevance of the labeled sample data to perform maximum value normalization using the method shown in formula (5);

[0110] H(w)=-∑ v∈上下文(w) P(v|w)·logP(v|w),

[0111]

[0112] in, is the conditional probability of w appearing in the context;

[0113] Step 2.3: Construct a linear regression model and perform minimum mean square error fitting on the model using formula (7);

[0114] S(w)=a·TF-IDF(w)+b·C norm (w)+c·H norm (w)+d (7)

[0115] Among them, a, b, and c represent the regression coefficient of word frequency, the regression coefficient of word co-occurrence rate, and the regression coefficient of information entropy respectively; d represents the fitting deviation;

[0116] Step 2.4: Use the regression model to adjust the anti-desensitization coefficient of word frequency, word co-occurrence rate and information entropy in the manner shown in formula (9) to form an anti-desensitization word list;

[0117]

[0118] W(w)=α·TF-IDF(w)+β·C norm (w)+γ·H norm (w) (9)

[0119] Among them, (α, β, γ) represents the fusion regression coefficient, and W (w) represents the anti-desensitization coefficient;

[0120] In this embodiment, the present invention uses a small number of annotated contract sample texts from a company's historical archives as a training set. Through small-sample training, it screens for anti-desensitization sample labels and sets anti-desensitization levels. For example, "Effective without a stamp" and "Agreed payment period is unclear" are marked as highly sensitive. Next, a data sliding window and the probability distribution of information entropy are used to perform maximum value normalization screening on the training contract samples. Anti-desensitization coefficients are derived based on a linear regression model, ultimately forming an anti-desensitization word list containing sensitive terms such as "supplementary agreement" and "internal information."

[0121] Step 3: Use the Trie structure to filter the anti-desensitization degree of the data set segments and form an anti-desensitization segment table;

[0122] Step 3.1: Use the anti-desensitization words as nodes and the number of searches as edges to build a dictionary tree Trie structure;

[0123] Step 3.2: Divide the dataset into segments;

[0124] Step 3.3: Use the Trie structure to obtain the anti-desensitization degree of the dataset segment;

[0125] Step 3.3.1: Use the Trie structure to match the dataset with the anti-desensitization word list to obtain the anti-desensitization words and their positions in the dataset;

[0126] Step 3.3.2: Use formula (10) to obtain the anti-desensitization degree of the dataset segment;

[0127] Score(S)=∑ w∈匹配到的候选词 W(w) (10)

[0128] Step 3.3.3: Set a dynamic threshold T and mark the anti-desensitization words with Score(S) ≥ T;

[0129] Step 3.4: Use the anti-desensitization degree to obtain the anti-desensitization segment table of the dataset segments;

[0130] In the embodiment, the present invention constructs a dictionary tree Trie structure based on the anti-desensitization word list obtained in step 2, and anti-desensitization words such as "automatic renewal", "supplementary agreement", and "final interpretation right" are inserted into the Trie structure according to the characters, and the end node records its corresponding anti-desensitization coefficient 3, 3, 1. Subsequently, the contract data set obtained in step 1 is divided into segments, and the anti-desensitization degree of the contract data set segment is obtained using the dictionary tree Trie segment; finally, the anti-desensitization degree is used to obtain the anti-desensitization segment table of the data set segment. For example, when scanning the segment "If there is no objection to this contract, it will be automatically postponed for one year", the "automatic postponement" term is matched, and the anti-desensitization degree of the segment is obtained as 3, and the threshold value T is set to 2, and the segment is added to the anti-desensitization segment table.

[0131] Step 4: Use the large model to label the text fragments of the chapter-level anti-desensitization dataset formed by fusing sentence anti-desensitization with adjacent sentence anti-desensitization;

[0132] Step 4.1: Construct a fine-grained screening pool of large model prompt words and sentence anti-desensitization E(R i );

[0133] Step 4.2: Use the large model to obtain sentence-level anti-desensitization through anti-desensitization fusion;

[0134] Step 4.2.1: Divide the text fragments of the dataset input to the large model into sentences, and obtain the initial sentence anti-desensitization degree by the method of formula (11);

[0135]

[0136] Among them, M(s) is the anti-desensitization degree of the anti-desensitization sentence table, W(s) is the anti-desensitization degree of the anti-desensitization word table, and f(w,s) represents the frequency of candidate word w appearing in sentence s;

[0137] Step 4.2.2: Use the anti-desensitization fusion method to obtain the sentence anti-desensitization degree;

[0138] Step 4.2.2.1: Set the fine-grained screening pool to exclude the set {R1, R2, ..., R m};

[0139] Step 4.2.2.2: The fine-grained screening pool discharge set, the initial sentence anti-desensitization and the sentence anti-desensitization are fused as shown in formula (12) to form the sentence-level anti-desensitization.

[0140]

[0141] Among them, δ i For rule R i The confidence factor output by the large model inference when triggered;

[0142] Step 4.3: Obtain the anti-desensitization degree of adjacent sentences in the text fragment of the dataset through similarity;

[0143] Step 4.3.1: Embedding vectors of consecutive sentences in the dataset text fragments i );

[0144] Step 4.3.2: Use the formula (13) to obtain the similarity Rel(s) between adjacent sentences. i ,s j );

[0145]

[0146] Among them, ||Embed(s i )|| represents sentence s i The norm of the embedding vector;

[0147] Step 4.3.3: Use formula (14) to obtain the anti-desensitization degree of adjacent sentences;

[0148] C(s i ,s j )=Rel(s i ,s j )×min{Sos(s i ),Sos(s j )} (14)

[0149] Among them, Sos(s i ) represents sentence s i sentence-level anti-desensitization;

[0150] Step 4.4: Fuse the sentence anti-desensitization score with the adjacent sentence anti-desensitization score to obtain the chapter-level anti-desensitization score as shown in formula (15);

[0151]

[0152] Where λ and η are equilibrium parameters and satisfy λ + η = 1;

[0153] Step 4.5: Set the chapter-level threshold T′ and set W passage The text segments of the dataset ≥T′ are marked as anti-desensitization; the anti-desensitization coefficients described in step 2.4 are interactively updated in a cyclic iterative manner using the anti-desensitization marks;

[0154] In the embodiment, the present invention embeds the system document "Contract Approval Process" into the prompt template, and builds a contract system fine-grained screening pool of large model prompt words and sentence anti-desensitization E(R) based on the regulations therein. i), such as R1: "The automatic renewal clause needs to state the termination conditions", the anti-desensitivity E(R1) = 2.5; the contract dataset fragments input to the large model are divided into sentences and the initial sentence anti-desensitivity is obtained. For example, the initial sentence anti-desensitivity 3 is calculated for the divided fragment s1: "......, if there is no objection, the contract will be extended for one year", and the large model is used to obtain the sentence-level anti-desensitivity through anti-desensitivity fusion. The model recognizes that it violates the fine-grained screening pool and excludes R1 from the set, and gives a confidence factor of 0.92, and calculates the sentence-level anti-desensitivity 3+2.5*0.92=5.3; then, BERT is used to construct the embedding vector of the contract sentence segment, and the large model obtains the anti-desensitivity C(s1,s2) of the adjacent sentences in the dataset text segment through similarity. j )=2.1; the anti-desensitization degree of all sentences in the input contract and the anti-desensitization degree of adjacent sentences are calculated and integrated as above to obtain the final chapter-level anti-desensitization degree.

[0155] Step 5: Generate an anti-desensitization report based on the anti-desensitization words and anti-desensitization degree of the marked anti-desensitization files;

[0156] On the other hand, to achieve the purpose of the present invention, based on the above method, the present invention further proposes a document anti-desensitization self-learning recognition system based on information entropy, including a document preprocessing module, a candidate sensitive word construction module, an initial screening and matching module, and a chapter-level association analysis module;

[0157] The file preprocessing module is used to perform format conversion preprocessing on original files from multiple sources, using OCR, text parsing and other technologies to convert multi-format files such as images, PDFs, Word, and texts into structured text data, which will serve as input to the candidate sensitive word construction module;

[0158] The candidate sensitive word construction module uses a dynamic weight calculation method that combines TF-IDF with word frequency co-occurrence and word information entropy to calculate the comprehensive sensitivity weight of each word based on training samples with sensitivity level labels, forming a table of candidate sensitive words and sensitivity weights; this candidate word table will serve as input to the fast initial screening and matching module;

[0159] The initial screening and matching module uses a dynamic weighted Trie matching algorithm to quickly scan and preliminarily screen structured text data, accurately locate matching candidate sensitive words and their locations in the text, and record the corresponding sensitivity weights; the initial screening results and matching details will serve as input for the detailed rule reasoning and paragraph-level association analysis modules;

[0160] The passage-level association analysis module combines a large model with preset rules to conduct an in-depth review of candidate sensitive texts. It also calculates the sensitivity score of each sentence and the overall sensitivity weight of the entire passage through sentence- and passage-level association analysis, generating a detailed reasoning report and sensitivity judgment results. These serve as input to the manual review feedback update module.

[0161] A computer-readable storage device storing a computer program, characterized in that when the computer program is executed, the method for preventing desensitization and self-learning recognition of files based on information entropy is implemented.

[0162] The above specific description further illustrates the purpose, technical solutions and beneficial effects of the invention in detail. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A file anti-desensitization self-learning identification method based on information entropy, characterized by: The following steps are included: Step 1: The original multimodal file is processed through character recognition and noise reduction to form a data set; Step 2: Use small sample training to perform maximum value normalization screening on labeled sample data using the data sliding window and information entropy probability distribution, and then use the fitted linear regression model to form an anti-desensitization word list; Step 2.1: Screen anti-desensitization sample labels through small sample training and set the anti-desensitization level; Step 2.2: Use the data sliding window and information entropy probability distribution to perform maximum value normalization screening on the labeled sample data; Step 2.3: Construct a linear regression model and perform minimum mean square error fitting on the model using formula (7); S(w)=a·TF-IDF(w)+b·C norm (w)+c·H norm (w)+d (7) Among them, a, b, and c represent the regression coefficient of word frequency, the regression coefficient of word co-occurrence rate, and the regression coefficient of information entropy respectively; d represents the fitting deviation; Step 2.4: Use the regression model to adjust the anti-desensitization coefficient of word frequency, word co-occurrence rate and information entropy in the manner shown in formula (9) to form an anti-desensitization word list; W(w)=α·TF-IDF(w)+β·C norm (w)+γ·H norm (w) (9) Among them, (α, β, γ) represents the fusion regression coefficient, and W (w) represents the anti-desensitization coefficient; Step 3: Use the Trie structure to filter the anti-desensitization degree of the data set segments and form an anti-desensitization segment table; Step 3.1: Use the anti-desensitization words as nodes and the number of searches as edges to build a dictionary tree Trie structure; Step 3.2: Divide the dataset into segments; Step 3.3: Use the Trie structure to obtain the anti-desensitization degree of the dataset segment; Step 3.4: Use the anti-desensitization degree to obtain the anti-desensitization segment table of the dataset segments; Step 4: Use the large model to label the text fragments of the chapter-level anti-desensitization dataset formed by fusing sentence anti-desensitization with adjacent sentence anti-desensitization; Step 4.1: Construct a fine-grained screening pool of large model prompt words and sentence anti-desensitization E(R i ); Step 4.2: Use the large model to obtain sentence-level anti-desensitization through anti-desensitization fusion; Step 4.3: Obtain the anti-desensitization degree of adjacent sentences in the text fragment of the dataset through similarity; Step 4.4: Fuse the sentence anti-desensitization score with the adjacent sentence anti-desensitization score to obtain the chapter-level anti-desensitization score as shown in formula (15); Where λ and η are equilibrium parameters and satisfy λ + η = 1; Step 4.5: Set the chapter-level threshold T′ and set W passage The text segments of the dataset ≥T′ are marked as anti-desensitization; the anti-desensitization coefficients described in step 2.4 are interactively updated in a cyclic iterative manner using the anti-desensitization marks; Step 5: Generate an anti-desensitization report for the marked anti-desensitization words and anti-desensitization degree of the anti-desensitization file.

2. The method for file anti-desensitization self-learning recognition based on information entropy according to claim 1, characterized in that: Step 1 is implemented as follows: Step 1.1: Divide the original multimodal files into file types; Step 1.2: Perform character recognition on the divided files through OCR to generate text data; Step 1.3: De-duplication and denoising of text data to obtain a dataset.

3. The method for file anti-desensitization self-learning recognition based on information entropy according to claim 1, characterized in that: Step 2.1 is implemented as follows: Step 2.1.1: Use small samples to construct labeled sample data for word training; Step 2.1.2: Segment the labeled sample data and remove non-desensitization-resistant words to obtain the desensitization-resistant sample labels; Step 2.1.3: Set the anti-desensitization level of the anti-desensitization sample label.

4. The method for file anti-desensitization self-learning recognition based on information entropy according to claim 1, characterized in that: The implementation method of step 2.2 is: Step 2.2.1: Obtain the word frequency of labeled sample data using the method shown in formula (1); Among them, w represents a word and d represents a single document; Step 2.2.2: Obtain the inverse document frequency of the labeled sample data using the method shown in formula (2); Where D represents the set of all documents; Step 2.2.3: Use word frequency and inverse document frequency to perform anti-desensitization measurement as shown in formula (3); TF-IDF(w,d)=TF(w,d)×IDF(w,D) (3) Step 2.2.4: Use the data sliding window to filter the anti-desensitization words and combine the context relevance of the labeled sample data to perform maximum normalization as shown in formula (4); Among them, C(w) represents the cumulative frequency of word w co-occurring with other words in the data sliding window; Step 2.2.5: Use information entropy to obtain the information distribution of anti-desensitization words and combine it with the contextual relevance of the labeled sample data to perform maximum value normalization using the method shown in formula (5); H(w)=-∑ v∈上下文(w) P(v|w)·logP(v|w), in, is the conditional probability of w appearing in the context.

5. The method for file anti-desensitization self-learning recognition based on information entropy according to claim 1, characterized in that: The implementation method of step 3.3 is: Step 3.3.1: Use the Trie structure to match the dataset with the anti-desensitization word list to obtain the anti-desensitization words and their positions in the dataset; Step 3.3.2: Use formula (10) to obtain the anti-desensitization degree of the dataset segment; Score(S)=∑ w∈匹配到的候选词 W(w) (10) Step 3.3.3: Set a dynamic threshold T and mark the anti-desensitization words with Score(S) ≥ T.

6. The method for file anti-desensitization self-learning recognition based on information entropy according to claim 1, characterized in that: The implementation method of step 4.2 is: Step 4.2.1: Divide the text fragments of the dataset input to the large model into sentences, and obtain the initial sentence anti-desensitization degree by the method of formula (11); Among them, M(s) is the anti-desensitization degree of the anti-desensitization sentence table, W(s) is the anti-desensitization degree of the anti-desensitization word table, and f(w,s) represents the frequency of candidate word w appearing in sentence s; Step 4.2.2: Use the anti-desensitization fusion method to obtain the sentence anti-desensitization degree.

7. The method for file anti-desensitization self-learning recognition based on information entropy according to claim 6, characterized in that: The implementation method of step 4.2.2 is: Step 4.2.2.1: Set the fine-grained screening pool to exclude the set {R1, R2, ..., R m }; Step 4.2.2.2: The fine-grained screening pool discharge set, the initial sentence anti-desensitization and the sentence anti-desensitization are fused as shown in formula (12) to form the sentence-level anti-desensitization. Among them, δ i For rule R i The confidence factor output by the large model inference when triggered.

8. The method for file anti-desensitization self-learning recognition based on information entropy according to claim 6, characterized in that: Step 4.3 is implemented as follows: Step 4.3.1: Embedding vectors of consecutive sentences in the dataset text fragments i ); Step 4.3.2: Use the formula (13) to obtain the similarity Rel(s) between adjacent sentences. i ,s j ); Among them, ||Embed(s i )|| represents sentence s i The norm of the embedding vector; Step 4.3.3: Use formula (14) to obtain the anti-desensitization degree of adjacent sentences; C(s i ,s j )=Rel(s i ,s j )×min{Sos(s i ),Sos(s j )} (14) Among them, Sos(s i ) represents sentence s i Sentence-level anti-desensitization.

9. A file anti-desensitization self-learning recognition system based on information entropy that implements the method of claim 1, characterized in that: It includes a file pre-processing module, a candidate sensitive word construction module, an initial screening and matching module, and a chapter-level association analysis module; The file preprocessing module is used to perform format conversion preprocessing on original files from multiple sources, using OCR, text parsing and other technologies to convert multi-format files such as images, PDFs, Word, and texts into structured text data, which will serve as input to the candidate sensitive word construction module; The candidate sensitive word construction module uses a dynamic weight calculation method that combines TF-IDF with word frequency co-occurrence and word information entropy to calculate the comprehensive sensitivity weight of each word based on training samples with sensitivity level labels, forming a table of candidate sensitive words and sensitivity weights; this candidate word table will serve as input to the fast initial screening and matching module; The initial screening and matching module uses a dynamic weighted Trie matching algorithm to quickly scan and preliminarily screen structured text data, accurately locate matching candidate sensitive words and their locations in the text, and record the corresponding sensitivity weights; The initial screening results and matching details will serve as input for the detailed rule reasoning and text-level association analysis modules; The chapter-level association analysis module combines a large model with preset rules to conduct an in-depth review of candidate sensitive texts. It also calculates the sensitivity score of each sentence and the comprehensive sensitivity weight of the entire chapter through sentence- and chapter-level association analysis, and generates a detailed reasoning report and sensitivity judgment results. It will serve as the input of the manual review feedback update module.

10. A computer-readable storage device storing a computer program, characterized in that: When the computer program is executed, it implements a file anti-desensitization self-learning identification method based on information entropy as described in any one of claims 1 to 8.

Images