Data security compliance evaluation and identification method and system

Abstract

The invention discloses a data security compliance evaluation and identification method and system, and relates to the technical field of data security compliance evaluation and identification, and the method comprises the steps: collecting regulation text data through an API interface, and carrying out the preprocessing of the regulation text data through a natural language processing library NLTK, and obtaining formatted regulation text data; mapping each word in the formatted law and regulation text data into a fixed-length vector by adopting a pre-calculation word embedding method, and generating a digital vector; marking important terms in the digital vector by adopting a named entity recognition NER method to obtain a key concept text; establishing an evaluation model based on a machine learning algorithm, inputting the digital vector into the evaluation model, calculating and outputting an evaluation value by adopting a weighted average method, setting an evaluation threshold value, comparing the evaluation threshold value with the evaluation value, judging whether the digital vector is compliant or not, and obtaining an evaluation result; and constructing a recognition model based on an anomaly detection algorithm, and inputting the key concept text into the recognition model.

Description

Technical Field

[0001] The present invention relates to the technical field of data security compliance assessment and identification, and in particular to a data security compliance assessment and identification method and system. Background Art

[0002] Data security compliance assessment and identification technology refers to the technology that uses advanced data analysis and machine learning algorithms to perform automated compliance assessment and anomaly detection on various regulatory texts, internal corporate policies, and operational procedures. Its core goal is to ensure that companies and organizations strictly abide by relevant laws and regulations during operations and avoid legal risks and economic losses caused by violations.

[0003] In the field of data security compliance assessment and identification technology, traditional assessment models rely on fixed threshold settings and are difficult to adapt to changes in different scenarios, resulting in inflexible and inaccurate assessment results. Existing anomaly detection methods mostly rely on a single algorithm, which is difficult to cope with the complex and changing regulatory text environment, resulting in unstable detection results. At the same time, in existing technologies, compliance assessments and vulnerability information are usually isolated, making it difficult to form an effective linkage mechanism, resulting in potential security risks not being discovered and repaired in a timely manner. Summary of the invention

[0004] In view of the above existing problems, the present invention is proposed.

[0005] Therefore, the present invention provides a data security compliance assessment and identification method to solve the problem that the traditional assessment model relies on fixed threshold settings and is difficult to adapt to changes in different scenarios, resulting in inflexible and inaccurate assessment results.

[0006] In order to solve the above technical problems, the present invention provides the following technical solutions: In a first aspect, the present invention provides a data security compliance assessment and identification method, which includes: Collect regulatory text data through the API interface, and use the natural language processing library NLTK to pre-process the regulatory text data to obtain formatted regulatory text data; Using a pre-computed word embedding method, each word in the formatted regulatory text data is mapped into a fixed-length vector and a digital vector is generated; The named entity recognition (NER) method is used to mark the important terms in the digital vector and obtain the key concept text; An evaluation model is established based on a machine learning algorithm, the digital vector is input into the evaluation model, and the output evaluation value is calculated using the weighted average method; Set the evaluation threshold and compare it with the evaluation value to determine whether the digital vector is compliant and obtain the evaluation result; A recognition model is built based on the anomaly detection algorithm, the key concept text is input into the recognition model, and the clustering algorithm is used to output the recognition results.

[0007] As a preferred solution of the data security compliance assessment and identification method of the present invention, wherein: the regulatory text data is collected through the API interface, and the regulatory text data is preprocessed using the natural language processing library NLTK to obtain formatted regulatory text data. The specific steps are: Use the requests library to send HTTP protocol requests to the API endpoint to obtain regulatory text data; Use the natural language processing library NLTK to filter meaningless words in regulatory text data and reduce noise; The words in the regulatory text data are restored, and the text in the regulatory text data is segmented into separate words to obtain formatted regulatory text data.

[0008] As a preferred solution of the data security compliance assessment and identification method of the present invention, the pre-calculated word embedding method is used to map each word in the formatted regulatory text data into a fixed-length vector and generate a digital vector. The specific steps are as follows: The word embedding library Word2Vec is used to perform word segmentation on the formatted regulatory text data, and all independent words after word segmentation are collected to form a non-repetitive vocabulary; Create an empty collection and fill it with vocabulary, count the frequency of occurrence of independent words, sort the vocabulary according to the frequency of occurrence of independent words, and form the final vocabulary; For independent words in the final vocabulary, the corresponding fixed-length vector is searched in the word embedding library Word2Vec, and the corresponding fixed-length vector found is used as the representation of the independent word to form a word-level digital vector; Create a zero vector of the same dimension as the word-level digital vector as an accumulator, accumulate all word-level digital vectors, and obtain the corresponding word-level digital vector for each independent word in the final vocabulary; Add each word-level digital vector to the accumulator one by one to get the average word vector; The obtained average word vector is used as the representation of the document-level digital vector to obtain the document-level digital vector and mark it as a digital vector.

[0009] As a preferred solution of the data security compliance assessment and identification method of the present invention, wherein: the named entity recognition (NER) method is used to mark important terms in the digital vector to obtain key concept texts, and the specific steps are: Using a named entity recognition (NER) model, input the formatted regulatory text data sentence by sentence or paragraph by paragraph into the named entity recognition (NER) model; The named entity recognition (NER) model scans the formatted regulatory text data and identifies entities; For the identified entities, the named entity recognition (NER) model assigns different category labels to them and outputs the annotated text information; The output annotated text information is marked as important terms in the text and forms a key concept text.

[0010] As a preferred solution of the data security compliance assessment and identification method of the present invention, wherein: the assessment model is established based on the machine learning algorithm, the digital vector is input into the assessment model, and the weighted average method is used to calculate the output assessment value. The specific steps are: The regulatory text data to be formatted Divide into training set and validation set; Use the scikit-learn library to import the LogisticRegression class to create a logistic regression instance and combine it with the formatted regulatory text data Build an evaluation model and set the initial parameter weights of the evaluation model ; A numeric vector containing the regulatory text data to be formatted Input into the evaluation model and calculate the evaluation value using the weighted average method , the expression is: ; in, is the word weight, It is The weight of the word, It is The word vector of each word, is the Sigmoid activation function, is the evaluation value, is the formatted regulatory text data, is a numeric vector of formatted regulatory text data; As a preferred solution of the data security compliance assessment and identification method of the present invention, wherein: the set assessment threshold is compared with the assessment value to determine whether the digital vector is compliant and obtain the assessment result, the specific steps are: Thresholds are set based on formatted regulatory text data in the validation set that was never used in training ; when ≥ , representing formatted regulatory text data A numeric vector in Below the set threshold , then the regulatory text is judged to be compliant; when < , determine the formatted regulatory text data A numeric vector in If it is not compliant, the formatted regulatory text data Make the modification, and after the modification, recalculate the formatted regulatory text data according to the digital vector expression A numeric vector in , re-enter it into the evaluation model to calculate the evaluation value, and set the number of iterations to , until ≥ .

[0011] As a preferred solution of the data security compliance assessment and identification method of the present invention, wherein: the recognition model is constructed based on the anomaly detection algorithm, the key concept text is input into the recognition model, and the clustering algorithm is used to output the recognition result. The specific steps are: Building a recognition model based on the Isolation Forest Anomaly Detection Algorithm and formatted regulatory text data; The isolation forest algorithm is used to isolate observation points by building multiple decision trees, and the IsolationForest class in the Python library is used to initialize the recognition model parameters; The key concept text in the formatted legal text data is converted into a TF-IDF value, the TF-IDF vector value of the key concept text in the formatted legal text data is calculated, and the key concept text in the formatted legal text data represented by the TF-IDF vector value is standardized. The expression is: ; in, It's words. Formatted regulatory text data Key concepts in the text, is all the key concept text, is the total number of documents in all key concept texts, Contains words The number of documents, It's a word In formatting regulatory text data The frequency of key concepts in the text, for TF-IDF vector value of ; The TF-IDF vector value is input into the recognition model, and the K-means clustering algorithm is used to classify the output of the recognition model. The expression is: ; in, is the cluster label, is the anomaly score matrix, Represents the anomaly score matrix Middle The anomaly score vector of samples, It is The center point of the cluster, is the set number of clusters, Formatted regulatory text data Key concepts in the text, Represents key text concepts The anomaly score, is the number of trees in the isolation forest algorithm, is the anomaly score matrix The elements in It is Clusters, is the isolation forest algorithm The path length, is the maximum path length of all trees in the isolation forest, yes The expected value of , that is, the average value of the path length; Setting anomaly score threshold With the anomaly score matrix Make a comparison; when ≥ , indicates formatted regulatory text data Key text concepts in Non-compliance; when < , indicates formatted regulatory text data Key text concepts in Compliance.

[0012] In a second aspect, the present invention provides a data security compliance assessment and identification system, including: a data processing module, a word embedding generation module, a key concept extraction module, an assessment model module, and an anomaly detection module; The data processing module is used to collect regulatory text data through an API interface, and pre-process the regulatory text data using a natural language processing library NLTK to obtain formatted regulatory text data; The word embedding generation module is used to map each word in the formatted regulatory text data into a fixed-length vector using a pre-calculated word embedding method and generate a digital vector; The key concept extraction module is used to mark important terms in the digital vector using a named entity recognition method to obtain a key concept text; The evaluation model module is used to establish an evaluation model based on a machine learning algorithm, input a digital vector into the evaluation model, calculate an output evaluation value using a weighted average method, set an evaluation threshold and compare it with the evaluation value, determine whether the digital vector is compliant, and obtain an evaluation result; The anomaly detection module is used to build a recognition model based on the anomaly detection algorithm, input the key concept text into the recognition model, and output the recognition result using the clustering algorithm.

[0013] In a third aspect, the present invention provides a computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: when the computer program is executed by the processor, any step of the data security compliance assessment and identification method described in the first aspect of the present invention is implemented.

[0014] In a fourth aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein: when the computer program is executed by a processor, it implements any step of the data security compliance assessment and identification method as described in the first aspect of the present invention.

[0015] The beneficial effects of the present invention are as follows: by collecting regulatory text data through an API interface, and using a requests library to send an HTTP protocol request to an API endpoint to obtain regulatory text data, automation and real-time updating are achieved, the timeliness and accuracy of regulatory text data are ensured, and changes in laws and regulations can be quickly responded to, thereby improving the timeliness and compliance of the system; by adopting a pre-calculated word embedding method, in particular, using a Word2Vec model to perform word segmentation on formatted regulatory text data, all independent words after word segmentation are collected to form a non-repetitive vocabulary, which not only improves the consistency and stability of text representation, but also enhances the recognition model's ability to understand semantic information, making subsequent evaluation and recognition more accurate; by adopting a named entity recognition (NER) model to scan formatted regulatory text data, entities are identified and assigned different category labels to them, text information with annotations is output, and key concept text is formed; the process accurately extracts key terms in the regulatory text, ensuring the pertinence and accuracy of subsequent anomaly detection, enabling the recognition method to focus on important regulatory clauses and concepts, and improving the effectiveness and practicality of the recognition results. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0017] Figure 1This is a flow chart of the data security compliance assessment and identification method in Example 1; Figure 2 Schematic diagram of the data security compliance assessment and identification system in Example 1. DETAILED DESCRIPTION

[0018] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are described in detail below in conjunction with the accompanying drawings.

[0019] In the following description, many specific details are set forth to facilitate a full understanding of the present invention, but the present invention may also be implemented in other ways different from those described herein, and those skilled in the art may make similar generalizations without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0020] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The term "in one embodiment" that appears in different places in this specification does not necessarily refer to the same embodiment, nor does it refer to a separate or selective embodiment that is mutually exclusive with other embodiments.

[0021] Example 1, reference Figure 1 and Figure 2 , which is the first embodiment of the present invention, provides a data security compliance assessment and identification method, including the following steps: S1. Collect regulatory text data through the API interface, and use the natural language processing library NLTK to pre-process the regulatory text data to obtain formatted regulatory text data; Furthermore, we use the requests library to send HTTP protocol requests to the API endpoint to obtain regulatory text data; Use the natural language processing library NLTK to filter meaningless words in regulatory text data and reduce noise; Use NLTK's built-in stop word list to check each word and remove it from the regulatory text data if it is a stop word; If the regulatory text data contains HTML formatted content, remove the HTML tags to retain the plain text; Restore the words in the regulatory text data and segment the text in the regulatory text data into separate words; Use NLTK's WordNetLemmatizer to perform lemmatization and segment continuous text into separate words or tokens to obtain formatted regulatory text data; It should be noted that in the process of collecting regulatory text data through the API interface and preprocessing the regulatory text data using the natural language processing library NLTK, the HTTP protocol request is sent to the API endpoint using the requests library to obtain the regulatory text data. This process not only ensures the reliability and real-time nature of the data source, but also denoises and standardizes the regulatory text data, effectively removes irrelevant information, improves the efficiency and accuracy of subsequent processing, and ensures the quality and consistency of the regulatory text data. In particular, steps such as stop word filtering, HTML tag removal, and word form restoration help reduce noise and retain key information, providing a high-quality data foundation for subsequent analysis.

[0022] S2, using a pre-computed word embedding method to map each word in the formatted regulatory text data into a fixed-length vector and generate a digital vector; Furthermore, the word embedding library Word2Vec is used to perform word segmentation on the formatted regulatory text data, and all independent words after word segmentation are collected to form a non-repetitive vocabulary; Create an empty collection and fill it with a vocabulary, count the frequency of occurrence of independent words, and sort the vocabulary from high frequency to low frequency according to the frequency of occurrence of independent words to form the final vocabulary; For independent words in the final vocabulary, the corresponding fixed-length vector is searched in the word embedding library Word2Vec, and the corresponding fixed-length vector found is used as the representation of the independent word to form a word-level digital vector; Create a zero vector of the same dimension as the word-level digital vector as an accumulator, accumulate all word-level digital vectors, and obtain the corresponding word-level digital vector for each independent word in the final vocabulary; Make sure you have a final vocabulary that contains all the individual words and load the word embedding library Word2Vec; Iterate over each individual word in the final vocabulary, check if it exists in the word embedding library Word2Vec, if so, get its vector, if not, replace it with a zero vector, and create a dictionary to store the word-level numeric vector of each word; Add each word-level digital vector to the accumulator one by one, and divide each word-level digital vector in the accumulator by the total number of independent words to get the average word vector. If the vocabulary is empty, return a zero vector. The average word vector obtained is used as the representation of the document-level digital vector to obtain the document-level digital vector and mark it as a digital vector. The expression is: ; in, Is formatted regulatory text data The number of words in Is formatted regulatory text data The words in It is The word vector of each word, Is formatted regulatory text data A numeric vector of is formatted regulatory text data; It should be noted that the use of pre-computed word embedding methods to map each word in the formatted regulatory text data into a fixed-length vector can not only capture the semantic relationship between words, but also represent the core features of the entire regulatory text by generating document-level digital vectors, ensuring the numerical operability of text information and the effectiveness of model training. When creating the final vocabulary, independent words are sorted according to their frequency of occurrence, so that high-frequency words can be retained first, further optimizing the quality of word vectors. The calculation of the average word vector enables texts of different lengths to be compared in a unified vector space, thereby improving the recognition model's ability to understand texts of different sizes.

[0023] S3, using named entity recognition (NER) method to mark important terms in the digital vector and obtain key concept text; Furthermore, a named entity recognition (NER) model is used to input the formatted regulatory text data sentence by sentence or paragraph by paragraph into the named entity recognition (NER) model; The named entity recognition (NER) model scans the formatted regulatory text data and identifies entities; For the identified entities, the named entity recognition (NER) model assigns different category labels to them and outputs annotated text information and entity types. ; Create a new text string, including the original text and annotation information, marking the original text with annotations as important terms in the text; Mark the output annotated text information as important terms in the text, create a list or dictionary, store all entities and their category labels as important terms in the text, and form key concept text. The expression is: ; in, is the word set after segmentation, Formatted regulatory text data Key concepts in the text, For the The entity type corresponding to the word, is formatted regulatory text data; It should be noted that the named entity recognition (NER) method is used to mark important terms in digital vectors, which can accurately extract key concepts in regulatory texts, such as legal names, agency names, dates, etc. Scanning texts and assigning category labels through the NER model not only enhances the structuring of text information, but also facilitates subsequent analysis and application. The annotated text information not only intuitively displays the important terms in the text, but also facilitates users to quickly locate and understand key content, thereby improving the accuracy and efficiency of interpreting regulatory texts. The key concept text formed serves as the basis for subsequent processing and can better support compliance assessments and other advanced analysis tasks.

[0024] S4. Establish an evaluation model based on a machine learning algorithm, input the digital vector into the evaluation model, and use a weighted average method to calculate the output evaluation value; Furthermore, the formatted regulatory text data Divide into training set and validation set; Use the scikit-learn library to import the LogisticRegression class to create a logistic regression instance and combine it with the formatted regulatory text data Build an evaluation model and set the initial parameter weights of the evaluation model ; A numeric vector containing the regulatory text data to be formatted Input into the evaluation model and calculate the evaluation value using the weighted average method , the expression is: ; in, is the word weight, It is The weight of the word, It is The word vector of each word, is the Sigmoid activation function, is the evaluation value, is the formatted regulatory text data, is a numeric vector of formatted regulatory text data; It should be noted that by establishing an evaluation model based on a machine learning algorithm, calculating the output evaluation value through the weighted average method, and setting the evaluation threshold for comparison with the evaluation value, it is possible to achieve automated judgment of the compliance of regulatory text data. The logistic regression algorithm is combined with the formatted regulatory text data to construct an efficient binary classification model that can distinguish between compliant and non-compliant texts.

[0025] S5. Set an evaluation threshold and compare it with the evaluation value to determine whether the digital vector is compliant and obtain an evaluation result; Furthermore, the threshold is set based on the formatted regulatory text data in the validation set that has never been used in training. , including selecting formatted regulatory text data that has never been involved in training as a validation set, inputting the digital vector of the validation set into the trained evaluation model, obtaining the evaluation value of each sample, performing k-fold cross validation on the validation set, and determining the threshold ; when ≥ , representing formatted regulatory text data A numeric vector in Below the set threshold , then the regulatory text is judged to be compliant; when < , determine the formatted regulatory text data A numeric vector in If it is not compliant, the formatted regulatory text data Make the modification, and after the modification, recalculate the formatted regulatory text data according to the digital vector expression A numeric vector in , re-enter it into the evaluation model to calculate the evaluation value, and set the number of iterations to , until ≥ ; It should be noted that the dynamic adjustment of the assessment threshold ensures the adaptability and robustness of the assessment model, especially when faced with new regulatory texts or changing compliance standards, it can respond flexibly. The process of recalculating the assessment value ensures the accuracy and reliability of the assessment results, making the compliance assessment of regulatory texts more scientific and reasonable.

[0026] S6. Build a recognition model based on anomaly detection algorithm, input key concept text into the recognition model, and use clustering algorithm to output recognition results; Furthermore, a recognition model is constructed based on the Isolation Forest Anomaly Detection Algorithm and the formatted regulatory text data, including importing the scikit-learn library for machine learning in Python, ensuring that the key concept text is converted into a numerical feature vector through TF-IDF, normalizing the numerical feature vector, creating an Isolation Forest model instance, and setting parameters, and training the Isolation Forest model using the formatted regulatory text data to obtain a recognition model; The isolation forest algorithm is used to isolate observation points by building multiple decision trees, and the IsolationForest class in the Python library is used to initialize the recognition model parameters; The key concept text in the formatted legal text data is converted into a TF-IDF value, the TF-IDF vector value of the key concept text in the formatted legal text data is calculated, and the key concept text in the formatted legal text data represented by the TF-IDF vector value is standardized. The expression is: ; in, It's words. Formatted regulatory text data Key concepts in the text, is all the key concept text, is the total number of documents in all key concept texts, Contains words The number of documents, It's a word In formatting regulatory text data The frequency of key concepts in the text, for TF-IDF vector value of ; The TF-IDF vector value is input into the recognition model, and the K-means clustering algorithm is used to classify the output of the recognition model. The expression is: ; in, is the cluster label, is the anomaly score matrix, Represents the anomaly score matrix Middle The anomaly score vector of samples, It is The center point of the cluster, is the set number of clusters, Formatted regulatory text data Key concepts in the text, Represents key text concepts The anomaly score, is the number of trees in the isolation forest algorithm, is the anomaly score matrix The elements in It is Clusters, is the isolation forest algorithm The path length, is the maximum path length of all trees in the isolation forest, yes The expected value of , that is, the average value of the path length; Setting anomaly score threshold With the anomaly score matrix Make a comparison; when ≥ , indicates formatted regulatory text data Key text concepts in Non-compliance; when < , indicates formatted regulatory text data Key text concepts in Compliance; It should be noted that by building a recognition model based on the anomaly detection algorithm and analyzing the key concept text through the isolation forest and K-means clustering algorithms, anomalies in the regulatory text can be effectively identified. The isolation forest algorithm can efficiently detect potential anomalies by constructing multiple decision trees to isolate observation points. The K-means clustering algorithm classifies these anomalies to help determine which key concept texts do not comply with regulations, providing clear judgment criteria for setting anomaly score thresholds, ensuring the accuracy and reliability of the recognition results, and being able to not only detect non-compliant parts in regulatory texts, but also provide valuable feedback for improving the texts, thereby improving the overall quality of regulatory texts.

[0027] This embodiment also provides a data security compliance assessment and identification system, including: Data processing module, word embedding generation module, key concept extraction module, evaluation model module and anomaly detection module; The data processing module is used to collect regulatory text data through the API interface and pre-process the regulatory text data using the natural language processing library NLTK to obtain formatted regulatory text data; A word embedding generation module, used for mapping each word in the formatted regulatory text data into a fixed-length vector using a pre-computed word embedding method, and generating a digital vector; The key concept extraction module is used to mark the important terms in the digital vector using the named entity recognition method to obtain the key concept text; The evaluation model module is used to establish an evaluation model based on a machine learning algorithm, input a digital vector into the evaluation model, calculate the output evaluation value using a weighted average method, set an evaluation threshold and compare it with the evaluation value to determine whether the digital vector is compliant and obtain the evaluation result; The anomaly detection module is used to build a recognition model based on the anomaly detection algorithm, input the key concept text into the recognition model, and use the clustering algorithm to output the recognition result.

[0028] This embodiment also provides a computer device, which is suitable for the data security compliance assessment and identification method, including: a memory and a processor; the memory is used to store computer executable instructions, and the processor is used to execute computer executable instructions to implement the data security compliance assessment and identification method proposed in the above embodiment.

[0029] The computer device may be a terminal, and the computer device includes a processor, a memory, a communication interface, a display screen, and an input device connected through a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be achieved through Wi-Fi, an operator network, NFC (near field communication) or other technologies. The display screen of the computer device may be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device may be a touch layer covering the display screen, or a key, trackball or touchpad provided on the housing of the computer device, or an external keyboard, touchpad or mouse, etc.

[0030] This embodiment also provides a storage medium on which a computer program is stored. When the program is executed by a processor, the data security compliance assessment and identification method proposed in the above embodiment is implemented; the storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (Static Random Access Memory, referred to as SRAM), electrically erasable programmable read-only memory (Electrically Erasable Programmable Read-Only Memory, referred to as EEPROM), erasable programmable read-only memory (Erasable Programmable Read Only Memory, referred to as EPROM), programmable read-only memory (Programmable Red-Only Memory, referred to as PROM), read-only memory (Read-Only Memory, referred to as ROM), magnetic storage, flash memory, magnetic disk or optical disk.

[0031] In summary, the present invention collects legal text data through an API interface, and uses the requests library to send HTTP protocol requests to the API endpoint to obtain legal text data, thereby realizing automation and real-time updating, ensuring the timeliness and accuracy of legal text data, and being able to quickly respond to changes in laws and regulations, thereby improving the timeliness and compliance of the system. By adopting a pre-calculated word embedding method, in particular, using the Word2Vec model to perform word segmentation on the formatted legal text data, all independent words after word segmentation are collected to form a non-repetitive vocabulary, which not only improves the consistency and stability of text representation, but also enhances the recognition model's ability to understand semantic information, making subsequent evaluation and recognition more accurate. By adopting a named entity recognition (NER) model to scan the formatted legal text data, entities are identified and assigned different category labels to them, text information with annotations is output, and key concept text is formed. The process accurately extracts key terms in the legal text, ensuring the pertinence and accuracy of subsequent anomaly detection, enabling the recognition method to focus on important legal clauses and concepts, and improving the effectiveness and practicality of the recognition results.

[0032] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present invention, which should all be included in the scope of the claims of the present invention.

Claims

1. A data security compliance assessment and identification method, characterized by: include: Collect regulatory text data through the API interface, and use the natural language processing library NLTK to pre-process the regulatory text data to obtain formatted regulatory text data; Using a pre-computed word embedding method, each word in the formatted regulatory text data is mapped into a fixed-length vector and a digital vector is generated; The named entity recognition (NER) method is used to mark the important terms in the digital vector and obtain the key concept text; An evaluation model is established based on a machine learning algorithm, the digital vector is input into the evaluation model, and the output evaluation value is calculated using the weighted average method; Set the evaluation threshold and compare it with the evaluation value to determine whether the digital vector is compliant and obtain the evaluation result; A recognition model is built based on the anomaly detection algorithm, the key concept text is input into the recognition model, and the clustering algorithm is used to output the recognition results.

2. The data security compliance assessment and identification method according to claim 1, characterized in that: The regulatory text data is collected through the API interface, and the regulatory text data is preprocessed using the natural language processing library NLTK to obtain formatted regulatory text data. The specific steps are: Use the requests library to send HTTP protocol requests to the API endpoint to obtain regulatory text data; Use the natural language processing library NLTK to filter meaningless words in regulatory text data and reduce noise; The words in the regulatory text data are restored, and the text in the regulatory text data is segmented into separate words to obtain formatted regulatory text data.

3. The data security compliance assessment and identification method according to claim 2, characterized in that: The pre-calculated word embedding method is used to map each word in the formatted regulatory text data into a fixed-length vector and generate a digital vector. The specific steps are as follows: The word embedding library Word2Vec is used to perform word segmentation on the formatted regulatory text data, and all independent words after word segmentation are collected to form a non-repetitive vocabulary; Create an empty collection and fill it with a vocabulary, count the frequency of occurrence of independent words, sort the vocabulary according to the frequency of occurrence of independent words, and form a final vocabulary; For independent words in the final vocabulary, the corresponding fixed-length vector is searched in the word embedding library Word2Vec, and the corresponding fixed-length vector found is used as the representation of the independent word to form a word-level digital vector; Create a zero vector of the same dimension as the word-level digital vector as an accumulator, accumulate all word-level digital vectors, and obtain the corresponding word-level digital vector for each independent word in the final vocabulary; Add each word-level digital vector to the accumulator one by one to get the average word vector; The obtained average word vector is used as the representation of the document-level digital vector to obtain the document-level digital vector and mark it as a digital vector.

4. The data security compliance assessment and identification method according to claim 3, characterized in that: The named entity recognition (NER) method is used to mark important terms in the digital vector to obtain key concept texts. The specific steps are as follows: Using a named entity recognition (NER) model, input the formatted regulatory text data sentence by sentence or paragraph by paragraph into the named entity recognition (NER) model; The named entity recognition (NER) model scans the formatted regulatory text data and identifies entities; For the identified entities, the named entity recognition (NER) model assigns different category labels to them and outputs the annotated text information; The output annotated text information is marked as important terms in the text and forms a key concept text.

5. The data security compliance assessment and identification method according to claim 4, characterized in that: The evaluation model is established based on the machine learning algorithm, the digital vector is input into the evaluation model, and the weighted average method is used to calculate the output evaluation value. The specific steps are as follows: The regulatory text data to be formatted Divide into training set and validation set; Use the scikit-learn library to import the LogisticRegression class to create a logistic regression instance and combine it with the formatted regulatory text data Build an evaluation model and set the initial parameter weights of the evaluation model ; A numeric vector containing the regulatory text data to be formatted Input into the evaluation model and calculate the evaluation value using the weighted average method , the expression is: ； in, is the word weight, It is The weight of the word, It is The word vector of each word, is the Sigmoid activation function, is the evaluation value, is the formatted regulatory text data, A numeric vector containing the formatted regulatory text data.

6. The data security compliance assessment and identification method according to claim 5, characterized in that: The evaluation threshold is set to be compared with the evaluation value to determine whether the digital vector is compliant and obtain the evaluation result. The specific steps are as follows: Thresholds are set based on formatted regulatory text data in the validation set that was never used in training ; when ≥ , representing formatted regulatory text data A numeric vector in Below the set threshold , then the regulatory text is judged to be compliant; when < , determine the formatted regulatory text data A numeric vector in If it is not compliant, the formatted regulatory text data Make the modification, and after the modification, recalculate the formatted regulatory text data according to the digital vector expression A numeric vector in , re-enter the evaluation model to calculate the evaluation value, and set the number of iterations to , until ≥ .

7. The data security compliance assessment and identification method according to claim 6, characterized in that: The recognition model is constructed based on the anomaly detection algorithm, key concept text is input into the recognition model, and the recognition result is output using the clustering algorithm. The specific steps are as follows: Building a recognition model based on the Isolation Forest Anomaly Detection Algorithm and formatted regulatory text data; The isolation forest algorithm is used to isolate observation points by building multiple decision trees, and the IsolationForest class in the Python library is used to initialize the recognition model parameters; The key concept text in the formatted legal text data is converted into a TF-IDF value, the TF-IDF vector value of the key concept text in the formatted legal text data is calculated, and the key concept text in the formatted legal text data represented by the TF-IDF vector value is standardized. The expression is: ； in, It's words. Formatted regulatory text data Key concepts in the text, is all the key concept text, is the total number of documents in all key concept texts, Contains words The number of documents, It's a word In formatting regulatory text data The frequency of key concepts in the text, for TF-IDF vector value of ; The TF-IDF vector value is input into the recognition model, and the K-means clustering algorithm is used to classify the output of the recognition model. The expression is: ； in, is the cluster label, is the anomaly score matrix, Represents the anomaly score matrix Middle The anomaly score vector of samples, It is The center point of the cluster, is the set number of clusters, Formatted regulatory text data Key concepts in the text, Represents key text concepts The anomaly score, is the number of trees in the isolation forest algorithm, is the anomaly score matrix The elements in It is Clusters, is the isolation forest algorithm The path length, is the maximum path length of all trees in the isolation forest, yes The expected value of , that is, the average value of the path length; Setting anomaly score threshold With the anomaly score matrix Make a comparison; when ≥ , indicates formatted regulatory text data Key text concepts in Non-compliance; when < , indicates formatted regulatory text data Key text concepts in Compliance.

8. A data security compliance assessment and identification system, based on the data security compliance assessment and identification method according to any one of claims 1 to 7, characterized in that: include: Data processing module, word embedding generation module, key concept extraction module, evaluation model module and anomaly detection module; The data processing module is used to collect regulatory text data through an API interface, and pre-process the regulatory text data using a natural language processing library NLTK to obtain formatted regulatory text data; The word embedding generation module is used to map each word in the formatted regulatory text data into a fixed-length vector using a pre-calculated word embedding method and generate a digital vector; The key concept extraction module is used to mark important terms in the digital vector using a named entity recognition method to obtain a key concept text; The evaluation model module is used to establish an evaluation model based on a machine learning algorithm, input a digital vector into the evaluation model, calculate an output evaluation value using a weighted average method, set an evaluation threshold and compare it with the evaluation value, determine whether the digital vector is compliant, and obtain an evaluation result; The anomaly detection module is used to build a recognition model based on the anomaly detection algorithm, input the key concept text into the recognition model, and output the recognition result using the clustering algorithm.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the data security compliance assessment and identification method described in any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the data security compliance assessment and identification method described in any one of claims 1 to 7 are implemented.

Images