A method, device and storage medium for adaptive data classification and grading

Abstract

The present application discloses an adaptive data classification and grading method, device and storage medium. The method includes: using network protocols to automatically identify and discover available data resources; preliminarily identifying target data of data resources and determining attribute content information related to the target data; using an adaptive algorithm to determine a sensitive data identification algorithm corresponding to the target data according to the determined attribute content information, and using the determined sensitive data identification algorithm to identify sensitive data corresponding to the target data; and classifying and grading the target data according to the sensitive data.

Description

Technical Field

[0001] The present application relates to the field of data security technology, and in particular to an adaptive data classification and grading method, device and storage medium. Background Art

[0002] Data security is now a focus of attention for the country and enterprises and institutions. Data classification and grading is the first step in data security governance and a prerequisite for doing a good job in data security.

[0003] In the prior art, the paper "Research on Automated Paths for Enterprise Data Classification and Grading" ("Big Data and Data Technology" 2024 Issue 6) discloses automated technical solutions for data detection, data preprocessing, sensitive data identification, classification and grading. However, due to the lack of uniformity in the classification and grading of various data in different industries and types of enterprises (for example, the data classification and grading standards of government units and industrial enterprises are completely different, and there are also huge differences between industrial enterprises in different industries), methods that rely on existing technical means still need to rely on a lot of manual configuration to ensure the accuracy of the final classification and grading results.

[0004] Invention application CN117972090A discloses a method for automatic classification and grading of government data based on a large model, defines the rules for data classification and grading, learns the definition of data classification and grading rules through a large model, and then analyzes and understands the input data through a large model, determines the classification and grading rule features that the input data conforms to, and combines the corresponding relationship between data categories, data sensitivity levels and classification and grading rule features to obtain the classification and grading results of the data. This invention solves the problem that most of the current data classification and grading methods are manual customized services, with high labor costs and low efficiency; or by mapping rules between fixed data and classification and grading results, classification and grading are performed by matching data keywords, resulting in a small matching range due to lack of semantic understanding, and fixed data and classification and grading results lead to separation from the actual application scenario of the data, and a high secondary modification rate. However, due to the inherent problems of large computing power consumption and long processing time of large language models, and they are not good at processing massive unstructured data other than text, this invention cannot cope with a variety of application scenarios, such as being unable to effectively cope with scenarios with massive unstructured data.

[0005] The large language models existing in the above-mentioned existing technologies have inherent high computing power consumption and long processing time, and are not good at processing massive unstructured data other than text. They are unable to effectively deal with technical problems in application scenarios with massive unstructured data, and no effective solution has been proposed so far. Summary of the invention

[0006] The present invention provides an adaptive data classification and grading method, which fully utilizes the advantages of the current large language model in text understanding, while avoiding its inherent defects of high computing power consumption and large data processing delay. The method introduces adaptive technology and can effectively deal with the problem of data grading and classification of large-scale enterprises and institutions.

[0007] The embodiments of the present disclosure provide an adaptive data classification and grading method to at least solve the technical problems that the large language models existing in the prior art have inherent high computing power consumption and long processing time, are not good at processing massive unstructured data other than text, and cannot effectively cope with application scenarios with massive unstructured data.

[0008] According to one aspect of an embodiment of the present disclosure, there is provided an adaptive data classification and grading method, including: utilizing network protocols to automatically identify and discover available data resources; preliminarily identifying target data of the data resources, and determining attribute content information related to the target data; utilizing an adaptive algorithm to determine a sensitive data identification algorithm corresponding to the target data based on the determined attribute content information, and utilizing the determined sensitive data identification algorithm to identify sensitive data corresponding to the target data; and classifying and grading the target data based on the sensitive data.

[0009] According to another aspect of the embodiments of the present disclosure, a storage medium is further provided. The storage medium includes a stored program, wherein the above method is executed by a processor when the program is running.

[0010] According to another aspect of the embodiments of the present disclosure, an adaptive data classification and grading device is also provided, including: a data resource discovery module, used to automatically identify and discover available data resources using network protocols; an attribute content determination module, used to preliminarily identify target data of the data resources, and determine attribute content information related to the target data; a sensitive data identification module, used to adopt an adaptive algorithm to determine a sensitive data identification algorithm corresponding to the target data according to the determined attribute content information, and use the determined sensitive data identification algorithm to identify sensitive data corresponding to the target data; and a classification and grading module, used to classify and grade the target data according to the sensitive data.

[0011] According to another aspect of the embodiments of the present disclosure, an adaptive data classification and grading device is also provided, including: a processor; and a memory connected to the processor, for providing the processor with instructions for processing the following processing steps: using network protocols to automatically identify and discover available data resources; preliminarily identifying target data of the data resources, and determining attribute content information related to the target data; using an adaptive algorithm to determine a sensitive data identification algorithm corresponding to the target data based on the determined attribute content information, and using the determined sensitive data identification algorithm to identify sensitive data corresponding to the target data; and classifying and grading the target data based on the sensitive data.

[0012] In the disclosed embodiments, the accuracy and efficiency of data classification and grading are improved through steps such as active data source detection, database data identification, file type identification, sensitive data identification, and automatic clustering. In particular, through adaptive technology, the optimal processing method can be selected according to different data types and features, thereby significantly reducing computing power consumption and processing delays, and improving the overall efficiency of data management. This solves the technical problem that the large language model existing in the prior art has inherent high computing power consumption and long processing time, is not good at processing massive unstructured data other than text, and cannot effectively cope with application scenarios with massive unstructured data. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] The drawings described herein are used to provide a further understanding of the present disclosure and constitute a part of the present application. The illustrative embodiments of the present disclosure and their descriptions are used to explain the present disclosure and do not constitute an improper limitation on the present disclosure. In the drawings:

[0014] Figure 1 is a hardware structure block diagram of a computing device for implementing the method according to Embodiment 1 of the present disclosure;

[0015] Figure 2 is a schematic diagram of a system for implementing the adaptive data classification and grading method described in Example 1 of the present disclosure;

[0016] Figure 3 is a flowchart of the adaptive data classification and grading method according to the first aspect of Embodiment 1 of the present disclosure;

[0017] Figure 4 is a flowchart of specific steps of step S306 in the method according to the first aspect of embodiment 1 of the present disclosure;

[0018] Figure 5 is a flowchart of the step of classifying and grading the target data by using automatic clustering in the method according to the first aspect of Embodiment 1 of the present disclosure;

[0019] Figure 6is a schematic diagram of the adaptive data classification and grading device according to Embodiment 2 of the present disclosure; and

[0020] Figure 7 It is a schematic diagram of the adaptive data classification and grading device described in Example 3 of the present disclosure. DETAILED DESCRIPTION

[0021] In order to enable those skilled in the art to better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are only embodiments of a part of the present disclosure, not all of the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in this field without making creative work should fall within the scope of protection of the present disclosure.

[0022] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product, or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products, or devices.

[0023] Example 1

[0024] According to this embodiment, a method embodiment of an adaptive data classification and grading method is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0025] The method embodiment provided in this embodiment can be executed in a server or a similar computing device. Figure 1 FIG. 1 shows a hardware structure block diagram of a computing device for implementing an adaptive data classification and grading method. Figure 1As shown, the computing device may include one or more processors (the processor may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA), a memory for storing data, a transmission device for communication functions, and an input / output interface. The memory, the transmission device, and the input / output interface are connected to the processor via a bus. In addition, it may also include: a display, a keyboard, and a cursor control device connected to the input / output interface. A person skilled in the art can understand that Figure 1 The structure shown is only for illustration and does not limit the structure of the above electronic device. Figure 1 More or fewer components as shown, or with Figure 1 Different configurations are shown.

[0026] It should be noted that the one or more processors and / or other data processing circuits described above may generally be referred to herein as "data processing circuitry". The data processing circuitry may be embodied in whole or in part as software, hardware, firmware, or any other combination thereof. In addition, the data processing circuitry may be a single independent processing module, or may be incorporated in whole or in part into any of the other components of the computing device. As involved in the embodiments of the present disclosure, the data processing circuitry acts as a processor control (e.g., selection of a variable resistor terminal path connected to an interface).

[0027] The memory can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the adaptive data classification and grading method in the embodiment of the present disclosure. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory, that is, the adaptive data classification and grading method of the above-mentioned application is realized. The memory may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory may further include a memory remotely arranged relative to the processor, and these remote memories may be connected to the computing device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0028] The transmission device is used to receive or send data via a network. The specific example of the above network may include a wireless network provided by a communication provider of the computing device. In one example, the transmission device includes a network adapter (Network Interface Controller, NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device can be a radio frequency (Radio Frequency, RF) module, which is used to communicate with the Internet wirelessly.

[0029] The display may be, for example, a touch screen liquid crystal display (LCD) that may enable a user to interact with a user interface of the computing device.

[0030] It should be noted that, in some optional embodiments, the above Figure 1 The computing device shown may include hardware elements (including circuits), software elements (including computer code stored on a computer-readable medium), or a combination of both hardware elements and software elements. Figure 1 This is merely one example of a particular embodiment and is intended to illustrate the types of components that may be present in the computing devices described above.

[0031] Figure 2 is a schematic diagram of a system according to the adaptive data classification and grading method described in this embodiment. Figure 2 As shown, the system includes: a data security system 100 (hereinafter referred to as "system") and servers 201-20n as data resources. The data security system 100 is used to classify and grade the data of the servers 201-20n (hereinafter referred to as "host"). It should be noted that the data security system 100 in the system can be applied to the hardware structure described above.

[0032] Under the above operating environment, according to the first aspect of this embodiment, a method for adaptive data classification and grading is provided. Figure 2 The data security system 100 shown in FIG. Figure 3 A schematic diagram showing the process of the method is shown in FIG. Figure 3 As shown, the method includes:

[0033] S302: Automatically identify and discover available data resources using network protocols;

[0034] S304: Preliminarily identify the target data of the data resource and determine the attribute content information related to the target data;

[0035] S306: using an adaptive algorithm to determine a sensitive data identification algorithm corresponding to the target data according to the determined attribute content information, and using the determined sensitive data identification algorithm to identify the sensitive data corresponding to the target data; and

[0036] S308: Classify and grade the target data according to sensitive data.

[0037] Specifically, in step S302, the data security system 100 performs selective network scanning based on ICMP and TCP protocols to automatically discover data resources, and the functions include host discovery, port scanning and version detection. Among them, active data source detection is one of the key steps of the present invention, which aims to automatically identify and discover available data resources through the effective use of network protocols. The process is mainly based on two network protocols: ICMP (Internet Control Message Protocol) and TCP (Transmission Control Protocol) to achieve efficient network scanning and data source detection.

[0038] The ICMP protocol is a connectionless network protocol, mainly used for network detection and fault diagnosis. By sending an ICMP echo request (Echo Request) message to the target host (for example, server 201), the system can determine whether the host is online and measure the round-trip time (RTT). In specific implementation, the system will construct an ICMP request packet and send it to the target IP address, and then wait to receive an ICMP echo reply (Echo Reply). If the target host is online and working normally, it will return a reply packet to confirm its existence.

[0039] The TCP protocol is a connection-oriented protocol suitable for detecting application services. Through the TCP three-way handshake process, the system can determine the open status of a specific port on the target host. The implementation method is: the system sends a TCP SYN (synchronization) message to the specified port of the target host. If the target host responds with a TCP SYN-ACK (synchronization response) message, it means that the port is open and the system can further perform service detection.

[0040] Active data source detection mainly includes the following three aspects:

[0041] 1) Host discovery: Through the echo request of ICMP protocol, the system can quickly identify the surviving hosts in the network. You can set an IP address range, and the system will send ICMP requests to each IP address in the range and record all the hosts that respond.

[0042] 2) Port scanning: Using the TCP protocol, the system can perform port scanning on discovered hosts and identify open service ports. By sending TCP SYN requests to common ports (such as 80, 443, 22, etc.) of each host, the system can determine which ports are open and record the corresponding service information.

[0043] 3) Version detection: After identifying the open port, the system can further perform version detection to obtain the specific version information of the service. This is usually achieved by sending a specific request packet and analyzing the returned response. For example, for an HTTP service, the system can send an HTTP GET request and parse the returned HTTP header information to extract the server version of the host.

[0044] In the specific implementation process, first configure the network scanning tool and set the target IP range. Then, perform ICMP detection and record all responding hosts. Next, perform TCP port scans on each surviving host to identify open ports. Finally, perform version detection on open ports to obtain service version information. In this way, the system can comprehensively and quickly identify data resources in the network, laying the foundation for subsequent data classification and grading.

[0045] For example, suppose you want to scan the 192.168.1.0 / 24 network. First, send an ICMP request to each IP address in the range and record the hosts that respond. Suppose you find that 192.168.1.10 and 192.168.1.20 are online. Next, perform a TCP port scan on these two hosts and find that port 80 and port 22 of 192.168.1.10 are open, while port 80 of 192.168.1.20 is open. Finally, perform version detection on port 80 of 192.168.1.10, send an HTTP request, get the response header information, and confirm that it is Apache version 2.4.41. Through this series of steps, the system successfully identifies the data resources in the network and provides basic data for subsequent data processing.

[0046] Then, the operation of step S304 can be divided into the following two aspects:

[0047] S3041: Database Data Identification: Aims to automatically identify readable databases and tables in the target database through efficient protocol identification technology, and perform data sampling to support subsequent data classification and grading; and / or

[0048] S3042: File type identification: Aims to automatically identify the file type on the target server through efficient file reading and encoding recognition technology, and provide support for subsequent data processing.

[0049] The database data identification in step S3041 includes the following sub-steps:

[0050] 1. Establishing JDBC connection: First, the system (i.e., data security system 100) establishes a connection with the target database (e.g., the database in servers 201-20n) through the Java Database Connection (JDBC) protocol. In specific implementation, the system will configure the corresponding JDBC driver according to the type of the target database (such as MySQL, PostgreSQL, Oracle, etc.), and use the connection string (including database URL, user name, and password) to connect. After the connection is successful, the system will obtain access rights to the database.

[0051] 2. Protocol identification engine initialization: After the connection is established, the system will initialize the protocol identification engine. The main function of this engine is to scan all databases and tables with readable permissions in the target database. By executing SQL query statements (such as SHOWDATABASES and SHOWTABLES), the system can obtain information about all databases and tables accessible to the current user.

[0052] 3. Adaptive random sampling: In order to effectively identify data, the system will use adaptive random sampling. The specific steps are as follows:

[0053] 1) Dynamic sampling strategy: The system dynamically adjusts the sampling ratio according to the size of the table and the data distribution. For example, for a large table, the system can choose a smaller sampling ratio (such as 1%), and for a small table, it can choose a larger ratio (such as 10%).

[0054] 2) Random sampling implementation: The system will use a random number generator to randomly select several rows of data from each table for sampling. You can use the SQL ORDER BY RAND() statement to implement random row selection, or implement a more efficient random sampling algorithm through programming.

[0055] 3) Data Identification: After sampling is completed, the system will analyze the sampled data to identify the type, format and content of the data. This can be achieved through technologies such as data type detection, format identification and content analysis.

[0056] 4. Result recording and feedback: Finally, the system will record the information of the identified database and table, including table name, field name, data type, sampled data, etc., and feed this information back to the subsequent data processing module for further classification and grading.

[0057] In the specific implementation process, the data security system 100 first configures the JDBC connection parameters to ensure that it can successfully connect to the target database. Then, the protocol recognition engine is initialized and SQL queries are executed to obtain databases and tables with readable permissions. Then, adaptive random sampling is implemented for each table, the sampling ratio is dynamically adjusted, and data rows are randomly selected. Finally, the recognition results are recorded and passed to subsequent modules.

[0058] For example, suppose we connect to a MySQL database. First, we use the JDBC connection string to connect to the database. After the connection is successful, we execute the SHOWDATABASES query and find that there are two readable databases: sales_db and inventory_db. Next, for the orders table in sales_db, the system decides to use a 5% sampling ratio based on the preset rules and randomly selects 100 records for analysis. By executing SELECT * FROM orders ORDERBYRAND() LIMIT100, the system obtains the randomly sampled data, analyzes its data type and content, and identifies the field types as int, varchar, and date, etc. Finally, the system records the recognition results to provide basic data support for subsequent data classification and grading.

[0059] In addition, the file type identification described in step S3042 may include the following sub-steps:

[0060] 1. Connect to the target server: First, the system (i.e., data security system 100) needs to establish a connection with the target server through means such as secure file transfer protocol (SFTP), file transfer protocol (FTP), or distributed file system (HDFS). In specific implementation, the system will configure the corresponding connection parameters according to the selected protocol, including server address, port, user name, and password. After the connection is successful, the system will obtain access rights to the target directory.

[0061] 2. Obtaining a file list: After the connection is established, the system will obtain a list of all files in the target directory. For SFTP and FTP protocols, you can use the LIST command to list the files in the directory; for HDFS, you can use the hdfs dfs-ls command to obtain file information. The system will record metadata such as the file name, size, and modification time for subsequent processing.

[0062] 3. Incremental file scanning: In order to improve efficiency and reduce the overhead of repeated scanning, the system will adopt an incremental file scanning strategy. The specific steps are as follows:

[0063] 1) Initial scan: When the system is connected for the first time, a comprehensive scan of the target directory will be performed to record the status of all files (such as file name, size, last modification time, etc.).

[0064] 2) Subsequent scans: In subsequent scans, the system will only check files that have changed since the last scan. By comparing the last modification time of the files, the system can identify newly added, modified or deleted files, thus avoiding repeated processing of unchanged files.

[0065] 4. Automatic identification of file codes: After identifying the file, the system will automatically identify the code of each file. The specific implementation methods include:

[0066] 1) Read file content: The system will select the appropriate reading method based on the file type. For example, for a text file, the content can be read line by line; for a binary file, it can be read block by block.

[0067] 2) Encoding detection: The system will use a preset encoding recognition algorithm (such as the chardet library or cchardet library) to analyze the file content and automatically identify the file's encoding format (such as UTF-8, ISO-8859-1, etc.). The recognition process usually involves counting the frequency of occurrence of characters in the file and matching them with the characteristics of known encodings.

[0068] 5. Result recording and feedback: Finally, the system will record the identified file information, including file name, file type, encoding format, etc., and feed this information back to the subsequent data processing module for further classification and grading.

[0069] In the specific implementation process, first configure the connection parameters to ensure that you can successfully connect to the target server. Then, obtain the list of files in the target directory and record the metadata of the files. Then, implement incremental file scanning to dynamically identify new or modified files. For each file, the system will read its content and perform encoding recognition, and finally record the recognition results and pass them to subsequent modules.

[0070] For example, suppose we connect to an FTP server and there are multiple files in the target directory. When we connect for the first time, the system obtains a list of files, including report.txt, data.csv, and image.png. In subsequent scans, the system finds that the last modification time of the report.txt file has changed, so it will re-read the contents of the file and perform encoding recognition. Through analysis, the system recognizes that the encoding format of report.txt is UTF-8 and records this information to support subsequent data processing.

[0071] Then, Figure 4 FIG. 4 is a schematic diagram showing the specific steps of step S306. Figure 4 As shown, the operation of step S306 includes:

[0072] S3061: Analyze the format and type of the target data, and determine the encoding format and identification strategy corresponding to the target data; and

[0073] S3062: Identify sensitive data in the target data using an identification algorithm corresponding to the determined identification strategy.

[0074] The operation of analyzing the format and type of the target data and determining the encoding format and identification strategy corresponding to the target data in step S3061 may include the following sub-steps:

[0075] S30611: Identify the data format of the target data, including: performing preliminary analysis on the input data to identify its format (such as text, image, audio, video, etc.). For text data, the file extension, MIME type or content features can be used for judgment; for image and audio data, the file header information can be used for identification.

[0076] S30612: After identifying the data format of the target data, further analyze the type of the target data. For example, text data can be subdivided into natural language text, code snippets, markup languages, etc.; image data can be divided into JPEG, PNG, GIF and other formats; audio data can be divided into MP3, WAV, AAC and other formats. This step can be achieved by building a data type dictionary, which contains various formats and their corresponding features.

[0077] S30613: Determine the encoding number of the target data, including: determining the encoding method of the target data according to the identified data format and type. For example, for text data, use UTF-8 or UTF-16 encoding; for image data, use JPEG compression or PNG lossless compression; for audio data, select a suitable sampling rate and bit rate for encoding. Use the following formula to calculate the storage size of the data:

[0078] Size=Duration×BitrateSize=Duration×Bitrate

[0079] Among them, Size is the storage size of the data, Duration is the duration of the audio or video, and Bitrate is the number of bits transmitted per second. Through this formula, the impact of different encoding methods on storage space can be evaluated.

[0080] S30614: Determine the recognition strategy for the target data. Specifically, after determining the data format and encoding method of the target data, select an appropriate recognition strategy. For text data, use regular expressions for pattern matching; for image data, use image recognition algorithms such as convolutional neural networks (CNN); for audio data, use acoustic feature extraction methods such as Mel-frequency cepstral coefficients (MFCC).

[0081] S30615: Determine the sensitive data identification algorithm corresponding to the identification strategy. Specifically, in order to realize the adaptive mechanism, the present invention proposes a modular identification framework, which can dynamically select the optimal identification algorithm according to the characteristics of the input data. In the specific implementation, in order to improve the processing efficiency, we adopt a dynamic decision tree model. The model branches according to the characteristics of the data (such as format, type, size, etc.), and finally selects a suitable algorithm for processing. The dynamic decision tree can flexibly adapt to data of different types and characteristics and automatically select the most appropriate processing method. In the data classification and grading scenarios of large industrial enterprises, the application of dynamic decision trees not only improves the efficiency and accuracy of data management, but also significantly reduces the need for manual intervention. This adaptive mechanism enables the system to operate efficiently in a complex and changeable industrial environment, ensuring that the sensitive information of the enterprise is effectively protected. Through the integration of feedback mechanisms, the system can be continuously optimized and adjusted to ensure its stability and reliability.

[0082] Specifically, the construction and application process of the dynamic decision tree model can be divided into the following sub-steps:

[0083] (I) Feature analysis and extraction

[0084] First, the system will conduct a comprehensive feature analysis and extraction of the input data. Feature analysis is the basis of the dynamic decision tree algorithm and determines the accuracy and efficiency of subsequent decisions. Features include but are not limited to:

[0085] Format: Identify the storage format of the data, such as text file (TXT), comma separated value file (CSV), JavaScript Object Notation (JSON), etc.

[0086] Type: Determine the type of data, which can be divided into structured data (such as database tables), unstructured data (such as free text), and semi-structured data (such as XML).

[0087] Size: Count the number of bytes or rows of data to assess the size of the data.

[0088] Content complexity: Use natural language processing techniques (such as TF-IDF) to quantify the complexity of the text.

[0089] Data sensitivity: Preliminary judgment of data sensitivity is made through predefined sensitive word libraries or pattern recognition algorithms.

[0090] Data distribution: Analyze the statistical distribution characteristics of data, such as mean, variance, skewness, and kurtosis.

[0091] Data completeness: Evaluate the completeness and consistency of the data, such as the proportion of missing values ​​and duplication rate.

[0092] Data correlation: Analyze the correlation between data features, such as the Pearson correlation coefficient.

[0093] Data distribution bias: Assess the degree of bias in the data distribution, such as the quantile range.

[0094] Data temporal characteristics: Analyze the temporal characteristics of data, such as the distribution and periodicity of timestamps.

[0095] Spatial characteristics of data: For geographic data, analyze its spatial characteristics, such as the distribution and spatial clustering of geographic coordinates.

[0096] (II) Dynamic decision tree construction

[0097] The following uses the scenario of implementing a data classification and grading system in a large manufacturing enterprise as an example to explain how to build a dynamic decision tree. The enterprise's data includes R&D design documents, production process records, sales contracts, customer information, and financial statements. These data need to be classified and graded according to their sensitivity for easy management and protection.

[0098] ① Feature selection

[0099] In this step, features that can effectively distinguish different categories and levels are extracted from the data.

[0100] Information gain calculation: Assume there is a batch of files, including design documents, contracts, and customer information. The system first calculates the feature information gain of each file. For example, the format of the file (such as PDF, Word, Excel) may be an important feature because data in different formats usually have different sensitivities.

[0101] The calculation formula of information gain is as follows:

[0102]

[0103] In this example, H is the entropy of the dataset, T is the set of all files, A is the file format feature, and T v is a subset of files in format v. Information gain measures how much the uncertainty of a dataset T is reduced given feature A. By calculating the information gain for each feature, the system can identify which features are most helpful for classification.

[0104] Specific examples:

[0105] Suppose there is a small dataset that contains 10 files. Each file has a specific format (PDF, Word, Excel) that can be used to determine whether the file belongs to the "sensitive" category. Here is the distribution of the dataset:

[0106] Sensitive files: 6 (4 PDFs, 1 Word, 1 Excel)

[0107] Non-sensitive files: 4 (1 PDF, 2 Word, 1 Excel)

[0108] 1) Calculate the initial entropy

[0109] Entropy is a measure of the uncertainty of a data set. First, calculate the entropy of the entire data set:

[0110]

[0111] Calculated:

[0112] H(T)=-(0.6×log20.6+0.4×log20.4)≈0.971

[0113] 2) Calculate the conditional entropy after splitting by format

[0114] The dataset is split based on the file format (PDF, Word, Excel) and the entropy of each subset is calculated.

[0115] PDF format (5 files: 4 sensitive, 1 non-sensitive):

[0116]

[0117] Word format (3 files: 1 sensitive, 2 non-sensitive):

[0118]

[0119] Excel format (2 files: 1 sensitive, 1 non-sensitive):

[0120]

[0121] 3) Calculate the information gain of format features

[0122] Now, calculate the conditional entropy after splitting using the file format:

[0123]

[0124] Calculated:

[0125] H(T|Format)=0.5×0.722+0.3×0.918+0.2×1≈0.837

[0126] Finally, the information gain is:

[0127] IG(T,Format)=H(T)-H(T|Format)≈0.971-0.837=0.134

[0128] By calculating information gain, we can use file format to split the data set to reduce the uncertainty (entropy) of the data, so as to better judge the sensitivity of the file. In this example, the information gain is 0.134, which means that the file format feature helps to distinguish sensitive and non-sensitive files to a certain extent. By comparing the information gain of different features, we can select the features that are most helpful for classification to build a decision tree.

[0129] Feature sorting: The system sorts features based on information gain and selects the feature with the highest information gain as the split feature for the current node. For example, file format, file size, and keyword frequency may be the top three important features.

[0130] ②Node Split

[0131] After selecting the optimal feature, the system needs to split the node according to the different values ​​of the feature and generate corresponding branches.

[0132] Splitting conditions: For the discrete feature of file format, the system splits according to each format. For example, PDF files may need further analysis of their text content, while Excel files need to be analyzed for their data table structure.

[0133] For continuous features (such as file size), the system needs to find the best split point. Suppose we divide the file size into two categories: less than 1MB and greater than 1MB. The system will determine this split point based on the principle of maximizing information gain.

[0134] Generate subnodes: Generate a subnode for each split condition and divide the dataset into the corresponding subnodes. For example, all PDF files are assigned to one subnode and all Excel files are assigned to another subnode.

[0135] ③Dynamic adjustment

[0136] Dynamic adjustment is the key feature of dynamic decision trees, enabling them to adapt to changes in data characteristics.

[0137] Real-time update: Assuming that the enterprise introduces a new file format (such as JSON), the system will update the decision tree structure in real time. Through the incremental learning algorithm, the system can update some nodes without rebuilding the entire tree.

[0138] Adaptive pruning: To prevent overfitting, the system dynamically prunes the decision tree. For example, if the splitting condition of a node contributes little to the classification result, the system will remove the node through the cost complexity pruning algorithm.

[0139] ④Path optimization

[0140] Path optimization aims to improve the efficiency of decision trees and reduce computational overhead by optimizing decision paths.

[0141] Path cost calculation: The system calculates the cost of each decision path, including computing time and resource consumption. For example, analyzing the text content of a large PDF file may require more computing resources, while analyzing the structure of a small Excel file is relatively simple.

[0142] The formula is as follows:

[0143]

[0144] Among them, N i It is the i-th node on the path, Time is the time consumption, and Resource is the resource consumption.

[0145] Actual case:

[0146] In a large manufacturing company, suppose we need to process a large number of production process records and sales contracts. Production process records are usually structured CSV files, while sales contracts are unstructured PDF files. In order to optimize the path, the system will give priority to the path of processing CSV files because their structured characteristics make parsing and classification more efficient.

[0147] For example, the path to process a CSV file may only require simple field parsing and statistical analysis, while the path to process a PDF file may require complex OCR (optical character recognition) and natural language processing. Therefore, the system calculates the cost of each path and prioritizes the path with the lower cost for processing.

[0148] Optimal path selection: The system chooses the path with the lowest cost to process data. For example, for a PDF file containing a large amount of text, the system may choose to perform OCR processing first and then keyword matching to minimize computing overhead.

[0149] ⑤ Feedback mechanism integration

[0150] The feedback mechanism is integrated to enable the decision tree to self-optimize based on the feedback of the recognition results.

[0151] Feedback collection: The system collects user feedback on classification results. For example, users can mark whether a file is correctly classified, and the system records this feedback.

[0152] Model update: Based on the feedback information, the system adjusts the structure of the decision tree and the node selection strategy. For example, if the splitting condition of a node leads to a high false positive rate, the system will re-evaluate the feature selection of that node. In this way, the system can continuously learn and improve to improve the accuracy and efficiency of classification.

[0153] In the present invention, the dynamic decision tree helps to select which algorithm (including: regularization + verification algorithm, Aho-Corasick automaton, NLP technology (natural language processing technology) and large language model) to identify sensitive data in the next step. For example, the decision tree output can be a number from 1 to 10, each representing an algorithm used subsequently (regularization + verification algorithm, Aho-Corasick automaton, NLP technology and large language model), thereby improving the efficiency and accuracy of sensitive data identification.

[0154] In addition, in step S3062, the operation of identifying sensitive data in the target data using the recognition algorithm corresponding to the determined recognition strategy includes identifying sensitive data in the target data using the recognition algorithm corresponding to the determined recognition strategy from the following recognition algorithms: regular expressions and verification algorithms; Aho-Corasick automata; NLP technology; and large language models.

[0155] The following is a detailed introduction to different sensitive data identification algorithms:

[0156] (I) Regular expression and verification algorithm implementation details:

[0157] In the present invention, regular expressions and verification algorithms are used to preliminarily screen and verify sensitive data to ensure the validity and accuracy of the data. The specific implementation steps are as follows:

[0158] 1. Application of regular expressions: First, define a series of regular expressions to match specific types of sensitive data. For example, for sensitive information such as email addresses, phone numbers, and ID numbers, you can design the following regular expressions:

[0159] 1) Email address: ^[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}$

[0160] 2) Phone number: ^\+? [0-9]{1,3}? [-.]? [0-9]{1,4}[-.]? [0-9]{1,4}[-.]? [0-9]{1,9}$

[0161] 3) ID number: ^\d{15}|\d{18}$

[0162] Through these regular expressions, the system can perform pattern matching on input data and quickly identify potentially sensitive information.

[0163] 2. Implementation of the verification algorithm: After the regular expression filters out sensitive data, the next step is to verify the validity of this data. Taking the ID card number as an example, the Luhn algorithm can be used for verification. The core idea of ​​the Luhn algorithm is to determine whether it meets specific verification rules by weighting and calculating the numbers. The specific steps are as follows:

[0164] 1) Weight each digit of the ID number from right to left, multiplying the odd digits by 1 and the even digits by 2.

[0165] 2) If the result after multiplication by 2 is greater than 9, subtract 9 from it.

[0166] 3) Add up all the processed numbers. If the sum is divisible by 10, the ID number is valid.

[0167] In this way, the system can effectively filter out invalid sensitive data and ensure the accuracy of subsequent processing.

[0168] 3. Data verification feedback mechanism: After completing the regular expression matching and verification algorithm verification, the system will generate a verification report, listing all the sensitive data that have passed and failed the verification. For data that has failed the verification, the system will provide detailed error information to help users make corrections.

[0169] In the specific implementation process, assume that the system receives a text file containing user information, including name, email, phone number, and ID number. First, the system uses regular expressions to match each line of data and identify fields that match the email and phone number formats. Then, the system applies the Luhn algorithm to the identified ID number for validity verification.

[0170] For example, if the recognized ID number is "123456789012345", the system will perform a weighted sum calculation from right to left, and the final sum is "30". Since 30 is divisible by 10, the system marks the ID number as valid. Finally, the system will generate a verification report that lists all valid and invalid sensitive data to ensure the accuracy and validity of the data, providing a reliable basis for subsequent data processing and management.

[0171] (II) Details of the implementation of the Aho-Corasick automaton:

[0172] In the present invention, the Aho-Corasick automaton is used for multi-pattern matching to achieve rapid identification of multiple sensitive data patterns. The specific implementation steps of the method are as follows:

[0173] 1. Construct Aho-Corasick automaton:

[0174] First, define a set of sensitive data patterns, such as email addresses, phone numbers, social security numbers, etc. Store these patterns in a list.

[0175] Next, we construct the Trie tree structure of the Aho-Corasick automaton. The Trie tree is a prefix tree used to efficiently store and retrieve a collection of strings. Each node represents a character, and the combination of characters on the path forms a pattern.

[0176] In the process of building the Trie tree, the characters of each sensitive data pattern are inserted to form a tree structure. The terminal node of each pattern will be marked as the end of the pattern.

[0177] 2. Build failure pointer:

[0178] 1) After the Trie tree is built, a failure pointer needs to be built for each node. The failure pointer is used to point to the next possible matching node when the match fails.

[0179] 2) In the specific implementation, start from the root node and use breadth-first search (BFS) to traverse the Trie tree. For each node, if the current character does not match, backtrack to its parent node through the failure pointer and continue to look for other possible matching paths.

[0180] 3) The construction of the failure pointer ensures that during the matching process, it can quickly jump to the next possible matching state, thereby improving the matching efficiency.

[0181] 3. Perform multi-pattern matching:

[0182] Once the Aho-Corasick automaton is built, the system can perform multi-pattern matching on the input data. The text data to be matched is input into the automaton, which reads the text character by character and matches it in the Trie tree.

[0183] When a pattern is matched, the system will record the starting and ending positions of the pattern and continue to match the next character. Due to the characteristics of the Aho-Corasick algorithm, the match can be completed within the time complexity of O(n+m+z), where n is the length of the text, m is the total length of the pattern, and z is the number of matching results.

[0184] 4. Result output and feedback mechanism:

[0185] After the matching is completed, the system will generate a matching result report, listing all the sensitive data patterns identified and their locations in the text. For sensitive data that is not matched, the system will provide corresponding feedback to help users correct and supplement the data.

[0186] In the specific implementation process, suppose the system receives a text data containing user information, including name, email, phone number and ID number. First, the system defines a set of sensitive data patterns, such as:

[0187] Email address pattern: ^[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}$

[0188] Phone number pattern: ^\+? [0-9]{1,3}? [-.]? [0-9]{1,4}[-.]? [0-9]{1,4}[-.]? [0-9]{1,9}$

[0189] ID number pattern: ^\d{15}|\d{18}$

[0190] Next, the system inserts these patterns into the Aho-Corasick automaton, builds a Trie tree and a failure pointer, and then inputs the text data to be matched into the automaton for matching.

[0191] For example, suppose the text to be matched is "The user's email is example@example.com, the phone number is +8613800138000, and the ID number is 123456789012345". The system reads the text character by character and matches it in the Trie tree. When it matches "example@example.com", the system records the start and end positions of the pattern and continues to match the next character.

[0192] Finally, the system generates a matching result report that lists all identified sensitive data patterns and their locations in the text, for example:

[0193] Email: example@example.com (position: 5-25)

[0194] Phone number: +8613800138000 (location: 27-43)

[0195] ID number: 123456789012345 (location: 45-65)

[0196] In this way, the system can efficiently and accurately identify a variety of sensitive data patterns, providing a reliable foundation for subsequent data processing and management.

[0197] (3) Implementation methods of NLP natural language processing technology.

[0198] In the present invention, natural language processing (NLP) technology is combined to effectively identify potential sensitive information by analyzing the context information of the text. The specific implementation steps of the method are as follows:

[0199] 1. Text preprocessing:

[0200] First, the input text data is preprocessed, including removing extra spaces, punctuation marks, and special characters. Then, the text is converted to lowercase to ensure consistency in subsequent processing.

[0201] Secondly, use word segmentation technology to segment the text into words or phrases. Word segmentation can be based on rules or statistics, such as using a dictionary or machine learning model.

[0202] 2. Contextual feature extraction:

[0203] After the text is segmented, context features are extracted. Context features can include information such as word frequency, part-of-speech tagging, and syntactic structure. Word frequency can be obtained by counting the number of times each word appears in the text, while part-of-speech tagging can use a pre-trained NLP model (such as spaCy or NLTK) to mark the part of speech of each word.

[0204] Syntactic structure analysis can be achieved through dependency parsing or component parsing to identify the relationship between words. These features will provide important contextual information for subsequent sensitive information identification.

[0205] 3. Construction of sensitive information identification model:

[0206] Based on the extracted context features, a sensitive information identification model is constructed. A variety of machine learning algorithms can be used for training, such as support vector machines (SVM), random forests, or deep learning models (such as LSTM or BERT). The input of the model is the extracted context features, and the output is the category of sensitive information (such as email, phone number, ID number, etc.).

[0207] During the model training process, it is necessary to prepare a labeled data set that contains known sensitive information and its context. By training on this data, the model can learn the characteristics of sensitive information.

[0208] 4. Sensitive information identification and feedback mechanism:

[0209] Once the model is trained, the system can identify sensitive information on new text data. The text to be analyzed is input into the model, and the model will output the identified sensitive information and its context. The system will generate a recognition result report that lists all identified sensitive information and its location in the text. At the same time, for unidentified sensitive information, the system will provide feedback to help users correct and supplement data.

[0210] During the specific implementation, assume that the system receives a text data containing user information, including name, email, phone number, and ID number. First, the system preprocesses the text, removes extra spaces and punctuation, and converts the text to lowercase. Next, the system segments the text into words. For example, "The user's email is example@example.com, the phone number is +8613800138000, and the ID number is 123456789012345" is segmented into "user", "of", "email", "is", "example@example.com", "phone number", "is", "+8613800138000", "ID number", "is", "123456789012345".

[0211] The system then extracts contextual features, including the frequency and part-of-speech tagging of each word. Assuming that "email" is tagged as a noun, the system will record its position and contextual information in the text. Next, the system uses the labeled dataset to train a sensitive information recognition model, and the model learns that "email" is often associated with the "@" symbol, and "phone number" often starts with "+" or a number.

[0212] After the model training is complete, the system inputs the text to be analyzed into the model for sensitive information identification. The model identifies "example@example.com" as an email, records its location, and identifies "+8613800138000" as a phone number. Finally, the system generates a recognition result report that lists all the identified sensitive information and its location in the text, for example:

[0213] Email: example@example.com (position: 10-30)

[0214] Phone number: +8613800138000 (location: 32-50)

[0215] ID number: 123456789012345 (location: 52-72)

[0216] In this way, the system can efficiently and accurately identify potentially sensitive information, providing a reliable foundation for subsequent data processing and management.

[0217] (IV) Method based on large language model.

[0218] By applying the Large Language Model (LLM), the present invention can perform in-depth analysis of the data to confirm its sensitivity and classify and grade it. The specific implementation steps are as follows:

[0219] 1. Data input and preprocessing:

[0220] First, the data to be analyzed is input into the system. The data can be text, documents, or other information formats. The system will pre-process the input data, including removing irrelevant format information, punctuation, and special characters to ensure the neatness and consistency of the data.

[0221] Next, the data is converted into a format suitable for processing by a large language model. This step is implemented using a pre-trained embedding model.

[0222] 2. Sensitivity analysis:

[0223] After data preprocessing is complete, the system will use the large language model to perform sensitivity analysis. The large language model can identify potentially sensitive information by analyzing the context, semantics, and structure of the text. For example, the model can identify content related to personal identity, financial information, medical records, etc.

[0224] The process of sensitivity analysis involves inputting preprocessed data into a large language model, which outputs a sensitivity score for each piece of data. The score can reflect the likelihood that the data is sensitive information based on the model's understanding of the data content.

[0225] Finally, in step S308, the target data is classified and graded according to the sensitive data, for example, using a large language model for classification and grading operations.

[0226] Specifically, according to the sensitivity score, the system will classify and grade the data. Based on multiple preset sensitivity levels, such as "high sensitivity", "medium sensitivity" and "low sensitivity", the system assigns the data to the corresponding category according to the score threshold.

[0227] In the specific implementation process, suppose the system receives a text data containing user information, including name, address, email, and credit card information. First, the system preprocesses the text to remove extra spaces and punctuation, and converts the text to lowercase. Next, the system converts the text into a vector representation for input into the large language model.

[0228] In the sensitivity analysis phase, the system inputs the preprocessed text into the large language model. The model analyzes the text content, identifies "email" and "credit card information" as potentially sensitive information, and generates a sensitivity score for each data segment. For example, the model may generate a high sensitivity score for "the user's email is example@example.com" and a medium sensitivity score for "the user's name is Zhang San".

[0229] Based on these scores, the system classifies and grades the data. Assuming the threshold is set at 0.7, content with a score higher than 0.7 is marked as "high sensitivity", content with a score between 0.4 and 0.7 is marked as "medium sensitivity", and content with a score lower than 0.4 is marked as "low sensitivity". Finally, the system generates a result report that lists the sensitivity level of each data fragment, for example:

[0230] Email: example@example.com (high sensitivity)

[0231] Name: Zhang San (medium sensitivity)

[0232] Address: Chaoyang District, Beijing (low sensitivity)

[0233] Credit card information: 1234-5678-9012-3456 (high sensitivity)

[0234] In this way, the system can efficiently and accurately confirm the sensitivity of the data and provide a reliable foundation for subsequent data management and protection.

[0235] In addition, preferably, the method of this embodiment further comprises: classifying and grading the target data by using automatic clustering. Specifically, the system 100 analyzes the data format to determine the encoding method, extracts text feature embedding and performs UMAP dimension reduction, and implements data classification and grading by combining machine learning clustering with active learning.

[0236] Automated clustering aims to achieve efficient data classification and grading through in-depth data analysis and feature extraction. The process can be divided into the following key steps:

[0237] 1. Data format analysis and encoding selection: The system first performs format analysis on the input data to determine the appropriate encoding method. The data format may include text, images, audio, etc. For different formats, the system will select the corresponding encoding method. For example, for text data, you can use the Bag of Words model (Bag of Words) or TF-IDF (Term Frequency-Inverse Document Frequency) for encoding; for image data, you can use the Convolutional Neural Network (CNN) to extract features.

[0238] 2. Feature extraction and embedding generation: After determining the encoding method, the system will extract features from the data and generate embeddings. Embedding is a representation method that maps high-dimensional data to a low-dimensional space, which can retain the semantic information of the data. For text data, the system can use pre-trained language models (such as Word2Vec or BERT) to generate embedding representations of the text. These embeddings can effectively capture the contextual information and semantic relationships of the text.

[0239] 3. UMAP dimensionality reduction: In order to speed up the subsequent clustering process, the system performs UMAP (Uniform Manifold Approximation and Projection) dimensionality reduction on the generated Embedding. UMAP is a nonlinear dimensionality reduction technology that can reduce the data dimension while maintaining the local structure of the data. Its mathematical formula is:

[0240]

[0241] Among them, P ij represents the similarity between point i and point j in high-dimensional space, and Q ij Represents the similarity between point i and point j in the low-dimensional space. By optimizing this objective function, UMAP can effectively map high-dimensional data to low-dimensional space.

[0242] 4) Combination of active learning and clustering algorithms: After completing dimensionality reduction, the system combines active learning machine learning clustering algorithms (such as K-Means, DBSCAN, etc.) to classify and grade the data. The core of active learning is to optimize the learning process of the model through continuous iteration and feedback. The system will select the most representative data samples for annotation based on the current clustering results, thereby improving the accuracy and efficiency of clustering.

[0243] In the specific implementation process, the system first analyzes the format of the input data and selects the appropriate encoding method. For example, for a set of text data, the system may choose TF-IDF encoding. Then, the system uses the pre-trained BERT model to generate the embedding representation of the text. Subsequently, the system performs UMAP dimensionality reduction on these embeddings to reduce the data dimension and retain important features. Finally, the system combines the active learning K-Means clustering algorithm to classify and grade the reduced data.

[0244] For example, suppose the system needs to cluster a set of customer feedback texts. First, the system analyzes the text format and selects TF-IDF encoding. Then, the BERT model is used to generate the Embedding representation of each feedback. Next, the system performs UMAP dimensionality reduction on these Embeddings to obtain a low-dimensional representation. Finally, the system applies the K-Means clustering algorithm to cluster similar customer feedback together, thereby achieving effective classification and analysis of customer feedback. In this way, companies can better understand customer needs and optimize products and services.

[0245] In addition, reference Figure 1 As shown, according to the second aspect of this embodiment, a storage medium is provided, wherein the storage medium includes a stored program, wherein when the program is run, a processor executes any one of the above methods.

[0246] Therefore, according to this embodiment, the accuracy and efficiency of data classification and grading are improved through steps such as active data source detection, database data identification, file type identification, sensitive data identification and automatic clustering. In particular, through adaptive technology, the optimal processing method can be selected according to different data types and features, thereby significantly reducing computing power consumption and processing delays, and improving the overall efficiency of data management. This solves the technical problem that the large language model in the prior art has inherent high computing power consumption and long processing time, is not good at processing massive unstructured data other than text, and cannot effectively cope with application scenarios with massive unstructured data.

[0247] It should be noted that, for the above-mentioned method embodiments, for the sake of simplicity, they are all described as a series of action combinations, but those skilled in the art should know that the present invention is not limited by the described action sequence, because according to the present invention, certain steps can be performed in other sequences or simultaneously. Secondly, those skilled in the art should also know that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present invention.

[0248] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a magnetic disk, or an optical disk), and includes a number of instructions for enabling a terminal device (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present invention.

[0249] Example 2

[0250] Figure 6 FIG. 6 shows an adaptive data classification and grading device 600 according to this embodiment, which corresponds to the method described in the first aspect of embodiment 1. Figure 6 As shown, the device 600 includes: a data resource discovery module 610, which is used to automatically identify and discover available data resources using network protocols; an attribute content determination module 620, which is used to preliminarily identify the target data of the data resource and determine the attribute content information related to the target data; a sensitive data identification module 630, which is used to adopt an adaptive algorithm to determine the sensitive data identification algorithm corresponding to the target data according to the determined attribute content information, and use the determined sensitive data identification algorithm to identify the sensitive data corresponding to the target data; and a classification and grading module 640, which is used to classify and grade the target data according to the sensitive data.

[0251] Optionally, the data resource discovery module 610 includes: a host identification submodule, which is used to identify the host of the data resource that responds in the network through the echo request of the ICMP protocol; a service port identification submodule, which is used to use the TCP protocol to perform port scanning on the discovered host and identify open service ports; and a version information identification submodule, which is used to perform version detection to obtain specific version information of the service.

[0252] Optionally, the attribute content determination module 620 includes: a database data identification submodule, used to identify database data in data resources and determine the type, format and content of target database data; and / or a file identification submodule, used to identify files in data resources and determine the type of target files.

[0253] Optionally, the database data identification submodule includes: a permission acquisition unit, used to establish a connection with the target database through the Java database connection protocol to obtain access rights to the target database; a protocol identification engine initialization unit, used to initialize the protocol identification engine, and the protocol identification engine is used to scan the database data with readable permissions in the target database; a random sampling unit, used to perform adaptive random sampling of database data according to the size of the data table and data distribution of the database data; and a sampling data identification unit, used to identify the sampling data of the adaptive random sampling and determine the type format and content of the sampling data.

[0254] Optionally, the file identification submodule includes: a server connection unit, used to establish a connection with a target server and obtain access rights to a target directory on the target server; a file list acquisition unit, used to obtain a file list under the target directory; a file scanning unit, used to scan the target file under the target directory according to the file list; and an encoding format identification unit, used to read the content of the target file and identify the encoding format of the target file.

[0255] The file scanning unit includes a file scanning subunit, which is used to scan the target file in an incremental scanning manner.

[0256] Optionally, the sensitive data identification module 630 includes: a target data analysis submodule, configured to analyze the format and type of the target data and determine the encoding format and identification strategy corresponding to the target data; and a sensitive data identification submodule, configured to identify sensitive data in the target data using an identification algorithm corresponding to the determined identification strategy.

[0257] Optionally, the sensitive data identification submodule includes: a target recognition algorithm selection unit, used to select a target recognition algorithm from the following recognition algorithms as the recognition algorithm corresponding to the determined recognition strategy: regular expression and verification algorithm; Aho-Corasick automaton; NLP technology; and a large language model; a sensitive data identification unit, used to identify sensitive data in the target data using the recognition algorithm corresponding to the determined recognition strategy.

[0258] Optionally, the classification and grading module 640 includes a classification and grading unit for classifying and grading the target data according to the sensitive data using a large language model.

[0259] Optionally, the device 600 further includes a clustering module for classifying and grading the target data using automated clustering.

[0260] Therefore, according to this embodiment, the accuracy and efficiency of data classification and grading are improved through steps such as active data source detection, database data identification, file type identification, sensitive data identification and automatic clustering. In particular, through adaptive technology, the optimal processing method can be selected according to different data types and features, thereby significantly reducing computing power consumption and processing delays, and improving the overall efficiency of data management. This solves the technical problem that the large language model in the prior art has inherent high computing power consumption and long processing time, is not good at processing massive unstructured data other than text, and cannot effectively cope with application scenarios with massive unstructured data.

[0261] Example 3

[0262] Figure 7 FIG. 7 shows an adaptive data classification and grading device 700 according to this embodiment, which corresponds to the method according to the first aspect of embodiment 1. Figure 7 As shown, the device 700 includes: a processor; and a memory, connected to the processor, for providing the processor with instructions for processing the following processing steps: using network protocols to automatically identify and discover available data resources; preliminarily identifying the target data of the data resources and determining the attribute content information related to the target data; using an adaptive algorithm to determine the sensitive data identification algorithm corresponding to the target data based on the determined attribute content information, and using the determined sensitive data identification algorithm to identify the sensitive data corresponding to the target data; and classifying and grading the target data based on the sensitive data.

[0263] Optionally, network protocols are used to automatically identify and discover available data resources, including: identifying the host of the data resource that responds in the network through an ICMP protocol echo request; using the TCP protocol to perform port scanning on the discovered host to identify open service ports; and performing version detection to obtain specific version information of the service.

[0264] Optionally, the target data of the data resource is preliminarily identified to determine the attribute content information related to the target data, including: identifying the database data in the data resource to determine the type, format and content of the target database data; and / or identifying the files in the data resource to determine the type of the target file.

[0265] Optionally, operations for identifying database data in data resources and determining the type, format and content of target database data include: establishing a connection with the target database through the Java database connection protocol to obtain access rights to the target database; initializing a protocol identification engine, which is used to scan database data with readable permissions in the target database; performing adaptive random sampling of database data based on the size of the data table and data distribution of the database data; and identifying the sampled data of the adaptive random sampling to determine the type format and content of the sampled data.

[0266] Optionally, the operations of identifying files in the data resources and determining the type of the target file include: establishing a connection with the target server and obtaining access rights to the target directory on the target server; obtaining a file list under the target directory; scanning the target files under the target directory based on the file list; and reading the content of the target file and identifying the encoding format of the target file.

[0267] Optionally, the operation of scanning the target files in the target directory according to the file list includes: scanning the target files in an incremental scanning manner.

[0268] Optionally, an adaptive algorithm is used to determine a sensitive data identification algorithm corresponding to the target data based on the determined attribute content information, and the operation of using the determined sensitive data identification algorithm to identify the sensitive data corresponding to the target data includes: analyzing the format and type of the target data to determine the encoding format and identification strategy corresponding to the target data; and using the identification algorithm corresponding to the determined identification strategy to identify the sensitive data in the target data.

[0269] Optionally, the operation of identifying sensitive data in the target data using the recognition algorithm corresponding to the determined recognition strategy includes: selecting a target recognition algorithm from the following recognition algorithms as the recognition algorithm corresponding to the determined recognition strategy: regular expressions and verification algorithms; Aho-Corasick automata; NLP technology; and large language models; identifying sensitive data in the target data using the recognition algorithm corresponding to the determined recognition strategy.

[0270] Optionally, the operation of classifying and grading the target data according to the sensitive data includes: utilizing a large language model to classify and grade the target data according to the sensitive data.

[0271] Optionally, the memory is further used to provide the processor with instructions for processing the following processing steps: classifying and grading the target data using automated clustering.

[0272] Therefore, according to this embodiment, the accuracy and efficiency of data classification and grading are improved through steps such as active data source detection, database data identification, file type identification, sensitive data identification and automatic clustering. In particular, through adaptive technology, the optimal processing method can be selected according to different data types and features, thereby significantly reducing computing power consumption and processing delays, and improving the overall efficiency of data management. This solves the technical problem that the large language model in the prior art has inherent high computing power consumption and long processing time, is not good at processing massive unstructured data other than text, and cannot effectively cope with application scenarios with massive unstructured data.

[0273] The serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.

[0274] In the above embodiments of the present invention, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0275] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0276] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0277] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0278] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, RandomAccess Memory), mobile hard disk, magnetic disk or optical disk and other media that can store program codes.

[0279] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. An adaptive data classification and grading method, characterized in that: include: Automatically identify and discover available data resources using network protocols; Preliminarily identifying the target data of the data resource and determining attribute content information related to the target data; Adopting an adaptive algorithm to determine a sensitive data identification algorithm corresponding to the target data according to the determined attribute content information, and identifying the sensitive data corresponding to the target data using the determined sensitive data identification algorithm; as well as The target data is classified and graded according to the sensitive data.

2. The method according to claim 1, characterized in that The operation of automatically identifying and discovering available data resources by using network protocols includes: Identify the host of the data resource that responds in the network through the ICMP protocol echo request; Use TCP protocol to perform port scan on discovered hosts and identify open service ports; Perform version detection to obtain specific version information of the service.

3. The method according to claim 1, characterized in that The operation of preliminarily identifying the target data of the data resource and determining the attribute content information related to the target data includes: Identify the database data in the data resource and determine the type, format and content of the target database data; and / or Identify the files in the data resources and determine the type of the target file.

4. The method according to claim 3, characterized in that Identify the database data in the data resources and determine the type, format and content of the target database data, including: Establishing a connection with a target database through a Java database connection protocol to obtain access rights to the target database; Initializing a protocol identification engine, the protocol identification engine is used to scan database data with readable permissions in the target database; Adaptively random sampling the database data according to the size and data distribution of the data table of the database data; and The sampling data of the adaptive random sampling is identified to determine the type format and content of the sampling data, and wherein, Identify files in data resources and determine the type of target files, including: Establishing a connection with a target server and obtaining access rights to a target directory on the target server; Get the file list under the target directory; Scanning the target files in the target directory according to the file list; and The content of the target file is read and the encoding format of the target file is identified.

5. The method according to claim 4, characterized in that The operation of scanning the target files in the target directory according to the file list includes: scanning the target files in an incremental scanning manner.

6. The method according to claim 3, characterized in that: The operation of using an adaptive algorithm to determine a sensitive data identification algorithm corresponding to the target data according to the determined attribute content information, and using the determined sensitive data identification algorithm to identify the sensitive data corresponding to the target data includes: Analyzing the format and type of the target data to determine the encoding format and identification strategy corresponding to the target data; and identifying sensitive data in the target data using an identification algorithm corresponding to the determined identification strategy, And wherein, the operation of identifying sensitive data in the target data using the identification algorithm corresponding to the determined identification strategy includes: Selecting a target recognition algorithm from the following recognition algorithms as a recognition algorithm corresponding to the determined recognition strategy: regular expression and verification algorithm; Aho-Corasick automaton; NLP technology; and large language model; Sensitive data in the target data is identified using an identification algorithm corresponding to the determined identification strategy, and wherein, The operation of classifying and grading the target data according to the sensitive data includes: using a large language model to classify and grade the target data according to the sensitive data.

7. The method according to claim 1, characterized in that It also includes classifying and grading the target data using automated clustering.

8. A storage medium, characterized in that: The storage medium includes a stored program, wherein when the program is run, the processor executes the method according to any one of claims 1 to 7.

9. An adaptive data classification and grading device, characterized in that: include: Data resource discovery module, used to automatically identify and discover available data resources using network protocols; An attribute content determination module, used to preliminarily identify the target data of the data resource and determine the attribute content information related to the target data; A sensitive data identification module, configured to use an adaptive algorithm to determine a sensitive data identification algorithm corresponding to the target data according to the determined attribute content information, and to identify the sensitive data corresponding to the target data using the determined sensitive data identification algorithm; as well as A classification and grading module is used to classify and grade the target data according to the sensitive data.

10. An adaptive data classification and grading device, characterized in that: include: processor; as well as A memory, connected to the processor, configured to provide the processor with instructions for processing the following processing steps: Automatically identify and discover available data resources using network protocols; Preliminarily identifying the target data of the data resource and determining attribute content information related to the target data; Adopting an adaptive algorithm to determine a sensitive data identification algorithm corresponding to the target data according to the determined attribute content information, and identifying the sensitive data corresponding to the target data using the determined sensitive data identification algorithm; as well as The target data is classified and graded according to the sensitive data.

Images