A method for processing enterprise data based on business asset sorting

Abstract

The present invention relates to an enterprise data processing method based on business asset combing applied in the field of data processing technology, including the steps of issuing rule knowledge base strategy, terminal PC data collection, document file classification processing, model and rule updating, asset combing result display and the like. The enterprise data processing method in the present application realizes the automatic management of enterprise digital information by combining data collection, rule knowledge base and advanced natural language processing model, thereby building an intelligent digital asset management system for the enterprise, greatly improving work efficiency and optimizing data management process. The enterprise data processing method in the present application utilizes rule knowledge base model for file classification and uses network model based on BERT model and CNN‑LSTM for in-depth analysis and hierarchical screening of text data. The network model can recognize and understand complex text content to ensure the accuracy and rationality of data classification.

Description

Technical Field

[0001] The present invention relates to an enterprise data processing method, and in particular to an enterprise data processing method based on business asset sorting and applied in the field of data processing technology. Background Art

[0002] Trade secrets refer to corporate information that is not known to the public, has practical value, and for which the right holder has taken protective measures, including technical information and business information. With the rapid development of the information digital age, enterprises have accumulated a large amount of digital assets, including personnel contract files, financial statements, engineering design drawings, program development codes, telephone consultation recordings, and various video materials. These data are not only an important part of corporate operations, but also a core strategic resource of the enterprise. Therefore, it is crucial to protect the security and integrity of these data.

[0003] In order to deal with data leakage and information security threats, the country has successively issued the "Cybersecurity Law" and the "Data Security Law" to strictly classify and grade data. These regulations require companies to establish a complete data management system and take corresponding protection measures for different types of data.

[0004] In this context, enterprises are faced with the challenge of how to comprehensively scan and sort out their asset data. Effective data scanning and sorting can not only help enterprises understand the status of their own data assets, but also identify potential security risks. Enterprises need to adopt advanced methods and technologies to ensure comprehensive sorting and classification of data, improve data management and protection capabilities, and effectively respond to the increasingly complex network security environment and regulatory requirements.

[0005] Traditional enterprise asset data classification is often based on manual processes and operations, requiring employees to review and classify data, relying on their knowledge and experience, which consumes a lot of time and human resources, and is prone to errors and inconsistencies. Although the current use of traditional machine learning algorithms such as support vector machine algorithms, naive Bayes algorithms, decision tree algorithms, and neural network algorithms to classify text categories continues to deepen, the data labeling process also relies on manual labor and is suitable for the classification of large quantities of labeled text data, but not for the classification of large amounts of unlabeled data. At the same time, traditional machine learning methods are not easy to extract features. Therefore, we propose an enterprise data processing method based on business asset combing. Summary of the invention

[0006] In view of the above-mentioned prior art, the technical problem to be solved by the present invention is that the traditional enterprise asset data classification and grading is often based on manual processes and operations, requiring employees to review and classify the data, relying on the employees' knowledge and experience, which consumes a lot of time and human resources, and is prone to errors and inconsistencies.

[0007] To solve the above problems, the present invention provides an enterprise data processing method based on business asset combing, comprising the following steps:

[0008] S1. Rule knowledge base strategy issuance:

[0009] S11, construction of rule knowledge base;

[0010] S12. Account registration and binding;

[0011] S13, rule knowledge base is issued;

[0012] S2. Terminal PC data collection:

[0013] S21, full disk scan;

[0014] S211, disk file scanning algorithm: the terminal PC scans the file system using a disk depth-first search algorithm;

[0015] S212, file rule matching and classification: during the scanning process, character string matching is performed according to the rules in the rule knowledge base, and the matching result includes the file name, path and the rule category that meets the requirements;

[0016] S22, file classification and uploading;

[0017] S221, classification result generation: after the scanning is completed, all matching files and their classification information are sorted into classification results in JSON format through the rule knowledge base;

[0018] S222, uploading classification results: the terminal PC uploads the file classification results to the data management server in segments through the HTTP protocol. During the uploading process, a JSON string is sent to the classification result processing endpoint of the server using a POST request;

[0019] S223, document upload: filter out files marked as document type, upload these document files together with their classification information, and perform further classification training;

[0020] S3. Document file classification processing:

[0021] S31, data input processing:

[0022] S311, document cleaning: remove unnecessary content in the document through regular expressions, and remove stop words according to the Chinese stop word list;

[0023] S312, document segmentation: split the document into tokens of word granularity through the Jieba tokenizer;

[0024] S32, BERT embedding encoding vector: The token after word segmentation passes through the BERT embedding layer to generate word embedding vector, type embedding vector and position vector;

[0025] S33, BERT weighted coding fusion: The first layer output vector of the low-level Transformer and the last layer output vector of the high-level Transformer are weightedly superimposed through residual connection design;

[0026] S34, constructing a CNN-LSTM decoding layer: The output of the BERT encoding layer is input into the textCNN network and the Bi-LSTM network respectively, and the output feature vector of the textCNN network and the output feature vector of the Bi-LSTM network are weighted averaged using a predetermined weight;

[0027] S35, fully connected layer classification output: the feature vector after weighted averaging in step S34 is input into the fully connected layer, and then the output of the fully connected layer is calculated by softmax to output the relevant probability and perform the final classification task;

[0028] S36, document classification result generation and storage: the document data is classified by a classification model composed of a BERT model and a CNN-LSTM network model, and the classification results are stored in a data management server;

[0029] S37, hierarchical model storage: storing the hierarchical model composed of the BERT model and the CNN-LSTM network model after each scanning task is completed in the data management server;

[0030] S4. Model and rule updates:

[0031] S41. Update of rule knowledge base: Based on user feedback and new file types, the rules of the rule knowledge base are regularly updated to ensure that they can adapt to the ever-changing data environment. After the rules are updated, they are sent to each terminal PC through an automated mechanism.

[0032] S5. Display of asset sorting results:

[0033] S51, database storage: after each grading and classification task is completed, the classification results of all files and the classification results of document files are stored in the MySQL database;

[0034] S52. Front-end display: Provide a RESTful API interface to transmit the classification results to the Web front-end through the HTTP protocol.

[0035] The above-mentioned enterprise data processing method based on business asset sorting realizes the automated management of enterprise digital information by combining data collection, rule knowledge base and advanced natural language processing models, and then builds an intelligent digital asset management system for the enterprise, which greatly improves work efficiency and optimizes data management processes.

[0036] As a further improvement of the present application, step S11 includes:

[0037] S111. Demand research: Cooperate with relevant departments of the enterprise to determine the file types, file suffixes and naming rules that need to be classified. Based on these requirements, define the structure and content of the rule knowledge base to ensure that it can cover all file types that need to be classified;

[0038] S112. Rule writing: writing specific rules of the rule knowledge base;

[0039] S113, Knowledge base testing: Testing the rules of the rule knowledge base to ensure the accuracy and coverage of the rules;

[0040] S114. Knowledge base optimization: Optimize the rules of the rule knowledge base based on the test results, eliminate false matches and improve the classification standards.

[0041] As a further improvement of the present application, step S12 includes:

[0042] S121. Account registration: When each terminal PC is registered, a unique identifier needs to be generated to create an account on the data management server and ensure the uniqueness of the device identity, which is used to create and maintain the account of the terminal on the data management server;

[0043] S122, account binding: using HTTP protocol, the unique identification and authentication information of the terminal PC is sent to the data management server through an API endpoint. After the server verifies the information, it binds the terminal PC to its corresponding account and returns confirmation information of successful binding;

[0044] S123, information storage: the account information, unique identifier, and binding status of the terminal PC are stored in the database through the data management server for subsequent policy distribution and data upload.

[0045] As a further improvement of the present application, step S13 includes:

[0046] S131, policy delivery: the data management server delivers the rule knowledge base through the HTTP protocol. The rule knowledge base is encoded in the JSON string format and sent to the terminal PC through an HTTP POST request.

[0047] S132, terminal reception and parsing: After receiving the rule knowledge base, the terminal PC parses the JSON string, extracts the rule content, and stores the rule in the local database, preparing for full disk scanning and classification tasks.

[0048] As a further improvement of the present application, step S4 further includes:

[0049] S42. Model retraining: Regularly retrain the grading model composed of the BERT model and the CNN-LSTM network model, and use the newly collected data files and grading results for fine-tuning to ensure that the network model always maintains efficient grading capabilities;

[0050] S43. Performance optimization: Optimize server performance, transmission speed, and data processing flow to ensure that the network model can efficiently handle the increasing amount of data.

[0051] To sum up, the enterprise data processing method in this application realizes the automated management of enterprise digital information by combining data collection, rule knowledge base and advanced natural language processing models, and then builds an intelligent digital asset management system for the enterprise, which greatly improves work efficiency and optimizes the data management process. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 This is a method flow chart of the first implementation mode of this application;

[0053] Figure 2 This is a structural block diagram of a hierarchical model composed of a BERT model and a CNN-LSTM network model in the first embodiment of the present application. DETAILED DESCRIPTION

[0054] An implementation of the present application is described in detail below with reference to the accompanying drawings.

[0055] The first implementation method:

[0056] Figure 1-2 A method for processing enterprise data based on business asset combing is shown, comprising the following steps:

[0057] S1. Rule knowledge base strategy issuance:

[0058] S11. Construction of rule knowledge base:

[0059] S111. Demand research: Cooperate with relevant departments of the enterprise to determine the file types, file suffixes and naming rules that need to be classified. Based on these requirements, define the structure and content of the rule knowledge base to ensure that it can cover all file types that need to be classified;

[0060] S112. Rule writing: writing specific rules of the rule knowledge base. The rules may include keywords in the file name, file extensions, etc. For example, a file name containing the word "contract" or a file ending with ".docx" will be classified as a "contract document" category;

[0061] S113, Knowledge base test: Test the rules of the rule knowledge base to ensure the accuracy and coverage of the rules. The test content includes whether the specified files can be accurately matched, whether there are omissions or misjudgments;

[0062] S114, Knowledge base optimization: optimize the rules of the rule knowledge base based on the test results, eliminate false matches and improve the classification standards to ensure the wide applicability and accuracy of the rule knowledge base;

[0063] S12. Account registration and binding:

[0064] S121, account registration: when each terminal PC is registered, a unique identifier (ID) needs to be generated to create an account on the data management server and ensure the uniqueness of the device identity, which is used to create and maintain the account of the terminal on the data management server;

[0065] S122, account binding: using HTTP protocol, the unique identification and authentication information of the terminal PC is sent to the data management server through an API endpoint. After the server verifies the information, it binds the terminal PC to its corresponding account and returns confirmation information of successful binding;

[0066] S123, information storage: the account information, unique identifier, and binding status of the terminal PC are stored in the database through the data management server for subsequent policy distribution and data upload;

[0067] S13. Rule knowledge base is issued:

[0068] S131, policy delivery: the data management server delivers the rule knowledge base through the HTTP protocol. The rule knowledge base is encoded in the JSON string format and sent to the terminal PC through an HTTP POST request.

[0069] S132, terminal reception and parsing: After receiving the rule knowledge base, the terminal PC parses the JSON string, extracts the rule content, and stores the rule in the local database, preparing for full disk scanning and classification tasks;

[0070] S2. Terminal PC data collection:

[0071] S21, Full Scan:

[0072] S211, disk file scanning algorithm: the terminal PC scans the file system using a disk depth-first search algorithm;

[0073] S212, file rule matching and classification: during the scanning process, character string matching is performed according to the rules in the rule knowledge base, and the matching result includes the file name, path and the rule category that meets the requirements;

[0074] S22. File classification and upload:

[0075] S221, classification result generation: After the scanning is completed, all matching files and their classification information are sorted into classification results in JSON format through the rule knowledge base. The classification information includes file name, path, matching rule, classification category, etc.;

[0076] S222, uploading classification results: the terminal PC uploads the file classification results to the data management server in segments through the HTTP protocol. During the uploading process, a JSON string is sent to the classification result processing endpoint of the server using a POST request;

[0077] S223, document upload: filter out files marked as document type, upload these document files together with their classification information, and perform further classification training;

[0078] S3. Document file classification processing:

[0079] S31, data input processing;

[0080] S311, document cleaning: remove unnecessary content in the document (such as special characters, HTML tags, extra spaces, etc.) through regular expressions, and remove stop words according to the Chinese stop word list;

[0081] S312, document segmentation: split the document into tokens of word granularity through the Jieba tokenizer;

[0082] S32, BERT embedding coding vector: After the word segmentation processing, the token passes through the BERT embedding layer to generate the word embedding vector, type embedding vector and position vector. In particular, the position vector in this model adopts the absolute position encoding method based on sine and cosine functions. The reference rules are as follows:

[0083]

[0084] Among them, pos represents the number of the word, i represents the dimension of embedding, and i∈[0,768);

[0085] S33, BERT weighted coding fusion:

[0086] Fusion principle: The Google team provides two versions of the model, BERT-base and BERT-large. The encoder of the BERT-base model consists of 12 layers of Transformers, and the encoding layer of the BERT-large model consists of 24 layers of Transformers. Different Transformer layers encode feature information in different levels. For example, low-level Transformers capture word-level feature information, while high-level Transformers tend to encode text-level feature information.

[0087] Fusion process: The first layer output vector of the low-level Transformer and the last layer output vector of the high-level Transformer are weightedly superimposed through residual connection design;

[0088] S34. Construct CNN-LSTM decoding layer:

[0089] Construction principle: The textCNN network uses multiple convolution kernels of different sizes to extract key information in sentences (similar to n-grams with multiple window sizes), so as to better capture local features in the text. The Bi-LSTM network uses a bidirectional LSTM layer to process text sequences, which helps to deal with long-distance text dependency problems and obtain contextual word order information of the text.

[0090] Construction process: The output of the BERT encoding layer is input into the textCNN network and the Bi-LSTM network respectively, and the output feature vector of the textCNN network and the output feature vector of the Bi-LSTM network are weighted averaged using the predetermined weights;

[0091] S35, fully connected layer classification output: the feature vector after weighted averaging in step S34 is input into the fully connected layer, and then the output of the fully connected layer is calculated by softmax to output the relevant probability and perform the final classification task;

[0092] S36. Generating and storing document classification results: The document data is classified by a classification model composed of a BERT model and a CNN-LSTM network model, and the classification results are stored in a data management server. The data must be structured and stored according to the document category and classification results to facilitate subsequent query and management.

[0093] S37, hierarchical model storage: storing the hierarchical model composed of the BERT model and the CNN-LSTM network model after each scanning task is completed in the data management server;

[0094] S4. Model and rule updates:

[0095] S41. Update of rule knowledge base: Based on user feedback and new file types, the rules of the rule knowledge base are regularly updated to ensure that they can adapt to the ever-changing data environment. After the rules are updated, they are sent to each terminal PC through an automated mechanism.

[0096] S42. Model retraining: Regularly retrain the grading model composed of the BERT model and the CNN-LSTM network model, and use the newly collected data files and grading results for fine-tuning to ensure that the network model always maintains efficient grading capabilities;

[0097] S43, Performance Optimization: Optimize server performance, transmission speed and data processing flow to ensure that the network model can efficiently handle the increasing amount of data;

[0098] S5. Asset sorting results display:

[0099] S51. Database storage: After each grading and classification task is completed, the classification results of all files and the grading results of document files are stored in the MySQL database, including the file name, file ID, file classification label, document grading label, etc.

[0100] S52. Front-end display: Provide a RESTful API interface to transmit the classification results to the Web front-end through the HTTP protocol, so that the front-end can display the classification results in the form of tables, charts, etc.

[0101] The advantages of the enterprise data processing method in this application are reflected in many aspects:

[0102] 1. Automated data processing: It realizes a fully automated process from data collection to data classification, reducing the reliance on manual intervention. By automatically scanning the terminal PC, it can quickly and efficiently collect a large amount of data, and automatically classify and filter according to preset rules;

[0103] 2. Accurate data classification: Use the rule knowledge base model to classify files and use the classification model composed of the BERT model and the CNN-LSTM network model to perform in-depth analysis and classification screening of text data. The network model can recognize and understand complex text content to ensure the accuracy and rationality of data classification. The context understanding ability of the classification model composed of the BERT model and the CNN-LSTM network model enables it to obtain higher classification accuracy when processing document data.

[0104] 3. Good scalability. This technology is highly scalable and can adapt to the needs of enterprises of different sizes. No matter how the amount of data changes, it can effectively respond and ensure the accuracy of classification and grading. It can flexibly adjust the rule knowledge base according to the specific needs of the enterprise to adapt to different types of files and data.

[0105] In view of current practical needs, the above-mentioned implementation mode adopted in this application is not limited to the scope of protection. Various changes made within the knowledge scope of technical personnel in this field without departing from the concept of this application still fall within the scope of protection of the present invention.

Claims

1. An enterprise data processing method based on business asset combing, characterized in that: The following steps are involved: S1. Rule knowledge base strategy issuance: S11, construction of rule knowledge base; S12. Account registration and binding; S13, rule knowledge base is issued; S2. Terminal PC data collection: S21, full disk scan; S211, disk file scanning algorithm: the terminal PC scans the file system using a disk depth-first search algorithm; S212, file rule matching and classification: during the scanning process, character string matching is performed according to the rules in the rule knowledge base, and the matching result includes the file name, path and the rule category that meets the requirements; S22, file classification and uploading; S221, classification result generation: after the scanning is completed, all matching files and their classification information are sorted into classification results in JSON format through the rule knowledge base; S222, uploading classification results: the terminal PC uploads the file classification results to the data management server in segments through the HTTP protocol. During the uploading process, a JSON string is sent to the classification result processing endpoint of the server using a POST request; S223, document upload: filter out files marked as document type, upload these document files together with their classification information, and perform further classification training; S3. Document file classification processing: S31, data input processing: S311, document cleaning: remove unnecessary content in the document through regular expressions, and remove stop words according to the Chinese stop word list; S312, document segmentation: split the document into tokens of word granularity through the Jieba tokenizer; S32, BERT embedding encoding vector: The token after word segmentation passes through the BERT embedding layer to generate word embedding vector, type embedding vector and position vector; S33, BERT weighted coding fusion: The first layer output vector of the low-level Transformer and the last layer output vector of the high-level Transformer are weightedly superimposed through residual connection design; S34, constructing a CNN-LSTM decoding layer: The output of the BERT encoding layer is input into the textCNN network and the Bi-LSTM network respectively, and the output feature vector of the textCNN network and the output feature vector of the Bi-LSTM network are weighted averaged using a predetermined weight; S35, fully connected layer classification output: the feature vector after weighted averaging in step S34 is input into the fully connected layer, and then the output of the fully connected layer is calculated by softmax to output the relevant probability and perform the final classification task; S36, document classification result generation and storage: the document data is classified by a classification model composed of a BERT model and a CNN-LSTM network model, and the classification results are stored in a data management server; S37, hierarchical model storage: storing the hierarchical model composed of the BERT model and the CNN-LSTM network model after each scanning task is completed in the data management server; S4. Model and rule updates: S41, rule knowledge base update: based on user feedback and new file types, the rules of the rule knowledge base are regularly updated. After the rules are updated, they are sent to each terminal PC through an automated mechanism; S5. Display of asset sorting results: S51, database storage: after each grading and classification task is completed, the classification results of all files and the classification results of document files are stored in the MySQL database; S52. Front-end display: Provide a RESTful API interface to transmit the classification results to the Web front-end through the HTTP protocol.

2. The enterprise data processing method based on business asset combing according to claim 1 is characterized in that: The step S11 comprises: S111. Demand research: Cooperate with relevant departments of the enterprise to determine the file types, file suffixes and naming rules that need to be classified, and define the structure and content of the rule knowledge base based on these requirements; S112. Rule writing: writing specific rules of the rule knowledge base; S113, Knowledge base test: Test the rules of the rule knowledge base, including the accuracy and coverage of the rules; S114. Knowledge base optimization: Optimize the rules of the rule knowledge base based on the test results, eliminate false matches and improve the classification standards.

3. The enterprise data processing method based on business asset combing according to claim 1 is characterized in that: The step S12 comprises: S121, account registration: when each terminal PC is registered, a unique identifier is generated; S122, account binding: using HTTP protocol, the unique identification and authentication information of the terminal PC is sent to the data management server through an API endpoint. After the server verifies the information, it binds the terminal PC to its corresponding account and returns confirmation information of successful binding; S123, information storage: the account information, unique identifier, and binding status of the terminal PC are stored in the database through the data management server for subsequent policy distribution and data upload.

4. The enterprise data processing method based on business asset combing according to claim 1 is characterized in that: The step S13 comprises: S131, policy delivery: the data management server delivers the rule knowledge base through the HTTP protocol. The rule knowledge base is encoded in the JSON string format and sent to the terminal PC through an HTTP POST request. S132, terminal reception and parsing: After receiving the rule knowledge base, the terminal PC parses the JSON string, extracts the rule content, and stores the rule in the local database, preparing for full disk scanning and classification tasks.

5. The enterprise data processing method based on business asset combing according to claim 1 is characterized in that: The step S4 further comprises: S42, Model retraining: Regularly retrain the grading model composed of the BERT model and the CNN-LSTM network model, and fine-tune it using newly collected data files and grading results; S43. Performance optimization: Optimize server performance, transmission speed, and data processing flow to ensure that the network model can efficiently handle the increasing amount of data.

Images