Data compliance transaction method and system based on big data mining technology
By building and training the data classification and grading model and combining with the alliance chain system, the problem of inaccurate data classification and grading levels in the existing technology is solved, and the accuracy and compliance of data compliance transactions are achieved.
Patent Information
- Application Number
- CN202411200894.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-29
- Publication Date
- 2024-12-03
AI Technical Summary
The data classification and hierarchy level cannot be accurately obtained in the prior art, resulting in inaccurate data compliance transactions, and the identification and resolution system based on the national industrial Internet cannot achieve data compliance transactions.
By obtaining server log data, building a data classification and grading model, training the optimal data classification and grading model, judging the classification level of the data to be classified, and setting data circulation rules in the alliance chain system to ensure that the transaction data complies with the purchaser's classification and grading level.
Accurate data classification and grading are realized, the accuracy problem in data compliance transactions is solved, and the compliant circulation of data in the alliance chain system is ensured.
Smart Images

Figure CN119066471A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of artificial intelligence technology, and specifically to a data compliance transaction system, method, electronic device and storage medium based on big data mining technology. Background Art
[0002] Data classification and grading is to formulate classification and grading standards suitable for data based on the natural attributes of data assets in combination with national and industry standards, and to classify and grade customer data according to the classification and grading standards to determine the security level of data assets.
[0003] Currently, data classification and grading are basically carried out after data is gathered in the business middle platform, which results in inaccurate data classification and grading and lax security control. In addition, the identification data open permission management function of the current identification resolution system based on the national industrial Internet is either fully open to the outside world or not open at all, making it impossible to achieve data compliance transactions.
[0004] Therefore, there is an urgent need for a data compliance transaction method based on big data mining technology that can achieve accurate data classification and grading levels. Summary of the invention
[0005] The purpose of the embodiments of the present invention is to provide a data compliance transaction system, method, electronic device and storage medium based on big data mining technology, so as to solve the problem that the accurate data classification and grading levels cannot be obtained in the prior art, and to realize data compliance transaction based on big data mining technology through accurate data classification and grading levels.
[0006] To achieve the above objectives, an embodiment of the present invention provides a data compliance transaction method based on big data mining technology, and the data compliance transaction method based on big data mining technology specifically includes:
[0007] Obtain component B1 in the front-end page A1 in the development environment, a back-end data interface or a data interaction call information request C1 for the component B1 operation docking, and obtain server log data based on the data interaction call information request C1, wherein the number of the component B1 is one or more, and the number of the data interaction call information request C1 is one or more;
[0008] Acquire the operation function items based on the server log data, and acquire the data sets, data tables, data fields and data processing rules corresponding to the operation function items under different request conditions;
[0009] Obtain the data sets, data tables, data fields, and data processing rules called by the backend trigger tasks and trigger mechanisms;
[0010] Perform data preprocessing on all data tables and all data fields to obtain preprocessed data;
[0011] Constructing a data classification and grading model, inputting the preprocessed data into the data classification and grading model for training, and obtaining an optimal data classification and grading model;
[0012] Constructing a data classification and grading keyword library, wherein the data classification and grading keyword library includes the classification and grading levels of the data table and the classification and grading levels of the data field;
[0013] Determine whether the data to be classified and graded exists in the data classification and grading keyword library, and when the data to be classified and graded exists in the data classification and grading keyword library, obtain the classification and grading level of the data to be classified and graded by matching;
[0014] When the data to be classified and graded does not exist in the data classification and graded keyword library, the data to be classified and graded is input into the optimal data classification and graded model to obtain the classification and graded level corresponding to the data to be classified and graded;
[0015] Build a consortium chain system, where the roles on the consortium chain system include data suppliers, data exchanges and data buyers;
[0016] In response to the data circulation rule setting request, setting the data circulation rule, wherein the data circulation rule includes a classification and grading level corresponding to each data buyer;
[0017] In response to the transaction request, determine whether the data to be traded meets the classification and grading level corresponding to the data buyer. If so, complete the transaction.
[0018] On the basis of the above technical solution, the present invention can also be improved as follows:
[0019] Furthermore, the data preprocessing is performed on all data tables and all data fields to obtain preprocessed data, including:
[0020] When the data to be classified and graded is unstructured data, the data to be classified and graded are extracted to obtain field data corresponding to the data to be classified and graded, corresponding data tables and entity relationships, and attribute features corresponding to each data table.
[0021] Furthermore, the constructing of the data classification and grading model, inputting the preprocessed data into the data classification and grading model for training, and obtaining the optimal data classification and grading model, comprises:
[0022] The threshold range corresponding to the correlation index between data tables and the threshold range corresponding to the correlation index between data fields are preset.
[0023] Furthermore, the constructing of the data classification and grading model, inputting the preprocessed data into the data classification and grading model for training, and obtaining the optimal data classification and grading model, further includes:
[0024] Input the data to be classified and graded into the optimal data classification and grading model, extract the corresponding key indicator data, and match the corresponding data fields, data tables and entity relationships, and attribute features corresponding to each data table based on the key indicator data;
[0025] Obtaining correlation indicators between data tables and correlation indicators between data fields, and determining whether the correlation indicators exceed a threshold range. When the correlation indicators exceed the threshold range, the current model is not met. Repeating the above operations until the correlation indicators meet the threshold range.
[0026] When the correlation index meets the threshold range, the classification level corresponding to the to-be-classified data is generated.
[0027] Furthermore, the construction of the alliance chain system includes:
[0028] In response to the data supplier's entry request, determine the data products that the data supplier applies for compliance registration, obtain the classification and grading identification data of the data tables and data fields corresponding to the data products, and automatically open the classification and grading identification data permissions to the corresponding data exchanges.
[0029] Furthermore, the construction of the alliance chain system also includes:
[0030] A data identification model is constructed, and the classification and grading levels corresponding to the current interface data of the data supplier are identified through the data identification model, wherein the classification and grading levels include public data, personal information, core data, important data and critical information infrastructure.
[0031] Furthermore, the construction of the alliance chain system also includes:
[0032] Construct an enterprise qualification analysis model. When a data buyer enters the data exchange, the classification and grading level corresponding to the data buyer is analyzed through the enterprise qualification analysis model.
[0033] In response to a transaction request, interface data that complies with the classification and grading level corresponding to the data buyer is provided to the data buyer.
[0034] A data compliance trading system based on big data mining technology, comprising:
[0035] An acquisition module is used to acquire component B1 in the front-end page A1 in the development environment, a back-end data interface or a data interaction call information request C1 for the component B1 operation docking, and acquire server log data based on the data interaction call information request C1, wherein the number of the component B1 is one or more, and the number of the data interaction call information request C1 is one or more;
[0036] Acquire the operation function items based on the server log data, and acquire the data sets, data tables, data fields and data processing rules corresponding to the operation function items under different request conditions;
[0037] Obtain the data sets, data tables, data fields, and data processing rules called by the backend trigger tasks and trigger mechanisms;
[0038] A preprocessing module is used to perform data preprocessing on all data tables and all data fields to obtain preprocessed data;
[0039] A construction module is used to construct a data classification and grading model, and input the preprocessed data into the data classification and grading model for training to obtain an optimal data classification and grading model;
[0040] Constructing a data classification and grading keyword library, wherein the data classification and grading keyword library includes the classification and grading levels of the data table and the classification and grading levels of the data field;
[0041] A classification and grading module, used to determine whether the data to be classified and graded exists in the data classification and grading keyword library, and when the data to be classified and graded exists in the data classification and grading keyword library, match and obtain the classification and grading level of the data to be classified and graded;
[0042] When the data to be classified and graded does not exist in the data classification and graded keyword library, the data to be classified and graded is input into the optimal data classification and graded model to obtain the classification and graded level corresponding to the data to be classified and graded;
[0043] The data circulation module is used to build a consortium chain system, where the roles in the consortium chain system include data suppliers, data exchanges and data buyers;
[0044] In response to the data circulation rule setting request, setting the data circulation rule, wherein the data circulation rule includes a classification and grading level corresponding to each data buyer;
[0045] In response to the transaction request, determine whether the data to be traded meets the classification and grading level corresponding to the data buyer. If so, complete the transaction.
[0046] An electronic device comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the steps of the method are implemented when the processor executes the computer program.
[0047] A non-transitory computer-readable storage medium stores a computer program, which implements the steps of the method when executed by a processor.
[0048] The embodiments of the present invention have the following advantages:
[0049] The data compliance transaction method based on big data mining technology in the present invention obtains operation function items based on the server log data, and obtains the data sets, data tables, data fields and data processing rules corresponding to the operation function items under different request conditions; obtains the data sets, data tables, data fields and data processing rules called by the back-end trigger tasks and trigger mechanisms; constructs a data classification and grading keyword library, wherein the data classification and grading keyword library includes the classification and grading levels of the data tables and the classification and grading levels of the data fields; determines whether the data to be classified and graded exists in the data classification and grading keyword library, and when the data to be classified and graded exists in the data classification and grading keyword library, matches and obtains the classification and grading levels of the data to be classified and graded; when the data to be classified and graded exists in the data classification and grading keyword library, matches and obtains the classification and grading levels of the data to be classified and graded; When the data to be classified and graded does not exist in the data classification and grading keyword library, the data to be classified and graded is input into the optimal data classification and grading model to obtain the classification and grading level corresponding to the data to be classified and graded; a consortium chain system is constructed, wherein the roles in the consortium chain system include data suppliers, data exchanges and data buyers; in response to a request for setting data circulation rules, data circulation rules are set, wherein the data circulation rules include the classification and grading level corresponding to each data buyer; in response to a transaction request, it is determined whether the data to be traded meets the classification and grading level corresponding to the data buyer, and if so, the transaction is completed; the problem that the accurate data classification and grading level cannot be obtained in the prior art is solved, and data compliance transactions based on big data mining technology are realized through accurate data classification and grading levels. BRIEF DESCRIPTION OF THE DRAWINGS
[0050] In order to more clearly illustrate the implementation methods of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for the implementation methods or the description of the prior art. Obviously, the drawings in the following description are only exemplary, and for ordinary technicians in this field, other implementation drawings can be derived from the provided drawings without creative work.
[0051] The structures, proportions, sizes, etc. illustrated in this specification are only used to match the contents disclosed in the specification so as to facilitate understanding and reading by persons familiar with the technology. They are not used to limit the conditions under which the present invention can be implemented, and therefore have no substantial technical significance. Any structural modification, change in proportion or adjustment of size shall still fall within the scope of the technical contents disclosed in the present invention without affecting the effects and purposes that can be achieved by the present invention.
[0052] Figure 1 It is a flow chart of the data compliance transaction method based on big data mining technology of the present invention;
[0053] Figure 2 This is an architecture diagram of the data compliance transaction system based on big data mining technology of the present invention;
[0054] Figure 3 This is a schematic diagram of the physical structure of the electronic device provided by the present invention.
[0055] The accompanying drawings are denoted as follows:
[0056] Acquisition module 10, preprocessing module 20, construction module 30, classification and grading module 40, data circulation module 50, electronic device 60, processor 601, memory 602, bus 603. DETAILED DESCRIPTION
[0057] The following is a description of the implementation of the present invention by specific embodiments. People familiar with the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0058] Figure 1 The flowchart of the data compliance transaction method embodiment based on big data mining technology of the present invention is as follows: Figure 1 As shown, a data compliance transaction method based on big data mining technology provided by an embodiment of the present invention includes the following steps:
[0059] S101, obtaining component B1 in the front-end page A1 in the development environment, a back-end data interface or a data interaction call information request C1 for the component B1 operation docking, and obtaining server log data based on the data interaction call information request C1;
[0060] Specifically: the number of components B1 is one or more, and the number of data interaction call information requests C1 is one or more.
[0061] S102, obtaining operation function items based on server log data, and obtaining data sets, data tables, data fields and data processing rules corresponding to the operation function items under different request conditions;
[0062] S103, obtaining the data set, data table, data field and data processing rules called by the backend trigger task and the trigger mechanism;
[0063] S102 and S103 specifically collect which fields of data tables are called by the operation functions of each system front end, and which components are used for display in the front end;
[0064] Collect the data of which front-end components are stored by the front-end operation functions of each system, and what rules are used to transform the stored data based on the front-end data, and in which field of the data table the data is stored;
[0065] How does the collection backend process the field data of the data table to obtain the field data of the new data table and data set?
[0066] Generate a unique A for the front-end page A1, and generate a unique D for each data entry element component D1 in A;
[0067] Generate a unique B for each component B1, generate a unique C for each data interaction call information request C1, generate a unique T based on different request conditions of C, and generate a unique TT based on T;
[0068] Associate the relationships between A, D, B, C, T, TT, and M;
[0069] Unique identification information is generated for A, D, B, C, T, TT and each association relationship between A, D, B, C, T, TT, and the identification information is stored.
[0070] Determine whether the association relationship between A, D, B, C, T, TT and A, D, B, C, T, TT, M has been stored. If so, do not store it.
[0071] Data storage is stored in the enterprise node system based on the national industrial Internet identification resolution system and completes the identification registration of relevant information in the national identification resolution system.
[0072] For example:
[0073] a. Obtain server log data, analyze the current front-end business system module, page and operation function name, and mark the operation function name with a unique ID1. Based on the data set called by the operating system, obtain the data table and data processing rules called by the data set;
[0074] If it is a query operation, obtain the component of the data indicator displayed on the front-end page. For the above unique ID1, if the server log data has differences in data sets, data tables, data processing rules, data displays, etc. under different conditions, generate different conditions to generate ID2 and the corresponding business system, system module, page, operation function name, called data table or data set, data processing rules, data indicators on the front-end display page and components;
[0075] If it is an addition, deletion or modification operation, the current front-end operation business system module, page and operation function name are analyzed in the same way, and the operation function name is marked with a unique ID1, then the business system, system module, page and component corresponding to the data processed by the addition, deletion or modification, and the data processing rules are obtained. If the server log data has differences in the corresponding components of the data processed under different conditions, the different conditions generate ID2 and the corresponding business system, system module, page and component, as well as the data processing rules;
[0076] For example, if the data processing is generated by a mechanism such as a task trigger on the backend, a unique ID1 is generated corresponding to the task trigger or mechanism. Similarly, ID2 is generated based on different conditions, as well as the data table and data rules called under the corresponding conditions, as well as the data table stored after processing or the generated data set.
[0077] b. A device based on identification resolution technology generates corresponding identification data from the acquired information and registers a unique identification ID3 corresponding to the above-mentioned storage related information;
[0078] c. For each ID1 and ID2, only a unique ID3 is generated. Based on the monitoring of the log data of each server, the device identifies the above-mentioned new operation or task / mechanism information and condition information, and determines whether it has been stored. If it has been stored, the acquisition in a and the storage in b are not performed.
[0079] S104, performing data preprocessing on all data tables and all data fields to obtain preprocessed data;
[0080] Specifically, when the data to be classified and graded is unstructured data, the data to be classified and graded is extracted to obtain field data corresponding to the data to be classified and graded, corresponding data tables and entity relationships, and attribute features corresponding to each data table.
[0081] The stored unstructured data includes documents, tables, pictures, videos, audio, etc., and the corresponding text information is converted. Based on the BERT model, the field data, the corresponding data tables and entity relationships, and the attribute features corresponding to each data table are extracted.
[0082] When the data to be classified and graded is structured data, obtain the source data metadata, obtain the table name, table name annotation, field, field corresponding annotation, and field description in the metadata; obtain the classification field data of the reference data table (for example, obtain the business type data of the business type field in the data table); build a data classification and grading keyword library based on different industry businesses, wherein the data classification and grading keyword library includes the classification and grading levels of the data table and the classification and grading levels of the data fields, determine whether the data to be classified and graded exists in the data classification and grading keyword library, and if so, match the data table name annotation, field annotation, field description and other information to obtain the data classification and grading levels of the corresponding fields of the data table.
[0083] S105, constructing a data classification and grading model, inputting the preprocessed data into the data classification and grading model for training, and obtaining an optimal data classification and grading model;
[0084] Specifically, the threshold range corresponding to the correlation index between data tables and the threshold range corresponding to the correlation index between data fields are preset.
[0085] Input the data to be classified and graded into the optimal data classification and grading model, extract the corresponding key indicator data, and match the corresponding data fields, data tables and entity relationships, and attribute features corresponding to each data table based on the key indicator data;
[0086] Obtaining correlation indicators between data tables and correlation indicators between data fields, and determining whether the correlation indicators exceed a threshold range. When the correlation indicators exceed the threshold range, the current model is not met. Repeating the above operations until the correlation indicators meet the threshold range.
[0087] When the correlation index meets the threshold range, the classification level corresponding to the to-be-classified data is generated.
[0088] The preprocessed data is randomly divided into two parts according to a ratio of 7:3 (not limited to), model training is performed based on 70% of the preprocessed data, and model verification is performed based on 30% of the preprocessed data, and a model with the best accuracy is formed through automatic parameter adjustment; the threshold range corresponding to the correlation index between data tables and the correlation index between data fields is set;
[0089] Import the data to be classified and graded (production data) into the model running data, extract the corresponding key indicator data, match the corresponding data fields, data tables and entity relationships, and the corresponding attribute characteristics of each data table;
[0090] Based on key indicator data, matching corresponding data tables and entity relationships, and corresponding attribute characteristics of each data table, analyze the correlation indicators between data tables and the correlation indicators between data fields, and compare them with the pre-set threshold range. If the threshold range is exceeded, it means that the matching corresponding data fields, data tables and entity relationships, and corresponding attribute characteristics of each data table are inaccurate, and the data is labeled as "not in line with the current model". If it meets the threshold range, the data meets the current model and the extracted data can be applied normally.
[0091] For data that "does not conform to the current model", repeat the above steps until it conforms to the threshold range.
[0092] S106, constructing a keyword library for data classification and grading;
[0093] Specifically, the data classification and grading keyword library includes the classification and grading levels of the data table and the classification and grading levels of the data field.
[0094] S107, determining whether the data to be classified and graded exists in the data classification and graded keyword library, and when the data to be classified and graded exists in the data classification and graded keyword library, matching and obtaining the classification and graded level of the data to be classified and graded;
[0095] S108, when the data to be classified and graded does not exist in the data classification and graded keyword library, the data to be classified and graded is input into the optimal data classification and graded model to obtain the classification and graded level corresponding to the data to be classified and graded;
[0096] S109, building a consortium chain system;
[0097] Specifically, the roles in the alliance chain system include data suppliers, data exchanges and data buyers; the data compliance transaction blockchain system of the alliance chain is connected with the national industrial Internet identification resolution system, that is, when the enterprise node of the data compliance transaction of the alliance chain applies for data compliance in a certain data exchange, the enterprise node of the alliance chain will call the interface data or data product data table name of the enterprise data compliance application to be applied for in the national industrial Internet identification resolution system (mainly based on the data interface table uploaded and stored by the enterprise node or the data table name involved in the data product) and sample data. When the enterprise node applies for compliance, the enterprise node system of the national industrial Internet identification resolution system automatically opens the query permission to this data exchange operation platform node, and the permission content includes the identification data query permission of the data table name corresponding to the data compliance applied for (including and limited to the data table, data field name, data field annotation, creation time and the classification and grading information corresponding to the data table and data field, and the association relationship of the data table is not open to the public). The data exchange operation and management node on the alliance chain corresponding to the data compliance transaction of the data exchange selected by the enterprise when applying has the permission to view the corresponding identification data. Other alliance chain nodes do not have the permission to view and download to local.
[0098] Specifically: based on the data trading alliance chain, the data supplier initiates a compliance registration application for the data table name of the interface data or data product (mainly based on the data interface table uploaded and stored by the enterprise node or the data table name involved in the data product) to a data exchange, and generates the corresponding data block. The data exchange node downloads the data block to the local, and the node system automatically initiates a query on the identification data of the data table involved in the data compliance application, and obtains the corresponding data table, data field name, data field annotation, creation time, and classification and grading information corresponding to the data table and data field to the local node, and automatically checks the overall classification and grading level of the data table of the current data compliance application based on the data compliance definition rules set by this data exchange.
[0099] Based on the data table and field information recognition model established by the data transaction alliance chain system, it is possible to identify whether the current data involves public data, personal information, core data, important data, or critical information infrastructure. Specifically, it is necessary to establish a corpus based on different identification categories, train the data model based on the corpus, generate the optimal model, import the data table, data table annotation, data field name, field annotation, sample data, etc. to be subject to compliance review into the model, and identify the data subject to compliance review based on the above different categories.
[0100] For data buyers who have settled in a data exchange, based on the classification and grading of data tables and data dictionaries corresponding to the data interfaces or data products of different data suppliers, as well as data circulation requirements, application scope, prohibited scope, whether it involves public data, whether it involves personal information, whether it contains important data, whether it involves key information infrastructure and other rules and requirements, based on the business and qualification review of the data buyers by the exchange operation management node, the data buyers can only obtain the corresponding data for transactions. Through the data compliance constraints of the alliance chain system, the mechanism of data compliance circulation can be better implemented.
[0101] S110, in response to the data circulation rule setting request, setting a data circulation rule;
[0102] Specifically, the data circulation rules include the classification and grading levels corresponding to each data buyer.
[0103] S111, in response to the transaction request, determining whether the data to be traded meets the classification and grading level corresponding to the data buyer, and if so, completing the transaction;
[0104] The data compliance transaction method based on big data mining technology obtains component B1 in the front-end page A1 in the development environment, the back-end data interface or data interaction call information request C1 for component B1 operation docking, and obtains server log data based on the data interaction call information request C1, wherein the number of the component B1 is one or more, and the number of the data interaction call information request C1 is one or more; obtains operation function items based on the server log data, and obtains data sets, data tables, data fields and data processing rules corresponding to the operation function items under different request conditions; obtains data sets, data tables, data fields and data processing rules called by back-end trigger tasks and trigger mechanisms; performs data preprocessing on all data tables and all data fields to obtain preprocessed data; constructs a data classification and grading model, and inputs the preprocessed data into the data classification and grading model for training to obtain the optimal data classification and grading model; constructs a data classification and grading keyword library, wherein, The data classification and grading keyword library includes the classification and grading level of the data table and the classification and grading level of the data field; judging whether the data to be classified and graded exists in the data classification and grading keyword library, when the data to be classified and graded exists in the data classification and grading keyword library, matching and obtaining the classification and grading level of the data to be classified and graded; when the data to be classified and graded does not exist in the data classification and grading keyword library, inputting the data to be classified and graded into the optimal data classification and grading model to obtain the classification and grading level corresponding to the data to be classified and graded; constructing an alliance chain system, wherein the roles on the alliance chain system include data suppliers, various data exchanges and data buyers; responding to a request for setting data circulation rules, setting data circulation rules, wherein the data circulation rules include the classification and grading level corresponding to each data buyer; responding to a transaction request, judging whether the data to be traded meets the classification and grading level corresponding to the data buyer, and if so, completing the transaction. The problem that the accurate data classification and grading level cannot be obtained in the prior art, and the data compliance transaction based on big data mining technology is realized through accurate data classification and grading level is solved.
[0105] Figure 2 Flow chart of an embodiment of a data compliance trading system based on big data mining technology of the present invention; Figure 2 As shown, an embodiment of the present invention provides a data compliance transaction system based on big data mining technology, comprising the following steps:
[0106] The acquisition module 10 is used to acquire the component B1 in the front-end page A1 in the development environment, the back-end data interface or the data interaction call information request C1 for the component B1 operation docking, and acquire the server log data based on the data interaction call information request C1, wherein the number of the component B1 is one or more, and the number of the data interaction call information request C1 is one or more;
[0107] Acquire the operation function items based on the server log data, and acquire the data sets, data tables, data fields and data processing rules corresponding to the operation function items under different request conditions;
[0108] Obtain the data sets, data tables, data fields, and data processing rules called by the backend trigger tasks and trigger mechanisms;
[0109] The preprocessing module 20 is used to perform data preprocessing on all data tables and all data fields to obtain preprocessed data;
[0110] The pre-processing module 20 is also used for:
[0111] When the data to be classified and graded is unstructured data, the data to be classified and graded are extracted to obtain field data corresponding to the data to be classified and graded, corresponding data tables and entity relationships, and attribute features corresponding to each data table.
[0112] A construction module 30 is used to construct a data classification and grading model, and input the pre-processed data into the data classification and grading model for training to obtain an optimal data classification and grading model;
[0113] The building block 30 is also used for:
[0114] The threshold range corresponding to the correlation index between data tables and the threshold range corresponding to the correlation index between data fields are preset.
[0115] Input the data to be classified and graded into the optimal data classification and grading model, extract the corresponding key indicator data, and match the corresponding data fields, data tables and entity relationships, and attribute features corresponding to each data table based on the key indicator data;
[0116] Obtaining correlation indicators between data tables and correlation indicators between data fields, and determining whether the correlation indicators exceed a threshold range. When the correlation indicators exceed the threshold range, the current model is not met. Repeating the above operations until the correlation indicators meet the threshold range.
[0117] When the correlation index meets the threshold range, the classification level corresponding to the to-be-classified data is generated.
[0118] Constructing a data classification and grading keyword library, wherein the data classification and grading keyword library includes the classification and grading levels of the data table and the classification and grading levels of the data field;
[0119] A classification and grading module 40 is used to determine whether the data to be classified and graded exists in the data classification and grading keyword library, and when the data to be classified and graded exists in the data classification and grading keyword library, match and obtain the classification and grading level of the data to be classified and graded;
[0120] When the data to be classified and graded does not exist in the data classification and graded keyword library, the data to be classified and graded is input into the optimal data classification and graded model to obtain the classification and graded level corresponding to the data to be classified and graded;
[0121] The data circulation module 50 is used to build a consortium chain system, wherein the roles in the consortium chain system include data suppliers, data exchanges and data buyers;
[0122] In response to the data circulation rule setting request, setting the data circulation rule, wherein the data circulation rule includes a classification and grading level corresponding to each data buyer;
[0123] In response to the transaction request, determine whether the data to be traded meets the classification and grading level corresponding to the data buyer. If so, complete the transaction.
[0124] The data circulation module 50 is also used for:
[0125] In response to the data supplier's entry request, determine the data products that the data supplier applies for compliance registration, obtain the classification and grading identification data of the data tables and data fields corresponding to the data products, and automatically open the classification and grading identification data permissions to the corresponding data exchanges.
[0126] A data identification model is constructed, and the classification and grading levels corresponding to the current interface data of the data supplier are identified through the data identification model, wherein the classification and grading levels include public data, personal information, core data, important data and critical information infrastructure.
[0127] Construct an enterprise qualification analysis model. When a data buyer enters the data exchange, the classification and grading level corresponding to the data buyer is analyzed through the enterprise qualification analysis model.
[0128] In response to a transaction request, interface data that complies with the classification and grading level corresponding to the data buyer is provided to the data buyer.
[0129] A data compliance transaction system based on big data mining technology of the present invention obtains component B1 in a front-end page A1 in a development environment, a back-end data interface or a data interaction call information request C1 to which component B1 is connected via an acquisition module 10, and obtains server log data based on the data interaction call information request C1, wherein the number of the component B1 is one or more, and the number of the data interaction call information request C1 is one or more; obtains operation function items based on the server log data, and obtains data sets, data tables, data fields and data processing rules corresponding to the operation function items under different request conditions; obtains data sets, data tables, data fields and data processing rules called by back-end trigger tasks and trigger mechanisms; performs data preprocessing on all data tables and all data fields via a preprocessing module 20 to obtain preprocessed data; constructs a data classification and grading model via a construction module 30, inputs the preprocessed data into the data classification and grading model for training, and obtains an optimal data classification and grading model; constructs data classification and grading keywords The data classification and grading keyword library includes the classification and grading level of the data table and the classification and grading level of the data field; the classification and grading module 40 is used to determine whether the data to be classified and graded exists in the data classification and grading keyword library. When the data to be classified and graded exists in the data classification and grading keyword library, the classification and grading level of the data to be classified and graded is obtained by matching; when the data to be classified and graded does not exist in the data classification and grading keyword library, the data to be classified and graded is input into the optimal data classification and grading model to obtain the classification and grading level corresponding to the data to be classified and graded; the alliance chain system is constructed by the data circulation module 50, wherein the roles on the alliance chain system include data suppliers, various data exchanges and data buyers; in response to the data circulation rule setting request, the data circulation rules are set, wherein the data circulation rules include the classification and grading level corresponding to each data buyer; in response to the transaction request, it is determined whether the data to be traded meets the classification and grading level corresponding to the data buyer, and if so, the transaction is completed. The data compliance transaction system based on big data mining technology solves the problem that the accurate data classification and grading level cannot be obtained in the prior art, and realizes the data compliance transaction based on big data mining technology through the accurate data classification and grading level.
[0130] Figure 3 A schematic diagram of the physical structure of an electronic device provided by an embodiment of the present invention, such as Figure 3 As shown, the electronic device 60 includes: a processor 601 (processor), a memory 602 (memory) and a bus 603;
[0131] The processor 601 and the memory 602 communicate with each other via the bus 603;
[0132] The processor 601 is used to call the program instructions in the memory 602 to execute the methods provided by the above-mentioned method embodiments, for example, including: obtaining the component B1 in the front-end page A1 in the development environment, the back-end data interface or data interaction call information request C1 for the component B1 operation docking, and obtaining the server log data based on the data interaction call information request C1, wherein the number of the component B1 is one or more, and the number of the data interaction call information request C1 is one or more; obtaining the operation function item based on the server log data, and obtaining the data set, data table, data field and data processing rules corresponding to the operation function item under different request conditions; obtaining the data set, data table, data field and data processing rules called by the back-end trigger task and trigger mechanism; performing data preprocessing on all data tables and all data fields to obtain preprocessed data; constructing a data classification and grading model, and inputting the preprocessed data into the data classification and grading model for training to obtain the optimal data classification and grading model; constructing a data classification and grading model. According to the classification and grading keyword library, wherein the data classification and grading keyword library includes the classification and grading level of the data table and the classification and grading level of the data field; judging whether the data to be classified and graded exists in the data classification and grading keyword library, when the data to be classified and graded exists in the data classification and grading keyword library, matching and obtaining the classification and grading level of the data to be classified and graded; when the data to be classified and graded does not exist in the data classification and grading keyword library, inputting the data to be classified and graded into the optimal data classification and grading model to obtain the classification and grading level corresponding to the data to be classified and graded; constructing an alliance chain system, wherein the roles on the alliance chain system include data suppliers, various data exchanges and data buyers; in response to a request for setting data circulation rules, setting data circulation rules, wherein the data circulation rules include the classification and grading level corresponding to each data buyer; in response to a transaction request, judging whether the data to be traded meets the classification and grading level corresponding to the data buyer, and if so, completing the transaction.
[0133] The present embodiment provides a non-transitory computer-readable storage medium, which stores computer instructions. The computer instructions enable a computer to execute the methods provided by the above-mentioned method embodiments, for example, including: obtaining component B1 in the front-end page A1 in the development environment, the back-end data interface or data interaction call information request C1 to which component B1 operates, and obtaining server log data based on the data interaction call information request C1, wherein the number of the components B1 is one or more, and the number of the data interaction call information request C1 is one or more; obtaining operation function items based on the server log data, and obtaining data sets, data tables, data fields and data processing rules corresponding to the operation function items under different request conditions; obtaining data sets, data tables, data fields and data processing rules called by back-end trigger tasks and trigger mechanisms; performing data preprocessing on all data tables and all data fields to obtain preprocessed data; constructing a data classification and grading model, and inputting the preprocessed data into the data classification and grading model for training to obtain the most An optimal data classification and grading model is provided; a data classification and grading keyword library is constructed, wherein the data classification and grading keyword library includes the classification and grading levels of the data table and the classification and grading levels of the data field; it is determined whether the data to be classified and graded exists in the data classification and grading keyword library, and when the data to be classified and graded exists in the data classification and grading keyword library, the classification and grading levels of the data to be classified and graded are obtained by matching; when the data to be classified and graded does not exist in the data classification and grading keyword library, the data to be classified and graded is input into the optimal data classification and grading model to obtain the classification and grading levels corresponding to the data to be classified and graded; an alliance chain system is constructed, wherein the roles on the alliance chain system include data suppliers, data exchanges and data buyers; in response to a request for setting data circulation rules, data circulation rules are set, wherein the data circulation rules include the classification and grading levels corresponding to each data buyer; in response to a transaction request, it is determined whether the data to be traded meets the classification and grading levels corresponding to the data buyer, and if so, the transaction is completed.
[0134] A person skilled in the art can understand that all or part of the steps of implementing the above method embodiment can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above method embodiment; and the aforementioned storage medium includes: ROM, RAM, magnetic disk or optical disk, etc., various storage media that can store program codes.
[0135] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, i.e., they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Those of ordinary skill in the art may understand and implement it without creative effort.
[0136] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods of each embodiment or some parts of the embodiment.
[0137] Although the present invention has been described in detail above by general description and specific embodiments, it is obvious to those skilled in the art that some modifications or improvements can be made to the present invention. Therefore, these modifications or improvements made without departing from the spirit of the present invention all belong to the scope of protection claimed by the present invention.
Claims
1. A data compliance transaction method based on big data mining technology, characterized in that: The data compliance transaction method based on big data mining technology specifically includes: Obtain component B1 in the front-end page A1 in the development environment, a back-end data interface or a data interaction call information request C1 for the component B1 operation docking, and obtain server log data based on the data interaction call information request C1, wherein the number of the component B1 is one or more, and the number of the data interaction call information request C1 is one or more; Acquire the operation function items based on the server log data, and acquire the data sets, data tables, data fields and data processing rules corresponding to the operation function items under different request conditions; Obtain the data sets, data tables, data fields, and data processing rules called by the backend trigger tasks and trigger mechanisms; Perform data preprocessing on all data tables and all data fields to obtain preprocessed data; Constructing a data classification and grading model, inputting the preprocessed data into the data classification and grading model for training, and obtaining an optimal data classification and grading model; Constructing a data classification and grading keyword library, wherein the data classification and grading keyword library includes the classification and grading levels of the data table and the classification and grading levels of the data field; Determine whether the data to be classified and graded exists in the data classification and grading keyword library, and when the data to be classified and graded exists in the data classification and grading keyword library, obtain the classification and grading level of the data to be classified and graded by matching; When the data to be classified and graded does not exist in the data classification and grading keyword library, the data to be classified and graded is input into the optimal data classification and grading model to obtain the classification and grading level corresponding to the data to be classified and graded; Build a consortium chain system, where the roles on the consortium chain system include data suppliers, data exchanges and data buyers; In response to the data circulation rule setting request, setting the data circulation rule, wherein the data circulation rule includes a classification and grading level corresponding to each data buyer; In response to the transaction request, determine whether the data to be traded meets the classification and grading level corresponding to the data buyer. If so, complete the transaction.
2. The data compliance transaction method based on big data mining technology according to claim 1 is characterized in that: The data preprocessing of all data tables and all data fields to obtain preprocessed data includes: When the data to be classified and graded is unstructured data, the data to be classified and graded are extracted to obtain field data corresponding to the data to be classified and graded, corresponding data tables and entity relationships, and attribute features corresponding to each data table.
3. The data compliance transaction method based on big data mining technology according to claim 1 is characterized in that: The constructing of the data classification and grading model, inputting the pre-processed data into the data classification and grading model for training, and obtaining the optimal data classification and grading model, comprises: The threshold range corresponding to the correlation index between data tables and the threshold range corresponding to the correlation index between data fields are preset.
4. The data compliance transaction method based on big data mining technology according to claim 3 is characterized in that: The constructing of the data classification and grading model, inputting the pre-processed data into the data classification and grading model for training, and obtaining the optimal data classification and grading model, further comprises: Input the data to be classified and graded into the optimal data classification and grading model, extract the corresponding key indicator data, and match the corresponding data fields, data tables and entity relationships, and attribute features corresponding to each data table based on the key indicator data; Obtaining correlation indicators between data tables and correlation indicators between data fields, and determining whether the correlation indicators exceed a threshold range. When the correlation indicators exceed the threshold range, the current model is not met. Repeating the above operations until the correlation indicators meet the threshold range. When the correlation index meets the threshold range, the classification level corresponding to the to-be-classified data is generated.
5. The data compliance transaction method based on big data mining technology according to claim 1 is characterized in that: The construction of the alliance chain system includes: In response to the data supplier's entry request, determine the data products that the data supplier applies for compliance registration, obtain the classification and grading identification data of the data tables and data fields corresponding to the data products, and automatically open the classification and grading identification data permissions to the corresponding data exchanges.
6. The data compliance transaction method based on big data mining technology according to claim 1 is characterized in that: The construction of the alliance chain system also includes: A data identification model is constructed, and the classification and grading levels corresponding to the current interface data of the data supplier are identified through the data identification model, wherein the classification and grading levels include public data, personal information, core data, important data and critical information infrastructure.
7. The data compliance transaction method based on big data mining technology according to claim 1 is characterized in that: The construction of the alliance chain system also includes: Construct an enterprise qualification analysis model. When a data buyer enters the data exchange, the classification and grading level corresponding to the data buyer is analyzed through the enterprise qualification analysis model. In response to a transaction request, interface data that complies with the classification and grading level corresponding to the data buyer is provided to the data buyer.
8. A data compliance trading system based on big data mining technology, characterized in that: include: An acquisition module is used to acquire component B1 in the front-end page A1 in the development environment, a back-end data interface or a data interaction call information request C1 for the component B1 operation docking, and acquire server log data based on the data interaction call information request C1, wherein the number of the component B1 is one or more, and the number of the data interaction call information request C1 is one or more; Acquire the operation function items based on the server log data, and acquire the data sets, data tables, data fields and data processing rules corresponding to the operation function items under different request conditions; Obtain the data sets, data tables, data fields, and data processing rules called by the backend trigger tasks and trigger mechanisms; A preprocessing module is used to perform data preprocessing on all data tables and all data fields to obtain preprocessed data; A construction module is used to construct a data classification and grading model, and input the preprocessed data into the data classification and grading model for training to obtain an optimal data classification and grading model; Constructing a data classification and grading keyword library, wherein the data classification and grading keyword library includes the classification and grading levels of the data table and the classification and grading levels of the data field; A classification and grading module, used to determine whether the data to be classified and graded exists in the data classification and grading keyword library, and when the data to be classified and graded exists in the data classification and grading keyword library, match and obtain the classification and grading level of the data to be classified and graded; When the data to be classified and graded does not exist in the data classification and grading keyword library, the data to be classified and graded is input into the optimal data classification and grading model to obtain the classification and grading level corresponding to the data to be classified and graded; The data circulation module is used to build a consortium chain system, where the roles in the consortium chain system include data suppliers, data exchanges and data buyers; In response to the data circulation rule setting request, setting the data circulation rule, wherein the data circulation rule includes a classification and grading level corresponding to each data buyer; In response to the transaction request, determine whether the data to be traded meets the classification and grading level corresponding to the data buyer. If so, complete the transaction.
9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.
10. A non-transitory computer readable medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.
