Scientific and technological financial platform data sharing method

Abstract

The invention provides a science and technology finance platform data sharing method, which comprises the following steps: extracting key field information and a metadata label from a data stream of a science and technology finance platform, the key field information comprising data content and a source identifier; analyzing a data type characteristic value according to the metadata label, and determining a data type and a corresponding sensitivity level by adopting a logistic regression model; querying a source credibility rule base based on the source identifier, and determining a risk level of the data source; according to the sensitivity level, the risk level and the receiver permission identifier, calculating to obtain a dynamic sensitivity score and generating a dynamic sensitivity tag; according to the risk level mapping table, in combination with the sensitivity level, determining a final desensitization mode of the field; and performing format verification on the processed data stream, transmitting the processed data stream, and recording execution information of the dynamic sensitivity label and the desensitization strategy.

Description

Technical Field

[0001] The present invention relates to the field of information technology, and in particular to a data sharing method for a technology and financial platform. Background Art

[0002] In the process of data sharing on technology and finance platforms, it is necessary to solve the technical problem of dynamic data sensitivity assessment to achieve the secure transmission of shared data. First, it is necessary to accurately identify and extract key field information and metadata tags in the financial data streams of different technology and finance platforms. Secondly, in the process of data type feature analysis, it is necessary to ensure the accuracy of sensitivity level determination. At the same time, the dynamic sensitivity scoring mechanism needs to comprehensively consider the risks of the data source, the data sensitivity level and the recipient's authority, determine the data desensitization strategy based on the dynamic sensitivity score, and desensitize the shared data according to the strategy to ensure the secure transmission of shared data. Summary of the invention

[0003] The present invention provides a technology finance platform data sharing method, which mainly includes:

[0004] Extract key field information and metadata labels from the data stream of the science and technology finance platform, the key field information including data content and source identifier; analyze data type feature values ​​according to the metadata label, and use a logistic regression model to determine the data category and its corresponding sensitivity level; query the source credibility rule library based on the source identifier to determine the risk level of the data source; calculate a dynamic sensitivity score and generate a dynamic sensitivity label according to the sensitivity level, risk level and recipient authority identifier; match a desensitization strategy from a pre-established risk level mapping table according to the dynamic sensitivity label, and perform masking, hashing or encryption processing on sensitive fields; perform format verification on the processed data stream and then transmit it, and record the execution information of the dynamic sensitivity label and the desensitization strategy.

[0005] Furthermore, the method of extracting key field information and metadata tags from the data stream of the science and technology finance platform includes: after obtaining the data packet transmitted by the science and technology finance platform source, preprocessing the data packet, using the Pandas library to remove noise and unify the format; using the DecisionTreeClassifier in Scikit-learn to establish a key field recognition model, and using the key field recognition model to extract content items and identifiers from the preprocessed data packet; for numerical data, using the Pandas library to calculate the mean, variance and extreme value, and generate numerical metadata tags; for text data, using the TfidfVectorizer in Scikit-learn to calculate the word frequency and inverse document frequency, and generate text metadata tags; using the Pandas library to merge the numerical metadata tags and the text metadata tags to generate the final metadata tags.

[0006] Furthermore, the method of analyzing the characteristic values ​​of data types according to metadata labels and determining data categories and their corresponding sensitivity levels using a logistic regression model includes: extracting metadata table structure information from a relational database to obtain label names and data types of all fields; screening fields whose data types are integer or floating point types, and storing their corresponding numeric label values ​​as a list; summing up all numeric label values ​​in the list and dividing by the number to obtain the arithmetic mean as the characteristic value; converting the characteristic value into an input vector acceptable to the logistic regression model in a fixed format; using the logistic regression model to determine the data category, and when the output category is equal to the sensitive data category in the predefined category list, converting the category value into an input format acceptable to the decision tree model; determining the data sensitivity level based on the decision tree model, and outputting the sensitivity level value in the predefined sensitivity level list.

[0007] Furthermore, the method of querying the source credibility rule base based on the source identifier to determine the risk level of the data source includes: standardizing the source code obtained from the data source identifier to obtain a standardized source code; using the standardized source code as input, matching it in a pre-established credibility rule base, and obtaining a weight value corresponding to the source code; calculating the risk value of the data source using a weighted average algorithm based on at least one weight value obtained; if the risk value is lower than a preset risk threshold, determining that the data source is a low-risk source; if the risk value is higher than the preset risk threshold, obtaining a characteristic value of the data source, the characteristic value including at least access frequency, data integrity and historical record; using the characteristic value as input, inputting it into a pre-trained risk level assessment model for classification calculation, and obtaining the risk level of the data source.

[0008] Furthermore, the method of calculating a dynamic sensitivity score and generating a dynamic sensitivity label based on the sensitivity level, risk level and recipient authority identifier includes: obtaining the sensitivity level weight, risk level weight and authority identifier weight for a preset dynamic weight algorithm parameter library; calculating the dynamic sensitivity score using a weighted sum formula based on the weight parameters obtained from the parameter library; comparing the dynamic sensitivity score result with a preset threshold; if the score is higher than the preset threshold, extracting the characteristic values ​​of the sensitivity level, risk level and authority identifier; inputting the processed characteristic values ​​into a pre-trained random forest classification model to obtain a corresponding classification result; converting the classification result into a corresponding dynamic sensitivity level according to a mapping rule pre-defined based on business needs; and generating a corresponding dynamic sensitivity label based on the dynamic sensitivity level.

[0009] Furthermore, the desensitization strategy is matched from a pre-established risk level mapping table according to the dynamic sensitivity label, and masking, hashing or encryption processing is performed on the sensitive field, including: extracting the type and content of the sensitive field for the dynamic sensitivity label obtained from a pre-established database; determining whether the field content contains sensitive information; if the field content meets the preset mask processing conditions, replacing the sensitive part of the field with an asterisk to obtain the masked field content; hashing the masked field content with a SHA256 algorithm to obtain the hashed field content; encrypting the hashed field content with an AES algorithm to obtain the encrypted field; extracting the character frequency and length from the encrypted field, and using the character frequency and length as feature values; inputting the feature values ​​into a pre-trained random forest model, and outputting the sensitivity level of the field through the random forest model; determining the final desensitization method of the field according to the pre-established risk level mapping table and the sensitivity level.

[0010] Furthermore, the final desensitization method of the field is determined based on a pre-established risk level mapping table in combination with the sensitivity level, including: obtaining a pre-established risk level mapping table, the risk level mapping table containing a correspondence between risk levels and sensitivity levels; obtaining the sensitivity level information of the target field from a data dictionary, the sensitivity level information in the data dictionary being consistent with the sensitivity level information in the risk level mapping table; searching the risk level mapping table for the corresponding risk level based on the sensitivity level information to obtain the risk level of the target field; selecting a desensitization method that matches the risk level from a preset desensitization method library, the desensitization method library containing desensitization methods corresponding to different risk levels; if the selected desensitization method contains adjustable parameters, adjusting the parameter values ​​based on the risk level to obtain adjusted desensitization method parameters; binding the adjusted desensitization method parameters to the target field to generate a final desensitization solution.

[0011] Furthermore, fields are extracted one by one from the data stream, and for each of the fields, a judgment is made using the preset field type and length rules to determine whether the field meets the preset requirements; if the type and length of the field match, execution information is generated according to the preset fixed format based on the final desensitization scheme; according to the preset log format requirements, the execution information is converted into structured log data; the converted structured log data is stored in a pre-configured MySQL database; for the data stream that has passed the verification, the Kafka message queue technology is used to transmit it to the preset target system; if the type and length of the field do not match, an error log is recorded, and the field is skipped to continue processing the next field.

[0012] The technical solution provided by the embodiment of the present invention may have the following beneficial effects:

[0013] The present invention discloses a data sharing method for a technology and finance platform. The method extracts key fields and metadata tags in the data stream, analyzes data features and determines the sensitivity level, and evaluates the credibility of the data source, thereby calculating a dynamic sensitivity score. Based on this score, the present invention matches the corresponding desensitization strategy from a preset risk level mapping table, and performs masking, hashing or encryption processing on sensitive fields. This dynamic desensitization method can flexibly select appropriate protection measures according to the sensitivity of the data and the source risk, and effectively balances data security and availability. By format checking the processed data and recording the execution information, the present invention ensures the traceability and compliance of the desensitization process. This innovative method significantly improves the security and efficiency of the technology and finance platform when processing sensitive data, and provides strong support for the secure sharing and application of financial data. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 This is a flow chart of the technology finance platform data sharing method of the present invention. DETAILED DESCRIPTION

[0015] In order to enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below in conjunction with the drawings in the embodiments of this specification. Obviously, the described embodiments are only part of the embodiments of this specification, not all of the embodiments. Based on the embodiments in this specification, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of this specification.

[0016] like Figure 1 The data sharing method of the technology finance platform in this embodiment may specifically include:

[0017] Step S101, extract key field information and metadata tags from the data stream of the technology finance platform, where the key fields include data content and source identifiers.

[0018] The data packets transmitted by the source of the science and technology finance platform are obtained. The data packets contain structured information and unstructured information. The Pandas library is used to preprocess the data packets, remove noise and unify the format to obtain consistent data. The DecisionTreeClassifier in Scikit-learn is used to establish a key field recognition model and obtain model parameters. The preprocessed data packets are processed by the key field recognition model to extract content items and identifiers. The content items are classified according to their data types. If they are numerical data, they enter the numerical data processing flow. If they are text data, they enter the text data processing flow. For numerical data, the Pandas library is used to calculate the mean, variance, and extreme value to generate numerical metadata labels. For text data, the TfidfVectorizer in Scikit-learn is used to calculate the word frequency and inverse document frequency to generate text metadata labels. The numerical metadata labels and text metadata labels are merged through the Pandas library to obtain the final metadata labels.

[0019] Exemplarily, a data packet transmitted by a source of a technology finance platform is obtained, and the data packet contains structured information such as transaction amount, user ID, and unstructured information such as user comments. The Pandas library is used to preprocess the data packet, remove noise such as missing values ​​and outliers, and unify the format such as standardizing the date field to "YYYY-MM-DD" to obtain consistent data. The DecisionTreeClassifier in Scikit-learn is used to establish a key field recognition model, set model parameters such as the maximum depth of 5 and the minimum sample split to 10, and obtain model parameters. The preprocessed data packet is processed by the key field recognition model to extract content items such as transaction amount, user comments, and identifiers such as user ID. The content item is classified according to the data type. If it is numerical data such as transaction amount, it enters the numerical data processing process. If it is text data such as user comments, it enters the text data processing process. For numerical data, the Pandas library is used to calculate the mean such as 1000, the variance such as 500, and the extreme value such as the maximum value of 2000, and generate a numerical metadata label. For text data, use TfidfVectorizer in Scikit-learn to calculate word frequency, such as "transaction" appears 3 times and inverse document frequency, such as 0.5, to generate text metadata tags. Use the Pandas library to merge numerical metadata tags and text metadata tags to obtain the final metadata tags. Associate the extracted key field information, such as transaction amount and user comments, with the generated metadata tags to determine the key field information set containing data content and source identifiers, such as user IDs.

[0020] Step S102: Analyze the data type feature value according to the final metadata tag, and use a logistic regression model to determine the data category and its corresponding sensitivity level.

[0021] Extract the metadata table structure information from the relational database of the final metadata tag, and obtain the tag name and data type of all fields. Filter the fields with integer or floating point data types, and store the corresponding numeric tag values ​​as a list. Sum all the numeric tag values ​​in the list and divide them by the number to obtain the arithmetic mean as the eigenvalue. Convert the eigenvalue into an input vector acceptable to the logistic regression model in a fixed format. Use the logistic regression model pre-trained with historical data to determine the data category and obtain the data category value. If the data category value is equal to the sensitive data category in the predefined category list, convert the category value into an input format acceptable to the decision tree model. Determine the data sensitivity level based on the decision tree model and obtain the sensitivity level value in the predefined sensitivity level list. Calculate the dynamic sensitivity score using the weighted sum formula based on the sensitivity level, risk level and recipient authority identifier. Compare the dynamic sensitivity score result with the preset threshold. If the score is higher than the preset threshold, generate the corresponding dynamic sensitivity label.

[0022] Exemplarily, the metadata table structure information is extracted from the relational database of the final metadata tag, and the tag names and data types of all fields are obtained. For example, the data type of the field name "user age" extracted from the MySQL database is integer, and the data type of the field name "user income" is floating point. The fields with integer or floating point data types are filtered, and the corresponding numerical tag values ​​are stored as lists. For example, the values ​​of the "user age" field [25, 30, 35] and the values ​​of the "user income" field [5000.0, 6000.0, 7000.0] are stored as two lists respectively. The arithmetic mean is obtained by summing all the numerical tag values ​​in the list and dividing them by the number. For example, the average value of the "user age" field is calculated to be 30, and the average value of the "user income" field is 6000.0. The characteristic value is converted into an input vector acceptable to the logistic regression model in a fixed format, for example, [30, 6000.0] is converted into a standardized vector of [0.3, 0.6]. Use the logistic regression model pre-trained with historical data to determine the data category and obtain the data category value. For example, the model output category value is "financial data". If the data category value is equal to the sensitive data category in the predefined category list, convert the category value to an input format acceptable to the decision tree model, for example, convert "financial data" to a one-hot encoding vector of [1,0,0]. The predefined category list contains data category items and category values. The category value of the current category can be obtained by querying the predefined category list; determine the data sensitivity level based on the decision tree model and obtain the sensitivity level value in the predefined sensitivity level list. For example, the model output sensitivity level value is "high".

[0023] Step S103, querying a source credibility rule base based on the source identifier to determine the risk level of the data source.

[0024] Based on the source identifier, a query is performed in the pre-established credibility rule library to obtain the source code corresponding to the source identifier. The credibility rule library contains the source code corresponding to the source identifier and the weight value corresponding to the source code. The obtained source code is standardized to obtain a standardized source code. The standardized source code is used as input and matched in the credibility rule library to obtain the weight value corresponding to the source code. According to at least one weight value obtained, a weighted average algorithm is used to calculate the risk value of the data source, where the risk value is equal to the sum of the products of each weight value and its corresponding score divided by the sum of the weight values. If the risk value is lower than the preset risk threshold, the data source is determined to be a low-risk source, and the data source risk determination process is terminated. If the risk value is higher than the preset risk threshold, the characteristic value of the data source is obtained, and the characteristic value includes access frequency, data integrity and historical records. The characteristic value is used as input and input into the pre-trained random forest model for classification calculation to obtain the risk level of the data source.

[0025] Exemplarily, based on the source identifier "S12345", a query is made in the pre-established credibility rule library to obtain the corresponding source code "C001". For the source code "C001", standardization is performed by removing special characters and unifying the case to obtain the standardized source code "C001-STD". The standardized source code "C001-STD" is used as input and matched in the credibility rule library to obtain the corresponding weight values ​​0.8 and 0.2. According to the weight values ​​0.8 and 0.2 and their corresponding scores 5 and 3, the risk value is calculated using the weighted average algorithm, and the result is (0.8×5+0.2×3) / (0.8+0.2)=4.6. If the risk value 4.6 is lower than the preset risk threshold 5, the data source is determined to be a low-risk source, and the risk determination process ends. If the risk value is higher than the preset risk threshold, the characteristic values ​​of the data source are obtained, including access frequency 120 times / day, data integrity 95%, and good historical records. The characteristic values ​​are input into the pre-trained risk level assessment model for classification calculation to obtain the risk level "medium risk". The risk level assessment model is obtained through unsupervised training based on the random forest model.

[0026] Step S104, based on the sensitivity level, risk level and recipient authority identifier, a dynamic sensitivity score is calculated and a dynamic sensitivity label is generated.

[0027] According to the sensitivity level, risk level and the receiver's permission identifier, the sensitivity level weight, risk level weight and permission identifier weight are obtained from the preset dynamic weight algorithm parameter library. The weighted sum formula is used to multiply the sensitivity level weight with the sensitivity level value, the risk level weight with the risk level value, and the permission identifier weight with the permission identifier value to calculate the dynamic sensitivity score. The dynamic sensitivity score is compared with the preset threshold, and the preset threshold is determined by the percentile method based on business needs. If the dynamic sensitivity score is higher than the preset threshold, the feature values ​​of the sensitivity level, risk level and permission identifier are extracted. The numerical features directly use the original values, and the categorical features are converted to binary vectors through unique hot encoding. The processed feature values ​​are input into the pre-trained random forest classification model to obtain the classification results. According to the mapping rules pre-defined based on business needs, the classification results are converted into the corresponding dynamic sensitivity level. Based on the dynamic sensitivity level, the corresponding dynamic sensitivity label is generated.

[0028] Exemplarily, according to the sensitivity level, risk level and the receiver's permission identification, a sensitivity level weight of 0.5, a risk level weight of 0.3 and a permission identification weight of 0.2 are obtained from the preset dynamic weight algorithm parameter library. These weight values ​​are determined based on the linear regression method of historical data. Using the weighted summation formula, the sensitivity level weight is multiplied by the sensitivity level value 3, the risk level weight is multiplied by the risk level value 4, and the permission identification weight is multiplied by the permission identification value 2, and the dynamic sensitivity score is calculated to be 3×0.5+4×0.3+2×0.2=3.1. The dynamic sensitivity score 3.1 is compared with the preset threshold value 2.8, and the preset threshold value is determined by the percentile method based on business needs. If the dynamic sensitivity score is higher than the preset threshold, the feature values ​​of sensitivity level 3, risk level 4 and permission identification 2 are extracted. The numerical features directly use the original values, and the categorical features are converted into binary vectors [1,0,1] through one-hot encoding. Historical data is obtained, and data features and sample labels are extracted. The feature selection method is used to screen out important features. Construct a training set based on the screened features and sample labels. Train a random forest classification model for the training set. If the model training is completed, obtain the prediction results. According to the prediction results, determine the model accuracy and recall rate. If the model evaluation result meets the preset threshold, it is determined that the model can be used for actual prediction, and a random forest classification model is obtained. Input the processed feature values ​​into the pre-trained random forest classification model, and use 100 decision trees for training to obtain a classification result of "high sensitivity". According to the mapping rules pre-defined based on business needs, convert the classification result "high sensitivity" into the corresponding dynamic sensitivity level "3". Based on the dynamic sensitivity level "3", generate the corresponding dynamic sensitivity label "high".

[0029] Step S105, matching the desensitization strategy from a pre-established risk level mapping table according to the dynamic sensitivity label, and performing masking, hashing or encryption processing on the sensitive field.

[0030] According to the dynamic sensitivity label, the desensitization strategy is matched from the pre-established risk level mapping table. Determine whether the sensitive field needs to be masked. If it meets the preset masking conditions, replace the sensitive part with an asterisk. The masked field content is hashed using the SHA256 algorithm. The hashed field content is encrypted using the AES algorithm. The character frequency and length are extracted from the encrypted field and used as feature values. The extracted feature values ​​are input into the pre-trained random forest model to output the sensitivity level of the field. According to the pre-established risk level mapping table, combined with the sensitivity level, the final desensitization method of the field is determined. If the desensitization method involves parameter adjustment, the desensitization parameter value is adjusted according to the sensitivity level and risk level. Based on the adjusted desensitization parameters, the sensitive field is subjected to the final masking, hashing or encryption processing.

[0031] For example, according to the dynamic sensitivity label "high", the desensitization strategy matched from the risk level mapping table is "mask + hash + encryption". It is judged that the sensitive field "ID number" meets the mask processing conditions, and the middle 8 digits are replaced with asterisks to obtain the mask result "110***********1234". The masked field content "110***********1234" is hashed using the SHA256 algorithm to generate a hash value "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855". The hash value is encrypted using the AES algorithm, and the key is "1234567890123456", generating the ciphertext "U2FsdGVkX18z7Z6X5X6X5X6X5X6X5X6X5X6X5X6X5X6X5X". Extract the character frequency and length feature values ​​from the ciphertext, the frequency is "0.05, 0.03, 0.02", and the length is "64". Obtain the feature value set in the historical data, and input the feature value set as training data into the random forest model. Use the cross-validation method to tune the parameters of the random forest model to determine the optimal model structure. According to the tuned model structure, use the training data to train the random forest model to obtain the initial model. Use the test data set to evaluate the performance of the initial model. If the accuracy is lower than the preset threshold, adjust the feature value set. Retrain the model based on the adjusted feature value set to obtain an optimized sensitivity level assessment model. Input the feature value into the pre-trained random forest model, and the model outputs a sensitivity level of "level 3". According to the risk level mapping table, combined with the sensitivity level "level 3", the final desensitization method is determined to be "encryption parameter adjustment". Adjust the key length of the AES encryption algorithm to "256 bits" and re-encrypt to generate a new ciphertext "U2FsdGVkX18z7Z6X5X6X5X6X5X6X5X6X5X6X5X6X5X6X5X". Based on the adjusted desensitization parameters, perform the final masking, hashing and encryption processing on the sensitive field "ID number" to generate the final desensitization result "U2FsdGVkX18z7Z6X5X6X5X6X5X6X5X6X5X6X5X6X5X6X5X.

[0032] Specifically, according to the pre-established risk level mapping table, combined with the sensitivity level, the final desensitization method of the field is determined, which specifically includes the following steps:

[0033] Obtain the pre-established risk level mapping table and extract the correspondence between risk level and sensitivity level. The risk level mapping table contains the correspondence between risk level and sensitivity level, which is used for the search operation in the subsequent steps. For the target field, obtain its sensitivity level information from the data dictionary. The sensitivity level information in the data dictionary is consistent with the sensitivity level information in the risk level mapping table, which is used for the search operation in the subsequent steps. According to the sensitivity level information, search for the corresponding risk level in the risk level mapping table. The search operation determines the risk level of the target field by matching the sensitivity level information in the data dictionary with the sensitivity level information in the risk level mapping table. Select a desensitization method that matches the risk level from the preset desensitization method library. The desensitization method library contains desensitization methods corresponding to different risk levels. The selection operation determines the applicable desensitization method by matching the risk level of the target field with the risk level in the desensitization method library. If the selected desensitization method contains adjustable parameters, adjust the parameter value according to the risk level. The parameter adjustment operation determines the specific value of the parameter value according to the risk level to ensure the applicability of the desensitization method. Bind the adjusted desensitization method parameters to the target field to generate the final desensitization solution. The binding operation generates the final desensitization solution by applying the adjusted desensitization method parameters to the target field.

[0034] Exemplarily, the sensitivity level can be divided into three levels: low, medium, and high, and the corresponding risk levels may be 1, 2, and 3. This mapping relationship enables the risk level of the data to be quickly determined according to its sensitivity, so as to take corresponding protection measures. As a core component of metadata management, the data dictionary not only stores the structural information of the data, but also contains important attributes such as the sensitivity level. By querying the data dictionary, the sensitivity level information of the target field can be quickly obtained. After determining the sensitivity level of the target field, the corresponding risk level is found in the risk level mapping table. For example, if the sensitivity level of a field is marked as "high", the corresponding risk level may be 3. The desensitization method library is a pre-defined desensitization strategy set that provides corresponding desensitization methods for different risk levels. For example, for data with a risk level of 1, a simple mask processing may be used; while for data with a risk level of 3, a more complex encryption algorithm may be required. This hierarchical processing ensures that the desensitization intensity matches the risk level of the data, which not only protects sensitive information, but also avoids the reduction of data availability caused by excessive desensitization. The parameter adjustment of the desensitization method can be carried out according to the risk level. For example, for mask processing, the number of bits of the mask can be adjusted according to the risk level. Low-risk data may only need to mask the last four digits, while high-risk data may need to mask all characters except the first two. This flexible parameter adjustment mechanism makes the desensitization process more accurate and effective. The generation of the final desensitization plan binds the adjusted desensitization parameters to the target field to form a complete desensitization plan. This plan not only includes the specific desensitization method, but also includes the specific parameters applied to the method. For example, for a field containing a personal ID number, the final desensitization plan may be: use masking, retain the first six digits and the last four digits, and replace the middle eight digits with asterisks. This systematic desensitization strategy formulation process not only improves the efficiency of data protection, but also enhances the traceability and consistency of the entire process. By integrating risk assessment, desensitization method selection, and parameter adjustment into a unified framework, it can better balance the needs of data security and data availability, providing a strong support for the organization's data management practices.

[0035] Step S106, performing format check on the processed data stream before transmission, and recording the execution information of the dynamic sensitivity state label and the desensitization strategy.

[0036] Fields are extracted from the data stream one by one, and for each field, the preset field type and length rules are used to judge whether the field meets the preset requirements; if the type and length of the field match, execution information is generated according to the preset fixed format according to the final desensitization scheme; according to the preset log format requirements, the execution information is converted into structured log data; the converted structured log data is stored in a pre-configured MySQL database; for the data stream that has passed the verification, the Kafka message queue technology is used to transmit it to the preset target system; if the type and length of the field do not match, an error log is recorded, and the field is skipped to continue processing the next field.

[0037] For example, first, you need to extract the fields one by one and match the type and length. For example, for a data stream containing user information, the name field may be set to a character type with a length of no more than 20 characters; the age field may be set to an integer type with a range of 0 to 150. Such preset rules help to quickly identify abnormal data. When the field passes the verification, according to the final desensitization scheme, the execution information is converted into structured log data for easy storage and analysis. Structured logs usually include information such as timestamps, operation types, field names, and processing results. This formatted log can help quickly locate problems and perform data statistics and analysis. MySQL provides powerful query functions and good performance, which can meet most log storage and retrieval needs. You can create a special log table containing various related fields, such as operation time, field name, sensitivity, etc. For the data that passes the verification, choose to use Kafka message queues for transmission. Different topics can be set for different types of data to ensure that the data can be accurately transmitted to the corresponding target system. When encountering a field that does not meet the preset rules, an error log is recorded and the processing is skipped. This ensures that the processing of the entire data stream will not be interrupted by individual abnormal data. The error log should contain enough information, such as field name, actual value, expected type and length, for subsequent troubleshooting and data correction. The entire process is designed to improve the efficiency and security of data processing. Through verification by preset rules, abnormal data can be discovered and processed in a timely manner. The application of dynamic sensitivity labels and desensitization rules helps to protect user privacy. The use of structured logs and message queues improves the maintainability and scalability of the system.

[0038] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical scheme and inventive concept of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

Claims

1. A data sharing method for a technology finance platform, characterized in that: include: Extract key field information and metadata tags from the data stream of the technology finance platform, wherein the key field information includes data content and source identifier; Analyze the data type feature value according to the metadata tag, and use a logistic regression model to determine the data category and its corresponding sensitivity level; Querying a source credibility rule base based on the source identifier to determine the risk level of the data source; According to the sensitivity level, risk level and recipient authority identifier, a dynamic sensitivity score is calculated and a dynamic sensitivity label is generated; Matching a desensitization strategy from a pre-established risk level mapping table according to the dynamic sensitivity label, and performing masking, hashing or encryption processing on the sensitive field; The processed data stream is format-checked and then transmitted, and the execution information of the dynamic sensitivity label and the desensitization strategy is recorded.

2. The technology finance platform data sharing method according to claim 1, characterized in that: The extracting of key field information and metadata tags from the data stream of the technology finance platform includes: After obtaining the data packets transmitted by the source of the science and technology finance platform, the data packets are preprocessed, and the Pandas library is used to remove noise and unify the format; Using DecisionTreeClassifier in Scikit-learn to establish a key field recognition model, and using the key field recognition model to extract content items and identifiers from the preprocessed data packets; For numerical data, use the Pandas library to calculate the mean, variance, and extreme value, and generate numerical metadata labels; For text data, use TfidfVectorizer in Scikit-learn to calculate word frequency and inverse document frequency and generate text metadata tags; Use the Pandas library to merge the numeric metadata labels and text metadata labels to generate the final metadata labels.

3. The technology finance platform data sharing method according to claim 2, characterized in that: Analyzing the data type characteristic value according to the metadata tag and using a logistic regression model to determine the data category and its corresponding sensitivity level includes: Extract metadata table structure information from the relational database of the final metadata tags, and obtain the tag names and data types of all fields; Filter fields whose data types are integer or floating point, and store their corresponding numeric tag values ​​as a list; Sum all the numeric label values ​​in the list and divide them by the number to get the arithmetic mean as the feature value; Convert the feature values ​​into an input vector acceptable to the logistic regression model in a fixed format; Use a logistic regression model pre-trained with historical data to determine the data category, and when the output category is equal to a sensitive data category in a predefined category list, convert the output category value into an input format acceptable to the decision tree model; The data sensitivity level is judged based on the decision tree model, and the output result is the sensitivity level value in the predefined sensitivity level list.

4. The technology finance platform data sharing method according to claim 3 is characterized in that: The querying of the source credibility rule base based on the source identifier to determine the risk level of the data source includes: Performing standardization processing on the source code obtained from the data source identifier to obtain a standardized source code; Taking the standardized source code as input, matching it in a pre-established credibility rule library to obtain a weight value corresponding to the source code; Calculate the risk value of the data source using a weighted average algorithm based on at least one weight value obtained; If the risk value is lower than a preset risk threshold, the data source is determined to be a low-risk source; If the risk value is higher than a preset risk threshold, obtaining characteristic values ​​of the data source, the characteristic values ​​including access frequency, data integrity and historical records; The characteristic value is used as input into a pre-trained risk level assessment model for classification calculation to obtain the risk level of the data source.

5. The technology finance platform data sharing method according to claim 4 is characterized in that: The step of calculating a dynamic sensitivity score and generating a dynamic sensitivity label according to the sensitivity level, risk level and recipient authority identifier includes: Obtain sensitivity level weights, risk level weights and permission identification weights for the preset dynamic weight algorithm parameter library; Calculating the dynamic sensitivity score using a weighted summation formula based on the weight parameters obtained from the parameter library; Compare the dynamic sensitivity score results with a preset threshold; If the score is higher than the preset threshold, extracting the characteristic values ​​of the sensitivity level, risk level and authority identification; The processed feature values ​​are input into the pre-trained random forest classification model to obtain the corresponding classification results; According to a mapping rule pre-defined based on business requirements, the classification result is converted into a corresponding dynamic sensitivity level; Based on the dynamic sensitivity level, a corresponding dynamic sensitivity label is generated.

6. The technology finance platform data sharing method according to claim 5, characterized in that: The step of matching a desensitization strategy from a pre-established risk level mapping table according to the dynamic sensitivity label and performing masking, hashing or encryption processing on the sensitive field includes: Get the dynamic sensitivity label and extract the type and content of the sensitive field; If the field content meets the preset mask processing conditions, the sensitive part of the field is replaced with an asterisk to obtain the field content after mask processing; The masked field content is hashed using the SHA256 algorithm to obtain the hashed field content; The AES algorithm is used to encrypt the hashed field content to obtain an encrypted field. Extracting character frequency and length from the encrypted field, and using the character frequency and length as feature values; Inputting the feature value into a pre-trained random forest model, and outputting the sensitivity level of the field through the random forest model; According to the pre-established risk level mapping table, combined with the sensitivity level, the final desensitization method of the field is determined.

7. The technology finance platform data sharing method according to claim 6, characterized in that: The method of determining the final desensitization method of the field according to the pre-established risk level mapping table and the sensitivity level includes: Obtaining a pre-established risk level mapping table, wherein the risk level mapping table includes a correspondence between risk levels and sensitivity levels; Acquire the sensitivity level information of the target field from the data dictionary, wherein the sensitivity level information in the data dictionary is consistent with the sensitivity level information in the risk level mapping table; According to the sensitivity level information, searching the corresponding risk level in the risk level mapping table to obtain the risk level of the target field; Selecting a desensitization method that matches the risk level from a preset desensitization method library, wherein the desensitization method library contains desensitization methods corresponding to different risk levels; If the selected desensitization method includes adjustable parameters, the parameter values ​​are adjusted according to the risk level to obtain adjusted desensitization method parameters; The adjusted desensitization method parameters are bound to the target field to generate a final desensitization solution.

8. The technology finance platform data sharing method according to claim 7, characterized in that: The processing of the processed data stream after format verification and transmission, and recording the execution information of the dynamic sensitivity label and the desensitization strategy, include: Extracting fields one by one from the data stream, and judging each field using a preset field type and length rule to determine whether the field meets the preset requirements; If the type and length of the field match, then generating execution information in a preset fixed format according to the final desensitization scheme; According to the preset log format requirements, convert the execution information into structured log data; Store the converted structured log data into a pre-configured MySQL database; For the data stream that has passed the verification, Kafka message queue technology is used to transmit it to the preset target system; If the type and length of the field do not match, an error log is recorded, the field is skipped, and the next field is processed.

Images