Data processing method and device, storage medium and computer program product

Abstract

The invention discloses a data processing method and device, a storage medium and a computer program product. The method comprises the steps of determining a file type of a to-be-processed file; extracting attribute information and key data of the to-be-processed file according to the file type; determining the first sensitivity of the to-be-processed file according to the attribute information; determining an importance degree weight corresponding to the key data; determining a second sensitivity of the to-be-processed file according to the importance degree weight; determining the comprehensive sensitivity of the to-be-processed file according to the first sensitivity and the second sensitivity; and determining the violation condition of the to-be-processed file according to the comprehensive sensitivity. According to the technical scheme, different emphasis violation judgments are adopted for different file types, and file sensitivity judgments are more in line with real use. The calculation of the file sensitivity is considered from multiple dimensions, the sensitive degree of the file is comprehensively analyzed, and the judgment result of file violation is more accurate.

Description

Technical Field

[0001] The present application relates to the field of data security technology, and specifically to a data processing method, device, storage medium and computer program product. Background Art

[0002] The main method of data leakage protection before is to set sensitive keywords or use regular expressions to determine whether there is specified illegal content in the file content. However, a single keyword and regular expression only judges the inclusion of the specified keyword, ignoring the integrity, logic and completeness of the file content, resulting in a high matching rate of irrelevant files. In addition, it will also cause the real sensitive files to not be audited because they do not contain the specified content, resulting in a waste of detection resources and serious missed audits, and it is also very difficult to manually review during subsequent backtracking. Summary of the invention

[0003] The purpose of the embodiments of the present application is to provide a data processing method, device, storage medium and computer program product to solve the technical problem of serious omission of illegal files in the prior art.

[0004] In order to achieve the above-mentioned purpose, the first aspect of the present application provides a data processing method, comprising:

[0005] Determine the file type of the file to be processed;

[0006] Extract attribute information and key data of the file to be processed according to the file type;

[0007] Determine a first sensitivity of the file to be processed according to the attribute information;

[0008] Determine the importance weights corresponding to key data;

[0009] Determine the second sensitivity of the file to be processed according to the importance weight;

[0010] Determining a comprehensive sensitivity of the file to be processed according to the first sensitivity and the second sensitivity;

[0011] The violations of the files to be processed are determined based on their comprehensive sensitivity.

[0012] In an embodiment of the present application, when the file type is a structured data type, the attribute information is the number of rows and columns of the file to be processed, and the key data is the data content of each column of the file to be processed.

[0013] In an embodiment of the present application, determining the second sensitivity of the file to be processed according to the importance weight includes: determining whether there is a preset title cell in each column of data content; when there is a preset title cell in each column of data content, determining the first credibility corresponding to each column of data content; when there is no preset title cell in each column of data content, determining the second credibility corresponding to each column of data content; determining the second sensitivity of the file to be processed according to all first credibility, all second credibility and all importance weights.

[0014] In an embodiment of the present application, determining the first credibility corresponding to each column of data content includes: matching the data content of each column with preset title cells based on regular expressions to determine the first credibility corresponding to each column of data content; in the case where the preset title cells do not exist in each column of data content, determining the second credibility corresponding to each column of data content, including: comparing the data lengths of all cells in each column of data content without preset title cells; in the case where the data lengths of all cells are the same, matching each cell in each column of data content based on regular expressions to determine the second credibility corresponding to each column of data content; in the case where the data lengths of all cells are not the same, extracting the first keyword in each column of data content to determine the second credibility corresponding to each column of data content based on the first keyword.

[0015] In an embodiment of the present application, when the file type is an unstructured data type, the attribute information is the number of pages of the file to be processed and the number of characters excluding space characters, and the key data is the second keyword in the file to be processed.

[0016] In an embodiment of the present application, determining the second sensitivity of the file to be processed according to the importance weight includes: determining the frequency of occurrence of each second keyword in the file to be processed; and determining the second sensitivity of the file to be processed according to the frequency of occurrence of all keywords and the importance weights of all second keywords.

[0017] In an embodiment of the present application, determining the first sensitivity of the file to be processed based on the attribute information includes: determining the content density of the file to be processed based on the number of pages of the file to be processed and the number of characters excluding space characters; determining the first sensitivity of the file to be processed based on the content density.

[0018] In an embodiment of the present application, extracting attribute information and key data of a file to be processed according to the file type includes: when the file type is an unstructured data type, determining an intra-class scatter matrix and an inter-class scatter matrix corresponding to the text content included in the file to be processed; determining a target matrix according to the intra-class scatter matrix and the inter-class scatter matrix; determining multiple target eigenvalues ​​in the target matrix, and determining a target eigenvector corresponding to each target eigenvalue in the target matrix; determining a target projection matrix according to all target eigenvectors; and performing dimensionality reduction processing on the file to be processed according to the target projection matrix to determine a second keyword in the text content included in the file to be processed.

[0019] A second aspect of the present application provides a data processing device, including:

[0020] a memory configured to store instructions;

[0021] The processor is configured to call instructions from the memory and implement the above data processing method when executing the instructions.

[0022] A third aspect of the present application provides a machine-readable storage medium, on which instructions are stored, and the instructions are used to enable a machine to execute the above-mentioned data processing method.

[0023] A fourth aspect of the present application provides a computer program product, which implements the above-mentioned data processing method when executed by a processor.

[0024] Through the above technical solution, different file types are judged with different focuses, and the file sensitivity judgment is more in line with actual use. Considering the calculation of file sensitivity from multiple dimensions and comprehensively analyzing the sensitivity of files, the judgment results of file violations can be more accurate, thus solving the problem of relying only on a single fixed content judgment for illegal file judgment, with many false positive files and missed audits of truly sensitive files.

[0025] Other features and advantages of the embodiments of the present application will be described in detail in the subsequent specific implementation section. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] The accompanying drawings are used to provide a further understanding of the embodiments of the present application and constitute a part of the specification. Together with the following specific implementations, they are used to explain the embodiments of the present application, but do not constitute a limitation on the embodiments of the present application. In the accompanying drawings:

[0027] Figure 1 A schematic diagram of a data processing method according to an embodiment of the present application is shown;

[0028] Figure 2 The structure block diagram of a data processing device according to an embodiment of the present application is schematically shown;

[0029] Figure 3 The schematic diagram shows a structure of a computer device according to an embodiment of the present application. DETAILED DESCRIPTION

[0030] In order to make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the technical scheme in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. It should be understood that the specific implementation methods described herein are only used to illustrate and explain the embodiments of the present application, and are not used to limit the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0031] It should be noted that if the embodiments of the present application involve directional indications (such as up, down, left, right, front, back...), the directional indications are only used to explain the relative position relationship, movement status, etc. between the components under a certain specific posture (as shown in the accompanying drawings). If the specific posture changes, the directional indication will also change accordingly.

[0032] In addition, if there are descriptions involving "first", "second", etc. in the embodiments of the present application, the descriptions of "first", "second", etc. are only used for descriptive purposes and cannot be understood as indicating or suggesting their relative importance or implicitly indicating the number of technical features indicated. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In addition, the technical solutions between the various embodiments can be combined with each other, but they must be based on the ability of ordinary technicians in the field to implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such combination of technical solutions does not exist and is not within the scope of protection required by this application.

[0033] Figure 1 The following schematically shows a flow chart of a data processing method according to an embodiment of the present application. Figure 1 As shown, an embodiment of the present application provides a data processing method, which may include the following steps.

[0034] S102, determining the file type of the file to be processed;

[0035] S104, extracting attribute information and key data of the file to be processed according to the file type;

[0036] S106, determining a first sensitivity of the file to be processed according to the attribute information;

[0037] S108, determining the importance weight corresponding to the key data;

[0038] S110, determining a second sensitivity of the file to be processed according to the importance weight;

[0039] S112, determining a comprehensive sensitivity of the file to be processed according to the first sensitivity and the second sensitivity;

[0040] S114, determining the violation status of the file to be processed according to the comprehensive sensitivity.

[0041] It can be understood that the pending files refer to files that need to be audited to prevent data leakage. The pending files include but are not limited to documents, tables, pictures and other files. Different pending files need to be audited in different content dimensions. First, it is necessary to determine the file type of the pending file, that is, to confirm whether the file belongs to a document, a table or a picture. The file type of the pending file can be divided into two categories. One category is structured data types, such as tables, charts, etc. The other category is unstructured data types, such as text and pictures containing text. Structured and unstructured data cover almost all data currently generated. Tables and documents are usually used as typical applications of two different data types. Extract the attribute information and key data of the file. The attribute information can specifically refer to information that can reflect the size of the file. For example, the attribute information of a table file includes the number of rows and columns of the table. Then, the size of the pending file can be determined based on the attribute information, so as to determine the first sensitivity of the pending file. Key data can specifically refer to the core data content in the pending file. Specifically, the key data of the table file is the data content of each column of data in the table. Different key data have different importance, so the importance weights can be divided for different key data in advance, and the second sensitivity corresponding to the key data in the to-be-processed file can be determined according to the importance weights. The comprehensive sensitivity of the to-be-processed file is determined according to the first sensitivity and the second sensitivity. Sensitivity refers to the importance of the data content of the to-be-processed file. The higher the sensitivity, the more serious the adverse effects caused by data leakage. The comprehensive sensitivity includes the sensitivity of different dimensions of the to-be-processed file, including the first sensitivity of attribute information and the second sensitivity of key data. The violation of the to-be-processed file is determined according to the comprehensive sensitivity, and different focuses are adopted for different file types. The judgment of file sensitivity is more in line with actual use. Considering the calculation of file sensitivity from multiple dimensions and comprehensively analyzing the sensitivity of the file can make the judgment result of file violation more accurate, thereby solving the problem that the judgment of illegal files only relies on a single fixed content judgment, there are many false positive files, and the audit of truly sensitive files is missed.

[0042] In an embodiment of the present application, when the file type is a structured data type, the attribute information is the number of rows and columns of the file to be processed, and the key data is the data content of each column of the file to be processed.

[0043] It can be understood that in daily use, tables are generally used to store structured data, so the judgment of table data focuses on the structuring of data, including the number of columns, rows, data content of each column and the consistency of column data content in the table. Specifically, 1) extract the content of the valid data column in the table, and count the number of columns containing data and the number of rows of data in each column, and make a preliminary judgment on the size of the file content according to the number of columns and rows. The number of columns and rows of the file corresponds to different first sensitivities. Different first sensitivities corresponding to different numbers of rows and columns of the document can be customized. For example: if the table is less than 5 columns and 50 rows, the first sensitivity is 0.3, if the table is 5-10 columns and 100 rows, the first sensitivity is 0.6, and if the table is greater than 10 columns and 100 rows, the first sensitivity is 1. The importance weight of key data is determined according to the importance of the data. Specifically, after the column data is integrated, extracted and judged, the importance weight is assigned to this column. The weight of important data is high, and the weight of low-level data is low or not included in the weight. Here, the data importance classification standard is given as follows. Core data include personal information (mobile phone number, ID card number, home address, bank card number, etc.), industry information (professional data content of various industries), etc. The weight of such data can be defined as 100%; important data include email, bank card type, company position, rank, etc. The weight of such data can be defined as 60%; low-level data include selection, gender, longitude and latitude, etc. The weight of such data can be defined as 20%.

[0044] In an embodiment of the present application, determining the second sensitivity of the file to be processed according to the importance weight includes: determining whether there is a preset title cell in each column of data content; when there is a preset title cell in each column of data content, determining the first credibility corresponding to each column of data content; when there is no preset title cell in each column of data content, determining the second credibility corresponding to each column of data content; determining the second sensitivity of the file to be processed according to all first credibility, all second credibility and all importance weights.

[0045] In the case where the file type is a structured data type, it can be determined that the key data of the file to be processed is the data content of each column of data. The data content of each column in the structured data is extracted, and the sensitivity of the data content is analyzed. Specifically, first determine whether there is a preset title cell in the data content of each column. The preset title cell refers to a cell that is pre-set by a technician in a combination of predefined and customized methods. Specifically, the text content of the title row can be defined. For example, if the content of the first row of cells in the column data content is consistent with the defined preset title cell, the first credibility corresponding to the column data content can be determined based on the rules defined for the preset title cell. Credibility refers to the correctness of the column data content. If there is no preset title cell in the column data content, a built-in rule or further extraction of the cell data content can be used to determine the second credibility corresponding to the column data content. It is understandable that the first credibility and the second credibility are relative, and the first credibility and the second credibility are calculated for different types of column data. For the column data with preset title cells, the corresponding importance weight can be determined according to the specific content of its title. For the column data without the preset title cell, it is necessary to further determine the specific classification of the column data according to the specific data content of the cell, so as to determine the importance weight. Further, the second sensitivity of the to-be-processed file is determined according to all the first credibility, all the second credibility and all the importance weights.

[0046] In an embodiment of the present application, determining the first credibility corresponding to each column of data content includes: matching the data content of each column with preset title cells based on regular expressions to determine the first credibility corresponding to each column of data content; in the case where the preset title cells do not exist in each column of data content, determining the second credibility corresponding to each column of data content, including: comparing the data lengths of all cells in each column of data content without preset title cells; in the case where the data lengths of all cells are the same, matching each cell in each column of data content based on regular expressions to determine the second credibility corresponding to each column of data content; in the case where the data lengths of all cells are not the same, extracting the first keyword in each column of data content to determine the second credibility corresponding to each column of data content based on the first keyword.

[0047] It can be understood that a regular expression is an object that describes a character pattern, and pattern matching, search and replacement operations can be performed in a string. For each column of data content with a preset title cell, the content in the data cell of the column usually has a certain regularity. Therefore, the correctness of this column of data can be calculated by a regular expression to determine the first credibility corresponding to the data content of the column. For example, if the content in the first row of cells is any one of a mobile phone number, contact information or telephone number, the data content of the cells in the column of data can be matched by a regular expression. For example, the matching rule can be set to (13[0-9]|14[5|7]|15[0|1|2|3|5|6|7|8|9]|18[0|1|2|3|5|6|7|8|9])\d{8}. The higher the accuracy, the higher the corresponding first credibility.

[0048] For each column of data content that does not have a preset title cell, the data lengths of all its cells are compared. When the data lengths of all cells are the same, each cell in each column of data content is matched based on a regular expression to determine the second credibility corresponding to each column of data content. When the data lengths of all cells are not the same, the regular expression method cannot uniformly match all cells, then the content of the cells in each column of data content can be segmented and extracted, and the first keyword in each column of data content can be extracted to determine the category to which each column of data content belongs based on the first keyword to determine the corresponding second credibility.

[0049] Specifically, for cells containing images, the data type of the cells can be identified to distinguish between image cells and text cells. For image cells, OCR technology is used to identify the content in the image, and the identified text content is used to determine the first credibility or second credibility of the corresponding entire column of data content through the aforementioned method.

[0050] Before identifying the content of each column of data, the content of each column of data in the table can be extracted, and the cells of the data content in the column can be deduplicated. For cells with duplicate content, one of them is retained to avoid the duplicate content affecting the calculation accuracy of the credibility.

[0051] In an embodiment of the present application, when the file type is an unstructured data type, the attribute information is the number of pages of the file to be processed and the number of characters excluding space characters, and the key data is the second keyword in the file to be processed.

[0052] It is understandable that documents generally store unstructured data, so it is necessary to conduct a comprehensive analysis of the overall content of the file. For document files, first obtain the attribute information of the document, mainly including the number of pages and the number of characters excluding space characters. Second, keywords can be extracted by identifying sentences in the document through a natural language recognition model. For example, the LDA algorithm is used to extract topics and keywords from the extracted content. The LDA algorithm can refine sentences into keywords and classify them.

[0053] In an embodiment of the present application, determining the first sensitivity of the file to be processed based on the attribute information includes: determining the content density of the file to be processed based on the number of pages of the file to be processed and the number of characters excluding space characters; determining the first sensitivity of the file to be processed based on the content density.

[0054] It can be understood that the content density of the document is first calculated. The smaller the number of document pages, the more words after removing spaces, which means that the content of the document is denser, and the probability of containing sensitive information is higher. Specifically, content density = number of words after removing spaces / number of document pages. The first sensitivity corresponds to different percentages between 0-100 according to different ranges of content density, providing a benchmark for the subsequent calculation of the second sensitivity of the entire document. For example: According to the conventional text size and paragraph layout, a page of A4 paper has about 500-700 words, and the number of words is less than 200, then the content density is low, corresponding to the first sensitivity of 30%. If the number of words is between 200-400, then the content density is medium, corresponding to the second sensitivity of 60. If the number of words is greater than 400, then the content density is high, corresponding to the second sensitivity of 100%.

[0055] In an embodiment of the present application, determining the second sensitivity of the file to be processed according to the importance weight includes: determining the frequency of occurrence of each second keyword in the file to be processed; and determining the second sensitivity of the file to be processed according to the frequency of occurrence of all keywords and the importance weights of all second keywords.

[0056] Specifically, the second sensitivity of the document can be calculated by the frequency of occurrence of the second keyword in the document. The sensitivity of the second keyword = importance weight × frequency of occurrence of the second keyword. If the second keyword is divided into core keywords, important keywords, and low-level keywords, and the importance weight of the core data = 100%, the importance weight of the important data = 60%, and the importance weight of the low-level data = 30%, then the second sensitivity = core keyword weight × frequency of occurrence of the core keyword + important keyword weight × frequency of occurrence of the important keyword + low-level keyword weight × frequency of occurrence of the low-level keyword. That is, the second sensitivity = the sum of the sensitivities of all second keywords.

[0057] In an embodiment of the present application, extracting attribute information and key data of a file to be processed according to the file type includes: when the file type is an unstructured data type, determining an intra-class scatter matrix and an inter-class scatter matrix corresponding to the text content included in the file to be processed; determining a target matrix according to the intra-class scatter matrix and the inter-class scatter matrix; determining multiple target eigenvalues ​​in the target matrix, and determining a target eigenvector corresponding to each target eigenvalue in the target matrix; determining a target projection matrix according to all target eigenvectors; and performing dimensionality reduction processing on the file to be processed according to the target projection matrix to determine a second keyword in the text content included in the file to be processed.

[0058] Specifically, the LDA algorithm is used to extract topics and keywords from the extracted content. The LDA algorithm can refine sentences into keywords and classify them. Input: Dataset D = {(x1, y1), (x2, y2), ..., (x m ,y m )} where any sample x i is an n-dimensional vector, y i ∈{C1,C2,...,C k}, dimension reduced to dimension d. Output: sample set D after dimension reduction ′ .

[0059] The LDA algorithm specifically includes the following steps:

[0060] 1. Calculate the intra-class scatter matrix S w ,The intra-class scatter matrix is ​​a matrix that represents the degree of “closeness” between sample points of the same type.

[0061] 2. Calculate the inter-class scatter matrix S b , the inter-class scatter matrix is ​​a matrix that represents the degree of “distance” between heterogeneous sample points.

[0062] 3. Calculate the target matrix S w -1 S b The purpose of constructing the target matrix is ​​to make the distance between the category centers of data of different categories as large as possible, while making the projection points of data of the same category as close as possible.

[0063] 4. Calculate S w -1 S b The largest d eigenvalues ​​and corresponding d eigenvectors (w1,w2,...,w d ), and get the projection matrix W.

[0064] 5. For each sample feature x in the sample set i , converted into a new sample z i =W T xi .

[0065] 6. Get the output sample set D ′ ={(z1,y1),(z2,y2),...,(z m ,y m )}

[0066] In addition to dimensionality reduction, the above LDA dimensionality reduction algorithm can also be used for classification. A common basic idea of ​​LDA classification is to assume that the sample data of each category conforms to the Gaussian distribution. After projection using LDA, the mean and variance of the projected data of each category can be calculated using maximum likelihood estimation, and then the probability density function of the Gaussian distribution of the category can be obtained. When a new sample arrives, we can project it, and then bring the projected sample features into the Gaussian distribution probability density function of each category, calculate the probability that it belongs to this category, and the category corresponding to the largest probability is the predicted category.

[0067] Furthermore, the comprehensive sensitivity of the file to be processed is the average of the first sensitivity and the second sensitivity. The comprehensive sensitivity is ultimately a value between 0 and 1. The larger the value, the higher the sensitivity. The sensitivity level can be determined based on the size of the comprehensive sensitivity. When the comprehensive sensitivity reaches a specified value, the file to be processed will be identified as a sensitive file. The anti-leakage system will log the sensitive documents and not log the insensitive documents.

[0068] This application provides a data processing method of a specific embodiment, which can be applied to a network data leakage prevention system, and can realize intelligent violation judgment of files in the network, ensure the security of important user data, and prevent data leakage without trace. According to the method described in detail in this application, the specific process is as follows:

[0069] 1. Employee A uses SMTP to send an Excel file containing basic information of 100 employees in the department to the outside of the company through the network of the data leakage prevention system. The Excel file contains the employee's name, mobile phone number, email address, address, and salary. The traffic analysis module of the data leakage prevention system analyzes the SMTP protocol traffic and restores the files sent in the traffic. The traffic analysis module sends the restored files to the detection module. After receiving the restored files, the detection module determines the type of the file. If the file type is Excel, the sensitivity detection logic of Excel is used to determine whether the file is sensitive.

[0070] 2. The detection module statistics file contains 5 columns of data, each column of data is 100 lines, and the first sensitivity is determined to be 60% based on the number of lines and columns in the file.

[0071] 3. The detection program continues to extract data from each column and confirms that the cell data is in text format and does not involve image content extraction. All cell data in each column are judged one by one. First, all cell data in the first column are judged. The first row of content extracted is "Name", which meets the conventional sensitive column title. The first row of this column is determined to be the column title. After calculating the length of each cell content, it is found that the length is between 2-4, and the data meets the column title format. The regular expression of the name is used to check the data in this column to confirm that this column meets the name characteristics. The data in this column is relatively important, so the cell content is deduplicated. No duplicate data is found during the processing, and all data is true. Names are set as important data in the system. The weight of important data is 60%. The sensitivity of this column = data weight 60% × data credibility 100%.

[0072] 4. Use step 3 to judge the data content of all columns in turn. The second column is judged as the mobile phone number, which is the core data, with a sensitivity of 100% × 100%. The third column is judged as the email address, which is the important data, with a sensitivity of 60% × 100%. The fourth column is judged as the address, which is the core data, with a sensitivity of 100% × 100%. The fifth column is judged as the salary, which is the low-level data, with a sensitivity of 20% × 100%.

[0073] 5. The second sensitivity of the final file column data is the sum of all column sensitivities / the number of file columns, that is, (60%×100%+100%×100%+60%×100%+100%×100%+20%×100%) / 5=68%.

[0074] 6. The comprehensive sensitivity of the file = (file size sensitivity 60% + file column data sensitivity 68%) / 2 = 64%.

[0075] 7. Define that a file is considered sensitive if its comprehensive sensitivity is greater than 30%. The sensitivity of this file is 64%, which is greater than the defined sensitivity and is therefore considered a sensitive file.

[0076] 8. The data leakage prevention system records the relevant information sent by this file.

[0077] Through the above technical solutions, different judgment criteria are adopted for the sensitivity judgment of different file types, so that the file sensitivity judgment is more in line with actual use. The sensitivity calculation method adopts multi-dimensional angle calculation to comprehensively analyze the sensitivity of the file, making the result more accurate. Based on natural language and combined with grammatical analysis, accurate judgment of the core content of the file is achieved. The LDA algorithm is used to extract and statistically classify the key words of the file, which can accurately record the details of the core content violations. Intelligent audit judgment is made based on the type, content, classification and industry of the file, so as to scientifically analyze and intelligently protect the user's core data and protect the security of the user's core data.

[0078] Figure 1 FIG. 1 is a flow chart of a data processing method in one embodiment. It should be understood that although Figure 1 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, Figure 1 At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.

[0079] Figure 2 The structure block diagram of a data processing device according to an embodiment of the present application is schematically shown. Figure 2 As shown, an embodiment of the present application provides a data processing device, which may include:

[0080] a memory configured to store instructions;

[0081] The processor is configured to call instructions from the memory and implement the above data processing method when executing the instructions.

[0082] Specifically, in the embodiment of the present application, the processor may be configured to:

[0083] Determine the file type of the file to be processed;

[0084] Extract attribute information and key data of the file to be processed according to the file type;

[0085] Determine a first sensitivity of the file to be processed according to the attribute information;

[0086] Determine the importance weights corresponding to key data;

[0087] Determine the second sensitivity of the file to be processed according to the importance weight;

[0088] Determining a comprehensive sensitivity of the file to be processed according to the first sensitivity and the second sensitivity;

[0089] The violations of the files to be processed are determined based on their comprehensive sensitivity.

[0090] In an embodiment of the present application, when the file type is a structured data type, the attribute information is the number of rows and columns of the file to be processed, and the key data is the data content of each column of the file to be processed.

[0091] In an embodiment of the present application, the processor may also be configured to:

[0092] Determining the second sensitivity of the file to be processed based on the importance weight includes: determining whether there is a preset title cell in each column of data content; when there is a preset title cell in each column of data content, determining the first credibility corresponding to each column of data content; when there is no preset title cell in each column of data content, determining the second credibility corresponding to each column of data content; determining the second sensitivity of the file to be processed based on all first credibility, all second credibility and all importance weights.

[0093] In an embodiment of the present application, the processor may also be configured to:

[0094] Determining the first credibility corresponding to each column of data content includes: matching each column of data content with preset title cells based on regular expressions to determine the first credibility corresponding to each column of data content; in the case that the preset title cells do not exist in each column of data content, determining the second credibility corresponding to each column of data content, including: comparing the data lengths of all cells in each column of data content without preset title cells; in the case that the data lengths of all cells are the same, matching each cell in each column of data content based on regular expressions to determine the second credibility corresponding to each column of data content; in the case that the data lengths of all cells are not the same, extracting the first keyword in each column of data content to determine the second credibility corresponding to each column of data content based on the first keyword.

[0095] In an embodiment of the present application, when the file type is an unstructured data type, the attribute information is the number of pages of the file to be processed and the number of characters excluding space characters, and the key data is the second keyword in the file to be processed.

[0096] In an embodiment of the present application, the processor may also be configured to:

[0097] Determining the second sensitivity of the file to be processed according to the importance weight includes: determining the occurrence frequency of each second keyword in the file to be processed; and determining the second sensitivity of the file to be processed according to the occurrence frequency of all keywords and the importance weights of all second keywords.

[0098] In an embodiment of the present application, the processor may also be configured to:

[0099] Determining the first sensitivity of the file to be processed according to the attribute information includes: determining the content density of the file to be processed according to the number of pages of the file to be processed and the number of characters excluding space characters; and determining the first sensitivity of the file to be processed according to the content density.

[0100] In an embodiment of the present application, the processor may also be configured to:

[0101] Extracting attribute information and key data of a file to be processed according to the file type includes: when the file type is an unstructured data type, determining an intra-class scatter matrix and an inter-class scatter matrix corresponding to the text content included in the file to be processed; determining a target matrix according to the intra-class scatter matrix and the inter-class scatter matrix; determining multiple target eigenvalues ​​in the target matrix, and determining a target eigenvector corresponding to each target eigenvalue in the target matrix; determining a target projection matrix according to all target eigenvectors; and performing dimensionality reduction processing on the file to be processed according to the target projection matrix to determine a second keyword in the text content included in the file to be processed.

[0102] An embodiment of the present application also provides a machine-readable storage medium, on which instructions are stored, and the instructions are used to enable a machine to execute the above-mentioned data processing method.

[0103] An embodiment of the present application also provides a computer program product, which implements the above-mentioned data processing method when the computer program is executed by a processor.

[0104] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 3 As shown. The computer device includes a processor A01, a network interface A02, a memory (not shown in the figure) and a database (not shown in the figure) connected through a system bus. Among them, the processor A01 of the computer device is used to provide computing and control capabilities. The memory of the computer device includes an internal memory A03 and a non-volatile storage medium A04. The non-volatile storage medium A04 stores an operating system B01, a computer program B02 and a database (not shown in the figure). The internal memory A03 provides an environment for the operation of the operating system B01 and the computer program B02 in the non-volatile storage medium A04. The database of the computer device is used to store data processing data. The network interface A02 of the computer device is used to communicate with an external terminal through a network connection. When the computer program B02 is executed by the processor A01, a data processing method is implemented.

[0105] Those skilled in the art will understand that Figure 3 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0106] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0107] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0108] These computer program instructions may also be stored in a computer readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0109] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0110] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0111] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0112] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0113] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0114] The above are only embodiments of the present application and are not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included within the scope of the claims of the present application.

Claims

1. A data processing method, characterized in that: The method comprises: Determine the file type of the file to be processed; Extracting attribute information and key data of the file to be processed according to the file type; Determine a first sensitivity of the file to be processed according to the attribute information; Determine the importance weight corresponding to the key data; Determining a second sensitivity of the to-be-processed file according to the importance weight; Determining a comprehensive sensitivity of the to-be-processed file according to the first sensitivity and the second sensitivity; The violation of the file to be processed is determined according to the comprehensive sensitivity.

2. The data processing method according to claim 1, characterized in that: In the case where the file type is a structured data type, the attribute information is the number of rows and columns of the file to be processed, and the key data is the data content of each column of the file to be processed.

3. The data processing method according to claim 2, characterized in that: Determining the second sensitivity of the to-be-processed file according to the importance weight includes: Determine whether there is a preset title cell in the data content of each column; In the case where there is a preset title cell in each column of data content, determining a first credibility corresponding to each column of data content; In the case that the preset title cell does not exist in each column of data content, determining a second credibility corresponding to each column of data content; The second sensitivity of the to-be-processed file is determined according to all the first credibility levels, all the second credibility levels, and all the importance weights.

4. The data processing method according to claim 3, characterized in that: Determining the first credibility corresponding to the content of each column of data includes: Matching the data content of each column in which the preset title cell exists based on a regular expression to determine a first credibility corresponding to the data content of each column; In the case that the preset title cell does not exist in each column of data content, determining the second credibility corresponding to each column of data content includes: Comparing the data lengths of all cells in each column of data content that do not contain the preset header cell; In the case that the data lengths of all cells are the same, matching each cell of each column of data content based on a regular expression to determine the second credibility corresponding to each column of data content; In the case that the data lengths of all cells are not the same, the first keyword in the content of each column of data is extracted to determine the second credibility corresponding to the content of each column of data according to the first keyword.

5. The data processing method according to claim 1, characterized in that: In the case where the file type is an unstructured data type, the attribute information is the number of pages of the to-be-processed file and the number of characters excluding space characters, and the key data is the second keyword in the to-be-processed file.

6. The data processing method according to claim 5, characterized in that: Determining the second sensitivity of the to-be-processed file according to the importance weight includes: Determine the occurrence frequency of each second keyword in the to-be-processed file; The second sensitivity of the to-be-processed file is determined according to the occurrence frequencies of all the keywords and the importance weights of all the second keywords.

7. The data processing method according to claim 5, characterized in that: Determining the first sensitivity of the to-be-processed file according to the attribute information includes: Determining the content density of the file to be processed according to the number of pages of the file to be processed and the number of characters excluding space characters; A first sensitivity of the file to be processed is determined according to the content density.

8. The data processing method according to claim 1, characterized in that: The extracting the attribute information and key data of the to-be-processed file according to the file type comprises: In the case where the file type is an unstructured data type, determining an intra-class scatter matrix and an inter-class scatter matrix corresponding to text content included in the to-be-processed file; Determine a target matrix according to the intra-class scatter matrix and the inter-class scatter matrix; Determine a plurality of target eigenvalues ​​in the target matrix, and determine a target eigenvector corresponding to each target eigenvalue in the target matrix; Determine the target projection matrix based on all target eigenvectors; A dimension reduction process is performed on the to-be-processed file according to the target projection matrix to determine a second keyword in the text content included in the to-be-processed file.

9. A data processing device, characterized in that: include: a memory configured to store instructions; A processor is configured to call the instructions from the memory and implement the data processing method according to any one of claims 1 to 8 when executing the instructions.

10. A machine-readable storage medium, characterized in that: The machine-readable storage medium stores instructions, which are used to enable a machine to execute the data processing method according to any one of claims 1 to 8.

11. A computer program product, characterized in that When the computer program is executed by a processor, the computer program implements the data processing method according to any one of claims 1 to 8.

Images