Data processing method and device, equipment and storage medium

Abstract

The invention provides a data processing method and device, equipment and a storage medium, and can be applied to the technical field of big data. The method comprises the following steps: in response to a received export request of a user for target data, analyzing the export request to obtain a user identity identifier and a data export identifier; matching based on the user identity identifier to obtain a user permission; obtaining an export weight score of each data in the target data according to a processing rule for the target data indicated by the user permission; under the condition that the sum of the export weight scores of all the data in the target data is larger than a preset score threshold value, dividing the target data into N target data subsets, and asynchronously obtaining all the target data subsets to obtain the target data; wherein the sum of the export weight scores of the data in the target data subset is smaller than a preset score threshold value; and matching predefined export configuration information based on the data export identifier, and writing the target data into a table by utilizing an export strategy indicated by the export configuration information.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of big data, and in particular to a data processing method and device, equipment, medium and program product. BACKGROUND

[0002] In the data processing process, data stored in a database or various computer systems often needs to be exported to a spreadsheet for data analysis and processing. Since different users have different data access permissions, the data needs to be desensitized according to the user permissions when exporting the data. However, in the case of a large amount of data to be exported, performance bottlenecks will be encountered, especially in scenarios where desensitization is required, both data export and desensitization will occupy system resources, and problems such as excessive memory occupation and processing timeout are likely to occur. SUMMARY

[0003] In view of the above problems, the present application provides a data processing method and device for improving data export efficiency, equipment, medium and program product.

[0004] According to a first aspect of the present application, a data processing method is provided, comprising: in response to receiving a user's export request for target data, parsing the export request to obtain a user identity and data export identifier; matching the user identity to obtain the user's permissions; according to the processing rules of the target data indicated by the user's permissions, obtaining the export weight scores of each data in the target data, the export weight scores indicating the resource occupation amount of performing data processing on the target data based on the processing rules; in the case that the sum of the export weight scores of each data in the target data is greater than a preset score threshold, dividing the target data into N target data subsets, and asynchronously obtaining each target data subset to obtain the target data; wherein the sum of the export weight scores of each data in the target data subset is less than the preset score threshold; N is a positive integer greater than 1; matching the data export identifier with the predefined export configuration information, and using the export strategy indicated by the export configuration information to write the target data into a table.

[0005] According to an embodiment of the present application, according to the processing rules of the target data indicated by the user's permissions, obtaining the export weight scores of each data in the target data comprises: in the case that the processing rules indicated by the user's permissions are to replace sensitive fields, obtaining the field length of the field to be replaced, and determining the export weight score according to the field length; the export weight score is proportional to the field length.

[0006] According to an embodiment of the present application, the table includes at least one data page, and a data export strategy indicated by export configuration information is used to write target data into the table, including: in the case that the export configuration information includes a data export identifier, obtaining a data page download configuration table and a table header configuration table corresponding to the export configuration information; instantiating an implementation class of the data page download configuration table to generate a table object of the corresponding data page; generating a table header of the corresponding data page based on the table header configuration table; and writing the target data into the data page based on a data acquisition strategy indicated by the table object.

[0007] According to an embodiment of the present application, the table header of the corresponding data page is generated based on the table header configuration table, including: obtaining a table header indication field in the data export identifier; in the case that the table header indication field indicates that the table header has a matching table header template, obtaining a table header configuration table corresponding to the table header template; and generating a table header of the corresponding data page based on the table header configuration table.

[0008] According to an embodiment of the present application, the data page download configuration table and the table header configuration table include a sorting field indicating a data page number, and the method further includes: sequentially arranging the sorting field, and generating the corresponding data page according to the sorting field.

[0009] According to an embodiment of the present application, the target data is written into the table, and the method further includes: obtaining a data format of the target data; and converting the target data according to a data conversion rule matched with the data format.

[0010] According to an embodiment of the present application, the method further includes: generating an audit fingerprint of the table; and the audit fingerprint includes a user identity, a timestamp, and metadata of the table.

[0011] A second aspect of the present application provides a data processing apparatus, including: an analysis module configured to, in response to receiving a data export request of target data from a user, analyze the data export request to obtain a user identity and a data export identifier; an authority acquisition module configured to match a user authority based on the user identity; a processing module configured to obtain an export weight score of each data in the target data according to a processing rule of the target data indicated by the user authority, the export weight score indicating a resource occupation amount of performing data processing on the target data based on the processing rule; an export module configured to, in the case that a sum of the export weight scores of each data in the target data is greater than a preset score threshold, divide the target data into N target data subsets, and obtain the target data by asynchronously acquiring each target data subset; wherein the sum of the export weight scores of each data in the target data subset is less than the preset score threshold; N is a positive integer greater than 1; and a table generation module configured to match predefined export configuration information based on the data export identifier, and write the target data into a table using a data export strategy indicated by the export configuration information.

[0012] The third aspect of the present application provides an electronic device, comprising: one or more processors; a memory for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors perform the above method.

[0013] The fourth aspect of the present application further provides a computer-readable storage medium having stored executable instructions, which, when executed by a processor, cause the processor to perform the above method.

[0014] The fifth aspect of the present application further provides a computer program product comprising a computer program, which, when executed by a processor, implements the above method.

[0015] According to the data processing method, device, equipment, medium and program product provided by the present application, the user authority is determined according to the user identity, the processing rule of the target output is obtained, and the export weight score is determined according to the resource occupation of data processing, so that the occupation of system processing resources by the two processing steps of data export and desensitization is fully considered. In the case that there may be a performance bottleneck in one-time data export, the data is divided into multiple target data subsets for asynchronous processing. Therefore, at least part of the problems such as too large memory occupation and processing timeout that may exist in one-time data export are solved, and the data export efficiency is improved. BRIEF DESCRIPTION OF DRAWINGS

[0016] The above and other objects, features and advantages of the present application will become more apparent from the following description of the preferred embodiments of the present application taken with reference to the accompanying drawings, in which:

[0017] Figure 1 The application scenario diagram of the data processing method, device, equipment, medium and program product according to the embodiments of the present application is schematically shown;

[0018] Figure 2 The flowchart of the data processing method according to the embodiments of the present application is schematically shown;

[0019] Figure 3 The structural block diagram of the data processing device according to the embodiments of the present application is schematically shown; and

[0020] Figure 4 The block diagram of the electronic device suitable for implementing the data processing method according to the embodiments of the present application is schematically shown. DETAILED DESCRIPTION

[0021] Embodiments of the present application will be described herein below with reference to the drawings. It is to be understood, however, that the description is merely exemplary and is not intended to limit the scope of the present application. In the following detailed description of embodiments of the application, numerous specific details are set forth in order to provide a thorough understanding of the embodiments of the application. However, it will be apparent to one skilled in the art that one or more embodiments of the application can be practiced without these specific details. In other instances, well-known structures and functions have not been described in detail in order to avoid obscuring aspects of the application.

[0022] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. As used herein, the term "includes" and tautological expressions thereof, means the inclusion of but not limited to.

[0023] All terms used herein including technical and scientific terms have the same meanings as commonly understood by one of ordinary skill in the art unless otherwise defined herein. It should be noted that the terms used herein should be interpreted as having a meaning that is consistent with the context of the specification, and not be interpreted in an idealized or overly formal way.

[0024] In the case where expressions such as "at least one of A, B, and C, and the like" are used, in general, it should be interpreted that the meaning is the same as "one or more of A, B, and C" (for example, "a system having at least one of A, B, and C" should include but not be limited to a system having A alone, a system having B alone, a system having C alone, a system having A and B together, a system having A and C together, a system having B and C together, and / or a system having A, B, and C together, etc.).

[0025] In the technical solutions of the present application, the collection, storage, use, processing, transmission, provision, disclosure, and application of user personal information comply with relevant laws and regulations, necessary processing measures are taken, and do not violate public order and good customs.

[0026] In the technical solutions of the present application, the acquisition, collection, storage, use, processing, transmission, provision, disclosure, and application of data comply with relevant laws and regulations, necessary processing measures are taken, and do not violate public order and good customs.

[0027] It should be noted that in the embodiments of the present application, some software, components, models, etc. may be mentioned, which should be considered as exemplary, and the purpose is only to illustrate the feasibility of the implementation of the technical solutions of the present application, but it does not mean that the applicant has or will necessarily use the scheme.

[0028] In a data processing process, it is often necessary to export data stored in a database or various computer systems to a spreadsheet for data analysis and processing. Because different users have different data access permissions, the data needs to be desensitized according to the user's permissions when exporting the data. However, in the case of a large amount of data to be exported, performance bottlenecks may occur, especially in scenarios where desensitization is required. Both data export and desensitization will consume system resources, which may lead to problems such as excessive memory usage and processing timeout. In related technologies, it can be determined whether to divide the export task according to whether the amount of data to be exported exceeds a preset threshold, but this method only considers the amount of data (i.e., the number of data to be exported) and assumes that the computational cost of each data export is the same, without considering scenarios where desensitization is required after data export. In addition, related technologies can filter the data that needs to be desensitized after exporting the required data, but this method requires secondary processing of the data, and the obtained data needs to be cached and written to a spreadsheet after the secondary desensitization is completed. However, this method will long-term occupy a large amount of memory, greatly affecting the performance of the system.

[0029] Figure 1 An application scenario diagram of the data processing method, apparatus, device and storage medium according to the embodiments of the present application is schematically shown.

[0030] As shown in Figure 1 According to the application scenario 100 of the embodiment, the terminal devices 101, 102 and 103, the network 104 and the server 105 can be included. The network 104 is used to provide a communication link medium between the terminal devices 101, 102, 103 and the server 105. The network 104 can include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0031] The user can use the terminal devices 101, 102 and 103 to interact with the server 105 through the network 104 to receive or send messages, etc. Various communication client applications can be installed on the terminal devices 101, 102 and 103, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social platform software, etc. (only as an example).

[0032] The terminal devices 101, 102 and 103 can be various electronic devices with display screens and support for web browsing, including but not limited to smartphones, tablet computers, laptop computers and desktop computers, etc.

[0033] The server 105 can be a server that provides various services, such as a background management server that provides support for a website browsed by a user using the terminal device 101, 102, or 103 (as an example only). The background management server can perform analysis and the like on received user requests and the like, and feed back a processing result (such as a webpage, information, or data obtained or generated according to a user request) to the terminal device.

[0034] A user can use the terminal device 101, 102, or 103 to upload offline form data to the server 105, so that the server 105 generates a corresponding report according to the uploaded data and stores the report in a database. The user can also use the terminal device 101, 102, or 103 to perform data processing on online generated data transmitted via the network 104, and upload the processed data to the server 105, so that the server 105 generates a corresponding report according to the uploaded data and stores the report in a database.

[0035] It should be noted that the data processing method provided by the embodiments of the present application can generally be executed by the server 105. Correspondingly, the data processing apparatus provided by the embodiments of the present application can generally be arranged in the server 105. The data processing method provided by the embodiments of the present application can also be executed by a server or a server cluster different from the server 105 and capable of communicating with the terminal device 101, 102, or 103 and / or the server 105. Correspondingly, the data processing apparatus provided by the embodiments of the present application can also be arranged in a server or a server cluster different from the server 105 and capable of communicating with the terminal device 101, 102, or 103 and / or the server 105.

[0036] It should be understood that Figure 1 the number of terminal devices, networks, and servers inmay be only illustrative. Any number of terminal devices, networks, and servers can be provided according to implementation needs.

[0037] The data processing method according to the embodiments of the present application will be described in detail below based on the scenario described in Figure 1 . Figures 2-4

[0038] Figure 2 A flowchart of the data processing method according to the embodiments of the present application is schematically shown.

[0039] As shown in Figure 2 , the data processing method of this embodiment includes operation S210 to operation S250.

[0040] In operation S210, in response to receiving a user export request for target data, the user identity and the data export identifier are parsed from the export request.

[0041] In operation S220, the user authority is matched based on the user identity.

[0042] In operation S230, the export weight score of each data in the target data is obtained according to the processing rule of the target data indicated by the user authority, and the export weight score indicates the resource occupation amount of performing data processing on the target data based on the processing rule.

[0043] In operation S240, in a case where the sum of the export weight scores of each data in the target data is greater than a preset score threshold, the target data is divided into N target data subsets, and each target data subset is obtained asynchronously to obtain the target data; wherein the sum of the export weight scores of each data in the target data subset is less than the preset score threshold; N is a positive integer greater than 1.

[0044] In operation S250, the target data is written into the table based on the data export identifier matching the predefined export configuration information, and the export strategy indicated by the export configuration information.

[0045] In the embodiments of the present application, the user authority is determined according to the user identity, the processing rule of the target output is obtained, and the export weight score is determined according to the resource occupation amount of data processing, so as to fully consider the occupation of system processing resources by the two processing steps of data export and desensitization. In the case that there may be a performance bottleneck in one-time data export, the data is divided into multiple target data subsets for asynchronous processing, which can solve the problems of too large memory occupation and processing timeout that may exist in one-time data export, and improve the data export efficiency.

[0046] In operation S210, the user identity and the data export identifier are obtained by analyzing the corresponding fields of the export request in the user's export request for the target data.

[0047] In operation S220, different user authorities correspond to different user identities, and different data correspond to different visible states. Correspondingly, different data can have different authority levels, and the data can be visible to the user only when the user authority and the authority level of the data match. For example, the authority level of the data can include public level, internal level and confidential level. For ordinary users without special identity, only the data of the public level can be displayed completely, and the data of the internal level and the confidential level cannot be visible to them. For users with certain authority, such as department managers, they can view the data of the internal level and the public level, but they may not have the permission to view the data of the confidential level or only have the permission to view part of the data of the confidential level, and the data needs to be processed, such as replacing the set bits in the mobile phone data with asterisks, or displaying only the value range identifier of a certain index without displaying the specific value.

[0048] In operation S230, after obtaining the user permission, different processing rules are set for different data in the target data according to matching of the user permission and the permission level of the target data, such as no processing, encryption processing (for example, replacing part of the content with an asterisk), fuzzy processing (for example, only displaying a numerical range identifier), and the like.

[0049] The data processing manners corresponding to different processing rules have different amounts of calculation, for example, the data that does not need to be processed does not need to consume additional calculation amount, the data that needs to be encrypted needs to perform operations such as truncation and character replacement, and the data that needs to be fuzzed needs to perform operations such as querying, mapping, and replacing (for example, finding a preset numerical range, mapping the data to an identifier corresponding to the specific numerical range, and replacing the existing data with the identifier). Therefore, in the embodiments of the present application, each piece of data in the target data is given an export weight score based on the processing rule of the target data indicated by the user permission. The export weight score indicates the resource occupation amount of performing data processing on the target data based on the processing rule. For example, the export weight score of the data that does not need to be processed can be 1, the export weight score of the data that needs to be encrypted can be 1.1, and the export weight score of the data that needs to be fuzzed can be 1.3. The export weight scores corresponding to different data processing manners can be preset or calculated by using a deep learning model.

[0050] In some embodiments, the export weight score can also be calculated according to the data processing amount. For example, when the processing rule indicated by the user permission is to replace a sensitive field, the field length of the field to be replaced is obtained, and the export weight score is determined according to the field length; the export weight score is proportional to the field length. The longer the field length of the field to be replaced, the greater the required calculation amount. The replacement of the sensitive field here corresponds to the encryption processing described above, in which part of the sensitive field is replaced by a character.

[0051] In this embodiment, the export weight score is determined according to the field length. On the one hand, since the calculation of the field length of the field to be replaced is also needed in the normal field replacement process, no additional calculation cost is introduced, the calculation speed is fast, and on the other hand, the different data processing calculation amounts of different data lengths are fully considered, the export weight scores of different data can be accurately determined, the division process of the target data subset is guided, the rationality of the division of the target data subset is ensured, and the efficiency of the data export processing is improved.

[0052] In operation S240, in the conventional method, it is determined whether the data needs to be divided based on the size relationship between the data amount of the target data and the preset data amount threshold, while in the embodiment of the present application, it is determined whether the data needs to be divided based on the size relationship between the sum of the export weight scores of each data in the target data and the preset score threshold. Compared with the conventional method, a weight representing the calculation amount size is added to each data, so that the influence of the data de-identification process on the system resource occupation in the export combined with de-identification scenario is fully considered, the data processing efficiency is improved, and the failure rate is reduced.

[0053] In the embodiment of the present application, if the sum of the export weight scores of each data in the target data is less than or equal to the preset score threshold, it indicates that the calculation amount consumed by exporting the target data once does not exceed the allowed upper limit, and the data can be exported and de-identified once. If the sum of the export weight scores of each data in the target data is greater than the preset score threshold, the target data needs to be divided into multiple target data subsets, and each subset is processed by using the asynchronous processing mode. The data of each target data subset can be directly written into the table after processing, or can be written into the cache first, and then the write operation is performed after all the data of the target data subsets are processed.

[0054] In operation S250, based on the matching between the data export identifier and the predefined export configuration information, the target data is written into the table by using the export strategy indicated by the export configuration information.

[0055] In order to improve the data processing efficiency, in the embodiment of the present application, the method specifically comprises:

[0056] In the case that the export configuration information with the data export identifier exists, the data page download configuration table and the table header configuration table corresponding to the export configuration information are obtained; the implementation class of the data page download configuration table is instantiated to generate the table object corresponding to the data page; the table header corresponding to the data page is generated based on the table header configuration table; and the target data is written into the data page based on the data acquisition strategy indicated by the table object.

[0057] In the embodiment of the present application, different export strategies are preset for different export requirements of data. The table objects of each data page are generated based on the preset implementation class, so that different to-be-imported data can be filled into different table objects through the interface mode, and the efficiency of writing the target data into the data page is improved.

[0058] In the data export, a table of export configuration information corresponding to the export process is first established. The table of export configuration information can include fields such as data export identifier, export name, table header indication field (indicating whether there is a matched table header template), template file path, and file name. The table of export configuration information is matched according to the data export identifier of the current export, and if the matching is implemented, the corresponding data page download configuration table is further matched according to the data export identifier. The data page download configuration table can include fields such as data export identifier, data page export identifier, data page name, implementation class, data start row, and sorting field. The implementation class is a preset data interface template, and the table object of the data page can be generated according to the implementation class. The table object points to each cell to be filled in the data page, so that the cells in the data page can be automatically assigned with the table object and the to-be-imported data and the corresponding cells can be accurately matched through the interface, thereby improving the data processing efficiency. The table of export configuration information and the data page download configuration table can be associated through the data export identifier. The table header configuration table can include fields such as data page export identifier, table header identifier, report name, data mapping identifier, and sorting field. The table header configuration table and the data page download configuration table can be associated through the data page export identifier.

[0059] The implementation class of the instantiated data page download configuration table is applied to the corresponding data page, and the corresponding table object of the cell in the data page is generated, so as to facilitate subsequent data filling.

[0060] The table header of the data page can be automatically generated according to the table header data indicated in the table header configuration table. For a complex table header, the table header template can be directly imported into the data page to implement the generation of the table header, so as to improve the table header generation efficiency. Specifically, in the embodiment of the present application, the table header of the corresponding data page is generated based on the table header configuration table, including: obtaining the table header indication field in the data export identifier; in the case that the table header indication field indicates that the table header has a matched table header template, obtaining the table header configuration table corresponding to the table header template; and generating the table header of the data page corresponding to the table header template based on the table header configuration table.

[0061] In the embodiment of the present application, the data page download configuration table and the table header configuration table include a sorting field indicating the page number of the data page, and the above method further includes: sequentially arranging the sorting field, and generating the corresponding data page according to the sorting field.

[0062] In the embodiment of the present application, in the data page generation process (including the table header generation and the data filling), a unique serial number is assigned to each data page based on the sorting field in the data page download configuration table and the table header configuration table, so as to generate the data page page by page, ensure that no data is missed, and ensure the accuracy of the data.

[0063] In the embodiments of the present application, writing the target data into the table further includes: obtaining a data format of the target data; and converting the target data according to a data conversion rule matched with the data format.

[0064] In the embodiments of the present application, for the target data exported into the table, the data format of the target data can be converted into a data format that is easier to read or process according to actual needs, for example, converting a date type into a YYYY-MM-DD format, converting a number type into a thousandth format, etc., which can improve the readability of the data and the efficiency of the user processing the data.

[0065] In the embodiments of the present application, after the table is generated, the method can further include: generating an audit fingerprint of the table; and the audit fingerprint includes a user identity, a timestamp, and metadata of the table.

[0066] In the embodiments of the present application, the audit fingerprint corresponding to the data export process is further generated after the table is generated, so as to facilitate judging the compliance and safety of the data export, and the compliance of the operation can be judged based on the user identity, the timestamp, and the metadata. The metadata of the table includes a file name, a file size, and a file hash value, and the data whether is tampered can be verified based on comparison of the file hash value, and the safety of the data export can be ensured.

[0067] Specifically, the audit fingerprint can be generated by a composite hash algorithm to generate an irreversible identification, and the user identity information, the export file information, and the user device features are composite encrypted. The user identity information includes a user identification, the export file information includes a timestamp of the export file, a file name of the file, a file size, and a file hash value, and the user device features include a device model of the user device and an address.

[0068] The following describes the process of writing the exported data into the table in operation S250 in the data processing method according to the embodiments of the present application.

[0069] For the table generation process, the table file required to be exported needs to be generated, so as to perform the data import operation subsequently. The following operations can be included.

[0070] (1) Obtain the corresponding export configuration information according to the data export identification and mark it; judge whether the export configuration information exists, if yes, go to (2); otherwise, output configuration error information, and go to (8).

[0071] (2) Loop to read the data page download configuration table, and perform the operation in (3) on each data page download configuration table respectively.

[0072] (3) According to the implementation class in the data page download configuration table, an implementation class object is instantiated and marked; it is judged whether the mark of the implementation class exists, if yes, go to (4); if not, output a configuration error and go to (8).

[0073] (4) The check parameter method of the implementation class object is called to check whether the parameters are valid, if yes, go to (5); otherwise, output a parameter error and go to (8).

[0074] (5) The check parameter method of the implementation class object is called to obtain the total number of data.

[0075] (6) It is detected whether the total number of data exceeds the maximum number limit, if not, go to (7), otherwise, output an "export data over limit error" and go to (8).

[0076] (7) It is judged whether the current data page download configuration table corresponds to the last data page, if yes, execute the "export data" step; if not, loop to read the next data page download configuration table and go to (3).

[0077] (8) End.

[0078] For the specific steps of the above "export data", the following operations can be included.

[0079] (1) It is judged whether the table header indication field in the export configuration information is equal to 1, if yes, it indicates that the table header has a matched table header template, go to (2); otherwise, go to (3).

[0080] (2) A workbook is created through the template file path in the export configuration information, go to (4).

[0081] (3) An empty workbook is created.

[0082] (4) Loop to read the data page download configuration table, and perform the operation in (5) for each data page download configuration table.

[0083] (5) It is judged whether the table header indication field in the export configuration information is equal to 1, if yes, go to (6); if not, go to (7).

[0084] (6) The data table corresponding to the data page name in the current data page download configuration table in the workbook is read, go to (10).

[0085] (7) An empty data page is created, go to (8).

[0086] (8) According to the data page export identifier, the table header configuration table is obtained from the table header configuration table in ascending order according to the sorting field, go to (9).

[0087] (9) write the report name in the current table header configuration table into the data page; go to (11).

[0088] (10) according to the data page export identifier, obtain the table header configuration table from the table header configuration table in ascending order according to the sorting field; go to (11).

[0089] (11) according to the implementation class in the current data page download configuration table, instantiate the implementation class object; go to (12).

[0090] (12) call the data list obtaining method of the implementation class object in the page mode, obtain the data list; go to (13).

[0091] (13) write the data from the data start row in the current data page download configuration table. The data start row of the current written data page is increased by 1; go to (14).

[0092] (14) loop to read each data item in the data list; go to (15).

[0093] (15) loop each item in the table header configuration table, mark the serial number of the list; go to (16).

[0094] (16) judge whether the data identifier mapping of the current item matches the field name of the data item in the data list, if yes, go to (17); if no, go to (18).

[0095] (17) perform data conversion on the field value of the current item to convert it into the required format; go to (18).

[0096] (18) judge whether the current item is the last data in the table header configuration table, if yes, go to (19), otherwise go to (15).

[0097] (19) judge whether the current item is the last data in the data list, if yes, go to (20); if no, go to (14).

[0098] (20) judge whether it is the last data page download configuration table, if yes, go to (21), otherwise, find the next data page download configuration table; go to (4).

[0099] (21) write the workbook into the table file, and return the file path and file name; go to (22).

[0100] (22) end.

[0101] The user can obtain the exported table file based on the returned file path and file name.

[0102] Based on the above data processing method, the application further provides a data processing device. The following will be combined with the data processing method to describe the data processing device. Figure 3The device is described in detail.

[0103] Figure 3 A structural block diagram of a data processing device according to an embodiment of the present application is schematically shown.

[0104] As Figure 3 shown, the data processing device 300 of the embodiment includes a parsing module 310, an authority obtaining module 320, a processing module 330, an exporting module 340, and a table generating module 350.

[0105] The parsing module 310 is configured to parse a user identity and a data export identifier from an export request of the user for target data in response to receiving the export request. In an embodiment, the parsing module 310 can be configured to perform the operation S210 described above, and details are not repeated here.

[0106] The authority obtaining module 320 is configured to match a user authority based on the user identity. In an embodiment, the authority obtaining module 320 can be configured to perform the operation S220 described above, and details are not repeated here.

[0107] The processing module 330 is configured to obtain an export weight score of each data in the target data according to a processing rule of the target data indicated by the user authority, the export weight score indicating a resource occupation amount of performing data processing on the target data based on the processing rule. In an embodiment, the processing module 330 can be configured to perform the operation S230 described above, and details are not repeated here.

[0108] The exporting module 340 is configured to divide the target data into N target data subsets in a case where a sum of the export weight scores of each data in the target data is greater than a preset score threshold, and to obtain the target data by asynchronously obtaining each target data subset; wherein the sum of the export weight scores of each data in the target data subset is less than the preset score threshold; and N is a positive integer greater than 1. In an embodiment, the exporting module 340 can be configured to perform the operation S240 described above, and details are not repeated here.

[0109] The table generating module 350 is configured to match a predefined export configuration information based on the data export identifier, and to write the target data into a table by using an export strategy indicated by the export configuration information. In an embodiment, the table generating module 350 can be configured to perform the operation S250 described above, and details are not repeated here.

[0110] According to an embodiment of the present application, the processing module 330 is further configured to obtain a field length of a field to be replaced in a case where the processing rule indicated by the user authority is to replace a sensitive field, and to determine the export weight score according to the field length; the export weight score is directly proportional to the field length.

[0111] According to an embodiment of the present application, the table includes at least one data page, and the table generation module 350 is further configured to, in a case where the export configuration information includes the data export identifier, acquire a data page download configuration table and a table header configuration table corresponding to the export configuration information; instantiate an implementation class of the data page download configuration table to generate a table object of the corresponding data page; generate a table header of the corresponding data page based on the table header configuration table; and write the target data into the data page based on a data acquisition strategy indicated by the table object.

[0112] According to an embodiment of the present application, the table generation module 350 is further configured to acquire a table header indication field in the data export identifier; in a case where the table header indication field indicates that the table header has a matched table header template, acquire a table header configuration table corresponding to the table header template; and generate a table header of a data page corresponding to the table header template based on the table header configuration table.

[0113] According to an embodiment of the present application, the data page download configuration table and the table header configuration table include a sorting field indicating a data page page number, and the table generation module 350 is further configured to sequentially arrange the sorting field and generate corresponding data pages according to the sorting field.

[0114] According to an embodiment of the present application, the table generation module 350 is further configured to acquire a data format of the target data; and convert the target data according to a data conversion rule matched with the data format.

[0115] According to an embodiment of the present application, the table generation module 350 is further configured to generate an audit fingerprint of the table; and the audit fingerprint includes a user identity, a timestamp, and metadata of the table.

[0116] According to an embodiment of the present application, any of the modules of the parsing module 310, the permission obtaining module 320, the processing module 330, the exporting module 340 and the table generating module 350 can be combined in one module, or any of the modules can be split into multiple modules. Alternatively, at least part of the functions of one or more of the modules can be combined with at least part of the functions of other modules, and implemented in one module. According to an embodiment of the present application, at least one of the parsing module 310, the permission obtaining module 320, the processing module 330, the exporting module 340 and the table generating module 350 can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on chip, a system on board, a system in package, an application specific integrated circuit (ASIC), or any other reasonable manner of integrating or packaging a circuit, etc. or implemented by hardware or firmware, or implemented in any one of software, hardware and firmware or in a proper combination of any of them. Alternatively, at least one of the parsing module 310, the permission obtaining module 320, the processing module 330, the exporting module 340 and the table generating module 350 can be at least partially implemented as a computer program module which, when executed, can perform the corresponding functions.

[0117] Figure 4 A block diagram of an electronic device suitable for implementing the data processing method according to an embodiment of the present application is schematically shown.

[0118] As shown in Figure 4 The electronic device 400 according to an embodiment of the present application includes a processor 401 which can perform various appropriate actions and processes according to programs stored in a read only memory (ROM) 402 or loaded from a storage portion 408 into a random access memory (RAM) 403. The processor 401 can include, for example, a general purpose microprocessor (e.g. a CPU), an instruction set processor and / or a related chipset and / or a special purpose microprocessor (e.g. an application specific integrated circuit (ASIC)), etc. The processor 401 can also include an on-board memory for cache use. The processor 401 can include a single processing unit or multiple processing units for performing different actions of the method processes according to embodiments of the present application.

[0119] In the RAM 403, various programs and data required for the operation of the electronic device 400 are stored. The processor 401, the ROM 402, and the RAM 403 are connected to each other via the bus 404. The processor 401 performs various operations of the method flow according to the embodiments of the present application by executing the programs in the ROM 402 and / or the RAM 403. It should be noted that the programs can also be stored in one or more memories other than the ROM 402 and the RAM 403. The processor 401 can also perform various operations of the method flow according to the embodiments of the present application by executing the programs stored in the one or more memories.

[0120] According to the embodiments of the present application, the electronic device 400 can further include an input / output (I / O) interface 405, which is also connected to the bus 404. The electronic device 400 can further include one or more of the following components connected to the I / O interface 405: an input part 406 including a keyboard, a mouse, etc.; an output part 407 including a display such as a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage part 408 including a hard disk, etc.; and a communication part 409 including a network interface card such as a LAN card, a modem, etc. The communication part 409 performs communication processing via a network such as the Internet. A drive 410 is also connected to the I / O interface 405 as necessary. A removable medium 411 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc. is mounted on the drive 410 as necessary, so that a computer program read out therefrom is installed in the storage part 408 as necessary.

[0121] The present application also provides a computer readable storage medium, which can be included in the device / apparatus / system described in the above embodiments; or can exist separately without being assembled into the device / apparatus / system. The above computer readable storage medium carries one or more programs, when the one or more programs are executed, the method according to the embodiments of the present application is implemented.

[0122] According to an embodiment of the present application, the computer readable storage medium can be a non-transitory computer readable storage medium, for example, can include but not limited to: a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In this application, a computer readable storage medium can be any tangible medium that contains or stores a program for use by or in connection with an instruction execution system, apparatus, or device. For example, according to an embodiment of the present application, the computer readable storage medium can include the ROM 402 and / or the RAM 403 described above and / or one or more memory other than the ROM 402 and the RAM 403.

[0123] Embodiments of the present application also include a computer program product, which includes a computer program containing program codes for executing the methods shown in the flowcharts. When the computer program product is run in a computer system, the program codes are used to make the computer system implement the data processing method provided by the embodiments of the present application.

[0124] The above functions defined in the system / device of the embodiments of the present application are performed when the computer program is executed by the processor 401. According to an embodiment of the present application, the system, device, module, unit, etc. described above can be implemented by computer program modules.

[0125] In one embodiment, the computer program can rely on tangible storage media such as optical storage media, magnetic storage media, etc. In another embodiment, the computer program can also be transmitted, distributed, downloaded and installed in the form of signals on a network medium, and be downloaded and installed through the communication part 409, and / or installed from the detachable medium 411. The program codes contained in the computer program can be transmitted by any appropriate network medium, including but not limited to: wireless, wired, etc., or any suitable combination of the foregoing.

[0126] In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 409, and / or installed from the detachable medium 411. When the computer program is executed by the processor 401, the above functions defined in the system of the embodiments of the present application are performed. According to an embodiment of the present application, the system, device, apparatus, module, unit, etc. described above can be implemented by computer program modules.

[0127] According to embodiments of the present application, program code for implementing the computer programs provided by embodiments of the present application can be written in any combination of one or more programming languages, and can be implemented in a high-level procedural and / or object-oriented programming language, and / or in assembly / machine language. Programming languages include, but are not limited to, Java, C++, python, "C", or the like. Program code can execute entirely on a user's computing device, partly on the user's device, as a stand-alone software package, partly on a remote computing device, or entirely on the remote computing device or server. In the latter scenario, the remote computing device can be connected to the user's computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computing device, such as through the Internet using an Internet Service Provider (ISP).

[0128] The computer program instructions can also be loaded onto a computer or other programmable information processing apparatus to cause a series of operations to be performed on the computer or other programmable information processing apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable information processing apparatus implement the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0129] Those skilled in the art will understand that features of the various embodiments and / or claims of the present application can be combined or / and integrated with one another, even though such a combination or integration is not expressly noted in the present application. In particular, features of the various embodiments and / or claims of the present application can be combined and / or integrated with one another, without departing from the spirit and teachings of the present application. All such combinations and / or integrations are within the scope of the present application.

[0130] The embodiments of the application have been described above. However, these embodiments are merely meant to be illustrative, and not meant to limit the scope of the application. Although each of the embodiments has been described above separately, this does not mean that the measures in the individual embodiments cannot be used advantageously in combination. The scope of the application is defined by the claims appended hereto and their equivalents. Various alternatives and modifications can be made to the embodiments of the application without departing from the scope of the application, and it is intended that all such alternatives and modifications be included within the scope of the application.

Claims

1. A data processing method, characterized in that, include: In response to receiving a user's request to export target data, the user's identity identifier and data export identifier are parsed from the export request. User permissions are obtained based on the user's identity identifier; According to the processing rules for the target data indicated by the user permissions, the derived weight score of each data in the target data is obtained, and the derived weight score indicates the resource consumption of performing data processing on the target data based on the processing rules; If the sum of the derived weight scores of all data in the target data is greater than a preset score threshold, the target data is divided into N target data subsets, and each target data subset is asynchronously acquired to obtain the target data; wherein, the sum of the derived weight scores of all data in the target data subset is less than the preset score threshold; N is a positive integer greater than 1; as well as Based on the data export identifier matching the predefined export configuration information, the target data is written into the table using the export strategy indicated by the export configuration information.

2. The method according to claim 1, characterized in that, The step of obtaining the derived weight score of each data point in the target data according to the processing rules for the target data indicated by the user permissions includes: When the processing rule indicated by the user permission is to replace a sensitive field, the field length of the field to be replaced is obtained, and the exported weight score is determined based on the field length; the exported weight score is proportional to the field length.

3. The method according to claim 1, characterized in that, The table includes at least one data page, and the step of writing the target data into the table using the export strategy indicated by the export configuration information includes: If export configuration information with the data export identifier exists, obtain the data page download configuration table and header configuration table corresponding to the export configuration information; Instantiate the implementation class of the data page download configuration table and generate the table object of the corresponding data page; Generate the header of the corresponding data page based on the header configuration table; Based on the data retrieval strategy indicated by the table object, the target data is written to the data page.

4. The method according to claim 3, characterized in that, The step of generating the header of the corresponding data page based on the header configuration table includes: Retrieve the header indicator field from the data export identifier; If the header indication field indicates that the header has a matching header template, obtain the header configuration table corresponding to the header template; Generate the header of the data page corresponding to the header template based on the header configuration table.

5. The method according to claim 3, characterized in that, The data page download configuration table and the header configuration table include a sorting field indicating the page number of the data page. The method further includes: arranging the sorting field in order and generating the corresponding data pages in sequence according to the sorting field.

6. The method according to claim 1, characterized in that, The step of writing the target data into the table further includes: Obtain the data format of the target data; The target data is converted according to the data conversion rules that match the data format.

7. The method according to claim 1, characterized in that, The method further includes: Generate an audit fingerprint for the table; the audit fingerprint includes the user's identity identifier, timestamp, and the table's metadata.

8. A data processing apparatus, characterized in that, include: The parsing module is used to respond to a user's export request for target data and parse the export request to obtain the user identity identifier and the data export identifier. The permission acquisition module is used to obtain user permissions based on the user identity identifier. The processing module is used to obtain the derived weight score of each data in the target data according to the processing rules for the target data indicated by the user permissions. The derived weight score indicates the resource consumption of performing data processing on the target data based on the processing rules. The export module is used to divide the target data into N target data subsets and asynchronously acquire each of the target data subsets to obtain the target data when the sum of the exported weight scores of each data in the target data subset is greater than a preset score threshold; wherein, the sum of the exported weight scores of each data in the target data subset is less than the preset score threshold; N is a positive integer greater than 1; as well as The table generation module is used to match the predefined export configuration information based on the data export identifier, and write the target data into a table using the export strategy indicated by the export configuration information.

9. An electronic device, comprising: One or more processors; Storage device for storing one or more programs. The characteristic is that, when the one or more programs are executed by the one or more processors, the one or more processors perform the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having executable instructions stored thereon, characterized in that, When executed by the processor, this instruction causes the processor to perform the method according to any one of claims 1 to 7.

11. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the method according to any one of claims 1 to 7.

Images