A general data export method, system, storage medium and electronic device

Abstract

The application provides a general data export method, system, storage medium and electronic equipment, wherein the method comprises: asynchronously executing a plurality of general data export tasks; wherein the execution process of each general data export task comprises the following steps: analyzing a plurality of data sources, data cleaning and conversion rules and data packaging and encryption modes indicated by the general data export task; concurrently extracting target data from the plurality of data sources through a data collector; performing data cleaning and conversion on the target data based on the data conversion rules; performing data packaging and encryption on the target data after data cleaning and conversion based on the data packaging and encryption mode to obtain a packaged and encrypted file; and storing the packaged and encrypted file. The general data export method and system of the application realize higher flexibility, provide better user experience, provide efficient performance, and also have cross-platform compatibility.

Description

Technical Field

[0001] This invention relates to the field of computer data processing technology, and in particular to a general data export method, system, storage medium, and electronic device. Background Technology

[0002] Currently, with the increasing demand for data from enterprises and the advancement of technology, more and more companies are realizing the importance of data for business decisions, which is driving the demand for efficient data export tools and services. Existing data export services are increasingly focusing on automation, reducing manual intervention, and improving efficiency.

[0003] Existing data export tools have the following drawbacks:

[0004] For complex data transformation logic, ETL tools still require writing code; the configuration process can be cumbersome and not user-friendly for non-technical personnel; some advanced features may require additional payment.

[0005] Open source solutions require a certain level of technical expertise to deploy and maintain; their documentation and support may not be as comprehensive as those of commercial products.

[0006] Specialized data export software and services may lack sufficient flexibility to handle complex business logic; they also have limited support for large amounts of data or high-performance requirements.

[0007] Database-provided export tools (such as MySQL Workbench, SQL Server Management Studio, etc.) can usually only handle a single type of data source and are not suitable for cross-platform data export; their functions are relatively limited and they are not suitable for complex data processing scenarios.

[0008] Therefore, a solution is urgently needed to address the aforementioned shortcomings. Summary of the Invention

[0009] One of the objectives of this invention is to provide a general data export method to address the shortcomings of the prior art.

[0010] This invention provides a general data export method, comprising:

[0011] Multiple general data export tasks are executed asynchronously; the execution flow of each general data export task includes the following steps:

[0012] It analyzes multiple data sources, data cleaning and transformation rules, and data packaging and encryption methods in the general data export task instructions;

[0013] The target data is extracted concurrently from multiple data sources using a data collector;

[0014] Based on data transformation rules, the target data is cleaned and transformed.

[0015] Based on the data packaging and encryption method, the target data after data cleaning and transformation is packaged and encrypted to obtain a packaged and encrypted file;

[0016] Store the packaged encrypted files.

[0017] Optionally, the steps for formulating the data cleaning and transformation rules include:

[0018] Users can select data cleaning and transformation rules from the first data cleaning and transformation rule library;

[0019] And / or,

[0020] This allows users to fine-tune the rules in the first data cleaning and transformation rule base to obtain data cleaning and transformation rules.

[0021] And / or,

[0022] Allows users to customize data cleaning and transformation rules;

[0023] And / or,

[0024] It allows users to visualize or encode data cleaning and transformation rules.

[0025] Optionally, the rules in the first data cleaning and transformation rule base include at least: date format conversion functions and string manipulation functions.

[0026] Optionally, the format of the packaged encrypted file includes at least: CSV, JSON, Excel, and ZIP.

[0027] Optionally, the storage destination for storing the packaged encrypted files may include at least: object storage systems, FTP server systems, cloud storage service systems, FSTP systems, and local file systems.

[0028] Optionally, implementation details of different storage destinations can be encapsulated through an abstraction layer; different storage destinations can be supported through plugin or adapter patterns.

[0029] Optionally, users can configure general data export tasks using task configuration templates.

[0030] This invention provides a general data export system, comprising:

[0031] The asynchronous execution module is used to asynchronously execute multiple general data export tasks; the execution flow of each general data export task includes the following steps:

[0032] It analyzes multiple data sources, data cleaning and transformation rules, and data packaging and encryption methods in the general data export task instructions;

[0033] The target data is extracted concurrently from multiple data sources using a data collector;

[0034] Based on data transformation rules, the target data is cleaned and transformed.

[0035] Based on the data packaging and encryption method, the target data after data cleaning and transformation is packaged and encrypted to obtain a packaged and encrypted file;

[0036] Store the packaged encrypted files.

[0037] The present invention provides a computer-readable storage medium storing a computer program, wherein a processor executes the computer program to implement the method described in any of the above embodiments.

[0038] An electronic device provided by an embodiment of the present invention includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the method described in any of the above embodiments.

[0039] This application has achieved the following beneficial effects:

[0040] 1. Greater flexibility: Supports a wider variety of data sources and target storage systems, and makes it easy to add new data sources;

[0041] 2. Improved user experience: The intuitive configuration interface allows even non-technical personnel to easily set up data export templates;

[0042] 3. High performance: capable of processing large amounts of data in a short time;

[0043] 4. Cross-platform compatibility: Not limited to a specific cloud platform or technology stack, it ensures functional integrity by implementing applications that can run independently and do not depend on a single cloud platform. It uses cross-process communication methods to solve the problem of technology stack dependence, making it widely applicable and not limited to a specific cloud platform or technology stack.

[0044] Other features and advantages of the invention will be set forth in the following description, and will be apparent in part from the description, or may be learned by practicing the invention. The objects and other advantages of the invention may be realized and obtained by means of the structures particularly pointed out in the written description and the accompanying drawings.

[0045] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0046] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings:

[0047] Figure 1 This is a schematic diagram of a general data export method according to an embodiment of the present invention;

[0048] Figure 2 This is a schematic diagram illustrating the application process of the general data export method in this embodiment of the invention;

[0049] Figure 3 This is a schematic diagram illustrating another application process of the general data export method in this embodiment of the invention;

[0050] Figure 4 This is a schematic diagram illustrating another application of the general data export method in this embodiment of the invention. Detailed Implementation

[0051] The preferred embodiments of the present invention will be described below with reference to the accompanying drawings. It should be understood that the preferred embodiments described herein are for illustration and explanation only and are not intended to limit the present invention.

[0052] This invention provides a general data export method, such as... Figure 1 As shown, it includes:

[0053] Multiple general data export tasks are executed asynchronously; the execution flow of each general data export task includes the following steps:

[0054] S1. Parse the multiple data sources, data cleaning and transformation rules, and data packaging and encryption methods of the general data export task instructions;

[0055] S2. Extract target data concurrently from multiple data sources using a data collector;

[0056] S3. Based on data transformation rules, perform data cleaning and transformation on the target data;

[0057] S4. Based on the data packaging and encryption method, the target data after data cleaning and transformation is packaged and encrypted to obtain a packaged and encrypted file;

[0058] S5. Store the packaged encrypted file;

[0059] The storage destinations for storing the packaged encrypted files include at least: object storage systems, FTP server systems, cloud storage service systems, FSTP systems, and local file systems;

[0060] Among them, users can configure general data export tasks through task configuration templates.

[0061] like Figure 2-4 As shown, users can create multiple general data export tasks, which can be executed asynchronously. When executing a general data export task, the system will specify the task execution requirements, including multiple data sources, data cleaning and transformation rules, and data packaging and encryption methods. After parsing these requirements, the system uses a data collector to concurrently extract target data from multiple data sources, improving data extraction efficiency. Next, based on the data transformation rules, the target data is cleaned and transformed. Then, based on the data packaging and encryption method, the cleaned and transformed target data is packaged and encrypted to obtain a packaged and encrypted file. Finally, the packaged and encrypted file is stored. When users configure general data export tasks using task configuration templates, they can perform configuration operations through the configuration interface.

[0062] This application has achieved the following beneficial effects:

[0063] 1. Greater flexibility: Supports a wider variety of data sources and target storage systems, and makes it easy to add new data sources;

[0064] 2. Improved user experience: The intuitive configuration interface allows even non-technical personnel to easily set up data export templates;

[0065] 3. High performance: capable of processing large amounts of data in a short time;

[0066] 4. Cross-platform compatibility: It is not limited to a specific cloud platform or technology stack and has broad applicability.

[0067] In one embodiment, the steps for formulating the data cleaning and transformation rules include:

[0068] Users can select data cleaning and transformation rules from the first data cleaning and transformation rule library;

[0069] And / or,

[0070] This allows users to fine-tune the rules in the first data cleaning and transformation rule base to obtain data cleaning and transformation rules.

[0071] And / or,

[0072] Allows users to customize data cleaning and transformation rules;

[0073] And / or,

[0074] Provides users with visual combinations or coding to obtain data cleaning and transformation rules;

[0075] The rules in the first data cleaning and transformation rule base include at least: date format conversion functions and string manipulation functions.

[0076] The steps for formulating data cleaning and transformation rules include four methods. First, the system provides built-in transformation rule functions, such as a specified date format conversion function (converting 2024-01-02 to January 02, 2024), string operation functions (such as removing prefixes). Users can select through the interface to add data processing functions to specified fields, and the execution flow between functions is executed in the order of addition. Second, the system also provides some common data transformation templates, and users can fine-tune the parameters to reuse the template. Third, it also supports users to save custom data transformation functions and data templates. Fourth, it supports users to combine them visually as needed (or can be implemented through coding), enhancing user-friendliness.

[0077] In one embodiment, the formats of the packaged and encrypted files at least include: CSV, JSON, Excel, and Zip.

[0078] In one embodiment, the implementation details of different storage destinations are encapsulated through an abstraction layer; different storage destinations are supported through the plugin or adapter pattern.

[0079] Different storage systems may use different protocols. Design an abstraction layer to encapsulate the specific implementation details of different storage systems, and support different storage systems through the plugin or adapter pattern; different storage systems (such as mysql, mssql, es, clickhouse, redis, plain, etc.) use different data storage protocols. By designing an abstraction layer for data acquisition to encapsulate the implementation details of different storage systems, it is convenient to support different storage systems through the plugin method or adapter pattern during the process.

[0080] In one embodiment, a general data export task can also be obtained through the following steps:

[0081] Based on the user's general data export requirements, determine multiple auxiliary design materials;

[0082] Based on each auxiliary design material, assist the user in designing a general data export task.

[0083] In the above technical solution, the general data export requirement refers to the user's general data export needs, such as: what kind of data to export in batches, what kind of data to export on a scheduled basis, what kind of data needs to be previewed before exporting, and how to record the export history when exporting different types of data; the auxiliary design materials are materials that help users design general data export tasks according to the general data export requirement, such as: the design history of general data export tasks related to the general data export requirement, and a table of recommended export task parameters (such as the table showing suitable encryption and packaging methods for different data types); based on the auxiliary design materials, users are assisted in designing general data export tasks.

[0084] Generally, when faced with large amounts of data requiring batch or scheduled export, users need to design relevant general data export tasks in advance. This design process is labor-intensive, costly, and inefficient. This invention addresses this by identifying multiple auxiliary design materials based on the user's general data export needs. These materials assist the user in designing general data export tasks, reducing labor costs, improving design efficiency, and making the process more user-friendly.

[0085] In one embodiment, the step of assisting users in designing a general data export task based on various auxiliary design materials includes:

[0086] Generate a timeline for displaying various auxiliary design materials;

[0087] Based on the material display timeline, various auxiliary design materials are displayed to the user in sequence;

[0088] Continuously acquire the user's viewing history of displayed auxiliary design materials;

[0089] Based on the viewing history, select the target auxiliary design material from the remaining undisplayed auxiliary design materials;

[0090] When the time interval between the target auxiliary design material's position on the material display timeline and the current time exceeds the interval threshold, the target auxiliary design material is displayed to the user in advance.

[0091] A general data export task for designs that receive user input.

[0092] In the above technical solution, the material display timeline is used to indicate how the system sequentially displays each auxiliary design material to the user. When a user views the displayed auxiliary design materials, a viewing history is generated, which reflects their specific design intent. Therefore, the target auxiliary design material can be selected from the remaining undisplayed auxiliary design materials. The interval threshold can be 5 minutes. When the interval between the current time and the current time exceeds the interval threshold, it means that if the user continues to wait for the target auxiliary design material to be displayed, the waiting time will be too long. To improve the user's design efficiency, the target auxiliary design material is displayed to the user in advance. If the interval does not exceed the interval threshold, it means that the target auxiliary design material will be displayed soon. Since the display order of each auxiliary design material on the material display timeline is reasonably planned, the user can wait. After the user completes the general data export task, they will input the data.

[0093] This invention, based on a material display timeline, controls the sequential display of auxiliary design materials to the user, eliminating the need for the user to manually browse all auxiliary design materials one by one, which is highly user-friendly and improves the user's design efficiency. When displaying auxiliary design materials to the user sequentially, the system selects target auxiliary design materials based on the user's viewing history, and then determines whether to display them to the user in advance based on the relationship between the interval and the interval threshold, further improving the user's design efficiency.

[0094] In one embodiment, generating the material display timeline for each auxiliary design material includes:

[0095] Based on the constraints of material cluster division, each auxiliary design material is divided into multiple material clusters;

[0096] Determine the external sorting value and internal arrangement rules for each material cluster;

[0097] Iterate through each material cluster in descending order of its external sorting value;

[0098] During each iteration, based on the internal arrangement rules of the traversed material clusters, multiple auxiliary design materials in the traversed material clusters are arranged on the timeline;

[0099] After traversing each cluster of materials, the timeline after all auxiliary design materials are arranged is used as the material display timeline;

[0100] The material cluster partitioning constraints include:

[0101] Constraint 1: The total number of auxiliary design materials in the same material cluster exceeds N; where N is a positive integer.

[0102] as well as,

[0103] Constraint 2: Within the same material cluster, only one auxiliary design material of the same category is allowed.

[0104] as well as,

[0105] Constraint 3: The set of material categories for auxiliary design materials in each material cluster must match and conform to the standard set of material categories;

[0106] The determination of the external sorting value and internal arrangement rules for each material cluster includes:

[0107] The external sort value for each material cluster is calculated using the following formula:

[0108]

[0109] Where P is the external sorting value of the material cluster, M is the total number of auxiliary design materials in the material cluster, and H is the total number of all auxiliary design materials. i H is the number of categories of auxiliary design materials that are the same as those in the other i-th material cluster, H-1 is the total number of other material clusters, and Y1 and Y2 are preset weight values;

[0110] When none of the auxiliary design materials in a material cluster have been used by the user, the standard arrangement rule corresponding to the standard material category set that matches the material category set of the material cluster will be used as the internal arrangement rule; otherwise, the internal arrangement rule will be generated as follows:

[0111] When arranging, the auxiliary design materials that the user has used in the material cluster are arranged from highest to lowest usage. Then, the first standard duration corresponding to the usage of the auxiliary design materials that the user has used in the material cluster is used as the first span duration on the time axis, and the time span of the used auxiliary design materials is set according to the first span duration.

[0112] The auxiliary design materials in the material cluster that have not been used by the user are arranged according to their relevance to the auxiliary design materials that have been used, and then arranged from largest to smallest after the last auxiliary design material that the user has used. The relevance of the auxiliary design materials that have not been used by the user in the material cluster and the corresponding second standard duration are used as the second span duration on the time axis. The time span of the used auxiliary design materials is set according to the first span duration.

[0113] In the above technical solution, under constraint three, the standard material category set includes categories representing multiple auxiliary design materials that collectively provide design assistance to the user. Through constraint three, auxiliary value is generated when the user sequentially views the auxiliary design materials in the same material cluster. Secondly, through constraints one and two, the user will see multiple auxiliary design materials in the same material cluster without repetition of material categories, improving the rationality of material cluster division. Under the constraint of material cluster division, each time the user sequentially views the auxiliary design materials in a material cluster, it is a design decision-making stage, and the user will receive sufficient and appropriate assistance.

[0114] In the formula for calculating the external ranking value, M / H is the ratio of the number of auxiliary design materials in the material cluster to the total number of all auxiliary design materials. The larger the ratio, the higher the priority for users to view, and it also avoids the aversion caused by viewing too much content in the later stages of the viewing process. It is the average number of categories of the same auxiliary design materials in the material cluster and other material clusters. The larger the average value, the higher the priority for users to view and understand in advance. When they view auxiliary design materials of the same category later, they can make comparisons and receive design assistance from a global perspective. Therefore, these two values ​​are assigned preset weight values ​​to calculate the external ranking value.

[0115] The standard material category set corresponds to a pre-set standard arrangement rule. This rule sorts the auxiliary design materials within the material cluster according to their usage frequency by different users, from highest to lowest. Frequency can be the total number of times used by different users throughout history. When no auxiliary design materials in a material cluster have been used by any user, the standard arrangement rule corresponding to the standard material category set that matches the material cluster's category set is used as the internal arrangement rule; otherwise, an internal arrangement rule is generated. Usage refers to the total number of times a user has used auxiliary design materials throughout history. Usage corresponds to a preset first standard duration. The higher the usage, the more familiar the user is with the auxiliary design materials, and the shorter the required viewing time, resulting in a shorter first standard duration. The first span duration refers to the length of the time interval in which the auxiliary design material is set on the timeline. Time span setting involves setting the time interval of the first span duration on the timeline and then placing the corresponding auxiliary design materials within that interval. First, the used auxiliary design materials are sorted according to… The auxiliary design materials are arranged from highest to lowest usage, and then the time span of used materials is set according to the first span duration. This allows users to prioritize viewing their used auxiliary design materials for different reasonable time intervals, improving their design decision-making efficiency. The relevance represents the degree of association between auxiliary design materials. The higher the relevance, the higher the priority of displaying the corresponding auxiliary design material. Therefore, the materials are arranged after the last user-used auxiliary design material in the relevance ranking from highest to lowest. The higher the relevance, the longer the user needs to spend viewing, and the longer the corresponding second standard duration. Similarly, the relevance and corresponding second standard duration of the auxiliary design materials in the material cluster are used as the second span duration on the timeline, and the time span of used auxiliary design materials is set according to the first span duration. Then, the user-unused auxiliary design materials are arranged after the user-used auxiliary design materials in the ranking.

[0116] The timeline for displaying various auxiliary design materials generated in this embodiment of the invention can help users view these materials in the best order and at the best pace, greatly improving the efficiency of users' general data export task design.

[0117] In one embodiment, selecting target auxiliary design materials from the remaining undisplayed auxiliary design materials based on viewing history includes:

[0118] Based on viewing history, determine the user's task design intent;

[0119] Determine the minimum intent difference between the task design intent and multiple standard task design intents;

[0120] Based on the minimum intent difference, generate the first material selection rule;

[0121] Based on the first material selection rule, select target auxiliary design materials from the remaining undisplayed auxiliary design materials;

[0122] And / or,

[0123] The viewing history is preprocessed to obtain the target history sequence; wherein, the content in the odd-numbered order of the target history sequence is the first target content in the auxiliary design materials that the user is interested in, and the content in the even-numbered order is the second target content of the viewing operation during the attention interval when the user looks at the first target content of two adjacent odd-numbered orders.

[0124] When the average of the first similarity between any two pairs of second target content exceeds the first similarity threshold, and the second similarity between the first even-numbered second target content in the target history sequence and the trigger content exceeds the second similarity threshold, a second material selection rule is generated based on each first target content.

[0125] Based on the second material selection rule, select the target auxiliary design material from the remaining undisplayed auxiliary design materials.

[0126] In the above technical solution, there are two ways to select target auxiliary design materials: First, viewing the history will reflect the user's task design intent. The standard task design intent is the complete design intent that the user will have when designing a general data export task. Based on the minimum intent difference between the task design intent and multiple standard task design intents, the remaining design intent that the user is most likely to reflect in the future can be further determined. Based on the minimum intent difference, a first material selection rule is generated, and the target auxiliary design materials are selected based on this rule. The minimum intent difference refers to the intent difference with the smallest degree of difference. The first material selection rule is to select auxiliary design materials that meet the minimum intent difference.

[0127] The second approach involves considering that viewing history doesn't necessarily reflect the user's task design intent. Instead, selection is based on the user's focus on displayed auxiliary design materials. Odd-numbered sequences refer to items 1, 3, 5, etc., while even-numbered sequences refer to items 2, 4, 6, etc. A target history sequence is generated, with odd-numbered sequences setting the first target content and even-numbered sequences setting the second target content. The attention gap refers to the time interval between periods when the user doesn't pay attention to any content between two adjacent odd-numbered first target contents. During this interval, the user might perform a viewing action, and the content viewed during this time becomes the second target content. The order of the first target content in each odd-numbered sequence also reflects the user's... The order in which the first target content is considered is as follows: the trigger content represents content that the user may want to use as auxiliary design materials, such as retrieving data to be exported and comparing it with the content in the auxiliary design materials; when the average of the first similarity between any two pairs of second target content exceeds the first similarity threshold and the second similarity between the first even-numbered second target content in the target history sequence and the trigger content exceeds the second similarity threshold, it is indicated that all first target content can be used as the basis for selecting target auxiliary design materials. Based on each first target content, a second material selection rule is generated, which selects auxiliary design materials related to the first target content.

[0128] By using the two methods described above for selecting target auxiliary design materials, the most suitable target auxiliary design materials can be selected in different situations, improving the system's adaptability and making it more intelligent.

[0129] This invention provides a general data export system, comprising:

[0130] The asynchronous execution module is used to asynchronously execute multiple general data export tasks; the execution flow of each general data export task includes the following steps:

[0131] It analyzes multiple data sources, data cleaning and transformation rules, and data packaging and encryption methods in the general data export task instructions;

[0132] The target data is extracted concurrently from multiple data sources using a data collector;

[0133] Based on data transformation rules, the target data is cleaned and transformed.

[0134] Based on the data packaging and encryption method, the target data after data cleaning and transformation is packaged and encrypted to obtain a packaged and encrypted file;

[0135] Store the packaged encrypted files.

[0136] This invention provides a computer-readable storage medium storing a computer program, which is executed by a processor to implement the method described in any of the above embodiments.

[0137] This invention provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the method described in any of the above embodiments.

[0138] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

Claims

1. A generic data export method, characterized by, The method comprises the following steps: Asynchronous execution of a plurality of general data export tasks; wherein the execution process of each general data export task comprises the following steps: Parsing a plurality of data sources, data cleaning and conversion rules, and data packaging and encryption methods indicated by the general data export task; Concurrently extracting target data from a plurality of data sources through a data collector; Based on the data conversion rules, the target data is cleaned and converted; Based on the data packaging and encryption method, the target data after data cleaning and conversion is packaged and encrypted to obtain a packaged and encrypted file; The packaged and encrypted file is stored; The general data export task is obtained by the following steps: Based on the user's general data export requirements, a plurality of auxiliary design materials are determined; Based on each auxiliary design material, the user is assisted in designing a general data export task; The method comprises the following steps: Generating a material display timeline of each auxiliary design material; Based on the material display timeline, each auxiliary design material is displayed to the user in sequence; Continuously obtaining the viewing history of the user viewing the displayed auxiliary design material; Based on the viewing history, a target auxiliary design material is selected from the remaining auxiliary design materials that have not been displayed; When the interval between the time position of the target auxiliary design material on the material display timeline and the current time exceeds the interval threshold, the target auxiliary design material is displayed to the user in advance; Receiving the user's input of the designed general data export task; The method comprises the following steps: Based on the material cluster division constraint, each auxiliary design material is divided into a plurality of material clusters; Determine the external sorting value and internal arrangement rule of each material cluster; Traverse each material cluster in descending order of external sorting value; Each time the material cluster is traversed, the internal arrangement rule of the traversed material cluster is used to arrange the plurality of auxiliary design materials in the traversed material cluster on the timeline; After traversing each material cluster, the timeline after arranging all auxiliary design materials is taken as the material display timeline; The material cluster division constraint comprises: Constraint one, the total number of auxiliary design materials in the same material cluster exceeds N; wherein N is a positive integer; Constraint two, the auxiliary design materials of the same material category in the same material cluster are unique; Constraint three, the material category set of the auxiliary design materials in each material cluster matches the standard material category set; The method comprises the following steps: Calculate the external sorting value of each material cluster, and the calculation formula is as follows: When the auxiliary design materials in the material cluster have not been used by the user, the standard arrangement rule corresponding to the standard material category set that matches the material category set of the material cluster is taken as the internal arrangement rule; otherwise, the internal arrangement rule is generated as follows: When arranging, the auxiliary design materials that have been used by the user in the material cluster are arranged in descending order of usage degree, and the first standard duration corresponding to the usage degree of the auxiliary design materials that have been used by the user in the material cluster is taken as the first span duration on the timeline, and the auxiliary design materials that have been used by the user are arranged in time span. wherein P is an external ordering value of the material cluster, M is the total number of the auxiliary design materials in the material cluster, H is the total number of all auxiliary design materials, P i is the number of the same auxiliary design material categories in the material cluster and the other ith material cluster, H-1 is the total number of the other material clusters, and Y1 and Y2 are preset weight values; ​ ​ The unused auxiliary design materials in the material cluster are arranged according to the association degrees between the unused auxiliary design materials and the last used auxiliary design material in the material cluster and the last used auxiliary design material arranged from large to small, and the association degrees of the unused auxiliary design materials in the material cluster and the corresponding second standard time length are taken as the second span time length on the time axis, and the used auxiliary design materials are set to span in time according to the first span time length.

2. The universal data export method of claim 1, wherein, The data cleaning and conversion rule making step comprises: The user selects a data cleaning and conversion rule from a first data cleaning and conversion rule library. And / or, The user fine-tunes the rules in the first data cleaning and conversion rule library to obtain a data cleaning and conversion rule. And / or, The user customizes a data cleaning and conversion rule. And / or, The user obtains a data cleaning and conversion rule through visual combination or coding.

3. The universal data export method of claim 2, wherein, The rules in the first data cleaning and conversion rule library at least include date format conversion functions and string operation functions.

4. The universal data export method of claim 1, wherein, The format of the packaged encrypted file at least includes CSV, JSON, Excel, and Zip.

5. The universal data export method of claim 1, wherein, The storage destination for storing the packaged encrypted file at least includes an object storage system, an FTP server system, a cloud storage service system, an FSTP system, and a local file system.

6. The universal data export method of claim 5, wherein, The implementation details of different storage destinations are encapsulated through an abstraction layer; different storage destinations are supported through a plug-in or adapter mode.

7. The universal data export method of claim 1, wherein, The user configures a general data export task through a task configuration template.

8. A general data export system, characterized by, It comprises: An asynchronous execution module for asynchronously executing a plurality of general data export tasks; wherein the execution process of each general data export task comprises the following steps: Parsing a plurality of data sources, data cleaning and conversion rules, and data packaging and encryption methods indicated by the general data export task; Concurrently extracting target data from a plurality of data sources through a data collector; Based on the data conversion rule, the target data is cleaned and converted; Based on the data packaging and encryption method, the target data after data cleaning and conversion is packaged and encrypted to obtain a packaged encrypted file; The packaged encrypted file is stored; The general data export task is obtained through the following steps: Based on the user's general data export requirements, a plurality of auxiliary design materials are determined; Based on each auxiliary design material, the user is assisted in designing a general data export task; The method comprises: Generating a material display time axis of each auxiliary design material; Based on the material display time axis, each auxiliary design material is displayed to the user in sequence; Continuously obtaining the viewing history of the user viewing the displayed auxiliary design materials; Based on the viewing history, a target auxiliary design material is selected from the remaining undisplayed auxiliary design materials; When the interval between the time position of the target auxiliary design material on the material display time axis and the current time exceeds the interval threshold, the target auxiliary design material is displayed to the user in advance; Receiving the user's input of the designed general data export task; The method comprises: Based on material cluster division constraints, each auxiliary design material is divided into a plurality of material clusters; determining an external ordering value and an internal arrangement rule for each of the material clusters; traversing the material clusters in descending order of the external ordering value; arranging the plurality of auxiliary design materials in the traversed material cluster on the time axis based on the internal arrangement rule of the traversed material cluster each time the traversal is performed; after the traversal of the material clusters is completed, taking the time axis after the arrangement of all the auxiliary design materials is completed as the material display time axis; wherein the material cluster division constraint comprises: constraint one, the total number of auxiliary design materials in the same material cluster exceeds N; wherein N is a positive integer; constraint two, the auxiliary design materials of the same material category in the same material cluster are unique; and constraint three, the material category set of the auxiliary design materials in each material cluster matches the standard material category set; wherein the determination of the external ordering value and the internal arrangement rule for each of the material clusters comprises: calculating the external ordering value of each material cluster, and the calculation formula is as follows: when the auxiliary design materials in the material cluster have not been used by the user, taking the standard arrangement rule corresponding to the standard material category set that matches the material category set of the material cluster as the internal arrangement rule; otherwise, generating the internal arrangement rule as follows: when arranging, arranging the auxiliary design materials that have been used by the user in the material cluster in descending order of the usage degree, taking the first standard time length corresponding to the usage degree of the auxiliary design materials that have been used by the user in the material cluster as the first span time length on the time axis, and performing time span setting on the auxiliary design materials that have been used by the user according to the first span time length; P = M / H * (H - 1) * (Y1 - Y2) / (H - 1) * (Y1 - Y2) + 1 i P = M / H * (H - 1) * (Y1 - Y2) / (H - 1) * (Y1 - Y2) + 1 arranging the auxiliary design materials that have not been used by the user in the material cluster in descending order of the association degree with the last auxiliary design material that has been used by the user and starting from the last auxiliary design material that has been used by the user, taking the second standard time length corresponding to the association degree of the auxiliary design materials that have not been used by the user in the material cluster as the second span time length on the time axis, and performing time span setting on the auxiliary design materials that have not been used by the user according to the second span time length. The computer readable storage medium stores a computer program, and the processor executes the computer program to implement the method of any one of claims 1-7. The electronic device comprises a memory and a processor, the memory stores a computer program, and the processor executes the computer program to implement the method of any one of claims 1-7.

9. A computer-readable storage medium, characterized in that, ​ 10. An electronic device, comprising: ​

Images