Data Export Method, System and Storage Medium Based on Big Data Platform

Abstract

This application is applicable to the field of artificial intelligence technology, and provides a data export method, system and storage medium based on a big data platform. The above method creates table configuration information through a configuration generation server. The table configuration information is used to determine the table structure, and the configuration distribution server automatically obtains and distributes the table configuration information, so that the data processing server can receive and load the table configuration information according to the load situation, generate the corresponding table configuration interface, so that the data to be exported can be exported according to the table configuration interface and its corresponding table structure, realizing the modularization and process of table export, thereby improving the response speed to table export requirements and greatly reducing the development time cost of table export scripts.

Description

Technical Field

[0001] This application belongs to the technical field of data processing, and particularly relates to a data export method, system, and storage medium based on a big data platform. Background Art

[0002] Currently, enterprises and institutions with data processing services or generating a large amount of data during the processing service usually choose to import the data into a database for unified storage and management. However, operating a database requires certain professional knowledge. For users lacking database professional knowledge, they are unable to export specified data in the database according to actual needs. Therefore, according to the actual needs of users, developers can design corresponding table export scripts to achieve the automatic export of specified data.

[0003] When the actual needs of users continue to increase and the content types of specified data also continue to increase, developers need to design corresponding table export scripts according to the content types of each specified data, and there is a certain amount of identical code among multiple table export scripts, resulting in excessive time consumption for developing table export scripts and repeatedly writing the same code. Therefore, how to solve the poor development efficiency of table export scripts has become an urgent problem to be solved currently. Summary of the Invention

[0004] In view of this, the embodiments of this application provide a data export method, system, and storage medium based on a big data platform to solve the problems of excessive time consumption for developing table export scripts, repeatedly writing the same code, and poor development efficiency of table export scripts.

[0005] The first aspect of the embodiments of this application provides a data export method based on a big data platform, which is applied to a data processing server built based on a big data platform. The method includes:

[0006] Receiving at least one piece of table configuration information sent by the configuration distribution server, where the table configuration information is used to determine the table structure;

[0007] Encapsulating each piece of the table configuration information to generate a corresponding table configuration interface;

[0008] Receiving the data to be exported and an export instruction sent by the requirement unit;

[0009] Obtaining the content type of the data to be exported;

[0010] Invoking a corresponding table configuration interface according to the content type of the data to be exported and the export instruction, where the content type of the data to be exported matches the table structure of the corresponding table configuration interface;

[0011] Determine the table structure through the corresponding table configuration interface, so that the data to be exported is exported according to the table structure.

[0012] In one embodiment, encapsulating each piece of the table configuration information to generate a corresponding table configuration interface includes:

[0013] Encapsulate each piece of the table configuration information to generate a corresponding annotation class;

[0014] Encapsulate each of the annotation classes to generate a corresponding annotation method;

[0015] Encapsulate each of the annotation methods to generate a corresponding table configuration interface.

[0016] In one embodiment, the method further includes:

[0017] Record the table configuration interface corresponding to the content type of each type of data to be exported, and establish a correspondence table between the content type and the interface.

[0018] In one embodiment, before calling the corresponding table configuration interface according to the content type of the data to be exported and the export instruction, it further includes:

[0019] Obtain the permission level of the export instruction;

[0020] Determine a corresponding list of table configuration interfaces according to the permission level of the export instruction;

[0021] Judge whether the table configuration interface corresponding to the content type of the data to be exported and the export instruction is in the list of table configuration interfaces;

[0022] When the table configuration interface corresponding to the content type of the data to be exported and the export instruction is not in the list of table configuration interfaces, stop exporting the data to be exported;

[0023] When the table configuration interface corresponding to the content type of the data to be exported and the export instruction is in the list of table configuration interfaces, execute calling the corresponding table configuration interface according to the content type of the data to be exported and the export instruction.

[0024] The data export method applied to a data processing server provided in the first aspect of the embodiments of the present application. The data processing server receives table configuration information sent by a configuration distribution server, generates a corresponding table configuration interface, and when receiving data to be exported and an export instruction, calls the corresponding table configuration interface in response to the export instruction, so that the data to be exported can be exported according to the table structure, realizing the modularization and process of table export, improving the response speed of table export requirements, and greatly reducing the time cost of developing table export scripts.

[0025] The second aspect of the embodiments of the present application provides a data export method based on a big data platform, which is applied to a configuration distribution server. The method includes:

[0026] When new table configuration information is added to the configuration generation server, read the table configuration information;

[0027] According to the load condition of the data processing server, send the table configuration information to the data processing server.

[0028] In one embodiment, the step of reading and, according to the load condition of the data processing server, sending the table configuration information to the data processing server includes:

[0029] Read and, according to the load condition of each data processing unit, send the table configuration information to the unloaded data processing unit.

[0030] The data export method provided by the second aspect of the embodiments of the present application and applied to a configuration distribution server realizes the automatic acquisition and distribution of table configuration information by the configuration distribution server automatically reading the newly added table configuration information when new table configuration information is added to the configuration generation server and distributing the table configuration information according to the load condition of the data processing server, and gives full play to the performance of the data processing server, improving the table export efficiency.

[0031] The third aspect of the embodiments of the present application provides a data export method based on a big data platform, which is applied to a configuration generation server. The method includes:

[0032] Add new table configuration information according to the input instruction of the user, where the table configuration information is used to determine the table structure.

[0033] The data export method provided by the third aspect of the embodiments of the present application and applied to a configuration generation server enables the configuration generation server to add new table configuration information by receiving the input instruction of the user, supports the user to set the table configuration information according to actual needs, thereby determining different table structures and improving the flexibility of configuring the table structure.

[0034] The fourth aspect of the embodiments of the present application provides a data export system based on a big data platform, including a configuration generation server, a configuration distribution server and a data processing server. The configuration distribution server is respectively connected to the configuration generation server and the data processing server;

[0035] The configuration generation server is used to execute the steps of the data export method provided by the third aspect of the embodiments of the present application;

[0036] The configuration distribution server is used to execute the steps of the data export method provided in the second aspect of the embodiments of the present application;

[0037] The data processing server is used to execute the steps of the data export method provided in the first aspect of the embodiments of the present application.

[0038] In one embodiment, the data processing server includes at least one data processing unit, and each data processing unit is connected to a corresponding requirement unit.

[0039] The data export system provided in the fourth aspect of the embodiments of the present application receives the input instruction of the user sent by the requirement unit through the configuration generation server to add table configuration information, so as to determine different table structures; it also automatically reads the newly added table configuration information through the configuration distribution server, and distributes the table configuration information according to the load condition of the data processing server, realizing the automatic acquisition and distribution of the table configuration information; and through the data processing server receiving the table configuration information sent by the configuration distribution server, generating the corresponding table configuration interface, when receiving the data to be exported and the export instruction, calling the corresponding table configuration interface in response to the export instruction, so that the data to be exported can be exported according to the table structure, realizing the modularization and process of the table export, improving the automation degree of the table export, also improving the response speed of the table export requirement, and greatly reducing the time cost of developing the table export script.

[0040] The fifth aspect of the embodiments of the present application provides a computer-readable storage medium, and the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, it realizes the steps of the data export method provided in the first aspect to the third aspect of the embodiments of the present application.

[0041] It can be understood that the beneficial effects of the above fifth aspect can be referred to the relevant descriptions in the above first aspect to the third aspect, and will not be elaborated here. Description of the Drawings

[0042] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following will briefly introduce the drawings required for use in the embodiments or the description of the prior art. Obviously, the following drawings are only some embodiments of the present application, and for those of ordinary skill in the art, other drawings can be obtained according to these drawings without creative efforts.

[0043] Figure 1 It is a schematic diagram of the scenario of the data export system provided in the embodiments of the present application;

[0044] Figure 2 It is a schematic diagram of the structure of the data processing server provided in the embodiments of the present application;

[0045] Figure 3 It is a schematic flowchart of a data export method applied to a data processing server provided by an embodiment of the present application;

[0046] Figure 4 It is a schematic flowchart of a data export method applied to a configuration distribution server provided by an embodiment of the present application;

[0047] Figure 5 It is a schematic flowchart of a data export method applied to a configuration generation server provided by an embodiment of the present application. Detailed implementation manners

[0048] In the following description, for the purpose of illustration rather than limitation, specific details such as specific system architectures, technologies, etc. are presented to thoroughly understand the embodiments of the present application. However, those skilled in the art should clearly understand that the present application can also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to avoid unnecessary details from interfering with the description of the present application.

[0049] It should be understood that when used in the specification and the appended claims of the present application, the term "comprising" indicates the presence of the described features, wholes, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components, and / or their combinations.

[0050] It should also be understood that the term "and / or" as used in the specification and the appended claims of the present application refers to any combination and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0051] As used in the specification and the appended claims of the present application, the term "if" can be interpreted as "when", "once", "in response to determining", or "in response to detecting" depending on the context. Similarly, the phrase "if determined" or "if detecting [the described condition or event]" can be interpreted as meaning "once determined", "in response to determining", "once detecting [the described condition or event]", or "in response to detecting [the described condition or event]" depending on the context.

[0052] In addition, in the description of the specification and the appended claims of the present application, the terms "first", "second", "third", etc. are only used for distinguishing descriptions and cannot be understood as indicating or implying relative importance.

[0053] References to "one embodiment" or "some embodiments" etc. described in the specification of this application mean that specific features, structures or characteristics described in connection with that embodiment are included in one or more embodiments of this application. Thus, statements such as "in one embodiment", "in some embodiments", "in other some embodiments", "in still other embodiments" etc. that appear in different places in this specification do not necessarily all refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized. The terms "comprising", "including", "having" and their variants all mean "including but not limited to", unless otherwise specifically emphasized.

[0054] In applications, when the data to be exported has multiple content types, it is necessary to develop corresponding table export scripts according to the multiple content types, and there is a certain amount of identical code in the development process of multiple table export scripts, resulting in excessive time consumption for developing table export scripts and repeatedly writing the same code, and the development efficiency of table export scripts is poor.

[0055] In view of the above technical problems, the embodiments of this application provide a data export method based on a big data platform. By configuring a generation server to create table configuration information, the table configuration information is used to determine the table structure, and the configuration distribution server automatically obtains and distributes the table configuration information, so that the data processing server can receive and load the table configuration information according to the load situation, generate the corresponding table configuration interface, so that the data to be exported can be exported according to the table configuration interface and its corresponding table structure, realizing the modularization and process of table export, thereby improving the response speed to table export requirements and greatly reducing the development time cost of table export scripts.

[0056] The data export method based on a big data platform provided by the embodiments of this application can be applied to any terminal device capable of recording or processing data. The terminal device can be a mobile phone, a tablet computer, a wearable device, a vehicle-mounted device, an augmented reality (AR) / virtual reality (VR) device, a notebook computer, an ultra-mobile personal computer (UMPC), a netbook, a personal digital assistant (PDA), etc. The embodiments of this application do not impose any restrictions on the specific type of the electronic device.

[0057] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Among them, artificial intelligence (AI) is a theory, method, technology, and application system that uses digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use knowledge to obtain the best results.

[0058] The server in the embodiments of the present application can be an independent server or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery network (CDN), and artificial intelligence platforms. Specifically, it can be a server built based on a big data platform.

[0059] As Figure 1 shown, the data export system based on a big data platform provided by the embodiments of the present application includes a configuration generation server 100, a configuration distribution server 200, and a data processing server 300. The configuration distribution server 200 is respectively connected to the configuration generation server 100 and the data processing server 300;

[0060] The data processing server 300 is used to execute Figure 3 the corresponding data export method applied to the data processing server;

[0061] The configuration distribution server 200 is used to execute Figure 4 the corresponding data export method applied to the configuration distribution server;

[0062] The configuration generation server 100 is used to execute Figure 5 the corresponding data export method applied to the configuration generation server.

[0063] In one embodiment, the data processing server 300 includes at least one data processing unit 301, and each data processing unit 301 is connected to a corresponding requirement unit 401.

[0064] Figure 2 An exemplary structural diagram of the data processing server 300 is shown. Specifically, the data processing server 300 can include at least one processor 302 and at least one memory 303. The memory 303 stores a computer program 304, and when at least one processor 302 executes the computer program 304, it can implement Figure 2 the corresponding data export method applied to the data processing server.

[0065] In an application, a data processing unit may include a processor and a memory, or multiple data processing units may include a processor and a memory.

[0066] In an application, the processor may be a Central Processing Unit (CPU), and the processor may also be other general-purpose processors, Digital Signal Processors (DSPs), Application Specific Integrated Circuits (ASICs), Field-Programmable Gate Arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0067] In an application, the memory may be an internal storage unit of a terminal device in some embodiments, such as a hard disk or memory of the terminal device. The memory may also be an external storage device of the terminal device in other embodiments, such as a plug-in hard disk, Smart Media Card (SMC), Secure Digital (SD) card, Flash Card, etc., equipped on the terminal device. Further, the memory may also include both the internal storage unit and the external storage device of the terminal device. The memory is used to store an operating system, application programs, a BootLoader, data, and other programs, such as program codes of computer programs. The memory may also be used to temporarily store data that has been output or will be output.

[0068] In an application, the structures of the configuration generation server 100 and the configuration distribution server 200 are the same as that of the above data processing server 300, and will not be described in detail here.

[0069] It can be understood that the structure illustrated in the embodiments of the present application does not constitute a specific limitation on the data export system. In other embodiments of the present application, the data export system may include more or fewer components than those illustrated, or combine certain components, or different components. For example, it may also include input / output devices, network access devices, etc. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0070] As Figure 3 shown, the data export method based on a big data platform provided by the embodiments of the present application is applied to a data processing server, and includes the following steps S301 to step S306:

[0071] Step S301: Receive at least one piece of form configuration information sent by a configuration distribution server, where the form configuration information is used to determine the form structure.

[0072] In an application, a data processing server is used to receive at least one piece of form configuration information sent by a configuration distribution server. The form configuration information can be written in a data interchange format. The data interchange format can record the form structure based on text and has the advantage of being applicable to any operating system (such as Windows, Linux, or macOS, etc.). Therefore, the form configuration information can be used to determine the form structure and can be read by servers of multiple different operating systems. The specific data interchange format of the form configuration information can be JSON (JavaScript Object Notation) language or XML (Extensible Markup Language) language. The embodiments of the present application do not make any limitations on the specific language type of the data interchange format.

[0073] In an application, each piece of form configuration information can be stored in a URL (Uniform Resource Locator). Using the URL as the carrier for storing the form configuration information can enable the form configuration information to be transmitted between different servers and has the advantages of fast transmission speed, high transmission stability, and difficult information leakage.

[0074] In an application, when the configuration distribution server and the data processing server are in one network environment, the communication range of the URL can be set within the above one network environment; when the configuration distribution server and the data processing server are in two different network environments, the communication range of the URL can be unrestricted. Among them, the network environment can specifically be a local area network or a wide area network, etc. Adjusting the communication range of the URL according to the network environment where the configuration distribution server and the data processing server are located can improve the flexibility of obtaining the form configuration information.

[0075] In one embodiment, step S301 includes:

[0076] The form configuration information includes at least one of the following:

[0077] Form name, title information, title attributes, document name, and document extension name.

[0078] In an application, the table configuration information may include multiple configuration contents such as table name, title information, title attributes, document name, and document extension name. Among them, the title information and title attributes can be used to determine the title of the table; the document extension name can be used to determine the table type. The table type can specifically be an EXCEL (workbook) document in formats such as.xls and.xlsx, or an XML document in formats such as.xml. Thus, the table structure can be determined according to the table configuration information. The embodiments of the present application do not impose any restrictions on the table type. In addition, the table name and document name can be set according to actual needs.

[0079] Step S302: Package each piece of table configuration information to generate a corresponding table configuration interface.

[0080] In an application, the data processing server can be used to parse the table configuration information, implement packaging each piece of table configuration information, and generate one or more corresponding table configuration interfaces. When the data processing server receives a data export requirement, it can call the table configuration interface adapted to the data to be exported according to the requirement of the requirement unit, improving the speed of data export. Among them, the number of table structures included in each piece of table configuration information determines the number of corresponding table configuration interfaces generated.

[0081] In one embodiment, step S302 includes:

[0082] Package each piece of table configuration information to generate a corresponding annotation class;

[0083] Package each annotation class to generate a corresponding annotation method;

[0084] Package each annotation method to generate a corresponding table configuration interface.

[0085] In an application, to package each piece of table configuration information to generate one or more corresponding table configuration interfaces, it needs to go through three steps: packaging the table configuration information into an annotation class, packaging the annotation class into an annotation method, and packaging the annotation method into a table configuration interface. Taking the generation of a corresponding table configuration interface from one piece of table configuration information as an example, the above three steps are described as follows:

[0086] The data processing server can perform parallel processing on multiple pieces of table configuration information according to its actual processing capacity. Among them, when processing a piece of table configuration information, it can first read each piece of content in the table configuration information and generate a corresponding annotation for each piece of configuration content. For example, if the configuration content "table name" in the table configuration information includes "order information", the configuration content "title information" includes "order ID" and "reservation date", and the configuration content "title attribute" includes "orderID" and "bookDate", then the data processing server can add the annotation "order information" to the configuration content "table name", add the annotations "order ID" and "reservation date" to the configuration content "title information", and add the annotations "orderID" and "bookDate" to the configuration content "title attribute", so as to generate three annotations corresponding to the configuration content of the table configuration information, and then form an annotation class corresponding to the table configuration information;

[0087] The data processing server can encapsulate the annotation class to generate a corresponding annotation method to endow the above annotation class with the processing logic for extracting data to be exported. The processing logic can be specifically implemented based on the reflection mechanism of JAVA. Specifically, through the reflection mechanism of JAVA, the corresponding data to be exported can be extracted according to the annotation class. For example, if the configuration content "table name" of the table configuration information of the data to be exported includes "order information", the configuration content "title information" includes "order ID" and "reservation date", and the configuration content "title attribute" includes "orderID" and "bookDate", it is easy to obtain that the configuration content included in the data to be exported is the same as the configuration content of the above three annotation classes, so that the data included in the three identical configuration contents can be extracted, that is, the two columns of data included in the table named "order information" with the title information of "order ID" and "reservation date" and the title attribute of "orderID" and "bookDate". It should be noted that the extraction of the data to be exported based on the three configuration contents of the table name, title information, and title attribute is only exemplary, and the data to be exported can be extracted according to the actual configuration content of the table configuration information set;

[0088] The data processing server can encapsulate the annotation method to generate a corresponding table configuration interface, so that when the data processing server calls any table configuration interface, it can quickly use the encapsulated annotation method to extract the data to be exported, and the table configuration interface can be set with a preset table type for determining the table type of the exported data. Specifically, it can be an EXCEL document with a file extension of.xls,.xlsx, etc. Therefore, through the table configuration interface, after inputting the data to be exported, the data can be automatically extracted and the exported data of the preset table type can be output, greatly improving the data export efficiency.

[0089] In an application, the table configuration interface can also be set with preset format settings, which can be used to determine font settings, cell settings, etc. Specifically, through the font settings, the font, font size, font color, underline, and bold font effects of different cells can be adjusted. Through the cell settings, the cell color, cell length and width, data type of the cell, cell border, etc. of different cells can be adjusted. The embodiments of the present application do not impose any restrictions on the specific content of the preset format settings.

[0090] Step S303: Receive the data to be exported and the export instruction sent by the requirement unit.

[0091] In an application, the data processing server can receive the data to be exported and the export instruction of the requirement unit in real time. Each piece of data to be exported can have a corresponding export instruction, and the data processing server can respond to the export instruction to export the data to be exported.

[0092] In an application, the requirement unit can send the data to be exported stored in any kind of database, and the data processing server can support data sent from different types of databases such as Oracle, MySQL, MongoDB, SQL Server, IBM Db2, and DM database.

[0093] Step S304: Obtain the content type of the data to be exported;

[0094] Step S305: Call the corresponding table configuration interface according to the content type of the data to be exported and the export instruction. The content type of the data to be exported matches the table structure of the corresponding table configuration interface.

[0095] In an application, the content type of the data to be exported can be obtained from the export instruction; or the corresponding content type can be obtained by analyzing the data to be exported and matching the corresponding table configuration interface. Specifically, the data processing server can read the first preset number of configuration contents of the data to be exported, such as table name, title information, title attributes, or document name, etc. The first preset number can be one or more. According to the read first preset number of configuration contents, a matching search is performed on multiple types of table configuration interfaces. If there is any kind of table configuration interface whose first preset number of configuration contents is the same as the first preset number of configuration contents of the above data to be exported, it means that the content type of the data to be exported matches the table structure of this kind of table configuration interface, and this kind of table configuration interface can be used as the corresponding table configuration interface.

[0096] In an application, the data processing server can also determine the corresponding table configuration interface to be called according to the export instruction or / and the content type of the data to be exported. Specifically, the export instruction can be used to determine the table configuration interface to be called; or, the export instruction can be used to determine multiple callable table configuration interfaces. According to the content type of the data to be exported, a matching search can be performed based on the above multiple callable table configuration interfaces to determine a corresponding table configuration interface. The data processing server can preset a content type and interface correspondence table, which includes the correspondence between the content type of the data to be exported and the table configuration interface, so that the table configuration interface can be determined by looking up the table.

[0097] In one embodiment, step S305 further includes:

[0098] Record the table configuration interface corresponding to the content type of each type of data to be exported, and establish a content type and interface correspondence table.

[0099] In an application, after obtaining the table configuration interface corresponding to the content type of any type of data to be exported, the content type of the data to be exported and the corresponding table configuration interface can be recorded, so as to generate a content type and interface correspondence table, which can be used to support the data processing server to determine the corresponding table configuration interface by looking up the table when obtaining the content type of the data to be exported.

[0100] In one embodiment, step S305 further includes:

[0101] Obtain the permission level of the export instruction;

[0102] Determine the corresponding table configuration interface list according to the permission level of the export instruction;

[0103] Judge whether the content type of the data to be exported and the table configuration interface corresponding to the export instruction are within the table configuration interface list;

[0104] When the content type of the data to be exported and the table configuration interface corresponding to the export instruction are not within the table configuration interface list, stop exporting the data to be exported;

[0105] When the content type of the data to be exported and the table configuration interface corresponding to the export instruction are within the table configuration interface list, execute calling the corresponding table configuration interface according to the content type of the data to be exported and the export instruction.

[0106] In an application, an export instruction may have permission levels, which may include at least two levels. Each permission level has a corresponding list of table configuration interfaces, and each list of table configuration interfaces includes at least one table configuration interface. For example, assume that the permission levels of the export instruction include three levels: the first permission level, the second permission level, and the third permission level. The third permission level corresponds to the third list of table configuration interfaces, the second permission level corresponds to the second list of table configuration interfaces, and the first permission level corresponds to the first list of table configuration interfaces. Among them, the number and type of table configuration interfaces included in each list of table configuration interfaces can be set according to actual needs.

[0107] In an application, the permission level of an export instruction can be determined by obtaining identity characteristics such as the department of the demand unit that sends the export instruction, the employee ID (Identity document), or the device MAC (Media Access Control Address). Among them, the corresponding relationship between the identity characteristics and the permission levels can be set according to actual needs.

[0108] In an application, obtain the corresponding list of table configuration interfaces according to the permission level of the export instruction, and determine whether the table configuration interface corresponding to the export instruction is within the list of table configuration interfaces according to the table configuration interface corresponding to the export instruction; if so, proceed to step S305 to export the data to be exported through the corresponding table configuration interface, and if not, stop exporting the data to be exported.

[0109] In an application, by obtaining the permission level of the export instruction, the corresponding list of table configuration interfaces can be determined, and it can be judged whether the table configuration interface corresponding to the export instruction belongs to the list of table configuration interfaces corresponding to the export instruction, so as to monitor whether the demand unit that sends the export instruction has the permission to call the corresponding table configuration interface, avoid the demand unit without permission from exporting the data to be exported, and improve the security of data export.

[0110] Step S306: Determine the table structure through the corresponding table configuration interface, so that the data to be exported is exported according to the table structure.

[0111] In an application, after obtaining the table configuration interface corresponding to the content type of the data to be exported, the data to be exported can be exported according to the table structure of the table configuration interface through the reflection mechanism of JAVA, so as to achieve the rapid export of the data to be exported.

[0112] In an application, a data processing server generates a corresponding table configuration interface by receiving the table configuration information sent by a configuration distribution server. When receiving the data to be exported and an export instruction, it calls the corresponding table configuration interface in response to the export instruction, enabling the data to be exported according to the table structure, realizing the modularization and process of table export, improving the response speed of table export requirements, and significantly reducing the time cost of developing table export scripts.

[0113] As Figure 4 shown, the data export method based on a big data platform provided by an embodiment of the present application is applied to a configuration distribution server and includes the following steps S401 and S402:

[0114] Step S401: When new table configuration information is added to a configuration generation server, read the table configuration information.

[0115] In an application, the configuration distribution server can monitor in real time whether new table configuration information is added to the configuration generation server, and when it detects that new table configuration information is added to the configuration generation server, read and save the table configuration information from the configuration generation server.

[0116] Step S402: Send the table configuration information to the data processing server according to the load condition of the data processing server.

[0117] In an application, the configuration distribution server can also read the load condition of the data processing server, and the load level is positively correlated with the number of table configuration information received by the data processing server. When the load condition of the data processing server is less than a preset threshold, the configuration distribution server can send a second preset number of table configuration information to the data processing server to update the table structure supported by the data processing server. Among them, the preset threshold can specifically be 60%, 70%, or 80%, etc., and the second preset number can be 3, 4, or 5, etc. The specific values of the preset threshold and the second preset number are not limited in the embodiments of the present application.

[0118] In one embodiment, step S402 includes:

[0119] Read and send the table configuration information to the idle data processing units according to the load condition of each data processing unit.

[0120] In an application, the data processing server includes at least one data processing unit. The configuration distribution server is respectively connected to each data processing unit. Each data processing unit is used to receive a piece of table configuration information and generate a corresponding table configuration interface. When the data processing server includes multiple data processing units, each data processing unit can receive a piece of table configuration information and generate a corresponding table configuration interface to support the corresponding table structure. The data distribution server can read whether each data processing unit receives the table configuration information. If so, it determines that the corresponding data processing unit is full. If not, it determines that the corresponding data processing unit is empty. The configuration distribution server can distribute the table configuration information to the empty data processing units.

[0121] In an application, when the configuration distribution server automatically reads the newly added table configuration information when adding table configuration information in the configuration generation server and distributes the table configuration information according to the load condition of the data processing server, it realizes the automatic acquisition and distribution of the table configuration information, and gives full play to the performance of the data processing server to improve the table export efficiency.

[0122] As Figure 5 shown, the data export method based on the big data platform provided by the embodiment of the present application is applied to the configuration generation server and includes the following step S501:

[0123] Step S501: Add table configuration information according to the input instruction of the user. The table configuration information is used to determine the table structure.

[0124] In an application, the configuration generation server adds table configuration information by receiving the input instruction of the user sent by the demand unit, which can support the user to set the table configuration information according to actual needs, so as to determine different table structures and improve the flexibility and convenience of configuring the table structure. The specific configuration content of the table configuration information is the same as that introduced in the above step S201 and will not be elaborated here.

[0125] In one embodiment, step S501 further includes:

[0126] Read the data to be exported of the demand unit, and add table configuration information according to the table structure of the data to be exported.

[0127] In an application, the configuration generation server can be connected to the demand unit and read the data to be exported of the demand unit, and generate configuration content according to the table structure of the data to be exported, and encapsulate the generated configuration content into table configuration information to realize the automatic generation of the table configuration information.

[0128] It should be understood that the sequence numbers of the steps in the above embodiments do not indicate the order of execution, and the execution order of each process should be determined by its function and internal logic, and should not constitute any limitation to the implementation process of the embodiments of the present application.

[0129] In the above embodiments, the descriptions of the various embodiments have their own emphases. For the parts not detailed or recorded in a certain embodiment, reference may be made to the relevant descriptions of other embodiments.

[0130] Those of ordinary skill in the art can realize that the modules and algorithm steps of the examples described in combination with the embodiments disclosed herein can be implemented by electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are executed in a hardware or software manner depends on the specific application and design constraints of the technical solution. Professional technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the present application.

[0131] In the embodiments provided in the present application, it should be understood that the disclosed terminal devices and methods can be implemented in other ways. For example, the terminal device embodiments described above are merely illustrative. For example, the division of the modules is only a logical function division, and there may be other division methods in actual implementation. For example, multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed couplings or direct couplings or communication connections to each other can be indirect couplings or communication connections through some interfaces, devices or modules, and can be in electrical, mechanical or other forms.

[0132] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or perform equivalent replacements for some of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

Claims

1. A data export method based on a big data platform, characterized in that, Applied to a data processing server built based on a big data platform, the data processing server is respectively connected to multiple demand units, and the data processing server is also connected to a configuration distribution server. The method includes: Receiving at least one piece of table configuration information sent by the configuration distribution server, where the table configuration information is used to determine the table structure; Encapsulating each piece of the table configuration information to generate a corresponding table configuration interface; Receiving the data to be exported and an export instruction sent by the demand unit; Obtaining the content type of the data to be exported; Invoking a corresponding table configuration interface according to the content type of the data to be exported and the export instruction, where the content type of the data to be exported matches the table structure of the corresponding table configuration interface; Determining the table structure through the corresponding table configuration interface, so that the data to be exported is exported according to the table structure; The encapsulating each piece of the table configuration information to generate a corresponding table configuration interface includes: Encapsulating each piece of the table configuration information to generate a corresponding annotation class; Encapsulating each of the annotation classes to generate a corresponding annotation method; Encapsulating each of the annotation methods to generate a corresponding table configuration interface.

2. The data export method according to claim 1, wherein The method further includes: Recording the table configuration interface corresponding to the content type of each piece of the data to be exported, and establishing a correspondence table between the content type and the interface.

3. The data export method according to claim 1, wherein Before invoking the corresponding table configuration interface according to the content type of the data to be exported and the export instruction, it further includes: Obtaining the permission level of the export instruction; Determining a corresponding list of table configuration interfaces according to the permission level of the export instruction; Judging whether the table configuration interface corresponding to the content type of the data to be exported and the export instruction is in the list of table configuration interfaces; When the table configuration interface corresponding to the content type of the data to be exported and the export instruction is not in the list of table configuration interfaces, stopping the export of the data to be exported; When the table configuration interface corresponding to the content type of the data to be exported and the export instruction is in the list of table configuration interfaces, executing the step of invoking the corresponding table configuration interface according to the content type of the data to be exported and the export instruction.

4. A data export method based on a big data platform, characterized in that, Applied to a configuration distribution server, the configuration distribution server is respectively connected to a data processing server and a configuration generation server. The method includes: When new table configuration information is added to the configuration generation server, reading the table configuration information; Sending the table configuration information to the data processing server according to the load condition of the data processing server; The data processing server includes at least one data processing unit, and the configuration distribution server is respectively connected to each data processing unit. Each data processing unit is used to receive a piece of table configuration information and generate a corresponding table configuration interface, specifically: encapsulating each piece of the table configuration information to generate a corresponding annotation class; encapsulating each of the annotation classes to generate a corresponding annotation method; encapsulating each of the annotation methods to generate a corresponding table configuration interface; Reading and sending the table configuration information to the data processing server according to the load condition of the data processing server, including: Reading and sending the table configuration information to the unloaded data processing unit according to the load condition of each data processing unit.

5. A data export method based on a big data platform, characterized in that, Applied to a configuration generation server, the configuration generation server is connected to a configuration distribution server, and the configuration distribution server is further connected to a data processing server. The method includes: Adding table configuration information according to a user input instruction, where the table configuration information is used to determine a table structure; The data processing server is used to encapsulate each piece of the table configuration information to generate a corresponding table configuration interface, including: Encapsulating each piece of the table configuration information to generate a corresponding annotation class; Encapsulating each of the annotation classes to generate a corresponding annotation method; Encapsulating each of the annotation methods to generate a corresponding table configuration interface.

6. A data export system based on a big data platform, characterized in that, Including a configuration generation server, a configuration distribution server, and a data processing server. The configuration distribution server is respectively connected to the configuration generation server and the data processing server; The configuration generation server is used to execute the steps of the method according to claim 5; The configuration distribution server is used to execute the steps of the method according to claim 4; The data processing server is used to execute the steps of the method according to any one of claims 1 to 3.

7. The system according to claim 6, wherein The data processing server includes at least one data processing unit, and each data processing unit is connected to a corresponding requirement unit.

8. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the data export method according to any one of claims 1 to 5.

Images