Data export methods and related products
By splitting and broadcasting the data export request to the second device cluster for asynchronous export, the problems of long export time of big data and the risk of server downtime are solved, and the export efficiency and user experience are improved.
Patent Information
- Application Number
- CN202010131856.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2020-02-29
- Publication Date
- 2024-11-19
- Estimated Expiration
- 2040-02-29
AI Technical Summary
The existing big data export technology has problems such as the export time is too long and complex, which may lead to the risk of server memory overflow and downtime, affecting the user experience.
By splitting the data export request into multiple sub-requests and broadcasting them to the second device cluster, using multiple second devices to export data asynchronously, avoiding the risk of downtime brought by exporting data by a single server and improving the export speed.
It improves data export efficiency, reduces user waiting time, reduces the risk of downtime of the second device, and improves user experience.
Smart Images

Figure CN111444011B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the technical field of data transmission, and in particular to a data export method and related products. Background Art
[0002] With the maturity of informatization, data collection work is becoming more and more perfect, and the types of data are increasing. Therefore, a large amount of data corresponding to a user can be stored in the database. Merchants can obtain the user's data from the database and make product recommendations for users through big data technology. Due to the large amount of data for each user, the export process is extremely difficult. There are currently two export methods: 1. Split the big data and export it in batches. This export method will consume too much business time and require users to repeatedly perform the same export operation, which reduces the user experience and affects the development of subsequent business applications. 2. "Export once, wait continuously", that is, initiate a data export request to the database server. The requesting user exports from the database to the specified storage path. After receiving the request, the database server exports the data to the specified storage path according to the data transmission rate it can bear. Due to the large amount of data, it will occupy too much memory resources, the waiting time is too long, and the performance of distributed services is not fully utilized.
[0003] Therefore, the current big data export technology is single, the export time is too long, and the export process is complicated. When exporting large amounts of data on a single server, it may cause server memory overflow, the server is at risk of crashing, and the user experience is low. Summary of the invention
[0004] An embodiment of the present application provides a data export method, which exports data asynchronously through multiple second devices, thereby improving data export efficiency and reducing the risk of downtime of the second devices.
[0005] In a first aspect, an embodiment of the present application provides a data export method, including:
[0006] The first device receives a data export request sent by a user terminal, where the data export request includes a data identifier of the data to be exported and a memory of the data to be exported;
[0007] If the memory of the data to be exported is greater than the first threshold, the first device splits the data export request into N sub-data export requests, and adds a device identifier of the second device in the second device cluster to each sub-data export request, and the device identifiers of any two sub-data export requests are different;
[0008] The first device broadcasts the N sub-data export requests to the second device cluster, so that the target second device exports the sub-data to be exported corresponding to the target sub-data export request to the user terminal, the target sub-data export request is any one of the N sub-data export requests, and the target second device is the second device in the second device cluster corresponding to the device identifier of the target sub-data request; and so that the user terminal synthesizes the data to be exported based on the received sub-data to be exported.
[0009] In a second aspect, an embodiment of the present application provides a data export device, including:
[0010] A receiving unit, configured to receive a data export request sent by a user terminal, wherein the data export request includes a data identifier of the data to be exported and a memory of the data to be exported;
[0011] a splitting unit, configured to split the data export request into N sub-data export requests if the memory of the data to be exported is greater than a first threshold, and add a device identifier of a second device in a second device cluster to each sub-data export request, and the device identifiers of any two sub-data export requests are different;
[0012] A broadcast unit is used to broadcast the N sub-data export requests to the second device cluster so that the target second device exports the sub-data to be exported corresponding to the target sub-data export request to the user terminal, the target sub-data export request is any one of the N sub-data export requests, and the target second device is the second device in the second device cluster corresponding to the device identifier of the target sub-data request; and so that the user terminal synthesizes the data to be exported based on the received sub-data to be exported.
[0013] In a third aspect, an embodiment of the present application provides an electronic device, comprising a processor, a memory, a communication interface, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the processor, and the program includes instructions for executing the steps in the method described in the first aspect.
[0014] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and the computer program enables a computer to execute the method described in the first aspect.
[0015] In a fifth aspect, an embodiment of the present application provides a computer program product, which includes a non-transitory computer-readable storage medium storing a computer program, and the computer is operable to cause the computer to execute the method described in the first aspect.
[0016] Implementing the embodiments of the present application has the following beneficial effects:
[0017] It can be seen that in the embodiment of the present application, when exporting data, the data export request is split through the first device, and the multiple sub-data requests after the split are broadcast to the second device cluster, so that the sub-data to be exported corresponding to the sub-data request is asynchronously exported to the user terminal by multiple second devices, avoiding the downtime risk caused by exporting data through a single server; moreover, by asynchronously exporting data through multiple second devices, the data export speed is improved, the user's waiting time is reduced, and the user experience is improved. BRIEF DESCRIPTION OF THE DRAWINGS
[0018] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.
[0019] Figure 1A A system architecture diagram for data export provided in an embodiment of the present application;
[0020] Figure 1B A flowchart of a data export method provided in an embodiment of the present application;
[0021] Figure 1C A schematic diagram of storing CSV data provided in an embodiment of the present application;
[0022] Figure 2 A flowchart of another data export method provided in an embodiment of the present application;
[0023] Figure 3 A flowchart of another data export method provided in an embodiment of the present application;
[0024] Figure 4 A schematic diagram of the structure of a data export device provided in an embodiment of the present application;
[0025] Figure 5 A functional unit composition frame of a data export device provided in an embodiment of the present application;
[0026] Figure 6 A block diagram of the functional units of another data export device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0027] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.
[0028] The terms "first", "second", "third" and "fourth" etc. in the specification and claims of the present application and the drawings are used to distinguish different objects, rather than to describe a specific order. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but optionally includes steps or units that are not listed, or optionally includes other steps or units inherent to these processes, methods, products or devices.
[0029] Reference to "embodiments" herein means that a particular feature, result, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.
[0030] In order to understand the embodiments of the present application, the technical knowledge involved in the present application is first introduced.
[0031] CSV, Comma-Separated Values (CSV), also known as character-separated values, is a file saving format. Specifically, files in CSV format store table data in plain text. Since data saving in CSV format is relatively convenient and fast, when using CSV format to store data, it is mainly used to save big data. For example, medical big data.
[0032] First of all, it should be noted that the data export involved in this application is illustrated by exporting big data in CSV format. The data export method in other formats is similar to that in CSV format and will not be described again.
[0033] See also Figure 1A , Figure 1A A system architecture diagram for data export provided in an embodiment of the present application includes: a user terminal 10, a first device 20, and a second device cluster (including multiple second devices 30), wherein any second device in the second device cluster stores data that the user terminal 10 needs to export:
[0034] The user terminal 10 submits 20 a data export request to the first device, the data export request including a data identifier of the data to be exported and a memory of the data to be exported;
[0035] When the memory of the data to be exported is greater than the first threshold, the first device 20 splits the data export request into N sub-data export requests, each sub-data request is used to export the sub-data to be exported, and each sub-data request includes a device identifier, which is the identifier of the second device (such as a device ID);
[0036] The first device 20 broadcasts the N sub-data export requests to the second device cluster, and each second device 30 can receive the N sub-data export requests, parse each sub-data export request, obtain the device identification of each sub-data export request, and when the device identification is its own device identification, use the sub-data export request as the target sub-data export request, search for the sub-data to be exported corresponding to the sub-data export request in the database, and export the sub-data to be exported to the user terminal 10.
[0037] The user terminal 10 receives N sub-data to be exported exported by N second devices 30, and uses the N sub-data to be exported to synthesize the data to be exported.
[0038] Optionally, the above-mentioned user terminals may include smart phones (such as Android phones, iOS phones, Windows Phone phones, etc.), tablet computers, PDAs, laptops, mobile Internet devices MID (Mobile Internet Devices, referred to as: MID) or wearable devices, etc. The above-mentioned electronic devices are only examples, not exhaustive, and include but are not limited to the above-mentioned electronic devices. In practical applications, the above-mentioned electronic devices may also include: smart vehicle terminals, computer equipment, etc. The first device may be a controller, a server, a cloud platform or a virtual machine, which is not limited in this application. The second device is a database server.
[0039] It can be seen that in the embodiment of the present application, when exporting data, the data export request is split through the first device, and the multiple sub-data requests after the split are broadcast to the second device cluster, so that the sub-data to be exported corresponding to the sub-data request is asynchronously exported to the user terminal by multiple second devices, avoiding the downtime risk caused by exporting data through a single server; moreover, by asynchronously exporting data through multiple second devices, the data export speed is improved, the user's waiting time is reduced, and the user experience is improved.
[0040] See also Figure 1B , Figure 1B A flow chart of a data export method provided in an embodiment of the present application. The method includes but is not limited to the following steps:
[0041] 101: The user terminal sends a data export request to the first device.
[0042] The data export request includes a data identifier of the data to be exported and a memory of the data to be exported. The identifier may be a name, a linked list, an index or other identifiers of the data to be exported.
[0043] In addition, the data export request also includes a storage path and a communication path (such as an IP address) of the user terminal, that is, the storage location of the data to be exported. The storage path can be specified by the user or automatically obtained by the user terminal, that is, when sending the data export request, the user terminal automatically detects the memory space of the user terminal, obtains the free memory space, and uses the free memory space corresponding to the size of the data to be exported as the storage location.
[0044] 102: If the memory of the data to be exported is greater than the first threshold, the first device splits the data export request into N sub-data export requests, and adds a device identifier of a second device in the second device cluster to each sub-data export request.
[0045] The first threshold may be 1G, 2G, 5G or other values. N is an integer greater than 1.
[0046] Optionally, the first device obtains the type of data to be exported, obtains multiple data types, obtains keywords corresponding to the multiple data types, classifies the keywords corresponding to the multiple data types, and constructs a sub-data export request for the data type corresponding to the keywords belonging to one category, obtains N sub-data export requests, and adds the device identifier of the second device in the second device cluster to each sub-data export request, and the device identifiers between any two sub-data export requests are different.
[0047] Specifically, the data types of the exported data are divided according to the attributes of the data. For example, the data export request is used to export the user's behavior records in the historical time period, and the user's video browsing records, news browsing records, shopping records, etc. are selected for export; because the video browsing records and news browsing records belong to the same online behavior, a sub-data export request can be constructed for the video browsing records and news browsing records, and the sub-data export request is used to export the user's online behavior.
[0048] Optionally, the first device obtains multiple data types corresponding to the data to be exported; and determines the data format corresponding to each data type, constructs a sub-data export request for the data types with the same data format, and obtains N sub-data export requests.
[0049] It should be noted that, in some cases, the same device identifier of the second device may be added to two sub-data export requests. For example, when the number of sub-data export requests is greater than the number of second devices, some second devices may be reused, that is, some device identifiers may be reused, and one second device may export sub-data to be exported corresponding to multiple sub-data export requests.
[0050] The data format includes one or a combination of documents, tables, videos, and pictures.
[0051] For example, if the data export request is used to export historical browsing videos, historical chat videos, shopping records, etc., a sub-data export request can be constructed for the historical browsing videos and historical chat videos, so as to export the sub-data to be exported in the data format through a second device. Since the data formats are the same, a unified export rule can be used for data export to improve export efficiency.
[0052] Furthermore, before the first device splits the data export request, the first device pre-stores the device identification of each second device. After the data export request is split into N sub-data export requests, N device identifications are randomly selected and a device identification of a second device is added to each sub-data export request, so as to export a sub-data to be exported corresponding to a sub-data export request through each second device.
[0053] It should be noted that each sub-data export request also includes the above-mentioned communication method and storage path.
[0054] In a possible implementation, before adding the device identifier, the first device first requests each second device to report the current network quality (i.e., report the uplink speed and / or downlink speed), and adds the device identifiers of the N second devices with the highest network quality to the N sub-data export request, and the N device identifiers are different.
[0055] In a possible implementation, before adding the device identifier, the first device first requests each second device to report the current load (such as CPU occupancy), and adds the device identifiers of N second devices with lower loads to the N sub-data export requests, and the N device identifiers are different.
[0056] 103: The first device broadcasts the N sub-data export requests to the second device cluster.
[0057] Optionally, the first device broadcasts the N sub-data export requests to the second device cluster through a message queue. Specifically, the first device packages the N sub-data requests, publishes the packaged data to a pre-established message queue, and notifies each second device in the second device cluster to consume the N sub-data requests from the message queue.
[0058] It can be understood that before broadcasting the N sub-data export requests to the second device cluster through the message queue, the first device creates a username, password, access path and port number for the message queue, and sends the username, password, access path and port number to each second device in the second device cluster so as to create a message queue using the username, password, access path and port number.
[0059] 104: The second device exports the sub-data to be exported corresponding to the target sub-data export request to the user terminal.
[0060] Among them, the second device corresponds to the target sub-data request, the target sub-data export request is any one of the N sub-data export requests, and the second device is the second device in the second device cluster corresponding to the device identifier of the target sub-data request.
[0061] Each second device in the second device cluster consumes the N sub-data export requests from the message queue, and then parses the device identification of each sub-data export request. When the device identification of the sub-data export request is parsed as its own device identification, the sub-data export request is used as the target sub-data export request, and the sub-data to be exported corresponding to the target sub-data export request is searched from the storage space. The sub-data to be exported is sent to the user terminal according to the communication mode of the user terminal.
[0062] 105: The user terminal synthesizes the data to be exported according to the N sub-data to be exported corresponding to the received N sub-data export requests.
[0063] After receiving the sub-data to be exported sent by N target second devices, the user terminal synthesizes the N sub-data to be exported to obtain the data to be exported.
[0064] It can be seen that in the embodiment of the present application, when exporting data, the data export request is split by the first device, and the multiple sub-data requests after the split are broadcast to the second device cluster, so that the sub-data to be exported corresponding to the sub-data request is asynchronously exported to the user terminal by multiple second devices, avoiding the downtime risk caused by exporting data through a single server; moreover, by asynchronously exporting data through multiple second devices, the data export speed is improved, the user's waiting time is reduced, and the user experience is improved.
[0065] In a possible implementation, before the target second device exports the sub-data to be exported corresponding to the target sub-data export request to the user terminal, the target second device parses the target sub-data export request to obtain the memory of the sub-data to be exported corresponding to the target sub-data export request. When the memory is greater than a second threshold, the target second device creates m threads corresponding to the target sub-data export request, and exports the sub-data to be exported corresponding to the target sub-data export request to the user terminal through the m threads.
[0066] Specifically, each second device's configuration file is configured with a preset maximum number of threads Max1. Therefore, the target second device obtains the preset maximum number of threads and the maximum amount of data that each thread can process Max2 by reading the configuration file; then obtains the amount of data K required to be exported by the target sub-data export request; creates the m threads according to the maximum amount of data that each thread can process and the amount of data required to be exported by the target sub-data export request, where m can be expressed by the following formula:
[0067]
[0068] Among them, since the implementation scheme of this application is mainly used to export data in CSV format, the data to be exported are all encapsulated in the database table. The amount of data to be exported can be reflected by the number of rows or columns in the database table. The maximum amount of data that each thread can process is also characterized by the number of rows or columns that each thread can export at one time. For example, if the data to be exported needs to export eight rows of data from the table, then the amount of data to be exported is determined to be eight; if each thread can export up to 10 rows of data at a time, then the maximum amount of data for the thread is determined to be 10.
[0069] In this example, the sub-data export request is split into one step, so that each second target device can asynchronously export the sub-data to be exported, thereby improving the data export speed.
[0070] In a possible implementation, before the target second device exports the sub-data to be exported corresponding to the target sub-data export request to the user terminal, the target second device stores the user's CSV data in a database according to a preset storage rule.
[0071] Specifically, the second device performs keyword recognition on the user's CSV data (including multiple sub-CSV data) to determine a keyword set corresponding to each sub-CSV data; based on the keyword set corresponding to each sub-CSV data, obtain multiple data dimensions corresponding to each sub-CSV data; obtain the importance level of each data dimension in the multiple data dimensions relative to the user; take the data dimension with the highest importance level as the root node, and use the other data dimensions in the multiple data dimensions except the data dimension with the highest importance level as child nodes to create a tree model corresponding to each sub-CSV data; store the CSV data corresponding to each data dimension in the multiple data dimensions on the node corresponding to the tree model (for example, store the CSV data corresponding to the root node on the root node), wherein the node includes the root node and child nodes of the tree model; store the tree model after storing the CSV data in a relational Oracle database.
[0072] Optionally, the implementation process of storing the CSV data corresponding to each data dimension in the multiple data dimensions on the node corresponding to the tree model can be: determining the subject domain corresponding to each data dimension in the multiple data dimensions of each sub-CSV data; obtaining the number of topics in the subject domain; obtaining the number of data corresponding to each topic in the subject domain, and determining the first topic with the largest number of data in the subject domain; creating an Excel table corresponding to the subject domain with the number of data of topics in the subject domain as the number of rows and the number of data corresponding to the first topic as the number of columns; adding each item of data in the CSV data to the Excel table; and hanging the Excel table with the added data at the node corresponding to each data dimension.
[0073] For example, if Figure 1C As shown, when the CSV data is the user's consumption record data, the sub-CSV data included are: online shopping data, dining data, and travel data. Keyword recognition is performed on each sub-CSV data, and the keywords "online shopping", "dining" and "travel" can be obtained. Relative to the user's consumption, online shopping data has the greatest impact on the user's consumption, so the online shopping dimension can be used as the root node. In addition, the online shopping data includes online shopping data on JD.com and Taobao, so the themes corresponding to the online shopping dimension are "JD.com", "Taobao", "Tmall" and "Pinduoduo", respectively. Therefore, the number of rows in the Excel table corresponding to the online shopping dimension is 4. For example, the number of online shopping on JD.com is 3 times, Taobao is 5 times, Tmall is 3 times, and Pinduoduo is 6 times, then the number of columns is 6. Then, after creating the Excel table, add the data corresponding to each theme to the Excel table, and attach the Excel table to the root node.
[0074] Adaptive, in accordance with Figure 1CAfter the user's CSV data is stored in the storage method shown, the target second device determines the index information of the sub-data to be exported according to the target sub-data export request, and determines the target node storing the sub-data to be exported from the tree model storing the data to be exported according to the sub-index information, and extracts the Excel table attached to the target node to obtain the sub-data to be exported, and sends the sub-data to be exported to the user terminal.
[0075] In this example, the second device stores the CSV data through a tree model (hierarchical storage), thereby improving the efficiency of searching for sub-data to be exported, thereby improving the data export speed.
[0076] See also Figure 2 , Figure 2 Another data export method provided in the embodiment of the present application includes but is not limited to the following steps:
[0077] 201: The user terminal sends a data export request to the first device.
[0078] 202: If the memory of the data to be exported is greater than the first threshold, the first device splits the data export request into N sub-data export requests, and adds a device identifier of a second device in the second device cluster to each sub-data export request.
[0079] 203: The first device creates a user name, password, access path, and port number for a message queue.
[0080] 204: The first device sends the user name, password, access path, and port number to the second device.
[0081] The second device is any second device in the second device cluster.
[0082] 205: The first device and the second device negotiate using the user name, password, access path, and port number to establish a message queue.
[0083] 206: The first device broadcasts the N sub-data export requests to the second device cluster through a message queue.
[0084] 207: The second device exports the sub-data to be exported corresponding to the target sub-data export request to the user terminal.
[0085] The second device is a second device corresponding to the device identifier of the sub-data request, and the sub-data request may be any one of the N sub-data requests.
[0086] 208: The user terminal synthesizes the data to be exported according to the N sub-data to be exported corresponding to the received N sub-data export requests.
[0087] It should be noted that Figure 2 The specific implementation process of each step of the method shown can be found in the above Figure 1B The specific implementation process of the method will not be described here.
[0088] It can be seen that in the embodiment of the present application, when exporting data, the data export request is split by the first device, and the multiple sub-data requests after the split are broadcast to the second device cluster, so that the sub-data to be exported corresponding to the sub-data request is asynchronously exported to the user terminal by multiple second devices, avoiding the downtime risk caused by exporting data through a single server; moreover, by asynchronously exporting data through multiple second devices, the data export speed is improved, the user's waiting time is reduced, and the user experience is improved.
[0089] See also Figure 3 , Figure 3 Another data export method provided in the embodiment of the present application includes but is not limited to the following steps:
[0090] 301: The user terminal sends a data export request to the first device.
[0091] 302: If the memory of the data to be exported is greater than the first threshold, the first device splits the data export request into N sub-data export requests, and adds a device identifier of a second device in the second device cluster to each sub-data export request.
[0092] 303: The first device creates the user name, password, access path and port number of the message queue.
[0093] 304: The first device sends the user name, password, access path and port number to the second device.
[0094] The second device is any second device in the second device cluster.
[0095] 305: The first device and the second device negotiate using the user name, password, access path, and port number to establish a message queue.
[0096] 306: The first device broadcasts the N sub-data export requests to the second device cluster through a message queue.
[0097] 307: If the memory of the target sub-data to be exported is greater than the second threshold, the second device creates m threads corresponding to the target sub-data export request.
[0098] The second device is a second device corresponding to the device identifier of the target sub-data request, and the target sub-data request may be any one of the N sub-data requests.
[0099] 308: The second device exports the sub-data to be exported corresponding to the target sub-data export request to the user terminal through m threads.
[0100] 309: The user terminal synthesizes the data to be exported according to the N sub-data to be exported corresponding to the received N sub-data export requests.
[0101] It should be noted that Figure 3 The specific implementation process of each step of the method shown can be found in the above Figure 1B The specific implementation process of the method will not be described here.
[0102] It can be seen that in the embodiment of the present application, when exporting data, the data export request is split by the first device, and the multiple sub-data requests after the split are broadcast to the second device cluster, so that the sub-data to be exported corresponding to the sub-data request is asynchronously exported to the user terminal by multiple second devices, avoiding the downtime risk caused by exporting data through a single server; moreover, by asynchronously exporting data through multiple second devices, the data export speed is improved, the user's waiting time is reduced, and the user experience is improved; moreover, when each second device exports data, the sub-data export request is further split, and the data is synchronously exported through multiple threads, further improving the data export speed.
[0103] See also Figure 4 , Figure 4 This is a schematic diagram of the structure of a data export device provided in an embodiment of the present application. Figure 4 As shown, the data export device 400 includes a processor, a memory, a communication interface, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the processor;
[0104] When the data export apparatus 400 is applied to the first device, the program includes instructions for executing the following steps:
[0105] Receiving a data export request sent by a user terminal, the data export request including a data identifier of the data to be exported and a memory of the data to be exported;
[0106] If the memory of the data to be exported is greater than the first threshold, split the data export request into N sub-data export requests, and add a device identifier of the second device in the second device cluster to each sub-data export request, and the device identifiers between any two sub-data export requests are different;
[0107] Broadcast the N sub-data export requests to the second device cluster so that the target second device exports the sub-data to be exported corresponding to the target sub-data export request to the user terminal, the target sub-data export request is any one of the N sub-data export requests, and the target second device is the second device in the second device cluster corresponding to the device identifier of the target sub-data request; and so that the user terminal synthesizes the data to be exported based on the received sub-data to be exported.
[0108] In a possible implementation manner, in terms of splitting the data export request into N sub-data export requests, the above program is specifically used to execute instructions of the following steps:
[0109] Acquire multiple data types corresponding to the data to be exported;
[0110] Obtain keywords corresponding to the multiple data types, classify the keywords corresponding to the multiple data types, and construct a sub-data export request for the data types corresponding to the keywords belonging to one category, to obtain N sub-data export requests;
[0111] Add a device ID to each child data export request.
[0112] In a possible implementation manner, in terms of splitting the data export request into N sub-data export requests, the above program is specifically used to execute instructions of the following steps:
[0113] Acquire multiple data types corresponding to the data to be exported;
[0114] Determine the data format corresponding to each data type, and construct a sub-data export request for the data types with the same data format, to obtain N sub-data export requests;
[0115] Add a device ID to each child data export request.
[0116] In a possible implementation manner, in terms of broadcasting the N sub-data export requests to the second device cluster, the above program is specifically used to execute instructions of the following steps:
[0117] The N sub-data export requests are broadcasted to the second device cluster through a message queue.
[0118] When the data export apparatus 400 is applied to the second device, the program includes instructions for executing the following steps:
[0119] Receiving N sub-data export requests broadcasted by the first device, the target sub-data export request is any one of the N sub-data export requests, and the target second device is a second device in the second device cluster corresponding to the device identifier of the target sub-data request;
[0120] The sub-data to be exported corresponding to the target sub-data export request is exported to the user terminal, so that the user terminal synthesizes the data to be exported according to the received sub-data to be exported.
[0121] In a possible implementation manner, in terms of exporting the sub-data to be exported corresponding to the target sub-data export request to the user terminal, the above program is specifically used to execute instructions of the following steps:
[0122] Parsing the target sub-data export request, and obtaining the memory of the sub-data to be exported corresponding to the target sub-data export request;
[0123] If the memory of the sub-data to be exported is greater than a second threshold, m threads corresponding to the target sub-data export request are created, and the sub-data to be exported corresponding to the target sub-data export request is exported to the user terminal through the m threads.
[0124] In a possible implementation manner, in terms of creating m threads corresponding to the target sub-data export request, the above program is specifically used to execute instructions of the following steps:
[0125] Get the preset maximum number of threads and the maximum amount of data that each thread can process;
[0126] Obtaining the amount of data required to be exported by the target sub-data export request;
[0127] The m threads are created according to the maximum amount of data that each thread can process and the amount of data that needs to be exported by the target sub-data export request.
[0128] See also Figure 5 , Figure 5 The present application provides a functional unit block diagram of a data export device, wherein the data export device uses a first device. The data export device 500 includes:
[0129] A receiving unit 510 is configured to receive a data export request sent by a user terminal, wherein the data export request includes a data identifier of the data to be exported and a memory of the data to be exported;
[0130] a splitting unit 520, configured to split the data export request into N sub-data export requests if the memory of the data to be exported is greater than a first threshold, and add a device identifier of a second device in a second device cluster to each sub-data export request, and the device identifiers of any two sub-data export requests are different;
[0131] The broadcast unit 530 is used to broadcast the N sub-data export requests to the second device cluster so that the target second device exports the sub-data to be exported corresponding to the target sub-data export request to the user terminal, the target sub-data export request is any one of the N sub-data export requests, and the target second device is the second device in the second device cluster corresponding to the device identifier of the target sub-data request; and so that the user terminal synthesizes the data to be exported based on the received sub-data to be exported.
[0132] In a possible implementation manner, in splitting the data export request into N sub-data export requests, the splitting unit 520 is specifically configured to:
[0133] Acquire multiple data types corresponding to the data to be exported;
[0134] Obtain keywords corresponding to the multiple data types, classify the keywords corresponding to the multiple data types, and construct a sub-data export request for the data types corresponding to the keywords belonging to one category, to obtain N sub-data export requests;
[0135] Add a device ID to each child data export request.
[0136] In a possible implementation manner, in splitting the data export request into N sub-data export requests, the splitting unit 520 is specifically configured to:
[0137] Acquire multiple data types corresponding to the data to be exported;
[0138] Determine the data format corresponding to each data type, and construct a sub-data export request for the data types with the same data format, to obtain N sub-data export requests;
[0139] Add a device ID to each child data export request.
[0140] In a possible implementation manner, in broadcasting the N sub-data export requests to the second device cluster, the broadcast unit 530 is specifically configured to:
[0141] The N sub-data export requests are broadcasted to the second device cluster through a message queue.
[0142] See also Figure 6 , Figure 6 The functional unit composition block diagram of a data export device provided in an embodiment of the present application, the data export device uses a second device. The data export device 600 includes: a receiving unit 610 and an export unit 620, wherein:
[0143] The receiving unit 610 is configured to receive N sub-data export requests broadcasted by the first device, wherein the target sub-data export request is any one of the N sub-data export requests, and the target second device is a second device in the second device cluster corresponding to the device identifier of the target sub-data request;
[0144] The exporting unit 620 is used to export the sub-data to be exported corresponding to the target sub-data export request to the user terminal, so that the user terminal can synthesize the data to be exported according to the received sub-data to be exported.
[0145] In a possible implementation manner, in exporting the sub-data to be exported corresponding to the target sub-data export request to the user terminal, the exporting unit 620 is specifically configured to:
[0146] Parsing the target sub-data export request, and obtaining the memory of the sub-data to be exported corresponding to the target sub-data export request;
[0147] If the memory of the sub-data to be exported is greater than a second threshold, m threads corresponding to the target sub-data export request are created, and the sub-data to be exported corresponding to the target sub-data export request is exported to the user terminal through the m threads.
[0148] In a possible implementation manner, in terms of creating m threads corresponding to the target sub-data export request, the export unit 620 is specifically configured to:
[0149] Get the preset maximum number of threads and the maximum amount of data that each thread can process;
[0150] Obtaining the amount of data required to be exported by the target sub-data export request;
[0151] The m threads are created according to the maximum amount of data that each thread can process and the amount of data that needs to be exported by the target sub-data export request.
[0152] An embodiment of the present application also provides a computer storage medium, wherein the computer-readable storage medium stores a computer program, and the computer program is executed by a processor to implement part or all of the steps of any data export method recorded in the above method embodiments.
[0153] An embodiment of the present application also provides a computer program product, which includes a non-transitory computer-readable storage medium storing a computer program, and the computer program is operable to cause a computer to execute part or all of the steps of any data export method recorded in the above method embodiments.
[0154] It should be noted that, for the aforementioned method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the described order of actions, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all optional embodiments, and the actions and modules involved are not necessarily required by the present application.
[0155] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.
[0156] In the several embodiments provided in the present application, it should be understood that the disclosed device can be implemented in other ways. For example, the device embodiments described above are only schematic, such as the division of the units, which is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, and the indirect coupling or communication connection of the device or unit can be electrical or other forms.
[0157] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
[0158] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above integrated unit may be implemented in the form of hardware or in the form of a software program module.
[0159] If the integrated unit is implemented in the form of a software program module and sold or used as an independent product, it can be stored in a computer-readable memory. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a memory, including a number of instructions to enable a computer device (which can be a personal computer, server or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned memory includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, disk or CD-ROM and other media that can store program codes.
[0160] A person skilled in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing related hardware through a program, and the program can be stored in a computer-readable memory, and the memory can include: a flash drive, a read-only memory (English: Read-Only Memory, abbreviated as: ROM), a random access memory (English: Random Access Memory, abbreviated as: RAM), a magnetic disk or an optical disk, etc.
[0161] The embodiments of the present application are introduced in detail above. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method and core idea of the present application. At the same time, for general technical personnel in this field, according to the idea of the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.
Claims
1. A data export method, characterized in that: include: The first device receives a data export request sent by a user terminal, where the data export request includes a data identifier of the data to be exported and a memory of the data to be exported; If the memory of the data to be exported is greater than a first threshold, the first device splits the data export request into N sub-data export requests, and adds a device identifier of a second device in a second device cluster to each sub-data export request, and the device identifiers of any two sub-data export requests are different; including: Acquire multiple data types corresponding to the data to be exported; Obtain keywords corresponding to the multiple data types, classify the keywords corresponding to the multiple data types, and construct a sub-data export request for the data types corresponding to the keywords belonging to one category, to obtain N sub-data export requests; add a device identifier of a second device in the second device cluster to each sub-data export request; or determine a data format corresponding to each data type, and construct a sub-data export request for data types with the same data format, to obtain N sub-data export requests; add the device identifier of the second device in the second device cluster to each sub-data export request; The first device broadcasts the N sub-data export requests to the second device cluster so that the target second device exports the sub-data to be exported corresponding to the target sub-data export request to the user terminal, including: The target second device parses the target sub-data export request to obtain the memory of the sub-data to be exported corresponding to the target sub-data export request; if the memory of the sub-data to be exported is greater than the second threshold, the target second device obtains the preset maximum number of threads and the maximum amount of data that each thread can process; the target second device obtains the amount of data required to be exported by the target sub-data export request; the target second device creates m threads according to the maximum amount of data that each thread can process and the amount of data required to be exported by the target sub-data export request, and exports the sub-data to be exported corresponding to the target sub-data export request to the user terminal through the m threads; The target sub-data export request is any one of the N sub-data export requests, and the target second device is the second device in the second device cluster corresponding to the device identifier of the target sub-data export request; and so that the user terminal synthesizes the data to be exported based on the received sub-data to be exported.
2. The method according to claim 1, characterized in that The first device broadcasts the N sub-data export requests to the second device cluster, including: The first device broadcasts the N sub-data export requests to the second device cluster through a message queue.
3. The method according to claim 2, characterized in that Before the first device broadcasts the N sub-data export requests to the second device cluster through a message queue, the method further includes: The first device creates a user name, password, access path and port number for the message queue; The first device sends the user name, the password, the access path and the port number to each second device in the second device cluster, so as to establish the message queue with each second device in the second device cluster according to the user name, the password, the access path and the port number.
4. A data export device, characterized in that: The data export device is used to execute the method according to any one of claims 1 to 3, and the data export device includes: A receiving unit, configured to receive a data export request sent by a user terminal, wherein the data export request includes a data identifier of the data to be exported and a memory of the data to be exported; a splitting unit, configured to split the data export request into N sub-data export requests if the memory of the data to be exported is greater than a first threshold, and add a device identifier of a second device in a second device cluster to each sub-data export request, and the device identifiers of any two sub-data export requests are different; A broadcasting unit is used to broadcast the N sub-data export requests to the second device cluster so that the target second device exports the sub-data to be exported corresponding to the target sub-data export request to the user terminal, the target sub-data export request is any one of the N sub-data export requests, and the target second device is the second device in the second device cluster corresponding to the device identifier of the target sub-data export request; and so that the user terminal synthesizes the data to be exported based on the received sub-data to be exported.
5. An electronic device, characterized in that: The method comprises a processor, a memory, a communication interface and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the processor, and the programs include instructions for executing the steps in the method according to any one of claims 1 to 3.
6. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and the computer program is executed by a processor to implement the method according to any one of claims 1 to 3.
