Data processing and exporting method and device based on distributed system and medium

Abstract

The invention discloses a data processing and exporting method and device based on a distributed system and a medium. The method comprises the steps of obtaining a to-be-processed data set and slicing the to-be-processed data set according to the to-be-processed data set and metadata description information to obtain a plurality of to-be-processed data slices; acquiring a current distributed node cluster corresponding to the current distributed system, and acquiring node registration processing capability information and node state monitoring information of each node; performing distribution processing on each to-be-processed data piece through a distributed node management method to obtain a data piece node distribution result and a data piece node distribution table; and through a data processing and exporting method, indicating each node to carry out data processing and exporting according to the data piece node distribution table, and obtaining and exporting a data processing result. The problems of low efficiency and slow data processing speed caused by processing data through a traditional single node are solved, and the efficiency and speed of data processing and the accuracy of data export are improved.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a data processing and exporting method, device and medium based on a distributed system. Background Art

[0002] With the development of big data technology, massive amounts of data continue to emerge, and the demand for processing large amounts of data is increasing. Improving the efficiency of data processing is also crucial.

[0003] In the process of implementing the present invention, the inventors found that the prior art has the following defects: At present, data is processed through a traditional single node, and this data processing method can no longer meet the requirements of efficient processing of big data, and is inefficient and slow in data processing speed; in the event of an abnormality in a single node, the fault recovery capability is poor. Summary of the invention

[0004] The present invention provides a data processing and exporting method, device and medium based on a distributed system, so as to improve the efficiency and speed of data processing and the accuracy of data export.

[0005] According to one aspect of the present invention, a data processing and exporting method based on a distributed system is provided, which includes:

[0006] Acquire a data set to be processed and metadata description information corresponding to the data set to be processed;

[0007] Slicing the data set to be processed according to the metadata description information to obtain a plurality of data slices to be processed;

[0008] Obtain the current distributed node cluster corresponding to the current distributed system, and obtain the node registration processing capacity information and node status monitoring information of each node in the current distributed node cluster;

[0009] Through the pre-set distributed node management method, each data slice to be processed is allocated and processed according to the node registration processing capacity information and node status monitoring information, the data slice node allocation result is obtained, and the corresponding data slice node allocation table is generated;

[0010] By using a preset data processing and exporting method and according to the data slice node allocation table, each node is instructed to perform data processing and exporting, and a data processing result is obtained and exported.

[0011] According to another aspect of the present invention, a data processing and exporting device based on a distributed system is provided, which includes:

[0012] A metadata description information acquisition module, used to acquire a data set to be processed and metadata description information corresponding to the data set to be processed;

[0013] A module for obtaining data slices to be processed, used for slicing the data set to be processed according to the metadata description information to obtain multiple data slices to be processed;

[0014] The node registration processing capability information and node status monitoring information acquisition module is used to acquire the current distributed node cluster corresponding to the current distributed system, and acquire the node registration processing capability information and node status monitoring information of each node in the current distributed node cluster;

[0015] The data slice node allocation table generation module is used to allocate and process each data slice to be processed according to the node registration processing capacity information and node status monitoring information through a pre-set distributed node management method, obtain the data slice node allocation result, and generate a corresponding data slice node allocation table;

[0016] The data processing result determination and export module is used to instruct each node to perform data processing and export through a preset data processing and export method and according to the data slice node allocation table, and obtain and export the data processing result.

[0017] According to another aspect of the present invention, there is provided an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the data processing and export method based on a distributed system as described in any embodiment of the present invention is implemented.

[0018] According to another aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the data processing and export method based on a distributed system described in any embodiment of the present invention when executed.

[0019] The technical solution of the embodiment of the present invention obtains a data set to be processed and metadata description information corresponding to the data set to be processed; slices the data set to be processed according to the metadata description information to obtain multiple data slices to be processed; obtains the current distributed node cluster corresponding to the current distributed system, and obtains the node registration processing capacity information and node status monitoring information of each node in the current distributed node cluster; through a pre-set distributed node management method, according to the node registration processing capacity information and the node status monitoring information, allocates and processes each data slice to be processed, obtains the data slice node allocation result, and generates a corresponding data slice node allocation table; through a pre-set data processing and exporting method, according to the data slice node allocation table, instructs each node to process and export data, obtains and exports the data processing result. The problem of low efficiency and slow data processing speed caused by processing data through a traditional single node is solved, and the efficiency and speed of data processing and the accuracy of data export are improved.

[0020] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0022] Figure 1 is a flow chart of a data processing and exporting method based on a distributed system provided according to Embodiment 1 of the present invention;

[0023] Figure 2 is a schematic diagram of the structure of a data processing and exporting device based on a distributed system provided according to Embodiment 2 of the present invention;

[0024] Figure 3 It is a structural schematic diagram of an electronic device provided according to Embodiment 3 of the present invention. DETAILED DESCRIPTION

[0025] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0026] It should be noted that the terms "target", "current", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0027] Embodiment 1

[0028] Figure 1 A flowchart of a data processing and exporting method based on a distributed system is provided in Embodiment 1 of the present invention. This embodiment is applicable to situations where data is processed and exported based on a distributed system. The method can be executed by a data processing and exporting device based on a distributed system, and the data processing and exporting device based on a distributed system can be implemented in the form of hardware and / or software.

[0029] Correspondingly, such as Figure 1 As shown, the method includes:

[0030] S110: Acquire a data set to be processed and metadata description information corresponding to the data set to be processed.

[0031] The data set to be processed may be a data set that needs to be processed and exported. The metadata description information may be data information describing the data set to be processed. Specifically, the metadata description information includes the total size of the data set and the number of shards.

[0032] Specifically, the total size of the data set may be information describing the size of the data set to be processed. The size of the number of shards may be information describing the number of shards of the data set to be processed.

[0033] S120. Slice the data set to be processed according to the metadata description information to obtain a plurality of data slices to be processed.

[0034] The data slice to be processed may be a plurality of data slices obtained by slicing the data set to be processed.

[0035] Optionally, slicing the data set to be processed according to the metadata description information to obtain multiple data slices to be processed, including: performing data processing operations on the data set to be processed according to the total size of the data set and the size of the number of slices to obtain multiple data slices to be processed corresponding to the data set to be processed; wherein the size of the data slice to be processed is calculated by dividing the total size of the data set by the size of the number of slices.

[0036] For example, assuming that the total size of the data set to be processed is 10TB and the number of shards is 1000, the 10TB can be sliced ​​into 1000 parts to obtain 1000 data slices to be processed, and the size of the data slices to be processed is 10GB.

[0037] Specifically, after slicing the data set to be processed according to the metadata description information to obtain multiple data slices to be processed, it also includes: matching the corresponding data slice identifier and data slice location information for each data slice to be processed, and jointly storing each data slice to be processed, data slice identifier and data slice location information in the memory corresponding to the distributed system.

[0038] In this embodiment, each data slice to be processed may be matched with a corresponding data slice identifier and data slice location information. Continuing with the previous example, 1000 data slices to be processed may be matched with corresponding data slice identifiers and data slice location information respectively.

[0039] For example, the data slice identifier of the data slice 1 to be processed is ID001, and the data slice location information is: / data / path1 / file001. The data slice identifier of the data slice 1000 to be processed is ID001000, and the data slice location information is: / data / path1 / file001000.

[0040] Furthermore, each data slice to be processed, as well as the data slice identifier and data slice location information corresponding to the data slice to be processed, can be jointly stored in a memory corresponding to the distributed system, and can be used in subsequent allocation processing of the data slices to be processed.

[0041] S130: Acquire a current distributed node cluster corresponding to the current distributed system, and acquire node registration processing capability information and node status monitoring information of each node in the current distributed node cluster.

[0042] The current distributed node cluster may be a node cluster in the current distributed system, including a plurality of nodes, and the plurality of nodes may be used to perform processing operations on the data slices to be processed respectively.

[0043] Specifically, the node registration processing capability information may be information describing the data processing capability of the node. The node status monitoring information may be information for monitoring the node status. Each node may correspond to a different node status. The node status monitoring information may include an idle state and a non-idle state.

[0044] S140. Allocate and process each data slice to be processed according to the node registration processing capability information and the node status monitoring information through a pre-set distributed node management method, obtain a data slice node allocation result, and generate a corresponding data slice node allocation table.

[0045] The distributed node management method may be a method for distributing the nodes through a distributed system, and may instruct different nodes to process the data slices to be processed. The data slice node allocation result may be the result of allocating the data slices to be processed to different nodes. The data slice node allocation table may be an allocation table consisting of the data slice node allocation result, the data slice identifier, and the data slice location information.

[0046] Specifically, the generating of the corresponding data slice node allocation table includes: generating the corresponding data slice node allocation table according to the obtained data slice node allocation result and in combination with the data slice identifier and the data slice location information. The data node allocation table thus obtained can be better applied to the node allocation processing of the data slice, thereby improving the accuracy and efficiency of the allocation of the data slice to be processed.

[0047] Optionally, the pre-set distributed node management method is used to allocate and process each data slice to be processed according to the node registration processing capability information and the node status monitoring information to obtain a data slice node allocation result, including: using the distributed node management method and judging whether each node is in a spatial state according to the node status monitoring information; if so, obtaining at least one target node in an idle state, and obtaining the node registration processing capability information corresponding to each target node; obtaining the size of the data slice to be processed, and judging whether the node registration processing capability information corresponding to each target node reaches the size of the data slice to be processed according to the node registration processing capability information; if so, allocating the data slice to be processed to each target node in sequence, and obtaining the data slice node allocation result.

[0048] In this embodiment, assuming that the current distributed node cluster includes 10 nodes, whether each node is in a space state is determined based on the node state monitoring information. Through the determination, it can be known that 8 nodes are in an idle state, and these 8 nodes are determined as target nodes.

[0049] Furthermore, the node registration processing capacity information corresponding to the eight target nodes is obtained. Assuming that the target node 1 is at full load, the node registration processing capacity information of the remaining seven target nodes is 10GB. Since the size of the data slice to be processed is 10GB, the 1000 data slices to be processed can be allocated to the seven target nodes in sequence, and the seven target nodes are instructed to perform data processing operations on the data slices to be processed in sequence, and the corresponding data slice node allocation results can be obtained.

[0050] Specifically, after determining whether each node is in a spatial state according to the node status monitoring information, it also includes: if not, the nodes in a non-idle state are discarded; after determining whether the node registration processing capability information corresponding to each target node reaches the size of the data slice to be processed according to the node registration processing capability information, it also includes: if not, the nodes that do not reach the size of the data slice to be processed are discarded.

[0051] Continuing with the previous example, since among the 10 nodes, 8 nodes are in an idle state and 2 are in a non-idle state, the 2 nodes in the non-idle state will be discarded.

[0052] Furthermore, among the eight target nodes, target node 1 is in a fully loaded state, so target node 1 needs to be discarded. The advantage of this setting is that a better node can be selected more accurately to process the data slice to be processed, thereby improving the efficiency and speed of the node's data processing of the data slice to be processed.

[0053] S150. Instruct each node to perform data processing and export through a preset data processing and export method and according to the data slice node allocation table, and obtain and export the data processing result.

[0054] The data processing result may be a merged result obtained by the node from processing the data of the data slice to be processed.

[0055] Optionally, the method uses a preset data processing and export method and instructs each node to perform data processing and export according to the data slice node allocation table to obtain and export data processing results, including: using a preset data processing and export method and instructing each node to perform data processing on the data in the corresponding data slice to be processed according to the data slice location information according to the data slice node allocation table to obtain data slice processing results; sequentially combining the data slice processing results according to the data slice identifier to obtain data processing results corresponding to the data set to be processed; and obtaining and exporting the data processing results according to a preset data export path.

[0056] In this embodiment, it is necessary to instruct each node to process one or more matching data slices to be processed according to the data slice node allocation table. Before data processing, the data of the corresponding data slice to be processed can be obtained in the memory according to the data slice location information.

[0057] For example, assume that there are 3 target nodes that meet the requirements, and need to process 8 data slices to be processed. Specifically, the data slices to be processed corresponding to target node 2 are: data slice 1 to be processed, data slice 4 to be processed, and data slice 7 to be processed; the data slices to be processed corresponding to target node 3 are: data slice 2 to be processed, data slice 5 to be processed, and data slice 8 to be processed; the data slices to be processed corresponding to target node 4 are: data slice 3 to be processed, and data slice 6 to be processed. The data slice processing results corresponding to each data slice to be processed can be obtained.

[0058] Assume that data slice identifier 1 is ID001; data slice identifier 2 is ID002; data slice identifier 3 is ID003; data slice identifier 4 is ID004; data slice identifier 5 is ID005; data slice identifier 6 is ID006; data slice identifier 7 is ID007; and data slice identifier 8 is ID008.

[0059] Furthermore, the data slice processing results can be sequentially combined according to the data slice identifiers, that is, combined in order from ID001 to ID008, to obtain the data processing results corresponding to the data set to be processed. Accordingly, the data processing results can be exported according to the data export path, assuming it is / data / path1 / file.

[0060] Optionally, it also includes: in the process of instructing each node to process the data in the corresponding data slice to be processed according to the data slice location information, real-time monitoring of the node status of each node is performed to obtain the real-time monitoring result of the node status; in the real-time monitoring result of the node status, if there is any abnormal state node, the data in the data slice to be processed corresponding to the abnormal state node is sent to other normal state nodes, and the node status monitoring information corresponding to the abnormal state node is modified, and the data node allocation table is updated.

[0061] In this embodiment, in the process of instructing each node to process the data in the corresponding data slice to be processed according to the data slice location information, it is also necessary to monitor the node status of each node in real time to obtain the node status real-time monitoring result. Specifically, the node status real-time monitoring result may include an abnormal state and a normal state. A node in an abnormal state is an abnormal state node; a node in a normal state is a normal state node.

[0062] Continuing with the previous example, in the real-time monitoring results of node status, assuming that the target node 4 is in an abnormal state, then the target node 4 is an abnormal state node, and the data in the data slice to be processed corresponding to the target node 4 needs to be sent to other normal state nodes, that is, the data slice to be processed 3 and the data slice to be processed 6 are sent to the target node 2 and the target node 3. The data slice to be processed 3 can be assigned to the target node 2, and the data slice to be processed 6 can be assigned to the target node 3.

[0063] Accordingly, it is necessary to modify the node status monitoring information corresponding to the abnormal state node and update the data node allocation table.

[0064] The technical solution of the embodiment of the present invention obtains a data set to be processed and metadata description information corresponding to the data set to be processed; slices the data set to be processed according to the metadata description information to obtain multiple data slices to be processed; obtains the current distributed node cluster corresponding to the current distributed system, and obtains the node registration processing capacity information and node status monitoring information of each node in the current distributed node cluster; through a pre-set distributed node management method, according to the node registration processing capacity information and the node status monitoring information, allocates and processes each data slice to be processed, obtains the data slice node allocation result, and generates a corresponding data slice node allocation table; through a pre-set data processing and exporting method, according to the data slice node allocation table, instructs each node to process and export data, obtains and exports the data processing result. The problem of low efficiency and slow data processing speed caused by processing data through a traditional single node is solved, and the efficiency and speed of data processing and the accuracy of data export are improved.

[0065] Embodiment 2

[0066] Figure 2 The structure diagram of a data processing and exporting device based on a distributed system provided in the second embodiment of the present invention. The data processing and exporting device based on a distributed system provided in this embodiment can be implemented by software and / or hardware, and can be configured in a terminal device or a server to implement a data processing and exporting method based on a distributed system in the embodiment of the present invention. Figure 2 As shown, the device includes: a metadata description information acquisition module 210, a data slice to be processed acquisition module 220, a node registration processing capacity information and node status monitoring information acquisition module 230, a data slice node allocation table generation module 240 and a data processing result determination and export module 250.

[0067] The metadata description information acquisition module 210 is used to acquire the data set to be processed and the metadata description information corresponding to the data set to be processed;

[0068] A module 220 for obtaining data slices to be processed is used to slice the data set to be processed according to the metadata description information to obtain a plurality of data slices to be processed;

[0069] The node registration processing capability information and node status monitoring information acquisition module 230 is used to acquire the current distributed node cluster corresponding to the current distributed system, and acquire the node registration processing capability information and node status monitoring information of each node in the current distributed node cluster;

[0070] The data slice node allocation table generation module 240 is used to allocate and process each data slice to be processed according to the node registration processing capability information and the node status monitoring information through a pre-set distributed node management method, obtain the data slice node allocation result, and generate a corresponding data slice node allocation table;

[0071] The data processing result determination and export module 250 is used to instruct each node to perform data processing and export according to the data slice node allocation table through a preset data processing and export method, and obtain and export the data processing result.

[0072] The technical solution of the embodiment of the present invention obtains a data set to be processed and metadata description information corresponding to the data set to be processed; slices the data set to be processed according to the metadata description information to obtain multiple data slices to be processed; obtains the current distributed node cluster corresponding to the current distributed system, and obtains the node registration processing capacity information and node status monitoring information of each node in the current distributed node cluster; through a pre-set distributed node management method, according to the node registration processing capacity information and the node status monitoring information, allocates and processes each data slice to be processed, obtains the data slice node allocation result, and generates a corresponding data slice node allocation table; through a pre-set data processing and exporting method, according to the data slice node allocation table, instructs each node to process and export data, obtains and exports the data processing result. The problem of low efficiency and slow data processing speed caused by processing data through a traditional single node is solved, and the efficiency and speed of data processing and the accuracy of data export are improved.

[0073] Based on the above embodiments, the metadata description information includes the total size of the data set and the number of shards;

[0074] On the basis of the above embodiments, the module 220 for obtaining the data slices to be processed can be specifically used to: perform data processing operations on the data slices to be processed according to the total size of the data set and the size of the number of slices, and obtain multiple data slices to be processed corresponding to the data set to be processed; wherein the size of the data slices to be processed is calculated by dividing the total size of the data set by the size of the number of slices.

[0075] On the basis of the above-mentioned embodiments, the data slice node allocation table generation module 240 can be specifically used for: through the distributed node management method, and according to the node status monitoring information, to determine whether each node is in a spatial state; if so, to obtain at least one target node in an idle state, and to obtain the node registration processing capacity information corresponding to each target node; to obtain the size of the data slice to be processed, and according to the node registration processing capacity information, to determine whether the node registration processing capacity information corresponding to each target node reaches the size of the data slice to be processed; if so, to allocate the data slice to be processed to each target node in sequence, and to obtain the data slice node allocation result.

[0076] On the basis of the above-mentioned embodiments, the data slice node allocation table generation module 240 can also be specifically used for: after judging whether each node is in a space state according to the node status monitoring information, if not, the nodes in a non-idle state are discarded; after judging whether the node registration processing capacity information corresponding to each target node reaches the size of the data slice to be processed according to the node registration processing capacity information, if not, the nodes that do not reach the size of the data slice to be processed are discarded.

[0077] On the basis of the above embodiments, it also includes a joint storage module, which can be specifically used for: after slicing the data set to be processed according to the metadata description information to obtain multiple data slices to be processed, matching the corresponding data slice identifier and data slice location information for each data slice to be processed, and jointly storing each data slice to be processed, the data slice identifier and the data slice location information in the memory corresponding to the distributed system.

[0078] Based on the above embodiments, the data slice node allocation table generation module 240 can also be specifically used to generate a corresponding data slice node allocation table according to the obtained data slice node allocation result and in combination with the data slice identifier and the data slice location information.

[0079] On the basis of the above-mentioned embodiments, the data processing result determination and export module 250 can be specifically used for: through a pre-set data processing and export method, and according to the data slice node allocation table, to instruct each node to perform data processing on the data in the corresponding data slice to be processed according to the data slice location information, to obtain the data slice processing result; according to the data slice identifier, to sequentially combine the data slice processing results to obtain the data processing results corresponding to the data set to be processed; obtain the data export path according to the pre-set data, and export the data processing results.

[0080] On the basis of the above embodiments, it can also be specifically used for: in the process of instructing each node to process the data in the corresponding data slice to be processed according to the data slice location information, real-time monitoring of the node status of each node is performed to obtain the real-time monitoring result of the node status; in the real-time monitoring result of the node status, if there is any abnormal state node, the data in the data slice to be processed corresponding to the abnormal state node is sent to other normal state nodes, and the node status monitoring information corresponding to the abnormal state node is modified, and the data node allocation table is updated.

[0081] The data processing and exporting device based on a distributed system provided in an embodiment of the present invention can execute the data processing and exporting method based on a distributed system provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0082] Embodiment 3

[0083] Figure 3 A schematic diagram of the structure of an electronic device 10 that can be used to implement the third embodiment of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0084] like Figure 3 As shown, the electronic device 10 includes at least one processor 11, and a memory connected to the at least one processor 11, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 to the random access memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the electronic device 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other through a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0085] A number of components in the electronic device 10 are connected to the I / O interface 15, including: an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0086] The processor 11 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 11 performs the various methods and processes described above, such as data processing and derivation methods based on distributed systems.

[0087] In some embodiments, the data processing and exporting method based on the distributed system can be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 18. In some embodiments, part or all of the computer program can be loaded and / or installed on the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of the data processing and exporting method based on the distributed system described above can be performed. Alternatively, in other embodiments, the processor 11 can be configured to perform the data processing and exporting method based on the distributed system by any other appropriate means (for example, by means of firmware).

[0088] The method includes: obtaining a data set to be processed and metadata description information corresponding to the data set to be processed; slicing the data set to be processed according to the metadata description information to obtain multiple data slices to be processed; obtaining a current distributed node cluster corresponding to the current distributed system, and obtaining node registration processing capability information and node status monitoring information of each node in the current distributed node cluster; using a pre-set distributed node management method, according to the node registration processing capability information and the node status monitoring information, allocating and processing each data slice to be processed, obtaining a data slice node allocation result, and generating a corresponding data slice node allocation table; using a pre-set data processing and exporting method, and according to the data slice node allocation table, instructing each node to perform data processing and exporting, obtaining and exporting the data processing result.

[0089] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0090] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the computer program is executed by the processor, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0091] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in combination with an instruction execution system, device or equipment. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0092] To provide interaction with a user, the systems and techniques described herein may be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0093] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0094] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client and server relationship is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services.

[0095] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.

[0096] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

[0097] Embodiment 4

[0098] Embodiment 4 of the present invention also provides a computer-readable storage medium, wherein the computer-readable instructions are used to execute a data processing and export method based on a distributed system when executed by a computer processor, the method comprising: obtaining a data set to be processed and metadata description information corresponding to the data set to be processed; slicing the data set to be processed according to the metadata description information to obtain multiple data slices to be processed; obtaining a current distributed node cluster corresponding to the current distributed system, and obtaining node registration processing capability information and node status monitoring information of each node in the current distributed node cluster; through a pre-set distributed node management method, each data slice to be processed is allocated and processed according to the node registration processing capability information and the node status monitoring information, a data slice node allocation result is obtained, and a corresponding data slice node allocation table is generated; through a pre-set data processing and export method, and according to the data slice node allocation table, each node is instructed to perform data processing and export, and the data processing result is obtained and exported.

[0099] Of course, the computer-readable storage medium provided in the embodiment of the present invention has computer executable instructions that are not limited to the method operations described above, and can also execute related operations in the data processing and export method based on a distributed system provided in any embodiment of the present invention.

[0100] Through the above description of the implementation methods, the technicians in the relevant field can clearly understand that the present invention can be implemented by means of software and necessary general hardware, and of course it can also be implemented by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as a computer floppy disk, a read-only memory (ROM), a random access memory (RAM), a flash memory (FLASH), a hard disk or an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present invention.

[0101] It is worth noting that in the above-mentioned embodiment of the data processing and exporting device based on a distributed system, the various units and modules included are only divided according to functional logic, but are not limited to the above-mentioned division, as long as the corresponding functions can be achieved; in addition, the specific names of the functional units are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of the present invention.

[0102] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A data processing and exporting method based on a distributed system, characterized in that: include: Acquire a data set to be processed and metadata description information corresponding to the data set to be processed; Slicing the data set to be processed according to the metadata description information to obtain a plurality of data slices to be processed; Obtain the current distributed node cluster corresponding to the current distributed system, and obtain the node registration processing capacity information and node status monitoring information of each node in the current distributed node cluster; Through the pre-set distributed node management method, each data slice to be processed is allocated and processed according to the node registration processing capacity information and node status monitoring information, the data slice node allocation result is obtained, and the corresponding data slice node allocation table is generated; By using a preset data processing and exporting method and according to the data slice node allocation table, each node is instructed to perform data processing and exporting, and a data processing result is obtained and exported.

2. The method according to claim 1, characterized in that The metadata description information includes the total size of the data set and the number of shards; The step of slicing the data set to be processed according to the metadata description information to obtain a plurality of data slices to be processed includes: According to the total size of the data set and the number of shards, a data processing operation is performed on the data set to be processed to obtain a plurality of data shards to be processed corresponding to the data set to be processed; The size of the data slice to be processed is calculated by dividing the total size of the data set by the number of slices.

3. The method according to claim 2, characterized in that The distributed node management method that is pre-set is used to allocate and process each data slice to be processed according to the node registration processing capability information and the node status monitoring information to obtain the data slice node allocation result, including: By using the distributed node management method and according to the node status monitoring information, it is determined whether each node is in a spatial state; If yes, then obtaining at least one target node in an idle state, and obtaining node registration processing capability information corresponding to each of the target nodes; Obtain the size of the data slice to be processed, and determine whether the node registration processing capacity information corresponding to each target node reaches the size of the data slice to be processed based on the node registration processing capacity information. If so, allocate the data slice to be processed to each target node in sequence and obtain the data slice node allocation result.

4. The method according to claim 3, characterized in that: After determining whether each node is in a space state according to the node state monitoring information, the method further includes: If not, the nodes in the non-idle state will be discarded; After determining, according to the node registration processing capability information, whether the node registration processing capability information corresponding to each target node reaches the size of the data slice to be processed, the method further includes: If not, the nodes that do not reach the size of the data slice to be processed will be discarded.

5. The method according to claim 4, characterized in that After slicing the data set to be processed according to the metadata description information to obtain a plurality of data slices to be processed, the method further includes: Matching a corresponding data slice identifier and data slice location information for each data slice to be processed, and jointly storing each data slice to be processed, the data slice identifier and the data slice location information in a memory corresponding to the distributed system; The generating of the corresponding data slice node allocation table includes: According to the obtained data slice node allocation result, in combination with the data slice identifier and the data slice location information, a corresponding data slice node allocation table is generated.

6. The method according to claim 5, characterized in that The method of pre-setting data processing and exporting is used to instruct each node to process and export data according to the data slice node allocation table, and obtain and export the data processing result, including: By using the pre-set data processing and export method and according to the data slice node allocation table, each node is instructed to process the data in the corresponding data slice to be processed according to the data slice location information to obtain the data slice processing result; According to the data slice identifier, the data slice processing results are sequentially combined and processed to obtain the data processing results corresponding to the data set to be processed; Obtain and export the data processing result according to a preset data export path.

7. The method according to claim 6, characterized in that Also includes: In the process of instructing each node to process the data in the corresponding data slice to be processed according to the data slice location information, the node status of each node is monitored in real time to obtain the node status real-time monitoring result; In the real-time monitoring results of node status, if there is any abnormal state node, the data in the to-be-processed data slice corresponding to the abnormal state node will be sent to other normal state nodes, and the node status monitoring information corresponding to the abnormal state node will be modified, and the data node allocation table will be updated.

8. A data processing and exporting device based on a distributed system, characterized in that: include: A metadata description information acquisition module, used to acquire a data set to be processed and metadata description information corresponding to the data set to be processed; A module for obtaining data slices to be processed, used for slicing the data set to be processed according to the metadata description information to obtain multiple data slices to be processed; The node registration processing capability information and node status monitoring information acquisition module is used to acquire the current distributed node cluster corresponding to the current distributed system, and acquire the node registration processing capability information and node status monitoring information of each node in the current distributed node cluster; The data slice node allocation table generation module is used to allocate and process each data slice to be processed according to the node registration processing capacity information and node status monitoring information through a pre-set distributed node management method, obtain the data slice node allocation result, and generate a corresponding data slice node allocation table; The data processing result determination and export module is used to instruct each node to perform data processing and export through a preset data processing and export method and according to the data slice node allocation table, and obtain and export the data processing result.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the distributed system-based data processing and export method according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the data processing and exporting method based on a distributed system as described in any one of claims 1 to 7 when executed.

Images