Data processing method, device, storage medium and electronic device

Abstract

The present application discloses a data processing method, device, storage medium and electronic device. The method includes: obtaining each data input from each data node; mapping each data into a corresponding data slice according to a mapping relationship; determining the priority of the storage area, wherein the storage area includes: a solid-state drive and a hard disk drive; preferentially inputting the data stored in the data slice into multiple buffer areas in the solid-state drive, and after the occupancy rate of the buffer area of ​​the solid-state drive reaches a preset threshold, writing the remaining data to the hard disk drive. The present application solves the technical problem of serious skew in storage data caused by the access data skew phenomenon generated by multi-user, multi-task and multi-priority access flows of big data in the process of cloud computing, as well as causing hot data competition and waste of cold data storage resources.

Description

Technical Field

[0001] The present application relates to the field of big data, and more specifically, to a data processing method, device, storage medium, and electronic device. Background Art

[0002] Related technologies produce computing nodes during calculations, and then encrypt and transmit them to the cloud platform for storage. However, there is still a problem that in the process of cloud computing, big data will cause access data skew for multi-user, multi-task and multi-priority access flows, resulting in serious skew in storage data based on unified management, thereby causing hot data competition and waste of cold data storage resources.

[0003] To address the above-mentioned problems, no effective solutions have been proposed so far. Summary of the Invention

[0004] The embodiments of the present application provide a data processing method, device, storage medium and electronic device to at least solve the technical problems of severe skew of storage data caused by the access data skew phenomenon generated by multi-user, multi-task and multi-priority access flows of big data in the process of cloud computing, as well as hot data competition and waste of cold data storage resources.

[0005] According to one aspect of an embodiment of the present application, a data processing method is provided, including: obtaining each data input by each data node; mapping each data into a corresponding data shard according to a mapping relationship; determining the priority of the storage area, wherein the storage area includes: a solid-state drive and a hard disk drive; preferentially inputting the data stored in the data shard into multiple buffer areas in the solid-state drive, and after the occupancy rate of the buffer area of ​​the solid-state drive reaches a preset threshold, writing the remaining data to the hard disk drive.

[0006] Optionally, each data is mapped and stored in a corresponding data shard according to the mapping relationship, including: determining the initial key-value pair corresponding to each data, mapping the initial key-value pair to a target binary group; and determining the data shard to which each data belongs according to the key value in the target binary group.

[0007] Optionally, the method further includes: determining access status of each data, dividing each data according to the access status, and classifying each data into hot data or cold data.

[0008] Optionally, after classifying each data as hot data or cold data, the method further includes: obtaining a global data copy load value, and when the task executed in the current period is a non-local task, determining the data block corresponding to the hot data, and automatically copying the data block from other nodes.

[0009] Optionally, after classifying each data as hot data or cold data, the method further includes: detecting the data block load stored on the data node at predetermined intervals, and obtaining the total number of data copies corresponding to the data node when the difference between the data block load and the normal load is less than a preset threshold; sending the erasure code to the data node when the total number of data copies is a predetermined number; receiving data information returned by the data block, and storing the data information separately in a cold data independent disk array.

[0010] Optionally, when the total number of data copies is not a predetermined number, the file in the data block on the data node is deleted and deletion information is reported, wherein the deletion information includes: the file name and location corresponding to the file.

[0011] Optionally, the access status of each data is determined, each data is divided according to the access status, and each data is classified as hot data or cold data, including: at least obtaining each file name and each access time corresponding to each data; determining the number of accesses corresponding to each file name, and determining that the data is hot data when the number of accesses is greater than a preset number of accesses; or determining that the data is hot data when the access time falls within a target time period; determining that the data is cold data when the number of accesses is less than a preset number of accesses; or determining that the data is cold data when the access time falls within the target time period.

[0012] According to another aspect of an embodiment of the present application, a data processing device is also provided, including: an acquisition module for acquiring various data transmitted by various data nodes; a mapping module for mapping various data into corresponding data slices according to the mapping relationship; a determination module for determining the priority of the storage area, wherein the storage area includes: a solid-state drive and a hard disk drive, and the data stored in the data slices are preferentially input into multiple buffer areas in the solid-state drive, and after the occupancy rate of the buffer area of ​​the solid-state drive reaches a preset threshold, the remaining data is written to the hard disk drive.

[0013] According to another aspect of an embodiment of the present application, a non-volatile storage medium is further provided, including: the storage medium includes a stored program, wherein when the program is running, the device where the storage medium is located is controlled to execute any data processing method.

[0014] According to another aspect of an embodiment of the present application, an electronic device is further provided, comprising: a processor; and a memory for storing processor-executable instructions; wherein the processor is configured to execute the instructions to implement any one of the data processing methods.

[0015] In an embodiment of the present application, a method of dividing data access heat is adopted, by obtaining each data transmitted by each data node; mapping each data into a corresponding data shard according to a mapping relationship; determining the priority of the storage area, wherein the storage area includes: a solid-state drive and a hard disk drive; the data stored in the data shard is preferentially input into multiple buffer areas in the solid-state drive, and after the occupancy rate of the buffer area of ​​the solid-state drive reaches a preset threshold, the remaining data is written to the hard disk drive, thereby achieving the purpose of reducing data redundancy in the big data cloud computing storage process, thereby achieving the technical effect of reducing hot data competition and avoiding waste of cold data storage resources, and further solving the serious skew of storage data caused by the access data skew phenomenon generated by multi-user, multi-task and multi-priority access flows of big data in the process of cloud computing, as well as the technical problems of hot data competition and waste of cold data storage resources. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0017] Figure 1 is a flow chart of a data processing method according to an embodiment of the present application;

[0018] Figure 2 is a flowchart of an optional data processing method according to an embodiment of the present application;

[0019] Figure 3 This is a data execution flow diagram of a data processing method according to an embodiment of the present application;

[0020] Figure 4 is a data flow diagram of a data processing method according to an embodiment of the present application;

[0021] Figure 5 is a schematic diagram of the structure of a device for a data processing method according to an embodiment of the present application;

[0022] Figure 6 is a schematic block diagram of an exemplary electronic device 600 according to an embodiment of the present application. DETAILED DESCRIPTION

[0023] In order to enable those skilled in the art to better understand the present invention, the following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of this application.

[0024] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in a sequence other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0025] According to an embodiment of the present application, an embodiment of a method for data processing is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0026] Figure 1 is a data processing method according to an embodiment of the present application, such as Figure 1 As shown, the method includes the following steps:

[0027] Step S102, obtaining each data input by each data node;

[0028] Step S104, storing each data mapping into the corresponding data shard according to the mapping relationship;

[0029] Step S106, determining the priority of the storage area, wherein the storage area includes: a solid state drive and a hard disk drive;

[0030] It should be noted that the solid-state drive has a higher priority than the hard disk drive.

[0031] Step S108: The data stored in the data slices are preferentially input into the plurality of buffer areas in the solid-state drive. After the occupancy rate of the buffer area of ​​the solid-state drive reaches a preset threshold, the remaining data is written into the hard disk drive.

[0032] It should be noted that the preset threshold may be 80%, that is, when the occupancy rate of the buffer area reaches 80%, the remaining data is written to the hard disk drive.

[0033] In an embodiment of the present application, a method of dividing data access heat is adopted, by obtaining each data transmitted by each data node; mapping each data into a corresponding data shard according to a mapping relationship; determining the priority of the storage area, wherein the storage area includes: a solid-state drive and a hard disk drive; the data stored in the data shard is preferentially input into multiple buffer areas in the solid-state drive, and after the occupancy rate of the buffer area of ​​the solid-state drive reaches a preset threshold, the remaining data is written to the hard disk drive, thereby achieving the purpose of reducing data redundancy in the big data cloud computing storage process, thereby achieving the technical effect of reducing hot data competition and avoiding waste of cold data storage resources, and further solving the serious skew of storage data caused by the access data skew phenomenon generated by multi-user, multi-task and multi-priority access flows of big data in the process of cloud computing, as well as the technical problems of hot data competition and waste of cold data storage resources.

[0034] In an exemplary embodiment of the present application, each data is mapped and stored in a corresponding data shard according to a mapping relationship, including: determining the initial key-value pair corresponding to each data, mapping the initial key-value pair to a target binary group; and determining the data shard to which each data belongs based on the key value in the target binary group.

[0035] Optionally, the method further includes: determining access status of each data, dividing each data according to the access status, and classifying each data into hot data or cold data.

[0036] It is understandable that data that is accessed more frequently is hot data, and data that is accessed less frequently is cold data.

[0037] In some optional embodiments of the present application, after classifying each data as hot data or cold data, the method also includes: obtaining a global data copy load value, and when the task executed in the current period is a non-local task, determining the data block corresponding to the hot data, and automatically copying the data block from other nodes.

[0038] In an exemplary embodiment of the present application, after classifying each data as hot data or cold data, the method further includes: detecting the data block load stored on the data node at predetermined intervals, and when the difference between the data block load and the normal load is less than a preset threshold, obtaining the total number of data copies corresponding to the data node; when the total number of data copies is a predetermined number, sending an erasure code to the data node; receiving data information returned by the data block, and storing the data information separately in a cold data independent disk array.

[0039] As an optional embodiment, when the total number of data copies is not a predetermined number, the file in the data block on the data node is deleted and the deletion information is reported, wherein the deletion information includes: the file name and location corresponding to the file.

[0040] It should be noted that the predetermined number of data copies is three copies, and copy operations cannot be performed on a single file. The storage location of the copies adopts a random allocation strategy.

[0041] In some optional embodiments of the present application, the access status of each data is determined, each data is divided according to the access status, and each data is classified as hot data or cold data, including: at least obtaining each file name and each access time corresponding to each data; determining the number of accesses corresponding to each file name, and determining that the data is hot data when the number of accesses is greater than a preset number of accesses; or, determining that the data is hot data when the access time falls within a target time period; determining that the data is cold data when the number of accesses is less than a preset number of accesses; or, determining that the data is cold data when the access time falls within the target time period.

[0042] It is understandable that when the access time belongs to a peak period, the data can be determined to be hot data; when the access time belongs to a non-peak period, the data can be determined to be cold data.

[0043] In order to facilitate those skilled in the art to better understand the technical solution of the present application, a specific embodiment is now described.

[0044] Figure 2 is a flow chart of an optional data processing method according to an embodiment of the present application, such as Figure 2 As shown, the process mainly includes the following steps:

[0045] (1) Partition the acquired data content according to data requirements and establish multiple databases based on the partitioned content;

[0046] (2) One or more input data nodes are stored in data shards. The data shards are processed and partitioned using mapping technology. The data is then sent to multiple buffer areas in the solid-state drive. When the buffer area is full, the backend will write the data in the buffer to the hard drive.

[0047] (3) The information collection module collects data access logs of upper-level nodes in the system to provide information for dynamic data partitioning. The dynamic hot and cold replica distinction module is used to dynamically divide data access heat, and the dynamic replica storage module manages and maintains the number of replicas.

[0048] It should be noted that the default number of copies of all files in the dynamic hot and cold copy distinction module is three. Copy operations cannot be performed on individual files. The storage location of the copies adopts a completely random allocation strategy, and the dynamic hot and cold copy distinction module adopts unified management for data storage and data access.

[0049] It should be noted that the dynamic copy storage module can mark and distinguish the hot and cold data according to the access status of the data. The data copies of the dynamic copy storage module are completely dynamic. The dynamic copy storage module adopts a feedback adjustment mechanism, which mainly includes a zero data copy copy increase mechanism and a data copy automatic decay mechanism to change the number of data blocks.

[0050] It can be understood that the zero-data copy replica increase mechanism is based on the logging module and can obtain the global data copy load value. If a non-local task is executed and its corresponding data block is hot data, when the data mapping is completed, the mapping task will automatically copy the data block from other nodes. The zero-data copy replica increase mechanism occurs after the data block copy is completed. The conventional mapping task will discard the mapping input data as a temporary file, while the zero-data copy replica increase mechanism will persist the data locally and report it to, thereby achieving visibility of the data block. Since the zero-data copy replica increase mechanism uses the execution process of the remote mapping task to perform the increase operation, the additional overhead it introduces is only the tiny traffic for reporting the metadata of the data block. Therefore, the zero-data copy replica increase mechanism can achieve replica increase with completely zero network overhead.

[0051] The automatic data copy decay mechanism is based on the calculation of the data block load. It will regularly scan the data block load stored on the entire node. When it is found that the data block load is significantly lower than the normal load value, the dynamic copy storage module will give priority to access and obtain the total number of copies. If the number of copies is not three, the corresponding file of the data block will be directly deleted and reported to achieve global visibility of data deletion. If the number of copies is equal to three, it will be handed over to the cold data independent disk array module for processing.

[0052] (4) For rarely accessed decaying data, a cold data independent disk array module is used to store additional data blocks for reliability;

[0053] It should be noted that the cold data independent disk array module uses a delayed loading mode to store data blocks. When the number of current copies of the dynamic hot and cold replica distinction module is three, the erasure code calculation operation will be sent to the data node. After receiving this information, the data node submits the information of the data block, and the cold data independent disk array module performs data reliability storage.

[0054] It is understandable that the RAID storage of data blocks in the cold data independent disk array module adopts a delayed loading mode. For a file data block, the master node will periodically summarize the locations of all its replicas. If the number of available replicas is less than three, it will automatically add a replica to ensure data reliability. In the dynamic hot and cold replica differentiation module, when the master node receives a message indicating the end of the data block replica lifecycle, it will monitor the number of replicas of the data block. If it finds that the current number of all replicas is three, it will issue an erasure code calculation operation to the data node. After receiving this message, the data node will not immediately delete the corresponding replica. Instead, it will submit the data block information, including <data block file name, data block segment ID number, and original data block data>, to the cold data independent disk array module for data reliability storage. After the data block completes storage in the cold data independent disk array module, the data node will delete the data block whose current lifecycle has ended and return this information to the master node, completing the entire data block storage process.

[0055] (5) All data in the data shards are written, and the mapping input is completed. There are multiple temporary buffer files in the solid-state drive. The data is strictly sorted and merged according to the key value by the reorganization end to form an intermediate data file containing multiple partitions and stored in the hard disk drive;

[0056] (6) During the file transfer process, the reorganization end globally merges the files, aggregates the key-value pairs with the same key value into a key group indexed by the key value, and transmits the key group content to the reduction end for application.

[0057] It is easy to notice that this application has the following beneficial effects by dividing data access popularity:

[0058] (1) The solution of the present application adopts a dynamic storage method, which combines a completely dynamic copy mode and an independent redundant disk array strategy to store data. Compared with the static solution, the dynamic copy can efficiently adapt to the changes in upper-level file access, thereby providing adaptive data storage. For hot data, dynamically increasing the number of copies can improve the availability of data under concurrency, reduce the generation of non-local tasks, reduce network transmission overhead, and alleviate the load imbalance of nodes, thereby improving the overall performance of the system. For cold data, compared with static independent redundant disk array operations, the dynamic strategy adopted by DHS can timely reduce the number of copies according to changes in data access. From the perspective of the system as a whole, it can save data storage costs.

[0059] (2) The present application solution adopts a dynamic hot and cold copy distinction module. The dynamic hot and cold copy distinction module adopts a dynamic copy mode. The load of the data block actually depends on the number of backups of the current data block. The access load of files in the same access state will change with the change of the number of copies. The more data backups there are, the more data blocks can share the pressure of upper-level access, so the load is lower. The fewer data backups, the opposite is true. For a file, since its upper-level access depends on the user, the dynamic hot and cold copy distinction module cannot intervene. Therefore, the dynamic hot and cold copy distinction module uses the concept of data block load to achieve the ultimate goal of adapting to upper-level access by adjusting the number of copies.

[0060] (3) The present application scheme adopts a cold data independent disk array module. For big data cloud computing, the core mechanism depends on the load abstraction and calculation of the data block. The load of the data block directly determines the number of copies of the data block, the disk load, and the node load. The disk load is the core parameter of the multi-disk scheduler. The node load will affect the priority of task scheduling and further affect the specific location of the added copies. The use of a cold data independent disk array module will change the task load of the node, thereby solving the redundancy problem caused by data skew in big data cloud computing, reducing redundancy to the minimum value, reducing the load of the big data cloud computing server, and improving the speed of big data cloud computing.

[0061] Figure 3 FIG. 1 is a data execution flow diagram of a data processing method according to an embodiment of the present application. Figure 3 As shown, the process mainly includes the following steps:

[0062] (1) Multiple databases are assigned to multiple servers for network interconnection, and each data shard is analyzed one by one during the data analysis process using mapping technology;

[0063] (2) The information collection module needs to record the following data for each access: the file name of the access, the node location of the data block after the file is divided, and the time of the access;

[0064] It should be noted that the format of the access information of the collected files is <file name, list <access time>>. The access information is used to divide the file heat. The mapping relationship between the file name and the node position of the file's split data block is used for heat calculation. The node heat is calculated based on the file data block distribution and the file heat, thereby supporting the subsequent task scheduler to balance the node load.

[0065] (3) The input of the mapping task is usually text data. The initial key-value pairs are <RAW data, line number>. Through the mapper, one or more key-value pairs of <RAW data, line number> are remapped into meaningful <Key, value> binary tuples;

[0066] It should be noted that the output results of the mapping will be partitioned and then sent to a buffer in the solid-state drive. Whenever the buffer is about to be filled 80%, the background process will write the data in the current buffer to the hard disk drive. When all the mapping inputs are completed, there may be multiple temporary buffer files in the hard disk drive, and these buffer files need to be merged. During the merging process, it is ensured that the data within each partition of the final merged file is strictly sorted according to the key values.

[0067] It can be understood that in order to ensure the speed of processing massive data, all the key-value pairs of the mapping output are strictly sorted in ascending order according to the key values. The advantage of strict ascending order is that it is convenient to quickly search for a certain key-value pair in the reduced data, improving the user's quick query of the result key-value pairs.

[0068] (4) When there are multiple reducers during the processing of data by the mapper, it is necessary to send a type of mapped data to the partitions of the corresponding reducer tasks. Each partition will ensure that the data therein is mapped to a unique reducer task, and the key-value pairs output by the mapping task will belong to a unique partition according to the key values;

[0069] (5) Reorganization copies the data that exceeds the available storage space of the solid-state drive to the hard disk drive as a temporary file, and the results of the reduction are organized in the form of key-value pairs and written to the server side.

[0070] It can be understood that the copy phase will cause the reducer to receive a large number of mapping results. Since they are split among multiple different files, after all the copies are completed, these files need to be globally merged to generate the final reduction input data. For the key-value pairs with the same key values, they are aggregated to form a value group indexed by the key values. At this time, the reorganization end has completed the transmission process of the mapping output results to the reducer end. <00​​​​​​​​​As shown in the solid-state drive partitions A, B, C, and D, each partition will ensure that the data in it is mapped to a unique reduction task, and the key-value pairs output by the mapping task will be attributed to a unique partition according to the key value. The reorganization end will copy the data that exceeds the available solid-state drive storage space to the hard disk drive as a temporary file, for example, Figure 4 Temporary files A, B, C and D in the file;

[0073] (2) During the copying phase, the reducer will receive a large number of mapping results. Since these results are divided into multiple different files, these files need to be globally merged after all the copies are completed to generate the reduced input data.

[0074] (3) Input the reduction input data into the reduction task.

[0075] Figure 5 is a schematic diagram of a device structure of a data processing method according to an embodiment of the present application, such as Figure 5 As shown, the device includes:

[0076] The acquisition module 50 is used to acquire each data input by each data node;

[0077] A mapping module 52 is used to store each data mapping into a corresponding data shard according to the mapping relationship;

[0078] The determination module 54 is used to determine the priority of the storage area, wherein the storage area includes: a solid-state drive and a hard disk drive, and the data stored in the data slices are preferentially input into the multiple buffer areas in the solid-state drive. After the occupancy rate of the buffer area of ​​the solid-state drive reaches a preset threshold, the remaining data is written to the hard disk drive.

[0079] In the device, an acquisition module 50 is used to acquire each data input by each data node; a mapping module 52 is used to map each data into a corresponding data slice according to a mapping relationship; a determination module 54 is used to determine the priority of the storage area, wherein the storage area includes: a solid-state drive and a hard disk drive, and the data stored in the data slice is preferentially input into multiple buffer areas in the solid-state drive. After the occupancy rate of the buffer area of ​​the solid-state drive reaches a preset threshold, the remaining data is written to the hard disk drive, thereby achieving the purpose of reducing data redundancy in the big data cloud computing storage process, thereby achieving the technical effect of reducing hot data competition and avoiding waste of cold data storage resources, and further solving the serious skew of storage data caused by the access data skew phenomenon generated by multi-user, multi-task and multi-priority access flows in the process of big data cloud computing, as well as the technical problems of hot data competition and waste of cold data storage resources.

[0080] According to another aspect of an embodiment of the present application, a non-volatile storage medium is further provided. The non-volatile storage medium includes a stored program, wherein when the program is running, the device where the non-volatile storage medium is located is controlled to execute any data processing method.

[0081] Specifically, the above storage medium is used to store program instructions for the following functions to implement the following functions:

[0082] Acquire each data input from each data node; store each data map into the corresponding data shard according to the mapping relationship; determine the priority of the storage area, where the storage area includes: solid-state drive and hard disk drive; preferentially input the data stored in the data shard into multiple buffer areas in the solid-state drive, and after the occupancy rate of the buffer area of ​​the solid-state drive reaches a preset threshold, write the remaining data to the hard disk drive.

[0083] Alternatively, in this embodiment, the storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or any suitable combination thereof. More specific examples of the storage medium may include an electrical connection based on one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof.

[0084] In an exemplary embodiment of the present application, a computer program product is further provided, including a computer program, which implements any of the above-mentioned data processing methods when executed by a processor.

[0085] Optionally, the computer program may implement the following steps when executed by a processor:

[0086] Acquire each data input from each data node; store each data map into the corresponding data shard according to the mapping relationship; determine the priority of the storage area, where the storage area includes: solid-state drive and hard disk drive; preferentially input the data stored in the data shard into multiple buffer areas in the solid-state drive, and after the occupancy rate of the buffer area of ​​the solid-state drive reaches a preset threshold, write the remaining data to the hard disk drive.

[0087] According to an embodiment of the present application, an electronic device is provided, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute any one of the above-mentioned data processing methods.

[0088] Optionally, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor, and the input / output device is connected to the processor.

[0089] Figure 6 6 is a schematic block diagram of an example electronic device 600 according to an embodiment of the present application. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present application described and / or claimed herein.

[0090] like Figure 6 As shown, the device 600 includes a computing unit 601, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 602 or a computer program loaded from a storage unit 608 into a random access memory (RAM) 603. Various programs and data required for the operation of the device 600 can also be stored in the RAM 603. The computing unit 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0091] Multiple components in device 600 are connected to I / O interface 605, including: input unit 606, such as a keyboard, mouse, etc.; output unit 607, such as various types of displays, speakers, etc.; storage unit 608, such as a magnetic disk, optical disk, etc.; and communication unit 609, such as a network card, modem, wireless communication transceiver, etc. Communication unit 609 allows device 600 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0092] The computing unit 601 can be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 601 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units that run machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 601 performs the various methods and processes described above, such as the data processing method. For example, in some embodiments, the data processing method can be implemented as a computer software program that is tangibly contained in a machine-readable medium, such as the storage unit 608. In some embodiments, part or all of the computer program can be loaded and / or installed on the device 600 via the ROM 602 and / or the communication unit 609. When the computer program is loaded into the RAM 603 and executed by the computing unit 601, one or more steps of the data processing method described above can be performed. Alternatively, in other embodiments, the computing unit 601 can be configured to perform the data processing method by any other appropriate means (e.g., by means of firmware).

[0093] Various embodiments of the systems and techniques described herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), system-on-chip systems (SOCs), programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include being implemented in one or more computer programs that are executable and / or interpreted on a programmable system that includes at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0094] The program code for implementing the methods of the present application can be written in any combination of one or more programming languages. Such program code can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the program code is executed by the processor or controller, the functions / operations specified in the flow charts and / or block diagrams are implemented. The program code can be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0095] In the context of the present application, a machine-readable medium can be a tangible medium that can contain or store a program for use by an instruction execution system, device or equipment or used in combination with an instruction execution system, device or equipment. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared or semiconductor system, device or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium can include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0096] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0097] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.

[0098] A computer system may include a client and a server. The client and server are generally remote from each other and typically interact through a communication network. The client-server relationship arises through computer programs running on the respective computers and having a client-server relationship with each other. The server may be a cloud server, a server in a distributed system, or a server integrated with a blockchain.

[0099] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0100] In the above embodiments of the present application, the description of each embodiment has its own focus. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.

[0101] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only exemplary. For example, the division of the units can be a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0102] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0103] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0104] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for enabling a computer device (which can be a personal computer, a server or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk.

[0105] The above is only a preferred embodiment of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A data processing method, characterized in that: include: Get each data passed in by each data node; Storing each data mapping into a corresponding data shard according to the mapping relationship; Determining a priority of a storage area, wherein the storage area includes: a solid state drive and a hard disk drive; inputting the data stored in the data slices into a plurality of buffer areas in the solid-state drive in a preferential manner, and writing the remaining data into the hard disk drive after the occupancy rate of the buffer areas in the solid-state drive reaches a preset threshold; The method further includes: determining the access status of each data, dividing the each data according to the access status, and classifying the each data as hot data or cold data; after classifying the each data as hot data or cold data, the method further includes: detecting the data block load stored on the data node at predetermined intervals, and when the difference between the data block load and the normal load is less than a preset threshold, obtaining the total number of data copies corresponding to the data node; when the total number of data copies is a predetermined number, issuing an erasure code to the data node; receiving data information returned by the data block, and storing the data information separately in a cold data independent disk array.

2. The method according to claim 1, characterized in that Storing each data mapping into the corresponding data shard according to the mapping relationship includes: Determine the initial key-value pairs corresponding to the respective data, and map the initial key-value pairs into target binary groups; The data shard to which each data belongs is determined according to the key value in the target binary group.

3. The method according to claim 1, characterized in that After classifying the data into hot data or cold data, the method further includes: Obtain a global data replica load value, and when the task executed in the current period is a non-local task, determine the data block corresponding to the hot data, and automatically copy the data block from other nodes.

4. The method according to claim 1, wherein When the total number of data copies is not the predetermined number, the file in the data block on the data node is deleted and deletion information is reported, wherein the deletion information includes: the file name and location corresponding to the file.

5. The method according to claim 1, wherein Determining access status of each data item, dividing each data item according to the access status, and classifying each data item as hot data or cold data, including: At least obtain the file names and access times corresponding to each data; Determining the number of accesses corresponding to each file name, and determining that the data is hot data when the number of accesses is greater than a preset number of accesses; or determining that the data is hot data when the access time falls within a target time period; If the number of accesses is less than a preset number of accesses, the data is determined to be cold data; or, if the access time falls within a target period, the data is determined to be cold data.

6. A data processing device, characterized in that: include: The acquisition module is used to obtain the data transmitted by each data node; A mapping module, configured to store each data mapping into a corresponding data shard according to a mapping relationship; a determination module, configured to determine a priority of a storage area, wherein the storage area includes a solid-state drive and a hard disk drive, preferentially inputting the data stored in the data slices into a plurality of buffer areas in the solid-state drive, and writing the remaining data to the hard disk drive after an occupancy rate of the buffer area of ​​the solid-state drive reaches a preset threshold; The data processing device is further used to: determine the access status of each data, divide the each data according to the access status, and classify the each data as hot data or cold data; after classifying the each data as hot data or cold data, the data processing device is further used to: detect the data block load stored on the data node at predetermined intervals, and when the difference between the data block load and the normal load is less than a preset threshold, obtain the total number of data copies corresponding to the data node; when the total number of data copies is a predetermined number, send an erasure code to the data node; receive data information returned by the data block, and store the data information separately in a cold data independent disk array.

7. A non-volatile storage medium, characterized in that: The storage medium includes a stored program, wherein when the program is run, the device where the storage medium is located is controlled to execute the data processing method according to any one of claims 1 to 5.

8. An electronic device, characterized in that: include: processor; a memory for storing instructions executable by the processor; The processor is configured to execute the instructions to implement the data processing method according to any one of claims 1 to 5.

Images