Data storage method, system, device, storage medium and program product

Abstract

The application discloses a data storage method, system and device, a storage medium and a program product, and belongs to the field of computer data processing. The method comprises the following steps: receiving distributed storage data sent by an execution node, wherein the distributed storage data is data stored by the execution node in a distributed storage manner; transferring the distributed storage data to a local disk for storage, wherein the local disk is a local storage area corresponding to the current device; in response to the local disk meeting data uploading conditions, uploading local storage data in the local disk to a cloud storage system according to a preset uploading strategy, wherein the preset uploading strategy is a strategy determined based on the average file size in the local disk; and in response to the uploading of the local storage data being completed, cleaning the local storage data from the local disk. The method achieves the effect of cooperation between local storage and the cloud storage system, and solves the problem that the large data workload cannot be effectively deployed in an offline hybrid cluster due to the fact that storage and calculation cannot be separated.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of computer data processing, and in particular to a data storage method, system, device, storage medium and program product. BACKGROUND

[0002] With the advent of the era of big data, due to the limited performance of a single computer device, it is not possible to complete a relatively complex computing task, so the concept of a distributed computing framework is introduced. The distributed computing framework can solve such huge computer problems, use servers to form a computing cluster and provide a parallel computing software framework, and implement complex operations such as communication between servers, load balancing, task computing and processing, and task storage.

[0003] In related technologies, a Shuffle service node in a distributed computing framework is generally used to directly store data to a local disk or directly store data to a distributed file system (Hadoop Distributed FileSystem, HDFS).

[0004] However, whether uploaded directly to a local disk or HDFS, the data is stored in each node by a Map node, which cannot separate data computing and storage, resulting in the problem that the data workload cannot be effectively deployed in an offline hybrid cluster. SUMMARY

[0005] The embodiments of the present application provide a data storage method, system, device, storage medium and program product, which are used to improve data processing efficiency. The technical solution is as follows:

[0006] On the one hand, a data storage method is provided, which is applied to a transfer service node in a distributed computing framework, and the method comprises:

[0007] receiving distributed storage data sent by an execution node, the distributed storage data being data stored by the execution node in a distributed storage manner;

[0008] transferring the distributed storage data to a local disk for storage, the local disk being a local storage area corresponding to the current device;

[0009] in response to the local disk meeting data upload conditions, uploading local storage data in the local disk to a cloud storage system according to a preset upload strategy, the preset upload strategy being a strategy determined based on an average file size in the local disk;

[0010] in response to the local storage data being uploaded, cleaning the local storage data from the local disk.

[0011] In an aspect, a data storage system is provided, applied in a distributed computing framework, the system comprising:

[0012] an execution node configured to perform task processing on candidate data to obtain distributed storage data, and send the distributed storage data to a transit service node, the distributed storage data being data stored in a distributed storage manner;

[0013] the transit service node configured to receive the distributed storage data, and transfer the distributed storage data to a local disk for storage, the local disk being a local storage area corresponding to a current device, and in response to the local disk meeting a data uploading condition, upload local storage data in the local disk to a cloud storage system according to a preset uploading strategy, the preset uploading strategy being a strategy determined based on an average file size in the local disk;

[0014] the cloud storage system configured to receive the local storage data uploaded by the local disk;

[0015] the transit service node is further configured to, in response to the local storage data being uploaded completely, clear the local storage data from the local disk.

[0016] In an aspect, a data storage apparatus is provided, the apparatus comprising:

[0017] a receiving module configured to receive distributed storage data sent by an execution node, the distributed storage data being data stored by the execution node in a distributed storage manner;

[0018] a storage module configured to transfer the distributed storage data to a local disk for storage, the local disk being a local storage area corresponding to a current device;

[0019] an uploading module configured to, in response to the local disk meeting a data uploading condition, upload local storage data in the local disk to a cloud storage system according to a preset uploading strategy, the preset uploading strategy being a strategy determined based on an average file size in the local disk;

[0020] a clearing module configured to, in response to the data being uploaded completely, clear the data from the local disk.

[0021] In an aspect, a computer device is provided, the computer device comprising a processor and a memory. The memory stores at least one instruction, at least one program, a code set or an instruction set, which are loaded and executed by the processor to implement the data storage method as described above.

[0022] In an aspect, a computer readable storage medium is provided, and the storage medium stores at least one instruction, at least one program, a code set or an instruction set, which are loaded and executed by the processor to implement the data storage method as described above.

[0023] In an aspect, a computer program product is provided, and the computer program product includes a computer program / instruction stored in a computer readable storage medium, and the computer program / instruction is read by the processor of a computer device, and the processor executes the computer program / instruction, so that the computer device is executed to implement the data storage method as described above.

[0024] The technical scheme provided by the embodiments of the present application has the beneficial effects that:

[0025] The data transmission process is established between the local disk and the cloud storage system, and the distributed storage data stored in the local disk is uploaded to the cloud storage system according to the preset uploading strategy, so as to realize the effect of cooperating with the local storage and the cloud storage system, solve the problem that the large data work load cannot be effectively deployed in the offline hybrid cluster due to the separation of storage and computing, and improve the data processing efficiency to a certain extent. BRIEF DESCRIPTION OF DRAWINGS

[0026] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0027] Figure 1 is a schematic diagram of a data storage system provided by the embodiments of the present application;

[0028] Figure 2 is a schematic diagram of a multi-level storage management unit based on Figure 1 the present application;

[0029] Figure 3 is a flowchart of a data storage method provided by the embodiments of the present application;

[0030] Figure 4 is a schematic diagram of a data uploading method based on Figure 3 the present application;

[0031] Figure 5 is a schematic diagram of another data uploading method based on Figure 3 the present application;

[0032] Figure 6 is a flow chart of a data storage method provided by another embodiment of the present application;

[0033] Figure 7 is based on Figure 6 is a flow chart of an active uploading method;

[0034] Figure 8 is a flow chart of a data storage method provided by another embodiment of the present application;

[0035] Figure 9 is a structural block diagram of a data storage device provided by an embodiment of the present application;

[0036] Figure 10 is a structural block diagram of a data storage device provided by another embodiment of the present application;

[0037] Figure 11 is a structural schematic diagram of a server provided by an exemplary embodiment of the present application;

[0038] Figure 12 is a server framework diagram provided by an embodiment of the present application. DETAILED DESCRIPTION

[0039] In order to make the purpose, technical solutions and advantages of the present application clearer, the embodiments of the present application will be further described in detail below with reference to the drawings.

[0040] First, the terms involved in the embodiments of the present application are introduced:

[0041] Distributed computing framework: the computing task is distributed to multiple servers, and each server undertakes a part of the computing task and the data storage task. It is necessary to complete the acquisition of data corresponding to each task, the merging of the computing result and the rollback operation of the error computing in the distributed computing process.

[0042] The embodiments of the present application mainly take the MapReduce distributed computing framework as an example for introduction. The MapReduce distributed computing framework is used for batch-oriented distributed computing framework. The MapReduce distributed computing framework is mainly divided into four stages, namely, Split node, Map node, Shuffle node and Reduce node.

[0043] The Split node mainly divides the input metadata into data blocks of a preset length, and hands the divided data blocks to the Map node for task processing. The number of Split determines the number of Map node tasks.

[0044] The Map node mainly maps the data blocks sliced by the Split node to obtain an intermediate calculation result.

[0045] The Shuffle node is an intermediate link between the Split node and the Map node, and mainly performs Partition, Sort, Spill, Merge, Fetch and the like. The Partition determines which partition each piece of data output by the Map node task will be included in, and which Reduce node will be given for task processing. The number of Reduce node tasks determines the number of partitions.

[0046] The specific execution process is that the metadata is first sliced by the Split node to obtain a Block block, and the corresponding Map node is distributed on each Block to perform intermediate result calculation, and the calculation result is saved in the form of key-value. At this time, the Shuffle node Hashes the Key value by modulo, and then forms a corresponding file according to the number of Reduce nodes. The Reduce node reads the corresponding file from the Map node, performs merging after reading is completed, and stores the merged file into the corresponding local disk. Then, the local storage data is uploaded to the cloud storage system via the local disk. The cloud storage system in the embodiment of the application includes at least one of HDFS, COS and S3. In the embodiment of the application, the transfer service node corresponds to the Shuffle node, and the execution node corresponds to the Map node.

[0047] Distributed storage data: data obtained after the execution node performs task processing on the metadata (the metadata corresponds to candidate data). The optional task processing includes but is not limited to data storage, data calculation, data merging, data filtering, data identification and data division.

[0048] In the related art, when a distributed computing framework is used to process data, the distributed storage data is usually directly stored in a local disk or directly written into HDFS. If the distributed storage data is directly stored into the corresponding local disk by the transfer service node (Shuffle service node), when facing some clusters with small local disks (for example, computing clusters and online service clusters), the capacity limit is easily reached, resulting in a high data overflow rate. If the distributed storage data is directly written into HDFS, when a large number of small Shuffle jobs (which can be understood as small tasks of distributed storage data) are encountered, a large number of small files need to be frequently written into HDFS, which increases the work load of the transfer service node (Shuffle service node) and causes the performance of the transfer service node (Shuffle service node) to decrease.

[0049] The embodiment of the application introduces a multi-level storage architecture of a local disk and a cloud storage system, cooperates local storage and cloud storage, provides a multi-level cache management manner, solves a cluster with limited local disk capacity, and can also reduce the work load of a transfer service node (Shuffle service node) and improve the performance of the transfer service node (Shuffle service node).

[0050] In the embodiment of the application, the cloud technology can be used to process the data storage system / method, wherein the cloud technology refers to a kind of hosting technology that unifies a series of resources such as hardware, software, network in a wide area network or in a wide area network, realizes the calculation, storage, processing and sharing of data.

[0051] The cloud technology is a general term of network technology, information technology, integration technology, management platform technology, application technology and the like applied based on the cloud computing business model, can form a resource pool, and is used on demand, flexible and convenient. The background service of the technical network system needs a large amount of calculation and storage resources, such as video websites, picture websites and more portal websites. With the high development and application of the Internet industry, in the future, every item may have its own identification mark, and needs to be transmitted to the background system for logical processing. Different degrees of data will be classified and processed, and various industry data interfaces need strong system backup support, which can only be realized through cloud computing.

[0052] The cloud technology includes cloud computing, cloud storage, database, and big data. The cloud computing refers to an IT infrastructure delivery and use mode, and refers to obtaining required resources in a demand-oriented and easily expandable manner through a network. The broad sense of cloud computing refers to a service delivery and use mode, and refers to obtaining required services in a demand-oriented and easily expandable manner through a network. Such services can be IT and software, Internet related, or other services. Cloud computing is the product of the development and integration of traditional computer and network technologies such as grid computing, distributed computing, parallel computing, utility computing, network storage technologies, virtualization, load balancing, etc. With the development of the Internet, real-time data flow, and diversified connection devices, and the promotion of search services, social networks, mobile commerce, and open collaboration, cloud computing has developed rapidly. Unlike previous parallel distributed computing, the generation of cloud computing will revolutionize the entire Internet model and enterprise management model from the concept.

[0053] Cloud storage is a new concept extended and developed on the basis of cloud computing concept. Distributed cloud storage system (hereinafter referred to as storage system) refers to a storage system that collects a large number of storage devices (storage device is also referred to as storage node) of various types through cluster application, grid technology and distributed storage file system, and cooperatively provides data storage and business access functions through application software or application interface. At present, the storage method of the storage system is as follows: a logical volume is created, and a physical storage space is allocated to each logical volume when the logical volume is created. The physical storage space may be a disk of a storage device or a disk group of several storage devices. A client stores data on a certain logical volume, that is, stores data on a file system. The file system divides the data into many parts, each part being an object. The object contains not only data but also additional information such as data ID (ID entity). The file system writes each object into the physical storage space of the logical volume, and records the storage location information of each object. Thus, when the client requests to access data, the file system can enable the client to access the data according to the storage location information of each object. The process of allocating the physical storage space to the logical volume is as follows: according to the capacity estimation of the object stored in the logical volume (the estimation often has a large margin relative to the actual capacity of the object to be stored) and the group of RAID (Redundant Array of Independent Disk), the physical storage space is divided into sections in advance. A logical volume can be understood as a section, so that the logical volume is allocated with the physical storage space.

[0054] A database, simply put, can be viewed as an electronic filing cabinet—a place to store electronic files, where users can perform operations such as adding, querying, updating, and deleting data. A "database" is a collection of data stored together in a certain way, shared by multiple users, with minimal redundancy, and independent of application programs. A Database Management System (DBMS) is a computer software system designed to manage databases, generally possessing basic functions such as storage, retrieval, security, and backup. DBMSs can be classified according to the database model they support, such as relational or XML (Extensible Markup Language); or according to the type of computer they support, such as server clusters or mobile phones; or according to the query language used, such as SQL (Structured Query Language) or XQuery; or according to performance priorities, such as maximum scale or maximum operating speed; or other classification methods. Regardless of the classification method used, some DBMSs can cross categories, for example, supporting multiple query languages ​​simultaneously.

[0055] Big data refers to data sets that cannot be captured, managed, and processed within a certain timeframe using conventional software tools. It represents massive, rapidly growing, and diverse information assets that require new processing models to achieve stronger decision-making, insightful discovery, and process optimization capabilities. With the advent of the cloud era, big data has attracted increasing attention. Big data requires specialized technologies to effectively process large amounts of data within a tolerable timeframe. Technologies suitable for big data include massively parallel processing databases, data mining, distributed file systems, distributed databases, cloud computing platforms, the internet, and scalable storage systems.

[0056] Secondly, please see Figure 1 It shows a schematic diagram of a data storage system involved in an embodiment of this application, combined with Figure 1 The data storage method provided in this application embodiment will be described in detail. The data storage system 100 includes an execution node 10, a relay service node 11, a local disk 12 corresponding to the relay service node 11, a cloud storage system 14, a registration node 15, a scheduling node 16, and a processing node (…). Figure 1 (Not shown in the image), the following is a detailed description of the data interaction process between the nodes in the data storage system 100:

[0057] The registration node 15 receives registration information from the transit service node 11, which includes at least one of identity document (ID) of the transit server, partition information, quantity information, and heartbeat information. The ID information is used to uniquely identify the transit server, the partition information is used to indicate the management service area corresponding to the transit service node 11, the quantity information is used to indicate the number of tasks corresponding to the transit service node 11 or the storage quantity (capacity) corresponding to the transit service node 11, and the heartbeat information is used to indicate the state of the transit service node 11 to the registration node 15 at a fixed frequency. The heartbeat information is used to indicate the working state of the transit service node 11, which includes a normal working state and an abnormal state. Optionally, the transit service node 11 sends the registration information to the registration node 15 while maintaining the sending of the heartbeat information to the registration node 15, and the heartbeat information carries the state information of the transit service node 11. Alternatively, the registration node 15 receives the registration information and the heartbeat information sent by the transit service node 11, and determines the state information of the transit service node 11 according to the heartbeat information.

[0058] Optionally, the registration node 15 receives registration information sent by at least one transit server, and classifies the at least one transit server according to the registration information in a preset manner, which includes at least one of partition information to which the transit server belongs, quantity information of the transit server, and working state. For example, the registration node receives registration information corresponding to the transit service node a, the transit service node b, and the transit service node c, wherein the transit service node a corresponds to partition 1, the transit service node b corresponds to partition 1, and the transit service node c corresponds to partition 3. The registration node classifies the transit service nodes a-c according to the partition, and obtains (partition 1: transit service node a, transit service node b), (partition 2: transit service node c).

[0059] After each transit service node sends the registration information to the registration node 15 to complete the registration, the task of storing candidate data is performed. The specific process is as follows: the scheduling node 16 sends a list acquisition request to the registration node 15, which is used to indicate a list of transit service nodes for executing the candidate data; the scheduling node 16 distributes the obtained transit service node list to the corresponding execution node 10 according to the partition information; and the execution node 10 sends distributed storage data (which is determined based on the candidate data) to the allocated transit service node according to the partition information.

[0060] The transit service node 11 aggregates the distributed storage data according to the partition information granularity, and stores the distributed storage data to the local disk 12. In response to the local disk 12 meeting the data uploading condition, the local storage data in the local disk 12 is uploaded to the cloud storage system 14 according to a preset uploading strategy, wherein the preset uploading strategy is a strategy determined based on the average file size in the local disk 12.

[0061] The transit service node 11 further includes a buffer 17 for receiving and storing the distributed storage data sent from the execution node 10. Optionally, when the storage capacity of the buffer 17 reaches a preset buffer capacity threshold, the distributed storage data stored in the buffer 17 is transferred to the local disk 12 for storage.

[0062] Optionally, after the local storage data is uploaded, the transit service node 11 clears the local storage data from the local disk 12.

[0063] The processing node receives an acquisition request, which is used to indicate the acquisition of the stored distributed storage data. Based on the acquisition request, the processing node determines the partition information corresponding to the distributed storage data to be acquired, and requests the distributed storage data to be acquired from the assigned transit service node 11 according to the determined partition information. The specific process of acquiring the distributed storage data to be acquired is that the processing node searches the local disk 12 corresponding to the assigned transit service node 11 for the distributed storage data to be acquired. If the distributed storage data to be acquired is found, the corresponding distributed storage data is directly acquired from the local disk 12. If the distributed storage data to be acquired is not found, the processing node acquires the distributed storage data to be acquired from the cloud storage system 14 and feeds back to the terminal device corresponding to the terminal device sending the acquisition request.

[0064] It is worth noting that the above system can be embedded in an existing distributed computing framework, or can form a new distributed computing framework for application. The application does not limit the application scenario of the above system. In the embodiments of the present application, the above system is applied to the MapReduce distributed computing framework for data storage.

[0065] In an optional embodiment, the specific process of the above system is as follows: Figure 2 , Figure 2 A multi-level cache management unit provided by an embodiment of the present application is shown, which is used to reflect the management process of data in the distributed computing framework formed by the above system, and includes a metadata management unit 21, a write control unit 22, an uploading unit 23, and a cleaning unit 24.

[0066] The metadata management unit 21 stores first directory information corresponding to the distributed storage data, first partition information corresponding to the execution nodes, second partition information corresponding to the uploaded distributed storage data, and a reading voucher. The first directory information includes the directory dimension corresponding to the distributed storage data, the size information and the quantity information of the files under the directory corresponding to the distributed storage data. Optionally, the size information and the quantity information can be calculated according to a single file or all files. For example, the distributed storage data is stored under the index directory, the index directory corresponds to files with a total size of 1G and a total quantity of 20, the first partition information includes first partition information corresponding to each execution node, and the list information obtained by aggregating the first partition information includes the capacity size information corresponding to each partition. For example, the execution node a corresponds to partition 1, the execution node b corresponds to partition 2, the execution node c corresponds to partition 2, and the execution node d corresponds to partition 1. The metadata management unit 21 aggregates the partition information corresponding to the execution nodes a-d to obtain a transfer service list: (partition 1: execution node a, execution node b), (partition 2: execution node b, execution node c). The transfer service list also includes the storage size corresponding to partition 1 and partition 2. The second partition information is used to indicate the partition list and the file size information of the distributed storage data uploaded from the local disk to the cloud storage system. The reading voucher is used to indicate the proof information that the read distributed storage data can be cleaned from the local disk. For example, the distributed storage data a carries a reading voucher, and the transfer service node directly cleans the distributed storage data from the local disk according to the reading voucher. Optionally, the metadata management unit 21 also stores a directory mapping relationship, which is used to reflect the directory mapping relationship between the local disk and the cloud storage system. For example, the index directory in the local disk corresponds to the index directory in the cloud storage system. The local disk can correspond to the same named directory as the cloud storage system, or can correspond to a differently named directory as the cloud storage system, which is not limited in the present application.

[0067] The write control 22 is used to indicate the write rule applied when writing the distributed storage data into the cloud storage system and / or the local disk. The write rule includes a high water level and a low water level. When the high water level is reached, the distributed storage data block is written into the corresponding directory. When the storage space returns to the low water level, the write state is restored. The write state corresponding to the high water level and the low water level can be controlled by a configuration parameter.

[0068] The uploading unit 23 is configured to instruct the distributed storage data to be uploaded from the local disk to the cloud storage system according to a preset strategy, the preset strategy is a strategy determined based on an average file size in the local disk, the preset strategy includes a first uploading manner and a second uploading manner (for details, see step 303 in the description, wherein the first uploading manner corresponds to an uploading manner corresponding to a case where the average file size reaches a preset threshold, and the second uploading manner corresponds to an uploading manner corresponding to a case where the average file size is less than the preset threshold), the first uploading manner corresponds to a first uploading subunit 230, and the second uploading manner corresponds to a second uploading subunit 231.

[0069] The cleaning unit 24 is configured to clean the distributed storage data that meets a preset cleaning condition, the preset cleaning condition includes active cleaning and passive cleaning, the active cleaning is configured to instruct the transit service node to clean the distributed storage data that has been read, and the passive cleaning is configured to instruct the transit service node to clean the distributed storage data whose heartbeat information is timed out.

[0070] In summary, the data storage system provided in the embodiment of the present application establishes a data transmission process between the local disk and the cloud storage system, uploads the distributed storage data stored in the local disk to the cloud storage system according to a preset uploading strategy, realizes the effect of cooperating with the local storage and the cloud storage system, solves the problem of poor data processing efficiency caused by the fact that storage and calculation cannot be separated, realizes the function of supporting mixed clusters of the cloud storage system, and solves the problem of limited local disk storage.

[0071] Please refer to Figure 3 which shows a data storage method provided in the embodiment of the present application, the method is applied to Figure 1 The data storage system shown in the transit service node, in the embodiment of the present application, the data storage system is applied to a MapReduce distributed computing framework, and the data storage method includes the following steps:

[0072] Step 301, receiving the distributed storage data sent by the execution node.

[0073] In the embodiment, the execution node is configured to perform task processing on the candidate data to obtain the distributed storage data, wherein the task processing includes at least one of data storage, data calculation, data merging, data screening, data identification, and data division.

[0074] Optionally, the execution node performs data aggregation processing according to the partition information provided by the scheduling node when performing the task processing, or the execution node merges the candidate data after performing all the tasks to obtain the distributed storage data.

[0075] In the embodiment of the present application, the execution node includes at least two execution nodes, and the specific process of the task processing performed by the at least two execution nodes includes at least one of the following manners:

[0076] First, the candidate data is allocated to at least two execution nodes in batches for task processing, the at least two execution nodes perform task processing to obtain corresponding distributed storage data, and before the at least two distributed storage data are sent to the transfer service node, the distributed storage data obtained by the at least two execution nodes are aggregated in granularity according to the partition information to obtain distributed storage data belonging to the same partition, and the distributed storage data belonging to the same partition is sent to the transfer service node.

[0077] Second, the candidate data is directly allocated to the specified execution node according to the partition information for task processing, that is, different execution nodes perform data processing tasks in corresponding partitions, for example, execution node A fixedly performs data processing tasks from partition a, and execution node B fixedly performs data processing tasks from partition b; that is, the distributed storage data obtained by the transfer service node is already aggregated according to the partition, and does not need to be separately aggregated by the transfer service node.

[0078] In the embodiment of the application, the execution node includes at least two execution nodes, the transfer service node includes at least two transfer service nodes, in the process of obtaining the distributed storage data, the candidate data can be aggregated according to the partition by the at least two execution nodes to obtain the distributed storage data; the distributed storage data sent by the at least two execution nodes can be aggregated according to the partition by the at least two transfer service nodes and stored; the candidate data can be once granularly aggregated by the at least two execution nodes, and the distributed storage data after the once granular aggregation can be twice granularly aggregated by the at least two transfer service nodes, which is not limited in the application.

[0079] In the embodiment, the distributed storage data is data stored in a distributed storage manner by the execution node, wherein the distributed storage manner includes but is not limited to aggregated storage according to the partition information, aggregated storage according to the execution node, etc.

[0080] Step 302, the distributed storage data is transferred to the local disk for storage.

[0081] Optionally, the transfer service node further includes at least one buffer and a corresponding local disk, the local disk can be used to indicate the local storage area corresponding to the transfer server node, or can be used to indicate the local storage area corresponding to the current device, which is not limited in the embodiment of the application.

[0082] The transfer service node stores the received distributed storage data in the buffer, and when the distributed storage data stored in the buffer reaches a preset capacity threshold, the distributed storage data stored in the buffer is transferred to the corresponding directory in the local disk for storage.

[0083] Optionally, when the distributed storage data to be stored in the cache reaches a preset capacity threshold or reaches the upper limit of the storage capacity of the cache, the distributed storage data stored before reaching the capacity threshold is transferred to the local disk for storage.

[0084] In this embodiment, the distributed storage data carries heartbeat information corresponding to the execution node, and the transfer service node receives the distributed storage data sent by the execution node in response to the heartbeat information being in a normal state, and ignores the data sent by the execution node in response to the heartbeat information being in an abnormal state, thereby ensuring the authenticity and validity of the distributed data. Optionally, each node in the data storage system provided in the embodiment of the application sends its own heartbeat information to the next interactive node when interacting with data, and the next interactive node selects to receive data / ignore data according to the received heartbeat information.

[0085] In this embodiment, after receiving the distributed storage data, the transfer service node judges the data size of the distributed storage data, and in response to the distributed storage data being smaller than a preset data size, transfers the distributed storage data to the local disk for storage; and in response to the distributed storage data being greater than or equal to the preset data size, directly uploads the distributed storage data to the cloud storage system according to a preset upload strategy. For specific details of the preset upload strategy, please refer to the content of step 303 below, which will not be described in detail here.

[0086] Optionally, in response to the distributed storage data being smaller than the preset data size, the distributed storage data is stored in a directory corresponding to the local disk, and when the distributed storage data stored in the directory corresponding to the local disk reaches a specified data size, the distributed storage data stored in the corresponding directory is merged and uploaded to the cloud storage system according to a preset upload strategy, thereby fully combining the storage performance of the local disk and the cloud storage system to achieve the purpose of storing small data in the local disk and storing large data in the cloud storage system.

[0087] Step 303, in response to the local disk meeting the data upload condition, uploading the local storage data in the local disk to the cloud storage system according to a preset upload strategy.

[0088] Optionally, the data upload condition is used to indicate a condition for uploading the distributed storage data from the local disk to the cloud storage system.

[0089] After meeting the data upload condition, the transfer service node uploads the local storage data in the local disk to the remote storage system according to a preset upload strategy, which is a strategy determined based on the average file size in the local disk. The specific uploading process includes at least one of the following modes:

[0090] First, in response to the average file size of the locally stored data reaching a preset threshold, the locally stored data is copied to the cloud storage system.

[0091] Specifically, please refer to Figure 4 , Figure 4 A schematic diagram corresponding to a data uploading method provided by an embodiment of the present application is shown. The transit service node stores the distributed stored data to the corresponding datafile and indexfile directories in the local disk 40. When the transit service node determines that the average file size of the data in the directory of the local disk 40 reaches a preset threshold, all files in the directory corresponding to the partition partition-n are directly copied and stored to the corresponding datafile and indexfile directories in the cloud storage system 41. After the locally stored data is completely copied, or after the locally stored data is completely read, the transit service node directly deletes the uploaded / read files from the corresponding datafile and indexfile directories in the local disk 40. For example, when the locally stored data is 1 Gb (reaching the preset threshold), the transit service node directly copies 1 Gb of locally stored data and directly uploads and stores the copied 1 Gb of data to the corresponding directories in the cloud storage system. In the embodiment of the present application, this uploading method is defined as a non-combine uploading (non-combine uploading) method.

[0092] Second, in response to the average file size of the locally stored data being less than a preset threshold, the locally stored data is serially read, and target data in the locally stored data that does not coincide with the data already stored in the cloud storage system is uploaded to the cloud storage system. The target data is the data in the locally stored data that is not uploaded to the cloud storage system and is retrieved by the transit service node.

[0093] In the embodiment of the present application, the transit service node detects the average file size of the locally stored data to be uploaded. If the average file size is less than a preset threshold, a list of already stored data is obtained from the cloud storage system. The transit service node retrieves whether the locally stored data to be uploaded is stored in the list of already stored data. The locally stored data (target data) that is not uploaded to the cloud storage system is additionally uploaded to the corresponding directory file in the cloud storage system.

[0094] Optionally, after the transit service node retrieves the locally stored data (target data) that is not uploaded, the target data is combined and combined data is obtained. The transit service node uploads the combined data to the corresponding directory in the cloud storage system.

[0095] For example, when the local storage data stores data A, data B, and data C, and the cloud storage system stores data A, the transit service node retrieves data B and data C that are not stored in the remote storage system when uploading data A to data C, and merges data B and data C to obtain data D when the average file size of data B and data C is less than the preset threshold. The transit service node stores data D in the cloud storage system. Details can be combined with the description of the above embodiment. Figure 5 , Figure 5 Another data uploading method provided by the embodiment of the application is shown in the schematic diagram. The transit service node aggregates the distributed storage data in the partition according to the partition information (partition-1, …, paritition-k, …, partition-n), and stores the aggregated distributed storage data in the local disk 50. The transit service node determines that the average file size of the distributed storage data corresponding to (partition-1, …, paritition-k, …, partition-n) is less than the preset threshold, and merges the distributed data corresponding to (partition-1, …, paritition-k, …, partition-n) according to the directory (datafile and indexfile) to obtain merged data. The transit service node uploads the merged data to the directory corresponding to the cloud storage system 51. For example, when the local disk stores 1048576 files of 1Kb, the transit service node determines that the average file size is less than the preset threshold when uploading the files, and merges the 1048576 files of 1Kb to obtain a file of 1Gb. The transit service node stores the merged file of 1Gb in the directory corresponding to the cloud storage system 51. In the embodiment of the application, this uploading method is defined as the combine uploading method.

[0096] Third, the transit service node receives an uploading instruction. When the usage space of a specified directory reaches the high water level, the local storage data in the local disk is uploaded to the directory corresponding to the cloud storage system using the uploading instruction. The uploading instruction is used to indicate that the uploading is forcibly performed without waiting for the reading of the local storage data to be completed. After the uploading is completed, the data in the specified file is cleared. For example, when the usage space of the directory directory reaches the high water level, the force uploading mode is used. After the uploading is completed, the truncate file is cleared, and the usage space of the directory is 0. The high water level is used to indicate that the usage space of the directory directory reaches the upper limit.

[0097] In step 304, the local storage data is cleared from the local disk in response to the completion of the uploading of the local storage data.

[0098] Optionally, when the local storage data is uploaded to the cloud storage system, the cloud storage system feeds back a received signal to the transfer service node, and the transfer server deletes the local storage data that has been uploaded from the corresponding directory of the local disk based on the received signal.

[0099] Optionally, the transfer server node reads the local storage data to be uploaded to the cloud storage system from the local disk, writes the local storage data into a reading voucher after reading is completed, and the transfer server uploads the local storage data to the cloud storage system according to the reading voucher and clears the local storage data from the local disk. The reading voucher is used to indicate that the local storage data has been read completely. In the embodiment of the application, uploading the local storage data to the cloud storage system and clearing the local storage data from the local disk can be processed in parallel or sequentially, and the order of sequential execution is not limited.

[0100] In summary, the data storage method provided by the embodiment of the application establishes a data transmission process between the local disk and the cloud storage system, uploads the distributed storage data stored in the local disk to the cloud storage system according to a preset uploading strategy, realizes the effect of cooperating with the local storage and the cloud storage system, solves the problem that the large data work load cannot be effectively deployed in the offline hybrid cluster due to the separation of storage and computing, and improves the data processing efficiency to a certain extent.

[0101] Please refer to Figure 6 which shows the data storage method provided by the embodiment of the application. The method is applied to Figure 1 The data storage system shown in the transfer service node includes the data storage method provided by the embodiment of the application. In the embodiment of the application, the data storage system is applied to the MapReduce distributed computing framework, and the data storage method includes the following steps.

[0102] Step 601, receiving the distributed storage data sent by the execution node.

[0103] In the embodiment, the distributed storage data is data stored by the execution node in a distributed storage manner, wherein the distributed storage manner includes but is not limited to aggregated storage according to partition information, aggregated storage according to the execution node, etc.

[0104] The specific process of this step is the same as that of step 301 described above, and this step will not be described again.

[0105] Step 602, transferring the distributed storage data to the local disk for storage.

[0106] The relay service node stores the received distributed storage data into a cache, and when the storage capacity of the cache reaches a preset threshold, the relay service node stores the distributed storage data stored in the cache into a corresponding directory in a local disk.

[0107] Optionally, the at least two execution nodes perform task processing on the candidate data to obtain corresponding distributed storage data, and the relay service node receives the distributed storage data, classifies the distributed storage data according to the partition information, and stores the distributed storage data into corresponding caches, that is, different caches correspond to different partitions. For example, the execution node A sends data a to d to the relay service node, the execution node B sends data e to g to the relay service node, the relay service node receives data a to d and data e to g, classifies the data according to the partition information, and obtains the classification result: (partition 1: data a, data e, data g), (partition 2: data b, data c, data d, data f), stores the data corresponding to partition 1 into cache m, and stores the data corresponding to partition 2 into cache n.

[0108] The step has the same specific process as the step 302, and thus will not be described herein.

[0109] In step 603, a data upload instruction is received, and local storage data in a local disk is uploaded to a cloud storage system according to a preset upload strategy based on the data upload instruction.

[0110] In the embodiments of the present application, the data upload instruction includes an active upload instruction and a passive upload instruction.

[0111] In the embodiments of the present application, the specific process of receiving the active upload instruction to upload the local storage data will be described with reference to steps 603a and 603b (for details, please refer to the description of steps 603a and 603b in combination with the description of the steps 603a and 603b). Figure 7 ).

[0112] In step 603a, an active upload instruction is received.

[0113] Optionally, the active upload instruction is used to indicate an active trigger of the upload instruction of the local storage data, and the active upload instruction includes size information, quantity information of the local storage data to be uploaded, and heartbeat information corresponding to an execution node.

[0114] In step 603b, the local storage data is uploaded to the cloud storage system according to a preset upload strategy based on the active upload instruction.

[0115] Optionally, the transit service node reads the local storage data to be uploaded in the local disk based on the active upload instruction, and uploads the read local storage data to the cloud storage system, wherein the preset upload strategy can refer to the above step 303, and details are not described herein. For example, the user actively selects the file in the index file in the local disk for uploading, the transit service node receives the upload instruction, obtains the index file from the local disk, and uploads the file of the local disk index file to the index file in the cloud storage system.

[0116] The passive upload instruction is summarized as follows, which includes but is not limited to the following ways:

[0117] First, in response to the storage capacity in the local disk exceeding the storage threshold, the local storage data in the local disk is uploaded to the cloud storage system according to the preset upload strategy, which includes judging the average file size of the local storage data in the local disk, selecting a non-combine upload mode or a combine upload mode based on the average file size, and uploading the local storage data to the cloud storage system. The non-combine upload mode and the combine upload mode can refer to the above step 303, and details are not described herein.

[0118] Second, the local storage is uploaded to the cloud storage system according to the preset upload strategy with a preset upload period. The preset period can be uploaded in hours, days, weeks or months. For example, the transit service node uploads the local storage data in weeks.

[0119] Third, the historical unused local storage data is uploaded to the cloud storage system. The historical unused local storage data is used to indicate data that has not been operated within a preset time period. The preset time period can be within a specified number of days, within a specified number of weeks, within a specified number of months, or within a specified number of years. For example, the transit service node retrieves the local storage data that has not been used for three months and uploads the retrieved local storage data to the cloud storage system according to the preset upload strategy, so as to avoid long-term unused data occupying the storage space of the local disk, improve the use of the storage capacity of the local disk, and the preset upload strategy can refer to the content described in step 303, and details are not described herein.

[0120] Step 604, in response to the completion of the local storage data upload, the local storage data is deleted from the local disk.

[0121] Optionally, after the local storage data is uploaded to the cloud storage system, the cloud storage system feeds back a received signal to the transit service node, and the transit server deletes the uploaded local storage data from the corresponding directory in the local disk based on the received signal.

[0122] The specific implementation process of step 604 can refer to step 304 described above, and will not be described here.

[0123] In summary, the data storage method provided by the embodiment of the application establishes a data transmission process between the local disk and the cloud storage system, uploads the distributed storage data stored in the local disk to the cloud storage system according to a preset uploading strategy, realizes the effect of cooperating with the local storage and the cloud storage system, and solves the problem of poor data processing efficiency caused by the separation of storage and calculation.

[0124] In the data storage method provided by the embodiment, the candidate data is aggregated by the execution node to obtain the distributed storage data, and the distributed storage data is aggregated by the relay service node to obtain the local storage data, so that the related data can be efficiently written in the cloud storage system.

[0125] Please refer to Figure 8 which shows the data storage method provided by the embodiment of the application. The method is applied to Figure 1 In the embodiment, the data storage system is applied to the MapReduce distributed computing framework, and the data storage method comprises the following steps:

[0126] Step 801: receiving the distributed storage data sent by at least two execution nodes.

[0127] In the embodiment, the data storage system comprises at least two execution nodes. When performing task processing, the scheduling node obtains a list corresponding to the relay service node required for performing the task processing from the registration node, and distributes the list to the at least two execution nodes. The at least two execution nodes perform task processing on the candidate data, which includes but is not limited to data storage, data calculation, data merging, data screening, data identification, and data division.

[0128] After the at least two execution nodes process the candidate data, the corresponding distributed storage data is obtained, and the distributed storage data is sent to the corresponding relay service node.

[0129] Optionally, when the execution node performs task processing on the candidate data, only the calculation operation is performed, and the storage operation is not performed, that is, the data calculation is performed by the execution node, and the data storage is performed by the relay service node, so as to realize the separation of calculation and storage.

[0130] Step 802: aggregating the distributed storage data sent by the at least two execution nodes according to the partition information to obtain partition aggregation storage data.

[0131] Optionally, the transit service node aggregates the received distributed storage data according to the partition information to obtain partition-aggregated storage data, or aggregates the received distributed storage data according to the execution nodes to obtain node-aggregated storage data.

[0132] Optionally, when each node in the data storage system performs data interaction processing, the node sends its heartbeat information to the next node, and the next node determines whether to store the data whose processing is timed out according to the received heartbeat information. For example, the transit service node receives the distributed storage data of the execution node a and the heartbeat information corresponding to the execution node, determines that the execution node a has an exception according to the heartbeat information, and the exception includes transmission interruption, transmission timeout, etc., and the transit service node clears the local storage data that has the exception.

[0133] In step 803, the partition-aggregated storage data is transferred to the local disk for storage.

[0134] Optionally, the partition-aggregated storage data or the node-aggregated storage data is stored in the buffer in the transit service node, and when the buffer reaches a preset storage capacity threshold, the partition-aggregated storage data or the node-aggregated storage data is transferred from the buffer to the local disk for storage.

[0135] In step 804, in response to the partition-aggregated storage data satisfying a data upload condition, the partition-aggregated storage data is uploaded to the cloud storage system according to a preset upload strategy.

[0136] In the embodiment of the application, the data upload condition includes the following manners:

[0137] First, a proactive upload instruction is received, and the partition-aggregated storage data or the node-aggregated storage data is proactively uploaded to the cloud storage system for storage. After the partition-aggregated storage data or the node-aggregated storage data is read or uploaded, the transit service node clears the partition-aggregated storage data or the node-aggregated storage data in the local disk.

[0138] Second, when the storage capacity of the local disk exceeds a threshold, the partition-aggregated storage data or the node-aggregated storage data in the local disk is uploaded to the cloud storage system for storage; or,

[0139] Third, the partition-aggregated storage data or the node-aggregated storage data is uploaded to the cloud storage system at a preset period.

[0140] Fourth, the historical unused partition-aggregated storage data or node-aggregated storage data is uploaded to the cloud storage system for storage.

[0141] The preset upload strategy can refer to the content described in step 303 above, and will not be described here again.

[0142] In summary, the data storage method provided by the embodiments of the present application establishes a data transmission process between a local disk and a cloud storage system, uploads distributed storage data stored in the local disk to the cloud storage system according to a preset uploading strategy, realizes the effect of cooperating with the local storage and the cloud storage system, solves the problem that the large data work load cannot be effectively deployed in the offline hybrid cluster due to the separation of storage and computing, and improves the data processing efficiency to a certain extent.

[0143] Please refer to Figure 9 which shows an execution flow chart corresponding to the data storage system provided by an embodiment of the present application. In the embodiment of the present application, the data storage system is applied to a MapReduce distributed computing framework, and the data storage system includes an execution node, a transfer service node and a cloud storage system. It can be understood that the execution node can include at least two execution nodes, and the transfer service node can also include at least two transfer service nodes. The data interaction process between the three nodes will be described in detail as follows.

[0144] S91, the execution node performs task processing on the candidate data to obtain distributed storage data.

[0145] Optionally, the execution node includes at least two execution nodes, and the candidate data is allocated to the at least two execution nodes for task processing. The task processing includes but is not limited to data storage, data calculation, data merging, data screening, data identification and data division. The candidate data is processed in batches by the at least two execution nodes to obtain at least two distributed storage data.

[0146] Optionally, the candidate data is allocated to a specified execution node according to partition information for task processing. The execution node processes the candidate data belonging to the same partition according to the task processing requirement to obtain corresponding distributed storage data.

[0147] S91, the execution node sends the distributed storage data to the transfer service node.

[0148] Optionally, the execution node classifies and aggregates the distributed storage data processed by the task according to the execution node type or the partition information, and sends the classified and aggregated distributed storage data to the transfer service node.

[0149] Optionally, after processing the candidate data, the execution node aggregates the distributed storage data processed by the same execution node, or aggregates the distributed storage data under the same partition, and sends the aggregated distributed storage data according to the execution node type or the partition information.

[0150] S93, the transfer service node transfers the received distributed storage data to a local disk.

[0151] Optionally, the transfer service node receives the distributed storage data sent by the execution node, and pre-stores the distributed storage data in a cache. When the storage capacity of the cache reaches a threshold, the transfer service node transfers the distributed storage data in the cache to a local disk for storage. The local disk is used to indicate a local storage area corresponding to the current device or a local storage area bound to the transfer service node.

[0152] S94, the transfer service node uploads the local storage data to a cloud storage system according to a preset uploading strategy.

[0153] Optionally, in response to the local storage data meeting a data uploading condition, the transfer service node uploads the local storage data to the cloud storage system according to the preset uploading strategy.

[0154] For details of the step S94, please refer to the above step 303, which will not be repeated here.

[0155] S95, the cloud storage system stores the local storage data under a corresponding directory file.

[0156] Optionally, the cloud storage system receives the local storage data sent by the transfer service node, detects a directory dimension corresponding to the local storage data, and the directory dimension is used to indicate a folder corresponding to the storage of the local storage data.

[0157] The cloud storage system stores the local storage data under a directory consistent with the local disk storage location.

[0158] In summary, the data storage system provided by the embodiments of the present application establishes a data transmission process between the local disk and the cloud storage system, uploads the distributed storage data stored in the local disk to the cloud storage system according to a preset uploading strategy, realizes the effect of cooperating with the local storage and the cloud storage system, solves the problem that the large data work load cannot be effectively deployed in the offline hybrid cluster due to the separation of storage and calculation, and improves the data processing efficiency to a certain extent.

[0159] For details, please refer to Figure 10 which shows a structural block diagram of a data storage device provided by an embodiment of the present application. The data storage device can be applied to Figure 1 the transfer service node in the data storage system shown in the figure. The data storage device comprises:

[0160] The receiving module 1010 is configured to receive distributed storage data sent by an execution node, wherein the distributed storage data is data stored by the execution node in a distributed storage manner.

[0161] The storage module 1020 is configured to transfer the distributed storage data to a local disk for storage, the local disk being a local storage area corresponding to the current device;

[0162] The uploading module 1030 is configured to, in response to the local disk meeting a data uploading condition, upload local storage data in the local disk to a cloud storage system according to a preset uploading strategy, the preset uploading strategy being a strategy determined based on an average file size in the local disk;

[0163] The clearing module 1040 is configured to, in response to the data uploading being completed, clear the data from the local disk.

[0164] In an optional embodiment, the uploading module 1030 is further configured to, in response to the average file size of the local storage data reaching a preset threshold, store the local storage data in the cloud storage system by replication;

[0165] The uploading module 1030 is further configured to, in response to the average file size of the local storage data being less than the preset threshold, serially read the local storage data; and upload target data in the local storage data that does not coincide with data already stored in the cloud storage system to the cloud storage system.

[0166] In an optional embodiment, as shown in Figure 11 The apparatus further includes a merging module 1050.

[0167] The merging module 1050 is configured to merge the target data in the local storage data that does not coincide with the data already stored in the cloud storage system to obtain merged data.

[0168] The uploading module 1030 is further configured to upload the merged data to the cloud storage system.

[0169] In an optional embodiment, the receiving module 1010 is further configured to receive an active uploading instruction, the active uploading instruction being used to trigger uploading of the local storage data.

[0170] The uploading module 1030 is further configured to, based on the active uploading instruction, upload the local storage data to the cloud storage system according to the preset uploading strategy.

[0171] In an optional embodiment, the uploading module 1030 is further configured to upload the local storage data in the local disk to the cloud storage system according to the preset uploading strategy in response to the storage capacity in the local disk exceeding a storage threshold; or upload the local storage data to the cloud storage system according to the preset uploading strategy at a preset uploading period.

[0172] In an optional embodiment, the receiving module 1010 is further configured to receive the distributed storage data sent by at least two execution nodes.

[0173] The storage module 1020 is further configured to aggregate the distributed storage data sent by the at least two execution nodes according to the partition information to obtain partition aggregated storage data, and transfer the partition aggregated storage data to the local disk for storage.

[0174] In an optional embodiment, the distributed storage data is data obtained after processing by the execution node task.

[0175] In summary, the data storage device provided by the embodiments of the present application establishes a data transmission process between the local disk and the cloud storage system, uploads the distributed storage data stored in the local disk to the cloud storage system according to a preset uploading strategy, realizes the effect of cooperation between the local storage and the cloud storage system, solves the problem that the large data work load cannot be effectively deployed in the offline hybrid cluster due to the separation of storage and computing, and improves the data processing efficiency to a certain extent.

[0176] Please refer to Figure 12 , Figure 12 Fig. 1 shows a structural schematic diagram of a server provided by an example embodiment of the present application. The server 1100 integrates Figure 1 Fig. 1 shows a data storage system. Specifically:

[0177] The server 1200 includes a central processing unit (CPU) 1201, a system memory 1204 including a random access memory (RAM) 1202 and a read-only memory (ROM) 1203, and a system bus 1205 connecting the system memory 1204 and the central processing unit 1201. The server 1200 further includes a basic input / output system (I / O system) 1206 that helps transfer information between various devices in the computer, and a mass storage device 1207 for storing an operating system 1213, an application program 1114, and other program modules 1215.

[0178] The basic input / output system 1206 includes the various components needed to display information, such as a display 1208 and input devices 1209, such as a mouse and keyboard, for inputting information. Both the display 1208 and input devices 1209 are connected to the central processing unit 1201 through an input / output controller 1210 connected to the system bus 1205. The basic input / output system 1206 can also include the input / output controller 1210 for receiving and processing input from a number of other devices, including a keyboard, mouse, or electronic stylus. Similarly, the input / output controller 1210 provides output to a display screen, printer, or other type of output device.

[0179] The mass storage device 1207 is connected to the central processing unit 1201 through a mass storage controller (not shown) connected to the system bus 1205. The mass storage device 1207 and its associated computer readable media provide non-volatile storage for the server 1200. That is, the mass storage device 1207 can include a computer readable medium (not shown) such as a hard disk or a compact disc read only memory (CD-ROM) drive.

[0180] Without loss of generality, computer readable media can include computer storage media and communication media. Computer storage media includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data. Computer storage media includes RAM, ROM, erasable programmable read only memory (EPROM, EEPROM, Electrically Erasable Programmable Read Only Memory), flash memory or other solid state memory, CD-ROM, digital versatile disks (DVD), or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices. Computer storage media would not, however, include communication media including wired or wireless signaling buses, plain old telephone service (POTS) and cellular networks, wireless data networks, and other networks that commonly carry computer readable instructions, data structures, program modules or other data. The system memory 1204 and mass storage device 1207 described above can collectively be referred to as memory.

[0181] According to various embodiments of the present application, the server 1200 can also be operated to run on a remote computer connected to the network through a network, such as the Internet. That is, the server 1200 can be connected to the network 1212 through the network interface unit 1211 connected to the system bus 1205, or can be connected to other types of networks or remote computer systems (not shown) using the network interface unit 1211.

[0182] The above-mentioned memory further includes one or more programs stored in the memory configured to be executed by the CPU.

[0183] One embodiment of the present application provides a computer readable storage medium, the storage medium stores at least one instruction, at least one program, a code set or an instruction set, the at least one instruction, the at least one program, the code set or the instruction set are loaded and executed by the processor to implement the data storage method as described above.

[0184] One embodiment of the present application provides a data storage device, the data storage device includes a processor and a memory, the memory stores at least one instruction, the instruction is loaded and executed by the processor to implement the data storage method as described above.

[0185] It should be noted that: the data storage device provided by the above-mentioned embodiment only divides the above-mentioned each function module for example when processing data, in actual application, the above-mentioned function can be completed by different function modules according to needs, that is, the internal structure of the data storage device is divided into different function modules to complete all or part of the functions described above. In addition, the data storage device and the data storage method provided by the above-mentioned embodiment belong to the same concept, and the specific implementation process is detailed in the method embodiment, which will not be repeated here.

[0186] It can be understood that in the specific embodiments of the present application, the local storage data and other related data involved need to obtain the permission or consent of the user when the above embodiments of the present application are applied to specific products or technologies, and the collection, use and processing of related data need to comply with relevant laws, regulations and standards of relevant countries and regions.

[0187] Those skilled in the art can understand that all or part of the steps of the above-mentioned embodiments can be completed by hardware, or by program to instruct related hardware, the program can be stored in a computer readable storage medium, the above-mentioned storage medium can be read only memory, disk or optical disk, etc.

[0188] The above description is only the preferred embodiment of the present application but not to limit the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A data storage method, characterized by, Applied to a relay service node in a distributed computing framework, the method comprises: Receiving distributed storage data sent by an execution node, the distributed storage data being data stored in a distributed storage manner by the execution node; Transferring the distributed storage data to a local disk for storage, the local disk being a local storage area corresponding to the current device; In response to the local disk meeting a data upload condition, uploading local storage data in the local disk to a cloud storage system according to a preset upload strategy, the preset upload strategy being a strategy determined based on an average file size in the local disk; wherein, in response to the average file size of the local storage data reaching a preset threshold, copying and storing the local storage data to the cloud storage system; in response to the average file size of the local storage data being less than the preset threshold, serially reading the local storage data; merging target data in the local storage data that does not coincide with data already stored in the cloud storage system to obtain merged data; and uploading the merged data to the cloud storage system; In response to the local storage data being uploaded, cleaning the local storage data from the local disk.

2. The method of claim 1, wherein, The response to the local disk meeting the data upload condition, uploading the local storage data in the local disk to the cloud storage system according to the preset upload strategy, comprises: Receiving an active upload instruction, the active upload instruction being used to trigger uploading the local storage data; Based on the active upload instruction, uploading the local storage data to the cloud storage system according to the preset upload strategy.

3. The method of claim 1, the response to the local disk meeting the data upload condition, uploading the local storage data in the local disk to the cloud storage system according to the preset upload strategy, comprises: In response to the storage capacity in the local disk exceeding a storage threshold, uploading the local storage data in the local disk to the cloud storage system according to the preset upload strategy; Or, Uploading the local storage data to the cloud storage system according to the preset upload strategy at a preset upload period.

4. The method of claim 1, wherein, The receiving of the distributed storage data sent by the execution node comprises: Receiving distributed storage data sent by at least two execution nodes; The transferring of the distributed storage data to the local disk for storage comprises: Aggregating the distributed storage data sent by the at least two execution nodes according to partition information to obtain partition-aggregated storage data; Transferring the partition-aggregated storage data to the local disk for storage.

5. The method of claim 1, wherein: The distributed storage data is data obtained after the execution node task processing.

6. A data storage system, characterized by Applied to a distributed computing framework, the system comprises: An execution node, configured to perform task processing on candidate data to obtain distributed storage data, and send the distributed storage data to a relay service node, the distributed storage data being data stored in a distributed storage manner; The transit service node is configured to receive the distributed storage data, transfer the distributed storage data to a local disk for storage, the local disk being a local storage area corresponding to a current device, and in response to the local disk meeting a data uploading condition, upload local storage data in the local disk to a cloud storage system according to a preset uploading strategy, the preset uploading strategy being a strategy determined based on an average file size in the local disk. The cloud storage system is configured to receive the local storage data uploaded by the local disk. The transit service node is further configured to, in response to the uploading of the local storage data being completed, clear the local storage data from the local disk.

7. A data storage device, characterized by The device comprises: A receiving module configured to receive distributed storage data sent by an execution node, the distributed storage data being data stored by the execution node in a distributed storage manner. A storage module configured to transfer the distributed storage data to a local disk for storage, the local disk being a local storage area corresponding to a current device. An uploading module configured to, in response to the local disk meeting a data uploading condition, upload local storage data in the local disk to a cloud storage system according to a preset uploading strategy, the preset uploading strategy being a strategy determined based on an average file size in the local disk. A clearing module configured to, in response to the uploading of the data being completed, clear the data from the local disk.

8. A computer device, comprising: The computer device comprises a processor and a memory, the memory storing at least one program, the at least one program being loaded and executed by the processor to implement the data storage method according to any one of claims 1 to 5.

9. A computer-readable storage medium, characterized in that, The storage medium stores at least one program, the at least one program being loaded and executed by the processor to implement the data storage method according to any one of claims 1 to 5.

10. A computer program product comprising computer instructions, characterized in that, The computer instructions are stored in a computer readable storage medium, and a processor of the computer device reads the computer instructions from the computer readable storage medium, and the processor executes the computer instructions, so that the computer device executes to realize the data storage method as claimed in any one of claims 1 to 5.

Images