Data processing method, apparatus, device, and system

Abstract

Disclosed are a data processing method, device, equipment and system, and relate to the field of big data. A control device is configured to control a computing device to perform big data distributed processing. The computing device is configured with storage media of multiple deployment modes, and the storage media of multiple deployment modes include at least two of local storage media in the computing device, global storage media composed of the local storage media, extended global storage media and extended local storage media. The method includes that the control device acquires a service request of a big data service, determines at least one storage medium of a deployment mode from the storage media of multiple deployment modes, sends a control instruction to the computing device, instructs the computing device to perform a map task in big data distributed processing on service data included in the service request, and stores obtained intermediate data to the at least one storage medium of the deployment mode. Thus, the storage space of the storage media in the computing device occupied for storing the intermediate data is reduced.

Description

[0001] This application claims priority to the Chinese Patent Application No. 202111234307.4 filed on October 22, 2021, and entitled “Distributed System”, the content of which is incorporated herein by reference in its entirety.

[0002] This application is a divisional application of the original application with the application number 202111583902.9 and the original filing date of December 22, 2021, the content of which is incorporated herein by reference in its entirety. TECHNICAL FIELD

[0003] The present application relates to the field of big data, and in particular to a data processing method, device, equipment and system. BACKGROUND

[0004] Big data is a mass, high growth rate and diversified information asset that requires innovative processing mode to have stronger decision-making ability, insight and process optimization ability. Generally, big data is distributedly processed by multiple computing devices based on a MapReduce model. Since data is distributed in different computing devices, the computing device performing a map task stores the intermediate data obtained after the map task. The computing device performing a reduce task can read the intermediate data from a remote end, and this data exchange process is called shuffle. Currently, the computing device stores the intermediate data to the memory or hard disk in the computing device. In the scenario of distributed processing of big data based on the MapReduce model, the storage space of the storage medium in the computing device is excessively occupied. SUMMARY

[0005] The present application provides a data processing method, device, equipment and system, thereby reducing the storage space of the storage medium in the computing device.

[0006] In a first aspect, a data processing method is provided, the method is executed by a control device, the control device is configured to control a computing device to perform distributed processing of big data, the computing device is configured with a plurality of deployment modes of storage media, the plurality of deployment modes of storage media includes at least two of a local storage medium in the computing device, a global storage medium composed of the local storage medium, an extended global storage medium and an extended local storage medium. The method includes that the control device acquires a service request of a big data service, determines at least one deployment mode of storage media from the plurality of deployment modes of storage media, and sends a control instruction to the computing device, instructing the computing device to perform a map task in the distributed processing of big data on service data included in the service request, and store the obtained intermediate data to the at least one deployment mode of storage media.

[0007] Thus, the control device can select the storage medium for storing the intermediate data from the storage media of the multiple deployment modes, so as to reduce the storage space of the storage medium (e.g., memory or disk) in the computing device for storing the intermediate data, release more available storage space for the computing device to perform the map task or the reduce task, and thus improve the processing efficiency of the computing device for processing the map task or the reduce task.

[0008] In a possible implementation, determining the storage medium of at least one deployment mode from the storage media of the multiple deployment modes includes: controlling the device to determine the storage medium of at least one deployment mode from the storage media of the multiple deployment modes according to a storage strategy. The storage strategy is used to indicate a method of selecting the storage medium according to the storage medium characteristics. The storage medium characteristics include at least one of write latency, read latency, total storage capacity, available storage capacity, access speed, central processing unit (CPU) consumption, energy consumption ratio, and reliability.

[0009] Example one, determining the storage medium of at least one deployment mode from the storage media of the multiple deployment modes according to the storage strategy includes: controlling the device to determine the storage medium of at least one deployment mode from the storage media of the multiple deployment modes according to the priority of the storage media of the multiple deployment modes indicated by the storage strategy. The priority of the storage media of the multiple deployment modes is determined according to the storage medium characteristics.

[0010] Example two, determining the storage medium of at least one deployment mode from the storage media of the multiple deployment modes according to the storage strategy includes: controlling the device to determine the storage medium of at least one deployment mode from the storage media of the multiple deployment modes according to the user demand and the storage medium characteristics of the storage media of the multiple deployment modes. The user demand is used to indicate the demand related to the storage medium characteristics. Thus, when the control device selects the storage medium for storing the intermediate data from the storage media of the multiple deployment modes, the control device dynamically selects the storage medium of at least one deployment mode that meets the user demand based on the user demand for the access speed or the reliability, and ensures the performance (e.g., data processing rate) and the reliability of the scene demand of the data processing relative to the storage of the intermediate data of different scenes by using the storage medium of one deployment mode.

[0011] The storage medium includes memory, hard disk, memory server, and storage-class memory (SCM). Thus, the memory server and the SCM are introduced in the big data distributed processing system, the storage medium for storing the intermediate data is expanded, the storage medium for storing the intermediate data has more possibilities, the storage space of the storage medium in the computing device for storing the intermediate data is reduced, and the reliability of the intermediate data is improved.

[0012] It should be noted that the storage strategy is configured in a big data service granularity or a task granularity in a big data service or a system global granularity.

[0013] In another possible implementation, after the control indication is sent to the computing device, the method further includes: the control device receiving a storage location of the intermediate data in the storage medium of at least one deployment mode sent by the computing device, the storage location of the intermediate data being used to read the intermediate data when the reduce task is executed. Further, the control device receives an intermediate data request sent by the computing device, the intermediate data request being used for the storage location where the intermediate data is located. The control device sends metadata indicating the storage location of the intermediate data to the computing device, the metadata indicating the storage location where the intermediate data is located. The computing device acquires the intermediate data according to the metadata.

[0014] In a second aspect, a data processing method is provided, the method being performed by a computing device, the computing device being configured with storage media of multiple deployment modes, the computing device being connected to a control device, the control device being used to control the computing device to perform big data distributed processing. The method includes: the computing device performing a map task in big data distributed processing on service data according to a control indication sent by the control device, to obtain intermediate data, and storing the intermediate data into a storage medium indicated by an identifier of the storage medium of at least one deployment mode. The control indication includes the service data and the identifier of the storage medium of at least one deployment mode, the storage medium of at least one deployment mode being determined by the control device from the storage media of multiple deployment modes according to a storage strategy, the storage strategy being used to indicate a method of selecting a storage medium according to a storage medium feature.

[0015] In this way, the storage media of multiple deployment modes are added in the system, so that the control device can select a storage medium for storing the intermediate data from the storage media of multiple deployment modes, and the computing device stores the intermediate data into the storage medium indicated by the identifier of the storage medium of at least one deployment mode according to the indication of the control device, to reduce the storage space of the storage medium (such as a memory or a disk) in the computing device for storing the intermediate data, release more available storage space for the computing device to perform the map task or the reduce task, and improve the processing efficiency of the computing device to process the map task or the reduce task.

[0016] In a possible implementation, the storing the intermediate data to the storage medium indicated by the identification of the at least one deployment mode of the storage medium comprises: if the at least one deployment mode of the storage medium comprises a plurality of storage media of one deployment mode, storing a plurality of pieces of the intermediate data to the plurality of storage media respectively; or if the at least one deployment mode of the storage medium comprises storage media of two or more deployment modes, storing a plurality of pieces of the intermediate data to the storage media of the two or more deployment modes respectively.

[0017] In another possible implementation, the plurality of deployment modes of the storage medium comprises a first storage medium and a second storage medium, and the method further comprises: if an available storage capacity of the first storage medium is less than a threshold value, storing data of the first storage medium into the second storage medium, and an access speed of the first storage medium is greater than an access speed of the second storage medium. Thus, after the intermediate data in the storage medium with the available storage capacity less than the threshold value is removed, the available storage capacity of the storage medium can be increased, so as to improve the processing efficiency of the intermediate data and ensure that the intermediate data can be normally written into the storage medium.

[0018] In another possible implementation, the method further comprises: the computing device sending, to the control device, available storage capacities of the plurality of deployment modes of the storage medium, so that the control device determines the at least one deployment mode of the storage medium from the plurality of deployment modes of the storage medium according to the available storage capacities of the storage medium.

[0019] In a third aspect, a control apparatus is provided, which comprises various modules for performing the data processing method in the first aspect or any possible design in the first aspect.

[0020] In a fourth aspect, a data processing apparatus is provided, which comprises various modules for performing the data processing method in the second aspect or any possible design in the second aspect.

[0021] In a fifth aspect, a control device is provided, which comprises at least one processor and a memory, the memory being configured to store a set of computer instructions; when the processor executes the set of computer instructions as an execution device in the first aspect or any possible implementation in the first aspect, the operation steps of the data processing method in the first aspect or any possible implementation in the first aspect are performed.

[0022] In a sixth aspect, a computing device is provided, which comprises at least one processor and a memory, the memory being configured to store a set of computer instructions; when the processor executes the set of computer instructions as an execution device in the second aspect or any possible implementation in the second aspect, the operation steps of the data processing method in the second aspect or any possible implementation in the second aspect are performed.

[0023] In a seventh aspect, a computer-readable storage medium is provided, comprising: computer software instructions; when the computer software instructions are run in a computing device, the computing device is caused to perform the operation steps of the method as described in the first aspect or any possible implementation manner of the first aspect, or, the computing device is caused to perform the operation steps of the method as described in the second aspect or any possible implementation manner of the second aspect.

[0024] In an eighth aspect, a computer program product is provided, when the computer program product is run on a computer, the computing device is caused to perform the operation steps of the method as described in the first aspect or any possible implementation manner of the first aspect, or, the computing device is caused to perform the operation steps of the method as described in the second aspect or any possible implementation manner of the second aspect.

[0025] On the basis of the implementation manners of the aspects provided by the present application, further combinations can be made to provide more implementation manners. BRIEF DESCRIPTION OF DRAWINGS

[0026] Figure 1 An architecture schematic diagram of a big data processing system provided by an embodiment of the present application;

[0027] Figure 2 A schematic diagram of big data distributed processing provided by an embodiment of the present application;

[0028] Figure 3 A storage medium deployment scenario schematic diagram provided by an embodiment of the present application;

[0029] Figure 4 A storage system schematic diagram of a three-layer structure provided by an embodiment of the present application;

[0030] Figure 5 A write intermediate data process schematic diagram provided by an embodiment of the present application;

[0031] Figure 6 A read intermediate data process schematic diagram provided by an embodiment of the present application;

[0032] Figure 7 A control device structure schematic diagram provided by an embodiment of the present application;

[0033] Figure 8 A data processing device structure schematic diagram provided by an embodiment of the present application;

[0034] Figure 9 A computing device structure schematic diagram provided by an embodiment of the present application. DETAILED DESCRIPTION

[0035] With the development of Internet, Internet of Things, network bandwidth, intelligent terminal and cloud computing services, the data type and data size develop at an unprecedented speed, and big data is generated. Data changes from a single processing object to a basic resource. Big data is a collection of data that cannot be captured, managed and processed by conventional software tools within a time range.

[0036] The source of big data can include management systems, web systems, physical systems and scientific experiment systems. The management system includes transaction processing systems and office automation systems. The web system includes Internet systems such as search engines. The physical system refers to systems related to physical objects and physical processes, such as real-time monitoring, real-time detection or production scheduling. The scientific experiment system refers to a physical system based on an experimental environment.

[0037] According to the data structure characteristics, big data is classified into structured data, unstructured data and semi-structured data. Structured data refers to data represented by a uniform structure, such as data stored in a database of a financial system, enterprise resource planning system or customer relationship management system. Unstructured data refers to data with variable field length, which is not easy to express by structured data, such as audio, video or image data. Semi-structured data is data between structured data and unstructured data, and semi-structured data belongs to the same collection of data with different attributes, such as data of a mail system.

[0038] Since the large amount of data contained in big data has a correlation relationship, the data analysis method, model or tool is used to analyze big data, mine the data relationship in big data, and use the data relationship of big data for prediction or decision-making. For example, analyzing user shopping trend data, pushing the user to the user's possible purchase items, and improving the user's shopping experience. Therefore, big data has the characteristics of large data volume, fast data growth rate, diverse data types and high utilization value.

[0039] Since the data volume of big data is very large, a single computing node cannot meet the computing needs of big data. Usually, a distributed file system (DFS) is used to store and manage big data, and multiple computing nodes in a computing cluster are used to process big data services.

[0040] Figure 1 An architecture diagram of a big data processing system provided by an embodiment of the present application is shown. As shown in Figure 1 The big data processing system 100 includes a client 110, a computing cluster 120 and a storage cluster 130.

[0041] The storage cluster 130 includes at least two storage nodes 131. One storage node 131 includes one or more controllers, network cards and a plurality of hard disks. The hard disks are used to store data. The hard disks can be magnetic disks or other types of storage media, such as solid state disks or shingled magnetic recording hard disks, etc. The network cards are used to communicate with the computing nodes 121 included in the computing cluster 120. The controllers are used to write data into the hard disks or read data from the hard disks according to read / write data requests sent by the computing nodes 121. In the process of reading / writing data, the controllers need to convert the addresses carried in the read / write data requests into addresses recognizable by the hard disks.

[0042] The client 110 communicates with the computing cluster 120 and the storage cluster 130 through the network 140. For example, the client 110 sends a service request of a big data service to the computing cluster 120 through the network 140, and requests the computing cluster 120 to perform distributed processing on service data included in the service request. The network 140 can be an enterprise intranet (such as a local area network (LAN)) or the Internet.

[0043] In some embodiments, the storage cluster 130 stores and manages big data based on the distributed file system 132 and the distributed database 133. The client 110 is installed with a client program 111, and the client 110 runs the client program 111 to display a user interface (UI) which is operated by the user 150 to access the distributed file system 132 and the distributed database 133 to obtain data and instruct the computing cluster 120 to process big data services. The client 110 is a computer connected to the network 140, and can also be referred to as a workstation. Different clients can share resources (such as computing resources and storage resources) on the network. In other embodiments, the system administrator 160 can access the distributed file system 132 and the distributed database 133 through the client 110 to call an application platform interface (API) 112 or a command-line interface (CLI) interface 113 to configure system information, etc. For example, the present application provides deployment information and storage strategies of storage media in a plurality of deployment modes for computing nodes or storage nodes.

[0044] The computing cluster 120 includes at least two computing nodes 121, which can communicate with each other. A computing node 121 is a computing device such as a server, a desktop computer, or a controller of a storage array, etc. A large data service submitted by the client 110 can be referred to as a job. The job can be split into multiple tasks, which are executed in parallel by multiple computing nodes, and all the tasks are completed to mark the completion of a job. A task is generally a processing procedure on a portion of data or a stage in a job, and all the tasks are scheduled to be completed in parallel or in series. In some embodiments, the computing cluster 120 performs distributed processing on a large data service based on a MapReduce model 134. The MapReduce model is a distributed programming model, i.e., a large data service is split into map tasks and reduce tasks, the map tasks are executed by multiple computing nodes 121, and the processing results are collected to execute the reduce tasks. In some embodiments, as shown in FIG. 1, the computing cluster 120 includes a control node 122 and at least two computing nodes 121. The control node and the computing node can be independent physical devices, and the control node can also be referred to as a control device or a name node. The computing node can be referred to as a computing device or a data node. The control node 122 is used to manage the namespace of the distributed file system and the access of the client 110 to the distributed file system. In addition, the control node 122 instructs the computing nodes to execute the map tasks and the reduce tasks. The computing nodes store the intermediate data (also referred to as map data or shuffle data) obtained by executing the map tasks based on the MapReduce model 134. The computing nodes that execute the reduce tasks read the intermediate data. Alternatively, the storage cluster 130 can also store the data after processing based on the MapReduce model 134 when storing the data. Figure 2

[0045] Embodiments of the present application provide a data processing method, in particular, a distributed processing technology for large data, and specifically, a technology for selecting a storage medium for storing intermediate data from multiple deployment modes of storage media, so as to reduce the storage space of the storage medium (such as memory or disk) in the computing device for storing the intermediate data, and release more available storage space for the computing nodes to execute the map tasks or the reduce tasks.

[0046] ​The plurality of deployment modes include a local deployment mode, a fusion deployment mode, and a separation deployment mode. The storage medium in the local deployment mode refers to a local storage medium. The storage medium in the fusion deployment mode refers to a global storage medium composed of local storage media. The storage medium in the separation deployment mode includes an extended global storage medium and an extended local storage medium. The storage medium in the plurality of deployment modes includes at least two of the local storage medium within the computing device, the global storage medium composed of local storage media, the extended global storage medium, and the extended local storage medium. For example, as shown in Figure 3 FIG. 1 is a schematic diagram of a storage medium deployment scenario provided by an embodiment of the present application. The storage medium 310 within each of the N computing nodes can refer to a storage medium in a local deployment mode. The global storage medium 320 includes storage media provided by at least two of the N computing nodes. The global storage medium refers to a storage medium that can be accessed by all computing nodes in the system and has global unified addressing. It should be understood that the storage capacity of the global storage medium is part of the storage capacity of the storage media of the N computing nodes, and the reserved storage capacity of the storage media of the N computing nodes is used for processing large data services by the computing nodes.

[0047] The extended global storage medium 330 refers to newly added storage media connected to the N computing nodes in the system. The newly added storage media have global unified addressing and can be accessed by the extended global storage medium 330 of the N computing nodes. The extended local storage medium 340 refers to newly added storage media connected to a computing node in the system. The newly added storage media are only accessed by the computing node connected thereto.

[0048] It should be noted that the memory is a memory device for storing programs and various data. The larger the capacity of the memory, the slower the access speed. Conversely, the smaller the capacity of the memory, the faster the access speed. The access speed refers to the data transmission speed when data is written to or read from the memory. The access speed can also be referred to as the read-write speed. According to the storage capacity and the access speed, the memory can be divided into different levels.

[0049] For example, Figure 4 FIG. 2 is a schematic diagram of a three-layer structure of a storage system provided by an embodiment of the present application. From the first layer to the third layer, the storage capacity increases gradually, the access speed decreases gradually, and the cost decreases gradually. For example, Figure 4As shown, the first level includes registers 411, level one cache 412, level two cache 413 and level three cache 414 in a central processing unit (CPU). The second level includes memory that can serve as the main memory of the computer system. For example, dynamic random access memory (DRAM) 421, double data rate synchronous dynamic random access memory (DDR SDRAM) 422, storage-class-memory (SCM) 423. The main memory can be referred to as main memory or memory, i.e., the memory that exchanges information with the CPU. The third level includes memory that can serve as the auxiliary memory of the computer system. For example, network memory 431, solid state disk (SSD) 432, hard disk drive (HDD) 433. The auxiliary memory can be referred to as auxiliary storage or external storage. Compared with the main memory, the external storage has a large storage capacity and a slow access speed. It can be seen that the closer the memory is to the CPU, the smaller the capacity, the faster the access speed, the larger the bandwidth, and the lower the delay. Therefore, the memory included in the third level stores data that is not frequently accessed by the CPU, thereby improving the reliability of the data. The memory included in the second level can serve as a cache device to store data frequently accessed by the CPU, thereby significantly improving the access performance of the system.

[0050] The storage media in various deployment modes described in the embodiments of the present application include memory, hard disk, memory server and SCM. For example, as shown in Table 1, the embodiments of the present application provide storage media in various deployment modes.

[0051] Table 1

[0052]

[0053]

[0054] As shown in Table 1, the local storage medium includes the memory and the hard disk, the access speed of the memory is higher than that of the hard disk, and the storage capacity of the memory is lower than that of the hard disk. The global storage medium composed of the local storage medium includes the DRAM and the SCM, and the access speed of the DRAM is higher than that of the SCM. The extended global storage medium and the extended local storage medium both include the memory server and the high-performance storage (HPS). Different storage media have different storage medium characteristics. The storage medium characteristics include at least one of a write latency, a read latency, a total storage capacity, an access speed, a CPU consumption, an energy consumption ratio, and reliability. The write latency refers to the latency of the computing node 121 writing the intermediate data to the storage medium. The read latency refers to the latency of the computing node 121 reading the intermediate data from the storage medium. The storage capacity refers to the total storage capacity of the storage medium that can store data. The access speed refers to the speed of the computing node 121 performing read and write operations on the storage medium. The CPU consumption refers to the CPU usage of the computing node 121 for writing the intermediate data to the storage medium or reading the intermediate data from the storage medium. The energy consumption ratio refers to the energy (such as electric energy) consumed per unit of time. The reliability refers to the degree of persistence of the data stored by the storage medium. The storage media of the various deployment modes in Table 1 are only illustrative, and in actual applications, the storage media of the various deployment modes can be configured according to business needs.

[0055] The embodiments of selecting the storage medium for storing the intermediate data from the storage media of the various deployment modes provided by the embodiments of the present application will be described in detail below with reference to the accompanying drawings.

[0056] Figure 5 A flowchart of a data processing method provided by the embodiments of the present application. Here, the client 110 and the computing cluster 120 are taken as examples for illustration. The computing cluster 120 includes the control node 122 and at least two computing nodes 121, the control node 122 is configured to control the computing nodes 121 to perform distributed processing of big data, and the computing nodes 121 are configured with storage media of various deployment modes. As shown in the figure, the method includes the following steps. Figure 5

[0057] S510, the control node 122 receives the service request of the big data service sent by the client 110.

[0058] ​The client 110 sends a service request of the big data service to the control node 122 in response to a user operation. The user operation can refer to an operation of operating a big data user interface to submit the big data service. The control node 122 can receive the service request of the big data service sent by the client 110 through a local area network or the Internet. The service request can include a service identifier and service data. The service identifier is used to uniquely indicate a big data service. The service data can be data for distributed processing of the big data computing node or identifier data indicating data to be processed.

[0059] The big data service includes a data analysis service, a data query service, and a data modification service, etc. For example, the big data service refers to analyzing personal data and purchase behavior data of a customer to draw a user portrait to classify the customer, so that a specific product or preferential product can be recommended to the specific customer, customer satisfaction is improved, and customer relationship is stabilized, etc. For another example, the big data service refers to analyzing historical sales of a product to predict future sales, finding reasons for a decrease in sales or reasons for an increase in sales, and recommending constructive suggestions to improve sales.

[0060] S520, the control node 122 determines at least one deployment mode of the storage medium from the plurality of deployment modes of the storage medium.

[0061] The control node 122 determines at least one deployment mode of the storage medium from the plurality of deployment modes of the storage medium according to a storage strategy. For example, the at least one deployment mode of the storage medium includes a global storage medium and an extended global storage medium. The storage strategy is used to indicate a method of selecting a storage medium according to a storage medium feature.

[0062] In some embodiments, the control node 122 is configured with a priority of the plurality of deployment modes of the storage medium determined according to a storage medium feature.

[0063] For example, the priority of the plurality of deployment modes of the storage medium is determined according to an access speed of the storage medium. The access speed of the global storage medium is higher than the access speed of the memory, the access speed of the memory is higher than the access speed of the hard disk, the access speed of the hard disk is higher than the access speed of the extended local storage medium, and the access speed of the extended local storage medium is higher than the access speed of the extended global storage medium. The priority of the storage medium (3) in the eight storage media shown in Table 1 is the highest, and the priority of the storage medium is in descending order of the priority of the DRAM, the priority of the SCM, the priority of the memory, the priority of the hard disk, the priority of the memory server, and the priority of the high-performance storage.

[0064] For example, the priority of the storage medium of the multiple deployment modes is determined according to the priority of the deployment mode. The priority of the local deployment mode is higher than the priority of the fusion deployment mode, and the priority of the fusion deployment mode is higher than the priority of the separation deployment mode. For example, the priority of the storage medium (1) in the eight storage media shown in Table 1 is the highest, and the priority is sequentially reduced to the storage medium (8), that is, the priority of the storage medium from high to low is the priority of the memory, the priority of the hard disk, the priority of the DRAM, the priority of the SCM, the priority of the memory server, and the priority of the high-performance storage. Alternatively, the priority of the fusion deployment mode is higher than the priority of the local deployment mode. The priority of the local deployment mode is higher than the priority of the separation deployment mode. The priority of the storage medium (3) in the eight storage media shown in Table 1 is the highest, and the priority of the storage medium from high to low is the priority of the DRAM, the priority of the SCM, the priority of the memory, the priority of the hard disk, the priority of the memory server, and the priority of the high-performance storage.

[0065] For example, the priority of the storage medium of the multiple deployment modes can be determined according to the characteristics of the multiple storage media. The priority of the storage medium of the multiple deployment modes is determined according to the priority of the deployment mode. The priority of the storage medium under the same deployment mode can be determined according to the access speed of the storage medium. For example, the priority of the storage medium (1) in the eight storage media shown in Table 1 is the highest, and the priority is sequentially reduced to the storage medium (8).

[0066] The control node 122 determines at least one storage medium of the multiple deployment modes from the multiple storage media according to the priority of the storage medium of the multiple deployment modes indicated by the storage strategy. For example, the storage medium characteristics can also include the available storage capacity. The available storage capacity refers to the remaining storage capacity obtained by subtracting the used storage capacity from the storage capacity of the storage medium. The control node 122 starts from the storage medium with the highest priority according to the priority of the storage medium of the multiple deployment modes, judges whether the available storage capacity of the storage medium with the highest priority is greater than the threshold value, if the available storage capacity of the storage medium with the highest priority is greater than the threshold value, it indicates that the storage medium with the highest priority has extra storage space to store the intermediate data, and then the storage medium with the highest priority is selected to store the intermediate data; if the available storage capacity of the storage medium with the highest priority is less than or equal to the threshold value, it indicates that the storage medium with the highest priority does not have extra storage space to store the intermediate data, then the available storage capacity of the storage medium with the second highest priority is judged, and the multiple storage media of the multiple deployment modes are sequentially traversed, and finally at least one storage medium of the multiple deployment modes is determined from the multiple storage media. When the control node 122 selects the storage medium according to the priority of the deployment mode, the at least one storage medium of the multiple deployment modes can include multiple storage media of one deployment mode.

[0067] In some embodiments, the control node 122 determines at least one deployment mode storage medium from the plurality of deployment mode storage media according to a user demand and storage medium features of the plurality of deployment mode storage media, the user demand indicating a demand related to the storage medium features.

[0068] For example, the control node 122 is configured with storage medium features of the plurality of deployment mode storage media. The service request contains a user demand, the user demand indicating an access speed range or a specific access speed, and the control node 122 determines at least one deployment mode storage medium from the plurality of deployment mode storage media that satisfies the user demand. For example, the user demand indicates an access speed of memory. The control node 122 determines at least one of local storage medium or global storage medium composed of local storage medium, such as memory, DRAM and SCM, from the plurality of deployment mode storage media.

[0069] For another example, the control node 122 is configured with an association relationship between storage medium and customer level. The service request contains a user demand, the user demand indicating a first customer level. The control node 122 determines a storage medium associated with the first customer level from the association relationship according to the first customer level, and determines the storage medium associated with the first customer level to store intermediate data.

[0070] In this way, when the control node 122 selects a storage medium to store intermediate data from the plurality of deployment mode storage media, at least one deployment mode storage medium that satisfies the user demand is dynamically selected based on the user demand of the user for the access speed or the reliability feature, and the performance and the reliability of the scene demand of the data processing are ensured relative to the storage of intermediate data of different scenes by using a deployment mode storage medium.

[0071] Optionally, the control node 122 can also select a storage medium with a storage capacity greater than a threshold from at least one deployment mode storage medium that satisfies the user demand as the storage medium to store the intermediate data.

[0072] It should be noted that in actual application, the storage strategy can be set according to business demand, scene demand or user demand, and the above description of the storage strategy is only for illustration. For example, the storage strategy can also refer to data localization preference, that is, the intermediate data is preferentially stored in local storage medium. For another example, the storage strategy can also refer to performance priority, storage capacity priority, cost priority, etc. according to the trade-off of application performance and cost.

[0073] Before the control node 122 selects the storage medium for storing the intermediate data using the storage strategy, i.e. before S520 is performed, the storage strategy and the configuration information of the storage medium of the multiple deployment modes can be pre-configured. The storage strategy provided by the embodiments of the present application is configured in the granularity of a big data service, or in the granularity of a task in a big data service, or in the granularity of the whole system, and then the control node 122 can use the storage strategy when selecting the storage medium for storing the intermediate data in processing a big data service, or processing a task in a big data service, or processing a task of the whole system.

[0074] In addition, the control node 122 can select a computing node 121 that is idle in the computing cluster 120 to perform the map task, or select a computing node 121 from the computing cluster 120 to perform the map task based on a balancing strategy.

[0075] S530, the control node 122 sends a control instruction to at least one computing node 121.

[0076] The control node 122 can divide the big data service into multiple map tasks, determine the computing node 121 for performing each map task, and determine the storage medium used by the computing node 121 for performing each map task to store the intermediate data according to the method described in S520.

[0077] The control node 122 sends a control instruction to at least one computing node 121 for performing the map task, and the control instruction is used to instruct the computing node 121 to perform the map task in the big data distributed processing of the service data, and store the obtained intermediate data to the storage medium of the at least one deployment mode. The control instruction can include the service data and the identification of the storage medium of the at least one deployment mode.

[0078] S540, the computing node 121 performs the map task in the big data distributed processing of the service data according to the control instruction sent by the control node 122, and obtains the intermediate data.

[0079] S550, the computing node 121 stores the intermediate data to the storage medium indicated by the identification of the storage medium of the at least one deployment mode.

[0080] The explanation of the storage medium of the at least one deployment mode can refer to the description of S520 above, and will not be repeated. The computing node 121 can write the intermediate data to the storage medium of the at least one deployment mode through a gather interface.

[0081] Optionally, the storage media of the plurality of deployment modes can be configured with different access interfaces, and the computing node 121 can invoke the corresponding access interface of the storage media to write the intermediate data to the storage media. For example, the computing node 121 invokes the access interface of the global storage media to write the intermediate data to the global storage media. For another example, the computing node 121 invokes the access interface of the extended global storage media to write the intermediate data to the extended global storage media.

[0082] If the storage media of at least one deployment mode includes a plurality of storage media of one deployment mode, a plurality of shard data of the intermediate data are respectively stored in the plurality of storage media of one deployment mode, for example, the plurality of storage media of one deployment mode includes Figure 3 the storage media 310 in the N computing nodes shown. Alternatively, if the storage media of at least one deployment mode includes storage media of two or more deployment modes, a plurality of shard data of the intermediate data are respectively stored in the storage media of two or more deployment modes, for example, the storage media of two or more deployment modes includes Figure 3 the global storage media 320 and the extended global storage media 330 shown.

[0083] S560, the computing node 121 sends the storage location of the intermediate data in the storage media of at least one deployment mode to the control node 122.

[0084] The computing node 121 stores the intermediate data in the storage media indicated by the identifier of the storage media of at least one deployment mode, generates metadata indicating the storage location of the intermediate data. The computing node 121 sends the metadata to the control node 122, so that the control node 122 instructs the computing node 121 performing the reduce task to read the intermediate data, for example, as Figure 6 shown, the computing node 121 performs S570 and the control node 122 performs S580.

[0085] S570, the computing node 121 sends an intermediate data request to the control node 122.

[0086] The intermediate data request is used for the storage location of the intermediate data. For example, the intermediate data request contains a service identifier or an object identifier, and the object identifier is used to indicate the identifier of the data to be obtained.

[0087] S580, the control node 122 sends the metadata indicating the storage location of the intermediate data to the computing node 121.

[0088] S590, the computing node 121 obtains the intermediate data according to the metadata.

[0089] The computing node 121 obtains the intermediate data through a gather interface according to the storage location of the intermediate data, and performs a reduce task according to the intermediate data. If the intermediate data is stored on multiple storage media, the computing node 121 obtains the shard data of the intermediate data from the multiple storage media.

[0090] It should be noted that the computing node performing the map task and the computing node performing the reduce task can be different computing nodes.

[0091] The data processing method provided by the embodiment of the present application extends the single deployment mode of the storage medium storing the intermediate data to multiple deployment modes, and realizes the extension of the storage location of the intermediate data. At the same time, according to the requirements of the business scenario, the appropriate storage strategy is flexibly configured and selected, and the balance of performance, cost performance, reliability, and energy consumption is achieved.

[0092] The computing node 121 is also configured to monitor the resource usage (such as the available storage capacity) of the accessible storage medium, record the distribution of the intermediate data in the storage medium, and allocate the intermediate data stored by the storage medium of the multiple deployment modes. For example, if the available storage capacity of the first storage medium is less than or equal to a threshold value, S5100 is performed, and the computing node 121 stores the data of the first storage medium into the second storage medium. The access speed of the first storage medium is greater than that of the second storage medium.

[0093] If the available storage capacity of the first storage medium is greater than the threshold value, S5110 is performed, and the computing node 121 stores the data removed from the third storage medium into the first storage medium. The access speed of the first storage medium is greater than that of the second storage medium. For example, if the available storage capacity of the first storage medium is less than or equal to 90% of the storage capacity, S5100 is performed; if the available storage capacity of the first storage medium is greater than 70% of the storage capacity, S5110 is performed.

[0094] It should be noted that the computing node 121 can perform S520, S5100, and S5110 asynchronously, that is, the computing node 121 can use different processor cores to perform S520, S5100, and S5110. Therefore, before the computing node 121 performs S520, the intermediate data in the storage medium with the available storage capacity less than the threshold value can be removed, so as to increase the available storage capacity of the storage medium, so as to improve the processing efficiency of the intermediate data, and ensure that the intermediate data can be normally written into the storage medium.

[0095] In some embodiments, the computing node 121 can also remove the intermediate data according to the priority of the intermediate data. For example, the priority of the intermediate data can be determined according to the partition identifier of the partition to which the intermediate data belongs. The priority of the intermediate data is determined according to the order from small to large of the partition identifier, and the intermediate data corresponding to the smaller partition identifier is more likely to be removed to the storage medium with slower access speed, and the intermediate data corresponding to the smaller partition identifier is more likely to be removed to the storage medium with faster access speed.

[0096] Therefore, when the computing node 121 performs the reduce task, it can read the intermediate data from the faster storage medium with a higher probability, and try to give full play to the performance advantage of completing the shuffle on the faster storage medium.

[0097] The storage medium to which the removed data is accepted is not limited in the embodiments of the present application. For example, the local storage medium is moved to the global storage medium composed of the local storage medium, the extended global storage medium, or the extended local storage medium. For another example, the memory is moved to the hard disk. For another example, the DRAM is moved to the hard disk.

[0098] The computing node 121 can also perform S5120, that is, the computing node 121 sends the available storage capacity of the storage medium of the plurality of deployment modes to the control node 122, so that the control node 122 determines at least one storage medium of the deployment mode from the plurality of deployment modes according to the available storage capacity of the storage medium.

[0099] It can be understood that, in order to realize the functions in the above embodiments, the computing device includes the corresponding hardware structure and / or software module for performing each function. Those skilled in the art should easily realize that, in combination with the units and method steps of the examples described in the embodiments disclosed in the present application, the present application can be realized in the form of hardware or a combination of hardware and computer software. Whether a certain function is realized in the form of hardware or computer software driving hardware depends on the specific application scenario and design constraints of the technical solution.

[0100] The data processing method provided by the embodiments is described in detail above in combination with Figure 1 to Figure 6 , the control device and the data processing device provided by the embodiments will be described below in combination with Figure 7 and Figure 8 .

[0101] Figure 7 The structure schematic diagram of the possible control device provided by the embodiments is provided. These control devices can be used to realize the functions of the control device or the control node in the above method embodiments, and thus can also realize the beneficial effects possessed by the above method embodiments. In the embodiments, the control device can be, for example, Figure 5The control node 122 shown can also be a module (such as a chip) applied to a server.

[0102] like Figure 7 As shown, the control device 700 includes a communication module 710, a decision module 720, a configuration module 730, and a storage module 740. The control device 700 is used to implement the above-mentioned... Figure 5 The method embodiment shown controls the function of node 122.

[0103] Communication module 710 is used to acquire business requests from big data services, the business requests including business data; and to send control instructions to computing devices, the control instructions instructing the computing devices to perform map tasks in big data distributed processing on the business data, and to store the resulting intermediate data in a storage medium of at least one deployment mode. For example, communication module 710 is used to execute... Figure 5 S510 and S530.

[0104] Decision module 720 is configured to determine at least one type of storage medium from a plurality of deployment modes, including at least two of local storage media within a computing device, global storage media composed of local storage media, extended global storage media, and extended local storage media. For example, decision module 720 is configured to perform... Figure 5 S520.

[0105] Configuration module 730 is used to configure storage media and storage strategies for multiple deployment modes. Storage module 740 is used to store multiple deployment modes and storage strategies so that decision module 720 can determine the storage media for at least one deployment mode from the storage media of multiple deployment modes according to the storage strategy. Storage module 740 is also used to store metadata, which indicates the storage location of intermediate data. Communication module 710 is also used to send metadata to computing devices so that control devices can instruct computing devices executing reduce tasks to read intermediate data. For example, communication module 710 is used to execute... Figure 5 S580.

[0106] Optionally, the decision module 720 is specifically used to determine at least one storage medium of a deployment mode from among the storage media of multiple deployment modes according to the priority of the storage media of multiple deployment modes indicated by the storage policy, wherein the priority of the storage media of multiple deployment modes is determined based on the characteristics of the storage media.

[0107] Optionally, the decision module 720 is specifically used to determine, based on user requirements and the storage media characteristics of storage media in multiple deployment modes, at least one storage media in a deployment mode that meets user requirements, wherein user requirements are used to indicate requirements related to storage media characteristics.

[0108] Figure 8 A structure diagram of a possible data processing apparatus is provided in this embodiment. The data processing apparatus can be used to implement the functions of the computing device or the computing node in the method embodiments described above, and thus can also achieve the beneficial effects possessed by the method embodiments described above. In this embodiment, the data processing apparatus can be a computing node 121 as shown in Figure 5 , or a module (such as a chip) applied to a server.

[0109] As shown in Figure 8 , the data processing apparatus 800 includes a communication module 810, a data processing module 820, and a storage module 830. The data processing apparatus 800 is used to implement the functions of the computing node 121 in the method embodiments shown in Figure 5 .

[0110] The communication module 810 is configured to receive a control instruction sent by a control device, the control instruction including service data and an identifier of at least one deployment mode storage medium, the at least one deployment mode storage medium being determined by the control device from a plurality of deployment mode storage media according to a storage strategy, the storage strategy being used to indicate a method of selecting a storage medium according to a storage medium feature.

[0111] The data processing module 820 is configured to perform a map task in a big data distributed processing on the service data according to the control instruction, to obtain intermediate data, and to store the intermediate data in a storage medium indicated by the identifier of the at least one deployment mode storage medium. For example, the data processing module 820 is configured to perform S540 and S550 in Figure 5 .

[0112] The storage module 830 is configured to store available storage capacities of the plurality of deployment mode storage media, so as to send the available storage capacities of the plurality of deployment mode storage media to the control device. For example, the communication module 810 is configured to perform S560 in Figure 5 .

[0113] The storage module 830 is configured to store metadata, the metadata indicating storage locations of the intermediate data. So as to send the metadata to the control device, so that a computing device performing a reduce task reads the intermediate data from the control device according to the metadata.

[0114] The data processing module 820 is specifically configured to store a plurality of shard data of the intermediate data in a plurality of storage media of one deployment mode if the at least one deployment mode storage medium includes the plurality of storage media of the one deployment mode, or to store the plurality of shard data of the intermediate data in a plurality of storage media of two or more deployment modes if the at least one deployment mode storage medium includes the plurality of storage media of the two or more deployment modes.

[0115] The data processing module 820 is further configured to store the data of the first storage medium into the second storage medium if the available storage capacity of the first storage medium is less than the threshold, and the access speed of the first storage medium is greater than the access speed of the second storage medium.

[0116] Optionally, the communication module 810 is further configured to send the intermediate data request. For example, the communication module 810 is configured to perform the following steps in S570. Figure 6

[0117] Optionally, the data processing module 820 is further configured to acquire the intermediate data according to the metadata. For example, the data processing module 820 is configured to perform the following steps in S590. Figure 6

[0118] Optionally, the data processing module 820 is further configured to monitor the resource usage (e.g., the available storage capacity) of the accessible storage medium, record the distribution of the intermediate data in the storage medium, and allocate the intermediate data stored in the storage medium of the plurality of deployment modes. For example, the data processing module 820 is configured to perform the following steps in S5100 and S5110. Figure 6

[0119] Optionally, the communication module 810 is further configured to send the available storage capacity of the storage medium of the plurality of deployment modes. For example, the communication module 810 is configured to perform the following steps in S5120. Figure 6

[0120] It should be understood that the control device 700 or the data processing device 800 of the embodiments of the present application can be implemented by an application-specific integrated circuit (ASIC) or a programmable logic device (PLD), which can be a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof. The data processing method shown in the above embodiments of the present application can also be implemented by software, and the control device 700 and each module thereof can also be software modules, and the data processing device 800 and each module thereof can also be software modules. Figure 5 Figure 6

[0121] The control device 700 or the data processing device 800 according to the embodiments of the present application can correspond to the method described in the embodiments of the present application, and the above and other operations and / or functions of each unit in the control device 700 or the data processing device 800 are respectively implemented​​​​​​Figure 5 or Figure 6 The corresponding flow of each method in the above embodiments is not described here for brevity.

[0122] Figure 9 A structural schematic diagram of a computing device 900 is provided for the embodiment. As shown in the figure, the computing device 900 includes a processor 910, a bus 920, a memory 930, a communication interface 940, and a memory unit 950 (which can also be referred to as a main memory unit). The processor 910, the memory 930, the memory unit 950, and the communication interface 940 are connected through the bus 920.

[0123] It should be understood that, in the embodiment, the processor 910 can be a CPU, and the processor 910 can also be other general-purpose processors, digital signal processors (DSPs), ASICs, FPGAs, or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor, etc.

[0124] The processor can also be a graphics processing unit (GPU), a neural network processing unit (NPU), a microprocessor, an ASIC, or one or more integrated circuits for controlling program execution of the solutions of the present application.

[0125] The communication interface 940 is configured to implement communication between the computing device 900 and external devices or devices. In the embodiment, the computing device 900 is configured to implement the functions of the control node 122 shown in the above embodiments, and the communication interface 940 is configured to send a control instruction indicating that the computing node 121 performs a map task in a big data distributed processing of service data and stores the obtained intermediate data to at least one storage medium of a deployment mode. Figure 5 Figure 5 or Figure 6 The communication interface 940 is configured to receive a control instruction and send the available storage capacity of the storage medium of the deployment mode to the control node 122 when the computing device 900 is configured to implement the functions of the computing node 121 shown in the above embodiments.

[0126] ​The bus 920 can include a pathway for conveying information among the aforementioned components (e.g., the processor 910, the memory unit 950, and the storage 930). The bus 920 can include, in addition to a data bus, a power bus, a control bus, and a state signal bus, etc. However, for the sake of clarity, the various buses are labeled as the bus 920 in the figure. The bus 920 can be a Peripheral Component Interconnect Express (PCIe) bus, or an extended industry standard architecture (EISA) bus, a unified bus (Ubus or UB), a compute express link (CXL), a cache coherent interconnect for accelerators (CCIX), etc. The bus 920 can be divided into an address bus, a data bus, a control bus, etc.

[0127] As an example, the computing device 900 can include multiple processors. The processor can be a multi-CPU processor. The processor herein can refer to one or more devices, circuits, and / or computing units for processing data (e.g., computer program instructions). In the present embodiment, the computing device 900 is configured to implement the functions of the control node 122 shown. Figure 5 When the processor 910 is configured to implement the functions of the control node 122 shown, the processor 910 can invoke the storage strategy stored in the storage 930, and determine at least one deployment mode of storage medium from multiple deployment modes of storage medium according to the storage strategy. The computing device 900 is configured to implement the functions of the control node 122 shown. Figure 5 When the processor 910 is configured to implement the functions of the computing node 121 shown, the processor 910 can perform a map task in a big data distributed processing on the service data according to the control indication sent by the control node 122, obtain intermediate data, and store the intermediate data to the storage medium indicated by the identification indication of the at least one deployment mode of storage medium.

[0128] It is worth noting that, Figure 9 The computing device 900 is only taken as an example including one processor 910 and one storage 930, and here, the processor 910 and the storage 930 are respectively used to indicate a type of device or equipment. In specific embodiments, the number of each type of device or equipment can be determined according to business needs.

[0129] The memory unit 950 can correspond to the storage medium used for storing information such as the deployment mode and the storage policy in the above method embodiments. The memory unit 950 can be a volatile memory or a non-volatile memory, or can include both volatile and non-volatile memories. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically EPROM (EEPROM), or a flash memory. The volatile memory can be a random access memory (RAM) used as an external cache. By way of example but not limitation, many forms of RAM can be used, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous dynamic RAM (SDRAM), double data rate synchronous dynamic RAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchlink DRAM (SLDRAM), and direct rambus RAM (DR RAM).

[0130] The memory 930 can correspond to the storage medium used for storing information such as computer instructions and storage policies in the above method embodiments, for example, a magnetic disk such as a mechanical hard disk or a solid state disk.

[0131] The above computing device 900 can be a general-purpose device or a special-purpose device. For example, the computing device 900 can be an edge device (e.g., a box carrying a processing-capable chip) and the like. Alternatively, the computing device 900 can also be a server or other computing-capable device.

[0132] It should be understood that the computing device 900 according to the present embodiment can correspond to the control apparatus 700 or the data processing apparatus 800 in the present embodiment, and can correspond to the respective subject performing the method in any one of Figure 5 or Figure 6 The above and other operations and / or functions of the respective modules in the control apparatus 700 or the data processing apparatus 800 are respectively for realizing the respective processes of the methods in Figure 5 or Figure 6 for brevity, will not be repeated here.

[0133] The steps of a method or algorithm described in connection with the present disclosure can be embodied directly in hardware, in a software module executed by a processor, or in a combination of the two. A software module can reside in RAM, flash memory, ROM, EPROM, EEPROM, registers, hard disk, a removable media, or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor such that the processor can read information from, and write information to, the storage medium. In the alternative, the storage medium can be integral to the processor. The processor and the storage medium can reside in an ASIC. Alternatively, the processor and the storage medium can reside as discrete components in a computing device. Moreover, in some embodiments, the processor and the storage medium can reside as discrete components in a computing device.

[0134] In the above embodiments, all or part of the embodiments can be implemented by software, hardware, firmware or any combination thereof. When implemented by software, all or part of the embodiments can be implemented in the form of a computer program product. The computer program product includes one or more computer programs or instructions. When the computer programs or instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of the present application are performed. The computer can be a general purpose computer, a special purpose computer, a computer network, a network device, a user equipment or other programmable apparatus. The computer programs or instructions can be stored in a computer readable storage medium or transmitted from one computer readable storage medium to another computer readable storage medium, for example, the computer programs or instructions can be transmitted from one website site, computer, server or data center to another website site, computer, server or data center through wired or wireless manner. The computer readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server, data center and the like integrated with one or more available media. The available media can be a magnetic medium, for example, a floppy disk, a hard disk, a magnetic tape; or an optical medium, for example, a digital video disc (digital video disc, DVD); or a semiconductor medium, for example, a solid state disk (solid state drive, SSD). The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto, any person skilled in the art can easily think of various equivalent modifications or replacements within the technical range disclosed in the present application, and these modifications or replacements should be covered in the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A data processing method, characterized by, The method comprises: The control device acquires a service request of a big data service, and the service request comprises service data; The control device determines a storage strategy according to the service request, and determines at least one deployment mode from a plurality of deployment modes configured by the computing device according to the storage strategy, each of the plurality of deployment modes corresponding to a storage medium with different performance; The storage medium comprises at least two of a local storage medium in the computing device, a global storage medium composed of the local storage medium, an extended global storage medium, and an extended local storage medium; wherein the global storage medium refers to a storage medium accessible by all computing nodes in the system; The control device sends a control instruction to the computing device, the control instruction being used to instruct the computing device to perform a map task in big data distributed processing on the service data, and store intermediate data obtained by processing the map task to a storage medium corresponding to the at least one deployment mode; The determination of the at least one deployment mode from the plurality of deployment modes configured by the computing device according to the storage strategy comprises: According to the priority of the storage medium of the plurality of deployment modes indicated by the storage strategy, the storage medium of the at least one deployment mode is determined from the storage medium of the plurality of deployment modes, and the priority of the storage medium of the plurality of deployment modes is determined according to the storage medium feature; Or, According to the user demand and the storage medium feature of the storage medium of the plurality of deployment modes, the storage medium of the at least one deployment mode satisfying the user demand is determined from the storage medium of the plurality of deployment modes, and the user demand is used to indicate the demand related to the storage medium feature.

2. The method of claim 1, wherein, The storage strategy is used to indicate a method of selecting a storage medium according to a storage medium feature, and the storage medium feature comprises at least one of write latency, read latency, total storage capacity, available storage capacity, access speed, CPU consumption, energy consumption ratio, and reliability.

3. The method according to claim 1 or 2, characterized in that, The storage medium comprises memory, hard disk, memory server, and SCM.

4. The method according to claim 1 or 2, characterized in that, After the control instruction is sent to the computing device, the method further comprises: Receiving a storage location of the intermediate data in the storage medium of the at least one deployment mode sent by the computing device, and the storage location of the intermediate data is used to read the intermediate data when performing a reduce task.

5. A data processing method, characterized by, The method comprises: The computing device receives a control indication sent by a control device, the control indication comprising service data and an identification of at least one deployment mode of a storage medium of the computing device, the storage medium of the computing device having a plurality of configuration modes; wherein the at least one deployment mode is determined according to a priority of the storage medium of the plurality of deployment modes indicated by a storage strategy, the priority of the storage medium of the plurality of deployment modes being determined according to a storage medium feature, or the at least one deployment mode is determined according to a user demand and a storage medium feature of the storage medium of the plurality of deployment modes, the user demand being used to indicate a demand related to the storage medium feature; the storage strategy being determined according to a service request of a big data service, the service request comprising service data; the storage medium comprising at least two of a local storage medium in the computing device, a global storage medium composed of local storage mediums, an extended global storage medium and an extended local storage medium; wherein the global storage medium refers to a storage medium accessible by all computing nodes in a system, each of the plurality of deployment modes corresponding to a storage medium with different performance; The computing device performs a map task in big data distributed processing on the service data according to the control indication, to obtain intermediate data; The computing device stores the intermediate data into a storage medium corresponding to the at least one deployment mode indicated by the identification of the at least one deployment mode.

6. The method of claim 5, wherein, Storing the intermediate data into the storage medium corresponding to the at least one deployment mode indicated by the identification of the at least one deployment mode comprises: If the storage medium of the at least one deployment mode comprises a plurality of storage mediums of one deployment mode, storing a plurality of shard data of the intermediate data into the plurality of storage mediums, respectively; or, if the storage medium of the at least one deployment mode comprises storage mediums of two or more deployment modes, storing a plurality of shard data of the intermediate data into the storage mediums of the two or more deployment modes, respectively. The method further comprises:

7. The method according to claim 5 or 6, characterized in that, If an available storage capacity of a first storage medium is less than a threshold value, storing data of the first storage medium into a second storage medium, the access speed of the first storage medium being greater than that of the second storage medium. The method further comprises:

8. The method of claim 7, wherein, Sending the available storage capacity of the storage medium of the plurality of deployment modes to the control device. Comprise:

9. A control device characterized by comprising: A communication module configured to obtain a service request of a big data service, the service request comprising service data; and a decision module configured to determine a storage strategy according to the service request, and determine at least one deployment mode from a plurality of deployment modes of a computing device according to the storage strategy; each of the plurality of deployment modes corresponding to a storage medium with different performance, the storage medium comprising at least two of a local storage medium in the computing device, a global storage medium composed of local storage mediums, an extended global storage medium and an extended local storage medium; wherein the global storage medium refers to a storage medium accessible by all computing nodes in a system. ​ The communication module is further configured to send a control instruction to the computing device, where the control instruction is used to instruct the computing device to perform a map task in big data distributed processing on the service data and store obtained intermediate data to the storage medium of the at least one deployment mode. When the decision module determines the at least one deployment mode from the multiple deployment modes of the computing device according to the storage strategy, the decision module is specifically configured to: determine the storage medium of the at least one deployment mode from the storage media of the multiple deployment modes according to a priority of the storage media of the multiple deployment modes indicated by the storage strategy, where the priority of the storage media of the multiple deployment modes is determined according to the storage medium features. Or, determine the storage medium of the at least one deployment mode from the storage media of the multiple deployment modes according to a user demand and the storage medium features of the storage media of the multiple deployment modes, where the user demand is used to instruct a demand related to the storage medium features.

10. The apparatus of claim 9, wherein, The storage strategy is used to instruct a method of selecting a storage medium according to a storage medium feature, and the storage medium feature includes at least one of a write latency, a read latency, a total storage capacity, an available storage capacity, an access speed, a CPU (Central Processing Unit) consumption, an energy consumption ratio, and reliability.

11. The apparatus of claim 9 or 10, wherein, The storage medium includes a memory, a hard disk, a memory server, and an SCM (Storage Class Memory).

12. The apparatus of claim 9 or 10, wherein The communication module is further configured to receive a storage location of the intermediate data in the storage medium of the at least one deployment mode sent by the computing device, where the storage location of the intermediate data is used to read the intermediate data when a reduce task is performed.

13. A data processing apparatus, characterized by The data processing apparatus is configured with storage media of multiple deployment modes, and the apparatus includes: a communication module configured to receive a control instruction sent by a control device, where the control instruction includes service data and an identification of at least one deployment mode of a storage medium of a computing device, the storage medium of the computing device has multiple configuration modes; the at least one deployment mode is determined by the control device from the multiple deployment modes according to a storage strategy, the storage strategy is determined according to a service request of big data service, the service request includes service data; the storage media of the multiple deployment modes include at least two of local storage media in the computing device, global storage media composed of the local storage media, extended global storage media, and extended local storage media; where the global storage media refers to storage media that can be accessed by computing nodes in a system; each of the multiple deployment modes corresponds to storage media with different performance; a data processing module configured to perform a map task in big data distributed processing on the service data according to the control instruction, obtain intermediate data, and store the intermediate data to a storage medium indicated by the identification of the at least one deployment mode of the storage medium. The at least one deployment mode is determined by the control device from the plurality of deployment modes according to a storage strategy, including: the at least one deployment mode is determined according to a priority of storage media of the plurality of deployment modes indicated by the storage strategy, the priority of storage media of the plurality of deployment modes is determined according to the storage media characteristics, or the at least one deployment mode is determined according to user demand and storage media characteristics of storage media of the plurality of deployment modes, the user demand is used to indicate the demand related to the storage media characteristics.

14. The apparatus of claim 13, wherein, The data processing module stores the intermediate data to the storage medium indicated by the identifier of the storage medium of the at least one deployment mode, and is specifically used for: If the storage medium of the at least one deployment mode contains a plurality of storage media of one deployment mode, the plurality of fragment data of the intermediate data is respectively stored to the plurality of storage media. Or, if the storage medium of the at least one deployment mode contains storage media of two or more deployment modes, the plurality of fragment data of the intermediate data is respectively stored to the storage media of the two or more deployment modes.

15. The apparatus of claim 13 or 14, wherein, The data processing module is further configured to, if the available storage capacity of the first storage medium is less than a threshold value, store data of the first storage medium into a second storage medium, the access speed of the first storage medium being greater than the access speed of the second storage medium.

16. The apparatus of claim 15, wherein, The communication module is further configured to send the available storage capacity of the plurality of deployment modes of storage media to the control device.

17. A control device, characterized by The control device includes a memory and at least one processor, the memory is used to store a set of computer instructions; when the processor executes the set of computer instructions, the operation steps of the method of any one of claims 1-4 are performed.

18. A computing device, comprising: The computing device includes a memory, at least one processor and a display, the memory is used to store a set of computer instructions; when the processor executes the set of computer instructions, the operation steps of the method of any one of claims 5-8 are performed.

19. A system, comprising: The system includes the control device of claim 17 and the computing device of claim 18, the control device is used to perform the operation steps of the method of any one of claims 1-4, and the computing device is used to perform the operation steps of the method of any one of claims 5-8.

Images