Data processing method, device, equipment and system

Abstract

The invention discloses a data processing method, device, equipment and system, and relates to the field of big data. The control device is used for controlling the computing device to execute big data distributed processing. The computing device is configured with storage media of multiple deployment modes, and the storage media of the multiple deployment modes comprise at least two of a local storage medium in the computing device, a global storage medium formed based on the local storage medium, an extended global storage medium and an extended local storage medium. The method comprises the steps that a control device obtains a service request of a big data service, determines a storage medium of at least one deployment mode from storage media of multiple deployment modes, sends a control instruction to a computing device, and instructs the computing device to execute a map task in big data distributed processing on service data included in the service request, and storing the obtained intermediate data in a storage medium of at least one deployment mode. Therefore, the storage space of the storage medium in the computing device occupied by storing the intermediate data is reduced.

Description

[0001] This application claims priority to the Chinese patent application filed with the State Intellectual Property Office on October 22, 2021, with application number 202111234307.4 and application name “Distributed System”, all contents of which are incorporated by reference in this application.

[0002] This application is a divisional application. The application number of the original application is 202111583902.9, and the original application date is December 22, 2021. The entire contents of the original application are incorporated into this application by reference. Technical Field

[0003] The present application relates to the field of big data, and in particular to a data processing method, device, equipment and system. Background Art

[0004] Big data is a massive, high-growth, and diverse information asset that requires innovative processing models to have stronger decision-making power, insight, and process optimization capabilities. Usually, multiple computing devices perform distributed processing on big data based on the MapReduce model. Since the data is distributed on different computing devices, the computing device that performs the map task stores the intermediate data obtained after the map task. The computing device that performs the reduce task may read the intermediate data from a remote end. This data exchange process is called shuffle. Currently, computing devices store intermediate data in the memory or hard disk within the computing device. In the scenario of distributed processing of big data based on the MapReduce model, too much storage space of the storage medium in the computing device is occupied. Summary of the invention

[0005] The present application provides a data processing method, apparatus, device and system, thereby reducing the storage space occupied by the storage medium in the computing device.

[0006] In a first aspect, a data processing method is provided, the method being executed by a control device, the control device being used to control a computing device to execute distributed processing of big data, the computing device being configured with storage media of multiple deployment modes, the storage media of multiple deployment modes including at least two of a local storage medium in the computing device, a global storage medium composed of local storage media, an extended global storage medium, and an extended local storage medium. The method comprises the control device acquiring a service request of a big data service, determining a storage medium of at least one deployment mode from the storage media of multiple deployment modes, sending a control instruction to the computing device, instructing the computing device to execute a map task in distributed processing of big data on the service data included in the service request, and storing the obtained intermediate data in a storage medium of at least one deployment mode.

[0007] In this way, the control device can select a storage medium for storing intermediate data from storage media in a variety of deployment modes to reduce the storage space occupied by the storage medium (such as memory or disk) in the computing device for storing intermediate data, thereby releasing more available storage space for the computing device to execute map tasks or reduce tasks, thereby improving the processing efficiency of the computing device in processing map tasks or reduce tasks.

[0008] In a possible implementation, determining at least one storage medium of a deployment mode from storage media of multiple deployment modes includes: controlling a device to determine at least one storage medium of a deployment mode from storage media of multiple deployment modes according to a storage policy. The storage policy is used to indicate a method of selecting a storage medium according to storage medium characteristics. The storage medium characteristics include at least one of write latency, read latency, total storage capacity, available storage capacity, access speed, central processing unit (CPU) consumption, energy consumption ratio, and reliability.

[0009] Example 1: determining at least one storage medium with a deployment mode from storage media with a plurality of deployment modes according to a storage policy, including: controlling a device to determine at least one storage medium with a deployment mode from storage media with a plurality of deployment modes according to the priorities of the storage media with a plurality of deployment modes indicated by the storage policy, wherein the priorities of the storage media with a plurality of deployment modes are determined based on characteristics of the storage medium.

[0010] Example 2: Determine at least one storage medium of a deployment mode from storage media of multiple deployment modes according to a storage strategy, including: the control device determines at least one storage medium of a deployment mode that meets the user's needs from the storage media of multiple deployment modes according to the user's needs and the storage medium characteristics of the storage media of multiple deployment modes, wherein the user's needs are used to indicate the needs related to the storage medium characteristics. In this way, when the control device selects a storage medium for storing intermediate data from the storage media of multiple deployment modes, based on the user's needs for characteristics such as access speed or reliability, the storage medium of at least one deployment mode that meets the user's needs is dynamically selected, thereby ensuring the performance of data processing (such as data processing rate) and reliability of the scenarios compared to using a storage medium of one deployment mode to store intermediate data of different scenarios.

[0011] Storage media include memory, hard disk, memory server and storage-class-memory (SCM). In this way, the introduction of memory server and SCM in the big data distributed processing system expands the storage media for storing intermediate data, making the storage media for storing intermediate data more possible, reducing the storage space occupied by the storage media in the computing device to store intermediate data, and improving the reliability of intermediate data.

[0012] It should be noted that the storage policy is configured at the granularity of the big data business, the granularity of the task in the big data business, or the granularity of the system global.

[0013] In another possible implementation, after sending a control instruction to the computing device, the method further includes: the control device receives the storage location of the intermediate data in the storage medium of at least one deployment mode sent by the computing device, and the storage location of the intermediate data is used to read the intermediate data when executing the reduce task. Furthermore, the control device receives the intermediate data request sent by the computing device, and the intermediate data request is for the storage location where the intermediate data is located. The control device sends metadata indicating the storage location of the intermediate data to the computing device, and the metadata indicates the storage location where the intermediate data is located. The computing device obtains the intermediate data according to the metadata.

[0014] In a second aspect, a data processing method is provided, the method is executed by a computing device, the computing device is configured with storage media of multiple deployment modes, the computing device is connected to a control device, and the control device is used to control the computing device to perform big data distributed processing. The method includes the computing device executing a map task in big data distributed processing on business data according to a control instruction sent by the control device, obtaining intermediate data, and storing the intermediate data in a storage medium indicated by an identifier of a storage medium of at least one deployment mode. The control instruction includes business data and an identifier of a storage medium of at least one deployment mode, and the storage medium of at least one deployment mode is determined by the control device from storage media of multiple deployment modes according to a storage policy, and the storage policy is used to indicate a method for selecting a storage medium based on storage medium characteristics.

[0015] In this way, storage media with multiple deployment modes are added to the system, so that the control device can select a storage medium for storing intermediate data from the storage media with multiple deployment modes, and the computing device stores the intermediate data in a storage medium indicated by an identifier of at least one deployment mode of storage media according to the instruction of the control device, so as to reduce the storage space of the storage medium (such as memory or disk) in the computing device occupied for storing intermediate data, release more available storage space for the computing device to execute map tasks or reduce tasks, and improve the processing efficiency of the computing device in processing map tasks or reduce tasks.

[0016] In one possible implementation, storing intermediate data in a storage medium indicated by an identifier of a storage medium of at least one deployment mode includes: if the storage medium of at least one deployment mode includes multiple storage media of one deployment mode, storing multiple shards of the intermediate data in the multiple storage media respectively; or, if the storage medium of at least one deployment mode includes storage media of more than two deployment modes, storing multiple shards of the intermediate data in the storage media of more than two deployment modes respectively.

[0017] In another possible implementation, the storage medium of multiple deployment modes includes a first storage medium and a second storage medium, and the method further includes: if the available storage capacity of the first storage medium is less than a threshold, storing the data of the first storage medium into the second storage medium, and the access speed of the first storage medium is greater than the access speed of the second storage medium. Thus, after the intermediate data in the storage medium whose available storage capacity is less than the threshold is removed, the available storage capacity of the storage medium can be increased, so as to improve the processing efficiency of the intermediate data and ensure that the intermediate data can be normally written into the storage medium.

[0018] In another possible implementation, the method also includes: the computing device sends the available storage capacity of storage media in multiple deployment modes to the control device, so that the control device determines at least one storage medium in the multiple deployment modes from the storage media in the multiple deployment modes based on the available storage capacity of the storage medium.

[0019] In a third aspect, a control device is provided, comprising modules for executing the data processing method in the first aspect or any possible design of the first aspect.

[0020] According to a fourth aspect, a data processing device is provided, comprising modules for executing the data processing method according to the second aspect or any possible design of the second aspect.

[0021] In a fifth aspect, a control device is provided, which includes at least one processor and a memory, the memory being used to store a set of computer instructions; when the processor executes the set of computer instructions as an execution device in the first aspect or any possible implementation of the first aspect, the operation steps of the data processing method in the first aspect or any possible implementation of the first aspect are executed.

[0022] In a sixth aspect, a computing device is provided, which includes at least one processor and a memory, the memory being used to store a set of computer instructions; when the processor executes the set of computer instructions as an execution device in the second aspect or any possible implementation of the second aspect, the operating steps of the data processing method in the second aspect or any possible implementation of the second aspect are executed.

[0023] In a seventh aspect, a computer-readable storage medium is provided, comprising: computer software instructions; when the computer software instructions are executed in a computing device, the computing device executes the operating steps of the method described in the first aspect or any possible implementation of the first aspect, or executes the operating steps of the method described in the second aspect or any possible implementation of the second aspect.

[0024] In an eighth aspect, a computer program product is provided. When the computer program product runs on a computer, the computing device executes the operating steps of the method described in the first aspect or any possible implementation of the first aspect, or executes the operating steps of the method described in the second aspect or any possible implementation of the second aspect.

[0025] Based on the implementations provided in the above aspects, this application can also be further combined to provide more implementations. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 A schematic diagram of the architecture of a big data processing system provided in an embodiment of the present application;

[0027] Figure 2 A schematic diagram of a large data distributed processing provided in an embodiment of the present application;

[0028] Figure 3 A schematic diagram of a storage medium deployment scenario provided in an embodiment of the present application;

[0029] Figure 4 A schematic diagram of a three-layer storage system provided in an embodiment of the present application;

[0030] Figure 5 A schematic diagram of a process of writing intermediate data provided in an embodiment of the present application;

[0031] Figure 6 A schematic diagram of a process of reading intermediate data provided by an embodiment of the present application;

[0032] Figure 7 A schematic diagram of the structure of a control device provided in an embodiment of the present application;

[0033] Figure 8 A schematic diagram of the structure of a data processing device provided in an embodiment of the present application;

[0034] Fig. 9 A schematic diagram of the structure of a computing device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0035] With the development of the Internet, the Internet of Things, network bandwidth, smart terminals, cloud computing and other services, the data types and data scales have developed at an unprecedented speed, and big data has emerged. Data has changed from a single processing object to a basic resource. Big data is a collection of data that cannot be captured, managed and processed by conventional software tools within a period of time.

[0036] The sources of big data can include management systems, web systems, physical systems, and scientific experiment systems. Management systems include transaction processing systems and office automation systems. Web systems include Internet systems, such as search engines. Physical systems refer to systems about physical objects and physical processes, such as real-time monitoring, real-time detection, or production scheduling. Scientific experiment systems refer to physical systems based on experimental environments.

[0037] Big data can be classified according to data structure characteristics, including structured data, unstructured data and semi-structured data. Structured data refers to data represented by a unified structure, such as data stored in a database such as a financial system, enterprise resource planning system or customer relationship management system. Unstructured data refers to data with variable field lengths that are not easily expressed using structured data, such as audio, video or image data. Semi-structured data is data between structured data and unstructured data. Semi-structured data belongs to the same category and has different attributes, such as data in a mail system.

[0038] Since the large amount of data contained in big data has correlations, data analysis methods, models or tools are used to analyze big data, explore the data relationships in big data, and use the data relationships in big data to make predictions or decisions. For example, by analyzing user shopping trend data, users can be pushed items that they may buy to improve their shopping experience. Therefore, big data has the characteristics of large data volume, fast data growth rate, diverse data types and high utilization value.

[0039] Since the amount of big data is very large, a single computing node cannot meet the computing needs of big data. Usually, a distributed file system (DFS) is used to store and manage big data, and multiple computing nodes in a computing cluster are used to process big data services.

[0040] Figure 1 This is a schematic diagram of the architecture of a big data processing system provided in an embodiment of the present application. Figure 1 As shown, the big data processing system 100 includes a client 110 , a computing cluster 120 , and a storage cluster 130 .

[0041] The storage cluster 130 includes at least two storage nodes 131. A storage node 131 includes one or more controllers, a network card and multiple hard disks. The hard disk is used to store data. The hard disk can be a magnetic disk or other types of storage media, such as a solid-state hard disk or a shingled magnetic recording hard disk. The network card is used to communicate with the computing node 121 included in the computing cluster 120. The controller is used to write data to the hard disk or read data from the hard disk according to the read / write data request sent by the computing node 121. In the process of reading and writing data, the controller needs to convert the address carried in the read / write data request into an address that the hard disk can recognize.

[0042] The client 110 communicates with the computing cluster 120 and the storage cluster 130 through the network 140. For example, the client 110 sends a service request of a big data service to the computing cluster 120 through the network 140, requesting the computing cluster 120 to perform distributed processing on the service data included in the service request. The network 140 may refer to an internal network of an enterprise (such as a local area network (LAN)) or the Internet.

[0043] In some embodiments, the storage cluster 130 stores and manages big data based on the distributed file system 132 and the distributed database 133. The client 110 is installed with a client program 111. The client 110 runs the client program 111 to display a user interface (UI). The user 150 operates the user interface to access the distributed file system 132 and the distributed database 133 to obtain data, and instructs the computing cluster 120 to process the big data business. The client 110 refers to a computer connected to the network 140, which can also be called a workstation. Different clients can share resources on the network (such as computing resources and storage resources). In other embodiments, the system administrator 160 can access the distributed file system 132 and the distributed database 133 through the client 110 to configure system information, such as the deployment information and storage strategy of the storage medium of the various deployment modes configured for the computing node or storage node provided in the embodiments of the present application.

[0044] The computing cluster 120 includes at least two computing nodes 121, and the computing nodes 121 can communicate with each other. The computing node 121 is a computing device, such as a server, a desktop computer, or a controller of a storage array. The big data business submitted by the client 110 can be called a job. The job can be divided into multiple tasks, and multiple computing nodes execute multiple tasks in parallel. The completion of all tasks marks the completion of a job. A task is generally a processing process for a part of data or a stage in a job, and all tasks are scheduled to be completed in parallel or serially. In some embodiments, the computing cluster 120 performs distributed processing on big data services based on the map-reduce (MapReduce) model 134. The MapReduce model is a distributed programming model, which decomposes big data services into map tasks and reduce tasks, and multiple computing nodes 121 execute map tasks, collect processing results and execute reduce tasks. In some embodiments, such as Figure 2 As shown, the computing cluster 120 includes a control node 122 and at least two computing nodes 121. The control node and the computing node can be independent physical devices, and the control node can also be called a control device or a naming node. The computing node can be called a computing device or a data node. The control node 122 is used to manage the namespace of the distributed file system and the access of the client 110 to the distributed file system. In addition, the control node 122 indicates the computing node that executes the map task and the computing node that executes the reduce task. The computing node stores the intermediate data (or called map data or shuffle data) obtained by executing the map task based on the MapReduce model 134. The computing node that executes the reduce task reads the intermediate data. Optionally, the storage cluster 130 can also process the data according to the MapReduce model 134 and then store it when storing data.

[0045] An embodiment of the present application provides a data processing method, and in particular, provides a technology for distributed processing of big data, specifically providing a technology for selecting a storage medium for storing intermediate data from storage media in a variety of deployment modes, so as to reduce the storage space occupied by the storage medium (such as memory or disk) in the computing device for storing intermediate data, and release more available storage space for computing nodes to execute map tasks or reduce tasks.

[0046] The multiple deployment modes include a local deployment mode, a fusion deployment mode, and a separation deployment mode. The storage medium in the local deployment mode refers to a local storage medium. The storage medium in the fusion deployment mode refers to a global storage medium composed of local storage media. The storage medium in the separation deployment mode includes an extended global storage medium and an extended local storage medium. The storage medium in the multiple deployment modes includes at least two of a local storage medium in a computing device, a global storage medium composed of local storage media, an extended global storage medium, and an extended local storage medium. For example, Figure 3 As shown, it is a schematic diagram of a storage medium deployment scenario provided in an embodiment of the present application. The storage medium 310 inside each of the N computing nodes may refer to the storage medium in the local deployment mode. The global storage medium 320 includes storage media provided by at least two of the N computing nodes. The global storage medium refers to a storage medium that is accessible to all computing nodes in the system and has a global unified addressing. It should be understood that the storage capacity of the global storage medium is part of the storage capacity in the storage medium of the N computing nodes, and the storage capacity reserved in the storage medium of the N computing nodes is used by the computing nodes to process big data services.

[0047] The extended global storage medium 330 refers to a newly added storage medium connected to N computing nodes in the system, and the newly added storage medium has a global unified address, and the N computing nodes can access the extended global storage medium 330. The extended local storage medium 340 refers to a newly added storage medium connected to a computing node in the system, and the newly added storage medium is only accessed by the computing node connected to it.

[0048] It should be noted that memory is a memory device used to store programs and various data. The larger the memory capacity, the slower the access speed. Conversely, the smaller the memory capacity, the faster the access speed. Access speed refers to the data transmission speed when writing data to or reading data from the memory. Access speed can also be called read and write speed. Memories can be divided into different levels based on storage capacity and access speed.

[0049] For example, Figure 4 A schematic diagram of a three-layer storage system provided in an embodiment of the present application. From the first layer to the third layer, the storage capacity increases step by step, the access speed decreases step by step, and the cost decreases step by step. Figure 4As shown, the first level includes registers 411, a first-level cache 412, a second-level cache 413, and a third-level cache 414 located in a central processing unit (CPU). The memory included in the second level can be used as the main memory of the computer system. For example, dynamic random access memory (DRAM) 421, double data rate synchronous dynamic random access memory (DDR SDRAM) 422, and storage-class-memory (SCM) 423. The main memory can be referred to as main memory or memory, that is, the memory that exchanges information with the CPU. The memory included in the third level can be used as auxiliary memory of the computer system. For example, network memory 431, solid-state drive (Solid State Disk or Solid State Drive, SSD) 432, hard disk drive (HDD) 433. Auxiliary memory can be referred to as auxiliary memory or external memory. Compared with the main memory, the external memory has a large storage capacity and a slow access speed. It can be seen that the closer the memory is to the CPU, the smaller the capacity, the faster the access speed, the larger the bandwidth, and the lower the latency. Therefore, the memory included in the third level stores data that the CPU does not frequently access, improving data reliability. The memory included in the second level can be used as a cache device to store data that the CPU frequently accesses, significantly improving the access performance of the system.

[0050] The storage media in various deployment modes described in the embodiments of the present application include memory, hard disk, memory server and SCM. For example, as shown in Table 1, the storage media in various deployment modes provided in the embodiments of the present application.

[0051] Table 1

[0052]

[0053]

[0054] As can be seen from Table 1, the local storage medium includes memory and hard disk, the access speed of memory is higher than the access speed of hard disk, and the storage capacity of memory is lower than the storage capacity of hard disk. The global storage medium composed of local storage media includes DRAM and SCM, and the access speed of DRAM is higher than the access speed of SCM. The extended global storage medium and the extended local storage medium both include memory servers and high performance storage (HPS). Different storage media have different storage medium characteristics. The storage medium characteristics include at least one of write latency, read latency, total storage capacity, access speed, CPU consumption, energy consumption ratio and reliability. Write latency refers to the latency of computing node 121 writing intermediate data to the storage medium. Read latency refers to the latency of computing node 121 reading intermediate data from the storage medium. Storage capacity refers to the total storage capacity of the storage medium that can store data. Access speed refers to the speed at which computing node 121 performs read and write operations on the storage medium. CPU consumption refers to the occupancy rate of the CPU of computing node 121 when computing node 121 writes intermediate data to the storage medium or reads intermediate data from the storage medium. Energy consumption ratio refers to the energy consumed per unit time (such as electrical energy). Reliability refers to the persistence of data stored in the storage medium. The storage media with various deployment modes in Table 1 are just examples. In actual applications, you can configure storage media with various deployment modes according to business needs.

[0055] The following will describe in detail an implementation method of selecting a storage medium for storing intermediate data from storage media in multiple deployment modes provided by an embodiment of the present application in conjunction with the accompanying drawings.

[0056] Figure 5 A flow chart of a data processing method provided in an embodiment of the present application. Here, the client 110 and the computing cluster 120 are used as an example for explanation. The computing cluster 120 includes a control node 122 and at least two computing nodes 121. The control node 122 is used to control the computing node 121 to perform big data distributed processing. The computing node 121 is configured with storage media with multiple deployment modes. Figure 5 As shown, the method includes the following steps.

[0057] S510 : The control node 122 receives a service request for a big data service sent by the client 110 .

[0058] The client 110 responds to the user operation and sends a service request for the big data service to the control node 122. The user operation may refer to the operation of the user operating the big data user interface to submit the big data service. The control node 122 may receive the service request for the big data service sent by the client 110 through a local area network or the Internet. The service request may include a service identifier and service data. The service identifier is used to uniquely indicate a big data service. The service data may be data for the computing node to perform distributed processing of big data or identification data indicating data to be processed.

[0059] Big data services include data analysis services, data query services, and data modification services. For example, big data services refer to analyzing customers' personal data and purchase behavior data to describe user portraits and classify customers, so that targeted products or preferential products can be recommended to specific customers, improving customer satisfaction and consolidating customer relationships. Another example is that big data services refer to analyzing the historical sales volume of products to predict future sales volume, discovering the reasons for the decline or increase in sales volume, and recommending constructive suggestions to increase sales.

[0060] S520. The control node 122 determines a storage medium of at least one deployment mode from storage media of multiple deployment modes.

[0061] The control node 122 determines at least one storage medium of a deployment mode from storage mediums of multiple deployment modes according to the storage policy. For example, the storage medium of at least one deployment mode includes a global storage medium and an extended global storage medium. The storage policy is used to indicate a method for selecting a storage medium according to storage medium characteristics.

[0062] In some embodiments, the control node 122 is configured with priorities of storage media in multiple deployment modes determined according to characteristics of the storage media.

[0063] For example, the priority of storage media in various deployment modes is determined based on the access speed of the storage media. The access speed of the global storage medium is higher than the access speed of the memory, the access speed of the memory is higher than the access speed of the hard disk, the access speed of the hard disk is higher than the access speed of the extended local storage medium, and the access speed of the extended local storage medium is higher than the access speed of the extended global storage medium. Among the eight storage media shown in Table 1, storage medium (3) has the highest priority. The priority of the storage media is from high to low: DRAM priority, SCM priority, memory priority, hard disk priority, memory server priority, and high-performance storage priority.

[0064] For another example, the priority of storage media of multiple deployment modes is determined based on the priority of the deployment mode. The priority of the local deployment mode is higher than the priority of the fusion deployment mode, and the priority of the fusion deployment mode is higher than the priority of the separation deployment mode. For example, among the eight storage media shown in Table 1, the storage medium (1) has the highest priority, and the priority is reduced to storage medium (8) in sequence, that is, the priority of the storage media is from high to low as follows: the priority of memory, the priority of hard disk, the priority of DRAM, the priority of SCM, the priority of memory server, and the priority of high-performance storage. Optionally, the priority of the fusion deployment mode is higher than the priority of the local deployment mode. The priority of the local deployment mode is higher than the priority of the separation deployment mode. Among the eight storage media shown in Table 1, the storage medium (3) has the highest priority, and the priority of the storage media is from high to low as follows: the priority of DRAM, the priority of SCM, the priority of memory, the priority of hard disk, the priority of memory server, and the priority of high-performance storage.

[0065] For another example, the priority of storage media in multiple deployment modes can be determined based on the comprehensive characteristics of multiple storage media. The priority of storage media in multiple deployment modes is determined based on the priority of the deployment mode. The priority of storage media in the same deployment mode can be determined based on the access speed of the storage medium. Among the 8 storage media shown in Table 1, storage medium (1) has the highest priority, and the priority decreases to storage medium (8) in turn.

[0066] The control node 122 determines at least one storage medium of a deployment mode from the storage mediums of a plurality of deployment modes according to the priorities of the storage mediums of a plurality of deployment modes indicated by the storage policy. For example, the storage medium feature may also include available storage capacity. Available storage capacity refers to the remaining storage capacity obtained by subtracting the used storage capacity from the storage capacity of the storage medium. The control node 122 determines whether the available storage capacity of the storage medium of the highest priority is greater than a threshold value according to the priorities of the storage mediums of the plurality of deployment modes, starting from the storage medium of the highest priority. If the available storage capacity of the storage medium of the highest priority is greater than the threshold value, it indicates that the storage medium of the highest priority has excess storage space to store intermediate data, and the storage medium of the highest priority is selected to store the intermediate data; if the available storage capacity of the storage medium of the highest priority is less than or equal to the threshold value, it indicates that the storage medium of the highest priority has no excess storage space to store intermediate data, and then determines whether the available storage capacity of the storage medium of the second highest priority is greater than the threshold value, and sequentially traverses the storage mediums of the plurality of deployment modes, and finally determines at least one storage medium of a deployment mode from the storage mediums of the plurality of deployment modes. When the control node 122 selects a storage medium according to the priority of the deployment mode, the storage medium of at least one deployment mode may include multiple storage media of one deployment mode.

[0067] In other embodiments, the control node 122 determines at least one storage medium in a deployment mode that meets the user demand from the storage medium in a plurality of deployment modes based on the user demand and the storage medium characteristics of the storage medium in a plurality of deployment modes, where the user demand is used to indicate the demand related to the storage medium characteristics.

[0068] For example, the control node 122 is configured with storage medium characteristics of storage media with multiple deployment modes. The service request includes user requirements, and the user requirements indicate an access speed range or a specific access speed. The control node 122 determines at least one storage medium with a deployment mode that meets the user requirements from the storage media with multiple deployment modes. For example, the user requirements indicate the access speed of the memory. The control node 122 determines a local storage medium or a global storage medium composed of local storage media from the storage media with multiple deployment modes, such as at least one of memory, DRAM, and SCM.

[0069] For another example, the control node 122 is configured with an association relationship between a storage medium and a customer level. The service request includes a user requirement, and the user requirement indicates a first customer level. The control node 122 determines a storage medium associated with the first customer level from the association relationship according to the first customer level, and determines that the storage medium associated with the first customer level is used to store intermediate data.

[0070] In this way, when the control node 122 selects a storage medium for storing intermediate data from storage media with multiple deployment modes, it dynamically selects a storage medium with at least one deployment mode that meets the user's needs based on the user's needs for characteristics such as access speed or reliability. This ensures the performance and reliability of data processing scenarios compared to using a storage medium with one deployment mode to store intermediate data in different scenarios.

[0071] Optionally, the control node 122 may also select a storage medium with an available storage capacity greater than a threshold from storage media of at least one deployment mode that meets user requirements as the storage medium for storing intermediate data.

[0072] It should be noted that in actual applications, storage policies can be set according to business needs, scenario needs, or user needs. The above description of storage policies is just an example. For example, storage policies can also refer to data localization preferences, that is, intermediate data is preferentially stored in local storage media. For another example, storage policies can also refer to performance priority, storage capacity priority, cost priority, etc. according to the application's balance between performance and cost.

[0073] Before the control node 122 uses the storage policy to select the storage medium for storing the intermediate data, that is, before executing S520, the storage policy and the configuration information of the storage medium of the multiple deployment modes can be pre-configured. The storage policy provided in the embodiment of the present application is configured at the granularity of the big data business or the granularity of the task in the big data business or the global granularity of the system, then the control node 122 can use the storage policy when selecting the storage medium for storing the intermediate data in processing the big data business or processing the task in the big data business or processing the global task of the system.

[0074] In addition, the control node 122 may select an idle computing node 121 in the computing cluster 120 to execute the map task, or select a computing node 121 from the computing cluster 120 to execute the map task based on a balancing strategy.

[0075] S530 : The control node 122 sends a control instruction to at least one computing node 121 .

[0076] The control node 122 may divide the big data service into multiple map tasks, determine the computing node 121 that executes each map task, and determine the storage medium used by the computing node 121 that executes each map task to store intermediate data according to the method described in S520.

[0077] The control node 122 sends a control instruction to at least one computing node 121 that executes a map task, and the control instruction is used to instruct the computing node 121 to execute a map task in big data distributed processing on the business data, and store the obtained intermediate data in a storage medium of at least one deployment mode. The control instruction may include the business data and the identification of the storage medium of at least one deployment mode.

[0078] S540, the computing node 121 executes the map task in the big data distributed processing on the business data according to the control instruction sent by the control node 122 to obtain the intermediate data.

[0079] S550. The computing node 121 stores the intermediate data in a storage medium indicated by an identifier of the storage medium of at least one deployment mode.

[0080] For explanations on the storage medium of at least one deployment mode, please refer to the above description of S520, which will not be described in detail. The computing node 121 can write the intermediate data into the storage medium of at least one deployment mode through a gather interface.

[0081] Optionally, storage media of various deployment modes may be configured with different access interfaces, and computing node 121 may call the corresponding access interface of the storage medium to write intermediate data to the storage medium. For example, computing node 121 calls the access interface of the global storage medium to write intermediate data to the global storage medium. For another example, computing node 121 calls the access interface of the extended global storage medium to write intermediate data to the extended global storage medium.

[0082] If the storage medium of at least one deployment mode includes multiple storage media of one deployment mode, multiple shard data of the intermediate data are stored in the multiple storage media respectively. For example, the multiple storage media of one deployment mode include Figure 3 Alternatively, if at least one storage medium of the deployment mode includes storage mediums of more than two deployment modes, multiple shard data of the intermediate data are stored in the storage mediums of more than two deployment modes, for example, the storage mediums of more than two deployment modes include Figure 3 A global storage medium 320 and an expanded global storage medium 330 are shown.

[0083] S560. The computing node 121 sends the storage location of the intermediate data in the storage medium of at least one deployment mode to the control node 122.

[0084] The computing node 121 stores the intermediate data in a storage medium indicated by the identification of the storage medium of at least one deployment mode, and generates metadata, where the metadata indicates the storage location of the intermediate data. The computing node 121 sends the metadata to the control node 122 so that the control node 122 instructs the computing node 121 executing the reduce task to read the intermediate data, for example, Figure 6 As shown, the computing node 121 executes S570 and the control node 122 executes S580.

[0085] S570 . The computing node 121 sends an intermediate data request to the control node 122 .

[0086] The intermediate data request is used for the storage location where the intermediate data is located. For example, the intermediate data request includes a business identifier or an object identifier, and the object identifier is used to indicate the identifier of the data to be obtained.

[0087] S580 : The control node 122 sends metadata indicating the storage location of the intermediate data to the computing node 121 .

[0088] S590 . The computing node 121 obtains intermediate data according to the metadata.

[0089] The computing node 121 obtains the intermediate data through a gather interface according to the storage location of the intermediate data, and executes the reduce task according to the intermediate data. If the intermediate data is stored on multiple storage media, the computing node 121 obtains the shard data of the intermediate data from the multiple storage media.

[0090] It should be noted that the computing node that executes the map task and the computing node that executes the reduce task can be different computing nodes.

[0091] The data processing method provided in the embodiment of the present application expands the single deployment mode of the storage medium for storing intermediate data to multiple deployment modes, thereby realizing the expansion of the storage location for storing intermediate data; at the same time, according to the requirements of the business scenario, it flexibly configures and selects appropriate storage strategies to achieve a balance between performance, cost-effectiveness, reliability, and energy consumption.

[0092] The computing node 121 is also used to monitor the resource usage of the accessible storage medium (such as available storage capacity), record the distribution of intermediate data in the storage medium, and deploy intermediate data stored in storage media of various deployment modes. For example, if the available storage capacity of the first storage medium is less than or equal to the threshold, S5100 is executed, and the computing node 121 stores the data of the first storage medium into the second storage medium. The access speed of the first storage medium is greater than the access speed of the second storage medium.

[0093] If the available storage capacity of the first storage medium is greater than the threshold, S5110 is executed, and the computing node 121 stores the removed data from the second storage medium into the first storage medium. The access speed of the first storage medium is greater than the access speed of the second storage medium. For example, if the available storage capacity of the first storage medium is less than or equal to 90% of the storage capacity, S5100 is executed; if the available storage capacity of the first storage medium is greater than 70% of the storage capacity, S5110 is executed.

[0094] It should be noted that the computing node 121 can asynchronously execute S520, S5100 and S5110, that is, the computing node 121 can use different processor cores to execute S520, S5100 and S5110. Therefore, before the computing node 121 executes S520, after the intermediate data in the storage medium whose available storage capacity is less than the threshold is removed, the available storage capacity of the storage medium can be increased, so as to improve the processing efficiency of the intermediate data and ensure that the intermediate data can be normally written to the storage medium.

[0095] In some embodiments, the computing node 121 may also remove the intermediate data according to the priority of the intermediate data. For example, the priority of the intermediate data may be determined according to the partition identifier of the partition to which the intermediate data belongs. The priority of the intermediate data is determined according to the order of the partition identifier from small to large, and the intermediate data corresponding to the smaller partition identifier is easier to be removed to the storage medium with a slower access speed, and the intermediate data corresponding to the smaller partition identifier is easier to be moved into the storage medium with a faster access speed.

[0096] Therefore, when computing node 121 executes a reduce task, it can read intermediate data from a faster storage medium with a greater probability, and maximize the performance advantage of completing shuffle on a faster storage medium.

[0097] The embodiments of the present application do not limit the storage medium that receives the data to be moved out. For example, the data is moved from a local storage medium to a global storage medium composed of local storage medium, an extended global storage medium, or an extended local storage medium. For another example, the data is moved from a memory to a hard disk. For another example, the data is moved from a DRAM to a hard disk.

[0098] The computing node 121 can also execute S5120, that is, the computing node 121 sends the available storage capacity of storage media in multiple deployment modes to the control node 122, so that the control node 122 can determine at least one storage medium in a deployment mode from the storage media in multiple deployment modes according to the available storage capacity of the storage medium.

[0099] It is understandable that in order to implement the functions in the above embodiments, the computing device includes hardware structures and / or software modules corresponding to the execution of each function. Those skilled in the art should easily realize that, in combination with the units and method steps of each example described in the embodiments disclosed in this application, the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application scenario and design constraints of the technical solution.

[0100] Combined with the above Figures 1 to 6 , describes in detail the data processing method provided by this embodiment, and will now be combined with Figure 7 and Figure 8 , describing the control device and data processing device provided according to this embodiment.

[0101] Figure 7 Schematic diagram of the structure of possible control devices provided in this embodiment. These control devices can be used to implement the functions of the control device or control node in the above method embodiment, so as to achieve the beneficial effects of the above method embodiment. In this embodiment, the control device can be as follows Figure 5The control node 122 shown may also be a module (such as a chip) applied to a server.

[0102] like Figure 7 As shown, the control device 700 includes a communication module 710, a decision module 720, a configuration module 730 and a storage module 740. The control device 700 is used to implement the above Figure 5 The method embodiment shown in FIG. 1 controls the functionality of node 122 .

[0103] The communication module 710 is used to obtain a business request for a big data business, the business request includes business data; and send a control instruction to a computing device, the control instruction is used to instruct the computing device to perform a map task in big data distributed processing on the business data, and store the obtained intermediate data in a storage medium of at least one deployment mode. For example, the communication module 710 is used to execute Figure 5 Among S510 and S530.

[0104] The decision module 720 is used to determine at least one storage medium of a deployment mode from the storage mediums of multiple deployment modes, wherein the storage mediums of the multiple deployment modes include at least two of a local storage medium in a computing device, a global storage medium composed of local storage media, an extended global storage medium, and an extended local storage medium. For example, the decision module 720 is used to execute Figure 5 Medium S520.

[0105] Configuration module 730 is used to configure storage media and storage strategies for multiple deployment modes. Storage module 740 is used to store multiple deployment modes and storage strategies so that decision module 720 can determine at least one storage medium for a deployment mode from storage media for multiple deployment modes according to the storage strategy. Storage module 740 is also used to store metadata, which indicates the storage location of intermediate data. Communication module 710 is also used to send metadata to computing devices so that the control device can instruct computing devices that execute reduce tasks to read intermediate data. For example, communication module 710 is used to execute Figure 5 Medium S580.

[0106] Optionally, the decision module 720 is specifically used to determine at least one storage medium of a plurality of deployment modes from the storage media of a plurality of deployment modes according to the priorities of the storage media of a plurality of deployment modes indicated by the storage policy, and the priorities of the storage media of a plurality of deployment modes are determined based on the characteristics of the storage media.

[0107] Optionally, the decision module 720 is specifically used to determine at least one storage medium in a deployment mode that meets user needs from storage media in multiple deployment modes based on user needs and storage medium characteristics of storage media in multiple deployment modes, where user needs are used to indicate needs related to storage medium characteristics.

[0108] Figure 8 The following is a schematic diagram of the structure of a possible data processing device provided in this embodiment. These data processing devices can be used to implement the functions of the computing device or computing node in the above method embodiment, and thus can also achieve the beneficial effects of the above method embodiment. In this embodiment, the data processing device can be as follows: Figure 5 The computing node 121 shown may also be a module (such as a chip) applied to a server.

[0109] like Figure 8 As shown, the data processing device 800 includes a communication module 810, a data processing module 820 and a storage module 830. The data processing device 800 is used to implement the above Figure 5 The function of computing node 121 in the method embodiment shown in FIG.

[0110] The communication module 810 is used to receive control instructions sent by the control device, the control instructions include business data and an identifier of a storage medium of at least one deployment mode, the storage medium of at least one deployment mode is determined by the control device from storage media of multiple deployment modes according to a storage strategy, and the storage strategy is used to indicate a method for selecting a storage medium based on storage medium characteristics.

[0111] The data processing module 820 is used to perform a map task in the big data distributed processing on the business data according to the control instruction, obtain intermediate data, and store the intermediate data in a storage medium indicated by the identification of the storage medium of at least one deployment mode. Figure 5 Among S540 and S550.

[0112] The storage module 830 is used to store the available storage capacity of the storage medium in various deployment modes, so as to send the available storage capacity of the storage medium in various deployment modes to the control device. Figure 5 Medium S560.

[0113] The storage module 830 is used to store metadata, which indicates the storage location of the intermediate data, so as to send the metadata to the control device so that the computing device executing the reduce task obtains the metadata from the control device and reads the intermediate data.

[0114] The data processing module 820 is specifically used to store multiple shards of intermediate data in multiple storage media respectively if the storage medium of at least one deployment mode includes multiple storage media of one deployment mode; or to store multiple shards of intermediate data in storage media of more than two deployment modes respectively if the storage medium of at least one deployment mode includes storage media of more than two deployment modes.

[0115] The data processing module 820 is also used to store the data of the first storage medium into the second storage medium if the available storage capacity of the first storage medium is less than a threshold, and the access speed of the first storage medium is greater than the access speed of the second storage medium.

[0116] Optionally, the communication module 810 is also used to send an intermediate data request. For example, the communication module 810 is used to execute Figure 6 Medium S570.

[0117] Optionally, the data processing module 820 is further configured to obtain intermediate data according to the metadata. Figure 6 Medium S590.

[0118] Optionally, the data processing module 820 is also used to monitor the resource usage of the accessible storage medium (such as available storage capacity), record the distribution of intermediate data in the storage medium, and allocate the intermediate data stored in the storage medium of various deployment modes. Figure 6 Among S5100 and S5110.

[0119] Optionally, the communication module 810 is also used to send the available storage capacity of the storage medium in multiple deployment modes. Figure 6 Medium S5120.

[0120] It should be understood that the control device 700 or the data processing device 800 of the embodiment of the present application can be implemented by an application-specific integrated circuit (ASIC) or a programmable logic device (PLD), and the above-mentioned PLD can be a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL) or any combination thereof. It can also be implemented by software Figure 5 or Figure 6 When the data processing method is shown, the control device 700 and its various modules may also be software modules, and the data processing device 800 and its various modules may also be software modules.

[0121] The control device 700 or the data processing device 800 according to the embodiment of the present application may correspond to executing the method described in the embodiment of the present application, and the above and other operations and / or functions of each unit in the control device 700 or the data processing device 800 are respectively to implement Figure 5 or Figure 6 For the sake of brevity, the corresponding processes of each method in are not repeated here.

[0122] Fig. 9 A schematic diagram of the structure of a computing device 900 provided in this embodiment. As shown in the figure, the computing device 900 includes a processor 910, a bus 920, a memory 930, a communication interface 940, and a memory unit 950 (also referred to as a main memory unit). The processor 910, the memory 930, the memory unit 950, and the communication interface 940 are connected via a bus 920.

[0123] It should be understood that in this embodiment, the processor 910 may be a CPU, and the processor 910 may also be other general-purpose processors, digital signal processors (DSP), ASICs, FPGAs or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0124] The processor may also be a graphics processing unit (GPU), a neural network processing unit (NPU), a microprocessor, an ASIC, or one or more integrated circuits for controlling the execution of the program of the present application.

[0125] The communication interface 940 is used to implement the communication between the computing device 900 and an external device or component. Figure 5 When the function of the control node 122 is shown, the communication interface 940 is used to send a control instruction to instruct the computing node 121 to perform a map task in the big data distributed processing on the business data and store the obtained intermediate data in a storage medium of at least one deployment mode. The computing device 900 is used to implement Figure 5 or Figure 6 When the computing node 121 functions as shown, the communication interface 940 is used to receive control instructions and send available storage capacity of storage media in multiple deployment modes to the control node 122.

[0126] The bus 920 may include a path for transmitting information between the above components (such as the processor 910, the memory unit 950, and the memory 930). In addition to the data bus, the bus 920 may also include a power bus, a control bus, and a status signal bus. However, for the sake of clarity, various buses are labeled as bus 920 in the figure. The bus 920 may be a Peripheral Component Interconnect Express (PCIe) bus, or an extended industry standard architecture (EISA) bus, a unified bus (Ubus or UB), a computer express link (CXL), a cache coherent interconnect for accelerators (CCIX), etc. The bus 920 can be divided into an address bus, a data bus, a control bus, etc.

[0127] As an example, the computing device 900 may include multiple processors. The processor may be a multi-core (multi-CPU) processor. The processor here may refer to one or more devices, circuits, and / or computing units for processing data (such as computer program instructions). In this embodiment, the computing device 900 is used to implement Figure 5 When the function of the control node 122 is executed, the processor 910 may call the storage policy stored in the memory 930, and determine at least one storage medium of the deployment mode from the storage mediums of the multiple deployment modes according to the storage policy. The computing device 900 is used to implement Figure 5 When the function of the computing node 121 is shown, the processor 910 can perform the map task in big data distributed processing on the business data according to the control instructions sent by the control node 122, obtain intermediate data, and store the intermediate data in the storage medium indicated by the identification of the storage medium of at least one deployment mode.

[0128] It is worth mentioning that Fig. 9 In the example, a computing device 900 including one processor 910 and one memory 930 is taken. Here, the processor 910 and the memory 930 are respectively used to indicate a type of device or equipment. In a specific embodiment, the number of each type of device or equipment can be determined according to business requirements.

[0129] The memory unit 950 may correspond to a storage medium for storing information such as a plurality of deployment modes and storage strategies in the above-mentioned method embodiment. The memory unit 950 may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static RAM (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct rambus RAM (DR RAM).

[0130] The memory 930 may correspond to a storage medium used to store information such as computer instructions and storage strategies in the above method embodiments, for example, a disk such as a mechanical hard disk or a solid-state hard disk.

[0131] The computing device 900 may be a general-purpose device or a dedicated device. For example, the computing device 900 may be an edge device (e.g., a box carrying a chip with processing capabilities), etc. Alternatively, the computing device 900 may also be a server or other device with computing capabilities.

[0132] It should be understood that the computing device 900 according to this embodiment may correspond to the control device 700 or the data processing device 800 in this embodiment, and may correspond to the device executing the Figure 5 or Figure 6 The corresponding subject in any method, and the above and other operations and / or functions of each module in the control device 700 or the data processing device 800 are respectively to achieve Figure 5 or Figure 6 For the sake of brevity, the corresponding processes of each method in are not repeated here.

[0133] The method steps in this embodiment can be implemented by hardware or by a processor executing software instructions. The software instructions can be composed of corresponding software modules, which can be stored in random access memory (RAM), flash memory, read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), registers, hard disks, mobile hard disks, CD-ROMs, or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor so that the processor can read information from the storage medium and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and the storage medium can be located in an ASIC. In addition, the ASIC can be located in a computing device. Of course, the processor and the storage medium can also exist in a computing device as discrete components.

[0134] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented by software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer programs or instructions. When the computer program or instruction is loaded and executed on a computer, the process or function described in the embodiment of the present application is executed in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, a network device, a user device or other programmable device. The computer program or instruction may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer program or instruction may be transmitted from one website site, computer, server or data center to another website site, computer, server or data center by wired or wireless means. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium may be a magnetic medium, for example, a floppy disk, a hard disk, a tape; it may also be an optical medium, for example, a digital video disc (DVD); it may also be a semiconductor medium, for example, a solid state drive (SSD). The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed in the present application, and these modifications or replacements should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. A data processing method, characterized in that: The method comprises: A control device obtains a service request of a big data service, wherein the service request includes service data; The control device determines at least one storage medium of a deployment mode from storage media of multiple deployment modes configured by the computing device, wherein the storage media of the multiple deployment modes include at least two of a local storage medium in the computing device, a global storage medium composed of local storage media, an extended global storage medium, and an extended local storage medium; The control device sends a control instruction to the computing device, wherein the control instruction is used to instruct the computing device to perform a mapping task in big data distributed processing on the business data, and store intermediate data obtained by processing the map task in a storage medium of at least one deployment mode.

2. The method according to claim 1, characterized in that Determining at least one storage medium of the deployment mode from the storage mediums of the multiple deployment modes includes: Determine the storage medium of at least one deployment mode from the storage media of the multiple deployment modes according to the storage strategy, wherein the storage strategy is used to indicate a method for selecting a storage medium based on storage medium characteristics, and the storage medium characteristics include at least one of write latency, read latency, total storage capacity, available storage capacity, access speed, central processing unit (CPU) consumption, energy consumption ratio and reliability.

3. The method according to claim 2, characterized in that The step of determining the storage medium of the at least one deployment mode from the storage mediums of the multiple deployment modes according to the storage policy includes: According to the priorities of the storage media of the multiple deployment modes indicated by the storage policy, the storage medium of at least one deployment mode is determined from the storage media of the multiple deployment modes, and the priorities of the storage media of the multiple deployment modes are determined based on the characteristics of the storage medium.

4. The method according to claim 2, characterized in that: The step of determining the storage medium of the at least one deployment mode from the storage mediums of the multiple deployment modes according to the storage policy includes: Based on user requirements and storage medium characteristics of the storage media in the multiple deployment modes, at least one storage medium in the deployment modes that meets the user requirements is determined from the storage media in the multiple deployment modes, and the user requirements are used to indicate requirements related to the storage medium characteristics.

5. The method according to any one of claims 1 to 4, characterized in that The storage medium includes memory, hard disk, memory server and storage-level memory SCM.

6. The method according to any one of claims 1 to 5, characterized in that After sending the control instruction to the computing device, the method further includes: Receive the storage location of the intermediate data in the storage medium of the at least one deployment mode sent by the computing device, where the storage location of the intermediate data is used to read the intermediate data when executing a reduce task.

7. A data processing method, characterized in that: The method comprises: The computing device receives a control instruction sent by the control device, the control instruction including business data and an identifier of at least one deployment mode of a storage medium of the computing device, the storage medium of the computing device having multiple configuration modes; The computing device performs a mapping task in big data distributed processing on the business data according to the control instruction to obtain intermediate data; The computing device stores the intermediate data in a storage medium corresponding to at least one deployment mode indicated by the identifier of the at least one deployment mode.

8. The method according to claim 7, characterized in that Storing the intermediate data in a storage medium corresponding to at least one deployment mode indicated by an identifier of the at least one deployment mode includes: If the storage medium of the at least one deployment mode includes multiple storage media of one deployment mode, the multiple shard data of the intermediate data are stored in the multiple storage media respectively; Alternatively, if the storage medium of the at least one deployment mode includes storage medium of two or more deployment modes, the multiple fragmented data of the intermediate data are respectively stored in the storage medium of the two or more deployment modes.

9. The method according to claim 7 or 8, characterized in that: The method further comprises: If the available storage capacity of the first storage medium is less than a threshold, the data of the first storage medium is stored in a second storage medium, and the access speed of the first storage medium is greater than the access speed of the second storage medium.

10. The method according to claim 9, characterized in that The method further comprises: The available storage capacity of the storage medium of the multiple deployment modes is sent to the control device.

11. A control device, characterized in that: include: A communication module, used to obtain a service request for a big data service, wherein the service request includes service data; A decision module, configured to determine at least one storage medium of a deployment mode from storage mediums of multiple deployment modes of a computing device, wherein the storage mediums of the multiple deployment modes include at least two of a local storage medium in the computing device, a global storage medium composed of local storage media, an extended global storage medium, and an extended local storage medium; The communication module is also used to send a control instruction to the computing device, wherein the control instruction is used to instruct the computing device to perform a mapping task in big data distributed processing on the business data, and store the obtained intermediate data in the storage medium of at least one deployment mode.

12. The device according to claim 11, characterized in that When the decision module determines at least one storage medium of a deployment mode from the storage mediums of multiple deployment modes of the computing device, it is specifically used to: Determine the storage medium of at least one deployment mode from the storage media of the multiple deployment modes according to the storage strategy, wherein the storage strategy is used to indicate a method for selecting a storage medium based on storage medium characteristics, and the storage medium characteristics include at least one of write latency, read latency, total storage capacity, available storage capacity, access speed, central processing unit (CPU) consumption, energy consumption ratio and reliability.

13. The device according to claim 12, characterized in that When the decision module determines the storage medium of the at least one deployment mode from the storage mediums of the multiple deployment modes according to the storage strategy, it is specifically used to: According to the priorities of the storage media of the multiple deployment modes indicated by the storage policy, the storage medium of at least one deployment mode is determined from the storage media of the multiple deployment modes, and the priorities of the storage media of the multiple deployment modes are determined based on the characteristics of the storage medium.

14. The device according to claim 12, characterized in that When the decision module determines the storage medium of the at least one deployment mode from the storage mediums of the multiple deployment modes according to the storage strategy, it is specifically used to: Based on user requirements and storage medium characteristics of the storage media in the multiple deployment modes, at least one storage medium in the deployment modes that meets the user requirements is determined from the storage media in the multiple deployment modes, and the user requirements are used to indicate requirements related to the storage medium characteristics.

15. The device according to any one of claims 11 to 14, characterized in that The storage medium includes memory, hard disk, memory server and storage-level memory SCM.

16. The device according to any one of claims 11 to 15, characterized in that The communication module is also used to receive the storage location of the intermediate data in the storage medium of the at least one deployment mode sent by the computing device, and the storage location of the intermediate data is used to read the intermediate data when executing the reduce task.

17. A data processing device, characterized in that: The data processing device is configured with storage media of multiple deployment modes, and the device includes: a communication module, configured to receive a control instruction sent by a control device, wherein the control instruction includes service data and an identifier of a storage medium of at least one deployment mode, wherein the storage medium of at least one deployment mode is determined by the control device from the storage media of the multiple deployment modes according to a storage policy, wherein the storage policy is used to indicate a method for selecting a storage medium according to storage medium characteristics; A data processing module is used to perform a mapping task in big data distributed processing on the business data according to the control instruction, obtain intermediate data, and store the intermediate data in a storage medium indicated by an identification of the storage medium of at least one deployment mode.

18. The device according to claim 17, characterized in that When the data processing module stores the intermediate data in the storage medium indicated by the identifier of the storage medium of the at least one deployment mode, it is specifically used to: If the storage medium of the at least one deployment mode includes multiple storage media of one deployment mode, the multiple shard data of the intermediate data are stored in the multiple storage media respectively; Alternatively, if the storage medium of the at least one deployment mode includes storage medium of two or more deployment modes, the multiple fragmented data of the intermediate data are respectively stored in the storage medium of the two or more deployment modes.

19. The device according to claim 17 or 18, characterized in that The data processing module is further used to store the data of the first storage medium into a second storage medium if the available storage capacity of the first storage medium is less than a threshold, and the access speed of the first storage medium is greater than the access speed of the second storage medium.

20. The device according to claim 19, characterized in that The communication module is also used to send the available storage capacity of the storage medium of the multiple deployment modes to the control device.

21. A control device, characterized in that: The control device includes a memory and at least one processor, wherein the memory is used to store a set of computer instructions; when the processor executes the set of computer instructions, the operation steps of the method described in any one of claims 1 to 6 are performed.

22. A computing device, characterized in that The computing device includes a memory, at least one processor and a display, wherein the memory is used to store a set of computer instructions; when the processor executes the set of computer instructions, the operation steps of the method described in any one of claims 7 to 10 are performed.

23. A system, characterized in that: The system includes a control device as described in claim 21 and a computing device as described in claim 22, wherein the control device is used to execute the operating steps of the method described in any one of claims 1 to 6, and the computing device is used to execute the operating steps of the method described in any one of claims 7 to 10.

Images