Data processing method, system, device and storage medium

By allocating separate distributed block storage devices and local storage space for each database instance, the problem of storage performance limitations of distributed storage devices is solved, improving the processing efficiency of temporary files and ensuring stable database operation.

CN113986846BActive Publication Date: 2024-11-12ALIBABA CLOUD COMPUTING CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202111087447.3
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-09-16
Publication Date
2024-11-12
Estimated Expiration
2041-09-16

AI Technical Summary

Technical Problem

In high concurrency scenarios, the storage performance limitations and local storage space limitations of distributed storage devices lead to temporary file system failures, affecting the normal operation of database instance processes and other related instances.

Method used

In the distributed file system, each database instance allocates a separate distributed block storage device and associated local storage space, prioritizes temporary files to local storage space, avoids the impact of database operation due to storage speed issues, and extends distributed blocks Storage devices solve the problem of insufficient storage space.

Benefits of technology

Improve the processing efficiency of temporary files in distributed databases, ensure the normal operation of database instances, avoid interrupts caused by storage speed problems, and improve the utilization rate of storage devices.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN113986846B_ABST
    Figure CN113986846B_ABST
Patent Text Reader

Abstract

The embodiments of the present application provide a data processing method, system, device and storage medium. The method includes: obtaining a temporary file; wherein the temporary file is generated after the first database instance performs a corresponding operation on the data in the distributed database; determining the distributed block storage device allocated to the first database instance in the file system of the distributed database; storing the temporary file in the first local storage space associated with the distributed block storage device. Based on the above embodiments, by separately allocating a corresponding distributed block storage device and an associated first local storage space to each database instance in the distributed file system, the generated temporary files can be stored and processed in a timely manner. When storing temporary files, they are preferentially stored in the first local storage space to avoid affecting the normal operation of the database due to the problem of temporary file processing speed, which can effectively improve the processing efficiency of temporary files in the distributed database.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to data processing methods, systems, devices and storage media. Background Art

[0002] With the development of database technology, users have higher and higher requirements on database performance during use. In order to meet users' requirements for computing performance and storage performance at the same time, a cloud-native relational database is proposed.

[0003] When executing a database instance based on a cloud-native relational database, some temporary files are generated. These temporary files usually do not need to be persistently stored, but due to the storage performance limitations of distributed storage devices or local storage space limitations, high-frequency temporary files generated in high-concurrency scenarios can easily cause file system failures, affecting the normal progress of the database instance process and even other related instances. Summary of the invention

[0004] In order to solve or improve the problems existing in the prior art, the embodiments of the present application provide a data processing method, system, device and storage medium.

[0005] In a first aspect, in one embodiment of the present application, a data processing method is provided. The method comprises:

[0006] Acquire a temporary file; wherein the temporary file is generated after the first database instance performs a corresponding operation on the data in the distributed database;

[0007] Determine a distributed block storage device allocated to the first database instance in a file system of the distributed database;

[0008] The temporary file is stored in a first local storage space associated with the distributed block storage device.

[0009] In a second aspect, in one embodiment of the present application, another data processing method is provided. The method comprises:

[0010] Generate a temporary file corresponding to the first database instance;

[0011] Sending a processing request for the temporary file to a distributed file system;

[0012] The drive letter of the distributed block storage device fed back by the distributed file system is received, so as to store the temporary file in a first local storage space associated with the distributed storage device.

[0013] In a third aspect, in one embodiment of the present application, a data processing system is provided. The system comprises:

[0014] A file system is used to obtain a temporary file; wherein the temporary file is generated after the first database instance performs a corresponding operation on the data in the distributed database; determine a distributed block storage device allocated to the first database instance in the file system of the distributed database; and store the temporary file in a first local storage space associated with the distributed block storage device;

[0015] A database is used to generate a temporary file corresponding to a first database instance; send a processing request for the temporary file to a distributed file system; receive a drive letter of a distributed block storage device fed back by the distributed file system, so as to store the temporary file in a first local storage space associated with the distributed storage device.

[0016] In a fourth aspect, in one embodiment of the present application, an electronic device is provided, including a memory and a processor; wherein:

[0017] The memory is used to store programs;

[0018] The processor is coupled to the memory and is used to execute the program stored in the memory to implement a data processing method described in the first aspect or another data processing method described in the second aspect.

[0019] In the fifth aspect, in one embodiment of the present application, a non-temporary machine-readable storage medium is provided, on which executable code is stored. When the executable code is executed by a processor of an electronic device, the processor executes the method described in the first aspect or the method described in the second aspect.

[0020] The technical solution provided in the embodiment of the present application obtains a temporary file; wherein the temporary file is generated after the first database instance performs corresponding operations on the data in the distributed database; determines the distributed block storage device allocated to the first database instance in the file system of the distributed database; and stores the temporary file in the first local storage space associated with the distributed block storage device. Based on the above embodiment, by separately allocating a corresponding distributed block storage device and an associated first local storage space to each database instance in the distributed file system, the generated temporary files can be stored and processed in a timely manner. When storing temporary files, they are preferentially stored in the first local storage space to avoid affecting the normal operation of the database due to the problem of temporary file processing speed, which can effectively improve the processing efficiency of temporary files in the distributed database. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, a brief introduction will be given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0022] Figure 1 A flowchart of a data processing method provided in an embodiment of the present application;

[0023] Figure 2 A schematic diagram of a process for storing temporary files in a distributed storage device provided in an embodiment of the present application;

[0024] Figure 3 A flowchart of a method for storing temporary files in a local storage space provided in an embodiment of the present application;

[0025] Figure 4 A flowchart of another data processing method provided in an embodiment of the present application;

[0026] Figure 5 A schematic diagram of a temporary file storage method provided in an embodiment of the present application;

[0027] Figure 6 A schematic diagram of the data processing system structure provided in an embodiment of the present application;

[0028] Figure 7 A schematic diagram of the structure of a data processing device provided in an embodiment of the present application;

[0029] Figure 8 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application;

[0030] Fig. 9 A schematic diagram of the structure of another data processing device provided in an embodiment of the present application;

[0031] Fig.10 A schematic diagram of the structure of another electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0032] In order to improve the data processing capability of the database, a database (POLARDB) with a structure that separates storage and computing is proposed. With the development of user needs, there will often be situations where database instance notifications are executed, and there may also be many temporary files generated by executing database instances. If there are many temporary files that need to be stored, the processing progress may be affected due to single-thread limitations, which may in turn affect the normal operation of more other instances. If temporary files are generated at a high frequency, the file system storing temporary files may also fail due to limited local storage space. In a distributed file system, since there are multiple instances on a physical machine and the file system space is limited, if a large amount of space is occupied by a certain instance, it will directly affect the space requirements of other instances. Therefore, a technical solution that can simultaneously have the ability to quickly store a large number of temporary files is needed. In the technical solution of this application, the specific working process will be described in the following embodiments.

[0033] It should be noted that the technical solution of this application is implemented based on a database (POLARDB) that separates storage and computing. The file system (PolarFS) in the database is a distributed file system with ultra-low latency and high availability. It is built with a lightweight user space network and I / O stack, and the corresponding kernel stack is abandoned. The purpose is to give full play to the potential of emerging hardware such as RDMA and NVMe SSD, and greatly reduce the end-to-end latency of distributed non-volatile data access. In actual applications, when a database instance performs related operations on data in a distributed database, the corresponding file system provides the instance with corresponding data storage space. There will be no other instances competing for storage space, and there will be no failure to store and process temporary files in a timely manner due to single-threaded problems.

[0034] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application.

[0035] In some processes described in the specification, claims and the above-mentioned figures of the present application, multiple operations appearing in a specific order are included, and these operations may not be executed or executed in parallel in the order in which they appear in this article. The serial numbers of the operations, such as 101, 102, etc., are only used to distinguish different operations, and the serial numbers themselves do not represent any execution order. In addition, these processes may include more or fewer operations, and these operations may be executed in sequence or in parallel. It should be noted that the descriptions of "first", "second", etc. in this article are used to distinguish different messages, devices, modules, etc., do not represent the order of precedence, and do not limit "first" and "second" to different types. In addition, the embodiments described below are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without making creative work are within the scope of protection of this application.

[0036] like Figure 1 A flow chart of a data processing method provided in an embodiment of the present application. In an actual application scenario, the execution subject of the method may be a computing node in a distributed database. The method specifically comprises the following steps:

[0037] 101: Acquire a temporary file; wherein the temporary file is generated after the first database instance performs a corresponding operation on the data in the distributed database.

[0038] 102: Determine a distributed block storage device allocated to the first database instance in the file system of the distributed database.

[0039] 103: Store the temporary file in a first local storage space associated with the distributed block storage device.

[0040] The temporary files mentioned here refer to those generated when the database instance performs corresponding operations on the data in the distributed database (for example, query sorting, large transaction query, etc.). For example, the temporary files generated by these large transactions can be / tmp / MLq9INFu(deleted) with a file name starting with ML; for another example, the temporary files generated by executing online DDL operations can be / tmp / ibCxlYQg(deleted) and / tmp / ib51nvZ1(deleted). These temporary files do not have a mandatory requirement for persistence and are only provided for temporary purposes. They can be deleted after use, and the temporary files do not need to be saved when the process is restarted.

[0041] The distributed block storage devices mentioned here can be understood as multiple block storage devices used to store related data (such as temporary files) in a distributed database. These distributed block storage devices can usually select one or more storage devices for temporary files generated by the database instance as needed during use. In actual applications, each database instance can be assigned its corresponding distributed block storage device. Of course, the distributed block storage devices of each database instance can be adjusted as needed during use (for example, increasing the storage space of the distributed block storage device, or allocating one more storage device to a certain instance), so as to better meet the storage space usage requirements of each database instance.

[0042] In actual applications, there may be situations where multiple database instances load and open the same database, and multiple databases are opened successively in the database instance process. In general, in the distributed database, the data storage space in the prior art can be shared, that is, when multiple database instances perform relevant data operations on the database, there may be a situation of competing for storage space. In the technical solution of this application, a temporary file refers to a file generated after any first database instance in the distributed database performs relevant operations on the database. Since the data processing capacity (data reading and writing speed) of the distributed block storage device is still slower than that of the local disk, when temporary files are generated at a high frequency, many temporary files need to be stored as soon as possible, but because the rate of the distributed block storage device is relatively slow, if it cannot be processed in time, it will cause the database instance process to be interrupted, and it will also affect the normal process of other related database instances. Therefore, when there is a storage demand for temporary files, the temporary files can be stored in the first local storage space associated with the distributed block storage device. The first local storage space mentioned here can be a local hard disk (for example, a solid state drive SSD), a virtual hard disk, a disk array, etc., which is specifically used to store temporary files. Alternatively, you can create multiple files on the local hard disk, each with independent storage space, and use the files on the hard disk to store temporary files generated based on the database instance. For example, create multiple files on the local hard disk of the Linux system, and use one of the files to store temporary files. This can effectively solve the problem of slow read and write speeds of distributed block storage devices affecting the read and write efficiency of temporary files, or causing interruptions to the database instance process.

[0043] In one or more embodiments of the present application, Figure 2 A flow chart of a method for storing a temporary file in a distributed storage device provided in an embodiment of the present application. As described in step 103, after the temporary file is stored in the first local storage space associated with the distributed block storage device, as described in step 104, Figure 2The method also includes the following steps: 201: if the temporary file is not successfully stored in the first local storage space, the temporary file is stored in the distributed block storage device. 202: if the temporary file is not successfully stored in the distributed block storage device, the distributed block storage device is expanded to obtain a second storage space, so as to store the temporary file in the expanded second storage space.

[0044] Since local storage devices usually have a maximum storage limit, the first local storage space allocated to the first database instance also has an upper limit. If the size of the temporary file that needs to be stored currently exceeds the size of the available space in the corresponding first local storage space, storage failure may occur. Or because the first local storage space is full, storage failure may also occur. Or the first local storage space is in a disabled state, which may also cause storage failure. For example, the first local storage space is 100G, 50G has been used, and there is 50G of available storage space remaining, but the total size of the temporary files to be stored is 70G. Therefore, the temporary files cannot be stored in the first local storage space.

[0045] In order to ensure that the process of the database instance is not interrupted, temporary files can be stored in the distributed block storage device. Generally speaking, the storage space of the distributed block storage device is relatively large compared to the local storage space. Moreover, in the distributed database, the size of the distributed block storage device allocated to the database instance can be adjusted as needed. When the storage space is insufficient, it can be expanded as needed. When the storage space is sufficient but the storage space of other database instances is insufficient, the excess storage space can be shared to solve the problem of insufficient storage space of other database instances and improve the utilization rate of the distributed block storage device.

[0046] In the process of storing temporary files in the distributed block storage device, the first storage space may be insufficient, resulting in the problem that the temporary files cannot be successfully stored in the first storage space. As mentioned above, the first storage space is allocated according to the needs of the database instance. Then, there will inevitably be some special circumstances that cause the size of the temporary file to exceed the size of the remaining available space in the previously allocated first storage space, which will cause the temporary file storage failure to occur. Since the first storage space is allocated based on the distributed block storage device, the first storage space is adjusted, that is, the space of the first storage space is expanded, so that there is a larger space to store the temporary files of the instance. After the expansion, the second storage space is obtained. The second storage space mentioned here is significantly larger than the first storage space. It should be noted that if the first storage space is expanded on the basis of the same distributed block storage device to obtain the second storage space, the drive letter disclosed to the outside can be unchanged. In other words, when a temporary file wants to be stored, the storage operation is still performed through the drive letter corresponding to the first storage space. If the first storage space and the second storage space are located on two different distributed block storage devices, then when temporary files need to be stored, two different drive letters are required to perform the storage operation on the temporary files. Regardless of the request, only the drive letter corresponding to the distributed block storage device is exposed to the outside, and no drive letter or file name related to the local storage space is exposed to the outside. This allows the storage needs for temporary files to be realized quickly, with large capacity and scalable without the outside world's perception.

[0047] In one or more embodiments of the present application, based on an allocation request for allocating space for the first database instance, a first block of storage space in the distributed block storage device is determined, and the first block of storage space in the distributed storage device is allocated to the first database instance.

[0048] In actual applications, there are multiple block storage devices in a distributed database. During data processing, these block storage devices will be allocated to the corresponding database instance as needed. When allocating, multiple block storage devices can be evenly divided so that the storage space of each database instance is the same. Of course, differential allocation can also be performed based on the differences between the database instances. For example, if it can be predicted that the temporary files generated by a database instance will occupy a large storage space, a larger storage space can be allocated for it. For example, when allocating the first storage space for a database instance, it is assumed that the first storage space is the entire space of a distributed block storage device. When in use, the temporary basis of the database instance can be stored according to the drive letter corresponding to the distributed storage device. Of course, if the distributed block storage device allows sharing storage space with other instances, the drive letter can also be provided to the corresponding database instance to implement temporary file storage.

[0049] In one or more embodiments of the present application, the first local storage space and the corresponding first local storage space identifier are determined based on the instance space allocation request, and an association relationship between the distributed block storage device and the first local storage space is established according to the first local storage space identifier on the index node.

[0050] As mentioned above, in order to meet the storage requirements for large quantities of temporary files, temporary files will be stored in external distributed block storage devices, because distributed block storage devices can be expanded according to actual storage requirements. However, sometimes it is necessary to store a batch of temporary files in a timely and fast manner. In order to avoid affecting the normal process of the database instance due to untimely storage, the temporary files can be stored in the first local storage space. It should be noted that the first local storage space identifier mentioned here includes: the local hard disk drive letter or any file name in multiple files established on the local hard disk. There is a one-to-one correspondence between the first local storage space and the first database instance. In other words, each database instance has its corresponding first local storage space, and the first local storage space is not allowed to be occupied by other database instances. This is because in a distributed database, it is common for multiple database instances to compete for shared storage space at the same time, and database instances that cannot grab sufficient storage space for storing temporary files are at risk of being interrupted. Moreover, it is difficult to determine when each database instance needs to store a large number of temporary files. Therefore, in order to ensure the normal execution of each database instance process, the local storage space is set as an independent space, and the local storage space for storing temporary files is not shared between each database instance.

[0051] When establishing the association between the distributed block storage device and the first local storage space, the first local storage space flag can be recorded in the index node (inode). That is, if the distributed block storage device is associated with the first local storage space flag, when there is a temporary file storage demand, it will be stored in the first local storage space first. On the contrary, if a distributed storage device is determined based on the drive letter, and the distributed storage device is not associated with the first local storage space flag, the temporary file will be directly stored in the distributed block storage device.

[0052] It should be noted that although the first local storage space and the distributed block storage device are used to store temporary files at the same time, in order to simplify the temporary file storage process, only the drive letter corresponding to the distributed block storage device can be made public. Figure 3 A flow chart of a method for storing temporary files in a local storage space provided in an embodiment of the present application. Figure 3 It can be seen that the specific steps include the following: 301: If the first local storage space identifier is marked on all the index nodes, the temporary file is stored in the first local storage space. 302: If the temporary file is not successfully stored in the first local storage space, the first local storage space is expanded to obtain a second local storage space, so that the temporary file is stored in the expanded second local storage space.

[0053] Specifically, after receiving the temporary file, it will first be checked whether the first local storage space identifier is marked in the index node according to the drive letter of the distributed block storage device. If it is marked, the temporary file will be stored in the first local storage space. If it is found that there is no mark after searching, the temporary file will be stored in the distributed storage device corresponding to the drive letter. If it is found that it has been marked, but the first local storage space is full, or the remaining space is less than the space required to store the temporary file, the first local storage space can also be expanded so that the second local storage space obtained after the expansion can successfully store the temporary file. If the temporary file still cannot be stored after the expansion, the temporary file will be stored in the distributed block storage device. Furthermore, since the second local storage space is not occupied, the second local storage space obtained after the expansion can also be reduced to the first local storage space to avoid unnecessary waste caused by excessive occupation of local storage space.

[0054] In one or more embodiments of the present application, if the temporary file is generated based on the second database instance performing a corresponding operation on the data in the distributed database, the temporary file is stored in a third local storage space associated with the distributed block storage device.

[0055] As mentioned above, each instance has its own dedicated local storage space. After receiving the temporary file, the source of the temporary file will be further determined, that is, which database instance the temporary file is generated based on. For example, if further determination finds that the temporary file is generated based on the second database instance, then the temporary file cannot be stored in the first local storage space, but needs to be stored in the corresponding third local storage space.

[0056] Based on the same idea, the embodiment of the present application also provides another data processing method. Figure 4 A flow chart of another data processing method provided in an embodiment of the present application. The method can be used in a storage node to cooperate with a computing node to achieve efficient storage of temporary files. The method includes:

[0057] 401: Generate a temporary file corresponding to the first database instance.

[0058] 402: Send a request for processing the temporary file to the distributed file system.

[0059] 403: Receive the drive letter of the distributed block storage device fed back by the distributed file system, so as to store the temporary file in the first local storage space associated with the distributed storage device.

[0060] In actual applications, when the first database instance performs corresponding operations on the data in the distributed database, corresponding temporary files will be generated. These temporary files usually need to be stored in the designated storage space. In order to achieve fast and efficient storage of temporary files, after the temporary file is generated, its feedback drive letter is received from the distributed file system, and the drive letter is the drive letter corresponding to the distributed block storage device allocated to the first database instance. Then, the temporary file is stored in the first local storage space associated with the distributed storage device. It should be noted that each database instance has its corresponding local storage space specifically used to store temporary files. It should be noted that since the association relationship between the first local storage space identifier and the distributed storage device has been established, when storing temporary files, it is only necessary to know the drive letter corresponding to the first database instance to achieve the storage of temporary files. The complex operation process of selecting the storage space during the storage process is avoided. For the specific implementation process, please refer to Figures 1 to 3 The various embodiments described above will not be repeated here.

[0061] In one or more embodiments of the present application, Figure 5 A flow chart of a temporary file storage method provided in an embodiment of the present application. Figure 5It can be seen that the specific steps include the following: 501: receiving the drive letter of the distributed block storage device fed back by the distributed file system, if the size of the temporary file exceeds the remaining space of the first local storage space, so as to store the temporary file in the distributed block storage device. 502: if the size of the temporary file exceeds the remaining space of the distributed block storage device, so as to expand the capacity of the distributed block storage device.

[0062] During the temporary file storage process, temporary files will be stored in the first local storage space first. If the size of the temporary file exceeds the available space in the first local storage space, and the local storage space of each database instance is not shared, if the temporary file cannot be successfully stored in the first local storage space, the temporary file will be stored in the distributed block storage device corresponding to the drive letter. Relative to the local storage space, distributed block storage devices can be called external storage devices, and these distributed storage devices can be expanded as needed. For example, if the size of the temporary file exceeds the available space of the distributed storage device, the distributed storage device corresponding to the first database instance can be expanded so that the temporary file can be successfully stored. For the specific implementation process, please refer to Figures 1 to 3 The various embodiments described above will not be repeated here.

[0063] Based on the same idea, the embodiment of the present application also provides a data processing system. Figure 6 A schematic diagram of the data processing system structure provided in the embodiment of the present application. Figure 6 As can be seen in the figure, the system includes:

[0064] The file system 61 is used to obtain a temporary file; wherein the temporary file is generated after the first database instance performs a corresponding operation on the data in the distributed database; determine the distributed block storage device allocated to the first database instance in the file system of the distributed database; and store the temporary file in a first local storage space associated with the distributed block storage device.

[0065] For a specific solution to implement temporary file storage based on the file system, see Figures 1 to 3 The various embodiments shown will not be repeated here.

[0066] Database 62 is used to generate a temporary file corresponding to the first database instance; send a processing request for the temporary file to the distributed file system; receive the drive letter of the distributed block storage device fed back by the distributed file system, so as to store the temporary file in the first local storage space associated with the distributed storage device.

[0067] For a specific solution to temporary file storage based on a database, see Figures 4 to 5 The various embodiments shown will not be repeated here.

[0068] based on Figure 6 It can be seen that the physical device of the system consists of a local storage disk and a distributed block storage disk. Then, it is encapsulated and presented to the outside in the form of a logical disk. The drive letter of the logical disk directly uses the drive letter of the distributed block storage disk (for example, the drive letter of the D drive is D:), so when processing temporary files of the database instance, you only need to know the drive letter of the distributed block storage disk. In other words, during use, the database does not need to perceive the existence of the local storage disk. After opening the temporary file, the distributed database calls the F_SETFL command of the fcntl interface to modify the storage method of the temporary file. The file system records the flag of the local storage disk at the inode level, indicating that the data block of the temporary file is allocated to the local storage disk.

[0069] Based on the same idea, the embodiment of the present application also provides a data processing device. Figure 7 A schematic diagram of the structure of a data processing device provided in an embodiment of the present application. The data processing device includes:

[0070] The acquisition module 71 is used to acquire a temporary file; wherein the temporary file is generated after the first database instance performs a corresponding operation on the data in the distributed database.

[0071] The determination module 72 is configured to determine a distributed block storage device allocated to the first database instance in the file system of the distributed database.

[0072] The storage module 73 is configured to store the temporary file in a first local storage space associated with the distributed block storage device.

[0073] Optionally, the storage module 73 is further configured to store the temporary file in the distributed block storage device if the temporary file is not successfully stored in the first local storage space.

[0074] Optionally, the determination module 72 is further configured to determine a first block of storage space in the distributed block storage device based on an allocation request for allocating space for the first database instance; and allocate the first block of storage space in the distributed storage device to the first database instance.

[0075] Optionally, the determination module 72 is further configured to determine the first local storage space and the corresponding first local storage space identifier based on the instance space allocation request; and establish an association relationship between the distributed block storage device and the first local storage space according to the first local storage space identifier on the index node.

[0076] Optionally, the storage module 73 is further configured to store the temporary file in the first local storage space if all the index nodes are marked with the first local storage space identifier.

[0077] Optionally, the storage module 73 is further configured to expand the distributed block storage device to obtain a second storage space if the temporary file is not successfully stored in the distributed block storage device, so as to store the temporary file in the expanded second storage space.

[0078] Optionally, the storage module 73 is further configured to expand the first local storage space to obtain a second local storage space if the temporary file is not successfully stored in the first local storage space, so as to store the temporary file in the expanded second local storage space.

[0079] Optionally, the first local storage space identifier includes: a local hard disk drive letter or any file name among multiple files established on the local hard disk.

[0080] Optionally, the storage module 73 is further configured to store the temporary file in a third local storage space associated with the distributed block storage device if the temporary file is generated based on a corresponding operation performed by the second database instance on data in the distributed database.

[0081] An embodiment of the present application also provides an electronic device. The electronic device is a master node electronic device in a computing unit. Figure 8 The electronic device includes a memory 801, a processor 802 and a communication component 803;

[0082] The memory 801 is used to store programs;

[0083] The processor 802 is coupled to the memory and is configured to execute the program stored in the memory to:

[0084] Acquire a temporary file; wherein the temporary file is generated after the first database instance performs a corresponding operation on the data in the distributed database;

[0085] Determine a distributed block storage device allocated to the first database instance in a file system of the distributed database;

[0086] The temporary file is stored in a first local storage space associated with the distributed block storage device.

[0087] The memory 801 can be configured to store various other data to support operations on the electronic device. Examples of these data include instructions for any application or method for operating on the electronic device. The memory can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0088] Furthermore, the processor 802 in this embodiment may specifically be: a programmable switching processing chip, in which a data replication engine is configured to replicate received data.

[0089] When the processor 802 executes the program in the memory, in addition to the above functions, it can also realize other functions, which can be specifically referred to the description of the above embodiments. Figure 8 As shown, the electronic device also includes: a power supply component 804 and other components.

[0090] The present application also provides a non-transitory machine-readable storage medium, wherein the non-transitory machine-readable storage medium stores executable code, and when the executable code is executed by a processor of an electronic device, the processor executes Figures 1 to 3 The method described in the corresponding embodiment.

[0091] Based on the same idea, the embodiment of the present application also provides a data processing device. Fig. 9 A schematic diagram of the structure of another data processing device provided in an embodiment of the present application. The data processing device includes:

[0092] The generating module 91 is used to generate a temporary file corresponding to the first database instance.

[0093] The sending module 92 is used to send a processing request for the temporary file to the distributed file system.

[0094] The receiving module 93 is configured to receive the drive letter of the distributed block storage device fed back by the distributed file system, so as to store the temporary file in the first local storage space associated with the distributed storage device.

[0095] Optionally, the receiving module 93 is further configured to receive a drive letter of a distributed block storage device fed back by the distributed file system, and if the size of the temporary file exceeds the remaining space of the first local storage space, store the temporary file in the distributed block storage device; and

[0096] If the size of the temporary file exceeds the remaining space of the distributed block storage device, the capacity of the distributed block storage device is expanded.

[0097] The present application also provides a computer program product, including a computer program / instruction. When the computer program / instruction is executed by a processor, the processor is enabled to implement Figure 4 The method described in the corresponding embodiment.

[0098] An embodiment of the present application also provides an electronic device. The electronic device is a standby node electronic device in a computing unit. Fig.10 This is a schematic diagram of the structure of another electronic device provided in an embodiment of the present application. The electronic device includes a memory 1001, a processor 1002 and a communication component 1003; wherein:

[0099] The memory 1001 is used to store programs;

[0100] The processor 1002 is coupled to the memory and is configured to execute the program stored in the memory to:

[0101] Generate a temporary file corresponding to the first database instance; send a processing request for the temporary file to the distributed file system; receive the drive letter of the distributed block storage device fed back by the distributed file system, so as to store the temporary file in the first local storage space associated with the distributed storage device.

[0102] The memory 1001 can be configured to store various other data to support operations on the electronic device. Examples of these data include instructions for any application or method for operating on the electronic device. The memory can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0103] Furthermore, the processor 1002 in this embodiment may specifically be: a programmable switching processing chip, in which a data replication engine is configured to replicate received data.

[0104] When the processor 1002 executes the program in the memory, in addition to the above functions, it can also realize other functions, which can be specifically referred to the description of the above embodiments. Fig.10 As shown, the electronic device also includes: a power supply component 1004 and other components.

[0105] The present application also provides a non-transitory machine-readable storage medium, wherein the non-transitory machine-readable storage medium stores executable code, and when the executable code is executed by a processor of an electronic device, the processor executes Figure 4 The method described in the corresponding embodiment.

[0106] The present application also provides a computer program product, including a computer program / instruction. When the computer program / instruction is executed by a processor, the processor is enabled to implement Figure 4 The method described in the corresponding embodiment.

[0107] Based on the above embodiment, a temporary file is obtained; wherein the temporary file is generated after the first database instance performs corresponding operations on the data in the distributed database; the distributed block storage device allocated to the first database instance in the file system of the distributed database is determined; and the temporary file is stored in the first local storage space associated with the distributed block storage device. Based on the above embodiment, by separately allocating a corresponding distributed block storage device and an associated first local storage space to each database instance in the distributed file system, the generated temporary files can be stored and processed in a timely manner. When storing temporary files, they are preferentially stored in the first local storage space to avoid affecting the normal operation of the database due to the problem of temporary file processing speed, which can effectively improve the processing efficiency of temporary files in the distributed database.

[0108] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0109] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0110] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A data processing method, the method comprising: Acquire a temporary file; wherein the temporary file is generated after the first database instance performs a corresponding operation on the data in the distributed database; Determine a distributed block storage device allocated to the first database instance in a file system of the distributed database; The temporary file is stored in a first local storage space associated with the distributed block storage device.

2. The method according to claim 1, further comprising: If the temporary file is not successfully stored in the first local storage space, the temporary file is stored in the distributed block storage device.

3. The method according to claim 1, determining the distributed block storage device allocated to the first database instance in the file system of the distributed database, comprising: Determining a first block storage space of the distributed block storage device based on an allocation request for allocating space for the first database instance; Allocate the first storage space in the distributed storage device to the first database instance.

4. The method according to claim 3, further comprising: Determine the first local storage space and the corresponding first local storage space identifier based on the instance space allocation request; An association relationship between the distributed block storage device and the first local storage space is established according to the first local storage space identifier on the index node.

5. The method according to claim 4, storing the temporary file in a first local storage space associated with the distributed block storage device, comprising: If the index node is marked with the first local storage space identifier, the temporary file is stored in the first local storage space.

6. The method according to claim 2, wherein storing the temporary file in the distributed block storage device comprises: If the temporary file is not successfully stored in the distributed block storage device, the distributed block storage device is expanded to obtain a second storage space, so as to store the temporary file in the expanded second storage space.

7. The method according to claim 1, wherein storing the temporary file in a first local storage space associated with the distributed block storage device comprises: If the temporary file is not successfully stored in the first local storage space, the first local storage space is expanded to obtain a second local storage space, so as to store the temporary file in the expanded second local storage space.

8. The method according to claim 1, wherein the first local storage space identifier comprises: The local hard disk drive letter or any file name among multiple files created on the local hard disk.

9. The method according to claim 1, further comprising: If the temporary file is generated based on the corresponding operation performed by the second database instance on the data in the distributed database, the temporary file is stored in a third local storage space associated with the distributed block storage device.

10. A data processing method, the method comprising: Generate a temporary file corresponding to the first database instance; Sending a processing request for the temporary file to a distributed file system; The drive letter of the distributed block storage device fed back by the distributed file system is received, so as to store the temporary file in a first local storage space associated with the distributed storage device.

11. The method according to claim 10, further comprising: receiving a drive letter of a distributed block storage device fed back by the distributed file system, and if the size of the temporary file exceeds the remaining space of the first local storage space, storing the temporary file in the distributed block storage device; as well as, If the size of the temporary file exceeds the remaining space of the distributed block storage device, the capacity of the distributed block storage device is expanded.

12. A data processing system, the system comprising: A file system is used to obtain a temporary file; wherein the temporary file is generated after the first database instance performs a corresponding operation on the data in the distributed database; determine a distributed block storage device allocated to the first database instance in the file system of the distributed database; and store the temporary file in a first local storage space associated with the distributed block storage device; A database is used to generate a temporary file corresponding to a first database instance; send a processing request for the temporary file to a distributed file system; receive a drive letter of a distributed block storage device fed back by the distributed file system, so as to store the temporary file in a first local storage space associated with the distributed storage device.

13. An electronic device comprising a memory and a processor; wherein: The memory is used to store programs; The processor, coupled to the memory, is configured to execute the program stored in the memory to implement the method of any one of claims 1 to 9; or the method of any one of claims 10 to 11.

14. A non-transitory machine-readable storage medium having executable code stored thereon, wherein when the executable code is executed by a processor of an electronic device, the processor is caused to execute the method as described in any one of claims 1 to 9; or the method as described in any one of claims 10 to 11 above.