Storage device saving method for big data processing

Abstract

To provide a method capable of minimizing a space occupied by storage data in a data storage system in which storage data is overlapped / copied and used for big data processing.SOLUTION: An operation method of a data storage system includes the steps of: selecting and setting first to third storage devices from a plurality of storage devices 20, 30, 40, 50, 60, 70, 80, and 90, and then sequentially transmitting storage data D1, D2, D3, D4, and D5 of a data processing device 10 to the second storage device following the first storage device and storing them; transmitting the storage data stored in the second storage device to the third storage device and then compressing and storing the data inside the third storage device; permitting access to the storage data by the data processing device after completing the storage of the storage data into the first storage device and the second storage device; and accessing the storage data in the second storage device if the storage data cannot be successfully accessed from the first storage device.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to a data storage system, and more particularly, to a method for saving storage devices for big data processing.

Background Art

[0002] Big data refers to a very large and complex data set. This generates an enormous amount of data that reaches several terabytes, petabytes, etc. within a short period of time. As the amount of data to be processed increases, there is a need for a file system that can store a huge amount of data in distributed servers and quickly process the stored data.

[0003] Hadoop refers to open source software that can cluster and distribute large amounts of data. Such Hadoop has the ability to analyze, store, and process large amounts of data, so it is the most spotlighted among all big data systems.

[0004] On the other hand, Hadoop uses a method of storing data in multiple servers and processing the data simultaneously on each of the stored servers. Therefore, the Hadoop Distributed File System (HDFS) divides big data into multiple unit data having a predetermined size. Then, the Hadoop Distributed File System uses a method of self-replicating the divided unit data to a plurality of storage devices so that the unit data is distributed and stored. For example, in the Hadoop Distributed File System (HDFS), a method of replicating unit data three times and storing it in a distributed manner can be used.

Summary of the Invention

Problems to be Solved by the Invention

[0005] Embodiments of the present invention provide a method capable of minimizing the space occupied by stored data in a data storage system that repeatedly replicates and uses stored data for big data processing.

Means for Solving the Problems

[0006] A method for operating a data storage system according to an embodiment of the present invention is a method for operating a data storage system including a data processing device and a plurality of storage devices. After selecting and setting the first to third storage devices among the plurality of storage devices, a first storage step of sequentially transmitting and storing the stored data of the data processing device to the first storage device and then to the second storage device; a second storage step of transmitting the stored data stored in the second storage device to the third storage device and then compressing and storing it inside the third storage device; after the storage of the stored data in the first storage device and the second storage device is completed, the data processing device includes a first access step of allowing access to the stored data, and a second access step of accessing the stored data in the second storage device if the stored data cannot be normally accessed from the first storage device after the first access step.

[0007] Also, the first access step and the second storage step can be performed in parallel.

[0008] Also, after the first access step, if the stored data cannot be normally accessed from the first storage device, it can further include a third storage step of decompressing the stored data stored in the third storage device in a compressed state and continuing to store it in the third storage device.

[0009] Also, the second access step and the third storage step can be performed in parallel.

[0010] Also, after the third storage step, reset the second storage device to the first storage device, and after resetting the third storage device to the second storage device, among the plurality of storage devices, a reset step of reselecting and setting the third storage device, and after the reset step, performing the second storage step to store the storage data in a compressed state in the third storage device can be further included.

[0011] Also, the first storage step may include a fourth storage step of transmitting the storage data generated by the data processing device to the first storage device and then storing it in the first storage device, and a fifth storage step of transmitting the storage data from the first storage device to the second storage device and then storing it in the second storage device.

[0012] Also, the fourth storage step and the fifth storage step can be performed in parallel.

[0013] A method for operating a data storage system according to still another embodiment of the present invention is a method for operating a data storage system including a data processing device and a plurality of storage devices. Among the plurality of storage devices, after selecting and setting the first to third storage devices, a first storage step of sequentially transmitting and storing the storage data of the data processing device to the first storage device and then to the second storage device, a second storage step of transmitting the storage data stored in the second storage device to the third storage device and then compressing and storing it inside the third storage device, and after the first and second storage information indicating that the storage of the storage data in each of the first and second storage devices is completed is transmitted to the data processing device, a first access step in which the data processing device permits access to the storage data, and after the first access step, if the storage data cannot be normally accessed from the first storage device, a second access step of accessing the storage data in the second storage device can be included.

[0014] Further, the first access step and the second storage step can be performed in parallel.

[0015] Also, after the first access step, if the storage data cannot be normally accessed from the first storage device, a third storage step of decompressing the storage data stored in the third storage device in a compressed state and continuing to store it in the third storage device can be further included.

[0016] Also, the second access step and the third storage step can be performed in parallel.

[0017] Also, after the third storage step, the second storage device is reset to the first storage device, and after the third storage device is reset to the second storage device, among the plurality of storage devices, a reset step of reselecting and setting the third storage device, and after the reset step, a step of performing the second storage step and storing the storage data in the third storage device in a compressed state can be further included.

[0018] Also, the first storage step can include a fourth storage step of transmitting the storage data generated by the data processing device to the first storage device and then storing it in the first storage device, a fifth storage step of transmitting the storage data from the first storage device to the second storage device and then storing it in the second storage device, a first transmission step of generating the second storage information in the second storage device in response to completion of the fifth storage step and transmitting it to the first storage device, and a second transmission step of generating the first storage information in the first storage device in response to completion of the fourth storage step after the first transmission step and transmitting it to the data processing device.

[0019] Also, the fourth storage step and the fifth storage step can be performed in parallel.

[0020] A data storage system according to still another embodiment of the present invention includes a plurality of storage devices, and stores internally generated storage data in a stacked manner in first to third storage devices among the plurality of storage devices. Each of the first and second storage devices stores the storage data in an uncompressed state, and the third storage device includes a data processing device that stores the storage data in a compressed state. When an operation of accessing the storage data fails in any one of the first and second storage devices, the data processing device can perform, in parallel, an operation of accessing the storage data in the remaining one of the storage devices and an operation of decompressing the storage data stored in the third storage device in a compressed state.

[0021] Further, the data processing device generates the storage data and transmits it to the first storage device. The first storage device performs, in parallel, an operation of storing the storage data transmitted from the data processing device internally in an uncompressed state and an operation of transmitting the storage data transmitted from the data processing device to the second storage device. The second storage device performs, in parallel, an operation of storing the storage data transmitted from the first storage device internally in an uncompressed state and an operation of transmitting the storage data transmitted from the first storage device to the third storage device. The third storage device can perform an operation of storing the storage data transmitted from the second storage device internally in a compressed state.

[0022] Further, in response to the storage data transmitted from the first storage device being stored internally in an uncompressed state, the second storage device generates second storage information and transmits it to the first storage device. In response to the storage data transmitted from the data processing device being stored internally in an uncompressed state and the second storage information being transmitted from the second storage device, the first storage device generates first storage information and transmits it to the data processing device. In response to the first storage information being transmitted from the first storage device, the data processing device permits an access operation to the storage data. In response to the storage data transmitted from the second storage device being stored internally in a compressed state, the third storage device can generate third storage information and transmit it to the data processing device.

[0023] In addition, the operation of storing the storage data transmitted from the second storage device internally in a compressed state by the third storage device and the operation of permitting an access operation to the storage data by the data processing device can be performed in parallel.

Advantages of the Invention

[0024] This technology can minimize the space occupied by storage data by storing some data in a compressed manner in a data storage system that duplicates and uses storage data for big data processing.

Brief Description of the Drawings

[0025]

Figure 1

Figure 2A

Figure 2B

Figure 3A

Figure 3B

Figure 3C

Figure 3D

Figure 4

Figure 5

Embodiments for Carrying Out the Invention

[0026] Hereinafter, preferred embodiments of the present invention will be described with reference to the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below, and can be configured in various different forms. Merely, these embodiments are provided to make the disclosure of the present invention complete and to fully inform those with ordinary knowledge of the scope of the present invention.

[0027] FIG. 1 is a diagram shown to explain the data storage system according to an embodiment of the present invention.

[0028] As shown in FIG. 1, the data storage system according to an embodiment of the present invention can include a data processing device 10 and a plurality of data storage devices 20, 30, 40, 50, 60, 70, 80, 90.

[0029] The data processing apparatus 10 can include an electronic device, for example, a portable electronic device such as a mobile phone, an MP3 player, a laptop computer, or an electronic device such as a desktop computer, a game machine, a TV, a projector, that is, a computer device or a wired / wireless electronic device.

[0030] In addition, the data processing device 10 includes at least one operating system (OS), and the operating system generally manages and controls the functions and operations of the data processing device 10, and provides an interaction between a user who uses a plurality of data storage devices 20, 30, 40, 50, 60, 70, 80, 90 and the data processing device 10. Here, the operating system supports functions and operations corresponding to the user's purpose and use. For example, it can be classified into a general operating system and a mobile operating system according to the mobility of the data processing device 10. Further, the general operating system in the operating system can be classified into a personal operating system and a corporate operating system according to the user's usage environment. As an example, the personal operating system is a system characterized by supporting service-providing functions for general users, including Windows (registered trademark) and Chrome (registered trademark), etc., and the corporate operating system is a system characterized by ensuring and supporting high performance, and can include Windows Server, Linux (registered trademark), and Unix (registered trademark), etc. Furthermore, the mobile operating system in the operating system is a system characterized by supporting a mobility service-providing function and a power-saving function of the system for users, and can include Android (registered trademark), iOS (registered trademark), Windows Mobile, etc.At this time, the data processing device 10 can include a plurality of operating systems, and performs an operating system for executing operations with a plurality of data storage devices 20, 30, 40, 50, 60, 70, 80, 90 corresponding to a user request. Here, the data processing device 10 transmits a plurality of commands corresponding to the user request to the plurality of data storage devices 20, 30, 40, 50, 60, 70, 80, 90, whereby the plurality of data storage devices 20, 30, 40, 50, 60, 70, 80, 90 can perform an operation corresponding to the command, that is, an operation corresponding to the user request.

[0031] In addition, each of the plurality of data storage devices 20, 30, 40, 50, 60, 70, 80, 90 operates in response to a request from the data processing device 10, and in particular, can store data accessed by the data processing device 10. In other words, each of the plurality of data storage devices 20, 30, 40, 50, 60, 70, 80, 90 can be used as a main memory device or an auxiliary memory device of the data processing device 10. Also, each of the plurality of data storage devices 20, 30, 40, 50, 60, 70, 80, 90 can be realized by any one of various types of storage devices. For example, the plurality of data storage devices 20, 30, 40, 50, 60, 70, 80, 90 can be realized by any one of various types of storage devices such as a solid state drive (SSD), MMC, eMMC (embedded MMC), RS-MMC (Reduced Size MMC), a multimedia card (MMC) in micro-MMC form, SD, mini-SD, a secure digital (SD) card in micro-SD form, a USB (Universal Storage Bus) storage device, a UFS (Universal Flash Storage) device, a CF (Compact Flash) card, a Smart Media card, a Memory Stick, and the like.

[0032] Also, the plurality of data storage devices 20, 30, 40, 50, 60, 70, 80, 90 can be integrated into one semiconductor device to form a memory card. As an example, it can form a memory card such as a PC card (PCMCIA: Personal Computer Memory Card International Association), a CompactFlash (registered trademark) card (CF), a SmartMedia (registered trademark) card (SM, SMC), a Memory Stick, a Multimedia Card (MMC, RS-MMC, MMCmicro), an SD card (SD, miniSD, microSD, SDHC), a Universal Flash Storage device (UFS), etc.

[0033] Also, as another example, the plurality of data storage devices 20, 30, 40, 50, 60, 70, 80, 90 can be a computer, UMPC (Ultra Mobile PC), workstation, net-book, PDA (Personal Digital Assistants), portable computer, web tablet, tablet computer, wireless phone, mobile phone, smart phone, e-book, PMP (portable multimedia player), portable game machine, navigation device, black box, digital camera, DMB (Digital Multimedia Broadcasting) player, 3-dimensional television, smart television, digital audio recorder, digital audio player, digital picture recorder, digital picture player, digital video recorder, digital video player, storage constituting a data center, a device capable of transmitting and receiving information in a wireless environment, one of various electronic devices constituting a home network, one of various electronic devices constituting a computer network, one of various electronic devices constituting a telematics network, an RFID (radio frequency identification) device, or one of various components constituting a computer system, etc.

[0034] In addition, each of the plurality of data storage devices 20, 30, 40, 50, 60, 70, 80, 90 may include a storage area (not shown). Further, the storage area (not shown) included in each of the plurality of data storage devices 20, 30, 40, 50, 60, 70, 80, 90 may include storage devices such as volatile memory devices such as DRAM (Dynamic Random Access Memory), SRAM (Static RAM), and non-volatile memory devices such as ROM (Read Only Memory), MROM (Mask ROM), PROM (Programmable ROM), EPROM (Erasable ROM), EEPROM (Electrically Erasable ROM), FRAM (registered trademark) (Ferromagnetic ROM), PRAM (Phase change RAM), MRAM (Magnetic RAM), RRAM (registered trademark) (Resistive RAM), flash memory, and the like.

[0035] Specifically, the data processing device 10 can write / read stored data to / from the plurality of data storage devices 20, 30, 40, 50, 60, 70, 80, 90. In particular, the data processing device 10 can replicate each of the plurality of stored data D1, D2, D3, D4, D5 and store them overlappingly in N data storage devices out of the plurality of data storage devices 20, 30, 40, 50, 60, 70, 80, 90. At this time, N can be set to a natural number of 2 or more. In the embodiment of the present invention, N is set to 3 for explanation.

[0036] More specifically, among the plurality of stored data D1, D2, D3, D4, D5, the data processing apparatus 10 can store the first stored data D1 redundantly in three of the plurality of data storage apparatuses 20, 30, 40, 50, 60, 70, 80, 90, that is, the first storage apparatus 20, the third storage apparatus 40, and the eighth storage apparatus 90. At this time, the first stored data D1 stored in the first storage apparatus 20 and the third storage apparatus 40 can be stored in an uncompressed state. In contrast, the first stored data D1 stored in the eighth storage apparatus 90 can be stored in a compressed state. In this way, the size of the first stored data D1 stored in the eighth storage apparatus 90 in a compressed state can be made smaller than the first stored data D1 stored in the first storage apparatus 20 and the third storage apparatus 40 in an uncompressed state, respectively.

[0037] Also, when performing an access operation on the first stored data D1, the data processing apparatus 10 can preferentially perform an access operation on the first stored data D1 in the first storage apparatus 20 or the third storage apparatus 40 in which the first stored data D1 is stored in an uncompressed state. If, for any one of the first storage apparatus 20 and the third storage apparatus 40, the access operation on the first stored data D1 fails, the data processing apparatus 10 can perform the access operation on the first stored data D1 in the remaining one (40 or 20) of the storage apparatuses and the operation of decompressing the first stored data D1 stored in the eighth storage apparatus 90 in a compressed state in parallel. Therefore, regardless of the result of the access operation on the first stored data D1 in the remaining one (40 or 20) of the storage apparatuses, the first stored data D1 that has been decompressed and switched to an uncompressed state can be stored in the eighth storage apparatus 90. That is, even when the access operation on the first stored data D1 in the remaining one (40 or 20) of the storage apparatuses fails, the data processing apparatus 10 can access the first stored data D1 that is in an uncompressed state in the eighth storage apparatus 90.

[0038] In the foregoing description, the operation in which the data processing apparatus 10 stores the first stored data D1 in three storage devices, namely, the first storage device 20, the third storage device 40, and the eighth storage device 90, and compresses and stores it in one of the storage devices can be directly applied to the second to fifth stored data D2, D3, D4, and D5 as well. That is, the second stored data D2 can be stored overlapping the first, second, and fourth storage devices 20, 30, and 50, and among them, it can be stored in a compressed state only in the fourth storage device 50. Similarly, the third stored data D3 can be stored overlapping the fifth, sixth, and seventh storage devices 60, 70, and 80, and among them, it can be stored in a compressed state only in the sixth storage device 70. Also, the fourth stored data D4 can be stored overlapping the third, sixth, and eighth storage devices 40, 70, and 90, and among them, it can be stored in a compressed state only in the third storage device 40. Further, the fifth stored data D5 can be stored overlapping the fourth, fifth, and seventh storage devices 50, 60, and 80, and among them, it can be stored in a compressed state only in the fifth storage device 60.

[0039] On the one hand, the data processing device 10 can classify three storage devices that respectively store multiple pieces of stored data D1, D2, D3, D4, D5 redundantly among the plurality of data storage devices 20, 30, 40, 50, 60, 70, 80, 90 as the "first storage device", the "second storage device", and the "third storage device", and classify the remaining storage devices as "general storage devices". At this time, each of the plurality of pieces of stored data D1, D2, D3, D4, D5 in the uncompressed state can be stored in the "first storage device" and the "second storage device", and each of the plurality of pieces of stored data D1, D2, D3, D4, D5 in the compressed state can be stored in the "third storage device". For example, based on whether redundant storage of the first piece of stored data D1 is possible, the data processing device 10 classifies the first storage device 20 among the plurality of data storage devices 20, 30, 40, 50, 60, 70, 80, 90 as the "first storage device", the third storage device 40 as the "second storage device", the eighth storage device 90 as the "third storage device", and the remaining second, fourth to seventh storage devices 30, 50, 60, 70, 80 as "general storage devices". For another example, based on whether redundant storage of the second piece of stored data D2 is possible, the data processing device 10 classifies the first storage device 20 among the plurality of data storage devices 20, 30, 40, 50, 60, 70, 80, 90 as the "first storage device", the second storage device 30 as the "second storage device", the fourth storage device 50 as the "third storage device", and the remaining third, fifth to eighth storage devices 40, 60, 70, 80, 90 as "general storage devices". For another example, based on whether redundant storage of the third piece of stored data D3 is possible, the data processing device 10 classifies the fifth storage device 60 among the plurality of data storage devices 20, 30, 40, 50, 60, 70, 80, 90 as the "first storage device", the seventh storage device 80 as the "second storage device", the sixth storage device 70 as the "third storage device", and the remaining first to fourth and eighth storage devices 20, 30, 40, 50, 90 as "general storage devices".For another example, based on whether the fourth stored data D4 can be redundantly stored, the data processing apparatus 10 classifies the sixth storage device 70 among the plurality of data storage devices 20, 30, 40, 50, 60, 70, 80, 90 as the "first storage device", the eighth storage device 90 as the "second storage device", the third storage device 40 as the "third storage device", and the remaining first, second, fourth, fifth, and seventh storage devices 20, 30, 50, 60, 80 as "general storage devices". For another example, based on whether the fifth stored data D5 can be redundantly stored, the data processing apparatus 10 classifies the fourth storage device 50 among the plurality of data storage devices 20, 30, 40, 50, 60, 70, 80, 90 as the "first storage device", the seventh storage device 80 as the "second storage device", the fifth storage device 60 as the "third storage device", and the remaining first to third, sixth, and eighth storage devices 20, 30, 40, 70, 90 as "general storage devices".

[0040] FIG. 2A and FIG. 2B are diagrams shown for explaining an example of a data writing operation of a data storage system according to an embodiment of the present invention.

[0041] First, as described with reference to FIG. 1, the data storage system according to the embodiment of the present invention can include a data processing apparatus 10 and a plurality of data storage devices 20, 30, 40, 50, 60, 70, 80, 90.

[0042] As shown in FIGS. 2A and 2B, an example of the writing operation for storing the storage data generated by the data processing apparatus 10 in three of the plurality of data storage apparatuses 20, 30, 40, 50, 60, 70, 80, 90 can be understood. That is, in FIGS. 2A and 2B, the data processing apparatus 10 selects each of the first storage apparatus 20, the second storage apparatus 30, and the third storage apparatus 40 among the plurality of data storage apparatuses 20, 30, 40, 50, 60, 70, 80, 90 described in FIG. 1 above and assumes that they are set to each of the "first storage apparatus", the "second storage apparatus", and the "third storage apparatus". Therefore, in the following description, the reference numeral of the "first storage apparatus" is defined as "20" indicating the first storage apparatus, the reference numeral of the "second storage apparatus" is "30" indicating the second storage apparatus, and the reference numeral of the "third storage apparatus" is defined as "40" indicating the third storage apparatus.

[0043] Specifically, after generating the storage data (11), the data processing apparatus 10 can transmit it to the first storage apparatus 20.

[0044] The first storage apparatus 20 can store the storage data transmitted from the data processing apparatus 10 in its internal storage space (21).

[0045] Also, the first storage apparatus 20 can transmit the storage data to the second storage apparatus 30. Referring to FIG. 2A according to the embodiment, the first storage apparatus 20 can transmit the storage data to the second storage apparatus 30 in the section where the operation of storing the storage data in its internal storage space (21) is performed. Therefore, the time point (22) when the storage of the storage data is completed in the first storage apparatus 20 can be later than the time point when the storage data is transmitted to the second storage apparatus 30. Referring to FIG. 2B according to another embodiment, the first storage apparatus 20 can transmit the storage data to the second storage apparatus 30 after the operation (21) of storing the storage data in its internal storage space is completed (22). Therefore, the time point (22) when the storage of the storage data is completed in the first storage apparatus 20 can be earlier than the time point when the storage data is transmitted to the second storage apparatus 30.

[0046] And the first storage device 20 can refrain from generating first storage information indicating that the storage data has been completely stored (22) in the internal storage space. That is, even after the storage data has been completely stored (22) in the internal storage space of the first storage device 20, the first storage device 20 can refrain from generating the first storage information until the second storage information indicating that the storage data transmitted to the second storage device 30 has been completely stored in the internal storage space of the second storage device 30 is transmitted from the second storage device 30. Therefore, the fact that the first storage device 20 has generated the first storage information can mean that the storage data has been completely stored in both the first storage device 20 and the second storage device 30. Further, the first storage device 20 can generate the first storage information and transmit it to the data processing device 10.

[0047] In this way, in response to the transmission of the first storage information from the first storage device 20, the data processing device 10 can know that the storage data has been stored in both the first storage device 20 and the second storage device 30. Therefore, the data processing device 10 can allow an access operation to the storage data (12) in response to the transmission of the first storage data from the first storage device 20.

[0048] And the second storage device 30 can store the storage data transmitted from the first storage device 20 in the internal storage space (31).

[0049] Also, the second storage device 30 can transmit the stored data to the third storage device 40. Referring to FIG. 2A according to the embodiment, the second storage device 30 can transmit the stored data to the third storage device 40 during the period in which the operation (31) of storing the stored data in the internal storage space is performed. Therefore, the time point (32) when the storage of the stored data in the second storage device 30 is completed can be later than the time point when the stored data is transmitted to the third storage device 40. Referring to FIG. 2B according to another embodiment, the second storage device 30 can transmit the stored data to the third storage device 40 after the operation (31) of storing the stored data in the internal storage space is completed (32). Therefore, the time point (32) when the storage of the stored data in the second storage device 30 is completed can be earlier than the time point when the stored data is transmitted to the third storage device 40.

[0050] Further, the second storage device 30 can generate second storage information indicating that the storage of the stored data in the internal storage space is completed (32) at the time point when the storage of the stored data in the internal storage space is completed. That is, the second storage device 30 can generate the second storage information in response to the completion (32) of the storage of the stored data in the internal storage space, regardless of whether the stored data transmitted to the third storage device 40 is stored in the storage space inside the third storage device 40, and transmit the second storage information to the first storage device 20.

[0051] And the third storage device 40 can compress and store the stored data transmitted from the second storage device 30 in the internal storage space (41). That is, the third storage device 40 can compress the stored data transmitted from the second storage device 30 so as to have a size smaller than the size of the stored data stored in each of the first storage device 20 and the second storage device 30, and store the compressed data in the internal storage space.

[0052] Further, the third storage device 40 can generate third storage information indicating that the storage of the stored data is completed (42) in a state where the stored data is compressed in the internal storage space, at the point in time when the storage of the stored data is completed (42) in a state where the stored data is compressed in the internal storage space. At this time, regardless of whether the stored data is stored in the storage space inside the third storage device 40, in response to the completion (32) of the storage of the stored data in the storage space inside the second storage device 30, the second storage device 30 transmits the second storage information to the first storage device 20. Therefore, there is no need to transmit the third storage information generated by the third storage device 40 to the second storage device 30. That is, the third storage device 40 can directly transmit the third storage information generated in response to the completion (42) of the storage of the stored data in a state where the stored data is compressed in the internal storage space to the data processing device 10.

[0053] On the other hand, since the stored data stored in the third storage device 40 is in a compressed state, it cannot be directly accessed by the data processing device 10. Also, the data processing device 10 can permit (12) an access operation to the stored data at the point in time when the storage of the stored data is completed in the first storage device 20 and the second storage device 30, that is, at the point in time when the first storage information is transmitted from the first storage device 20. Therefore, the operation of the third storage device 40 to compress and store the stored data in the internal storage space can be performed in parallel with the operation (12) of the data processing device 10 to permit access to the stored data from the first storage device 20 or the second storage device 30. For example, at the point in time when the data processing device 10 accesses the stored data from the first storage device 20, the third storage device 40 can be in the middle of performing the operation of compressing and storing the stored data in the internal storage space.

[0054] FIGS. 3A to 3D are diagrams shown to explain an example of a data read operation of a data storage system according to an embodiment of the present invention.

[0055] As shown in FIGS. 3A to 3D, in the state where the writing operation described in FIGS. 2A and 2B above is completed, that is, the data processing apparatus 10 generates storage data and stores it overlappingly in the first storage device 20, the second storage device 30, and the third storage device 40. It can be assumed that the third storage device 40 stores the data in a compressed state. Therefore, it can be in a state where the storage data is stored (23) in the first storage device 20, the storage data is stored (33) in the second storage device 30, and the storage data is stored in a compressed manner (43) in the third storage device 40.

[0056] As shown in FIG. 3A (a), the data processing apparatus 10 can transmit an access request for the storage data to the first storage device 20.

[0057] In response to the access request transmitted from the data processing apparatus 10, the first storage device 20 can attempt to access the storage data stored in the internal storage space. As a result, when the access to the storage data is successful (24), the first storage device 20 can transmit an access success response to the data processing apparatus 10.

[0058] In such a case, since the access to the storage data requested by the data processing apparatus 10 is successful in the first storage device 20, no operation needs to be performed on the second storage device 30 and the third storage device 40.

[0059] As shown in FIG. 3A (b), the data processing apparatus 10 can transmit an access request for the storage data to the first storage device 20.

[0060] In response to the access request transmitted from the data processing apparatus 10, the first storage device 20 can attempt to access the storage data stored in the internal storage space. As a result, when the access to the storage data fails (25), the first storage device 20 can transmit an access failure response to the data processing apparatus 10.

[0061] In response to the access failure response transmitted from the first storage device 20, the data processing device 10 can send an access request for the stored data to the second storage device 30. Further, in response to the access failure response transmitted from the first storage device 20, the data processing device 10 can send a decompression request for the stored data to the third storage device 40. That is, when an access failure response is transmitted from the first storage device 20, the data processing device 10 can send an access request again to the second storage device 30 in which the stored data is replicated and stored in an uncompressed state, and can send a decompression request for the stored data to the third storage device 40 in which the stored data is replicated and stored in a compressed state.

[0062] In response to the access request transmitted from the data processing device 10, the second storage device 30 can attempt to access the stored data stored in the internal storage space. As a result, when the stored data is successfully accessed (34), the second storage device 30 can send an access success response to the data processing device 10.

[0063] In response to the decompression request transmitted from the data processing device 10, the third storage device 40 can decompress the stored data stored in a compressed state in the internal storage space (44).

[0064] At this time, the operation of accessing the stored data stored in the internal storage space of the second storage device 30 and the operation of decompressing the stored data stored in a compressed state in the internal storage space in the third storage device 40 can be performed in parallel.

[0065] As shown in FIG. 3B, the data processing device 10 can send an access request for the stored data to the first storage device 20.

[0066] The first storage device 20 can attempt to access the stored data in the internal storage space in response to an access request transmitted from the data processing device 10. As a result, when the stored data fails to be accessed (25), the first storage device 20 can transmit an access failure response to the data processing device 10.

[0067] The data processing device 10 can transmit an access request for the stored data to the second storage device 30 in response to the access failure response transmitted from the first storage device 20. Also, the data processing device 10 can transmit a decompression request for the stored data to the third storage device 40 in response to the access failure response transmitted from the first storage device 20. That is, when an access failure response is transmitted from the first storage device 20, the data processing device 10 can transmit an access request again to the second storage device 30 where the stored data is replicated and stored in an uncompressed state, and can transmit a decompression request for the stored data to the third storage device 40 where the stored data is replicated and stored in a compressed state.

[0068] The second storage device 30 can attempt to access the stored data in the internal storage space in response to an access request transmitted from the data processing device 10. As a result, when the stored data fails to be accessed (36), the second storage device 30 can transmit an access failure response to the data processing device 10.

[0069] The data processing device 10 can send an access request for the stored data to the third storage device 40 in response to the access failure response transmitted from the second storage device 30. At this time, the third storage device 40 can be in the middle of performing a decompression operation on the stored data stored in a compressed state inside in response to the decompression request transmitted from the data processing device 10 corresponding to the access failure response transmitted from the first storage device 20. If, when an access request is transmitted from the data processing device 10, the decompression operation for the stored data stored in a compressed state inside the third storage device 40 is not completed, after the decompression operation is completed and the stored data is switched to an uncompressed state, the third storage device 40 can perform an access operation on the stored data in response to the access request from the data processing device 10.

[0070] The third storage device 40 can attempt to access the stored data stored in the internal storage space in response to the access request transmitted from the data processing device 10. As a result, if the stored data is successfully accessed (47), the third storage device 40 can send an access success response to the data processing device 10. If the stored data fails to be accessed by the third storage device 40, although not shown in the drawings, the access operation for the stored data will ultimately be in a failed state, and the access operation for the stored data will not be able to be completed normally.

[0071] As shown in FIG. 3C, it is assumed that the read operation described in (b) of FIG. 3A described above has been completed, that is, the access operation of the stored data attempted by the data processing apparatus 10 to the first storage apparatus 20 has failed (25), the access operation of the stored data attempted to the second storage apparatus 30 has succeeded (34), the decompression request to the third storage apparatus 40 has been completed, and the uncompressed stored data is stored inside the third storage apparatus 40. Therefore, the stored data that was stored in the first storage apparatus 20 but the access failed is in an invalidated (26) state, the stored data is stored (33) in the second storage apparatus 30, and the stored data can be stored (45) in the third storage apparatus 40.

[0072] Specifically, the stored data that was stored in the first storage apparatus 20 but the access failed can be invalidated (26). Therefore, the first storage apparatus 20 can be reset to a general storage apparatus, that is, the first storage apparatus 20 that does not store the stored data redundantly (27). In other words, among the plurality of storage apparatuses 20, 30, 40, 50, 60, 70, 80, 90 by the data processing apparatus 10 in FIGS. 2A and 2B described above, the first storage apparatus 20 that was selected and set as the "first storage apparatus" can be reset by the data processing apparatus 10 to a "general storage apparatus" that is not the "first storage apparatus" while the stored data stored inside is invalidated (26) in FIG. 3C (27).

[0073] After being stored in the second storage device 30, the stored data for which access has been successful can then be stored in the second storage device 20 (33). At this time, the data processing device 10 may have reset the first storage device 20, which had been previously selected and set as the "first storage device", to a "general storage device". Thereby, the data processing device 10 can reset the second storage device 30, which had been previously selected and set as the "second storage device", to the "first storage device". That is, in FIGS. 2A and 2B described above, among the plurality of storage devices 20, 30, 40, 50, 60, 70, 80, 90, the second storage device 30, which had been selected and set as the "second storage device" by the data processing device 10, can be reset to the "first storage device" by the data processing device 10 in FIG. 3C (35).

[0074] After being compression-stored in the third storage device 40, the stored data for which a decompression operation has been performed in response to a decompression request can be stored in the third storage device 40 in an uncompressed state (45). At this time, the data processing device 10 may have reset the first storage device 20, which had been previously selected and set as the "first storage device", to a "general storage device", and reset the second storage device 30, which had been previously selected and set as the "second storage device", to the "first storage device". Thereby, the data processing device 10 can reset the third storage device 40, which had been previously selected and set as the "third storage device", to the "second storage device". That is, in FIGS. 2A and 2B described above, among the plurality of storage devices 20, 30, 40, 50, 60, 70, 80, 90, the third storage device 40, which had been selected and set as the "third storage device" by the data processing device 10, can be reset to the "second storage device" by the data processing device 10 in FIG. 3C (46).

[0075] Then, the data processing device 10 can select the fourth storage device 50, which was classified as a "general storage device" because the storage data was not stored in it among the plurality of storage devices 20, 30, 40, 50, 60, 70, 80, 90 by the data processing device 10 in FIGS. 2A and 2B described above, and reconfigure it as the "third storage device". That is, the fourth storage device 50, which was selected and configured as a "general storage device" among the plurality of storage devices 20, 30, 40, 50, 60, 70, 80, 90 by the data processing device 10 in FIGS. 2A and 2B described above, can be reconfigured as the "third storage device" by the data processing device 10 in FIG. 3C (51).

[0076] As described in the above explanation, due to the storage data for which the access failed, the storage devices corresponding to each of the "first storage device", the "second storage device", and the "third storage device" can be reconfigured. In summary, in FIGS. 2A and 2B described above, the "first storage device" corresponded to the first storage device 20, but in FIG. 3C in which the reconfiguration operation (35) was performed, the "first storage device" can be in a form corresponding to the second storage device 30. Similarly, in FIGS. 2A and 2B described above, the "second storage device" corresponded to the second storage device 30, but in FIG. 3C in which the reconfiguration operation (46) was performed, the "second storage device" can be in a form corresponding to the third storage device 40. Similarly, in FIGS. 2A and 2B described above, the "third storage device" corresponded to the third storage device 40, but in FIG. 3C in which the reconfiguration operation (51) was performed, the "third storage device" can be in a form corresponding to the fourth storage device 50. Therefore, in the following description of FIG. 3C, the reference numeral of the "first storage device" is defined as "30" indicating the second storage device, the reference numeral of the "second storage device" is defined as "40" indicating the third storage device, and the reference numeral of the "third storage device" is defined as "50" indicating the fourth storage device.

[0077] After the above-described reset operations (27, 35, 46, 51), the first storage device 30 and the second storage device 40 can be in a state where the storage data is stored in an uncompressed state. On the other hand, the third storage device 50 can be in a state where no storage data is stored. Therefore, the data processing device 10 can transmit the storage data stored in the second storage device 40 to the third storage device 50.

[0078] The third storage device 50 can compress and store the storage data transmitted from the second storage device 40 in the internal storage space (52). That is, the third storage device 50 can compress the storage data transmitted from the second storage device 40 so as to have a size smaller than the size of the storage data stored in each of the first storage device 30 and the second storage device 40 and store it in the internal storage space.

[0079] In addition, the third storage device 50 can generate third storage information indicating that the storage is completed (53) in a state where the storage data is compressed in the internal storage space at the time when the storage is completed (53) in a state where the storage data is compressed in the internal storage space. At this time, since the second storage device 40 already stores the storage data in an uncompressed state, it is not necessary to transmit the third storage information generated by the third storage device 50 to the second storage device 40. That is, the third storage device 50 can directly transmit the third storage information generated in response to the completion (53) of the storage in a state where the storage data is compressed in the internal storage space to the data processing device 10.

[0080] On the other hand, since the storage data stored in the third storage device 50 is in a compressed state, it cannot be directly accessed by the data processing device 10. Also, since the storage data has already been stored in the first storage device 30 and the second storage device 40 in a superimposed manner before being stored in the third storage device 50 in a compressed state, as described in FIG. 2B, the data processing device 10 can be in a state where access operations to the storage data have already been permitted (12). Therefore, the operation of the third storage device 50 to compress and store the storage data in its internal storage space can be performed in parallel with the operation of the data processing device 10 to permit access to the storage data from the first storage device 30 or the second storage device 40. For example, when the data processing device 10 accesses the storage data from the first storage device 30, the third storage device 50 can be in the middle of performing the operation of compressing and storing the storage data in its internal storage space.

[0081] As shown in FIG. 3D, assume that the read operation described in FIG. 3B above has been completed, that is, the access operation to the storage data attempted by the data processing device 10 on the first storage device 20 has failed (25), then the access operation to the storage data attempted on the second storage device 30 has also failed (36), and the decompression request for the third storage device 40 has been completed and the access operation to the third storage device 40 has been successful (47) with the uncompressed storage data stored inside the third storage device 40. Therefore, the storage data stored in the first storage device 20 but with access failure is in an invalidated (26) state, the storage data stored in the second storage device 30 but with access failure is also in an invalidated (37) state, and the storage data can be in a state where it is stored (45) in the third storage device 40.

[0082] Specifically, the stored data that was stored in the first storage device 20 but access to which failed can be invalidated (26). Also, the stored data that was stored in the second storage device 30 but access to which failed can be invalidated (37). However, the third storage device 40 can be in a state (45) where the stored data is stored normally. That is, the stored data is stored only in the third storage device 40 alone and can be in a state where it is not stored redundantly in any storage device. Therefore, the data processing device 10 can interrupt the access operation to the stored data (13).

[0083] In such a state, each of the first storage device 20 and the second storage device 30 can be reset to a general storage device, that is, the first storage device 20 and the second storage device 30 that do not store the stored data redundantly (27, 38). In other words, in FIGS. 2A and 2B described above, among the plurality of storage devices 20, 30, 40, 50, 60, 70, 80, 90 selected and set as the "first storage device" by the data processing device 10, the first storage device 20 can be reset by the data processing device 10 to a "general storage device" that is not the "first storage device" while the stored data stored inside is invalidated (26) in FIG. 3D (27). Also, in FIGS. 2A and 2B described above, among the plurality of storage devices 20, 30, 40, 50, 60, 70, 80, 90 selected and set as the "second storage device" by the data processing device 10, the second storage device 30 can be reset by the data processing device 10 to a "general storage device" that is not the "second storage device" while the stored data stored inside is invalidated (37) in FIG. 3D (38).

[0084] After being compressed and stored in the third storage device 40, the stored data for which the decompression operation and the access operation have all been successful can then be stored in the third storage device 40 continuously (45). At this time, the data processing device 10 may have reset each of the first and second storage devices 20 and 30 that had been selected and set as the "first storage device" and the "second storage device" to the "general storage device". Thereby, the data processing device 10 can reset the third storage device 40 that had been selected and set as the "third storage device" to the "first storage device". That is, in FIGS. 2A and 2B described above, among the plurality of storage devices 20, 30, 40, 50, 60, 70, 80, 90 by the data processing device 10, the third storage device 40 that had been selected and set as the "third storage device" can be reset to the "first storage device" by the data processing device 10 in FIG. 3D (48).

[0085] Then, the data processing device 10 can select the fourth and fifth storage devices 50 and 60 that had been classified as the "general storage device" because the stored data was not stored among the plurality of storage devices 20, 30, 40, 50, 60, 70, 80, 90 by the data processing device 10 in FIGS. 2A and 2B described above, and reset them to the "second storage device" and the "third storage device". That is, in FIGS. 2A and 2B described above, among the plurality of storage devices 20, 30, 40, 50, 60, 70, 80, 90 by the data processing device 10, the fourth and fifth storage devices 50 and 60 that had been selected and set as the "general storage device" can be reset to the "second storage device" and the "third storage device" by the data processing device 10 in FIG. 3C (54, 61).

[0086] As described above, due to the stored data for which access has failed, the storage devices corresponding to each of the "first storage device", "second storage device", and "third storage device" can be reset. To summarize, in FIGS. 2A and 2B described above, the "first storage device" corresponded to the first storage device 20, but in FIG. 3D in which the reset operation (48) was performed, the "first storage device" can be in a form corresponding to the third storage device 40. Similarly, in FIGS. 2A and 2B described above, the "second storage device" corresponded to the second storage device 30, but in FIG. 3D in which the reset operation (54) was performed, the "second storage device" can be in a form corresponding to the fourth storage device 50. Similarly, in FIGS. 2A and 2B described above, the "third storage device" corresponded to the third storage device 40, but in FIG. 3D in which the reset operation (61) was performed, the "third storage device" can be in a form corresponding to the fifth storage device 60. Therefore, in the following description of FIG. 3D, the reference numeral of the "first storage device" is defined as "40" indicating the third storage device, the reference numeral of the "second storage device" is defined as "50" indicating the fourth storage device, and the reference numeral of the "third storage device" is defined as "60" indicating the fifth storage device.

[0087] After the reset operations (27, 38, 48, 54, 61) described above, the first storage device 40 can be in a state in which the stored data in the non-compressed state is stored. On the other hand, the second storage device 50 and the third storage device 60 can be in a state in which no stored data is stored. Therefore, the data processing device 10 can transmit the stored data stored in the first storage device 40 to the second storage device 50.

[0088] The second storage device 50 can store the stored data transmitted from the first storage device 40 in the internal storage space (55).

[0089] In addition, the second storage device 50 can transmit the stored data to the third storage device 60. As illustrated in the drawings according to the embodiment, the second storage device 50 can transmit the stored data to the third storage device 60 in a section where the operation (55) of storing the stored data in the internal storage space is performed. Therefore, the time point (56) when the storage of the stored data is completed in the second storage device 50 can be later than the time point when the stored data is transmitted to the third storage device 60. Different from what is illustrated in the drawings according to other embodiments, the second storage device 50 can transmit the stored data to the third storage device 60 after the operation (55) of storing the stored data in the internal storage space is completed (56). Therefore, the time point (56) when the storage of the stored data is completed in the second storage device 50 can be earlier than the time point when the stored data is transmitted to the third storage device 60.

[0090] Further, the second storage device 50 can generate second storage information indicating that the storage of the stored data in the internal storage space is completed (56) at the time point when the storage of the stored data in the internal storage space is completed. That is, the second storage device 50 can generate the second storage information in response to the completion (56) of the storage of the stored data in the internal storage space and transmit it to the first storage device 40 regardless of whether the stored data transmitted to the third storage device 60 is stored in the internal storage space of the third storage device 60.

[0091] At this time, since the stored data has already been stored in the first storage device 40, the first storage device 40 can generate the first storage information in response to the transmission of the second storage information from the second storage device 50 and transmit it to the data processing device 10.

[0092] In response to the transmission of the first storage information from the first storage device 40, the data processing device 10 can know that the storage data is stored redundantly in the first storage device 40 and the second storage device 50. Therefore, the data processing device 10 can allow the access operation to the storage data again in response to the transmission of the first storage data from the first storage device 40 (12).

[0093] Then, the third storage device 60 can compress and store the storage data transmitted from the second storage device 50 in the internal storage space (62). That is, the third storage device 60 can compress the storage data transmitted from the second storage device 50 so as to have a size smaller than the size of the storage data stored in each of the first storage device 40 and the second storage device 50, and store it in the internal storage space.

[0094] Also, the third storage device 60 can generate third storage information indicating that the storage is completed (63) with the storage data compressed in the internal storage space at the time when the storage is completed (63) with the storage data compressed in the internal storage space. At this time, regardless of whether the storage data is stored in the internal storage space of the third storage device 60, in response to the completion (56) of the storage of the storage data in the internal storage space of the second storage device 50, the second storage device 50 has transmitted the second storage information to the first storage device 40. Therefore, it is not necessary to transmit the third storage information generated by the third storage device 60 to the second storage device 50. That is, the third storage device 60 can directly transmit the third storage information generated in response to the completion (63) of the storage with the storage data compressed in the internal storage space to the data processing device 10.

[0095] On the other hand, since the storage data stored in the third storage device 60 is in a compressed state, it cannot be directly accessed by the data processing device 10. Further, when the storage data is completely stored overlappingly in the first storage device 40 and the second storage device 50, that is, when the first storage information is transmitted from the first storage device 40, the data processing device 10 can again permit (12) the access operation to the storage data. Therefore, the operation of the third storage device 60 to compress and store the storage data in the internal storage space can be performed in parallel with the operation (12) of the data processing device 10 to permit access to the storage data from the first storage device 40 or the second storage device 50. For example, when the data processing device 10 accesses the storage data from the first storage device 40, the third storage device 60 can be in the middle of performing the operation of compressing and storing the storage data in the internal storage space.

[0096] FIG. 4 is a diagram shown to explain the data writing operation of the data storage system according to an embodiment of the present invention applied to a Hadoop Distributed File System (HDFS).

[0097] As shown in FIG. 4, the data processing device 10 according to an embodiment of the present invention applied to a Hadoop Distributed File System (HDFS) can include an HDFS client 101 and a name node 102.

[0098] Assume that among the plurality of data storage devices 20, 30, 40, 50, 60, 70, 80, 90 described in FIG. 1 above, the three data nodes 20, 30, 40 connected to the Hadoop Distributed File System have selected and set each of the first storage device 20, the second storage device 30, and the third storage device 40 as the "first data node", the "second data node", and the "third data node", respectively. Therefore, in the following description, the reference numeral of the "first data node" is defined as "20" which refers to the first storage device, the reference numeral of the "second data node" is "30" which refers to the second storage device, and the reference numeral of the "third data node" is defined as "40" which refers to the third storage device.

[0099] Specifically, the HDFS client 101 is a Hadoop client and can perform functions such as requesting data writing and data reading from the plurality of data nodes 20, 30, 40 through cooperation with the name node 102.

[0100] When a data write request to the plurality of data nodes 20, 30, 40 is requested by the HDFS client 101, the name node 102 can notify and manage the data nodes 20, 30, 40 of information regarding how the data written by the HDFS client 101 should be stored in which data node.

[0101] More specifically, before transmitting a data write request to the plurality of data nodes 20, 30, 40, the HDFS client 101 first connects to the name node 102 to check whether the execution of the write request is possible and, if possible, which data node the write operation can be performed on (S1). When a response indicating that the execution of the write request is possible is transmitted from the name node 102 (S2), the HDFS client 101 can transmit the write request to the plurality of data nodes 20, 30, 40.

[0102] First, the HDFS client 101 can send a write request to the first data node 20.

[0103] In response to the write request transmitted from the HDFS client 101, the first data node 20 can store the write data in its internal storage space in an uncompressed state. At this time, the first data node 20 can receive an instruction for uncompressed storage of the write data from the name node 102 (A). That is, when the write request and the write data are transmitted from the HDFS client 101, the first data node 20 can receive an instruction for uncompressed storage of the write data from the name node 102, and thus store the write data in its internal storage space in an uncompressed state. In addition to the operation of storing the write data in its internal storage space in an uncompressed state, the first data node 20 can send the write request transmitted from the HDFS client 101 to the second data node 30 (S4).

[0104] In response to the write request transmitted from the first data node 20, the second data node 30 can store the write data in its internal storage space in an uncompressed state. At this time, the second data node 30 can receive an instruction for uncompressed storage of the write data from the name node 102 (B). That is, when the write request and the write data are transmitted from the first data node 20, the second data node 30 can receive an instruction for uncompressed storage of the write data from the name node 102, and thus store the write data in its internal storage space in an uncompressed state. In addition to the operation of storing the write data in its internal storage space in an uncompressed state, the second data node 30 can send the write request transmitted from the first data node 20 to the third data node 40 (S6).

[0105] In response to the write request transmitted from the second data node 30, the third data node 40 can store the write data in a compressed state in its internal storage space. At this time, the third data node 40 can receive an instruction for storing the write data in a compressed state from the name node 102 (C). That is, when the write request and the write data are transmitted from the second data node 30, the third data node 40 can receive an instruction for storing the write data in a compressed state from the name node 102, and thereby store the write data in a compressed state in its internal storage space.

[0106] Then, in response to the completion of the operation of storing the write data in an uncompressed state in its internal storage space, the second data node 30 can transmit a storage completion response ACK2 to the first data node 20 (S5). At this time, separately from the operation of transmitting the storage completion response ACK2 to the first data node 20, the second data node 30 can notify the name node 102 that the operation of storing the write data in an uncompressed state in its internal storage space has been completed (B).

[0107] Then, in response to the completion of the operation of storing the write data in an uncompressed state in its internal storage space and the transmission of the storage completion response ACK2 from the second data node 30, the first data node 20 can transmit a storage completion response ACK1 to the HDFS client 101 (S7). At this time, separately from the operation of transmitting the storage completion response ACK1 to the HDFS client 101, the first data node 20 can notify the name node 102 that the operation of storing the write data in an uncompressed state in its internal storage space has been completed (A).

[0108] After the storage completion response ACK1 is input from the first data node 20, the HDFS client 101 can start an access operation on the write data.

[0109] Then, the third data node 40 can notify the name node 102 that the operation of storing the write data in the internal storage space in a compressed state has been completed (C).

[0110] Through the operations as described above, the name node 102 can confirm that the uncompressed write data is stored in the first data node 20 and the second data node 30, and the compressed write data is stored in the third data node 40.

[0111] FIG. 5 is a diagram showing the data reading operation of the data storage system according to the embodiment of the present invention applied to the Hadoop distributed file system (HDFS: Hadoop Distributed File System).

[0112] As shown in FIG. 5, the data processing apparatus 10 according to the embodiment of the present invention applied to the Hadoop distributed file system (HDFS: Hadoop Distributed File System) can include an HDFS client 101 and a name node 102.

[0113] And it is assumed that the three data nodes 20, 30, and 40 connected to the Hadoop distributed file system each select the first storage device 20, the second storage device 30, and the third storage device 40 among the plurality of data storage devices 20, 30, 40, 50, 60, 70, 80, 90 described in FIG. 1 above and are set as the "first data node", the "second data node", and the "third data node" respectively. Therefore, in the following description, the reference numeral of the "first data node" is defined as "20" indicating the first storage device, the reference numeral of the "second data node" is "30" indicating the second storage device, and the reference numeral of the "third data node" is defined as "40" indicating the third storage device.

[0114] Specifically, the HDFS client 101 is a Hadoop client that can perform functions such as requesting data writing and data reading from multiple data nodes 20, 30, and 40 through cooperation with the NameNode 102.

[0115] When a data read request for multiple data nodes 20, 30, and 40 is requested by the HDFS client 101, the NameNode 102 can notify and manage the data nodes 20, 30, and 40 of information regarding how the data read-requested by the HDFS client 101 is stored in which data node.

[0116] More specifically, before transmitting a data read request to multiple data nodes 20, 30, and 40, the HDFS client 101 first connects to the NameNode 102 to check whether the execution of the read request is possible and, if possible, which data node stores the read data (F1). When a response indicating that the execution of the read request is possible is transmitted from the NameNode 102 (F2), the HDFS client 101 can transmit the read request to multiple data nodes 20, 30, and 40.

[0117] At this time, the state of how which data is stored in multiple data nodes 20, 30, and 40 before the HDFS client 101 makes a read request can be assumed to be the state where the writing operation described in FIG. 4 above is completed. That is, it can be a state where specific data is replicated and stored on top of each other in the first data node 20, the second data node 30, and the third data node 40. However, it can be assumed that specific uncompressed data is stored in each of the first data node 20 and the second data node 30, and compressed specific data is stored in the third data node 40.

[0118] In such a state, the HDFS client 101 can send a read request to the first data node 20 (F3). In response to the read request from the HDFS client 101, the first data node 20 can read the specific data stored therein and then send it to the HDFS client 101. However, in the drawing, it is assumed that the operation of reading the specific data from the first data node 20 fails (F4).

[0119] As assumed, since the operation of reading the specific data from the first data node 20 fails (F4), the HDFS client 101 can further send a read request to the second data node 30 (F5). In response to the read request from the HDFS client 101, the second data node 30 can read the specific data stored therein and then send it to the HDFS client 101 (F6).

[0120] At this time, since it is assumed that the operation of reading the specific data from the first data node 20 fails (F4), regardless of whether the read request to the second data node 30 is successful or not, the HDFS client 101 can request the decompression of the specific data stored in the internal storage space of the third data node 40 in a compressed state via the name node 102 (F7).

[0121] In response to the decompression request from the name node 102, the third data node 40 can perform a decompression operation on the specific data stored in the internal storage space in a compressed state.

[0122] The third data node 40 can notify the name node 102 that the execution of the decompression operation on the specific data has been completed.

[0123] Through the operations as described above, the name node 102 can confirm that the uncompressed specific data is stored not only in the first data node 20 and the second data node 30 but also in the third data node 40.

[0124] For reference, although not specifically illustrated in the drawings, after the operation in which specific data in an uncompressed state is stored in the third data node 40, the HDFS client 101 and the name node 102 will operate such that after further setting a new data node, the specific data can be stored in a compressed state in the storage space of the newly set data node. Also, after the operation in which specific data in an uncompressed state is stored in the third data node 40, the HDFS client 101 and the name node 102 can invalidate the specific data stored in the storage space of the data node where a previous read request failed, that is, the first data node 20. Therefore, the HDFS client 101 and the name node 102 can operate to store the specific data redundantly in three data nodes, store it in an uncompressed state in two data nodes, and store it in a compressed state in the remaining one data node.

[0125] As described above, the present invention is not limited by the foregoing embodiments and the attached drawings, and it will be apparent to those of ordinary skill in the technical field to which the present invention pertains that various substitutions, modifications, and changes are possible without departing from the technical idea of the present invention.

Claims

1. In an operation method of a data storage system including a data processing device and a plurality of storage devices, after selecting and setting first to third storage devices among the plurality of storage devices, storing the stored data of the data processing device in the first storage device, and sequentially transmitting and storing the stored data stored in the first storage device in a second storage device, a first storage step; after transmitting the stored data stored in the second storage device to the third storage device, a second storage step of compressing and storing the data inside the third storage device; after the first storage step is completed, a first access step in which the data processing device accesses the stored data stored in the first storage device; if the first access step fails, a second access step in which the stored data stored in the second storage device is accessed; if the first access step fails, a third storage step of decompressing the stored data stored in the third storage device in a compressed state; including, An operation method of a data storage system in which the second access step and the third storage step are performed in parallel.

2. The operation method of the data storage system according to claim 1, wherein the first access step and the second storage step are performed in parallel.

3. The third storage step is when the first access step fails, a step of decompressing the stored data stored in the third storage device in a compressed state; after the step of decompressing, a step of continuing to store the stored data in a decompressed state in the third storage device; The operation method of the data storage system according to claim 2, including.

4. After the third storage step, reset the second storage device to the first storage device, and after resetting the third storage device to the second storage device, among the plurality of storage devices, a reset step of reselecting and setting the third storage device; After the reset step, perform the second storage step to store the stored data in the third storage device in a compressed state; The method for operating a data storage system according to claim 3, further comprising:

5. The first storage step includes: A fourth storage step of transmitting the stored data generated by the data processing device to the first storage device and then storing it in the first storage device; A fifth storage step of transmitting the stored data from the first storage device to the second storage device and then storing it in the second storage device; The method for operating a data storage system according to claim 1, comprising:

6. The method for operating a data storage system according to claim 5, wherein the fourth storage step and the fifth storage step are performed in parallel.

7. In a method for operating a data storage system including a data processing device and a plurality of storage devices, After selecting and setting the first to third storage devices among the plurality of storage devices, storing the stored data of the data processing device in the first storage device, and sequentially transmitting and storing the stored data stored in the first storage device to the second storage device; a first storage step; A second storage step of transmitting the stored data stored in the second storage device to the third storage device and then compressing and storing it inside the third storage device; After the first and second storage information indicating that the storage of the stored data in each of the first and second storage devices is completed is transmitted to the data processing device, the data processing device accesses the stored data stored in the first storage device; a first access step; When the first access step fails, a second access step of accessing the storage data stored in the second storage device, When the first access step fails, a third storage step of decompressing the storage data stored in the third storage device in a compressed state, including The method of operating a data storage system in which the second access step and the third storage step are performed in parallel.

8. The method of operating a data storage system according to claim 7, wherein the first access step and the second storage step are performed in parallel.

9. The third storage step is When the first access step fails, a step of decompressing the storage data stored in the third storage device in a compressed state, After the decompressing step, a step of continuously storing the decompressed storage data in the third storage device, The method of operating a data storage system according to claim 8, including

10. After the third storage step, reset the second storage device to the first storage device, and after resetting the third storage device to the second storage device, among the plurality of storage devices, a reset step of reselecting and setting the third storage device; After the reset step, perform the second storage step to store the storage data in the third storage device in a compressed state; The method of operating a data storage system according to claim 9, further including

11. The first storage step is After transmitting the storage data generated by the data processing device to the first storage device, a fourth storage step of storing it in the first storage device, After transmitting the stored data from the first storage device to the second storage device, a fifth storage step of storing the data in the second storage device; In response to completion of the fifth storage step, a first transmission step of generating the second storage information in the second storage device and transmitting the information to the first storage device; After the first transmission step, in response to completion of the fourth storage step, a second transmission step of generating the first storage information in the first storage device and transmitting the information to the data processing device; The method for operating a data storage system according to claim 7, comprising:

12. The method for operating a data storage system according to claim 11, wherein the fourth storage step and the fifth storage step are performed in parallel.

13. A plurality of storage devices; A data processing device that stores internally generated stored data in the first to third storage devices among the plurality of storage devices in an overlapping manner, stores the stored data in an uncompressed state in each of the first and second storage devices, and stores the stored data in a compressed state in the third storage device; Comprising: The data processing device: When an operation of accessing the stored data fails in any one of the first and second storage devices, the data processing device performs in parallel an operation of accessing the stored data in the remaining one of the storage devices and an operation of decompressing the stored data stored in the third storage device in a compressed state. A data storage system.

14. The data processing device generates the stored data and transmits the data to the first storage device. The first storage device performs in parallel an operation of storing the stored data transmitted from the data processing device internally in an uncompressed state and an operation of transmitting the stored data transmitted from the data processing device to the second storage device. The second storage device performs in parallel the operation of storing the storage data transmitted from the first storage device internally in an uncompressed state and the operation of transmitting the storage data transmitted from the first storage device to the third storage device. The data storage system according to claim 13, wherein the third storage device performs an operation of storing the storage data transmitted from the second storage device internally in a compressed state.

15. In response to the storage data transmitted from the first storage device being completely stored internally in an uncompressed state, the second storage device generates second storage information and transmits it to the first storage device. In response to the storage data transmitted from the data processing device being completely stored internally in an uncompressed state and the second storage information being transmitted from the second storage device, the first storage device generates first storage information and transmits it to the data processing device. In response to the first storage information being transmitted from the first storage device, the data processing device permits an access operation to the storage data. The data storage system according to claim 14, wherein in response to the storage data transmitted from the second storage device being completely stored internally in a compressed state, the third storage device generates third storage information and transmits it to the data processing device.

16. The data storage system according to claim 15, wherein the operation of storing the storage data transmitted from the second storage device internally in a compressed state in the third storage device and the operation of permitting an access operation to the storage data in the data processing device are performed in parallel.

Images