Data storage system and method

Abstract

An embodiment of the present invention discloses a data storage system and method. The system includes: a distributed message cache cluster, a data processing package, and a database; wherein the distributed message cache cluster is used to receive and store data to be stored sent by at least one producer; the data processing package is used to read the data to be stored stored by the distributed message cache cluster, and generate a data stream based on the read data to be stored, and send the data stream to the database; the database is used to receive the data stream and store the data to be stored in the data stream. The technical solution of the embodiment of the present invention solves the technical problem of the risk of data loss in existing data storage systems, fully reduces the risk of data loss, and further improves data security.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a data storage system and method. Background Art

[0002] With the advent of the big data era, data storage systems are increasingly required to store increasing amounts of data. Existing technologies typically store data directly in databases. Due to the large volume of data to be stored, data storage is slow, leading to data backlogs. This creates a technical risk of data loss if the database server fails or experiences a power outage, compromising data security. Summary of the Invention

[0003] The present invention provides a data storage system and method to reduce the risk of data loss during data storage and further improve data security.

[0004] According to one aspect of the present invention, a data storage system is provided, which includes: a distributed message cache cluster, a data processing package and a database; wherein,

[0005] The distributed message cache cluster is used to receive and store data to be stored sent by at least one producer;

[0006] The data processing program package is used to read the data to be stored stored in the distributed message cache cluster, generate a data stream based on the read data to be stored, and send the data stream to the database;

[0007] The database is used to receive the data stream and store the data to be stored in the data stream.

[0008] According to another aspect of the present invention, a data storage method is provided. The method comprises:

[0009] Receive and store data to be stored from at least one producer through a distributed message cache cluster;

[0010] Reading the data to be stored stored in the distributed message cache cluster through a data processing program package, generating a data stream based on the read data to be stored, and sending the data stream to a database;

[0011] The data stream is received through the database, and the data to be stored in the data stream is stored.

[0012] According to another aspect of the present invention, a computer-readable storage medium is provided, on which a computer program is stored. When the program is executed by a processor, the data storage method provided by any embodiment of the present invention is implemented.

[0013] The technical solution of the embodiment of the present invention is to form a data storage system through a distributed message cache cluster, a data processing package and a database. The data to be stored sent by at least one producer is received and stored by the distributed message cache cluster. In the embodiment of the present invention, a large amount of data to be stored can be stored in the distributed message cache cluster to alleviate the storage pressure of the database. The data to be stored stored in the distributed message cache cluster is read by the data processing package, and a data stream is generated based on the read data to be stored, and the data stream is sent to the database, so that the data to be stored in the distributed message cache cluster can be stored in the database, which can avoid data loss caused by failure of the server where the database is deployed. The data stream is received by the database, and the data to be stored in the data stream is stored. The technical solution of the embodiment of the present invention solves the technical problem of the risk of data loss in the existing data storage system, achieves a substantial reduction in the risk of data loss, and further improves the security of the data.

[0014] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present invention, nor is it intended to limit the scope of the present invention. Other features of the present invention will become readily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0016] Figure 1 A schematic structural diagram of a data storage system provided in accordance with the first embodiment of the present invention;

[0017] Figure 2 A schematic structural diagram of an optional data storage system provided in the second embodiment of the present invention;

[0018] Figure 3 A flowchart of a data storage method provided in Example 3 of the present invention. DETAILED DESCRIPTION

[0019] In order to enable those skilled in the art to better understand the solutions of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of the present invention.

[0020] It is understandable that the data involved in this technical solution (including but not limited to the data itself, the acquisition or use of the data) must comply with the requirements of relevant laws, regulations and relevant provisions.

[0021] Example 1

[0022] Figure 1 This is a schematic diagram of the structure of a data storage system provided in Example 1 of the present invention. This embodiment is applicable to situations where data is stored. The system can be implemented in hardware and / or software. The system specifically includes the following structure:

[0023] A distributed message cache cluster 10, a data processing program package 20 and a database 30; wherein the distributed message cache cluster 10 is used to receive and store data to be stored sent by at least one producer.

[0024] Among them, the distributed message cache cluster 10 can be a pre-built data storage cluster (such as a kafka cluster), which can be used to receive one or more data to be stored produced by at least one producer. The producer can be understood as an application that generates data to be stored (such as a social media application). The number of producers can be one, two or more. A producer can produce one or more data to be stored. The type of data to be stored can include at least one of text, numerical values, images, audio and video. In actual applications, the distributed message cache cluster 10 can receive data to be stored produced by multiple producers, wherein the number of data to be stored is multiple.

[0025] The data processing package 20 may be a data storage package developed using a programming language. The programming language may include at least one of the Java programming language, the C++ programming language, the C# programming language, and the C programming language, without specific limitation herein. The database 30 may include a PostgreSQL database 30, a MongoDB database 30, a Redis database 30, or an HBase database 30, among others. In an embodiment of the present invention, the database 30 is a PostgreSQL database 30. The advantage of using a PostgreSQL database 30 is that it supports database management systems with features such as transactions, subqueries, multi-version parallel control systems, and data integrity checks.

[0026] Specifically, the distributed message cache cluster 10 receives and stores the data to be stored sent by at least one producer. It can be understood that one or more producers write the data to be stored to the distributed message cache cluster (kafka cluster). In order to facilitate subsequent data processing, when the producer writes data to the distributed message cache cluster 10, the data to be stored can be classified according to the actual data storage requirements. Then, the same type of data to be stored can be stored in the corresponding topic (data topic) of the distributed message cache cluster 10 respectively. Among them, the data storage requirements can be data type, data size, and the scenario corresponding to the data. Optionally, before receiving the data to be stored produced by the producer, the distributed message cache cluster 10 can be built according to the actual data storage requirements.

[0027] The data processing package 20 is used to read the data to be stored stored in the distributed message cache cluster 10 , generate a data stream based on the read data to be stored, and send the data stream to the database 30 .

[0028] Specifically, the data processing package 20 can be used to pull the data to be stored in the distributed message cache cluster 10. The pulled data to be stored can then be converted into a data stream. The data stream is further forwarded to the database 30. To avoid computer performance degradation caused by frequent execution of the data processing package 20, the data processing package 20 can be used to pull the data to be stored in the distributed message cache cluster 10 based on a preset time interval (e.g., 5 seconds / time).

[0029] It is understood that the data processing package 20 can act as a consumer of the distributed message cache cluster 10, consuming data stored in the distributed message cache cluster 10. The number of data processing packages 20 can be one, two, or more. In actual applications, if the amount of data to be stored in the distributed message cache cluster 10 is large, that is, if the amount of data to be stored is large, multiple consumers can be set up, that is, multiple data processing packages 20 can be deployed as consumers. On this basis, in order to facilitate improving the efficiency of data storage, multiple consumers can be grouped, and each group can be separately regarded as a consumer group.

[0030] In the embodiment of the present invention, the group identifiers of consumers in the same consumer group are the same, which can avoid the phenomenon of repeated data storage. It should be noted that if different consumers belong to different consumer groups, consumers in different consumer groups can consume the same data to be stored.

[0031] Optionally, after pulling the data to be stored, it can be stored in a pre-created data storage collection (dataMap) based on the data theme. The data stored in the data storage collection can then be traversed. After the traversal is complete, a data list (dataList) can be obtained. The resulting data list can then be converted into a data stream (InputStream) to subsequently improve data storage efficiency.

[0032] Optionally, the data processing package 20 is further configured to establish a connection with the database 30 by executing a preset database 30 connection statement. The preset database 30 connection statement may include the database 30 connection address (IP), database 30 owner identification information, and a database 30 connection password. The preset database 30 connection statement may be used to connect to the database 30, allowing the database 30 to parse the data stream and store the data to be stored in the data stream.

[0033] The database 30 is used to receive the data stream and store the data to be stored in the data stream.

[0034] Specifically, the data stream sent by the data processing program package 20 is received by the database 30. After the database 30 receives the data stream, it can store the data to be stored in the data stream.

[0035] To facilitate subsequent use and analysis of data, in an embodiment of the present invention, the database 30 may include at least one data table, which is specifically used to receive a data stream and store the data to be stored in the data stream into the data table.

[0036] Specifically, based on the preset data storage rules, the data to be stored in the data stream is stored in the corresponding data table. In order to avoid storage anomalies caused by empty data (NULL) during the data storage process, the empty data is usually converted into an empty string. For example, NULL is as "".

[0037] Optionally, the data processing package 20 includes a data copy-into-warehouse statement corresponding to the database 30; the data processing package 20 is used to parse the data stream by executing the data copy-into-warehouse statement, obtain the data to be stored contained in the data stream, and store the data to be stored in the data table.

[0038] The data copy-in statement may be a predefined programming statement that can be used to copy the data to be stored in the data stream to the database 30. In an embodiment of the present invention, the data copy-in statement may be a statement defined based on the data entry method (copyIn method) in the Copy syntax.

[0039] Exemplarily, copyIn(”COPY table1(name, gender, age) FROM STDIN delimiter ‘,’ NULL as “”, fileInputStream);

[0040] Among them, table1 represents the table name. name (age), gender (gender), and age (age) are respectively the column names of table1. FROM STDIN means the data is obtained from the text stream. delimiter ‘,’ means that in the text, each line of data uses the English character ‘,’ as the delimiter. Exemplarily, “Zhang, male, 14”. fileInputStream represents the text stream. It can be understood that the data stream in the embodiments of the present invention can be a text stream.

[0041] Optionally, the distributed message cache cluster 10 is specifically configured to receive the data to be stored sent by at least one producer, determine the data topics of each data to be stored, classify the data to be stored according to the data topics, and store them.

[0042] Optionally, the data processing package 20 includes: a predefined data consumption method; the data processing package 20 is specifically configured to read the data to be stored corresponding to each data topic from the distributed message cache cluster 10 by calling the data consumption method, convert the read data to be stored into a data stream, and send the data stream to the database 30. In order to avoid the phenomenon of garbled characters after converting the data to be stored into a data stream, a data encoding format (such as UTF-8) is predefined in the data consumption method, so that the data to be stored can be converted into a data stream based on the preset data encoding format.

[0043] [[ID=,12]]For the convenience of subsequent data management and analysis, the data processing package 20 can be used to store the data to be stored corresponding to each data topic read into the data storage linked list corresponding to each data topic, traverse the data to be stored in each data storage list, and convert it into a data stream.

[0044] Among them, the corresponding relationship between the data topic and the data storage linked list can be one-to-one or one-to-many. In one embodiment, if the amount of data to be stored corresponding to a data topic is large, then for this data topic, one or more data linked lists can be created to store the data to be stored corresponding to this data topic.

[0045] Based on the embodiments of the present invention, the data processing package 20 is further configured to read the pre-configured data storage configuration file, determine the data tables corresponding to each data topic, and store the data to be stored corresponding to each data topic into the corresponding data tables.

[0046] Among them, the data storage configuration file can be used to configure the correspondence between data topics and data tables. Exemplarily, the data topics include Topic A and Topic B, and the data tables include Data Table 1 and Data Table 2. The data storage configuration file can configure the correspondence between Topic A and Data Table 1, and the correspondence between Topic B and Data Table 2; or, the data storage configuration file can configure the correspondence between Topic B and Data Table 1, and the correspondence between Topic A and Data Table 2. It should be noted that the correspondence between the data topics and data tables configured in the data storage configuration file can be set according to actual needs. Optionally, in the data storage configuration table, the arrangement order of each column of data in the data table can be pre-configured so that the data to be stored subsequently can be stored in the database 30 according to this arrangement order.

[0047] Optionally, the data storage configuration file is in the Extensible Markup Language (XML) format. In the embodiment of the present invention, the advantage of using the XML format data storage configuration file is that the corresponding relationship between the data table and the data subject can be flexibly configured.

[0048] The technical solution of the embodiment of the present invention is to form a data storage system through a distributed message cache cluster, a data processing package and a database. The data to be stored sent by at least one producer is received and stored by the distributed message cache cluster. In the embodiment of the present invention, a large amount of data to be stored can be stored in the distributed message cache cluster to alleviate the storage pressure of the database. The data to be stored stored in the distributed message cache cluster is read by the data processing package, and a data stream is generated based on the read data to be stored, and the data stream is sent to the database, so that the data to be stored in the distributed message cache cluster can be stored in the database, which can avoid data loss caused by failure of the server where the database is deployed. The data stream is received by the database, and the data to be stored in the data stream is stored. The technical solution of the embodiment of the present invention solves the technical problem of the risk of data loss in the existing data storage system, achieves a substantial reduction in the risk of data loss, and further improves the security of the data.

[0049] Example 2

[0050] As an optional example of an embodiment of the present invention, an embodiment of the present invention provides a schematic diagram of the structure of a data storage system. In this embodiment of the present invention, the producer can be an external application. The distributed message cache cluster can be a Kafka cluster. The data processing package can be the DataToDb program. The database can be a PostgreSQL database. The PostgreSQL database can include data tables, such as Table 1, Table 2, and Table 3.

[0051] like Figure 2As shown, the Kafka cluster receives data to be stored from external applications, and classifies and stores the data according to the data topic of the data to be stored. For example, the data to be stored is stored in Topic 1, Topic 2, and Topic 3.

[0052] By calling or executing the DataToDb program, the data to be stored in the Kafka cluster can be read, the read data to be stored can be converted into a text stream, and the text stream can be sent to the PostgerSQL database, that is, the text stream can be written to the PostgerSQL database. Specifically, writing the text stream to the PostgerSQL database can include: storing the data to be stored in the text stream into the PostgerSQL database. Optionally, storing the data to be stored in the text stream into the PostgerSQL database includes: according to the data subject of the data to be stored in the text stream, storing the data to be stored in the data table corresponding to the data subject.

[0053] The technical solution of the embodiment of the present invention solves the technical problem of data loss risk in existing data storage systems, fully reduces the risk of data loss, and further improves data security.

[0054] Example 3

[0055] Figure 3 This is a flow chart of a data storage method provided in Example 3 of the present invention. This method and the data storage system in the aforementioned embodiments are based on the same inventive concept. For details not fully described in the data storage method embodiments, reference can be made to the aforementioned data storage system embodiments. This method is applied to a data storage system consisting of a distributed message cache cluster, a data processing package, and a database. The method specifically includes the following steps:

[0056] S210: Receive data to be stored sent by at least one producer through a distributed message cache cluster and store the data.

[0057] S220: Read the data to be stored in the distributed message cache cluster through a data processing program package, generate a data stream based on the read data to be stored, and send the data stream to a database.

[0058] S230: Receive the data stream through the database, and store the data to be stored in the data stream.

[0059] Optionally, the database includes at least one data table, and the data stream is received through the database, and the data to be stored in the data stream is stored in the data table.

[0060] Optionally, the data processing package includes a data copy-into-database statement corresponding to the database; the data processing package executes the data copy-into-database statement to parse the data stream, obtain the data to be stored contained in the data stream, and store the data to be stored in the data table.

[0061] Optionally, the distributed message cache cluster receives data to be stored sent by at least one producer, determines a data subject of each data to be stored, and classifies and stores the data to be stored according to the data subject.

[0062] Optionally, the data processing package includes: a predefined data consumption method; calling the data consumption method through the data processing package, reading the data to be stored corresponding to each of the data topics from the distributed message cache cluster, converting the read data to be stored into the data stream, and sending the data stream to the database.

[0063] Optionally, the data processing package stores the read data to be stored corresponding to each of the data topics into a data storage linked list corresponding to each data topic, traverses the data to be stored in each data storage list, and converts it into the data stream.

[0064] Optionally, the data processing package reads a pre-configured data storage configuration file, determines a data table corresponding to each data topic, and stores the data to be stored corresponding to each data topic in the corresponding data table.

[0065] Optionally, the file format of the data storage configuration file is an extensible markup language file format.

[0066] Optionally, the data processing package establishes a connection with the database by executing a preset database connection statement.

[0067] The technical solution of the embodiment of the present invention is to form a data storage system through a distributed message cache cluster, a data processing package and a database. The data to be stored sent by at least one producer is received and stored by the distributed message cache cluster. In the embodiment of the present invention, a large amount of data to be stored can be stored in the distributed message cache cluster to alleviate the storage pressure of the database. The data to be stored stored in the distributed message cache cluster is read by the data processing package, and a data stream is generated based on the read data to be stored, and the data stream is sent to the database, so that the data to be stored in the distributed message cache cluster can be stored in the database, which can avoid data loss caused by failure of the server where the database is deployed. The data stream is received by the database, and the data to be stored in the data stream is stored. The technical solution of the embodiment of the present invention solves the technical problem of the risk of data loss in the existing data storage system, achieves a substantial reduction in the risk of data loss, and further improves the security of the data.

[0068] Note that the above are only preferred embodiments of the present invention and the technical principles employed. Those skilled in the art will understand that the present invention is not limited to the specific embodiments described herein, and that various obvious changes, readjustments, and substitutions can be made by those skilled in the art without departing from the scope of protection of the present invention. Therefore, although the present invention has been described in detail through the above embodiments, the present invention is not limited to the above embodiments and may include many other equivalent embodiments without departing from the concept of the present invention. The scope of the present invention is determined by the scope of the appended claims.

[0069] Example 4

[0070] A fourth embodiment of the present invention further provides a computer-readable storage medium having a computer program stored thereon, wherein when the program is executed by a processor, for example, the data storage method provided in the above embodiment of the present invention includes:

[0071] Receive and store data to be stored from at least one producer through a distributed message cache cluster;

[0072] Reading the data to be stored stored in the distributed message cache cluster through a data processing program package, generating a data stream based on the read data to be stored, and sending the data stream to a database;

[0073] The data stream is received through the database, and the data to be stored in the data stream is stored.

[0074] The computer storage medium of the embodiment of the present invention may adopt any combination of one or more computer-readable media. The computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or component, or any combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In this document, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device.

[0075] A computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, which carries computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transport a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0076] Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0077] The computer program code for performing the operations of the embodiments of the present invention can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0078] Note that the above are only preferred embodiments of the present invention and the technical principles employed. Those skilled in the art will understand that the present invention is not limited to the specific embodiments described herein, and that various obvious changes, readjustments, and substitutions can be made by those skilled in the art without departing from the scope of protection of the present invention. Therefore, although the present invention has been described in detail through the above embodiments, the present invention is not limited to the above embodiments and may include many other equivalent embodiments without departing from the concept of the present invention. The scope of the present invention is determined by the scope of the appended claims.

Claims

1. A data storage system, characterized in that: include: Distributed message cache cluster, data processing package and database; among them, The distributed message cache cluster is used to receive and store data to be stored sent by at least one producer; The data processing program package is used to read the data to be stored stored in the distributed message cache cluster, generate a data stream based on the read data to be stored, and send the data stream to the database; The database is configured to receive the data stream and store the data to be stored in the data stream; The database includes at least one data table, specifically configured to receive the data stream and store the data to be stored in the data stream into the data table; The data processing program package includes a data copy and storage statement corresponding to the database; The data processing program package is used to parse the data stream by executing the data copy-into-database statement, obtain the data to be stored contained in the data stream, and store the data to be stored in the data table.

2. The system according to claim 1, wherein: The distributed message cache cluster is specifically used to receive data to be stored sent by at least one producer, determine the data subject of each data to be stored, classify the data to be stored according to the data subject, and store it.

3. The system according to claim 2, characterized in that: The data processing package includes: a predefined data consumption method; The data processing package is specifically used to read the to-be-stored data corresponding to each of the data topics from the distributed message cache cluster by calling the data consumption method, convert the read to-be-stored data into the data stream, and send the data stream to the database.

4. The system according to claim 3, characterized in that The data processing program package is used to store the read data to be stored corresponding to each data topic into the data storage linked list corresponding to each data topic, traverse the data to be stored in each data storage list, and convert it into the data stream.

5. The system according to claim 3, wherein: The data processing package is also used to: Read the pre-configured data storage configuration file, determine the data table corresponding to each data topic, and store the data to be stored corresponding to each data topic in the corresponding data table.

6. The system according to claim 5, characterized in that The file format of the data storage configuration file is an extensible markup language file format.

7. The system according to claim 1, wherein: The data processing program package is further used to establish a connection with the database by executing a preset database connection statement.

8. A data storage method, characterized in that: include: Receive and store data to be stored from at least one producer through a distributed message cache cluster; Reading the data to be stored stored in the distributed message cache cluster through a data processing program package, generating a data stream based on the read data to be stored, and sending the data stream to a database; receiving the data stream through the database, and storing the data to be stored in the data stream; The database includes at least one data table, the data stream is received through the database, and the data to be stored in the data stream is stored in the data table; The data processing program package includes a data copy and storage statement corresponding to the database; The data processing package executes the data copy-into-database statement to parse the data stream, obtain the data to be stored contained in the data stream, and store the data to be stored in the data table.

Images