Data storage method and device, electronic equipment and storage medium

Abstract

The invention discloses a data storage method and apparatus, an electronic device and a storage medium. The method comprises the steps of obtaining to-be-written first business data in a business system; based on a write-in sequence of databases in a database cluster, determining a first database used for storing the first business data, the write-in sequence being periodically determined according to the difference between each database in the database cluster and a second database in reference data volume, the second database is the database with the maximum reference data volume in the database cluster; and writing the first business data into the first database.

Description

Technical Field

[0001] The present application relates to the field of database technology applications, and in particular to a data storage method, device, electronic device and storage medium. Background Art

[0002] As the amount of data continues to grow, the traditional single database storage method can no longer meet the needs of modern applications and faces increasing storage pressure. Data sub-database storage has become an important solution. Common sub-database storage methods include storage methods based on business features and based on modular operations.

[0003] However, common sub-database storage methods are often overly dependent on specific business characteristics, and have problems such as poor versatility and scalability, difficult maintenance, and data migration. Summary of the invention

[0004] The purpose of the embodiments of the present application is to provide a data storage method, device, electronic device and storage medium for solving the problems of poor versatility, poor scalability, difficult maintenance, data migration, etc. in the process of storing data in a target sub-library.

[0005] In order to achieve the above objectives, the present application embodiment adopts the following technical solutions: In a first aspect, an embodiment of the present application provides a data storage method, including: Acquire first business data to be written in the business system; Determine a first database for storing the first business data based on a write order of databases in the database cluster, wherein the write order is periodically determined according to a difference in reference data volume between each database in the database cluster and a second database, and the second database is a database with the largest reference data volume in the database cluster; The first business data is written into the first database.

[0006] In a second aspect, an embodiment of the present application provides a data storage device, including: An acquisition unit, used for acquiring first business data to be written in the business system; a determining unit, configured to determine a first database for storing the first business data based on a writing order of databases in a database cluster, wherein the writing order is periodically determined according to a difference in reference data volume between each database in the database cluster and a second database, and the second database is a database with the largest reference data volume in the database cluster; A writing unit is used to write the first business data into the first database.

[0007] In a third aspect, an embodiment of the present application provides an electronic device, comprising: a processor; and a memory for storing instructions executable by the processor; wherein the processor is configured to execute the instructions to implement the method described in the first aspect.

[0008] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium. When instructions in the storage medium are executed by a processor of an electronic device, the electronic device can execute the method described in the first aspect.

[0009] At least one of the above-mentioned technical solutions adopted in the embodiments of the present application can achieve the following beneficial effects: by obtaining the first business data to be written in the business system, based on the writing order of the databases in the database cluster, determining the first database for storing the first business data, and writing the first business data into the first database, wherein the writing order is periodically determined according to the difference in the amount of reference data between each database in the database cluster and the second database, the second database is the database with the largest amount of reference data in the database cluster, and based on the difference in the amount of reference data between each database in the database cluster and the second database, determining the writing order of the databases in the database cluster, and then writing the business data of the business system into the corresponding database, not only effectively solves the storage pressure and performance bottleneck faced by traditional single database storage, but also does not rely on specific business characteristics, thereby adapting to various complex business data, having strong versatility, and avoiding modular operations. When adding a new database, there is no need to migrate the data in the database cluster, having strong scalability, easy maintenance and management, making data storage more flexible and convenient. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings: Figure 1 A schematic diagram of a data storage method provided for an embodiment of the present application; Figure 2 A function graph of an impact factor and hard disk usage rate provided in one embodiment of the present application; Figure 3 A schematic diagram of the structure of a circular linked list provided for one embodiment of the present application; Figure 4 A schematic diagram of an application scenario of a data storage method provided by an embodiment of the present application; Figure 5 A schematic diagram of a flow chart of another data storage method provided for an embodiment of the present application; Figure 6A histogram of the amount of data in a database in a database cluster provided for one embodiment of the present application; Figure 7 A schematic diagram of the structure of a data storage device provided in one embodiment of the present application; Figure 8 A schematic diagram of the structure of an electronic device provided for one embodiment of the present application. DETAILED DESCRIPTION

[0011] In order to make the purpose, technical solution and advantages of the present application clearer, the technical solution of the present application will be clearly and completely described below in combination with the specific embodiments of the present application and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present application.

[0012] The terms "first", "second", etc. in this specification and claims are used to distinguish similar objects, and are not used to describe a particular order or precedence. It should be understood that the terms used in this way can be interchanged where appropriate so that the embodiments of the present application are implemented in a sequence other than those illustrated or described herein. In addition, "and / or" in this specification and claims means at least one of the connected objects, and the character " / " generally means that the objects associated with each other are in an "or" relationship.

[0013] As the amount of data continues to grow, the traditional single database storage method can no longer meet the needs of modern applications and faces increasing storage pressure. Data sub-database storage has become an important solution. Common sub-database storage methods include storage methods based on business features and based on modular operations.

[0014] However, common sub-database storage methods are often overly dependent on specific business characteristics, and have problems such as poor versatility and scalability, difficult maintenance, and data migration.

[0015] Specifically, storage methods based on business characteristics, for example: there is a certain business, the company conducts this business in different provinces, the business system can use the province code as the identification of the database sub-library, and the order data from different provinces can be saved in the sub-library corresponding to the province code. This method requires that the business data has very clear sub-library conditions as identification to be feasible, and its versatility is poor. Storage methods based on modulus operations, for example: taking user data as an example, when preparing to insert a user data, the system first generates a user ID, and then uses the ID to modulo the number of database sub-libraries. The modulo result is the sub-library to which this data is to be stored. Assuming that the system has 5 sub-libraries at the beginning, for a data with ID 2001, 2001mod5=1, at this time, the data with ID 2001 should be stored in sub-library No. 1; if the system is expanded to 6 sub-libraries, for the data with ID 2001, 2001mod6=3, at this time, the sub-library corresponding to the data with ID 2001 becomes library No. 3. This method has poor scalability. When adding a sub-database, a large amount of existing data needs to be migrated, and subsequent maintenance is more difficult.

[0016] In view of this, an embodiment of the present application provides a data storage method, which obtains the first business data to be written in the business system, determines the first database for storing the first business data based on the writing order of the databases in the database cluster, and writes the first business data into the first database, wherein the writing order is periodically determined according to the difference in the amount of reference data between each database in the database cluster and the second database, the second database is the database with the largest amount of reference data in the database cluster, and determines the writing order of the databases in the database cluster based on the difference in the amount of reference data between each database in the database cluster and the second database, and then writes the business data of the business system into the corresponding database. This method not only effectively solves the storage pressure and performance bottleneck faced by traditional single database storage, but also does not rely on specific business characteristics, thereby adapting to various complex business data, has strong versatility, and avoids modular operations. When a new database is added, there is no need to migrate the data in the database cluster. This method has strong scalability and is easy to maintain and manage, making data storage more flexible and convenient.

[0017] The technical solutions provided by various embodiments of the present application are described in detail below in conjunction with the accompanying drawings.

[0018] See also Figure 1 , is a flow chart of a data storage method provided by an embodiment of the present application. Figure 1 As shown, an embodiment of the present application provides a data storage method, which may include the following steps: S102: Acquire first business data to be written in the business system.

[0019] A business system usually refers to a system, application service or device used to process or manage specific business and business data, such as a customer relationship management system, enterprise resource planning system or other business management tools. Business data is the data generated or processed in a business system, such as information about business operations, customers, transactions, etc.

[0020] The business data generated or processed in the business system is usually stored and managed in a database. The embodiment of the present application adopts a sub-database storage method to store the business data generated in the business system, which can effectively solve the pressure and bottleneck faced by traditional single database storage. The embodiment of the present application records the business data to be written into the database cluster in the business system as the first business data.

[0021] The embodiment of the present application shows some implementation methods of the above S102. Of course, it should be understood that the above S102 can also be implemented in other ways, and the embodiment of the present application does not limit this.

[0022] S104: Determine a first database for storing first business data based on a write order of databases in the database cluster.

[0023] A database cluster is a whole composed of multiple databases. The multiple databases in the cluster are used to store business data generated by the business system.

[0024] The writing order is periodically determined based on the difference in reference data volume between each database in the database cluster and the second database, where the second database is the database with the largest reference data volume in the database cluster. The reference data volume is an indicator that reflects the storage situation of the database and is related to the data volume and storage space of the database.

[0025] As an optional implementation, the above writing order can be determined by the following method, which includes the following steps: S140, periodically monitor the performance data of each database in the database cluster, where the performance data includes the data volume.

[0026] S142: Determine the reference data volume of each database based on the data volume of each database.

[0027] S143, sorting the databases in the database cluster based on the difference between the reference data volume of each database and the reference data volume of the second database to obtain a write order.

[0028] The embodiment of the present application dynamically adjusts the write order of the database by periodically monitoring the performance data of each database in the database cluster, especially the data volume. By real-time monitoring the load status of each database, the write order can more accurately reflect the performance of the current database, thereby avoiding the performance bottleneck and unbalanced load problems caused by the fixed write strategy, thereby improving the overall performance and response speed of the system. It not only effectively solves the storage pressure and scalability problems faced by the traditional single database storage method, but also provides a more flexible and efficient data storage method for diversified business data, and has strong versatility and scalability.

[0029] As an optional method, in the above S140, by installing a device monitoring component such as Prometheus on the nodes corresponding to each database of the database cluster, the system calls the monitoring component interface to regularly pull the relevant performance data of the database. Currently, other monitoring methods can also be used, such as the monitoring function of the database itself, to obtain the performance data of the database cluster. The embodiment of the present application shows some implementation methods of the above S140. Of course, it should be understood that the above S140 can also be implemented in other ways, and the embodiment of the present application is not limited to this.

[0030] Performance data is various indicators used to evaluate the operating status of a database. As an example, performance data includes the response time, query speed, throughput, number of connections, error rate, and data volume of the database. The data volume is the actual size or quantity of data stored in the database, which can be measured in bytes, number of records, etc. In an embodiment of the present application, the reference data volume is determined based on the specific data volume of the database, and the reference data volume is the size or quantity of data stored in the database under specific conditions, such as performance requirements, capacity limitations, etc. As an optional method, in the above S142, the data volume of the database can be directly used as the reference data volume of the database. When the storage space or hard disk capacity of the database in the database cluster is the same, the data volume of the database is directly used as the reference data volume of the database. In the case where the storage space or hard disk capacity of the databases in the database cluster are different, it is necessary to consider the impact of the database's hard disk usage on its data volume. When the hard disk usage reaches a certain threshold, the database's data volume will be magnified to a certain extent, and then it is necessary to determine the impact factor of each database based on the hard disk usage, and then determine the reference data volume of each database. The write order of the databases in the database cluster is determined according to the actual storage situation of the database, and then true load balancing and uniform storage are achieved. By introducing the impact factor, better compatibility is achieved for databases with different storage space in the database cluster.

[0031] Specifically, as an optional implementation, in the case where the performance data includes hard disk usage, the above S142 may include the following steps: determining an impact factor of each database based on the hard disk usage of each database, where the impact factor is used to describe the impact of the hard disk usage of the database on the data volume of the database; For each database, the reference data volume of the database is determined based on the product of the data volume of the database and the impact factor of the database.

[0032] Hard disk utilization refers to the ratio of the currently used storage space of a storage device, such as a hard disk or solid-state drive, to the total storage space, that is, hard disk utilization = hard disk used capacity / hard disk total capacity, usually expressed as a percentage, reflecting the utilization of the storage device, that is, the database. The hard disk is the physical medium for storing data in the database, so the hard disk utilization directly affects the performance of the database. When the hard disk utilization approaches a certain threshold, its data volume will be greatly magnified. In the case where the total storage space of the database in the database cluster is different, the storage status of different databases cannot be evaluated and compared based on the data volume of the database alone, and thus it is impossible to determine which databases can still write data and which databases have reached the total capacity of the hard disk. Therefore, the embodiment of the present application determines the influencing factor of each database by introducing the hard disk utilization rate, and based on the influencing factor, determines the final reference data volume of the database to obtain the actual storage situation of the database, and then determines the write order. For example: Assuming that based on the hard disk usage rate of the database, the impact factor of the database is determined to be 9.545941546, and the amount of data in the database obtained through monitoring is 70,000, then the reference data amount of the database is 9.545941546×70,000=668215.9082.

[0033] The embodiment of the present application enhances the ability to dynamically adjust the amount of database reference data by incorporating the hard disk usage rate into performance data and introducing the concept of influencing factors, allowing the actual hard disk usage situation to be considered when evaluating database storage performance, thereby more accurately determining the amount of reference data for each database. As the load of the database system changes, the amount of reference data can be adjusted in real time, which can effectively avoid performance bottlenecks caused by insufficient storage resources, and can effectively solve the storage pressure and performance problems existing in the traditional single database storage method. Through detailed monitoring and analysis of the hard disk usage rate, a more flexible and efficient multi-database management solution can be achieved, ensuring that each database runs in the best state, and further improving the system's response speed and processing capabilities.

[0034] As an optional manner, the impact factor of each database may be determined based on the difference between a preset usage rate threshold and the hard disk usage rate of each database.

[0035] As an optional manner, the impact factor of each database may be determined in the following manner, including the following steps: obtaining a usage difference of the database based on a difference between a preset usage threshold and a hard disk usage of the database; An exponential operation is performed on the ratio of the preset usage rate threshold and the usage rate difference to obtain the impact factor of the database.

[0036] The preset usage rate threshold is a pre-set standard value. In practical applications, in order to ensure the safety and optimal range of database hard disk usage, the usage rate threshold is usually set to 90%. Of course, it can also be set according to actual needs, and this embodiment of the application does not limit this.

[0037] As an example, the impact factor of a database can be calculated by the formula: y=(n / (nx))^z, where y represents the impact factor, n represents the usage threshold, x represents the database disk usage, and z is an index, which is set according to objective conditions and is usually set to 1.5. For example, assuming that the usage threshold is 90% and z is 1.5, the changes in the database disk usage and impact factor can be calculated by Figure 2 The graph shown shows that when the hard disk usage rate x=0, the impact factor y=1; as x increases, y gradually increases, and when x>0.6, y begins to increase sharply; when x=0.8, that is, the hard disk usage rate reaches 80%, the impact factor y is already as high as 25 or more.

[0038] Here, an example is used to illustrate the process of determining the impact factor, which should not be understood as a limitation on the method of the embodiment of the present application. For example: Assume that the usage threshold n is 90%, z is 1.5, the hard disk usage of a database is 70%, and the difference between the preset usage threshold and the hard disk usage of the database is , The usage difference of the database is 90%-70%=20%, and the usage difference is 20%. The ratio of the preset usage threshold and the usage difference is exponentially calculated, that is, (90% / 20%)^1.5=9.545941546, and the impact factor of the database is 9.545941546.

[0039] The embodiment of the present application introduces a preset usage rate threshold, and based on the difference between the preset usage rate threshold and the hard disk usage rate of the database, obtains the usage rate difference to accurately calculate the impact factor of the database, thereby providing a more detailed basis for the performance evaluation of each database. Then, performing an exponential operation on the ratio of the preset usage rate threshold to the usage rate difference can amplify the impact of the usage rate difference, making it more sensitive to changes in hard disk usage, so that the database system can more effectively judge the storage capacity and pressure of the database under dynamically changing load conditions, thereby making more appropriate adjustments in the write strategy, which not only solves the storage pressure problem of the traditional single database storage mode, but also improves the versatility and scalability of the system, ensuring that even under high load conditions, each database can maintain efficient performance and stability, thereby optimizing the overall data storage and management strategy.

[0040] As an optional implementation, the above S143 may include the following steps: obtaining a reference data volume difference of each database based on a difference between the reference data volume of each database and the reference data volume of the second database; Based on the sum of the reference data volume differences of the databases in the database cluster, a total reference data volume difference value is obtained; Based on the ratio between the reference data volume difference of each database and the total reference data volume difference, the difference ratio of each database is obtained; Based on the difference ratio of each database, the databases in the database cluster are sorted to obtain the write order.

[0041] The difference ratio of each database can be expressed as a percentage based on the ratio between the reference data volume difference of each database and the total reference data volume difference, and the final difference ratio of each database can be obtained by rounding. A specific example is used here to illustrate the process of determining the difference ratio of each data, which should not be understood as a limitation on the method of the embodiment of the present application. For example: Assume that the database cluster includes multiple data such as database A, database B, database C, database D, and database E, and the reference data volumes of these multiple databases are 70,000, 99,000, 40,000, 100,000, 10,000 and 0, respectively, among which the reference data volume of database C is the largest, that is, database C is the second database, and the difference between the reference data volume of each database and the reference data volume of the second database, that is, the reference data volume difference of each database is 100,000-70,000=30,000, 100,000-99,000=100 respectively. 0, 100000-40000=60000, 100000-100000=0, 100000-10000=90000, 100000-0=100000, and then the sum of the reference data volume differences is 30000+1000+60000+0+90000+100000=281000, then the difference proportion of database A is 30000 / 281000=0.106761566, and the final difference proportion of database A is 11% by rounding, and the same applies to the other databases.

[0042] As an example, the order of databases in the database cluster can be determined according to the difference ratio of each database, and the write order can be obtained by sorting from small to large or from large to small according to the difference ratio of each database. For example: Assuming that the difference ratios of data in the database cluster are: database A is 21%, database B is 11%, database C is 32%, and database D is 36%, in order from small to large, the write order is database B, database A, database C, and database D. Assuming that the business data to be written in the business system acquired last time is written to database A, the business data currently acquired should be written to database C.

[0043] As an optional method, databases whose difference ratios meet a preset ratio threshold are selected as candidate databases, and the candidate databases are sorted based on their difference ratios to obtain a write order. By sorting databases that have a large difference in reference data volume from the second database, and then determining the write order based on the difference ratios of the databases to be sorted, load balancing of data in the database cluster is achieved.

[0044] The embodiment of the present application optimizes the data writing process by sorting the databases in the database cluster and determining the writing order, thereby effectively improving the efficiency and performance of database management. By calculating the difference between the reference data volume of each database and the reference data volume of the second database, the relative storage burden of each database can be quantified. The difference not only reflects the storage pressure of the database in the cluster, but also reflects the usage of each database. Next, based on the sum of the reference data volume difference and the reference data volume difference of each database, the difference ratio of each database is calculated to ensure that the actual storage status of each database can be taken into account when sorting, so as to determine their priority in the data writing process, which not only effectively solves the competition problems that may occur when writing to a traditional single database, but also ensures that in a cluster environment, databases with lighter loads can receive write requests first, avoiding performance degradation and resource waste caused by improper write order. Through the intelligent sorting mechanism, the database cluster can distribute the load more evenly when efficiently processing write requests, thereby improving the performance and stability of the overall system.

[0045] As another optional implementation, after sorting the databases in the database cluster based on the difference ratio of each database to obtain the write order, the method may further include the following steps: creating a circular linked list; For each database, generating at least one element corresponding to the database based on a difference between a reference data volume of the database and a reference data volume of a second database; Based on the writing order, at least one element corresponding to each database is written into the circular linked list.

[0046] A circular linked list is a data structure consisting of a set of nodes, each of which contains data and a pointer to the next node. The last node of the circular linked list points to the first node of the linked list, thereby forming a closed loop. In an embodiment of the present application, the number of nodes in the circular linked list is the same as the total number of elements generated.

[0047] The embodiment of the present application can generate at least one element corresponding to each database based on the difference between the reference data volume of the database and the reference data volume of the second database, and the database identifier of the database, each element is the content of the database identifier, and the number of elements is the same as the difference. As another optional method, the difference ratio of each database can also be determined based on the difference between the reference data volume of the database and the reference data volume of the second database. Please refer to the previous embodiment for details, and then generate at least one element corresponding to each database according to the difference ratio of each database and the database identifier of each database, and write the generated elements into the nodes of the circular linked list according to the determined writing order. Here, a specific example is used to illustrate the process of creating the above-mentioned circular linked list, which should not be understood as limiting the method of the embodiment of the present application. For example, assuming that the difference percentages of the databases involved in the sorting are: database 0 is 11%, database 2 is 21%, database 4 is 32%, and database 5 is 36%, then at least one element corresponding to database 0 is 11 "0", at least one element corresponding to database 2 is 21 "2", at least one element corresponding to database 4 is 32 "4", and at least one element corresponding to database 5 is 36 "5". The generated elements are written into the circular linked list in the determined writing order of database 0, database 2, database 4, and database 5. For details, please refer to Figure 3 The structural diagram of the circular linked list shown.

[0048] The embodiment of the present application creates a circular linked list after sorting the database cluster and determining the writing order, generates at least one element corresponding to each database based on the reference data volume difference of each database, and stores the elements corresponding to each database through the circular linked list, thereby realizing efficient management of the database writing order, providing a flexible and efficient way to store and manage database elements, so that the writing order of each database can be dynamically adjusted in an unfixed structure, solving the shortcomings of database writing order management, especially in the face of frequent writing and updating operations, the circular linked list can effectively avoid the performance loss caused by the order adjustment. By generating elements based on the reference data volume difference and storing them in the circular linked list, the circular linked list can accurately reflect the relative load of each database during the writing process, thereby realizing more intelligent writing scheduling, which can not only improve the efficiency of data writing, but also reduce the waste of resources caused by uneven load, thereby optimizing the overall performance of the system. Through continuous dynamic adjustment and optimization, the responsiveness of the system in a high concurrency environment is enhanced, ensuring the stability and consistency of the database cluster when processing large-scale data writing, thereby improving the reliability and operation efficiency of the entire system.

[0049] As an optional implementation, the above S104 may include the following steps: writing the first business data to be written in the business system into the business queue, and repeatedly performing the following operations until the writing of the first business data in the business queue is completed: Traversing the first service data in the service queue, and using the currently traversed first service data as the second service data; The elements in the circular linked list are traversed in a loop, and a database corresponding to a currently traversed element is determined as a first database for storing the second business data.

[0050] A business queue refers to a data structure or storage mechanism used to temporarily store first business data. The business data generated by the business system in real time is sequentially placed in the business queue, while the data is traversed from the business queue to obtain the second business data, and the second business data is written into the corresponding database, so as to achieve the decoupling of business data acquisition and writing, which helps to improve the writing efficiency and response speed of the system, avoid the backlog of a large amount of data, and avoid the system performance bottleneck caused by instantaneous high load.

[0051] The above process of determining the first database is illustrated here by an example, which should not be understood as a limitation on the method of the embodiment of the present application. For example: suppose that a plurality of first business data obtained from the business system are sequentially written into the business queue, including: business data 1, business data 2, business data 3 and business data 4; there are 5 elements in the circular linked list, and the order is: C, C, A, B, B. Suppose that the business queue is traversed, and the data currently traversed is business data 1, that is, the second business data. At the same time, the circular linked list is traversed, and the element currently traversed is A, then the database used to store business data 1 is determined to be database A, that is, the first database; then, the next traversed data is business data 2, and the traversed element is B, then the database used to store business data 2 is determined to be database B, then, the next traversed data is business data 3, and the traversed element is B, then the database used to store business data 3 is determined to be database B, then, the next traversed data is business data 4, and the traversed element is C, then the database used to store business data 4 is determined to be database C, and the business queue and the circular linked list are continuously traversed to determine the first database used to store business data.

[0052] The embodiment of the present application writes the first business data to be written into the business queue, and stores the write order of the database in the database cluster through a circular linked list, and determines the database corresponding to each business data by traversing the business queue and the circular linked list, thereby optimizing the data storage process. By introducing the business queue, data processing is made more orderly and efficient. By storing the first business data in the business queue, the system can ensure that data loss or erroneous storage will not occur during the writing process due to the different order of data arrival, thereby improving the integrity and consistency of the data. By combining the traversal mechanism of the circular linked list, the system can dynamically select the most suitable database for storage based on the current load situation, effectively avoiding the performance degradation caused by overload of a database, thereby improving the responsiveness of the overall system. Solve the problems of low writing efficiency and uneven load of traditional databases in high concurrency environments, ensure efficient data writing in high concurrency scenarios, enable database clusters to better cope with changes in business traffic, and improve the scalability and stability of the system.

[0053] S106: Write the first business data into the first database.

[0054] As an optional method, after obtaining the business data, the business data can be directly written into the corresponding database according to the writing order of the databases in the database cluster. As another optional method, after determining the writing order of the databases in the database cluster, a circular linked list can be created based on the difference between the reference data volume of the database and the reference data volume of the second database. After obtaining the business data, the business data obtained in real time is added to the business queue, and by traversing the business queue and the circular linked list, the database corresponding to the currently traversed element is determined to be the database used to store the currently traversed business data, and then the currently traversed business data is written into the database corresponding to the currently traversed element.

[0055] As an optional method, when writing the currently traversed business data into the database corresponding to the currently traversed element, it includes: based on the currently traversed element, generating a primary key ID corresponding to the currently traversed business data, and writing the primary key ID and the currently traversed business data into the database corresponding to the currently traversed element. Among them, the primary key ID consists of three parts: database identifier, timestamp, and random number. The database identifier is the element currently traversed. The timestamp is the time when the primary key ID is generated, that is, the time when the corresponding business data is written to the corresponding database. The random number is generated by a random function. By generating a primary key ID and storing it in the database together with the business data, when querying the data later, it can be determined in which sub-library the data to be queried is located and which data in the sub-library the data to be queried is based on the determined primary key ID. Realize fast data query without performing query operations on all databases.

[0056] The embodiment of the present application shows some implementation methods of the above S106. Of course, it should be understood that the above S106 can also be implemented in other ways, and the embodiment of the present application does not limit this.

[0057] The embodiment of the present application obtains the first business data to be written in the business system, determines the first database for storing the first business data based on the writing order of the databases in the database cluster, and writes the first business data into the first database, wherein the writing order is periodically determined according to the difference in the amount of reference data between each database in the database cluster and the second database, the second database is the database with the largest amount of reference data in the database cluster, and determines the writing order of the databases in the database cluster based on the difference in the amount of reference data between each database in the database cluster and the second database, and then writes the business data of the business system into the corresponding database. This not only effectively solves the storage pressure and performance bottleneck faced by traditional single database storage, but also does not rely on specific business characteristics, thereby adapting to various complex business data, having strong versatility, and avoiding modular operations. When adding a new database, there is no need to migrate the data in the database cluster. This has strong scalability, is easy to maintain and manage, and makes data storage more flexible and convenient.

[0058] The following is an actual application scenario to illustrate the data storage method provided in the embodiment of the present application, which should not be understood as a limitation on the method of the embodiment of the present application. Figure 4 , is a schematic diagram of an application scenario structure of a data storage method provided by an embodiment of the present application, the application scenario includes: application service, sub-library information statistics module, and database cluster. Among them, the database cluster includes multiple databases, and the database identifier, that is, the sub-library number is numbered from 0 to n. The application service is used to process the business logic, obtain the sub-library number from the circular linked list of the sub-library statistics module, generate a primary key ID based on the obtained sub-library number, and store the primary key ID and the corresponding business data together in the database corresponding to the sub-library number. For details, please refer to Figure 5 The flowchart of another database storage method is shown in FIG. The sub-database information statistics module monitors the databases in the database cluster, uses a scheduled task to regularly count the performance data such as the data volume and hard disk usage of each database, and creates or updates a circular linked list based on the acquired performance data. The circular linked list contains the sub-database numbers of the databases in the database cluster.

[0059] Specifically, here we use a specific example to illustrate the creation and update of a circular linked list. For example, suppose there are 6 databases in the database cluster, including sub-database 0, sub-database 1, sub-database 2, sub-database 3, sub-database 4 and sub-database 5. Sub-database 5 is a newly added database with 0 data volume. The data volume of the remaining databases is as follows: Figure 6Assuming that the storage space of each database in the database cluster is the same, the data volume of each database is used as the reference data volume. Figure 6 It can be seen that sub-library 3 has the largest amount of data, and sub-library 3 is the second database. The reference data volume of the second database is denoted as Rmax, and the reference data volume of the remaining databases is denoted as Rn. The difference in reference data volume between the second database and each database in the database cluster is denoted as Dn, that is, Dn=Rmax-Rn, and the total difference Dtotal=D0+D1+D2+D3+D4+D5. Then, the ratio of the reference data volume difference of each database to the total difference Dn / Dtotal is the difference ratio of each database, as shown in Table 1. After rounding, the difference ratio of each database is shown in the last column of Table 1.

[0060] Table 1

[0061] Among them, the proportion of sub-library 1 and sub-library 3 does not meet the threshold and is not included in the sorting. Therefore, only sub-library 0, sub-library 2, sub-library 4 and sub-library 5 are sorted. They are sorted from small to large according to the proportion, and the final write order is determined to be sub-library 0, sub-library 2, sub-library 4, and sub-library 5.

[0062] Create a circular linked list with a length of 100. Based on the sub-library number, generate corresponding elements for sub-library 0, sub-library 2, sub-library 4, and sub-library 5 according to their respective difference ratios. For example, sub-library 0 corresponds to 11 elements, and 11 "0"s are placed in the circular linked list. Figure 3 As shown. When traversing, start from the first element of the circular linked list, and take out the values ​​of the elements in turn according to the direction of the arrows in the figure. The database corresponding to the value is the first database that stores the business data currently traversed. When the last element is taken, taking the next one will return to the first element. When the performance data obtained by statistics is updated, the circular linked list is updated synchronously, and the circular traversal starts from the first element according to the updated circular linked list.

[0063] When the storage space of each database in the database cluster is different, it is necessary to further determine the reference data volume of each database based on the data volume of each database. Assuming that the hard disk usage impact factor is y, the hard disk usage is x, the usage threshold is n=90%, and the index z=1.5, then the impact factor y=(0.9 / (0.9-x))^1.5, the hard disk usage of each database is brought into the calculation to obtain the impact factor corresponding to each database, and then, based on the product of the quantity and the impact factor, the reference data volume of each database is obtained, as shown in Table 2.

[0064] Table 2

[0065] After adding the hard disk usage rate, the second database changes from sub-database 3 to sub-database 2, which has the largest reference data volume. The same method is used to calculate the difference ratio, determine the writing order, and create or update the circular linked list, which will not be repeated here.

[0066] It is worth noting that when adding a new database to the database cluster, it is necessary to add the new sub-database information to the application service and sub-database information statistics module data source configuration. The sub-database information statistics module can only take effect by restarting the system, and then periodically monitor the performance data of each database and update the circular linked list.

[0067] The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0068] With the above Figure 1 Corresponding to the data storage method shown, the present application embodiment also provides a data storage device. Figure 7 , is a structural diagram of a data storage device provided in an embodiment of the present application, the device 700 includes: an acquisition unit 710, a determination unit 720 and a writing unit 730.

[0069] The acquisition unit 710 is used to acquire first business data to be written in the business system.

[0070] The determination unit 720 is used to determine a first database for storing the first business data based on a write order of databases in the database cluster, wherein the write order is periodically determined according to a difference in reference data volume between each database in the database cluster and a second database, and the second database is a database with the largest reference data volume in the database cluster.

[0071] The writing unit 730 is configured to write the first business data into the first database.

[0072] Optionally, the device 700 further includes: a monitoring unit; The monitoring unit is used to perform the following steps to determine the write order, including: periodically monitoring the performance data of each database in the database cluster, the performance data including the data volume; Based on the data volume of each database, determine the reference data volume of each database; Based on the difference between the reference data volume of each database and the reference data volume of the second database, the databases in the database cluster are sorted to obtain the write order.

[0073] Optionally, when the monitoring unit sorts the databases in the database cluster based on the difference between the reference data volume of each database and the reference data volume of the second database to obtain the write order, the monitoring unit may perform the following steps: obtaining a reference data volume difference of each database based on the difference between the reference data volume of each database and the reference data volume of the second database; Based on the sum of the reference data volume differences of the databases in the database cluster, a reference data volume difference sum is obtained; Based on the ratio between the reference data volume difference of each database and the sum of the reference data volume differences, obtaining the difference ratio of each database; Based on the difference ratio of each database, the databases in the database cluster are sorted to obtain the write order.

[0074] Optionally, in the case where the performance data further includes a hard disk usage rate, the monitoring unit may perform the following steps when determining the reference data volume of each database based on the data volume of each database: determining an impact factor of each database based on the hard disk usage rate of each database, the impact factor being used to describe the degree of impact of the hard disk usage rate of the database on the data volume of the database; For each database, a reference data volume of the database is determined based on the product of the data volume of the database and the impact factor of the database.

[0075] Optionally, when determining the impact factor of each database based on the hard disk usage of each database, the monitoring unit may perform the following steps: obtaining a usage difference of the database based on a difference between a preset usage threshold and the hard disk usage of the database; An exponential operation is performed on the ratio of the preset usage rate threshold and the usage rate difference to obtain an impact factor of the database.

[0076] Optionally, the apparatus 700 further includes: a creating unit; The creating unit is configured to, after the monitoring unit sorts the databases in the database cluster based on the difference between the reference data amount of each database and the reference data amount of the second database to obtain the write order, perform the following steps: create a circular linked list; For each database, generating at least one element corresponding to the database based on a difference between a reference data volume of the database and a reference data volume of the second database; Based on the writing order, at least one element corresponding to each database is written into the circular linked list.

[0077] Optionally, when determining the first database for storing the first business data based on the writing order of the databases in the database cluster, the determining unit 730 may perform the following steps: writing the first business data to be written in the business system into a business queue, and repeatedly performing the following operations until the writing of the first business data in the business queue is completed: Traversing the first service data in the service queue, and using the currently traversed first service data as the second service data; The elements in the circular linked list are traversed in a loop, and a database corresponding to a currently traversed element is determined as a first database for storing the second business data.

[0078] Obviously, the data storage device provided in the embodiment of the present application can be used as Figure 1 The execution subject of the data storage method shown, for example Figure 1 In the data storage method shown in FIG. 1 , step S102 may be performed by Figure 7 The acquisition unit 710 in the data storage device shown in FIG. 1 is executed, and step S104 can be performed by Figure 7 The determination unit 720 in the data storage device shown in FIG. 1 is executed, and step S106 can be performed by Figure 7 The writing unit 730 in the data storage device shown is executed.

[0079] According to another embodiment of the present application, Figure 7 The various units in the data storage device shown can be separately or completely combined into one or several other units to constitute, or one (some) of the units can be further divided into multiple functionally smaller units to constitute, which can achieve the same operation without affecting the realization of the technical effects of the embodiments of the present application. The above-mentioned units are divided based on logical functions. In practical applications, the function of one unit can also be implemented by multiple units, or the functions of multiple units can be implemented by one unit. In other embodiments of the present application, the data storage device may also include other units. In practical applications, these functions can also be implemented with the assistance of other units, and can be implemented by the collaboration of multiple units.

[0080] According to another embodiment of the present application, the program can be executed on a general computing device such as a computer including a central processing unit (CPU), a random access memory (RAM), a read-only memory (ROM), and other processing elements and storage elements. Figure 1 A computer program (including program code) for each step involved in the corresponding method shown in FIG. Figure 7 The data storage device shown in and the data storage method of the embodiment of the present application are implemented. The computer program can be recorded on, for example, a computer-readable storage medium, and transferred to an electronic device through the computer-readable storage medium and run therein.

[0081] Figure 8 This is a schematic diagram of the structure of an electronic device according to an embodiment of the present application. Figure 8 At the hardware level, the electronic device includes a processor, and optionally also includes an internal bus, a network interface, and a memory. The memory may include a memory, such as a high-speed random access memory (RAM), and may also include a non-volatile memory (non-volatile memory), such as at least one disk storage. Of course, the electronic device may also include hardware required for other services.

[0082] The processor, network interface and memory can be interconnected through an internal bus, which can be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 8 Only one bidirectional arrow is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0083] The memory is used to store the program. Specifically, the program may include a program code, and the program code includes a computer operation instruction. The memory may include a memory and a non-volatile memory, and provides instructions and data to the processor.

[0084] The processor reads the corresponding computer program from the non-volatile memory into the memory and then runs it, forming a data processing device at the logical level. The processor executes the program stored in the memory and is specifically used to perform the following operations: Acquire first business data to be written in the business system; Determine a first database for storing the first business data based on a write order of databases in the database cluster, wherein the write order is periodically determined according to a difference in reference data volume between each database in the database cluster and a second database, and the second database is a database with the largest reference data volume in the database cluster; The first business data is written into the first database.

[0085] The above application Figure 7 The method performed by the data storage device disclosed in the illustrated embodiment can be applied to a processor or implemented by a processor. The processor may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by an integrated logic circuit of hardware in the processor or an instruction in the form of software. The above processor may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it may also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present application can be implemented or executed. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc. The steps of the method disclosed in the embodiments of the present application can be directly embodied as being executed by a hardware decoding processor, or executed by a combination of hardware and software modules in a decoding processor. The software module can be located in a storage medium mature in the art such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory, and the processor reads the information in the memory and completes the steps of the above method in combination with its hardware.

[0086] The electronic device may also perform Figure 1 method, and realizing the data storage device in Figure 1 , Figure 7 The functions of the illustrated embodiment will not be described in detail in the embodiments of the present application.

[0087] Of course, in addition to software implementation methods, the electronic device of the present application does not exclude other implementation methods, such as logic devices or a combination of software and hardware, etc. That is to say, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0088] The embodiment of the present application also provides a computer-readable storage medium, which stores one or more programs, wherein the one or more programs include instructions, which, when executed by a portable electronic device including a plurality of application programs, enable the portable electronic device to execute Figure 1 The method of the embodiment shown is specifically used to perform the following operations: Acquire first business data to be written in the business system; Determine a first database for storing the first business data based on a write order of databases in the database cluster, wherein the write order is periodically determined according to a difference in reference data volume between each database in the database cluster and a second database, and the second database is a database with the largest reference data volume in the database cluster; The first business data is written into the first database.

[0089] In short, the above description is only a preferred embodiment of the present application and is not intended to limit the protection scope of the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

[0090] The systems, devices, modules or units described in the above embodiments may be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, the computer may be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0091] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0092] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0093] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

Claims

1. A data storage method, characterized in that: include: Acquire first business data to be written in the business system; Determine a first database for storing the first business data based on a write order of databases in the database cluster, wherein the write order is periodically determined according to a difference in reference data volume between each database in the database cluster and a second database, and the second database is a database with the largest reference data volume in the database cluster; The first business data is written into the first database.

2. The method according to claim 1, characterized in that The writing order is determined in the following manner: Periodically monitoring performance data of each database in the database cluster, wherein the performance data includes data volume; Based on the data volume of each database, determine the reference data volume of each database; Based on the difference between the reference data volume of each database and the reference data volume of the second database, the databases in the database cluster are sorted to obtain the write order.

3. The method according to claim 2, characterized in that The step of sorting the databases in the database cluster based on the difference between the reference data volume of each database and the reference data volume of the second database to obtain the write order includes: Obtaining a reference data volume difference of each database based on a difference between the reference data volume of each database and the reference data volume of the second database; Based on the sum of the reference data volume differences of the databases in the database cluster, a reference data volume difference sum is obtained; Based on the ratio between the reference data volume difference of each database and the sum of the reference data volume differences, obtaining the difference ratio of each database; Based on the difference ratio of each database, the databases in the database cluster are sorted to obtain the write order.

4. The method according to claim 2, characterized in that: The performance data also includes hard disk usage; The step of determining the reference data volume of each database based on the data volume of each database includes: Based on the hard disk usage rate of each database, determining the impact factor of each database, the impact factor is used to describe the impact degree of the hard disk usage rate of the database on the data volume of the database; For each database, a reference data volume of the database is determined based on the product of the data volume of the database and the impact factor of the database.

5. The method according to claim 4, characterized in that For each database, the impact factor of the database is determined in the following manner: Based on the difference between the preset usage rate threshold and the hard disk usage rate of the database, obtaining the usage rate difference of the database; An exponential operation is performed on the ratio of the preset usage rate threshold and the usage rate difference to obtain an impact factor of the database.

6. The method according to claim 2, characterized in that After sorting the databases in the database cluster based on the difference between the reference data volume of each database and the reference data volume of the second database to obtain the write order, the method further includes: Create a circular linked list; For each database, generating at least one element corresponding to the database based on a difference between a reference data volume of the database and a reference data volume of the second database; Based on the writing order, at least one element corresponding to each database is written into the circular linked list.

7. The method according to claim 6, characterized in that The determining, based on the writing order of databases in the database cluster, a first database for storing the first business data includes: Writing the first business data to be written in the business system into the business queue, and repeatedly performing the following operations until the writing of the first business data in the business queue is completed: Traversing the first service data in the service queue, and using the currently traversed first service data as the second service data; The elements in the circular linked list are traversed in a loop, and a database corresponding to a currently traversed element is determined as a first database for storing the second business data.

8. A data storage device, characterized in that: include: An acquisition unit, used for acquiring first business data to be written in the business system; a determining unit, configured to determine a first database for storing the first business data based on a writing order of databases in a database cluster, wherein the writing order is periodically determined according to a difference in reference data volume between each database in the database cluster and a second database, and the second database is a database with the largest reference data volume in the database cluster; A writing unit is used to write the first business data into the first database.

9. An electronic device, characterized in that: include: processor; a memory for storing instructions executable by the processor; The processor is configured to execute the instructions to implement the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: When the instructions in the storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the method as claimed in any one of claims 1 to 7.

Images