Data storage method and device, storage medium and electronic equipment

Abstract

The invention discloses a data storage method and device, a storage medium and electronic equipment. The data storage method comprises the steps that when it is monitored that a preset adjustment condition is met, original fragments needing to be adjusted are determined, and data access logs corresponding to all service data stored in the original fragments are acquired; according to the data access log, determining access hotspot data in the business data; according to the service association information corresponding to each piece of access hotspot data, adjusting a fragmentation storage strategy of each piece of service data in a database to obtain an adjusted fragmentation storage strategy; and according to the adjusted fragment storage strategy, determining a target fragment required for storing each piece of business data, so as to migrate at least part of business data in each piece of business data from the original fragment to the target fragment for storage. According to the scheme, the access delay of the data is effectively reduced, and the access efficiency and the resource utilization rate of the database are further improved.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a data storage method, device, storage medium and electronic device. Background Art

[0002] With the rapid development of information technology, the amount of business data has also increased dramatically. The traditional single-node database system can no longer meet the growing business needs. The emergence of data sharding technology has solved the performance bottleneck of database systems when processing large-scale data.

[0003] Data sharding, as a key data processing technology, is used to divide large databases into smaller and more manageable parts. These "parts" are called shards. Each shard is a subset of the database and is usually deployed on different nodes, which effectively reduces the burden on a single storage node and improves the overall performance and stability of the system.

[0004] However, existing data storage methods often adopt a fixed shard storage strategy, that is, certain data is fixedly stored in one shard. Since the access frequency of these data is too high, the access delay corresponding to the shard will also increase, thereby affecting data access efficiency.

[0005] Therefore, how to reduce the access delay in the process of accessing business data is an urgent problem to be solved. Summary of the invention

[0006] This specification provides a data storage method, device, storage medium and electronic device to partially solve the above problems existing in the prior art.

[0007] This manual adopts the following technical solutions:

[0008] This specification provides a data storage method, including:

[0009] When it is monitored that the preset adjustment conditions are met, the original shards that need to be adjusted are determined, and the data access logs corresponding to the business data stored in the original shards are obtained, wherein the data access logs are used to reflect the access popularity of the business data;

[0010] According to the data access log, determining each access hotspot data in the each business data;

[0011] According to the business association information corresponding to each access hotspot data, the shard storage strategy of each business data in the database is adjusted to obtain an adjusted shard storage strategy;

[0012] According to the adjusted shard storage strategy, a target shard required for storing each business data is determined, so as to migrate at least part of the business data from the original shard to the target shard for storage.

[0013] Optionally, the step of monitoring that a preset adjustment condition is met specifically includes:

[0014] When the preset adjustment time is reached, it is determined that the adjustment condition is met.

[0015] Optionally, the step of monitoring that a preset adjustment condition is met specifically includes:

[0016] When it is monitored that the access delay of the database when being accessed by the user exceeds a preset threshold, it is determined that the adjustment condition is met.

[0017] Optionally, the step of adjusting the shard storage strategy of each business data in the database according to the business association information corresponding to each access hotspot data specifically includes:

[0018] With the optimization goals of minimizing the access delay when the database is accessed by users and maximizing the resource utilization corresponding to the database, the shard storage strategy of each business data in the database is adjusted according to the business association information corresponding to each access hotspot data.

[0019] Optionally, the step of determining, according to the adjusted shard storage strategy, a target shard required for storing each business data, so as to migrate at least part of the business data from the original shard to the target shard for storage, specifically includes:

[0020] Determine a data migration batch corresponding to each business data stored in the original shard;

[0021] According to the data migration batch, at least part of the business data in each business data is migrated from the original shard to the target shard for storage in sequence.

[0022] Optionally, the step of determining, according to the adjusted shard storage strategy, a target shard required for storing each business data, so as to migrate at least part of the business data from the original shard to the target shard for storage, specifically includes:

[0023] Copying at least part of the business data to obtain mirror data;

[0024] Migrating each of the mirrored data to the target shard;

[0025] When it is determined that the migration of each mirror data is completed, at least part of the business data stored in the original slice is deleted.

[0026] Optionally, before the step of deleting at least part of the business data stored in the original slice, the method further includes:

[0027] For each business data, if the business data is split into sub-data and stored in multiple target shards, the sub-data are obtained from the multiple target shards, and the sub-data are reassembled to obtain reassembled data;

[0028] Performing a consistency check on the reorganized data and the business data;

[0029] The step of deleting at least part of the business data stored in the original slice specifically includes:

[0030] If it is determined that the split business data passes the consistency check, at least part of the business data stored in the original shard is deleted.

[0031] This specification provides a data storage device, including:

[0032] A monitoring module, used to determine the original shards that need to be adjusted when it is monitored that the preset adjustment conditions are met, and to obtain the data access logs corresponding to each business data stored in the original shards, wherein the data access logs are used to reflect the access popularity of each business data;

[0033] A determination module, used to determine each access hotspot data in each business data according to the data access log;

[0034] An adjustment module, used to adjust the shard storage strategy of each business data in the database according to the business association information corresponding to each access hotspot data, to obtain an adjusted shard storage strategy;

[0035] A migration module is used to determine the target shard required to store the business data according to the adjusted shard storage strategy, so as to migrate at least part of the business data from the original shard to the target shard for storage.

[0036] This specification provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the above-mentioned data storage method is implemented.

[0037] This specification provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above-mentioned data storage method when executing the program.

[0038] At least one of the above technical solutions adopted in this specification can achieve the following beneficial effects:

[0039] In the data storage method provided in this specification, when it is monitored that the preset adjustment conditions are met, the original shards that need to be adjusted are determined, and the data access logs corresponding to the business data stored in the original shards are obtained; according to the data access logs, the access hotspot data are determined in the business data; according to the business association information corresponding to each access hotspot data, the shard storage strategy of each business data in the database is adjusted to obtain the adjusted shard storage strategy; according to the adjusted shard storage strategy, the target shard required to store each business data is determined, so as to migrate at least part of the business data in each business data from the original shard to the target shard for storage. This solution effectively reduces the access delay of data and further improves the access efficiency and resource utilization of the database.

[0040] It can be seen from the above method that this solution can determine the access hotspot data based on the data access log. In this way, the shard storage strategy of the business data can be dynamically adjusted according to the business-related information corresponding to the access hotspot data. Compared with the current method of storing data in fixed shards, this solution can avoid the problem of increased access delay caused by excessive access frequency of a certain shard by dynamically adjusting the shard storage strategy, effectively reducing data access delay and further improving data access efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] The drawings described herein are used to provide a further understanding of this specification and constitute a part of this specification. The illustrative embodiments and descriptions of this specification are used to explain this specification and do not constitute an improper limitation on this specification. In the drawings:

[0042] Figure 1 A schematic diagram of a data storage method provided in this specification;

[0043] Figure 2 A schematic diagram of a data storage device provided in this specification;

[0044] Figure 3 A method corresponding to the Figure 1 Schematic diagram of electronic equipment. DETAILED DESCRIPTION

[0045] In order to make the purpose, technical solutions and advantages of this specification more clear, the technical solutions of this specification will be clearly and completely described below in combination with the specific embodiments of this specification and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of this specification, not all of them. Based on the embodiments in this specification, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this specification.

[0046] The technical solutions provided by the embodiments of this specification are described in detail below in conjunction with the accompanying drawings.

[0047] Figure 1 This is a flow chart of a data storage method provided in this specification, comprising the following steps:

[0048] S101: When it is monitored that a preset adjustment condition is met, the original shard that needs to be adjusted is determined, and the data access log corresponding to each business data stored in the original shard is obtained, wherein the data access log is used to reflect the access popularity of each business data.

[0049] In current database storage systems, efficient management and access to data are crucial. Traditional database sharding technologies mainly rely on predefined sharding strategies or simple hash sharding, which have significant limitations when processing large-scale, diverse business data. Although these technologies have improved data access efficiency to a certain extent, they still face problems such as uneven data distribution, access hot spots, and downtime maintenance when adjusting sharding strategies.

[0050] Based on this, this specification provides a data storage method that generates an optimal sharding strategy based on the analysis results of data access logs, and dynamically adjusts sharding and reorganizes data without affecting system operation.

[0051] In this specification, the execution subject for implementing a data storage method may be a designated device such as a server. For ease of description, the following will only take the server as an execution subject as an example to illustrate a data storage method provided in this specification.

[0052] During the process of users executing business, the server can generate a data access log corresponding to the accessed business data stored in each shard of the database in response to the operations performed by the users during the execution of the business. The data access log can be used to reflect the access popularity of the business data (such as the access frequency corresponding to the business data within a specified range).

[0053] In addition, the data access log can also record the behavioral data generated when users access business data (such as login, logout, verification, transactions, and interaction with virtual scenes, etc.) and the storage information corresponding to the business data (such as the storage address of the shard, the storage space occupied, etc.).

[0054] After the data storage date is generated, the server can store it in the cloud. In actual applications, there are many ways to store the above data access logs. Among them, the server can use the Logstash tool to collect the access logs of the game server, store them in Elasticsearch, and then use Kibana for real-time monitoring and analysis.

[0055] For example, in a multiplayer online role-playing game, data access logs containing player login, logout, transaction, and combat behavior data are collected and transmitted through Logstash and stored in Elasticsearch for subsequent analysis.

[0056] In addition, the server can use the Fluentd tool as a log collection agent to transfer logs to the Kafka cluster for storage and processing.

[0057] For example, a game server can send players' data access logs to Fluentd, and Fluentd sends the logs to a Kafka cluster for distributed storage, ensuring high throughput and low latency.

[0058] In addition, the server can also use the log collection and management services of the cloud service provider to automatically collect and store data access logs.

[0059] For example, in the AWS environment, game servers send data access logs to CloudWatch Logs, leveraging its automated and scalable features for log management.

[0060] When it is detected that the preset adjustment conditions are met, the server can determine the original shards that need to be adjusted, and obtain the data access logs corresponding to each business data stored in the original shards.

[0061] In this specification, the server can adjust the shard storage strategy of the business data based on the access log at a preset time interval, and when the preset adjustment time is reached, it is determined that the adjustment condition for adjusting the shard storage strategy is met. Among them, the above-mentioned preset adjustment time can be set according to actual conditions, and this specification does not make specific restrictions on this.

[0062] In addition, the server can also monitor the access status of the database, and when it is monitored that the access delay of the database when being accessed by the user exceeds the preset threshold, it is determined that the adjustment condition for adjusting the shard storage strategy is met. Among them, the above preset threshold can be set according to actual conditions, and this specification does not make specific restrictions on this.

[0063] Of course, the server may also determine whether the above-mentioned specified conditions are met based on other performance indicators such as data throughput and resource utilization corresponding to each data slice, and this specification does not make specific limitations on this.

[0064] It should be noted that when the above adjustment conditions are met, the server can adjust the shard storage strategy for the business data in all original shards, and at this time the server can obtain the data access logs corresponding to the business data in all data shards. Of course, the server can also adjust the shard storage strategy for the business data in only some of the original shards (such as shards with high access latency or low remaining storage space), and at this time the server only needs to obtain the data access logs corresponding to the shards to be adjusted.

[0065] S102: Determine access hotspot data from the business data according to the data access log.

[0066] The server may determine each access hotspot data in the business data according to the data access log, and determine the business association information corresponding to each access hotspot data.

[0067] Among them, the above-mentioned access hotspot data can be business data that has been accessed by users more frequently in a recent period of time. In actual applications, the server can pre-set a time range. For each business data, when the frequency of access to the business data in the recent time range is higher than a preset frequency threshold, the business data can be used as access hotspot data. Among them, the above-mentioned time range and frequency threshold can be set according to actual conditions, and this manual does not make specific limitations on this.

[0068] Furthermore, the server may further determine the business association information corresponding to each access hotspot data. The business association information may be determined by the server according to the data access log, or may be obtained by the server by parsing the access hotspot data.

[0069] For each access hotspot data, the business association information corresponding to the access hotspot data may include the business type involved in the access hotspot data (such as login, logout, registration, transaction, interaction, etc.), user information (such as the user's geographic location, the server where the user is located, etc.) and time information (such as the timestamp corresponding to when each business data is accessed), etc.

[0070] Specifically, the server may pre-generate an information description statement, and then input the information description statement and the data access log into a preset big model, so as to determine each access hotspot data in the business data through the big model, and determine the business association information corresponding to each access hotspot data.

[0071] Among them, the above-mentioned information description statement is used to specify the output content and format of the big model. For example, the information description statement input by the server in the big model can be: "Determine the hot spot data with higher access frequency in the data access log, determine the business-related information involved in each hot spot data, and display the above content in XX format, wherein the business-related information needs to include one or more of the business type, user information and time information."

[0072] In actual applications, the server can use the TensorFlow framework to build a deep learning model, use scikit-learn for feature extraction and model training, and use the trained neural network model to determine the access hotspot data and the business-related information corresponding to the access hotspot data.

[0073] In addition, the server can use models such as Apache Spark to perform distributed processing and analysis on large-scale data access logs and use Hadoop for data storage and management.

[0074] In addition, the server can also use the Neo4j graph database to store and analyze data access logs, and perform complex graph calculations and pattern recognition through GraphX ​​to obtain access hotspot data and its corresponding business-related information.

[0075] S103: According to the business association information corresponding to each access hotspot data, the shard storage strategy of each business data in the database is adjusted to obtain an adjusted shard storage strategy.

[0076] The server can adjust the shard storage strategy of the business data in the database according to the business association information corresponding to each access hotspot data and the storage information corresponding to each hotspot data, with the goal of minimizing the access delay when the database is accessed by the user and maximizing the resource utilization corresponding to the database, thereby determining a shard storage strategy with the minimum access delay and the maximum resource utilization as the adjusted shard strategy.

[0077] In this specification, the above-mentioned shard storage strategy may include a shard strategy for the database and a storage strategy for business data. The shard strategy for the database is used to determine the sharding rules of the database memory, through which the number of shards the database is divided into and the storage space and address corresponding to each shard can be determined; the storage strategy for business data is used to determine the corresponding relationship between each business data and each shard in the database, through which the storage strategy can determine in which target shard of the database each business data will be stored.

[0078] In practical applications, there are many ways to determine the adjusted shard storage strategy. Among them, the server can use reinforcement learning algorithms, use tools such as OpenAI Gym to build a simulation environment, use TensorFlow to implement the reinforcement learning algorithm, and simulate the effects of different shard storage strategies in the simulation environment to generate the optimal shard storage strategy.

[0079] In addition, the server can use genetic algorithms to simulate the evolutionary process based on the DEAP (Distributed Evolutionary Algorithms in Python) framework to generate multiple sharding strategies, evaluate the effectiveness of the strategies through fitness functions, implement genetic algorithm selection, crossover and mutation operations, and thus explore the optimal sharding storage strategy.

[0080] In addition, the server can also tune multiple parameters of the sharding strategy through a hybrid optimization algorithm combined with Bayesian optimization and random search, and perform hyperparameter optimization of the sharding strategy in the Hyperopt or Optuna framework.

[0081] S104: Determine a target shard required for storing each business data according to the adjusted shard storage strategy, so as to migrate at least part of the business data from the original shard to the target shard for storage.

[0082] After determining the adjusted shard storage strategy, the server may determine a target shard required to store each business data, so as to migrate at least part of each business data from the original shard to the target shard for storage.

[0083] It should be noted that under the adjusted shard storage strategy, the business data in all original shards may not be migrated, and some business data will still be stored in the previous original shards under the adjusted shard storage strategy.

[0084] Furthermore, in the process of migrating business data in the original shard, in order to reduce the impact on actual business, the server can determine the data migration batch corresponding to each business data stored in the original shard, and then migrate each business data from the delayed shard to the target shard for storage in turn according to the data migration batch corresponding to each business data.

[0085] In addition, in the process of migrating business data, in order not to affect the normal operation of the business, the server can migrate the business data through a double-write mechanism. For each business data that needs to be migrated, the server can copy the business data in the original shard where the business data is located to obtain the mirror data corresponding to the business data, and then migrate the mirror data to the target shard allocated to the business data under the adjusted shard storage strategy. When it is determined that the migration of each mirror data is completed, the server deletes the uncopied business data stored in the original shard.

[0086] In this specification, there may be multiple data migration methods corresponding to the adjusted shard storage strategy, wherein the server may migrate business data in an original shard to a target shard based on the adjusted shard storage strategy;

[0087] The server can also merge the business data in multiple original shards, and then store the merged business data in a target shard;

[0088] In addition, the server can also split the business data in a certain original shard into multiple sub-data and store these sub-data in multiple target shards.

[0089] Of course, the server can also migrate business data from multiple original shards to multiple target shards.

[0090] Specifically, for each business data, if the business data is split into sub-data and stored in multiple target shards, the server can obtain each sub-data from the multiple target shards, and reorganize the sub-data to obtain reorganized data, and then perform a consistency check on the reorganized data and the business data (that is, verify whether the hash value of the reorganized data is consistent with the hash value of the business data).

[0091] If it is determined that the split business data passes the consistency check, at least part of the business data stored in the original shard is deleted. Otherwise, the reorganized data is deleted and the business data in the original shard is migrated again.

[0092] In actual applications, the server can use distributed database technology to implement dynamic sharding in the Cassandra database and reorganize data in the background to ensure high availability of the system.

[0093] In addition, the server can migrate business data from relational databases to distributed databases through data migration tools such as Apache Sqoop, and reorganize the data according to the new sharding strategy to ensure that the system runs seamlessly during the migration process.

[0094] In addition, the server can use non-disruptive data reorganization tools such as Percona Toolkit to perform online table reorganization and sharding adjustments in the MySQL database to ensure that user access is not affected during the reorganization process.

[0095] Furthermore, during the actual execution of business, the server can monitor the operation of the database in real time, and dynamically adjust the shard storage strategy based on the monitoring data.

[0096] In this manual, there are many ways to monitor the operation of the database. Among them, the server can use the Prometheus tool to collect database system performance indicators and use Grafana for real-time visualization and monitoring.

[0097] For example, you can deploy Prometheus in the game server to collect performance indicators such as CPU, memory, and disk IO in real time, display the system status through Grafana, and adjust the sharding strategy in real time based on the monitoring data.

[0098] In addition, the server can use Kubernetes and Ansible to achieve automated operation and maintenance and dynamic adjustment to ensure efficient operation of the system.

[0099] For example, you can run a game server in a Kubernetes environment, use Ansible scripts to automatically adjust the sharding strategy, and perform dynamic optimization based on real-time monitoring data.

[0100] In addition, the server can use the stream processing framework to implement an adaptive control system to dynamically adjust the sharding strategy based on the real-time data stream.

[0101] For example, the server can process real-time data streams through Apache Flink, analyze player behavior and system performance, and adjust sharding strategies in real time to ensure that the system can still run efficiently under high load.

[0102] From the above method, it can be seen that this solution can improve the database storage performance. Through intelligent sharding and dynamic adjustment, it avoids the overload of certain shards, balances the system load, and improves the overall data access efficiency. The optimized sharding strategy reduces the average delay of data access and improves the user experience.

[0103] The overall solution has high availability, and data reorganization and sharding adjustments are performed without affecting system operation, reducing downtime and ensuring high system availability. Real-time monitoring and dynamic adjustment mechanisms enable the system to quickly respond to changes in access patterns and maintain efficient operation.

[0104] The database storage resources are optimized, resource waste is avoided through intelligent sharding, and the resource utilization of the server is improved. The optimized system performance and resource utilization reduce hardware and maintenance costs.

[0105] Since the adjusted shard storage strategy is determined based on the business association information, in this case, the server can ensure that the access delay of each shard is minimized and the resource utilization is maximized. In addition, since the business association information contains the business type, user information and time information, the server can store the business information with related business type, user information (such as the user's zone and service) and time information in the same or similar storage address shards, thereby shortening the distance between business data with business associations, further improving data access efficiency and reducing data access delay.

[0106] The above are one or more data storage methods implemented in this specification. Based on the same idea, this specification also provides corresponding data storage devices, such as Figure 2 shown.

[0107] Figure 2 A schematic diagram of a data storage device provided in this specification includes:

[0108] The monitoring module 201 is used to determine the original shards that need to be adjusted when the preset adjustment conditions are met, and obtain the data access logs corresponding to the business data stored in the original shards, wherein the data access logs are used to reflect the access popularity of the business data;

[0109] A determination module 202, configured to determine each access hotspot data in each business data according to the data access log;

[0110] The adjustment module 203 is used to adjust the shard storage strategy of each business data in the database according to the business association information corresponding to each access hotspot data, so as to obtain the adjusted shard storage strategy;

[0111] The migration module 204 is used to determine the target shard required to store the business data according to the adjusted shard storage strategy, so as to migrate at least part of the business data from the original shard to the target shard for storage.

[0112] Optionally, the monitoring module 201 is specifically configured to determine that the adjustment condition is satisfied when a preset adjustment time is reached.

[0113] Optionally, the monitoring module 201 is specifically configured to determine that the adjustment condition is satisfied when it is monitored that an access delay when the database is accessed by a user exceeds a preset threshold.

[0114] Optionally, the adjustment module 203 is specifically used to adjust the shard storage strategy of each business data in the database according to the business association information corresponding to each access hotspot data, with the goal of minimizing the access delay when the database is accessed by a user and maximizing the resource utilization corresponding to the database.

[0115] Optionally, the migration module 204 is specifically used to determine a data migration batch corresponding to each business data stored in the original shard; according to the data migration batch, at least part of the business data in each business data is migrated from the original shard to the target shard for storage in sequence.

[0116] Optionally, the migration module 204 is specifically used to copy at least part of the business data to obtain mirror data; migrate the mirror data to the target shard; and when it is determined that the migration of the mirror data is completed, delete at least part of the business data stored in the original shard.

[0117] Optionally, before deleting at least part of the business data stored in the original shard, the migration module 204 is further configured to, for each business data, if the business data is split into sub-data and stored in multiple target shards, obtain the sub-data from the multiple target shards, reorganize the sub-data to obtain reorganized data; and perform consistency check between the reorganized data and the business data;

[0118] The migration module 204 is specifically configured to delete at least part of the business data stored in the original shard if it is determined that the split business data passes the consistency check.

[0119] This specification also provides a computer-readable storage medium, which stores a computer program, which can be used to execute the above Figure 1 A data storage method is provided.

[0120] This manual also provides Figure 3 The one shown corresponds to Figure 1 A schematic diagram of the electronic device. Figure 3As mentioned above, at the hardware level, the electronic device includes a processor, an internal bus, a network interface, a memory, and a non-volatile memory, and may also include other hardware required for the business. The processor reads the corresponding computer program from the non-volatile memory into the memory and then runs it to achieve the above Figure 1 Of course, in addition to the software implementation, this specification does not exclude other implementations, such as logic devices or a combination of software and hardware, etc., that is, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0121] For the improvement of a technology, it can be clearly distinguished whether it is a hardware improvement (for example, improvement of the circuit structure of diodes, transistors, switches, etc.) or a software improvement (improvement of the method flow). However, with the development of technology, many improvements of the method flow today can be regarded as direct improvements of the hardware circuit structure. Designers almost always obtain the corresponding hardware circuit structure by programming the improved method flow into the hardware circuit. Therefore, it cannot be said that the improvement of a method flow cannot be implemented with a hardware entity module. For example, a programmable logic device (PLD) (such as a field programmable gate array (FPGA)) is such an integrated circuit whose logical function is determined by the user's programming of the device. Designers can "integrate" a digital system on a PLD by programming themselves, without having to ask chip manufacturers to design and make dedicated integrated circuit chips. Moreover, nowadays, instead of manually making integrated circuit chips, this kind of programming is mostly implemented by "logic compiler" software, which is similar to the software compiler used when developing and writing programs, and the original code before compilation must also be written in a specific programming language, which is called hardware description language (HDL). There is not only one HDL, but many kinds, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, RHDL (Ruby Hardware Description Language), etc. The most commonly used ones are VHDL (Very-High-Speed ​​Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art should also know that it is only necessary to program the method flow slightly in the above-mentioned hardware description languages ​​and program it into the integrated circuit, and then it is easy to obtain the hardware circuit that implements the logic method flow.

[0122] The controller can be implemented in any appropriate manner, for example, the controller can take the form of a microprocessor or processor and a computer-readable medium storing a computer-readable program code (such as software or firmware) that can be executed by the (micro)processor, a logic gate, a switch, an application-specific integrated circuit (ASIC), a programmable logic controller, and an embedded microcontroller. Examples of controllers include, but are not limited to, the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicone Labs C8051F320. The memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art also know that in addition to implementing the controller in a purely computer-readable program code manner, the controller can be implemented in the form of a logic gate, a switch, an application-specific integrated circuit, a programmable logic controller, and an embedded microcontroller by logically programming the method steps. Therefore, this controller can be considered as a hardware component, and the devices included therein for implementing various functions can also be regarded as structures within the hardware component. Or even, the devices for implementing various functions can be regarded as both software modules for implementing the method and structures within the hardware component.

[0123] The systems, devices, modules or units described in the above embodiments may be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, the computer may be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0124] For the convenience of description, the above device is described in various units according to their functions. Of course, when implementing this specification, the functions of each unit can be implemented in the same or multiple software and / or hardware.

[0125] Those skilled in the art will appreciate that the embodiments of this specification may be provided as methods, systems, or computer program products. Therefore, this specification may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, this specification may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0126] This specification is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of this specification. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0127] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0128] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable device. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0129] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0130] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0131] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0132] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0133] Those skilled in the art will appreciate that the embodiments of this specification may be provided as methods, systems or computer program products. Therefore, this specification may take the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware. Moreover, this specification may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0134] This specification may be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. This specification may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules may be located in local and remote computer storage media, including storage devices.

[0135] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0136] The above description is only an embodiment of the present specification and is not intended to limit the present specification. For those skilled in the art, the present specification may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present specification shall be included in the scope of the claims of the present specification.

Claims

1. A data storage method, characterized in that: include: When it is monitored that the preset adjustment conditions are met, the original shards that need to be adjusted are determined, and the data access logs corresponding to the business data stored in the original shards are obtained, wherein the data access logs are used to reflect the access popularity of the business data; According to the data access log, determining each access hotspot data in the each business data; According to the business association information corresponding to each access hotspot data, the shard storage strategy of each business data in the database is adjusted to obtain an adjusted shard storage strategy; According to the adjusted shard storage strategy, a target shard required for storing each business data is determined, so as to migrate at least part of the business data from the original shard to the target shard for storage.

2. The method according to claim 1, characterized in that The step of monitoring that a preset adjustment condition is met specifically includes: When the preset adjustment time is reached, it is determined that the adjustment condition is met.

3. The method according to claim 1, characterized in that The step of monitoring that a preset adjustment condition is met specifically includes: When it is monitored that the access delay of the database when being accessed by the user exceeds a preset threshold, it is determined that the adjustment condition is met.

4. The method according to claim 1, characterized in that The step of adjusting the shard storage strategy of each business data in the database according to the business association information corresponding to each access hotspot data specifically includes: With the optimization goals of minimizing the access delay when the database is accessed by users and maximizing the resource utilization corresponding to the database, the shard storage strategy of each business data in the database is adjusted according to the business association information corresponding to each access hotspot data.

5. The method according to claim 1, characterized in that The step of determining the target shard required for storing the business data according to the adjusted shard storage strategy, so as to migrate at least part of the business data from the original shard to the target shard for storage, specifically includes: Determine a data migration batch corresponding to each business data stored in the original shard; According to the data migration batch, at least part of the business data in each business data is migrated from the original shard to the target shard for storage in sequence.

6. The method according to claim 1, characterized in that The step of determining the target shard required for storing the business data according to the adjusted shard storage strategy, so as to migrate at least part of the business data from the original shard to the target shard for storage, specifically includes: Copying at least part of the business data to obtain mirror data; Migrating each of the mirrored data to the target shard; When it is determined that the migration of each mirror data is completed, at least part of the business data stored in the original slice is deleted.

7. The method according to claim 6, characterized in that Before the step of deleting at least part of the business data stored in the original slice, the method further includes: For each business data, if the business data is split into sub-data and stored in multiple target shards, the sub-data are obtained from the multiple target shards, and the sub-data are reassembled to obtain reassembled data; Performing a consistency check on the reorganized data and the business data; The step of deleting at least part of the business data stored in the original slice specifically includes: If it is determined that the split business data passes the consistency check, at least part of the business data stored in the original shard is deleted.

8. A data storage device, characterized in that: include: A monitoring module, used to determine the original shards that need to be adjusted when it is monitored that the preset adjustment conditions are met, and to obtain the data access logs corresponding to each business data stored in the original shards, wherein the data access logs are used to reflect the access popularity of each business data; A determination module, used to determine each access hotspot data in each business data according to the data access log; An adjustment module, used to adjust the shard storage strategy of each business data in the database according to the business association information corresponding to each access hotspot data, to obtain an adjusted shard storage strategy; A migration module is used to determine the target shard required to store the business data according to the adjusted shard storage strategy, so as to migrate at least part of the business data from the original shard to the target shard for storage.

9. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the method according to any one of claims 1 to 7 is implemented.

Images