Data storage method, system, terminal device and computer storage medium

Abstract

The application relates to the technical field of financial technology, and discloses a data storage method, a data storage system, a terminal device and a computer storage medium. The data storage method comprises the following steps: a data storage system for high-concurrency data is used to monitor data information of high-concurrency data and a data type to which the high-concurrency data belongs; according to a data consumption state of a local data consumption queue corresponding to the data information and the data type, the local data consumption queue is dynamically adjusted; the high-concurrency data is processed by calling the local data consumption queue which is dynamically adjusted, and the processed high-concurrency data is stored in a corresponding database. The application does not need to replace the system scheme, avoids huge resource consumption caused by system change, can improve the service efficiency of the system for high-concurrency data, and effectively avoids the problem that a data system cannot quickly process storage or service breakdown due to large database storage pressure.

Description

Technical Field

[0001] This invention relates to the field of financial technology (Fintech), and more particularly to a data storage method, system, terminal device, and computer storage medium. Background Technology

[0002] With the development of computer technology, more and more technologies are being applied in the financial field. The traditional financial industry is gradually transforming into financial technology. However, due to the requirements of security, real-time performance and stability in the financial industry, higher demands are also being placed on technology.

[0003] Currently, with the rapid growth of information and data, more and more internet servers are encountering high-concurrency data storage situations. Although most data storage systems currently use NoSQL (Not Only SQL, non-relational database caching) to solve the problem of high-concurrency data storage, completely replacing the data storage solution with NoSQL for a relatively old data storage system requires huge changes to the data storage system and consumes a lot of resources. However, if the traditional data storage solution is still used, it will lead to excessive database pressure under high-concurrency data storage, which can easily cause problems such as data storage request timeouts and system server crashes.

[0004] In summary, how to improve the service efficiency of data storage systems to cope with high-concurrency data storage while consuming fewer resources is a technical problem that the industry urgently needs to solve. Summary of the Invention

[0005] The main objective of this invention is to provide a data storage method, system, terminal device, and computer storage medium, aiming to solve the technical problem of how to improve the service efficiency of a data storage system to cope with high-concurrency data storage while consuming fewer resources.

[0006] To achieve the above objectives, the present invention provides a data storage method, which is applied to a data storage system oriented towards high-concurrency data. The data storage method includes the following steps:

[0007] Monitor the data information of high-concurrency data and the data type to which the high-concurrency data belongs;

[0008] Based on the data information and the data consumption status of the local data consumption queue corresponding to the data type, the local data consumption queue is dynamically adjusted;

[0009] The dynamically adjusted local data consumption queue is invoked to process the high-concurrency data, and the processed high-concurrency data is stored in the corresponding database.

[0010] Furthermore, to achieve the above objectives, the present invention also provides a data storage system, which is applied to a data storage system for high-concurrency data, the data storage system comprising:

[0011] The monitoring module is used to monitor the data information of high-concurrency data and the data type to which the high-concurrency data belongs;

[0012] The queue dynamic adjustment module is used to dynamically adjust the local data consumption queue based on the data information and the data consumption status of the local data consumption queue corresponding to the data type.

[0013] The concurrent data processing module is used to call the dynamically adjusted local data consumption queue to process the high-concurrency data and store the processed high-concurrency data in the corresponding database.

[0014] In this invention, each functional module of the data storage system implements the steps of the data storage method described above during operation.

[0015] In addition, to achieve the above objectives, the present invention also provides a terminal device, the terminal device comprising: a memory, a processor, and a data storage program stored in the memory and executable on the processor, wherein the data storage program, when executed by the processor, implements the steps of the data storage method as described above.

[0016] In addition, to achieve the above objectives, the present invention also provides a computer storage medium storing a data storage program, which, when executed by a processor, implements the steps of the data storage method described above.

[0017] In addition, to achieve the above objectives, the present invention also provides a computer program product comprising a computer program that, when executed by a processor, implements the steps of the data storage method described above.

[0018] This invention provides a data storage method, system, terminal device, computer storage medium, and computer program product. It monitors the data information and data type of high-concurrency data through a data storage system for high-concurrency data; dynamically adjusts the local data consumption queue based on the data information and the data consumption status of the local data consumption queue corresponding to the data type; processes the high-concurrency data by calling the dynamically adjusted local data consumption queue; and stores the processed high-concurrency data in the corresponding database.

[0019] When using a data storage system for high-concurrency data, this invention first monitors the data information and data type of the high-concurrency data. Then, based on the monitored data information and the data consumption status of the currently configured local data consumption queue corresponding to the data type, it dynamically adjusts the local data consumption queue in real time. Finally, it processes the high-concurrency data by calling the dynamically adjusted local data consumption queue and stores the processed high-concurrency data in the corresponding database.

[0020] Compared to traditional methods that completely replace storage solutions to serve high-concurrency data, this invention only monitors the data information of high-concurrency data. Based on this data information and the real-time data consumption status of the local data consumption queue within the system, it dynamically allocates data consumption queues in real time, thereby prioritizing system resources for processing and storing high-concurrency data. Thus, this invention eliminates the need for system-wide solution changes, avoiding the enormous resource consumption associated with system modifications. Furthermore, it significantly improves the system's efficiency in serving high-concurrency data, effectively preventing data systems from experiencing storage or service outages due to high database storage pressure. Attached Figure Description

[0021] Figure 1 This is a schematic diagram of the device structure of the terminal device hardware operating environment involved in the embodiments of the present invention;

[0022] Figure 2 This is a flowchart illustrating an embodiment of the data storage method of the present invention;

[0023] Figure 3 This is a schematic diagram illustrating an application scenario of a local data consumption queue involved in an embodiment of the data storage method of the present invention;

[0024] Figure 4 This is a schematic diagram illustrating an application scenario of a monitoring data consumption queue according to an embodiment of the data storage method of the present invention;

[0025] Figure 5 This is a schematic diagram illustrating an application scenario for storing high-concurrency data, based on an embodiment of the data storage method of the present invention.

[0026] Figure 6 This is a schematic diagram of the data routing application process for high-concurrency data in an embodiment of the data storage method of the present invention;

[0027] Figure 7 This is a schematic diagram of the functional modules of an embodiment of the data storage system of the present invention.

[0028] The realization of the objective, functional features and advantages of the present invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0029] It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0030] Reference Figure 1 , Figure 1 This is a schematic diagram of the hardware operating environment of the terminal device involved in the embodiments of the present invention.

[0031] The terminal device in this embodiment of the invention can be a terminal device configured as a data storage system that provides data services for high-concurrency data. The terminal device can be a server, smartphone, PC (Personal Computer), tablet computer, portable computer, etc.

[0032] like Figure 1 As shown, the terminal device may include: a processor 1001, such as a CPU; a communication bus 1002; a user interface 1003; a network interface 1004; and a memory 1005. The communication bus 1002 is used to enable communication between these components. The user interface 1003 may include a display screen and an input unit such as a keyboard; optionally, the user interface 1003 may also include a standard wired interface or a wireless interface. The network interface 1004 may optionally include a standard wired interface or a wireless interface (such as a Wi-Fi interface). The memory 1005 may be high-speed RAM or non-volatile memory, such as a disk drive. Optionally, the memory 1005 may also be a storage device independent of the aforementioned processor 1001.

[0033] Those skilled in the art will understand that Figure 1 The terminal device structure shown does not constitute a limitation on the terminal device and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0034] like Figure 1 As shown, the memory 1005, which serves as a computer storage medium, may include an operating system, a network communication module, a user interface module, and a data storage program.

[0035] exist Figure 1 In the terminal shown, the network interface 1004 is mainly used to connect to the backend server and communicate with the backend server; the user interface 1003 is mainly used to connect to the client and communicate with the client; and the processor 1001 can be used to call the data storage program stored in the memory 1005 and execute various embodiments of the following data storage methods.

[0036] Based on the above hardware structure, various embodiments of the data storage method of the present invention are proposed.

[0037] It's important to note that with the rapid growth of information and data, more and more internet servers are encountering high-concurrency data storage scenarios. While most data storage systems currently address high-concurrency data storage by employing NoSQL (Not Only SQL, non-relational database caching), completely replacing an older data storage system with NoSQL requires significant changes and consumes substantial resources. However, if traditional data storage solutions are still used, high-concurrency data storage can lead to excessive database pressure, potentially causing data storage requests to time out and resulting in server crashes.

[0038] To address the above-mentioned issues, this invention provides a data storage method. Please refer to... Figure 2 , Figure 2 This is a flowchart illustrating a first embodiment of the data storage method of the present invention. In this embodiment, the data storage method is applied to a terminal device configured as a data storage system providing data services for high-concurrency data. Specifically, the terminal device can act as a server for high-concurrency data in the data storage system. The data storage method of the present invention includes:

[0039] Step S10: Monitor the data information of high-concurrency data and the data type to which the high-concurrency data belongs;

[0040] During the process of providing data services to high-concurrency data, the terminal device continuously monitors the data information of the high-concurrency data and simultaneously detects which type of data the high-concurrency data is currently, such as new, deleted, queried, or updated.

[0041] It should be noted that, in this embodiment, the terminal device, configured as a data storage system providing data services for high-concurrency data, provides data services to high-concurrency data according to a pre-configured dynamic queue allocation mechanism. That is, based on the data information of the high-concurrency data being processed (also known as concurrent data characteristics, such as the size and concurrency speed of the concurrent data), the number of local data consumption queues is dynamically adjusted to process and store the high-concurrency data. Thus, the terminal device continuously monitors the data information of the high-concurrency data throughout the entire process.

[0042] Specifically, for example, the terminal device can obtain data information such as the amount of high-concurrency data, the data concurrency speed, and even the data initiation and termination time from the connected data service platform in real time through the method of embedding points. Then, it can perform real-time analysis on the data information obtained in real time and the relevant status of the local data consumption queue collected from the terminal to provide corresponding services for the high-concurrency data on the data service platform.

[0043] Furthermore, in this embodiment, after acquiring high-concurrency data from the data service platform, the terminal device can detect whether the data type of the high-concurrency data is update, deletion, query, or creation. Specifically, the terminal device detects whether the currently acquired high-concurrency data is of the data type (update, delete, query, or creation) by recognizing the data type identifier (encapsulated in data characters) carried by the high-concurrency data.

[0044] It should be noted that, in this embodiment, the terminal device, as a data storage system providing data services for high-concurrency data, can also employ NIO (Java non-blocking IO) to distribute high-concurrency data generated by multiple data service platforms or clients to a local data consumption queue through multiplexing. It should be understood that, based on different design needs of actual applications, in any feasible implementation, the terminal device can use any different data publishing methods to allow the local data consumption queue to simultaneously accept high-concurrency data transmitted by multiple clients. The data storage method of this invention does not limit the specific publishing method for high-concurrency data.

[0045] Step S20: Based on the data information and the data consumption status of the local data consumption queue corresponding to the data type, dynamically adjust the local data consumption queue;

[0046] It should be noted that, in this embodiment, the terminal device pre-constructs a one-to-one correspondence between each data type of high-concurrency data and its corresponding local data consumption queue, based on the data types such as creation, update, deletion, and query. Thus, after the terminal device acquires high-concurrency data in real time and identifies its data type, it can process the high-concurrency data of that data type by dynamically adjusting the local data consumption queue corresponding to that data type and calling the adjusted local data consumption queue.

[0047] After monitoring and obtaining real-time data information of high-concurrency data, the terminal device further analyzes this data information together with the real-time data consumption status of the local data consumption queue that corresponds one-to-one with the data type of the high-concurrency data collected from the local device. In this way, the local data consumption queue is dynamically adjusted in real time according to the pre-configured dynamic queue allocation mechanism.

[0048] It should be noted that in this embodiment, the pre-configured dynamic queue allocation mechanism of the data consumption system dynamically allocates local consumption queues based on the size of different concurrent data volumes and the data consumption speed of the local data consumption queues. This ensures that the processing needs of high-concurrency data are met in real time while avoiding resource consumption. The specific principle of the dynamic queue allocation mechanism configured in this embodiment is as follows: when a local data consumption queue receives a particularly large amount of concurrent data within a certain time period, leading to a rapid increase in concurrent data, but the data consumption speed provided by the local data consumption queue is insufficient to meet the processing speed of this concurrent data, the data storage system uses this dynamic queue allocation mechanism to dynamically increase the number of local data consumption queues to meet the processing speed of this concurrent data. Conversely, when the concurrent data volume decreases, the data storage system reduces the number of local data consumption queues according to the dynamic queue allocation mechanism, thereby realizing the functions of system resource reclamation and dynamic queue allocation.

[0049] Furthermore, it should be understood that, based on different design requirements of practical applications, in any other feasible implementation, the terminal device may of course adopt other processes different from those described in this embodiment to achieve dynamic adjustment of the local data consumption queue. The data storage method of the present invention does not limit the specific implementation method of dynamically adjusting the local data consumption queue. Any other means of dynamically adjusting the data consumption queue are within the technical concept of the present invention and should therefore be included in the protection scope of the present invention.

[0050] Further, in a feasible embodiment, the data information of the high-concurrency data continuously monitored by the terminal device is: the data volume of the high-concurrency data, while the real-time data consumption status of the local data consumption queue collected by the terminal device from its own end is: the data consumption speed of the local data consumption queue. Step S20 above may include:

[0051] Step S201: Obtain the data consumption speed of the local data consumption queue corresponding to the data type;

[0052] After monitoring and obtaining real-time data information of high-concurrency data and the data type to which the high-concurrency data belongs, the terminal device collects the data consumption speed of the local data consumption queue corresponding to the data type in real time.

[0053] Specifically, for example, in this embodiment, the terminal device obtains data information such as the amount of high-concurrency data, the data concurrency speed, and even the data initiation and termination time of high-concurrency data from the connected data service platform in real time through the method of embedding points. When it is determined that the data type of the high-concurrency data is "new" based on the identification of the data type encapsulated by the high-concurrency data, it immediately collects the data consumption speed of the local data consumption queue corresponding to the "new" data type at the current moment from the local terminal.

[0054] Step S202: Detect whether the data volume and the data consumption speed meet the preset consumption queue increase conditions, wherein the consumption queue increase conditions are: the data consumption speed does not meet the high-concurrency data consumption of the data volume;

[0055] When the terminal device analyzes the amount of high-concurrency data it acquires together with the real-time data consumption speed of the local data consumption queue collected from the local end, it first checks whether the data consumption speed is sufficient to process the high-concurrency data of that size. If it detects that the data consumption speed is insufficient to process the high-concurrency data, it determines that the preset conditions for increasing the consumption queue have been met.

[0056] Specifically, for example, a terminal device can detect whether the data consumption speed of its local data consumption queue is sufficient to handle a certain volume of high-concurrency data by checking if the number of pending messages accumulated in the local data consumption queue exceeds a threshold (assuming the threshold is 5000 messages). That is, if the terminal device detects that the data service platform subscribed to by local data consumption queue 1 has 30,000 high-concurrency data messages at the current moment, and there is a queue in local data consumption queue 1 with more than 5000 pending messages, it determines that the current data consumption speed of local data consumption queue 1 is insufficient to handle the 30,000 high-concurrency data messages. Therefore, the terminal device determines that local data consumption queue 1 has met the conditions for adding more consumption queues for this high-concurrency data at the current moment.

[0057] Step S203: If the condition for increasing the consumption queue is met, then the number of local data consumption queues is dynamically increased.

[0058] When the terminal device detects and determines that the preset conditions for increasing the consumption queue have been met, it immediately and dynamically increases the number of local data consumption queues so that the increased number of local data consumption queues can meet the consumption speed for processing high-concurrency data.

[0059] Specifically, for example, after the terminal device detects and determines that the data consumption speed of the local data consumption queue 1 at the current moment is no longer sufficient to process the 30,000 high-concurrency data entries, and further determines that the local data consumption queue 1 has met the conditions for increasing the consumption queue for the high-concurrency data at the current moment, the terminal device increases the number of local data consumption queues 1 so that the increased number of local data consumption queues 1 can meet the processing needs of the 30,000 high-concurrency data entries.

[0060] It should be noted that, in this embodiment, when the terminal device increases the number of local data consumption queues, assuming that three-quarters of the local data consumption queues each have more than 5,000 pending message data, the terminal device can add a new local data consumption queue with the number of queues equal to one-quarter of the current total number of local data consumption queues. If the total number of local data consumption queues multiplied by one-quarter results in a decimal, it can be rounded down to obtain the number of queues to be added.

[0061] Step S204: Detect whether the data volume and the data consumption speed have reached the preset consumption queue recycling condition, wherein the consumption queue reduction condition is: the data consumption speed consumes a surplus of high-concurrency data of the data volume;

[0062] The terminal device analyzes the amount of high-concurrency data it acquires, along with the real-time data consumption speed of the local data consumption queue collected from the local end. This analysis determines whether the preset conditions for increasing the consumption queue have been met. It also checks whether the data consumption speed is sufficient to process the high-concurrency data of this size and whether there is still some surplus. If a surplus is detected, it determines that the preset conditions for recycling the consumption queue have been met.

[0063] Specifically, for example, the terminal device can detect whether the number of pending messages accumulated in the local data consumption queue is less than a threshold (assuming the threshold is 1000 messages) to determine whether the data consumption speed of the local data consumption queue still has a surplus for consuming a certain amount of high-concurrency data. That is, when the terminal device detects that the data service platform subscribed to by local data consumption queue 2 has 10,000 high-concurrency data messages at the current moment, and there is a queue in local data consumption queue 2 with less than 1000 pending messages, it determines that the data consumption speed of local data consumption queue 2 at the current moment has a surplus for consuming the 10,000 high-concurrency data messages. Therefore, the terminal device determines that local data consumption queue 2 has met the consumption queue reclamation conditions for the high-concurrency data at the current moment.

[0064] Step S205: If the consumption queue recycling condition is met, the number of local data consumption queues is dynamically reduced.

[0065] When the terminal device detects and determines that the preset consumption queue recycling conditions have been met, it immediately and dynamically reduces the number of local data consumption queues so that the reduced number of local data consumption queues can meet the consumption speed for processing high-concurrency data.

[0066] It should be noted that, in this embodiment, when the terminal device reduces the number of local data consumption queues, it sequentially reclaims queues in the local data consumption queues that have accumulated fewer than 1,000 pending message data entries, until all queues in the local data consumption queues have accumulated more than or equal to 1,000 pending message data entries, or until only one secondary data consumption queue remains in the remaining local data consumption queues after reclamation.

[0067] Furthermore, in this embodiment, please refer to Figure 3 As shown in the application scenario, the local data consumption queues on the terminal device can include: a primary data consumption queue (illustrated) and secondary data consumption queues (illustrated as select local message queue, update local message queue, delete local message queue, and create local message queue). The number of primary data consumption queues corresponds one-to-one with the types of high-concurrency data. That is, when the data service platform connected to the terminal device includes four types of high-concurrency data: delete, update, create, and select, the terminal device needs to configure at least the following four primary data consumption queues on its local end: select primary message queue, update primary message queue, delete primary message queue, and create primary message queue. A single primary data consumption queue can be subscribed to by one or more secondary data consumption queues. Furthermore, this primary data consumption queue, along with the one or more secondary data consumption queues subscribed to it, can be collectively referred to as a local consumption queue cluster. Alternatively, only the one or more secondary data consumption queues subscribed to the same primary data consumption queue can be collectively referred to as a local consumption queue cluster.

[0068] In this embodiment, by subdividing high-concurrency data into different categories, i.e., distributing the datasets to the corresponding main data consumption queues, the data in each main data consumption queue is prevented from becoming too large. Furthermore, once data arrives at a main data consumption queue, one or more corresponding slave data consumption queues will subscribe to the data and process it accordingly. Thus, this invention first addresses the issue by distributing the data, distributing high-concurrency data twice, which significantly reduces server load, increases system concurrency, and effectively prevents server crashes caused by excessive high-concurrency data.

[0069] In addition, please refer to the following: Figure 4 In the application scenario shown, the terminal device uses a sub-thread to monitor all configured local data consumption queues (the local consumption queue cluster shown in the diagram) in real time and records the queue data of the slave data consumption queues in each queue, as well as the number of pending message data accumulated in each slave data consumption queue. Each local data consumption queue, by default, has four slave data consumption queues that simultaneously subscribe to the master data consumption queue during initialization.

[0070] In this embodiment, the step of "dynamically increasing the number of local data consumption queues" in step S203 above may include:

[0071] Step S2031: Dynamically increase the number of slave data consumption queues and detect whether the number of slave data consumption queues has reached a preset maximum value, wherein the preset maximum value is: the maximum number of slave data consumption queues that simultaneously subscribe to a single master data consumption queue;

[0072] When the terminal device dynamically increases the number of local data consumption queues, it prioritizes increasing the number of slave data consumption queues within that local data consumption queue. After each increase of a slave data consumption queue, it checks whether the total number of slave data consumption queues has reached the maximum number of slave data consumption queues that can simultaneously subscribe to all slave data consumption queues of the current local data consumption queue's master data consumption queue.

[0073] Specifically, for example, when a terminal device dynamically increases the number of local data consumption queues, it prioritizes the number of four slave data consumption queues that simultaneously subscribe to a master data consumption queue within that local data consumption queue: 4 multiplied by one-quarter to get 1. Then, it adds one slave data consumption queue that also subscribes to the master data consumption queue within that local data consumption queue. After each addition of the slave data consumption queue, it checks whether the total number of slave data consumption queues has exceeded the maximum number of slave data consumption queues that the master data consumption queue is allowed to subscribe to.

[0074] Step S2032: If yes, then increase the number of master data consumption queues and continue to dynamically increase the number of slave data consumption queues;

[0075] When a terminal device dynamically increases the number of local data consumption queues, if the total number of such data consumption queues after adding a new data consumption queue has reached the maximum number of data consumption queues that the main data consumption queue in the local data consumption queue can be subscribed to, and it is still necessary to add more data consumption queues, the terminal device will first add a new main data consumption queue, and then add a new data consumption queue and have the new data consumption queue subscribe to the new main data consumption queue.

[0076] Specifically, for example, assuming that in a local data consumption queue, the maximum number of slave data consumption queues that the master data consumption queue can be subscribed to is 10, then when the terminal device first increases the number of slave data consumption queues subscribed to the master data consumption queue to 10, and still needs to continue to increase the number of slave data consumption queues, the terminal device adds a new master data consumption queue that is the same as the master data consumption queue. Then, it continues to add new slave data consumption queues, and makes the newly added slave data consumption queues directly subscribe to the new master data consumption queue.

[0077] Furthermore, in a feasible embodiment, step S203 above, the step of "dynamically increasing the number of local data consumption queues," may further include:

[0078] Step S2033: Obtain real-time memory data, and stop increasing the number of local data consumption queues when the real-time memory data reaches the preset limit for increasing the consumption queue.

[0079] In this embodiment, to prevent the number of local data consumption queues from increasing endlessly, which could lead to system memory overflow and other problems, the terminal device can monitor memory usage in real time through a sub-thread, and then control the cessation of increasing the number of local data consumption queues based on this usage. Furthermore, the real-time memory data obtained by the terminal device through monitoring memory usage via a sub-thread includes, but is not limited to: freeMemory (size of free memory), totalMemory (current total memory size), and maxMemory (maximum available memory size). The preset limit for increasing the consumption queue is that the ratio of freeMemory to totalMemory is less than or equal to a preset constant value, which can be optionally set to 7 / 10, or the ratio of totalMemory to maxMemory is greater than or equal to a preset constant value, such as 7 / 10. During the dynamic addition of local data consumption queues, the terminal device monitors memory usage via a sub-thread, obtaining the aforementioned real-time memory data. When the ratio of freeMemory to totalMemory is less than or equal to 7 / 10, or the ratio of totalMemory to maxMemory is greater than or equal to 7 / 10, it determines that the limit for adding consumption queues has been reached, and immediately stops adding new master data consumption queues and / or new slave data consumption queues. This effectively prevents excessive system memory consumption by the local data consumption queues, which could negatively impact other system functions.

[0080] Furthermore, in this embodiment, the step of "dynamically reducing the number of local data consumption queues" in step S205 above may further include:

[0081] Step S2051: Among the multiple data consumption queues, determine the target queue with the least amount of message data to be processed.

[0082] When a terminal device dynamically reduces the number of local data consumption queues, it first detects and determines the target queue with the fewest accumulated pending message data at the current moment among the multiple slave data consumption queues of the local data consumption queue.

[0083] Step S2052: Transfer all pending message data in the target queue and reclaim the target queue to reduce the number of data consumption queues;

[0084] After identifying the target queue among multiple data consumption queues with the fewest pending message data at the current moment, the terminal device immediately transfers all pending message data accumulated in the target queue to other queues among the multiple data consumption queues excluding the target queue. After the target queue finishes processing the message data currently being consumed, the target queue is immediately reclaimed. This process of reclaiming target queues continues until the terminal device determines that the above-mentioned consumption queue reclamation conditions have not been met, thereby reducing the number of data consumption queues.

[0085] Specifically, for example, a terminal device uses a sub-thread to monitor all local consumption queue clusters in real time and records the existing master and slave data consumption queues in each cluster (such as the speed of data processing, creation time, memory usage, etc.), as well as the number of pending messages accumulated in each slave data consumption queue in the cluster. Therefore, when the terminal device determines that the local data consumption queues have met the consumption queue reclamation conditions for high-concurrency data (assuming that the reclamation condition is: three-quarters of the slave data consumption queues have fewer than 1000 pending messages), and needs to dynamically reduce the number of local data consumption queues, the following two methods can be used:

[0086] Method 1: The terminal device first selects the target queue with the smallest accumulated message count from all local data consumption queues. Then, it transfers all pending message data accumulated in this target queue to one or more of the remaining queues. After the target queue finishes processing its message data, it immediately reclaims the target queue. This process is repeated, reclaiming each target queue one by one, until the total number of pending message data accumulated in all data consumption queues exceeds 1000, or until only one data consumption queue remains.

[0087] Method 2: The terminal device uses a step-by-step cumulative transfer to reclaim data from the data consumption queue. That is, the terminal device first selects the target queue with the smallest number of accumulated messages at the current moment. Then, it accumulates all the unprocessed message data in the target queue with the messages in the second-to-last data consumption queue with the number of accumulated unprocessed message data. If the total number of unprocessed message data after accumulation is greater than 1000, the accumulation is abandoned. If the total number of unprocessed message data after accumulation is less than 1000, the accumulation is performed and the above process continues until the total number of accumulated unprocessed message data exceeds 1000, or until the condition that "three-quarters of the queues have less than 1000 accumulated unprocessed message data" is no longer met in the data consumption queues.

[0088] It should be noted that, in this embodiment, in the second method described above, after the terminal device adds all the pending message data in the target queue to the second-to-last queue with the number of pending message data entries, it immediately reclaims the target queue after the message data being processed in the target queue has been processed.

[0089] Step S2053: When the number of the slave data consumption queues is reduced to zero, and the target master data consumption queues subscribed to by the slave data consumption queues have the same master data consumption queues, the target master data consumption queues are reclaimed to reduce the number of master data consumption queues.

[0090] When a terminal device dynamically reduces the number of local data consumption queues, if all slave data consumption queues subscribed to the current target master data consumption queue have been reduced to zero, and the target master data consumption queue still has one or more master data consumption queues of the same type, then the terminal device will further reclaim the target master data consumption queue, thereby reducing the number of master data consumption queues in the local data consumption queue.

[0091] Specifically, for example, suppose that the current local data consumption queue 3 contains two main data consumption queues of the same type (e.g., both are select main message queues): main data consumption queue 1 and main data consumption queue 2. Main data consumption queue 1 has 5 subscribed slave data consumption queues, while main data consumption queue 2 has 3 subscribed slave data consumption queues. Thus, after the terminal device reclaims all 3 slave data consumption queues subscribed to main data consumption queue 2 according to either "Method 1" or "Method 2" (the unprocessed message data accumulated in these 3 slave data consumption queues is transferred to one or more of the 5 slave data consumption queues subscribed to main data consumption queue 1), it further reclaims main data consumption queue 2 as well.

[0092] Step S30: The dynamically adjusted local data consumption queue is invoked to process the high-concurrency data, and the processed high-concurrency data is stored in the corresponding database.

[0093] After the terminal device dynamically adjusts the local data consumption queue in real time according to the pre-configured dynamic queue allocation mechanism, it calls the dynamically adjusted local data consumption queue to process high-concurrency data and then directly stores the processed high-concurrency data into the corresponding database.

[0094] Furthermore, in a feasible embodiment, step S30 above, the step of "calling the dynamically adjusted local data consumption queue to process the high-concurrency data" may include:

[0095] Step S301: Invoke the dynamically adjusted local data consumption queue;

[0096] Step S302: Determine the target data consumption queue for processing the high-concurrency data from the local data consumption queue according to the preset hash algorithm;

[0097] Step S303: According to the preset guaranteed scheduling algorithm, the target data consumption queue processes the high-concurrency data fairly.

[0098] It should be noted that in this embodiment, the terminal device uses a hash algorithm and a guaranteed scheduling algorithm to determine the data content that multiple data consumption queues subscribe to from the main data consumption queue, thereby achieving fairness in the entire local data consumption queue. That is, it ensures that the number of pending message data accumulated in one data consumption queue will not far exceed that of other data consumption queues.

[0099] After the terminal device dynamically adjusts the local data consumption queue in real time according to the pre-configured dynamic queue allocation mechanism, it calls the dynamically adjusted local data consumption queue and processes the high-concurrency data according to the preset hash algorithm and guaranteed scheduling algorithm.

[0100] Specifically, for example, the terminal device calls a dynamically adjusted local data consumption queue. The main data consumption queue in the local data consumption queue receives the high-concurrency data to be processed. Then, the hash algorithm (n-1)&hash (where n represents the number of slave data consumption queues in the local data consumption queue and hash represents the hash code of the high-concurrency data to be processed) is used to determine which slave data consumption queue in the local data consumption queue that subscribes to the main data consumption queue will consume and process the high-concurrency data to be processed.

[0101] Then, during the process of the terminal device subscribing to and consuming high-concurrency data to be processed from the main data consumption queue in each of the slave data consumption queues, the scheduling algorithm ensures fairness among the slave data consumption queues. That is, among the n slave data consumption queues in the current local data consumption queue, when new message data to be processed (i.e., high-concurrency data to be processed) is randomly assigned to a slave data consumption queue based on a hash algorithm until the difference in the number of message data to be processed between the slave data consumption queues is greater than or equal to 100, the terminal device selects the target queue with the smallest number of message data to be processed at the current moment, and then assigns the new message data to be processed received randomly based on the hash algorithm to the target queue, until the number of message data to be processed accumulated in the target queue exceeds the number of message data to be processed accumulated in any other slave data consumption queue closest to the target queue. Next, it checks whether there are any target queues in all data consumption queues where the difference in the number of pending messages is greater than or equal to 100. If they exist, the above process continues; otherwise, the hash algorithm is used to randomly allocate new pending message data.

[0102] Furthermore, in a feasible embodiment, step S30 above, "storing the processed high-concurrency data in the corresponding database," may include:

[0103] Step S302: Obtain the data number of the processed high-concurrency data;

[0104] Step S303: According to the number range in which the data number is located, the processed high-concurrency data is routed to the database corresponding to the number range for storage. Different number ranges correspond to different databases, and each database stores the same data table.

[0105] After the terminal device calls the dynamically adjusted local data consumption queue and processes the high-concurrency data according to the preset hash algorithm and guaranteed scheduling algorithm, it further obtains the data number of each of the high-concurrency data after consumption. Then, according to the number range of the data number, the high-concurrency data after consumption is routed to the same data table in different databases corresponding to the number range for storage.

[0106] It should be noted that, in this embodiment, it should be understood that, based on different design requirements of actual applications, in other feasible implementation methods, the terminal device may of course adopt a different routing method than the data number-based method used here to route high-concurrency data that has been consumed to the same data table in different databases for storage. The data storage method of the present invention is not limited to the specific routing rules for high-concurrency data storage.

[0107] In addition, please refer to the following: Figure 5 In one feasible embodiment of the application scenario illustrated, the terminal device pre-establishes a correspondence between multiple databases and their respective number ranges based on different data number ranges. Then, after consuming high-concurrency data through a dynamically adjusted local data consumption queue, the high-concurrency data is routed to the same table in different databases for storage according to the number range of the data number. This achieves separate storage of high-concurrency data, horizontally splitting it and increasing the concurrency capacity of the storage system. Furthermore, it prevents a single table in a database from becoming excessively large; that is, the maximum number of records in a single table within a database will not exceed 5 million, thus avoiding slow response times for data update or deletion operations (in traditional storage solutions, data from the same table is stored in only one table). Simultaneously, by routing high-concurrency data to different databases, multiple commands correspond to multiple databases, forming a many-to-many relationship, avoiding the many-to-one relationship in traditional storage solutions and significantly reducing the data storage pressure on the databases.

[0108] Furthermore, in this embodiment, the terminal device calculates the data number for the high-concurrency data in advance during the storage process of the high-concurrency data after consumption processing.

[0109] When a terminal device stores high-concurrency data that has undergone consumption processing into the corresponding database, it first calculates the data number of the high-concurrency data. Then, based on the number range of that data number, it routes the high-concurrency data to the same data table in different databases corresponding to that number range for storage. Specifically, see, for example... Figure 6 The application process shown is as follows: after the terminal device calls the dynamically adjusted local data consumption queue to consume high-concurrency data, it first calculates the data number of the high-concurrency data, then obtains the data number through database routing and determines the number range in which the data number is located, and then routes the high-concurrency data to the database corresponding to the number range according to the data number, so that the high-concurrency data is stored in the corresponding data table in the database.

[0110] Based on the above, the data storage method of the present invention may further include the following when calculating the data number of high-concurrency data:

[0111] Step A: When it is detected that the data type of the high-concurrency data is newly created, a new data number is calculated for the high-concurrency data;

[0112] Step B: When it is detected that the data type of the high-concurrency data is deletion, query, or update, the original data number is determined for the high-concurrency data.

[0113] In this embodiment, after consuming high-concurrency data by calling a dynamically adjusted local data consumption queue, the terminal device detects whether the data type of the high-concurrency data is update, deletion, query, or creation. If the terminal device detects that the data type is creation, it increments the primary key of the same data table in the database that stores creation-type data by one to generate a new data ID for the current high-concurrency data. Alternatively, if the terminal device detects that the data type is deletion, query, or update, it does not generate a new data ID for the high-concurrency data; instead, it directly determines the existing data ID of the high-concurrency data in the database table that stores deletion, query, or creation-type data.

[0114] It should be noted that, in this embodiment, in addition to calculating a new data number for high-concurrency data based on the primary key of the database table, the terminal device can also randomly generate a new data number for newly created high-concurrency data based on hash calculation. For example, the hash value of the hash calculation for high-concurrency data can be directly used as the new data number.

[0115] In this embodiment, during the process of providing data services to high-concurrency data, the terminal device continuously monitors the data information of the high-concurrency data and simultaneously detects which data type the high-concurrency data is currently, such as creation, deletion, query, or update. After monitoring the real-time data information of the high-concurrency data, the terminal device further analyzes this data information together with the real-time data consumption status of the local data consumption queue, which corresponds one-to-one with the data type of the high-concurrency data, collected from the local device. This allows for real-time dynamic adjustment of the local data consumption queue according to a pre-configured dynamic queue allocation mechanism. After real-time dynamic adjustment of the local data consumption queue according to the pre-configured dynamic queue allocation mechanism, the terminal device uses the dynamically adjusted local data consumption queue to process the high-concurrency data and then directly stores the processed high-concurrency data into the corresponding database.

[0116] Compared to traditional methods that completely replace storage solutions to serve high-concurrency data, this invention only monitors the data information of high-concurrency data. Based on this data information and the real-time data consumption status of the local data consumption queue within the system, it dynamically allocates data consumption queues in real time, thereby tilting system resources to process and store high-concurrency data. Thus, this invention avoids the huge resource consumption caused by system changes by eliminating the need for a complete system overhaul. Furthermore, it significantly improves the system's efficiency in serving high-concurrency data, effectively preventing data systems from experiencing storage or service outages due to high database storage pressure.

[0117] Furthermore, the present invention also provides a data storage system applied to a data storage system oriented towards high-concurrency data. Please refer to... Figure 7 , Figure 7 This is a functional module diagram of an embodiment of the data storage system of the present invention. Figure 7 As shown, the data storage system of the present invention includes:

[0118] The monitoring module 10 is used to monitor the data information of high-concurrency data and the data type to which the high-concurrency data belongs;

[0119] The queue dynamic adjustment module 20 is used to dynamically adjust the local data consumption queue according to the data information and the data consumption status of the local data consumption queue corresponding to the data type;

[0120] The concurrent data processing module 30 is used to call the dynamically adjusted local data consumption queue to process the high-concurrency data and store the processed high-concurrency data in the corresponding database.

[0121] Furthermore, the data information refers to: the amount of high-concurrency data, and the data consumption status refers to: the data consumption speed of the local data consumption queue;

[0122] The queue dynamic adjustment module 20 includes:

[0123] The first acquisition unit is used to acquire the data consumption speed of the local data consumption queue corresponding to the data type;

[0124] The first detection unit is used to detect whether the data volume and the data consumption speed have reached the preset consumption queue increase condition, wherein the consumption queue increase condition is: the data consumption speed does not meet the high-concurrency data consumption of the data volume;

[0125] The queue dynamic addition unit is used to dynamically increase the number of local data consumption queues if the first detection unit detects that the consumption queue addition condition has been met.

[0126] The second detection unit is used to detect whether the data volume and the data consumption speed have reached the preset consumption queue recycling condition, wherein the consumption queue reduction condition is: the data consumption speed consumes a surplus of high-concurrency data of the data volume;

[0127] The queue dynamic reduction unit is used to dynamically reduce the number of the local data consumption queues if the consumption queue recycling condition is detected.

[0128] Furthermore, the local data consumption queue includes a master data consumption queue and slave data consumption queues, wherein a single master data consumption queue is simultaneously subscribed to by one or more slave data consumption queues;

[0129] The queue dynamically adds units, including:

[0130] The sub-unit is added from the queue to dynamically increase the number of the slave data consumption queues and to detect whether the number of the slave data consumption queues has reached a preset maximum value, wherein the preset maximum value is: the maximum number of slave data consumption queues that simultaneously subscribe to a single master data consumption queue;

[0131] The main queue adds sub-units to increase the number of the main data consumption queue and continue to dynamically increase the number of the slave data consumption queues;

[0132] The queue dynamic reduction unit includes:

[0133] A determination subunit is used to determine the target queue with the least amount of currently pending message data among multiple data consumption queues;

[0134] The queue reclamation subunit is used to transfer all pending message data in the target queue and reclaim the target queue to reduce the number of data consumption queues.

[0135] The main queue reclamation subunit is used to reclamate the target main data consumption queue to reduce the number of main data consumption queues when the number of the slave data consumption queues is reduced to zero and the target main data consumption queues subscribed to by the slave data consumption queues have the same main data consumption queue.

[0136] Furthermore, the dynamic addition of units to the queue also includes:

[0137] The limiting subunit is used to acquire real-time memory data and stop increasing the number of local data consumption queues when the real-time memory data reaches the preset limit condition for increasing the consumption queue.

[0138] Furthermore, the concurrent data processing module 30 includes:

[0139] The data processing unit is configured to invoke the dynamically adjusted local data consumption queue; determine the target data consumption queue for processing the high-concurrency data from the local data consumption queue according to a preset hash algorithm; and ensure that the target data consumption queue processes the high-concurrency data fairly according to a preset guaranteed scheduling algorithm.

[0140] Furthermore, the concurrent data processing module 30 also includes:

[0141] The second acquisition unit is used to acquire the data number of the processed high-concurrency data;

[0142] The storage unit is used to route the processed high-concurrency data to the database corresponding to the number range according to the number range in which the data number is located, wherein different number ranges correspond to a database, and each database stores the same data table.

[0143] Furthermore, the data types include: creation, deletion, query, and update. The data storage system of this invention also includes:

[0144] The new number generation module is used to calculate a new data number for the high-concurrency data when it is detected that the data type of the high-concurrency data is newly created.

[0145] The original number determination module is used to determine the original data number for the high-concurrency data when it is detected that the data type of the high-concurrency data is deletion, query or update.

[0146] The functions of each module in the above data storage system correspond to the steps in the above data storage method embodiments, and their functions and implementation processes will not be described in detail here.

[0147] The present invention also provides a computer storage medium storing a data storage program, which, when executed by a processor, implements the steps of the data storage method as described in any of the above embodiments.

[0148] The specific embodiments of the computer storage medium of the present invention are basically the same as the embodiments of the data storage method described above, and will not be repeated here.

[0149] The present invention also provides a computer program product, the computer program product comprising a computer program, which, when executed by a processor, implements the steps of the data storage method as described in any of the above embodiments.

[0150] The specific embodiments of the computer program product of the present invention are basically the same as the embodiments of the data storage method described above, and will not be repeated here.

[0151] The above are merely preferred embodiments of the present invention and do not limit the scope of the patent. Any equivalent structural or procedural transformations made based on the description and drawings of the present invention, or direct or indirect applications in other related technical fields, are similarly included within the scope of patent protection of the present invention.

Claims

1. A data storage method, characterized in that, The data storage method is applied to a data storage system oriented towards high-concurrency data, and the data storage method includes the following steps: Monitor the data information of high-concurrency data and the data type to which the high-concurrency data belongs; Based on the data information and the data consumption status of the local data consumption queue corresponding to the data type, the local data consumption queue is dynamically adjusted; The dynamically adjusted local data consumption queue is invoked to process the high-concurrency data, and the processed high-concurrency data is stored in the corresponding database. The data information refers to: the amount of high-concurrency data, and the data consumption status refers to: the data consumption speed of the local data consumption queue; The step of dynamically adjusting the local data consumption queue based on the data information and the data consumption status of the local data consumption queue corresponding to the data type includes: Obtain the data consumption speed of the local data consumption queue corresponding to the data type; The system detects whether the data volume and the data consumption speed meet the preset conditions for increasing the consumption queue, wherein the conditions for increasing the consumption queue are: the data consumption speed does not meet the requirements for consuming the data volume of high-concurrency data; If the detection meets the condition for increasing the consumption queue, then the number of local data consumption queues is dynamically increased; and / or, The system detects whether the data volume and the data consumption speed have reached the preset consumption queue recycling conditions, wherein the consumption queue reduction condition is: the data consumption speed consumes a surplus of high-concurrency data of the data volume; If the detection meets the conditions for recycling the consumption queue, the number of local data consumption queues will be dynamically reduced.

2. The data storage method as described in claim 1, characterized in that, The local data consumption queue includes a master data consumption queue and slave data consumption queues, wherein a single master data consumption queue is simultaneously subscribed to by one or more slave data consumption queues; The step of dynamically increasing the number of local data consumption queues includes: The number of slave data consumption queues is dynamically increased, and it is detected whether the number of slave data consumption queues has reached a preset maximum value, wherein the preset maximum value is: the maximum number of slave data consumption queues that simultaneously subscribe to a single master data consumption queue; If so, then increase the number of the master data consumption queue and continue to dynamically increase the number of the slave data consumption queues; The step of dynamically reducing the number of local data consumption queues includes: Among the multiple data consumption queues, determine the target queue with the least amount of pending message data; Transfer all pending message data in the target queue and reclaim the target queue to reduce the number of data consumption queues; When the number of slave data consumption queues is reduced to zero, and the target master data consumption queues subscribed to by the slave data consumption queues have the same master data consumption queue, the target master data consumption queues are reclaimed to reduce the number of master data consumption queues.

3. The data storage method as described in claim 1, characterized in that, The step of dynamically increasing the number of local data consumption queues further includes: Acquire real-time memory data, and stop increasing the number of local data consumption queues when the real-time memory data reaches the preset limit condition for adding consumption queues.

4. The data storage method as described in claim 1, characterized in that, The step of calling the dynamically adjusted data consumption queue to process the high-concurrency data includes: Invoke the dynamically adjusted local data consumption queue; The target data consumption queue for processing the high-concurrency data is determined from the local data consumption queue according to a preset hash algorithm; According to the preset guaranteed scheduling algorithm, the target data consumption queue processes the high-concurrency data fairly.

5. The data storage method as described in claim 1, characterized in that, The step of storing the processed high-concurrency data into the corresponding database includes: Obtain the data ID of the processed high-concurrency data; According to the number range in which the data number is located, the processed high-concurrency data is routed to the database corresponding to the number range for storage. Different number ranges correspond to different databases, and each database stores the same data table.

6. The data storage method according to any one of claims 1 to 5, characterized in that, The data types include: create, delete, query, and update; the methods also include: When the data type of the high-concurrency data is detected to be newly created, a new data number is calculated for the high-concurrency data; or, When it is detected that the data type of the high-concurrency data is deletion, query, or update, the original data number is determined for the high-concurrency data.

7. A data storage system, characterized in that, The data storage system is applied to a data storage system oriented towards high-concurrency data, and the data storage system includes: The monitoring module is used to monitor the data information of high-concurrency data and the data type to which the high-concurrency data belongs; The queue dynamic adjustment module is used to dynamically adjust the local data consumption queue based on the data information and the data consumption status of the local data consumption queue corresponding to the data type. The concurrent data processing module is used to call the dynamically adjusted local data consumption queue to process the high-concurrency data and store the processed high-concurrency data in the corresponding database. The data information refers to: the amount of high-concurrency data, and the data consumption status refers to: the data consumption speed of the local data consumption queue; The queue dynamic adjustment module is specifically used to obtain the data consumption speed of the local data consumption queue corresponding to the data type; detect whether the data volume and the data consumption speed have reached a preset consumption queue increase condition, wherein the consumption queue increase condition is: the data consumption speed does not meet the high-concurrency data consumption of the data volume; if the consumption queue increase condition is reached, the number of local data consumption queues is dynamically increased; and / or, detect whether the data volume and the data consumption speed have reached a preset consumption queue reduction condition, wherein the consumption queue reduction condition is: the data consumption speed has a surplus in consuming the high-concurrency data of the data volume; if the consumption queue reduction condition is reached, the number of local data consumption queues is dynamically reduced.

8. A terminal device, characterized in that, The terminal device includes: a memory, a processor, and a data storage program stored in the memory and executable on the processor, wherein the data storage program, when executed by the processor, implements the steps of the data storage method as described in any one of claims 1 to 6.

9. A computer storage medium, characterized in that, The computer storage medium stores a data storage program, which, when executed by a processor, implements the steps of the data storage method as described in any one of claims 1 to 6.

Images