Data storage method, device, storage medium and electronic device

Abstract

This application discloses a data storage method, device, storage medium, and electronic device, relating to the field of big data. The method comprises: receiving target data to be stored, storing the target data in a cache, and writing the target data to a message queue via the cache; determining whether the target data has been successfully written to the message queue; if the target data has not been successfully written to the message queue, sending the target data to a target engine and periodically rewriting the target data in the target engine to the message queue; and if the target data has been successfully written to the message queue, monitoring the message queue and storing the target data in the message queue in a database cluster. This application solves the problem in the related art of MySQL databases being unable to concurrently store large amounts of data.

Description

Technical Field

[0001] The present application relates to the field of big data, and more specifically, to a data storage method, device, storage medium, and electronic device. Background Art

[0002] With the rapid development of information technology, data is growing explosively. As data increases, so do the requirements for data storage. The data storage process in existing solutions typically involves sending a user request to a gateway, which performs load balancing and forwards the request to a backend application service. The backend application service receives the request, generates the corresponding data, stores it in a corresponding database, and then feeds the data back to the user. The database typically uses MySQL.

[0003] However, because the MySQL database's main bottleneck lies in input and output performance, it doesn't perform well in scenarios with high concurrency. In our business systems, if there are many users or high concurrency, the MySQL database can become a system bottleneck, causing data storage system lags. Furthermore, in situations with high instantaneous concurrency, the MySQL database can't effectively withstand traffic spikes. However, the MySQL database is a well-established relational database, so optimizing the data storage system is necessary to address these storage issues.

[0004] With regard to the problem in related technologies that the MySQL database cannot concurrently store large amounts of data, no effective solution has yet been proposed. Summary of the Invention

[0005] The present application provides a data storage method, device, storage medium and electronic device to solve the problem in the related art that the MySQL database cannot concurrently store large amounts of data.

[0006] According to one aspect of the present application, a data storage method is provided. The method includes: receiving target data to be stored, storing the target data in a cache, and writing the target data to a message queue through the cache; determining whether the target data is successfully written to the message queue; if the target data is not successfully written to the message queue, sending the target data to a target engine and periodically rewriting the target data in the target engine to the message queue; if the target data is successfully written to the message queue, monitoring the message queue and storing the target data in the message queue in a database cluster.

[0007] Optionally, after sending the target data to the target engine, the method further includes: determining the data attribute information of the target data through preset rules in the target engine to obtain the target attribute information; determining the data processing method corresponding to the target data according to the target attribute information and the preset mapping relationship to obtain the target data processing method, wherein the preset mapping relationship is used to characterize the relationship between multiple preset data attribute information and corresponding data processing methods, and the data processing method includes at least one of the following: a data storage method, an early warning notification method, and a data deletion method; when the target data processing method is a data storage method, executing an operation of periodically rewriting the target data in the target engine into the message queue; when the target data processing method is a early warning notification method, determining the organization that sends the target data, generating alarm information for the target data, and sending the alarm information to the organization; when the target data processing method is a data deletion method, deleting the target data in the target engine and deleting the target data in the cache.

[0008] Optionally, the data attribute information of the target data is determined by preset rules in the target engine, and the target attribute information of the target data is obtained, including: judging whether the target data carries an importance identifier and / or a user-preset identifier; in the case that the target data carries an importance identifier, determining the importance identifier as the target attribute information; and / or in the case that the target data carries a user-preset identifier, determining the user-preset identifier as the target attribute information; in the case that the target data does not carry the importance identifier and the user-preset identifier, inputting the target data into a natural language processing model to obtain an attribute identifier corresponding to the target data, and determining the attribute identifier as the target attribute information, wherein the natural language processing model is a model obtained by training through attribute identifiers and historical data.

[0009] Optionally, before periodically obtaining the target data from the target engine and rewriting the target data into the message queue, the method further includes: obtaining the number of write failures in writing the target data into the message queue, and determining whether the number of write failures is greater than a number threshold; when the number of write failures is less than or equal to the number threshold, executing the step of periodically rewriting the target data in the target engine into the message queue; when the number of write failures is greater than the number threshold, determining the organization that sends the target data, generating alarm information for the target data, and sending the alarm information to the organization.

[0010] Optionally, after storing the target data in the database cluster via the message queue, the method further includes: receiving a data acquisition request sent by the user end, wherein the data acquisition request is used to acquire the target data; acquiring the target data from the cache, and sending the target data to the user end.

[0011] Optionally, after storing the target data in the database cluster through the message queue, the method further includes: comparing the target data in the cache with the target data in the database cluster according to a preset period to obtain a comparison result; when the comparison result indicates that the version of the target data in the cache is inconsistent with the version of the target data in the database cluster, updating the target data with a lower version to the target data with a higher version, wherein the version indicates an updated version of the data; when the comparison result indicates that the data content of the target data in the cache is inconsistent with the data content of the target data in the database cluster, sending the target data in the cache to the database cluster through the message queue to update the target data in the database cluster.

[0012] Optionally, after storing the target data in the database cluster through the message queue, the method further includes: after receiving a modification instruction, modifying the target data in the cache according to the modification instruction, and sending the modified target data to the message queue through the cache, wherein the modification instruction is used to modify the target data; sending the modified target data to the database cluster through the message queue, and updating the target data in the database cluster to the modified target data.

[0013] According to another aspect of the present application, a data storage device is provided. The device includes: a storage unit for receiving target data to be stored, storing the target data in a cache, and writing the target data into a message queue through the cache; a judgment unit for judging whether the target data is successfully written into the message queue; a first sending unit for sending the target data to a target engine if the target data is not successfully written into the message queue, and periodically rewriting the target data in the target engine into the message queue; and a monitoring unit for monitoring the message queue if the target data is successfully written into the message queue, and storing the target data in the message queue into a database cluster.

[0014] According to another aspect of an embodiment of the present invention, a computer storage medium is provided. The computer storage medium is used to store a program. When the program is running, the device where the computer storage medium is located is controlled to execute a data storage method.

[0015] According to another aspect of an embodiment of the present invention, an electronic device is provided, comprising one or more processors and a memory; the memory stores computer-readable instructions, and the processor is used to execute the computer-readable instructions, wherein a data storage method is executed when the computer-readable instructions are executed.

[0016] Through this application, the following steps are adopted: receiving target data to be stored, storing the target data in a cache, and writing the target data into a message queue through the cache; judging whether the target data is successfully written into the message queue; in the case that the target data is not successfully written into the message queue, sending the target data to the target engine, and regularly rewriting the target data in the target engine into the message queue; in the case that the target data is successfully written into the message queue, monitoring the message queue, and storing the target data in the message queue into the database cluster. The problem that the MySQL database in the related art cannot concurrently store a large amount of data is solved. By first storing the target data into a cache with a high query rate per second, and sending the target data to the message queue through the cache, due to the decoupling, asynchrony and peak-cutting properties of the message queue, the target data can be smoothly written into the MySQL database, and the data that cannot be successfully written is stored in the target engine, and regularly rewritten into the message queue, thereby achieving the effect of concurrently storing a large amount of data by assisting the MySQL database with the message queue and the target engine. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The accompanying drawings, which constitute part of this application, are intended to provide a further understanding of this application. The exemplary embodiments and descriptions of this application are intended to explain this application and do not constitute an improper limitation on this application. In the accompanying drawings:

[0018] Figure 1 is a flowchart of a data storage method provided according to an embodiment of the present application;

[0019] Figure 2 is a schematic diagram of an optional data storage system according to an embodiment of the present invention;

[0020] Figure 3 is a schematic diagram of a data storage device provided according to an embodiment of the present application;

[0021] Figure 4 A schematic diagram of an electronic device provided according to an embodiment of the present application. DETAILED DESCRIPTION

[0022] It should be noted that, in the absence of conflict, the embodiments and features of the embodiments in this application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0023] In order to enable those skilled in the art to better understand the present invention, the following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of this application.

[0024] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present application described here. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0025] It should be noted that the relevant information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for display and analysis, etc.) involved in this disclosure are all information and data authorized by the user or fully authorized by all parties. For example, an interface is set up between this system and the relevant user or organization. Before obtaining relevant information, it is necessary to send an acquisition request to the aforementioned user or organization through the interface, and obtain the relevant information after receiving the consent information fed back by the aforementioned user or organization.

[0026] It should be noted that the data storage method, device, storage medium and electronic device determined in the present disclosure can be used in the field of big data, and can also be used in any field other than the field of big data. The application field of the data storage method, device, storage medium and electronic device determined in the present disclosure is not limited.

[0027] For ease of description, some nouns or terms involved in the embodiments of the present application are explained below:

[0028] Message queue: A "first-in, first-out" data structure in the basic data structure, which can be used to solve problems such as application decoupling, asynchronous messaging, and traffic peak shaving.

[0029] Cache: A memory that can exchange data at high speed and has a faster access speed than RAM.

[0030] Query rate per second: A measure of how much traffic a particular query server handles within a specified time period.

[0031] According to an embodiment of the present application, a data storage method is provided.

[0032] Redis: Remote Dictionary Server, an open source database that provides APIs in multiple languages.

[0033] Figure 1 This is a flow chart of a data storage method according to an embodiment of the present application. Figure 1 As shown, the method includes the following steps:

[0034] Step S101 : receiving target data to be stored, storing the target data in a cache, and writing the target data into a message queue through the cache.

[0035] Specifically, the target data can be generated based on user needs and stored when the data is fed back to the user. Since the MySQL database cannot store large amounts of data with high concurrency, the data must first be stored in a cache and then temporarily stored in a message queue. The message queue can control the data storage capacity according to the maximum concurrent storage capacity of the MySQL database, thereby ensuring that the data can be stored normally in the MySQL database.

[0036] Step S102: Determine whether the target data is successfully written into the message queue.

[0037] Specifically, when writing target data into the message queue, a write exception may occur. At this time, a large amount of write failure data will be generated. Therefore, it is necessary to determine whether the target data is successfully written when writing the target data into the message queue, so as to process the unsuccessfully written data in a timely manner.

[0038] Step S103: If the target data is not successfully written into the message queue, the target data is sent to the target engine, and the target data in the target engine is rewritten into the message queue at a regular interval.

[0039] Specifically, if the target data is not written successfully, the target data needs to be sent to the target engine, processed and stored in the target engine, and the target plug-in will periodically rewrite the target data in the target engine into the message queue to ensure that all data can be added to the database cluster through the message queue.

[0040] Step S104: When the target data is successfully written into the message queue, the message queue is monitored, and the target data in the message queue is stored in the database cluster.

[0041] Specifically, the target data successfully written into the message queue will be monitored in real time by the target plug-in, and when the message queue instructs to send the target data to the database cluster, the target plug-in will add the target data to the database cluster, thereby completing the persistent storage of the target data.

[0042] In this embodiment, if Figure 2 As shown, the aforementioned data storage method can be executed by an optional data storage system as an execution subject, and the data storage system at least includes a background application, a target plug-in, a cache component, a message queue, a target engine and a MySQL database.

[0043] Specifically, such as Figure 2 As shown, after the background application generates the corresponding target data according to the user request, it sends the target data to the target plug-in. At this time, the target plug-in receives the target data, stores the target data in the cache component, and sends the target data to the message queue through the cache component. Optionally, since the target plug-in has a temporary storage component, the target data can also be sent to the cache component and the message queue at the same time through the target plug-in. The purpose of sending the target data to the message queue through the cache component is to ensure data consistency and prevent the occurrence of different data in the message queue and the cache component.

[0044] For example, the cache component can be Redis. First, determine the configuration file corresponding to the target plug-in. The main configurations in the configuration file include: the address of the message queue to be monitored, the topic (flag) name of the subscribed message queue, the interval time for pulling failed messages, the threshold for pulling failed messages, the scheduled task thread pool configuration, the Redis address, the Redis connection pool configuration, etc. Based on these configurations, create the corresponding thread pool and start the target plug-in. After the target plug-in is started, it first connects to Redis and creates a connection pool for the Redis service. Then, based on the configuration information of the message queue, it connects to the message queue and subscribes to the topic specified in the configuration file to monitor the topic. Based on the thread pool configuration information, create a scheduled task thread pool to periodically pull data stored in the database that failed to be written to the message queue. During the operation of the target plug-in, when data needs to be written through the target plug-in, the corresponding routing key and the data to be written can be specified. The task is submitted to the thread pool provided within the target plug-in through asynchronous writing. The thread in the thread pool then writes to Redis and then writes to the message queue. Since the target plug-in subscribes to the corresponding topic of the message queue when it is started, once a message is written to the topic in the message queue, the thread in the thread pool subscribed to the topic in the target plug-in will receive the message. The thread will then smoothly persist the received message to the database, completing the data storage.

[0045] Furthermore, when writing data to the message queue, a write exception may occur. At this time, the target plug-in will send the target data that failed to be written to the target engine, and process and store the target data in the target engine. The target plug-in will periodically rewrite the target data in the target engine to the message queue to ensure that all data can be added to the database cluster through the message queue, where the database cluster can be a MySQL database.

[0046] If the target data can be successfully written into the message queue, the target plug-in monitors the message queue in real time, and when the message queue instructs to send the target data to the database cluster, the target plug-in adds the target data to the database cluster.

[0047] For example, consider the case where a large number of users have added a product to their favorites on the shopping mall interface. To quickly respond to user requests and generate favorites data, the target plugin stores the generated data in a cache. This data is then added to the message queue and immediately returned to the user, indicating the successful addition. The message queue pushes this data to applications subscribed to the corresponding topic for consumption. After consuming the data, the consumer persists it to a MySQL database. Because the message queue can control push speeds and offers high throughput and message reliability, it can withstand traffic spikes even when large amounts of data are written, preventing anomalies in the data storage system and MySQL database.

[0048] The data storage method provided by the embodiment of the present application receives the target data to be stored, stores the target data in a cache, and writes the target data into a message queue through the cache; determines whether the target data is successfully written into the message queue; if the target data is not successfully written into the message queue, sends the target data to the target engine, and periodically rewrites the target data in the target engine into the message queue; if the target data is successfully written into the message queue, monitors the message queue, and stores the target data in the message queue into the database cluster. The problem that the MySQL database in the related art cannot concurrently store a large amount of data is solved. By first storing the target data in a cache with a high query rate per second, and sending the target data to the message queue through the cache, due to the decoupling, asynchrony and peak-cutting nature of the message queue, the target data can be smoothly written into the MySQL database, and the data that cannot be successfully written can be stored in the target engine and rewritten into the message queue periodically, thereby achieving the effect of concurrently storing a large amount of data by assisting the MySQL database with the message queue and the target engine.

[0049] In order to perform corresponding processing on different target data in the target engine, optionally, in the data storage method provided in the embodiment of the present application, after sending the target data to the target engine, the method further includes: determining the data attribute information of the target data through preset rules in the target engine to obtain the target attribute information; determining the data processing method corresponding to the target data according to the target attribute information and the preset mapping relationship to obtain the target data processing method, wherein the preset mapping relationship is used to characterize the relationship between multiple preset data attribute information and the corresponding data processing methods, and the data processing method includes at least one of the following: a data storage method, an early warning notification method, and a data deletion method; when the target data processing method is a data storage method, executing the operation of periodically rewriting the target data in the target engine into the message queue; when the target data processing method is a early warning notification method, determining the mechanism for sending the target data, generating alarm information for the target data, and sending the alarm information to the mechanism; when the target data processing method is a data deletion method, deleting the target data in the target engine and deleting the target data in the cache.

[0050] Specifically, after the target data that cannot be successfully written into the message queue is sent to the target engine, the target data needs to be processed in the target engine to determine a reasonable disposal of the target data that cannot be successfully written into the message queue.

[0051] First, preset rules are required to determine the data attribute information of the target data to obtain the target attribute information, wherein the target attribute information can be the data attributes carried by the target data itself, for example, whether the target data is important data, whether it is abnormal data, whether it is encrypted data, etc. After determining the target attribute information, it is necessary to determine the data processing method corresponding to the target data based on the target attribute information and the preset mapping relationship to obtain the target data processing method, that is, the data processing method of the target data is determined according to the correspondence between the target attribute information and the data processing method in the preset mapping relationship.

[0052] For example, the target attribute information may be that the target data is important data, and in the preset mapping relationship, the data processing method corresponding to the important data is the data storage method, then the data processing method of the target data is the data storage method.

[0053] Furthermore, in the case where the target data processing method is a data storage method, it indicates that the target engine has stored the target data in the database in the target engine, and then the target data can be periodically rewritten into the message queue by the target data in the target engine; in the case where the target data processing method is an early warning notification method, it indicates that an abnormality has occurred in the data, and it is necessary to notify the system maintenance personnel corresponding to the target data, that is, the organization corresponding to the target data, and send the abnormal alarm information to the corresponding organization; in the case where the target data processing method is a data deletion method, it proves that the data is relatively important and cannot be returned to the requesting party. It is necessary to delete the target data in the target engine and the target data in the cache, thereby protecting the data. This embodiment achieves the effect of reasonable disposal of different data by performing different operations on different data in the target engine.

[0054] In order to accurately determine the attribute information of the target data, optionally, in the data storage method provided in the embodiment of the present application, the data attribute information of the target data is determined by preset rules in the target engine, and obtaining the target attribute information of the target data includes: judging whether the target data carries an importance identifier and / or a user-preset identifier; in the case where the target data carries an importance identifier, determining the importance identifier as the target attribute information; and / or in the case where the target data carries a user-preset identifier, determining the user-preset identifier as the target attribute information; in the case where the target data does not carry the importance identifier and the user-preset identifier, inputting the target data into a natural language processing model to obtain the attribute identifier corresponding to the target data, and determining the attribute identifier as the target attribute information, wherein the natural language processing model is a model obtained by training through attribute identifiers and historical data.

[0055] Specifically, there may be a corresponding identifier in the target data, which is an identifier manually added when the target data is created. After the target data is sent to the target engine, the target engine will check whether the target data carries a status according to the importance rule, that is, whether the target data carries an importance identifier. If the target data carries an importance identifier, it can be indicated that the target data is important data.

[0056] Furthermore, it is necessary to determine whether the target data complies with the user-defined rules according to the user-defined rules, that is, whether the target data carries a user-preset identifier. If the target data carries a user-preset identifier, it is determined that the target data complies with the user-defined rules.

[0057] When the target data does not meet the importance rules or the user-defined rules, that is, when the target data does not carry the importance identifier and the user-preset identifier, the target data is input into the natural language processing model, and the target attribute information of the target data is determined by the natural language processing model, wherein the natural language processing model can be a trained NPL natural language processing model, and the NPL natural language processing model can be trained by a large number of attribute identifiers and corresponding historical data to obtain a trained NPL natural language processing model.

[0058] Optionally, in the data storage method provided in the embodiment of the present application, before periodically obtaining the target data from the target engine and rewriting the target data into the message queue, the method also includes: obtaining the number of write failures for writing the target data into the message queue, and determining whether the number of write failures is greater than a number threshold; when the number of write failures is less than or equal to the number threshold, executing the step of periodically rewriting the target data in the target engine into the message queue; when the number of write failures is greater than the number threshold, determining the organization that sends the target data, generating alarm information for the target data, and sending the alarm information to the organization.

[0059] It should be noted that, since the target data needs to be rewritten into the message queue after being periodically obtained from the target engine, there is a possibility that the target data will still fail to be written into the message queue again. Therefore, in order to prevent falling into an infinite loop of repeated write failures, the number of write failures in the target data is changed each time the write to the message queue fails, so that the number of write failures can be used to determine whether it is necessary to perform the operation of repeatedly writing to the message queue again.

[0060] Specifically, the number of target data write failures can be compared with a threshold number of times, and a determination can be made based on the comparison result whether to continue executing the target data write operation. If the number of write failures is less than or equal to the threshold number of times, the target data write operation can be executed again. If the number of write failures is greater than the threshold number of times, it indicates that there have been too many failures and there may be problems with the data. Therefore, it is necessary to notify the system maintenance personnel corresponding to the target data, that is, the organization corresponding to the target data, and send the abnormal alarm information to the corresponding organization, thereby informing the requester that the data is in an abnormal state and cannot be stored. This embodiment achieves the effect of avoiding the phenomenon of falling into an infinite loop of repeated write failures.

[0061] Optionally, in the data storage method provided in an embodiment of the present application, after storing the target data in a database cluster through a message queue, the method further includes: receiving a data acquisition request sent by a user terminal, wherein the data acquisition request is used to obtain the target data; obtaining the target data from the cache, and sending the target data to the user terminal.

[0062] Specifically, when the user requests to obtain the target data again, due to the large concurrency of the cache and the fast data transmission speed, the target data can be obtained directly from the cache and the cached data can be directly returned to the user end, thereby achieving the effect of the data storage system quickly responding to user needs.

[0063] Optionally, in the data storage method provided in an embodiment of the present application, after the target data is stored in the database cluster through a message queue, the method further includes: comparing the target data in the cache with the target data in the database cluster according to a preset period to obtain a comparison result; when the comparison result indicates that the version of the target data in the cache is inconsistent with the version of the target data in the database cluster, updating the target data with a lower version to the target data with a higher version, wherein the version indicates an updated version of the data; when the comparison result indicates that the data content of the target data in the cache is inconsistent with the data content of the target data in the database cluster, sending the target data in the cache to the database cluster through a message queue to update the target data in the database cluster.

[0064] It's important to note that because the cache is designed for high concurrency and fast transmission, and the database cluster is designed for persistent storage, ensuring consistency between the target data in the cache and the database cluster is crucial. Therefore, the target data must maintain consistent versions and content in both the cache and the database cluster.

[0065] Specifically, when the version of the target data in the cache is inconsistent with the version of the target data in the database cluster, it is necessary to determine the higher version of the data in the two storage locations, and when the version of the target data in the cache is higher, the higher version data is sent to the database cluster through the message queue to replace the lower version target data in the database cluster. When the version of the target data in the database cluster is higher, the target data in the database cluster is sent to the cache through the target plug-in to replace the lower version target data in the cache, thereby updating the lower version target data to the higher version target data, so that the version of the target data in the cache and the version of the target data in the database cluster are both the latest versions, that is, the higher version.

[0066] Furthermore, when the data content of the target data in the cache is inconsistent with the data content of the target data in the database cluster, it is necessary to modify the data in the database cluster based on the target data in the cache to ensure data accuracy.

[0067] Optionally, in the data storage method provided in an embodiment of the present application, after storing the target data in the database cluster through a message queue, the method further includes: after receiving a modification instruction, modifying the target data in the cache according to the modification instruction, and sending the modified target data to the message queue through the cache, wherein the modification instruction is used to modify the target data; sending the modified target data to the database cluster through the message queue, and updating the target data in the database cluster to the modified target data.

[0068] Specifically, there will be a scenario where the target data in the database cluster is modified. When modifying the target data in the database cluster, the target data in the cache is first modified according to the modification instruction, and the modified target data is sent to the message queue through the cache, and then the modified data is sent to the database cluster, so that the unmodified target data in the database cluster is replaced by the modified target data, thereby completing the update of the target data.

[0069] It should be noted that when the modification instruction is to delete the target data, the data in the cache is deleted first, and the modified target data cannot be added to the message queue. In this case, the deletion instruction is directly sent to the message queue, and the deletion operation of the target data is executed in the database cluster.

[0070] For example, consider a scenario where a user wants to remove a favorite item. Upon receiving the user's cancellation request, the plugin first locates the user's favorited items in the cache based on the user's ID and then deletes the item from the user's favorites. Favorited items are stored in the cache using a hash structure, where the key is the user ID, the hash key is the favorited item ID, and the value is the item's detailed data. A message is then sent to the message queue to delete the user's favorited item. The target plugin then sends the delete instruction from the message queue to the MySQL database and deletes the favorited item record from the MySQL database based on the user ID and item ID.

[0071] It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and that, although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0072] The present application also provides a data storage device. It should be noted that the data storage device of the present application can be used to execute the data storage method provided in the present application. The data storage device provided in the present application is introduced below.

[0073] Figure 3Schematic diagram of a data storage device according to an embodiment of the present application. Figure 3 As shown, the device includes: a storage unit 31, a judgment unit 32, a first sending unit 33, and a monitoring unit 34.

[0074] The storage unit 31 is configured to receive target data to be stored, store the target data in a cache, and write the target data into a message queue through the cache.

[0075] The judging unit 32 is used to judge whether the target data is successfully written into the message queue.

[0076] The first sending unit 33 is configured to send the target data to the target engine if the target data is not successfully written into the message queue, and to rewrite the target data in the target engine into the message queue at a regular interval.

[0077] The monitoring unit 34 is configured to monitor the message queue and store the target data in the message queue into the database cluster when the target data is successfully written into the message queue.

[0078] The data storage device provided in the embodiment of the present application receives the target data to be stored through the storage unit 31, stores the target data in the cache, and writes the target data into the message queue through the cache. The judgment unit 32 judges whether the target data is successfully written into the message queue. If the target data is not successfully written into the message queue, the first sending unit 33 sends the target data to the target engine and periodically rewrites the target data in the target engine into the message queue. If the target data is successfully written into the message queue, the monitoring unit 34 monitors the message queue and stores the target data in the message queue into the database cluster. This solves the problem in the related art that the MySQL database cannot concurrently store large amounts of data. By first storing the target data in a cache with a high query rate per second and then sending the target data to the message queue through the cache, due to the decoupling, asynchrony and peak-cutting properties of the message queue, the target data can be smoothly written into the MySQL database, and the data that cannot be successfully written is stored in the target engine and rewritten into the message queue periodically, thereby achieving the effect of assisting the MySQL database in concurrently storing large amounts of data through the message queue and the target engine.

[0079] Optionally, in the data storage device provided in the embodiment of the present application, the device also includes: a first determination unit, used to determine the data attribute information of the target data through preset rules in the target engine to obtain the target attribute information; a second determination unit, used to determine the data processing method corresponding to the target data according to the target attribute information and the preset mapping relationship, to obtain the target data processing method, wherein the preset mapping relationship is used to characterize the relationship between multiple preset data attribute information and the corresponding data processing methods, and the data processing method includes at least one of the following: a data storage method, an early warning notification method, and a data deletion method; a first execution unit, used to, when the target data processing method is a data storage method, execute the operation of periodically rewriting the target data in the target engine into the message queue; a third determination unit, used to, when the target data processing method is a early warning notification method, determine the mechanism for sending the target data, generate alarm information for the target data, and send the alarm information to the mechanism; a deletion unit, used to, when the target data processing method is a data deletion method, delete the target data in the target engine and delete the target data in the cache.

[0080] Optionally, in the data storage device provided in the embodiment of the present application, the first determination unit includes: a judgment module for judging whether the target data carries an importance identifier and / or a user-preset identifier; a first determination module for determining the importance identifier as target attribute information when the target data carries an importance identifier; and / or a second determination module for determining the user-preset identifier as target attribute information when the target data carries a user-preset identifier; a third determination module for inputting the target data into a natural language processing model when the target data does not carry the importance identifier and the user-preset identifier, obtaining the attribute identifier corresponding to the target data, and determining the attribute identifier as the target attribute information, wherein the natural language processing model is a model obtained by training through attribute identifiers and historical data.

[0081] Optionally, in the data storage device provided in the embodiment of the present application, before periodically obtaining the target data from the target engine and rewriting the target data into the message queue, the device also includes: a first acquisition unit, used to obtain the number of write failures for writing the target data into the message queue, and determine whether the number of write failures is greater than a number threshold; a second execution unit, used to execute the step of periodically rewriting the target data in the target engine into the message queue when the number of write failures is less than or equal to the number threshold; a fourth determination unit, used to determine the organization that sends the target data when the number of write failures is greater than the number threshold, generate alarm information for the target data, and send the alarm information to the organization.

[0082] Optionally, in the data storage device provided in an embodiment of the present application, after the target data is stored in the database cluster through a message queue, the device also includes: a receiving unit for receiving a data acquisition request sent by a user terminal, wherein the data acquisition request is used to obtain the target data; a second acquisition unit for obtaining the target data from the cache and sending the target data to the user terminal.

[0083] Optionally, in the data storage device provided in the embodiment of the present application, after the target data is stored in the database cluster through the message queue, the device also includes: a comparison unit, used to compare the target data in the cache with the target data in the database cluster according to a preset period to obtain a comparison result; a first update unit, used to update the target data with a lower version to the target data with a higher version when the comparison result indicates that the version of the target data in the cache is inconsistent with the version of the target data in the database cluster, wherein the version indicates an updated version of the data; a second sending unit, used to send the target data in the cache to the database cluster through the message queue to update the target data in the database cluster when the comparison result indicates that the data content of the target data in the cache is inconsistent with the data content of the target data in the database cluster.

[0084] Optionally, in the data storage device provided in an embodiment of the present application, after the target data is stored in the database cluster through the message queue, the device also includes: a third sending unit, which is used to modify the target data in the cache according to the modification instruction after receiving the modification instruction, and send the modified target data to the message queue through the cache, wherein the modification instruction is used to modify the target data; a second update unit, which is used to send the modified target data to the database cluster through the message queue, and update the target data in the database cluster to the modified target data.

[0085] The above-mentioned data storage device includes a processor and a memory. The above-mentioned storage unit 31, judgment unit 32, first sending unit 33, monitoring unit 34, etc. are all stored in the memory as program units, and the processor executes the above-mentioned program units stored in the memory to realize corresponding functions.

[0086] The processor includes a kernel, which retrieves the corresponding program unit from the memory. One or more kernels can be set. By adjusting the kernel parameters, the problem of the MySQL database being unable to concurrently store large amounts of data in the related art is solved.

[0087] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM, and the memory includes at least one memory chip.

[0088] An embodiment of the present invention provides a computer-readable storage medium having a program stored thereon, which implements the data storage method when executed by a processor.

[0089] An embodiment of the present invention provides a processor, which is used to run a program, wherein the data storage method is executed when the program is running.

[0090] like Figure 4 As shown, an embodiment of the present invention provides an electronic device, wherein the electronic device 40 includes a processor, a memory, and a program stored in the memory and executable on the processor. When the processor executes the program, the following steps are implemented: receiving target data to be stored, storing the target data in a cache, and writing the target data into a message queue through the cache; determining whether the target data is successfully written into the message queue; if the target data is not successfully written into the message queue, sending the target data to a target engine, and periodically rewriting the target data in the target engine into the message queue; if the target data is successfully written into the message queue, monitoring the message queue, and storing the target data in the message queue into a database cluster. The device herein may be a server, a PC, a PAD, a mobile phone, etc.

[0091] The present application also provides a computer program product, which, when executed on a data processing device, is suitable for executing an initialization program having the following method steps: receiving target data to be stored, storing the target data in a cache, and writing the target data into a message queue through the cache; determining whether the target data is successfully written into the message queue; if the target data is not successfully written into the message queue, sending the target data to a target engine, and periodically rewriting the target data in the target engine into the message queue; if the target data is successfully written into the message queue, monitoring the message queue, and storing the target data in the message queue into a database cluster.

[0092] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0093] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the steps in the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0094] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0095] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0096] In a typical configuration, a computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory.

[0097] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0098] Computer-readable media includes permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. The information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include transitory computer-readable media (transitory media), such as modulated data signals and carrier waves.

[0099] It should also be noted that the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, commodity, or apparatus that includes a series of elements includes not only those elements but also other elements not explicitly listed, or includes elements inherent to such process, method, commodity, or apparatus. In the absence of further limitations, an element defined by the phrase "comprises a ..." does not exclude the presence of other identical elements in the process, method, commodity, or apparatus that includes the element.

[0100] The above are merely embodiments of the present application and are not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application should all be included within the scope of the claims of the present application.

Claims

1. A data storage method, characterized in that: include: Receive target data to be stored, store the target data in a cache, and write the target data into a message queue through the cache, wherein the message queue controls the data storage capacity according to the maximum concurrent storage capacity of the database, so that the data is normally stored in the database; Determining whether the target data is successfully written into the message queue; If the target data is not successfully written into the message queue, the target data is sent to the target engine, and the target data in the target engine is rewritten into the message queue at a regular interval; When the target data is successfully written into the message queue, monitoring the message queue and storing the target data in the message queue into a database cluster; After receiving the modification instruction, the target data in the cache is modified according to the modification instruction, and the modified target data is sent to the message queue through the cache.

2. The method according to claim 1, characterized in that After sending the target data to the target engine, the method further includes: Determine the data attribute information of the target data by using preset rules in the target engine to obtain target attribute information; Determining a data processing method corresponding to the target data based on the target attribute information and a preset mapping relationship to obtain a target data processing method, wherein the preset mapping relationship is used to characterize the relationship between multiple preset data attribute information and corresponding data processing methods, and the data processing method includes at least one of the following: a data storage method, an early warning notification method, and a data deletion method; In a case where the target data processing method is the data storage method, executing the operation of rewriting the target data in the target engine into the message queue at a regular interval; In a case where the target data processing method is the early warning notification method, determining an organization to send the target data, generating warning information of the target data, and sending the warning information to the organization; In a case where the target data processing method is the data deletion method, the target data in the target engine is deleted, and the target data in the cache is deleted.

3. The method according to claim 2, characterized in that Determining the data attribute information of the target data by using preset rules in the target engine, obtaining the target attribute information of the target data includes: Determining whether the target data carries an importance identifier and / or a user-preset identifier; In the case where the target data carries the importance identifier, determining the importance identifier as the target attribute information; and / or in the case where the target data carries the user-preset identifier, determining the user-preset identifier as the target attribute information; In the case that the target data does not carry the importance identifier and the user-preset identifier, the target data is input into a natural language processing model to obtain an attribute identifier corresponding to the target data, and the attribute identifier is determined as the target attribute information, wherein the natural language processing model is a model obtained by training through attribute identifiers and historical data.

4. The method according to claim 1, wherein Before periodically acquiring the target data from the target engine and rewriting the target data into the message queue, the method further includes: Obtaining a number of write failures for writing the target data into the message queue, and determining whether the number of write failures is greater than a number threshold; When the number of write failures is less than or equal to the number threshold, executing the step of rewriting the target data in the target engine into the message queue at a scheduled time; When the number of write failures is greater than the number threshold, an organization to which the target data is sent is determined, alarm information of the target data is generated, and the alarm information is sent to the organization.

5. The method according to claim 1, wherein After storing the target data in the database cluster through the message queue, the method further includes: receiving a data acquisition request sent by a user terminal, wherein the data acquisition request is used to acquire the target data; The target data is obtained from the cache, and the target data is sent to the user terminal.

6. The method according to claim 1, characterized in that After storing the target data in the database cluster through the message queue, the method further includes: Comparing the target data in the cache with the target data in the database cluster according to a preset period to obtain a comparison result; If the comparison result indicates that the version of the target data in the cache is inconsistent with the version of the target data in the database cluster, updating the target data with a lower version to the target data with a higher version, wherein the version indicates an updated version of the data; When the comparison result indicates that data content of the target data in the cache is inconsistent with data content of the target data in the database cluster, the target data in the cache is sent to the database cluster through the message queue to update the target data in the database cluster.

7. The method according to claim 1, characterized in that After storing the target data in the database cluster through the message queue, the method further includes: The modification instruction is used to modify the target data; The modified target data is sent to the database cluster through the message queue, and the target data in the database cluster is updated to the modified target data.

8. A data storage device, characterized in that include: a storage unit, configured to receive target data to be stored, store the target data in a cache, and write the target data into a message queue through the cache, wherein the message queue controls the data storage capacity according to the maximum concurrent storage capacity of the database so that the data is normally stored in the database; a judging unit, configured to judge whether the target data is successfully written into the message queue; a first sending unit, configured to send the target data to a target engine if the target data is not successfully written into the message queue, and to rewrite the target data in the target engine into the message queue at a regular interval; A monitoring unit, configured to monitor the message queue and store the target data in the message queue into a database cluster when the target data is successfully written into the message queue; The third sending unit is configured to, after receiving the modification instruction, modify the target data in the cache according to the modification instruction, and send the modified target data to the message queue through the cache.

9. A computer storage medium, characterized in that The computer storage medium is used to store a program, wherein when the program is running, the device where the computer storage medium is located is controlled to execute the data storage method according to any one of claims 1 to 7.

10. An electronic device, characterized in that: The device comprises one or more processors and a memory, wherein the memory is used to store one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the data storage method according to any one of claims 1 to 7.

Images