Data storage method and device, equipment and storage medium

Abstract

The invention provides a data storage method and device, equipment and a storage medium. The method comprises the steps that a data storage request sent by an application layer is received, the data storage request carries to-be-stored data and a target database type, and the data storage request is used for indicating the proxy service to write the to-be-stored data into a database corresponding to the target database type; target connection information is obtained according to the target database type, the target connection information comprises a first address of a database corresponding to the target database type, the first address is located in a first data center, and the data center where the proxy service is located is the first data center; writing the to-be-stored data into a first target database corresponding to the first address, and writing the to-be-stored data into a second target database corresponding to a second address of a second data center through a message queue; the second data center is in communication connection with the first data center. According to the method, the data multi-write efficiency is improved.

Description

Technical Field

[0001] This application relates to the technical field of data storage, and in particular, to a data storage method, apparatus, device, and storage medium. Background Art

[0002] Cloud computing refers to a system with extremely strong computing power formed through a computer network (mostly the Internet), which can store and aggregate relevant resources and can be configured on demand to provide personalized services to users. The carrier of cloud computing is the data center. In order to improve the high availability and stability of the business system, the business system can adopt a construction mode of dual-active or multi-active data centers, and this dual-active or multi-active data center construction mode is required to support multi-writing of data.

[0003] Currently, each application program directly writes data into the databases of each data center respectively to achieve multi-writing of data by each application program in the construction mode of dual-active or multi-active data centers. However, the efficiency of such data multi-writing is relatively low. Summary of the Invention

[0004] This application provides a data storage method, apparatus, device, and storage medium to solve the technical problem of relatively low accuracy and poor timeliness of the data written into each data center.

[0005] In a first aspect, this application provides a data storage method applied to a proxy service. The method includes:

[0006] Receiving a data storage request sent by the application layer. The data storage request carries the data to be stored and the target database type, and the data storage request is used to instruct the proxy service to write the data to be stored into the database corresponding to the target database type;

[0007] Obtaining target connection information according to the target database type. The target connection information includes the first address of the database corresponding to the target database type, and the first address is in the first data center, and the data center where the proxy service is located is the first data center;

[0008] Writing the data to be stored into the first target database corresponding to the first address, and writing the data to be stored into the second target database corresponding to the second address in the second data center through a message queue; the second data center is communicatively connected to the first data center.

[0009] In a possible implementation manner, obtaining target connection information according to the target database type includes:

[0010] Real-time monitoring the change of configuration parameters in the registration / configuration center in the first data center, and correspondingly updating the connection information in the proxy service according to the change of the configuration parameters to obtain the updated connection information;

[0011] Obtain the target connection information from the updated connection information according to the target database type.

[0012] In a possible implementation, writing the data to be stored into the first target database corresponding to the first address and writing the data to be stored into the second target database corresponding to the second address in the second data center through a message queue includes:

[0013] Obtain the connection between the proxy service and the first target database corresponding to the first address;

[0014] Write the data to be stored into the first target database corresponding to the first address, and write the data to be stored and the target database type into the message queue, so that the consumer program in the second data center writes the data to be stored obtained from the message queue into the second target database corresponding to the second address; the second address is obtained by the consumer program according to the target database type from the registration / configuration center in the second data center, and the consumer program in the second data center has subscribed to the message queue in advance.

[0015] In a possible implementation, before receiving the data storage request sent by the application layer, the method further includes:

[0016] The proxy service receives the connection request sent by the application layer;

[0017] In response to the connection request, establish a long connection with the application layer.

[0018] In a possible implementation, before writing the data to be stored into the first target database corresponding to the first address, the method further includes:

[0019] The proxy service determines whether the number of data storage requests received within a preset time window is less than or equal to the number of tokens generated within the preset time window;

[0020] If so, execute the step of writing the data to be stored into the first target database corresponding to the first address.

[0021] In a possible implementation, the method further includes:

[0022] Obtain the response duration of the first target database;

[0023] When the response duration is greater than or equal to the preset duration, return service unavailable information to the application layer and fuse the connection with the application layer; and,

[0024] Send a probe message to the first target database at a preset period to determine the response duration of the first target database, and when the response duration is less than the preset duration, resume the connection with the application layer.

[0025] In a second aspect, the present application provides a data storage device, which is applied to proxy services and includes a receiving module, an obtaining module, and a writing module. Among them,

[0026] The receiving module is configured to receive a data storage request sent by the application layer. The data storage request carries the data to be stored and the target database type, and the data storage request is used to instruct the proxy service to write the data to be stored into the database corresponding to the target database type;

[0027] The obtaining module is configured to obtain target connection information according to the target database type. The target connection information includes the first address of the database corresponding to the target database type, and the first address is in the first data center, and the data center where the proxy service is located is the first data center;

[0028] The writing module is configured to write the data to be stored into the first target database corresponding to the first address, and write the data to be stored into the second target database corresponding to the second address in the second data center through a message queue; the second data center is communicatively connected to the first data center.

[0029] In a possible implementation manner, the obtaining module is specifically configured to:

[0030] Real-time monitor the change of the configuration parameters of the registration / configuration center in the first data center, and update the connection information in the proxy service correspondingly according to the change of the configuration parameters to obtain the updated connection information;

[0031] Obtain the target connection information from the updated connection information according to the target database type.

[0032] In a possible implementation manner, the writing module is specifically configured to:

[0033] Obtain the connection between the proxy service and the first target database corresponding to the first address;

[0034] Write the data to be stored into the first target database corresponding to the first address, and write the data to be stored and the target database type into the message queue, so that the consumer program in the second data center writes the data to be stored obtained from the message queue into the second target database corresponding to the second address; the second address is obtained by the consumer program in the second data center from the registration / configuration center according to the target database type, and the consumer program in the second data center has subscribed to the message queue in advance.

[0035] In a possible implementation manner, the device further includes:

[0036] The proxy service receives a connection request sent by the application layer;

[0037] In response to the connection request, establish a long connection with the application layer.

[0038] In a possible implementation, the apparatus further includes:

[0039] The proxy service determines whether the number of received data storage requests within a preset time window is less than or equal to the number of tokens generated within the preset time window;

[0040] If so, perform the step of writing the data to be stored into the first target database corresponding to the first address.

[0041] In a possible implementation, the apparatus further includes:

[0042] Obtain the response duration of the first target database;

[0043] When the response duration is greater than or equal to the preset duration, return service unavailable information to the application layer and fuse the connection with the application layer; and,

[0044] Send a probe message to the first target database at a preset period to determine the response duration of the first target database. When the response duration is less than the preset duration, resume the connection with the application layer.

[0045] In a third aspect, the present application provides a data storage system, which includes a first data center and a second data center connected by communication. Both the first data center and the second data center include an application layer, a proxy layer, a component layer, a message queue, and a registration / configuration center. The proxy layer includes a proxy service, and the component layer includes at least one type of database;

[0046] The application layer is used to send a data storage request to the proxy service; the data storage request carries the data to be stored and the target database type;

[0047] The registration / configuration center is used to configure and save configuration parameters;

[0048] The proxy service is used to execute the data storage method described in the first aspect;

[0049] The message queue is used to temporarily store the data to be stored and the target database type until the consumer program in the second data center retrieves the data to be stored from the message queue and writes the data to be stored into the second target database corresponding to the second address in the second data center;

[0050] The database is used to store the data to be stored.

[0051] In a fourth aspect, the present application provides an electronic device, including: a processor and a memory communicatively connected to the processor; the memory stores computer-executable instructions; the processor executes the computer-executable instructions stored in the memory, so that the processor executes the data storage method described in the first aspect.

[0052] Fifth aspect, the present application provides a computer-readable storage medium storing computer-executable instructions, which are used to implement the data storage method described in the first aspect when executed by a processor.

[0053] Sixth aspect, the present application provides a computer program product including a computer program, which implements the data storage method described in the first aspect when executed by a processor.

[0054] Seventh aspect, the present application provides a chip storing a computer program, which implements the data storage method described in the first aspect when executed by the chip.

[0055] In a possible implementation, the chip is a chip in a chip module.

[0056] The present application provides a data storage method, apparatus, device, and storage medium, which are applied to proxy services. This method uniformly performs asynchronous multi-writing of data of each application program in two or more data centers through proxy services and message queues, which helps to improve the overall data multi-writing efficiency. In addition, the proxy service isolates each application program from each database, and the data storage requests of each application program are uniformly sent to the database through the proxy service, which can reduce the number of database connections, thereby reducing the pressure on the database and helping to improve the stability of the database. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] The accompanying drawings herein are incorporated into the specification and constitute a part of this specification, showing embodiments consistent with the present application, and are used together with the specification to explain the principles of the present application.

[0058] Figure 1 It is a schematic diagram of the architecture of a data storage system for a dual-active data center provided by an embodiment of the present application;

[0059] Figure 2 It is a flowchart of a data storage method provided by an embodiment of the present application Figure 1 ;

[0060] Figure 3 It is a flowchart of a data storage method provided by an embodiment of the present application Figure 2 ;

[0061] Figure 4 It is a schematic diagram of the structure of a data storage apparatus provided by an embodiment of the present application;

[0062] Figure 5 It is a schematic diagram of the hardware structure of an electronic device provided by an embodiment of the present application.

[0063] Through the above-mentioned drawings, specific embodiments of the present application have been shown, and more detailed descriptions will be provided hereinafter. These drawings and textual descriptions are not intended to limit the scope of the concept of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. Detailed Description of the Embodiments

[0064] Here, exemplary embodiments will be described in detail, and examples are shown in the drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present application. On the contrary, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0065] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in the present application are all information and data authorized by the user or fully authorized by all parties. Moreover, the collection, use, and processing of relevant data need to comply with relevant laws, regulations, and standards, and corresponding operation entrances are provided for users to choose to authorize or reject.

[0066] Cloud computing is another important transformation of Internet technology. It provides on-demand, dynamically scalable, and inexpensive computing services through the network and is a service related to information technology, software, and the Internet. The carrier of cloud computing is the data center. To improve the high availability and stability of the business system, the business system can adopt a construction mode of dual-active or multi-active data centers. Exemplarily, the dual-active mode can include a primary data center / backup data center, dual operating data centers, and dual-active data centers, etc.

[0067] Among them, in the dual-active mode of the primary data center / backup data center, the main business systems run in the primary data center, while the backup data center provides cold backup or hot backup for the business system. When the primary data center fails or becomes unavailable, the backup data center will automatically take over to ensure the continuous operation of the business.

[0068] Among them, in the dual-active mode of dual operating data centers, application programs are deployed in different data centers, and the application programs in different data centers back up each other. If the application program in a certain data center fails, the application program in the backup data takes over the production business without affecting the production business.

[0069] Among them, in the active-active data center mode, multiple data centers of the business system undertake business at the same time, and business is diverted according to the resource situation of the data. When a data center fails, the business diversion ratio is adjusted, the business in the failed data center is cut off, and other data centers take over the business of the failed data center.

[0070] In order to enable multiple data centers to work better in coordination, a dual-active or multi-active data center construction model is needed to support multiple data writes.

[0071] Currently, each application program writes data directly into a database (also referred to as a system component) of each data center to achieve multiple data writes for each application program in an active-active or active-active data center construction mode.

[0072] However, when multiple data are written in this way through direct writing by applications, on the one hand, each application needs to connect to the database of each data center respectively, and each application's connection to the database of a data center corresponds to a set of connection information. When there are a large number of applications, it is easy to cause a large number of connection information, chaotic configuration, and increased difficulty in later management and maintenance; at the same time, each application needs to connect to the database of the data center where it is located, which also leads to a large number of database connections, which easily leads to high database pressure and database instability; and when each application writes multiple data separately, it handles the delays, timeouts, write failures and other exceptions that may occur in the multiple writing process separately, and no unified scheduling is performed. Therefore, when a large number of applications write multiple data concurrently, the overall data multiple writing efficiency is low. On the other hand, the above scheme cannot realize the rapid modification of connection information. For example, if there are one thousand applications that need to modify the password of a database cluster, each application needs to modify the database cluster password separately, which is inefficient; and the connection information configuration such as the password modified by the one thousand applications can only take effect after the one thousand applications are restarted. Therefore, the configuration takes a long time to take effect, resulting in low business system operation efficiency and user perception. On the other hand, when multiple data are written by direct writing through applications, the same data is written to databases in different data centers for each application. However, the network environment and other conditions of different data centers are different, resulting in different effects and times for applications to write the same data in different data centers, which easily leads to inconsistent data written to the two databases.

[0073] In view of this, an embodiment of the present application provides a data storage method. In this method, a proxy service is added to the active-active data center. The proxy service isolates each application from each database, and through the proxy service and the message queue, asynchronous multi-writing of data of each application in two or more data centers is unified, which helps to improve the overall data multi-writing efficiency. In addition, the proxy service isolates each application from each database, and the data storage requests of each application are uniformly sent to the database through the proxy service, which can reduce the number of connections to the database, thereby reducing the pressure on the database and helping to improve the stability of the database.

[0074] To improve the reliability and high availability of the business system, the data storage method of the embodiment of the present application adopts the active-active mode of the active-active data center. Among them, the business system can be, for example, a call record management system.

[0075] In the active-active data center, both data centers maintain normal operation and can jointly undertake the business to improve the comprehensive business performance and system resource efficiency of the data center. The two data centers back up data for each other. When a single data center is unavailable, there is zero loss of production data, and all production services can be transferred to the other data center.

[0076] The active-active data center can include the following advantages:

[0077] Avoid resource waste. If the active / standby data center mode is used for active-active construction, the resources of the standby data center are in a restricted state for a long time and the resources are not fully utilized. However, the active-active data center can make full use of the resources in the two data centers.

[0078] The active-active data center naturally has disaster tolerance capabilities. If one data center fails, the other data center can take over the production business without affecting the production business and being imperceptible to users.

[0079] Exemplarily, Figure 1 shows a schematic diagram of the architecture of a data storage system of an active-active data center provided by an embodiment of the present application. As Figure 1 shown, the architecture can include a first data center and a second data center, and the first data center and the second data center are communicatively connected. Both the first data center and the second data center can include an application layer, a proxy layer, a component layer, a message queue, and a registration / configuration center. The proxy layer includes a proxy service, and the component layer includes at least one type of database. The application layer includes multiple applications.

[0080] For ease of description, an embodiment of the present application takes the application layer, proxy layer, component layer, message queue, and registration / configuration center of the first data center as an example for description.

[0081] In the embodiment of the present application, the data center can be deployed on the kubernetes (k8s) platform. Among them, the proxy service can be deployed on the k8s platform in the form of a deamonset.

[0082] Among them, the application layer can be used to send a data storage request to the proxy service in the proxy layer.

[0083] The configuration center can be used to configure and save configuration parameters.

[0084] The proxy service in the proxy layer can be used to write the data to be stored carried in the data storage request into the database of the first data center, and asynchronously write the data to be stored into the message queue.

[0085] In the embodiment of the present application, the decorator pattern can be used in the data center, and the proxy service can be added to the data center through the decorator pattern.

[0086] In order to achieve high performance, high throughput, and low latency, the proxy service in the embodiment of the present application adopts the Google Remote Procedure Call (grpc) protocol. The grpc protocol has the following advantages: based on the HTTP / 2 standard protocol, it has better performance and supports stream processing combination; based on the protocol buffers binary protocol, the transmission efficiency is higher; it has a high degree of compatibility with the golang programming language. Among them, the stream processing combination based on the grpc protocol supports the iterator operation of the database.

[0087] In the embodiment of the present application, the proxy service supports inter-process communication (IPC) socket, and the IPC socket can also be called uds, and uds realizes the communication between different processes on the same host. The uds does not need to go through the network protocol stack, does not need to calculate checksums, pack and unpack, maintain sequence numbers and responses, etc., but only copies the application layer data from one process to another process, which helps to improve the data writing efficiency.

[0088] The message queue can be used to temporarily store the data to be stored until the consumer program of the second data center obtains the data to be stored from the message queue and writes the data to be stored into the database of the second data center.

[0089] The database in the component layer can be used to store the data to be stored.

[0090] The following uses specific embodiments to elaborate in detail on the technical solution of this application and how the technical solution of this application solves the above technical problems. These several specific embodiments below can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of this application will be described below in conjunction with the accompanying drawings.

[0091] Figure 2 Flow schematic of a data storage method provided by an embodiment of this application Figure 1 This method can be applied to Figure 1 the proxy service in the proxy layer shown. For ease of description, an embodiment of this application takes the proxy service of the proxy layer of the first data center shown as an example for illustration. As Figure 1 shown, this method may include: Figure 2 shown, this method may include:

[0092] S201. Receive a data storage request sent by the application layer. The data storage request carries the data to be stored and the target database type, and the data storage request is used to instruct the proxy service to write the data to be stored into the database corresponding to the target database type.

[0093] Among them, the data to be stored may vary according to different business systems. The specific content of the data to be stored in the embodiments of this application is not limited, as long as it is data that can be stored in the database. Exemplarily, assuming the business system is a call record management system, the data to be stored may be call record data.

[0094] In the embodiments of this application, the database may include, for example, databases of types such as remote dictionary server (redis), MongoDB, MySQL, etc. The target database type is the database type corresponding to the database that the application program in the application layer expects to use to store the data to be stored.

[0095] Exemplarily, assuming the application program expects to store the data to be stored in MongoDB, the data storage request is used to instruct the proxy service to write the data to be stored into MongoDB.

[0096] In the embodiments of this application, the proxy service writing the data to be stored into the database corresponding to the target database type may be that the proxy service directly writes the data to be stored into the target database, or the proxy service writes the data to be stored into the target database through a message queue.

[0097] S202. Obtain target connection information according to the target database type. The target connection information includes the first address of the database corresponding to the target database type, and the first address is in the first data center, and the data center where the proxy service is located is the first data center.

[0098] In the embodiments of the present application, the connection information may include, for example, the IP address, port number, username, password, connection method, authentication method, polling method, etc. of the database. The connection information such as the IP address, port number, username, password, connection method, authentication method, polling method, etc. corresponding to each database is respectively encapsulated into a connection information group corresponding to each database and uniformly managed to maintain the consistency of the system architecture of the data center and reduce the number of connections to the database.

[0099] In the embodiments of the present application, the first address may include the IP address and port number of the database corresponding to the target database type in the first data center.

[0100] In a possible implementation, the proxy service may pre-obtain the connection information corresponding to each database from the registration / configuration center, and then determine the target connection information from the connection information corresponding to each database according to the target database type. Alternatively, the proxy service may also obtain the target connection information from the connection information of each database preset in the registration / configuration center in real time according to the target database type.

[0101] S203. Write the data to be stored into the first target database corresponding to the first address, and write the data to be stored into the second target database corresponding to the second address in the second data center through the message queue; the second data center is communicatively connected to the first data center.

[0102] In the embodiments of the present application, the database types of the first target database and the second target database are both the target database type. For example, assuming that the application expects to store the data to be stored in a redis database, the first target database is the redis database in the first data center, and the second target database is the redis database in the second data center.

[0103] In the embodiments of the present application, the first address may be the IP address and port number of the first target database, and the second address may be the IP address and port number of the second target database.

[0104] In the embodiments of the present application, the proxy service may configure a configuration item for whether multi-write is supported. For example, when the configuration item is 1, it indicates that the proxy service supports multi-write; when the configuration item is 0, it indicates that the proxy service does not support multi-write. Before the proxy service writes the data to be stored into the second target database corresponding to the second address in the second data center through the message queue, the user may set the configuration item to support multi-write.

[0105] In a possible implementation, the proxy service directly writes the data to be stored into the first target database according to the first address, and asynchronously writes the data to be stored and the target database type into the message queue, and writes the data to be stored into the second target database corresponding to the second address in the second data center through the message queue.

[0106] Similarly, when a third data center is communicatively connected to the first data center and data multi-writing is required, the proxy service can also write data to multiple data centers through a message queue. Among them, the third data center may include at least one data center.

[0107] In the embodiments of the present application, the proxy service and the message queue are used to uniformly perform asynchronous multi-writing of data of each application program in two or more data centers, which helps to improve the overall data multi-writing efficiency. In addition, the proxy service isolates each application program from each database, and the data storage requests of each application program are uniformly sent to the database through the proxy service, which can reduce the number of connections to the database, thereby reducing the pressure on the database and helping to improve the stability of the database.

[0108] In a possible implementation, the method may further include: the proxy service obtains the response duration of the first target database; when the response duration is greater than or equal to a preset duration, the proxy service returns service unavailable information to the application layer and fuses the connection with the application layer; and, the proxy service sends a probe message to the first target database at a preset period to determine the response duration of the first target database. When the response duration is less than the preset duration, the proxy service resumes the connection with the application layer.

[0109] In the embodiments of the present application, the proxy supports configuring a fusing metric, and the fusing metric may be, for example, the response duration of the database.

[0110] Exemplarily, the proxy service can obtain the response duration of the first target database. When the response duration is greater than or equal to a preset duration (i.e., the fusing threshold), the proxy service can return service unavailable information to the application layer and actively fuse the connection with the application layer. At the same time, the proxy service can also regularly send small-traffic probe messages to the first target database at a preset period to determine the response duration of the first target database. When the response duration is less than the preset duration, the proxy service resumes the connection with the application layer, that is, resumes data writing to the first target database. Among them, the service unavailable information is used to notify the application layer that the first database suspends service and cannot perform writing of data to be stored.

[0111] Exemplarily, the fusing metric may further include the number of database exceptions, the exception ratio, the number of slow queries, etc.

[0112] In the embodiments of the present application, when an exception occurs in the first target database, the service for the application layer is stopped to reduce the storage pressure on the first target database.

[0113] Based on the above embodiments, in order to more clearly describe the technical solution of the present application, exemplarily, please refer to Figure 3 , Figure 3The flowchart of a data storage method provided by an embodiment of the present application is shown Figure 2 . This method can be applied to Figure 1 the proxy service in the proxy layer shown. For ease of description, an embodiment of the present application takes the proxy service of the proxy layer of the first data center shown Figure 1 as an example for illustration. As shown Figure 3 , this method may include:

[0114] S301. The application layer obtains the proxy service address.

[0115] Exemplarily, when an application program in the application layer needs to store data, it obtains the address of the pre-configured proxy service from the registration / configuration center.

[0116] S302. The application layer initiates a connection request to the proxy service corresponding to the proxy service address, and receives the connection response returned by the proxy service, and establishes a connection with the proxy service.

[0117] In an embodiment of the present application, the connection between the application layer and the proxy service is a long connection. Within a certain period of time, the application layer can continuously send data storage requests to the proxy service based on this long connection without having to establish a connection with the proxy service again until the long connection is disconnected. This way of establishing a connection once and using it multiple times can reduce the number of connection establishments, thereby reducing the connection establishment time and improving the data writing efficiency.

[0118] S303. The application layer sends a data storage request to the proxy service.

[0119] After the application layer receives the connection response returned by the proxy service, the application layer can send a data storage request to the proxy service.

[0120] S304. The proxy service receives the data storage request sent by the application layer.

[0121] This step is similar to or the same as step S201 above, and will not be elaborated here.

[0122] S305. The proxy service monitors the change of the configuration parameters of the registration / configuration center in the first data center in real time, and correspondingly updates the connection information in the proxy service according to the change of the configuration parameters to obtain the updated connection information; obtains the target connection information from the updated connection information according to the target database type.

[0123] Among them, the configuration parameters may include the connection information corresponding to different database types.

[0124] In a possible implementation, the proxy service can obtain the connection information corresponding to each database type from the registration / configuration center in real time. At the same time, the proxy service can synchronously update the connection information as the configuration parameters configured by the user in the registration / configuration center change.

[0125] Exemplarily, the proxy service can monitor the change of the configuration parameters of the registration / configuration center in the first data center in real time. For example, if it is monitored that the database password has changed, the database password in the connection information in the proxy service is updated in real time to obtain the updated connection information. Further, the proxy service can obtain the target connection information from the updated connection information according to the target database type.

[0126] In the embodiment of the present application, the proxy service can monitor the change of the configuration parameters of the registration / configuration center in the first data center in real time, and can synchronously update the connection information as the configuration parameters configured by the user in the registration / configuration center change. That is, when the configuration parameters of the registration / configuration center change, the proxy service can directly synchronously change the connection information and take effect directly without restarting the application program. Therefore, the configuration effective time is shortened and the user is unaware. At the same time, when the configuration parameters of the registration / configuration center change, compared with the method of modifying the connection information separately in each application program in the prior art, in the embodiment of the present application, only the connection information in the proxy service needs to be modified, realizing the unified and rapid modification of the connection information, and the modification efficiency is relatively high.

[0127] Optionally, before step S305, the method may further include:

[0128] The proxy service determines whether the number of data storage requests received within a preset time window is less than or equal to the number of tokens generated within the preset time window.

[0129] If the proxy service determines that the number of data storage requests received within the preset time window is less than or equal to the number of tokens generated within the preset time window, step S305 is executed; if the proxy service determines that the number of data storage requests received within the preset time window is greater than the number of tokens generated within the preset time window, a message indicating access failure is returned to the application program to cause the application program to reduce the request speed, and a flow limiting error message is sent to the client of the proxy service to cause the client of the proxy service to wait and try after receiving the flow limiting error message to reduce the loss of data to be stored.

[0130] Wherein, the preset time window and the number of tokens generated within the preset time window can be set according to the actual scenario. Exemplarily, the preset time window and the number of tokens generated within the preset time window can be the optimal parameters selected according to the component stress test results.

[0131] In a possible implementation, the proxy service supports rate limiting using the token bucket model to protect the database from crashing. For example, the proxy service can generate tokens at a constant rate, and one token is used for one data storage request. When the proxy service determines that the number of data storage requests received within a preset time window is less than or equal to the number of tokens generated within the preset time window, the proxy service can normally process the data storage requests. When the proxy service determines that the number of data storage requests received within the preset time window is greater than the number of tokens generated within the preset time window, it means that the speed at which the application initiates data storage requests is greater than the token generation speed. After the tokens are exhausted, the proxy service returns a message indicating access failure to the application.

[0132] S306. The proxy service obtains a connection to the first target database corresponding to the first address; writes the data to be stored into the first target database corresponding to the first address, and writes the data to be stored and the target database type into the message queue, so that the consumer program in the second data center writes the data to be stored obtained from the message queue into the second target database corresponding to the second address; the second address is obtained by the consumer program from the registration / configuration center in the second data center according to the target database type, and the consumer program in the second data center has subscribed to the message queue in advance.

[0133] In the embodiments of the present application, the proxy service can maintain a grpc database connection pool, which can be used to cache and reuse database connections. When the grpc database connection pool is initialized, a certain number of database connections will be created and put into the database connection pool. The number of initially created grpc database connections is determined by the set minimum number of connections. At the same time, the maximum number of connections in the grpc database connection pool is set to prevent the proxy service from having an endless connection with the database. When a database connection needs to be established, the proxy service only needs to take out the connection to the first target database from the grpc database connection pool to reduce the time for re-establishing the connection and improve the connection efficiency.

[0134] In a possible implementation, after the proxy service establishes a connection with the first target database, it writes the data to be stored into the first target database and asynchronously writes the data to be stored and the target database type into the message queue. Among them, the asynchronous writing into the message queue has no impact on the process of writing into the first target database.

[0135] After the data to be stored and the target database type are asynchronously written into the message queue, the message queue can return a response message, which is used to indicate whether the write is successful. If the response message indicates a write failure, the proxy service writes the data to be stored and the target database type into the local disk file of the first data center, and after being collected by the file collection module, it is written into the message queue again; if the response message indicates a write success, the message queue can send the data to be stored and the target database type to the consumer program in the second data center. The consumer program in the second data center obtains the second address from the registration / configuration center in the second data center according to the target database type, and writes the data to be stored into the second target database based on the second address.

[0136] Similarly, when there is a third data center communicating with the first data center and data multi-writing is required, the message queue can send the data to be stored and the target database type to the consumer program in the third data center, and the consumer program in the third data center writes the data to be stored into the database in the third data center.

[0137] S307. The proxy service regularly audits the data to be stored in the first database and the second database.

[0138] In the embodiment of the present application, the proxy service can regularly audit the data to be stored in the first database and the second database. If there are differences, corrections and completions are made to ensure the final consistency of the data to be stored in the first database and the data to be stored in the second database.

[0139] In the embodiment of the present application, through the proxy service and the message queue, the asynchronous multi-writing of data of each application program in two or more data centers is unified, which helps to improve the overall data multi-writing efficiency. In addition, the proxy service isolates each application program from each database, and the data storage requests of each application program are uniformly sent to the database through the proxy service, which can reduce the number of connections to the database, thereby reducing the pressure on the database and helping to improve the stability of the database.

[0140] Figure 4 It is a schematic structural diagram of a data storage device provided by an embodiment of the present application. Please refer to Figure 4 , the data storage device 40 includes a receiving module 401, an obtaining module 402, and a writing module 403, where

[0141] The receiving module 401 is configured to receive a data storage request sent by the application layer. The data storage request carries the data to be stored and the target database type, and the data storage request is used to instruct the proxy service to write the data to be stored into the database corresponding to the target database type;

[0142] An obtaining module 402, configured to obtain target connection information according to a target database type, where the target connection information includes a first address of a database corresponding to the target database type, the first address is in a first data center, and the data center where the proxy service is located is the first data center;

[0143] A writing module 403, configured to write data to be stored into a first target database corresponding to the first address, and write the data to be stored into a second target database corresponding to a second address in a second data center through a message queue; the second data center is communicatively connected to the first data center.

[0144] In a possible implementation manner, the obtaining module 402 is specifically configured to:

[0145] Monitor in real time changes in configuration parameters of a registration / configuration center in the first data center, and update the connection information in the proxy service correspondingly according to the changes in the configuration parameters to obtain updated connection information;

[0146] Obtain the target connection information from the updated connection information according to the target database type.

[0147] In a possible implementation manner, the writing module 403 is specifically configured to:

[0148] Obtain a connection between the proxy service and the first target database corresponding to the first address;

[0149] Write the data to be stored into the first target database corresponding to the first address, and write the data to be stored and the target database type into the message queue, so that a consumer program in the second data center writes the data to be stored obtained from the message queue into the second target database corresponding to the second address; the second address is obtained by the consumer program in the second data center from a registration / configuration center according to the target database type, and the consumer program in the second data center has subscribed to the message queue in advance.

[0150] In a possible implementation manner, the apparatus 40 further includes:

[0151] The proxy service receives a connection request sent by an application layer;

[0152] Responds to the connection request and establishes a long connection with the application layer.

[0153] In a possible implementation manner, the apparatus 40 further includes:

[0154] The proxy service determines whether the number of data storage requests received within a preset time window is less than or equal to the number of tokens generated within the preset time window;

[0155] If so, execute the step of writing the data to be stored into the first target database corresponding to the first address.

[0156] In a possible implementation, the apparatus 40 further includes:

[0157] Obtain the response duration of the first target database;

[0158] When the response duration is greater than or equal to the preset duration, return information indicating that the service is unavailable to the application layer and fuse the connection with the application layer; and,

[0159] Send a probe message to the first target database at a preset period to determine the response duration of the first target database. When the response duration is less than the preset duration, resume the connection with the application layer.

[0160] The data storage apparatus provided by the embodiments of the present application can be used to execute the above method embodiments, and its implementation principle and technical effects are similar, which will not be elaborated here in the embodiments of the present application.

[0161] It should be understood that the above apparatus embodiments are merely illustrative, and the apparatus of the present application can also be implemented in other ways. For example, the division of units / modules in the above embodiments is only a logical function division, and there can be other division methods in actual implementation. For example, multiple units, modules or components can be combined, or can be integrated into another system, or some features can be ignored or not executed.

[0162] The embodiments of the present application provide an electronic device, Figure 5 which is a schematic hardware structure diagram of an electronic device provided by the embodiments of the present application.

[0163] As Figure 5 shown, the electronic device 50 includes: a processor 501 and a memory 502; the memory 502 stores computer-executable instructions; the processor 501 executes the computer-executable instructions stored in the memory 502, so that the electronic device 50 executes the above method.

[0164] When the memory 502 is independently provided, the electronic device 50 further includes a bus 503 for connecting the memory 502 and the processor 501.

[0165] Figure 5 The electronic device 50 shown in the embodiment can execute the steps in the above method embodiments, and its implementation principle and beneficial effects are similar, which will not be elaborated here.

[0166] The embodiments of the present application provide a chip. The chip includes a processor, and the processor is used to call a computer program in the memory to execute the technical solutions in the above embodiments. Its implementation principle and technical effects are similar to those of the above related embodiments, which will not be elaborated here.

[0167] The embodiments of the present application also provide a computer-readable storage medium. The computer-readable storage medium stores computer-executable instructions. When the computer-executable instructions are executed by a processor, the above-mentioned method is implemented. The methods described in the above embodiments can be implemented in whole or in part by software, hardware, firmware, or any combination thereof. If implemented in software, the functions can be stored as one or more instructions or codes on a computer-readable medium or transmitted on a computer-readable medium. The computer-readable medium can include a computer storage medium and a communication medium, and can also include any medium that can transfer a computer program from one place to another. The storage medium can be any target medium accessible by a computer.

[0168] In a possible implementation, the computer-readable medium may include RAM, ROM, a compact disc read-only memory (CD-ROM), or other optical disc storage, a magnetic disk storage, or any other magnetic storage device, or any other medium targeted to carry or store the required program code in the form of instructions or data structures and accessible by a computer. Moreover, any connection is properly termed a computer-readable medium. For example, if software is transmitted from a website, server, or other remote source using coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technologies such as infrared, radio, and microwave, then the coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwave are included in the definition of the medium. As used herein, disk and optical disc include optical disc, laser disc, optical disc, digital versatile disc (DVD), floppy disk, and Blu-ray disc, where disks typically reproduce data magnetically, while optical discs reproduce data optically using lasers. The above combinations should also be included within the scope of the computer-readable medium.

[0169] The embodiments of the present application provide a computer program product. The computer program product includes a computer program that, when run, causes a computer to execute the above-mentioned method.

[0170] In the above embodiments, the descriptions of the various embodiments have their respective focuses. For parts not detailed in a certain embodiment, reference can be made to the relevant descriptions of other embodiments. The technical features of the above embodiments can be combined arbitrarily. For the sake of brevity of description, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, it should be considered as within the scope described in this specification.

[0171] Other embodiments of the present application will be readily apparent to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present application, which follow the general principles of the present application and include known common general knowledge or conventional technical means in the technical field not disclosed in the present application. The specification and examples are only to be considered as exemplary, and the true scope and spirit of the present application are pointed out by the appended claims.

[0172] It should be understood that the present application is not limited to the exact structures described above and shown in the drawings, and various modifications and changes can be made without departing from its scope. The scope of the present application is only limited by the appended claims.

Claims

1. A data storage method, characterized in that, Applied to a proxy service, the method includes: Receiving a data storage request sent by the application layer, where the data storage request carries data to be stored and a target database type, and the data storage request is used to instruct the proxy service to write the data to be stored into the database corresponding to the target database type; Obtaining target connection information according to the target database type, where the target connection information includes a first address of the database corresponding to the target database type, and the first address is in a first data center, and the data center where the proxy service is located is the first data center; Writing the data to be stored into a first target database corresponding to the first address, and writing the data to be stored into a second target database corresponding to a second address in a second data center through a message queue; the second data center is communicatively connected to the first data center.

2. The method according to claim 1, wherein The obtaining target connection information according to the target database type includes: Listening in real time for changes in configuration parameters of a registration / configuration center in the first data center, and correspondingly updating connection information in the proxy service according to the changes in the configuration parameters to obtain updated connection information; Obtaining the target connection information from the updated connection information according to the target database type.

3. The method according to claim 2, characterized in that, The writing the data to be stored into a first target database corresponding to the first address, and writing the data to be stored into a second target database corresponding to a second address in a second data center through a message queue includes: Obtaining a connection between the proxy service and a first target database corresponding to the first address; Writing the data to be stored into a first target database corresponding to the first address, and writing the data to be stored and the target database type into the message queue, so that a consumer program in the second data center writes the data to be stored obtained from the message queue into a second target database corresponding to the second address; the second address is obtained by the consumer program according to the target database type from a registration / configuration center in the second data center, and the consumer program in the second data center has subscribed to the message queue in advance.

4. The method according to any one of claims 1 to 3, characterized in that Before receiving the data storage request sent by the application layer, the method further includes: The proxy service receiving a connection request sent by the application layer; Establishing a long connection with the application layer in response to the connection request.

5. The method according to claim 4, characterized in that, Before writing the data to be stored into a first target database corresponding to the first address, the method further includes: The proxy service determining whether the number of data storage requests received within a preset time window is less than or equal to the number of tokens generated within the preset time window; If so, performing the step of writing the data to be stored into a first target database corresponding to the first address.

6. The method according to claim 5, wherein The method further includes: Obtaining a response duration of the first target database; When the response duration is greater than or equal to a preset duration, returning service unavailable information to the application layer and fusing the connection with the application layer; and, Send a detection message to the first target database at a preset period to determine the response duration of the first target database. When the response duration is less than the preset duration, restore the connection with the application layer.

7. A data storage device, characterized in that, Applied to the proxy service, including a receiving module, an obtaining module, and a writing module, where The receiving module is configured to receive a data storage request sent by the application layer. The data storage request carries the data to be stored and the target database type, and the data storage request is used to instruct the proxy service to write the data to be stored into the database corresponding to the target database type; The obtaining module is configured to obtain target connection information according to the target database type. The target connection information includes the first address of the database corresponding to the target database type, and the first address is in the first data center, and the data center where the proxy service is located is the first data center; The writing module is configured to write the data to be stored into the first target database corresponding to the first address, and write the data to be stored into the second target database corresponding to the second address in the second data center through the message queue; the second data center is communicatively connected to the first data center.

8. A data storage system, characterized in that, The data storage system includes a first data center and a second data center that are communicatively connected. Both the first data center and the second data center include an application layer, a proxy layer, a component layer, a message queue, and a registration / configuration center. The proxy layer includes a proxy service, and the component layer includes at least one type of database; The application layer is configured to send a data storage request to the proxy service; the data storage request carries the data to be stored and the target database type; The registration / configuration center is configured to configure and save configuration parameters; The proxy service is configured to execute the method according to any one of claims 1 to 6; The message queue is configured to temporarily store the data to be stored and the target database type until the consumer program in the second data center obtains the data to be stored from the message queue and writes the data to be stored into the second target database corresponding to the second address in the second data center; The database is configured to store the data to be stored.

9. An electronic device, characterized in that, Includes: A processor, and a memory communicatively connected to the processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory to implement the method according to any one of claims 1 to 6.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, and when the computer-executable instructions are executed by a processor, they are used to implement the method according to any one of claims 1 to 6.

Images