Data import method, device, system and equipment for pure memory database and medium
By determining the number of threads and target nodes in a pure in-memory database, the data import process was optimized, resolving the failure issue caused by data exceeding memory capacity and improving operational efficiency and data consistency.
Patent Information
- Application Number
- CN202511691741.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-11-18
- Publication Date
- 2026-03-03
Smart Images
Figure CN121597751A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of data transmission technology, and in particular to a method, apparatus, system, device, and medium for importing data into a pure memory database. Background Technology
[0002] With the development of the digital economy, more and more companies are launching online business services and providing services to users online. This online processing generates various business data. To properly store and manage this data, different companies set up and implemented databases to manage their business data.
[0003] One commonly used type of database is the pure in-memory database (also known as an in-memory database, such as ShuangzhaoDB). In-memory databases are used for reading, writing, and performing CRUD operations to achieve online data management. To quickly import business data from external systems, the operation of importing business data from external systems into a pure in-memory database currently relies heavily on its data structure characteristics. Generally, import tools (such as redis-loader, command-line tools, or scripting languages like Python) can be used. These tools can execute batch insert commands (such as *2 CRLF format) through a client application to import batches of business data into the database.
[0004] However, using the above method to import data into a pure in-memory database presents the following technical problems: the amount of business data imported in different batches varies, often resulting in the data volume exceeding the memory capacity. Once the data volume exceeds the memory capacity, the import process will fail. For example, MySQL's default `max_allowed_packet` parameter limits the size of data packets transmitted in a single batch; exceeding this limit will cause the import to fail. Therefore, multiple batches of import are necessary, which not only increases the number of operations and reduces efficiency, but also increases the risk of errors, leading to inconsistencies in the imported data and ultimately data corruption. Summary of the Invention
[0005] This invention provides a method, apparatus, system, device, and medium for importing data into a pure in-memory database, which can solve the technical problems of existing pure in-memory databases requiring multiple batch processing for data import, resulting in low operational efficiency and easy inconsistency in imported data, making it difficult to guarantee data consistency.
[0006] A first aspect of this invention provides a data import method for a purely in-memory database, the method comprising: Receive business data, determine the number of threads required for the business data, and create the corresponding number of transmission threads; The target node is determined from the pure in-memory database based on the key value of the service data transmitted by each of the transmission threads, and the corresponding channel transmission quantity is determined by the number of threads and the number of the target nodes. The target node is allocated a corresponding number of transmission threads according to the transmission quantity, so that the target node updates the transmission channel according to the transmission quantity and uses the transmission threads to import the service data into the target node through the transmission channel.
[0007] A second aspect of the present invention provides a data import apparatus for a pure in-memory database, the apparatus comprising: The receiving module is used to receive business data, determine the number of threads required for the business data, and create a corresponding number of transmission threads. The determination module is used to determine the target node from a pure memory database based on the key value of the service data transmitted by each transmission thread, and to determine the corresponding channel transmission quantity using the number of threads and the number of nodes of the target node; The import module is used to allocate a corresponding number of transmission threads to the target node according to the transmission quantity, so that the target node updates the transmission channel according to the transmission quantity and uses the transmission threads to import the service data into the target node through the transmission channel.
[0008] A third aspect of this invention provides a data import system for a pure in-memory database, the system comprising: a pure in-memory database and several business terminals; The pure memory database is connected to each of the business terminals. The pure memory database has several data storage node units. Each node unit has an import manager. The import manager is applicable to the data import method of the pure memory database as described above.
[0009] A fourth aspect of the present invention provides an electronic device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the data import method for a pure memory database as described above.
[0010] A fifth aspect of the present invention provides a computer-readable storage medium storing a computer-executable program for causing a computer to execute the data import method for a pure in-memory database as described above.
[0011] Compared to existing technologies, the data import method, apparatus, system, device, and medium for a pure in-memory database provided in this invention offer the following advantages: After receiving business data, this invention can determine the required number of threads and create a corresponding number of transmission threads; based on the key values of the business data transmitted by each transmission thread, it determines the target node from the pure in-memory database, and uses the number of threads and the number of nodes in the target node to determine the number of channel transmissions corresponding to the target node; it allocates a corresponding number of transmission threads to the target node according to the transmission number, so that the target node updates the transmission channel according to the transmission number, and uses the transmission threads to import the business data into the target node through the transmission channel. By matching the target node, this invention avoids the situation where the import process fails due to the amount of business data exceeding the memory capacity, thereby avoiding multiple data transmissions. This not only reduces the number of transmissions and improves data transmission efficiency, but also reduces the probability of data transmission errors and avoids inconsistencies in the imported data caused by multiple transmissions, thus ensuring data consistency; furthermore, by transmitting data to the target node simultaneously through threads, it can further improve data transmission efficiency. Attached Figure Description
[0012] Figure 1 This is a schematic diagram of the structure of a data import system for a pure in-memory database according to an embodiment of the present invention; Figure 2 This is a schematic diagram of the structure of an import manager provided in an embodiment of the present invention; Figure 3 This is a flowchart illustrating a data import method for a pure in-memory database according to an embodiment of the present invention; Figure 4 This is a schematic diagram of the construction of the transmission thread for Kafka streaming data import according to an embodiment of the present invention; Figure 5 This is a schematic diagram of the structure of a data import device for a pure memory database provided in an embodiment of the present invention. Detailed Implementation
[0013] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0014] With the development of the digital economy, more and more companies are launching online business services and providing services to users online. This online processing generates various business data. To properly store and manage this data, different companies set up and implemented databases to manage their business data.
[0015] One commonly used type of database is the pure in-memory database (also known as an in-memory database, such as ShuangzhaoDB). In-memory databases are used for reading, writing, and performing CRUD operations to achieve online data management. To quickly import business data from external systems, the operation of importing business data from external systems into a pure in-memory database currently relies heavily on its data structure characteristics. Generally, import tools (such as redis-loader, command-line tools, or scripting languages like Python) can be used. These tools can execute batch insert commands (such as *2 CRLF format) through a client application to import batches of business data into the database.
[0016] However, using the above method to import data into a pure in-memory database presents the following technical problems: the amount of business data imported in different batches varies, often resulting in the data volume exceeding the memory capacity. Once the data volume exceeds the memory capacity, the import process will fail. For example, MySQL's default `max_allowed_packet` parameter limits the size of data packets transmitted in a single batch; exceeding this limit will cause the import to fail. Therefore, multiple batches of import are necessary, which not only increases the number of operations and reduces efficiency, but also increases the risk of errors, leading to inconsistencies in the imported data and ultimately data corruption.
[0017] To address the aforementioned issues, the following detailed embodiments will be used to describe and explain a data import method, apparatus, device, and medium for a pure in-memory database provided in this application.
[0018] To address the technical issues of existing pure in-memory databases requiring multiple batches for data import, resulting in low operational efficiency and inconsistencies that make it difficult to guarantee data consistency, this paper refers to... Figure 1 The diagram shows a structural schematic of a data import system for a pure in-memory database according to an embodiment of the present invention.
[0019] As an example, the data import system for the pure in-memory database may include: a pure in-memory database and several business terminals, each business terminal corresponding to an external data system; The pure memory database is connected to each of the business terminals, and the pure memory database has several data storage node units.
[0020] Reference Figure 2 Each node unit is equipped with an import manager, which is applicable to the data import method of the pure in-memory database.
[0021] In one embodiment, a pure in-memory database (also known as an in-memory database) is a database that operates directly on data stored in memory. Compared to disk, memory offers data read and write speeds several orders of magnitude faster, and storing data in memory significantly improves application performance compared to accessing it from disk.
[0022] In practice, the external data system can be a system that processes business services. When the external data system generates business data, it can transfer the business data to a pure in-memory database. The pure in-memory database can then import the business data into various node units for storage.
[0023] In one embodiment, the import operation can be performed by the import manager, see reference. Figure 2 The diagram shows a schematic of the structure of an import manager provided in an embodiment of the present invention.
[0024] The process includes the initialization of the import manager, the import processing of streaming business data, and the node change and reconfiguration module. The entire process can be completed by the import manager to shut down and stop the corresponding data import devices.
[0025] During the import process, the import manager can execute a data import method for a pure in-memory database, which includes: Receive business data, determine the number of threads required for the business data, and create the corresponding number of transmission threads; The target node is determined from the pure in-memory database based on the key value of the service data transmitted by each of the transmission threads, and the corresponding channel transmission quantity is determined by the number of threads and the number of the target nodes. The target node is allocated a corresponding number of transmission threads according to the transmission quantity, so that the target node updates the transmission channel according to the transmission quantity and uses the transmission threads to import the service data into the target node through the transmission channel.
[0026] In this embodiment, the present invention provides a data import system for a pure in-memory database. Its advantages are as follows: the present invention can create a corresponding transmission thread and match the corresponding target node according to the business data, and call the transmission thread to transmit the business data to the target node. By matching, the import process can be avoided if the amount of business data exceeds the memory capacity, thus eliminating the need for multiple data transmissions. This not only reduces the number of transmissions and improves the efficiency of data transmission, but also reduces the probability of data transmission errors and avoids inconsistencies in the imported data caused by multiple transmissions, thereby ensuring data consistency.
[0027] Reference Figure 3 The diagram shows a flowchart of a data import method for a pure in-memory database according to an embodiment of the present invention.
[0028] As an example, the data import method for the pure in-memory database may include: S11. Receive service data, determine the number of threads required for the service data, and create the corresponding number of transmission threads.
[0029] In one embodiment, business data transmitted from an external data system can be received, the number of threads required to import this business data can be determined, and then a corresponding number of transmission threads can be created based on the thread data. Creating a corresponding number of transmission threads allows transmission threads to be started simultaneously to perform business data transmission operations, enabling the import of business data at the same time without multiple operations, thereby improving processing efficiency.
[0030] In one embodiment, the pure in-memory database can be shuangzhaoDB. All interactions with shuangzhaoDB exhibit ACID transaction properties, ensuring data consistency and reliability. Furthermore, startup, shutdown, configuration changes, and anomaly recovery are all automatically managed by shuangzhaoDB without manual intervention. The shuangzhaoDB cluster utilizes a ZooKeeper cluster to maintain high reliability and consistency, ensuring normal system recovery even if less than half of the nodes fail.
[0031] To enable importing business data into a pure in-memory database, the following preparatory work can be performed, including: The first step is to configure the relevant import settings for ShuangzhaoDB. Taking importing Kafka data as an example, the configuration requires setting the Kafka broker, subscribed Kafka topics, the group the Kafka client belongs to, and other attributes required by the Kafka client. The key configuration is the stored procedure needed to process the imported data. ShuangzhaoDB provides a stored procedure execution engine that supports the SQL standard to ensure the integrity of database operations. The stored procedures involved in ShuangzhaoDB are typically written in Java, compiled, and resident in memory. ShuangzhaoDB also performs optimization processing based on the system's partitioning.
[0032] The second step is to write the corresponding ShuangzhaoDB stored procedure for the data import requirements. The third step is to create relevant streaming data tables in the ShuangzhaoDB database to store the imported streaming data. These streaming data tables can be described as follows: CREATE STREAM alerts; ( {column-definition} [,...] ); It's important to note that Streams only support INSERT operations and do not store historical data (i.e., only support data in transit). SELECT, UPDATE, or DELETE operations are not supported; otherwise, an error will occur. Based on stream tables, views can be defined in the system to perform real-time aggregation or analysis of streaming data. Stream tables can be combined with stored procedures to complete transaction processing, thereby ensuring ACID consistency of transactions.
[0033] To create corresponding transmission threads for business data, as an example, receiving business data, determining the number of threads required for the business data, and creating the corresponding number of transmission threads may include the following sub-steps: S111. Receive service data and store the service data in the cache area of a pure memory database.
[0034] S112. Determine the number of threads required to transmit the business data based on the preset total number of partitions, wherein the preset total number of partitions is the total number of partitions in the Kafka topic.
[0035] S113. Create a corresponding number of transmission threads in the thread pool based on the number of threads.
[0036] When receiving business data, the import manager can be initialized. The import manager can manage the entire lifecycle of the imported data. Specifically, the import manager can connect to the data importer and control the data importer to perform import operations. In one embodiment, the initialization of the import manager requires the following operations: First, the deployment configuration file is read, relevant configuration information is created for each data importer, and the existence of related stored procedures is checked based on the Catalog directory data. If they exist, the relevant data importer is loaded; otherwise, the relevant data importer is not created.
[0037] Second, based on the importer-related content in the shuanzhaoDB database's deployment.xml file, the relevant configuration packages are loaded, and the relevant processors are started according to the type attribute in the importer to handle the data import. It should be noted that the ShuangzhaoDB database starts with a configuration file that describes the import system's configuration and configures the relevant processors. These processors will handle the import-related processes subsequently.
[0038] Third, load the relevant formatter converter and associate the formatter with the corresponding format.
[0039] Fourth, initialize the channel distributor. The channel distributor is responsible for managing all channel information and interacting with Zookeeper to handle possible future node changes.
[0040] After initialization, the system can receive business data from external data systems. It's important to note that the data transfer process with external data systems is asynchronous; that is, there's no need for the import of business data from external systems and the main database process to wait for each other. After receiving business data, it can be cached. The main thread processes the data according to a schedule, and once processing is complete, it notifies the import thread to clear the relevant data. This avoids blocking the main database transaction flow.
[0041] Furthermore, the import operations for different business data will vary depending on the data importer. Currently supported data importers include general data import and Kafka streaming data import. (See reference...) Figure 4 The diagram illustrates the construction of a transmission thread for Kafka streaming data import according to an embodiment of the present invention.
[0042] For importing streaming business data into Kafka, the corresponding initialization operations can be performed, and the basic process is as follows: First, define the relevant Kafka clients based on Kafka's properties, and start the corresponding clients in separate threads.
[0043] Second, create the initial consumer and query the number of topic partitions - create the first consumer - get the total number of partitions for the Kafka topic.
[0044] Third, calculate the required number of consumers, and use the number of consumers as the number of threads to obtain the total number of threads.
[0045] Fourth, based on the creation of the consumer thread pool, a fixed-size thread pool can be created, and transmission threads can be built within the thread pool. In one embodiment, taking the creation of the remaining Kafka consumers as an example, all consumer tasks can be submitted to the thread pool for execution.
[0046] In practice, you can create a first Kafka consumer, and then create the remaining Kafka consumers within a thread pool. Each consumer corresponds to a specific type of business data. The logical steps are: create the first consumer to obtain the Kafka partition, then determine the number of consumers based on the partition, thereby determining the number of threads, and finally create transmission threads based on the number of threads. After creating the remaining Kafka partitions, all consumers share these thread pools.
[0047] In one embodiment, the initialization operation performed for importing streaming business data into Kafka can be performed by a Kafka client. The Kafka client can connect to an import manager and control the Kafka client to perform the above operations through the import manager.
[0048] Additionally, it's worth noting that when creating the transmission threads, an independent thread pool executor can be created for each data importer, ensuring resource isolation. Furthermore, the implementation employs atomic references and CAS (compare and set) atomic operations. These atomic references and CAS operations guarantee that the thread execution process is not interrupted, thus ensuring thread safety and preventing data transmission interruptions, thereby avoiding inconsistencies caused by transmission interruptions. Moreover, the data importer needs to be started according to the configured running mode. The data importer's startup modes include: Global mode: All mode will launch all importers; Distributed mode: In distributed mode, the importer will register the relevant channels and callback functions with the channel allocator, which will determine the master node in the cluster to handle the data import.
[0049] Additionally, if partition relocation is required, a message is sent to the channel allocator to complete the relevant work. If not, the data is retrieved and processed. Partition relocation may occur due to changes in the nodes of the shuangzhaoDB database cluster, leading to a change in the number of nodes and consequently, changes in data partitions, necessitating partition relocation. Whether partition relocation is required is monitored by Kafka's ZooKeeper.
[0050] In an optional embodiment, determining the number of threads required to transmit the service data based on the preset total number of partitions may include the following sub-steps: S121. If the number of consumers in the Kafka topic is a fixed configuration value, then the number of threads required to transmit the business data shall be determined according to the fixed configuration value.
[0051] S122. If the number of consumers in the Kafka topic is not a fixed configuration value, then obtain the total number of partitions in the Kafka topic, and calculate the number of threads by the ratio of the total number of partitions to the number of hosts in the pure in-memory database.
[0052] Specifically, the number of consumers per node can be calculated based on the number of partitions and nodes, and this number of consumers is used as the number of threads.
[0053] If a fixed number of consumers is pre-configured, the fixed configuration value can be used as the number of threads required for business data.
[0054] If a fixed number of consumers is not pre-configured, the number of consumers per node can be obtained by dividing the number of partitions of the topic by the number of hosts (nodes) of the database, and thus the number of threads can be determined.
[0055] When receiving business data, the system may have initiated multiple requests before execution, and the same business data may have been received in previous operations. To avoid duplication, as an example, the process of receiving business data, determining the number of threads required for the business data, and creating a corresponding number of transmission threads may further include the following sub-steps: S21. After receiving business data and storing the business data in the cache area of a pure memory database, obtain the partition sequence number corresponding to the business data and the historical sequence number from the preset partition tracker, wherein the historical sequence number is the sequence number recorded by the Kafka processor at a previous time node.
[0056] S22. If the partition sequence number and the historical sequence number are different, the historical sequence number is updated synchronously using the partition sequence number.
[0057] Specifically, after receiving business data, the partition sequence number corresponding to the received business data and the historical sequence number can be obtained from the preset partition tracker. The historical sequence number is the sequence number recorded by the Kafka processor when it received business data at a previous time node, that is, the sequence number recorded by the partition tracker when the Kafka processor received business data at the previous time.
[0058] To prevent duplicate data acquisition, the partition sequence number and the historical sequence number must be compared. If they are different, indicating a gap between the partition sequence number and the historical sequence number, the same business data has already been collected. To avoid duplicate acquisition, the historical sequence number can be updated synchronously using the partition sequence number. Specifically, the historical sequence number can replace the partition sequence number, and subsequent business data can be received based on the historical sequence number.
[0059] S12. Determine the target node from the pure memory database based on the key value of the service data transmitted by each transmission thread, and determine the corresponding channel transmission quantity using the number of threads and the number of nodes of the target node.
[0060] In one embodiment, each transmission thread corresponds to one consumer's business data. To avoid the import process failing due to the business data exceeding the memory capacity, the key value of the business data transmitted by each transmission thread can be obtained. Based on the key value, a target node is determined from multiple nodes in a pure in-memory database. The target node is the node that matches the key value. Then, the number of communication transmissions corresponding to each target node can be determined using the number of threads and the number of target nodes.
[0061] In an optional embodiment, determining the target node from a pure in-memory database based on the key value of the service data transmitted by each of the transmission threads, and determining the corresponding channel transmission quantity using the number of threads and the number of target nodes, may include the following sub-steps: S121. Obtain the key value of the business data transmitted by each transmission thread, and perform hash calculation on the key value using a distributed hash algorithm to obtain the data hash value.
[0062] S122. Select nodes that match each data hash value from multiple nodes in the pure memory database to obtain the target node.
[0063] S123. Calculate the ratio of the number of threads to the number of target nodes to obtain the number of channel transmissions corresponding to each target node.
[0064] Specifically, the key-value pairs of the business data transmitted by each transmission thread can be obtained. These key-value pairs are then hashed using a distributed hash algorithm to obtain data hash values. Each data hash value is matched against multiple nodes in a pure in-memory database, and the nodes that match the data hash values are selected to obtain the target node.
[0065] Reference Figure 4 As can be seen, each node unit can correspond to an external data system, and each node unit may have multiple nodes. Through the above processing, one or more target nodes can be selected from multiple nodes.
[0066] Finally, the ratio of the number of threads to the number of target nodes can be calculated to obtain the number of transmissions corresponding to each target node.
[0067] Specifically, based on the load of each node, a distributed hash algorithm can be used to allocate channels for each transmission thread to different nodes. During actual data processing, the key value of the input streaming business data can be used to perform hash calculations using a distributed hash algorithm to determine which node should process the data, thereby creating the relevant channels and executing transmission threads for data transmission.
[0068] In one embodiment, each node unit has one or more, and each node unit is provided with a data importer. In order for the data importer to execute a transmission thread to receive imported data, all data importers can be registered with channels, and each channel corresponds to a transmission thread.
[0069] To ensure load balancing across all target nodes, the number of channels on each target node is typically averaged. This is calculated as: number of channels per target node = number of channels / number of nodes. This gives the transmission volume for each target node. Each target node can handle multiple channels, and each channel corresponds to one data import process and is handled by an independent thread. In other words, each channel corresponds to one thread.
[0070] S13. Allocate a corresponding number of transmission threads to the target node according to the transmission quantity, so that the target node updates the transmission channel according to the transmission quantity, and uses the transmission threads to import the service data into the target node through the transmission channel.
[0071] The number of transmission threads allocated to each target node can be determined based on the number of transmissions, with each transmission thread corresponding to one channel. It's possible that the target node has already completed a transmission and has a corresponding number of channels. To update this, it's necessary to first determine if the target node is a master node. If the target node is a master node, a channel allocation process will be implemented. The basic process includes: first, obtaining the number of channels already registered and the number of transmissions allocated in this instance; then, determining the channels to be added and removed based on the number of registered channels and the number of transmissions allocated, and finally, calculating the number of channels allocated to each node based on the number of nodes.
[0072] After the update is completed, the transmission thread can be executed, which allows business data to be imported into the target node through the transmission channel.
[0073] In one specific operational method, to update the channel list for each target node, the registration information of the data importer can be used to determine whether the current node is the master node of the partition. If the current node is the master node of the partition, it will initiate the channel allocation process. If the current node is not the master node, it will receive the channel allocation information from the master node and update its own channel allocation information. This allows the corresponding number of transmission threads to be allocated to the target nodes according to the transmission volume, ensuring that each channel is evenly distributed among the target nodes.
[0074] After allocation, channel information for each node can be recorded in each Zookeeper node. Specifically, based on Zookeeper, the necessary Zookeeper file directory structure can be initialized, and a monitoring mechanism can be initialized, including: changes in the main channel list; changes in nodes; and changes in cluster operation mode.
[0075] Each non-master node can read the list of channels assigned to it after receiving the corresponding channel information, compare it with the previously assigned channel list, generate channel allocation change information, and send the channel allocation information event through EventBus. The importer receives the notification and starts or stops channel processing accordingly.
[0076] In addition, changes in the number of partitions in a Kafka topic can lead to changes in the number of channels. Each node stores the current channel allocation information. When the channel information changes, the node receives the new channel information and compares it with the old information to generate the changes.
[0077] In an optional embodiment, after importing business data, the data's correctness can be determined. If the data is correct, the execution engine of the shuangzhaoDB database is invoked to execute the relevant stored procedures, saving the data in the corresponding database tables, and submitting the updated partition tracker to the channel coordinator. This is primarily determined based on the preceding sequence number.
[0078] In this embodiment, the present invention provides a data import method for a pure in-memory database. Its advantages are as follows: After receiving business data, the present invention can determine the number of threads required for the business data and create a corresponding number of transmission threads; based on the key values of the business data transmitted by each transmission thread, the target node is determined from the pure in-memory database, and the number of threads and the number of nodes in the target node are used to determine the number of channel transmissions corresponding to the target node; according to the number of transmissions, a corresponding number of transmission threads are allocated to the target node, so that the target node updates the transmission channel according to the number of transmissions, and the business data is imported into the target node through the transmission channel using the transmission threads. By matching the target node, the present invention can avoid the situation where the import process fails due to the amount of business data exceeding the memory capacity, thereby avoiding multiple data transmissions. This not only reduces the number of transmissions and improves the efficiency of data transmission, but also reduces the probability of data transmission errors and avoids inconsistencies in the imported data caused by multiple transmissions, thus ensuring data consistency; moreover, by transmitting data to the target node simultaneously through threads, the data transmission efficiency can be further improved.
[0079] This invention also provides a data import device for a pure in-memory database, see [link to documentation]. Figure 5 The diagram shows a structural schematic of a data import device for a pure memory database according to an embodiment of the present invention.
[0080] As an example, the data import device for the pure in-memory database may include: The receiving module 201 is used to receive business data, determine the number of threads required for the business data, and create a corresponding number of transmission threads; The determining module 202 is used to determine the target node from the pure memory database based on the key value of the service data transmitted by each transmission thread, and to determine the corresponding channel transmission quantity using the number of threads and the number of nodes of the target node; Import module 203 is used to allocate a corresponding number of transmission threads to the target node according to the transmission quantity, so that the target node updates the transmission channel according to the transmission quantity and uses the transmission threads to import the service data into the target node through the transmission channel.
[0081] Optionally, the receiving module is further configured to: Receive business data and store the business data in the cache area of a pure in-memory database; The number of threads required to transmit the business data is determined based on the preset total number of partitions, wherein the preset total number of partitions is the total number of partitions in the Kafka topic; Based on the stated number of threads, create a corresponding number of transmission threads in the thread pool.
[0082] Optionally, the receiving module is further configured to: If the number of consumers in the Kafka topic is a fixed configuration value, then the number of threads required to transmit the business data will be determined according to the fixed configuration value.
[0083] Optionally, the receiving module is further configured to: If the number of consumers for a Kafka topic is not a fixed configuration value, then obtain the total number of partitions for the Kafka topic, and calculate the number of threads by the ratio of the total number of partitions to the number of hosts for a pure in-memory database.
[0084] Optionally, the receiving module is further configured to: Obtain the partition sequence number corresponding to the business data and the historical sequence number from the preset partition tracker, wherein the historical sequence number is the sequence number recorded by the Kafka processor at a previous time node; If the partition serial number and the historical serial number are different, the historical serial number is updated synchronously using the partition serial number.
[0085] Optionally, the determining module is further configured to: Obtain the key value of the business data transmitted by each of the transmission threads, and perform hash calculation on the key value using a distributed hash algorithm to obtain the data hash value; The target node is obtained by filtering nodes that match each data hash value from multiple nodes in a pure in-memory database. Calculate the ratio of the number of threads to the number of target nodes to obtain the number of channel transmissions corresponding to each target node.
[0086] Those skilled in the art will understand that, for ease of description and brevity, the specific working process of the device described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.
[0087] Furthermore, this application also provides an electronic device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the data import method for a pure memory database as described in the above embodiments.
[0088] Furthermore, embodiments of this application also provide a computer-readable storage medium storing a computer-executable program, the computer-executable program being used to cause a computer to execute the data import method for a pure in-memory database as described in the above embodiments.
[0089] In the description of the embodiments of the present invention, it should be noted that the terms "above," "below," etc., indicating the orientation or positional relationship are based on the orientation or positional relationship shown in the accompanying drawings, and are only for the convenience of describing the embodiments of the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of the present invention. When an element such as a layer, region, or substrate is referred to as being "above" or "on top of" another element, it may be directly on the other element, or there may be an intermediate element. Conversely, when an element is referred to as being "directly on" or "above" another element, there is no intermediate element. It should also be understood that when an element is referred to as being "below" or "under" another element, it may be directly below or under the other element, or there may be an intermediate element. Conversely, when an element is referred to as being "directly below" or "under" another element, there is no intermediate element. Unless otherwise explicitly specified and limited, the terms "installation," "connection," and "linking" should be interpreted broadly. For example, they can refer to fixed connections, detachable connections, or integral connections; they can refer to mechanical connections or electrical connections; they can refer to direct connections or indirect connections through an intermediate medium; and they can refer to the internal connection between two components. Those skilled in the art can understand the specific meaning of the above terms in this invention according to the specific circumstances.
[0090] Those skilled in the art will understand that embodiments of this application may also include computer program products. Therefore, this application may take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application may take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0091] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), devices, and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.
[0092] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.
[0093] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.
[0094] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the technical principles of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.
Claims
1. A method for importing data into a purely in-memory database, characterized in that, The method includes: Receive business data, determine the number of threads required for the business data, and create the corresponding number of transmission threads; The target node is determined from the pure in-memory database based on the key value of the service data transmitted by each of the transmission threads, and the corresponding channel transmission quantity is determined by the number of threads and the number of the target nodes. The target node is allocated a corresponding number of transmission threads according to the transmission quantity, so that the target node updates the transmission channel according to the transmission quantity and uses the transmission threads to import the service data into the target node through the transmission channel.
2. The data import method for a pure in-memory database according to claim 1, characterized in that, The process of receiving service data, determining the number of threads required for the service data, and creating a corresponding number of transmission threads includes: Receive business data and store the business data in the cache area of a pure in-memory database; The number of threads required to transmit the business data is determined based on the preset total number of partitions, wherein the preset total number of partitions is the total number of partitions in the Kafka topic; Based on the stated number of threads, create a corresponding number of transmission threads in the thread pool.
3. The data import method for a pure in-memory database according to claim 2, characterized in that, The step of determining the number of threads required to transmit the service data based on the preset total number of partitions includes: If the number of consumers in the Kafka topic is a fixed configuration value, then the number of threads required to transmit the business data will be determined according to the fixed configuration value.
4. The data import method for a pure in-memory database according to claim 2, characterized in that, The step of determining the number of threads required to transmit the service data based on the preset total number of partitions includes: If the number of consumers for a Kafka topic is not a fixed configuration value, then obtain the total number of partitions for the Kafka topic, and calculate the number of threads by the ratio of the total number of partitions to the number of hosts for a pure in-memory database.
5. The data import method for a pure in-memory database according to claim 2, characterized in that, The process of receiving service data, determining the number of threads required for the service data, and creating a corresponding number of transmission threads further includes: After receiving business data and storing the business data in the cache area of a pure memory database, the partition sequence number corresponding to the business data and the historical sequence number are obtained from the preset partition tracker, wherein the historical sequence number is the sequence number recorded by the Kafka processor at a previous time node. If the partition serial number and the historical serial number are different, the historical serial number is updated synchronously using the partition serial number.
6. The data import method for a pure in-memory database according to any one of claims 1-5, characterized in that, The step of determining the target node from a pure in-memory database based on the key value of the service data transmitted by each transmission thread, and determining the corresponding channel transmission quantity using the number of threads and the number of target nodes, includes: Obtain the key value of the business data transmitted by each of the transmission threads, and perform hash calculation on the key value using a distributed hash algorithm to obtain the data hash value; The target node is obtained by filtering nodes that match each data hash value from multiple nodes in a pure in-memory database. Calculate the ratio of the number of threads to the number of target nodes to obtain the number of channel transmissions corresponding to each target node.
7. A data import device for a pure in-memory database, characterized in that, The device includes: The receiving module is used to receive business data, determine the number of threads required for the business data, and create a corresponding number of transmission threads. The determination module is used to determine the target node from a pure memory database based on the key value of the service data transmitted by each transmission thread, and to determine the corresponding channel transmission quantity using the number of threads and the number of nodes of the target node; The import module is used to allocate a corresponding number of transmission threads to the target node according to the transmission quantity, so that the target node updates the transmission channel according to the transmission quantity and uses the transmission threads to import the service data into the target node through the transmission channel.
8. A data import system for a pure in-memory database, characterized in that, The system includes: a pure in-memory database and several business terminals; The pure memory database is connected to each of the business terminals. The pure memory database has several data storage node units. Each node unit has an import manager. The import manager is applicable to the data import method of the pure memory database according to any one of claims 1-6.
9. An electronic device, comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, when the processor executes the computer program, it implements the data import method for a pure memory-based database as described in any one of claims 1-6.
10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer-executable program, which is used to cause a computer to perform the data import method for a pure memory database as described in any one of claims 1-6.