Real-time data processing method and device of distributed database and storage medium

By asynchronously applying for physical partitions in batches and dynamically adjusting the number of partitions, the problem of delay in real-time business processing by distributed databases is solved, data processing efficiency and system reliability are improved, and data processing with high availability and low latency are ensured.

CN118981498APending Publication Date: 2024-11-19JINZHUAN INFORMATION TECHNOLOGY CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411007515.4
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-25
Publication Date
2024-11-19

AI Technical Summary

Technical Problem

In real-time business processing, distributed databases face dynamic management of rapidly generating data, delay in storage application, time-consuming data acquisition, and high latency issues during cluster downtime, affecting the system's high availability and user experience.

Method used

The asynchronous batch application method is used to apply for physical partitions for real-time data in the middleware of distributed databases, and persist after data is successfully written. The number of physical partitions is dynamically adjusted through timing tasks to ensure that the data can be processed and stored in a timely manner.

Benefits of technology

It improves data writing efficiency, reduces user waiting time, enhances the reliability and availability of the system, avoids write delays caused by waiting for partition application, and realizes real-time or near-real-time data processing.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118981498A_ABST
    Figure CN118981498A_ABST
Patent Text Reader

Abstract

The invention relates to a real-time data processing method and device for a distributed database and a storage medium, and the method comprises the steps: obtaining to-be-processed real-time data, applying for a physical partition for the real-time data on middleware of the distributed database in an asynchronous batch application mode, and starting from the successful application of the first batch of physical partitions, applying for the real-time data on the middleware of the distributed database; writing the real-time data into the corresponding physical partitions, and after all the real-time data are successfully written into the corresponding physical partitions, persisting the real-time data into the distributed database from the middleware. Through a mode of asynchronously applying for physical partitions in batches, the advantages of asynchronous processing and layered architecture can be effectively combined, the writing efficiency, the resource utilization rate and the flexibility of the system are improved, and meanwhile, the reliability and consistency of data are enhanced.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of database operation and maintenance, and in particular to a real-time data processing method, device and storage medium for a distributed database. Background Art

[0002] With the high integration of information technology and production and life, a large amount of data is generated in the process of rapid development of society, and distributed database technology has gradually become popular. The use of distributed databases makes data continuously available and has a certain fault tolerance, which can deal with the problem of data loss caused by the downtime of some nodes. Therefore, in industries such as finance and communications, the application of distributed databases is becoming more and more extensive, which can meet the needs of large-scale data storage and processing while maintaining the availability and reliability of the system.

[0003] Although distributed databases have many advantages, they still face many challenges in real-time business processing. Real-time data is generated quickly, and the business needs to dynamically add new tables during the processing. If the data is processed after the entire business is completed, the real-time requirements cannot be met. In addition, there is dynamically added and dynamically deleted data in the actual production environment, which causes the middleware to take longer to apply for storage space, affecting the user experience. At the same time, the increase in data volume will significantly increase the time it takes for system components to obtain data from the middleware, and in the event of some cluster outages, the system must consider the high time consumption of data copy synchronization and physical partition application. Therefore, how to effectively and timely process real-time changing data and minimize user waiting time while ensuring high system availability has become a difficult problem that scientific research and technical personnel urgently need to overcome. Summary of the invention

[0004] The present application provides a real-time data processing method, device and storage medium for a distributed database to solve the problems of dynamic management of data quickly generated by the distributed database in real-time business processing, storage application delay, time-consuming data acquisition and high latency when the cluster is down, so as to ensure high availability of the system and reduce user waiting time.

[0005] In a first aspect, the present application provides a real-time data processing method for a distributed database, the method comprising:

[0006] When the distributed database receives a real-time task, it obtains the real-time data to be processed;

[0007] Applying physical partitions for the real-time data on the middleware of the distributed database in an asynchronous batch application manner, and writing the real-time data into corresponding physical partitions starting from the successful application of the first batch of physical partitions;

[0008] After all the real-time data are successfully written into the corresponding physical partition, the real-time data is persisted from the middleware to the distributed database.

[0009] In a possible implementation, applying for physical partitions for the real-time data on the middleware of the distributed database in an asynchronous batch application manner includes:

[0010] Determine the number of initialized physical partitions and the target number of physical partitions actually required for the real-time data according to the data volume of the real-time data, and apply for physical partitions for the real-time data on the middleware of the distributed database according to the number of initialized physical partitions;

[0011] In the case where the number of the initialized physical partitions is less than the target number of the physical partitions, starting a scheduled task, the scheduled task being used to periodically apply for physical partitions for the real-time data on the middleware of the distributed database according to a set number;

[0012] When the number of successfully applied physical partitions reaches the target, the scheduled task is terminated.

[0013] In a possible implementation manner, determining the number of physical partitions to be initialized according to the data volume of the real-time data includes:

[0014] Comparing the data volume of the real-time data with a preset first threshold and a second threshold respectively, wherein the first threshold is smaller than the second threshold;

[0015] In the case where it is found that the data volume of the real-time data is less than or equal to the first threshold, the first threshold is determined as the number of initialized physical partitions;

[0016] In the case where it is found that the amount of the real-time data is greater than or equal to the second threshold, the second threshold is determined as the number of initialized physical partitions;

[0017] When it is compared that the data volume of the real-time data is greater than the first threshold and less than the second threshold, an initialization ratio value is determined according to the data volume of the real-time data, and the initialization ratio value is multiplied by the target physical partition number to obtain the initialized physical partition number.

[0018] In a possible implementation manner, determining the initialization ratio value according to the data volume of the real-time data includes:

[0019] The data volume of the real-time data is substituted into a pre-fitted initialization ratio value calculation formula to obtain an initialization ratio value.

[0020] In a possible implementation manner, applying for physical partitions for the real-time data on the middleware of the distributed database according to the number of initialized physical partitions includes:

[0021] Determining the number of free physical partitions on the middleware;

[0022] Comparing the number of initialized physical partitions with the number of free physical partitions;

[0023] In the case that it is found that the number of the idle physical partitions is greater than or equal to the number of the initialized physical partitions, selecting a target physical partition from the idle physical partitions according to the number of the initialized physical partitions and allocating it to the real-time data;

[0024] When it is found that the number of free physical partitions is smaller than the number of initialized physical partitions, all the free physical partitions are allocated to the real-time data, and physical partitions are created for the real-time data on the middleware according to the difference between the number of initialized physical partitions and the number of free physical partitions.

[0025] In a possible implementation manner, after writing the real-time data into the corresponding physical partition, the method further includes:

[0026] Establishing a mapping relationship between the real-time data and the physical partition into which it is written;

[0027] The method further comprises:

[0028] After deleting the data in the distributed database, deleting the mapping relationship between the data and the physical partition where the data is written;

[0029] The determining the number of idle physical partitions on the middleware includes:

[0030] The physical partitions on the middleware where no data is written and the physical partitions where data is written but where the mapping relationship does not exist are determined as idle physical partitions, and the number of the idle physical partitions is determined.

[0031] In a possible implementation manner, after writing the real-time data into the corresponding physical partition, the method further includes:

[0032] Establishing a mapping relationship between the real-time data and the physical partition into which it is written;

[0033] After all the real-time data are successfully written into the corresponding physical partitions, the mapping relationship between the real-time data and the physical partitions into which the real-time data are written is persisted in the distributed database.

[0034] In a second aspect, the present application provides a real-time data processing device for a distributed database, the device comprising:

[0035] A real-time data acquisition module is used to acquire the real-time data to be processed when the distributed database receives a real-time task;

[0036] A partition application and data writing module, used to apply for physical partitions for the real-time data on the middleware of the distributed database in an asynchronous batch application manner, and to write the real-time data into the corresponding physical partitions starting from the successful application of the first batch of physical partitions;

[0037] The data persistence module is used to persist the real-time data from the middleware to the distributed database after all the real-time data are successfully written into the corresponding physical partition.

[0038] In a possible implementation, the partition application and data writing module includes:

[0039] A physical partition number initialization unit, used to determine the number of initialized physical partitions and the target number of physical partitions actually required for the real-time data according to the data volume of the real-time data, and apply for physical partitions for the real-time data on the middleware of the distributed database according to the number of initialized physical partitions;

[0040] A scheduled task starting unit, used for starting a scheduled task when the number of the initialized physical partitions is less than the target number of physical partitions, wherein the scheduled task is used for periodically applying for physical partitions for the real-time data on the middleware of the distributed database according to a set number;

[0041] The scheduled task ending unit is used to end the scheduled task when the number of successfully applied physical partitions reaches the target.

[0042] In a possible implementation manner, the physical partition number initialization unit includes:

[0043] A data volume threshold evaluation subunit, used to compare the data volume of the real-time data with a preset first threshold and a second threshold respectively, wherein the first threshold is smaller than the second threshold;

[0044] A small data volume processing subunit is used to determine the first threshold as the number of initialized physical partitions when comparing that the data volume of the real-time data is less than or equal to the first threshold;

[0045] A large data volume processing subunit, configured to determine the second threshold as the number of initialized physical partitions when the data volume of the real-time data is greater than or equal to the second threshold;

[0046] The threshold interval processing subunit is used to determine the initialization ratio value according to the data volume of the real-time data when it is compared that the data volume of the real-time data is greater than the first threshold and less than the second threshold, and multiply the initialization ratio value by the target physical partition number to obtain the initialized physical partition number.

[0047] In a possible implementation manner, the threshold interval processing subunit is specifically configured to:

[0048] The data volume of the real-time data is substituted into a pre-fitted initialization ratio value calculation formula to obtain an initialization ratio value.

[0049] In a possible implementation manner, the physical partition number initialization unit includes:

[0050] An idle physical partition detection subunit, used to determine the number of idle physical partitions on the middleware;

[0051] A physical partition number comparison subunit, used to compare the number of initialized physical partitions with the number of free physical partitions;

[0052] a target physical partition allocation subunit, configured to select a target physical partition from the free physical partitions according to the number of the initialized physical partitions and allocate it to the real-time data if the number of the free physical partitions is greater than or equal to the number of the initialized physical partitions;

[0053] The physical partition creation and allocation subunit is used to allocate all the free physical partitions to the real-time data when it is compared that the number of the free physical partitions is less than the number of the initialized physical partitions, and to create physical partitions for the real-time data on the middleware according to the difference between the number of the initialized physical partitions and the number of the free physical partitions.

[0054] In a possible implementation, the device further includes:

[0055] A mapping relationship establishing module, used to establish a mapping relationship between the real-time data and the physical partition to which it is written;

[0056] The device also includes:

[0057] A mapping relationship deletion module, used to delete the mapping relationship between the data and the physical partition where the data is written after deleting the data in the distributed database;

[0058] The idle physical partition detection subunit is specifically used for:

[0059] The physical partitions on the middleware where no data is written and the physical partitions where data is written but where the mapping relationship does not exist are determined as idle physical partitions, and the number of the idle physical partitions is determined.

[0060] In a possible implementation, the device further includes:

[0061] A mapping relationship establishing unit, used to establish a mapping relationship between the real-time data and the physical partition into which it is written;

[0062] The mapping relationship persistence unit is used to persist the mapping relationship between the real-time data and the physical partition into which it is written into the distributed database after all the real-time data are successfully written into the corresponding physical partition.

[0063] In a third aspect, the present application provides an electronic device, including a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus.

[0064] Memory, used to store computer programs;

[0065] The processor is used to implement the real-time data processing method of the distributed database described in any one of the first aspects above when executing the program stored in the memory.

[0066] In a fourth aspect, the present application further provides a computer storage medium storing computer executable instructions, wherein the computer executable instructions are used to execute the real-time data processing method of a distributed database described in any one of the above items of the present application.

[0067] The above technical solution provided by the embodiment of the present application has the following advantages over the prior art: the method provided by the embodiment of the present application can improve the efficiency of writing data by applying for physical partitions in batches asynchronously, so that the data processing process is not blocked, thereby realizing real-time or near real-time data writing. The application for physical partitions and data writing are performed separately, which can better utilize the resources of the middleware and database and avoid write delays caused by waiting for partition application. After the data is written to the physical partition, it is persisted to ensure that they will not be lost before the data is successfully stored in the warehouse, thereby enhancing the reliability of the system. In summary, by applying for physical partitions in batches asynchronously, the advantages of asynchronous processing and layered architecture can be effectively combined to improve writing efficiency, resource utilization and system flexibility, while enhancing data reliability and consistency. BRIEF DESCRIPTION OF THE DRAWINGS

[0068] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0069] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0070] One or more embodiments are exemplarily described by pictures in the corresponding drawings, and these exemplified descriptions do not constitute limitations on the embodiments. Elements with the same reference numerals in the drawings represent similar elements, and unless otherwise stated, the figures in the drawings do not constitute proportional limitations.

[0071] Figure 1 A flow chart of an embodiment of a real-time data processing method for a distributed database provided in an embodiment of the present application.

[0072] Figure 2 A flow chart of another embodiment of a real-time data processing method for a distributed database provided in an embodiment of the present application.

[0073] Figure 3 A flow chart of an embodiment of a real-time data processing method for a distributed database provided in an embodiment of the present application.

[0074] Figure 4 A flow chart of an embodiment of a real-time data processing method for a distributed database provided in an embodiment of the present application.

[0075] Figure 5 A block diagram of an embodiment of a real-time data processing device for a distributed database provided in an embodiment of the present application.

[0076] Figure 6 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0077] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0078] The disclosure below provides many different embodiments or examples to implement different structures of the present invention. In order to simplify the disclosure of the present invention, the parts and settings of specific examples are described below. Of course, they are only examples, and the purpose is not to limit the present invention. In addition, the present invention can repeat reference numbers and / or letters in different examples. This repetition is for the purpose of simplification and clarity, and does not itself indicate the relationship between the various embodiments and / or settings discussed.

[0079] In order to solve the technical problems in the prior art of dynamic management of distributed databases that quickly generate data in real-time business processing, storage application delays, time-consuming data acquisition, and high latency when clusters are down, the present application provides a real-time data processing device for a distributed database that can realize asynchronous batch application of physical partitions, effectively combine the advantages of asynchronous processing and layered architecture, improve write efficiency, resource utilization, and system flexibility, and enhance data reliability and consistency.

[0080] Figure 1 The following is a flow chart of an embodiment of a real-time data processing method for a distributed database provided in an embodiment of the present application. Figure 1 As shown, the following steps are included:

[0081] Step 101: When the distributed database receives a real-time task, it obtains real-time data to be processed.

[0082] A distributed database is a database system that stores data in multiple physical nodes. These nodes can be located in the same location or in different geographical locations, and are connected and coordinated through a network. The key features of a distributed database include data sharding, redundant backup, and fault tolerance, which allow data to be distributed and shared among multiple nodes, thereby improving the availability, performance, and scalability of the system. This architecture enables the system to handle massive amounts of data, cope with high concurrent access, and ensure that normal services can be maintained when a node fails.

[0083] Real-time tasks refer to tasks that need to be processed and analyzed immediately when data is generated or received. Such tasks usually have extremely high requirements for timeliness and aim to convert data into usable information with the lowest possible latency.

[0084] In the context of distributed databases, when the amount of new real-time data increases dramatically, the system must quickly and effectively partition and process this data to ensure business continuity and rapid response.

[0085] For example, in a high-availability cluster scenario, multiple nodes may have inconsistent data copies due to downtime. At the same time, during the execution of real-time tasks, the system needs to add new data copies in real time to maintain data consistency and availability. In this case, real-time tasks not only include the rapid processing of new data, but also involve the effective management of logical partitions (such as topics) and physical partitions (such as partitions) to ensure that the system can continue to efficiently process business when facing a large amount of new data. When the amount of data increases dramatically, the partitioning process may involve high resource consumption, including computing resources and network bandwidth. This will not only delay the processing speed of data, but may also affect the overall throughput of the system. Therefore, in the face of this data growth challenge, real-time tasks need to have the ability to dynamically adjust the partitioning strategy to maintain the efficiency of the system under high load. In addition, it is necessary to implement a monitoring mechanism to analyze data inflow and system performance in real time so that timely adjustments can be made to ensure that the system can continue to provide services efficiently and meet business needs when processing large-scale data.

[0086] In an embodiment of the present application, the real-time data to be processed includes multiple table data, and one table data corresponds to one physical partition. Real-time data involves a variety of different data types and a large amount of dynamic information. In the management of real-time data, data is usually allocated to multiple tables according to its characteristics and usage requirements. Each table represents a specific data type or business function. For example, the user behavior data table is used to record the user's clicks, browsing and interactive behaviors to help analyze user preferences and behavior patterns; the transaction record table is used to save detailed information on purchase behavior to facilitate financial reporting and auditing. Physical partitioning refers to the actual physical distribution of data on the storage medium. Each data table corresponds to a physical partition, that is, the data of each table is stored in a separate, independent storage area.

[0087] Structured management of real-time data By dividing real-time data into multiple tables and performing physical partitioning, the system can achieve efficient data writing and access. Newly generated data can be immediately written to the corresponding table and updated to the corresponding physical partition, ensuring that the system always reflects the latest information status.

[0088] Step 102: Apply for physical partitions for real-time data on the middleware of the distributed database in an asynchronous batch application manner, and write the real-time data into the corresponding physical partitions starting from the successful application of the first batch of physical partitions.

[0089] Asynchronous means that an operation can continue without waiting for the previous operation to complete. In the context of databases and middleware, asynchronous operations allow programs to initiate database connections or write operations without waiting for the read results immediately, and can perform other tasks during the waiting process. Batch application means that the data processing process is not completed all at once, but is divided into multiple batches, so that the use of resources can be effectively controlled and the efficiency of operations can be improved.

[0090] Middleware is used to provide a service layer between applications and infrastructure in a distributed system. In the process of processing real-time data, middleware is responsible for receiving partition applications from various applications, managing the status of partitions, and ensuring that data can be correctly and efficiently written to the corresponding physical partitions. It provides a unified interface so that developers can easily make asynchronous batch applications and improve the efficiency of data writing.

[0091] In an embodiment of the present application, the middleware will initiate multiple asynchronous application requests to create a new physical partition for real-time data. In this process, the system will package a batch of requests to reduce network delay and improve efficiency. Since the application is asynchronous, multiple requests can be processed at the same time, which can make full use of system resources and reduce the response time of a single request. In each application request, a callback function can be set to perform corresponding operations when the application is successful, so that the system can respond to the application results in a timely manner. In this process, the middleware will continue to monitor the application status of each physical partition until all requests are completed. When a partition application is successful, the corresponding callback logic will be triggered to send a notification to the system or related modules to inform that the new partition is available. After the physical partition application is successful, the middleware will start preparing to write the real-time data to the corresponding physical partition. The middleware will execute the data routing logic to determine which specific physical partition each real-time data should be written to. Once the target partition for writing is determined, the middleware will asynchronously initiate a write operation and send the data in batches to the corresponding physical partition to improve the writing efficiency and reduce the burden of a single write. During the writing process, the middleware needs to maintain control over the transaction to ensure the consistency of the data when writing, such as the transaction rollback mechanism and retry logic, to deal with possible errors or failures.

[0092] During the entire process, the middleware will continue to monitor the status of the written data to ensure that all data is successfully written to the new partition. All partition applications and data writing operations will be recorded in the log for subsequent auditing and analysis.

[0093] Step 103: After all the real-time data is successfully written into the corresponding physical partition, the real-time data is persisted from the middleware to the distributed database.

[0094] Middleware is often used to process data in real time, such as receiving, forwarding, and preliminary processing. However, middleware is not always able to store data persistently. If the middleware fails, data may be lost. Writing data to a distributed database ensures that the data is preserved in the event of a failure. Compared with middleware, distributed databases provide powerful data query and analysis capabilities. Storing real-time data in a database makes it easy to perform complex queries, analysis, and report generation to support business decisions.

[0095] After being processed by the middleware, the data is stored in a distributed database that can efficiently store and manage massive amounts of data. In this process, the data is transformed from a temporary storage state to long-term storage for subsequent query and analysis.

[0096] In an embodiment of the present application, by applying for physical partitions in batches asynchronously, the efficiency of writing data can be improved, so that the data processing process is not blocked, thereby achieving real-time or near real-time data writing. The application of physical partitions and data writing are carried out separately, which can better utilize the resources of the middleware and database and avoid write delays caused by waiting for the application partition. After the data is written to the physical partition, it is persisted to ensure that they will not be lost before the data is successfully stored, thereby enhancing the reliability of the system. In summary, by applying for physical partitions in batches asynchronously, the advantages of asynchronous processing and layered architecture can be effectively combined to improve write efficiency, resource utilization and system flexibility, while enhancing data reliability and consistency.

[0097] Figure 2 A flow chart of another embodiment of a real-time data processing method for a distributed database provided in an embodiment of the present application. Figure 2 The embodiment shown in Figure 1 Based on the embodiment shown, the following steps are also included:

[0098] Step 201: Establish a mapping relationship between real-time data and the physical partition into which it is written.

[0099] The mapping relationship between real-time data and the physical partition to which it is written is a structure or record that indicates to which specific physical partition a piece of real-time data is written. It usually contains the following information:

[0100] Data identifier: A data identifier is a unique identifier for each piece of real-time data, usually automatically assigned by the system when the data is generated. It can be a number, a string, or a combination of numbers, used to uniquely identify a record. Through the data identifier, the system can quickly find and access specific data records, while ensuring that each piece of data is unique throughout the system, avoiding duplicate records, and facilitating the management and maintenance of real-time data, such as update, delete, and other operations.

[0101] Partition identifier: A partition identifier is an identifier of a specific physical partition to which real-time data is written, and is usually closely related to the data storage architecture. Each physical partition may be divided according to different partitioning strategies (such as based on time, region, or data type). The partition identifier is used to indicate where the data is stored on the hardware, which facilitates the system to manage physical storage. When querying, the search scope can be quickly narrowed down to access the specified physical partition instead of searching the entire database, thereby improving the retrieval speed and facilitating the optimization and maintenance of partitions, such as re-dividing or merging partitions after analyzing the data distribution.

[0102] Write timestamp: Write timestamp refers to the specific time when real-time data is written to the physical partition, usually in a time format (such as UTC time). This timestamp is automatically recorded when the data is generated to ensure that the time attribute of the data is accurately reflected. It can be used to sort or group data for time, facilitate time series analysis and trend observation; help the system determine the validity and freshness of data, and facilitate the cleaning and management of expired data; provide time clues for data, facilitate auditing, tracing and compliance inspection, and ensure the transparency of system operations.

[0103] After establishing the mapping relationship between real-time data and the physical partitions to which it is written, the system can track the source and location of each piece of data, quickly locate the physical storage location of specific data, improve the efficiency of data storage and access, ensure fast retrieval and data management, and facilitate the time sorting and compliance audit of data. This helps the system optimize storage, simplify maintenance, and improve overall performance when processing large-scale real-time data.

[0104] Step 202: After all the real-time data is successfully written into the corresponding physical partition, the mapping relationship between the real-time data and the physical partition into which it is written is persisted in the distributed database.

[0105] Persistent mappings ensure that the data storage structure and associated information are still available even in the event of a system crash, restart, or other failure. This allows the system to be restored to its previous state and continue to access data quickly.

[0106] In practical applications, the mapping relationship can be formatted into a structure suitable for storage, such as JSON, CSV, or other data formats supported by the database. Use the database API or query language (such as SQL, NoSQL) to write the formatted mapping data into the distributed database, check whether the same mapping relationship already exists, and prevent duplicate writing; ensure the atomicity of the write operation, that is, either all succeed or all fail, to maintain data consistency. After successful writing, execute a query to verify whether the mapping relationship is stored correctly.

[0107] In this process, the system implements a strategy of creating physical partitions on demand. When new real-time data needs to be stored, the system dynamically creates the corresponding physical partitions instead of pre-generating all possible partitions. For data that needs to be deleted in real time, only the corresponding mapping relationship needs to be updated or deleted without processing the physical partitions. In this way, the system can efficiently determine which data has been deleted, thereby reducing unnecessary resource consumption and processing time.

[0108] In the embodiment of the present application, by establishing a mapping relationship, a one-to-one correspondence between real-time data and its physical storage location is ensured, which helps to maintain data consistency. When restoring or querying data, the corresponding physical partition can be quickly located; the persistent mapping relationship can speed up the data query process, and the system can quickly find the physical partition according to the mapping relationship, thereby improving the efficiency of data retrieval. At the same time, since the mapping relationship is persisted in a distributed database, the integrity and consistency of the data can be restored even in the event of a system crash or failure, thereby enhancing the fault tolerance of the system.

[0109] Figure 3 The following is a flow chart of another embodiment of a real-time data processing method for a distributed database provided in the present application. Figure 3 As shown, the following steps are included:

[0110] Step 301: When the distributed database receives a real-time task, it obtains real-time data to be processed.

[0111] The description of step 301 above can refer to the relevant description in the above embodiment, which will not be repeated here.

[0112] Step 302: Compare the data volume of the real-time data with the preset first threshold and second threshold respectively, the first threshold is less than the second threshold; if the comparison shows that the data volume of the real-time data is less than or equal to the first threshold, execute step 303; if the comparison shows that the data volume of the real-time data is greater than or equal to the second threshold, execute step 304; if the comparison shows that the data volume of the real-time data is greater than the first threshold and less than the second threshold, execute step 305.

[0113] Step 303: Determine the first threshold as the number of initialized physical partitions.

[0114] Step 304: Determine the second threshold as the number of initialized physical partitions.

[0115] Step 305: Determine an initialization ratio value according to the data volume of the real-time data, and multiply the initialization ratio value by the target number of physical partitions to obtain the number of initialized physical partitions.

[0116] For ease of understanding, steps 302 to 305 are described in a unified manner as follows:

[0117] In the embodiment of the present application, the first threshold is the minimum partition quantity requested by the system when performing asynchronous initialization. That is, when the actual data volume is lower than this value (e.g., 100), the system can request the space of the first threshold without affecting the service processing speed. Therefore, setting a minimum threshold can ensure that the system has sufficient resources when starting.

[0118] The second threshold is the maximum number of partitions that the system can accept when performing asynchronous initialization. Exceeding this value (for example, 4000) may cause the system to be overloaded, thus affecting performance. Therefore, setting a maximum threshold is to prevent the system from being inefficient or causing other problems due to too many partitions.

[0119] By properly setting the minimum and maximum thresholds, the system can maintain efficient operation under varying workloads.

[0120] When the amount of real-time data is small and does not exceed the first threshold, it can be considered that the system load is light, so the number of physical partitions can be set to a smaller value to ensure the effective use of system resources and avoid waste of resources caused by excessive partitioning. When the amount of real-time data reaches or exceeds the second threshold, it means that the system load is heavy. In order to improve processing capacity and performance, the number of physical partitions needs to be increased to accommodate more data processing needs.

[0121] In one embodiment, when the data volume of real-time data is greater than a first threshold and less than a second threshold, an initialization ratio value calculation formula can be pre-fitted, and the data volume of the real-time data is substituted into the pre-fitted initialization ratio value calculation formula to obtain the initialization ratio value, and the initialization ratio value is multiplied by the target physical partition number to obtain the initialized physical partition number.

[0122] In another embodiment, a physical partition number algorithm formula may be pre-fitted, and then the data volume of the real-time data is substituted into the pre-fitted physical partition number algorithm formula. The physical partition number algorithm formula is as follows:

[0123] f(x)=αln(x)+βx γ

[0124] Among them, α, β, and γ are values ​​pre-fitted according to the actual business data model, for example, α=1.274, β=0.0135, and γ=0.585; x is the amount of real-time business data, and f(x) is the number of initialized physical partitions.

[0125] By flexibly adjusting the number of partitions when the load changes, it can be adapted to the current data volume to improve the processing efficiency of the system.

[0126] Step 306: Apply for physical partitions for real-time data on the middleware of the distributed database according to the number of initialized physical partitions; and write the real-time data into the corresponding physical partitions starting from the successful application of the first batch of physical partitions.

[0127] After determining the number of physical partitions to be initialized through the above steps, an application request is issued on the middleware according to the number of physical partitions to be initialized. Once the requested physical partition is successfully created or allocated, the system will receive a confirmation response. This confirmation indicates that the new physical partition is ready and can accept data writing. From the moment the confirmation response is issued, the system begins to write real-time data to the newly requested physical partition. This process involves the distribution and scheduling of data to ensure that data can be written to each partition evenly and efficiently. By writing real-time data to multiple physical partitions, the system can achieve parallel processing and significantly improve the throughput of data processing.

[0128] Step 307: when the number of initialized physical partitions is less than the target number of physical partitions, start a scheduled task, which is used to periodically apply for physical partitions for real-time data on the middleware of the distributed database according to the set number.

[0129] Step 308: When the number of successfully applied physical partitions reaches the target, the scheduled task is terminated.

[0130] For ease of understanding, steps 307 to 308 are described in a unified manner as follows:

[0131] The target number of physical partitions refers to the number of physical partitions required in a data storage or data processing system to effectively store and process real-time data. This number is set based on the analysis of storage requirements. By rationally planning the target number of physical partitions, you can ensure that the system can efficiently and flexibly respond to changes in data inflow.

[0132] When the number of initialized physical partitions is less than the set target number of physical partitions, continuing to create physical partitions is not only a supplement to the current data processing capabilities, but also an active adaptation to future business growth. It can ensure the flexibility of data storage, improve system performance, enhance reliability, and optimize the rational use of resources.

[0133] In one embodiment, when the number of initialized physical partitions is less than the set target number of physical partitions, a scheduled task will be started. The main functions of this task include:

[0134] Periodic application: The scheduled task is triggered once every minute, and partitions are applied for on the distributed database middleware according to the set number (for example, 100 physical partitions are applied for each time).

[0135] Multiple tests have shown that the impact of scheduled tasks on business is negligible, so tasks can be run in parallel.

[0136] When the number of successfully applied physical partitions reaches the target number of physical partitions, the scheduled task will end automatically to avoid unnecessary system burden and ensure the smooth operation of other business tasks.

[0137] Step 309: After all the real-time data are successfully written into the corresponding physical partition, the real-time data is persisted from the middleware to the distributed database.

[0138] The description of the above step 309 can refer to the relevant description in the above embodiment, which will not be repeated here.

[0139] In the embodiment of the present application, by comparing the amount of real-time data with a preset threshold, the number of physical partitions can be adjusted dynamically. This adaptive partition management mechanism can optimize resource allocation according to actual data flow changes, and improve the flexibility and coping ability of the system. At the same time, real-time data writing and physical partition application are parallel operations, which can reduce the write delay caused by insufficient partitions. This parallel processing ensures that the system can maintain good performance under high load conditions and respond to real-time task requirements in a timely manner. By setting two thresholds (the first threshold and the second threshold), only necessary physical partitions are applied when the amount of data is small, avoiding the waste of resources caused by excessive partitioning. When the amount of data reaches a certain scale, more partitions are applied. This control mechanism of partition request effectively reduces the consumption of system resources. Physical partitions are applied periodically as needed, and partitions can be adjusted in time according to changes in real-time data, avoiding the problem of unbalanced system load when the amount of data increases dramatically, and improving the load balancing ability of the system. In summary, through the above scheme, the efficiency, flexibility and reliability of real-time data processing of distributed databases can be improved, ensuring that the system can run stably in a variety of business scenarios.

[0140] Figure 4 The following is a flow chart of another embodiment of a real-time data processing method for a distributed database provided in the present application. Figure 4 As shown, the following steps are included:

[0141] Step 401: When the distributed database receives a real-time task, it obtains real-time data to be processed.

[0142] The description of the above step 401 can refer to the relevant description in the above embodiment, which will not be repeated here.

[0143] Step 402: Determine the number of initialized physical partitions and the number of target physical partitions actually required for the real-time data according to the data volume of the real-time data.

[0144] Step 403: determine the physical partitions on the middleware that have not been written with data and the physical partitions that have been written with data but do not have a mapping relationship as idle physical partitions, and determine the number of idle physical partitions.

[0145] Step 404: Compare the number of initialized physical partitions with the number of free physical partitions. If the comparison shows that the number of free physical partitions is greater than or equal to the number of initialized physical partitions, execute step 405; if the comparison shows that the number of free physical partitions is less than the number of initialized physical partitions, execute step 406.

[0146] Step 405: Select a target physical partition from the idle physical partitions according to the number of initialized physical partitions and allocate it to the real-time data.

[0147] Step 406: Allocate all free physical partitions to the real-time data, and create physical partitions for the real-time data on the middleware according to the difference between the number of initialized physical partitions and the number of free physical partitions.

[0148] For ease of understanding, steps 402 to 406 are described in a unified manner as follows:

[0149] An idle physical partition refers to a physical partition in a data storage system or middleware that is not actually used or assigned to any data storage. That is, a partition that has not yet written any data. The partition has not stored any information during its life cycle and is completely idle. In addition, partitions that have been written with data but currently have no mapping relationship with other data or partitions can also be considered idle. These partitions can be used for previous operations, but are no longer used at this stage due to data processing or cleaning.

[0150] Determine the number of initial physical partitions and the target number of physical partitions actually required for real-time data based on the amount of real-time data. In this step, the total amount of real-time data needs to be evaluated first, which involves analyzing the current input stream, data size, processing capacity, etc. The initial number of physical partitions can ensure the number of partitions required to effectively process and store real-time data, while the target number of physical partitions is the possible demand calculated based on the actual demand for real-time data. In other words, the core is to determine how the system divides the storage area through data volume analysis to optimize the subsequent data processing efficiency.

[0151] Physical partitions on the middleware that have not been written to data and physical partitions that have been written to data but do not have a mapping relationship are identified as free physical partitions, and the number of free physical partitions is determined. Here, the middleware refers to the abstraction layer between the application and the underlying operating system or database. Physical partitions that have not been written to data are those that have not been used or stored any data. Physical partitions that have been written to data but do not have a mapping relationship are partitions that should have data but are not actually used for some reason (such as data migration or uninstallation). Through the above operations, all currently available free physical partitions can be found to ensure that there are enough resources to process new real-time data.

[0152] Compare the number of initialized physical partitions with the number of free physical partitions. This step is based on the comparison in the previous two steps to determine whether the currently available resources (free physical partitions) can meet the initially set requirements (the number of initialized physical partitions). The next allocation strategy is determined by whether the number of free resources is sufficient.

[0153] In the case that the number of free physical partitions is greater than or equal to the number of initialized physical partitions, target physical partitions are selected from the free physical partitions according to the number of initialized physical partitions and allocated to the real-time data. If the number of free physical partitions is greater than or equal to the required number of initial physical partitions in the previous step of comparison, the system will directly select a corresponding number of partitions from these free partitions to allocate to the storage of real-time data. This ensures that the system fully utilizes the available physical resources, avoids waste of resources, and improves the efficiency of data processing.

[0154] When it is found that the number of free physical partitions is less than the number of initialized physical partitions, all free physical partitions are allocated to real-time data, and physical partitions are created for real-time data on the middleware according to the difference between the number of initialized physical partitions and the number of free physical partitions. If the number of free physical partitions is less than the number of initialized physical partitions, the system will first allocate all available free partitions to real-time data. In this case, since the available resources are insufficient to meet the demand, new physical partitions need to be created on the middleware according to the difference between the actual number of free partitions and the required number of initialized physical partitions. The system needs to dynamically adjust resource allocation to ensure that real-time data can be effectively stored to prevent data loss or processing delays.

[0155] By establishing a dynamic physical partition management mechanism, storage resources can be flexibly allocated according to the needs of real-time data. This mechanism not only improves the effectiveness of storage, but also ensures that the system can maintain good performance and responsiveness when facing growing data demands.

[0156] Step 407: when the number of initialized physical partitions is less than the target number of physical partitions, start a scheduled task, which is used to periodically apply for physical partitions for real-time data on the middleware of the distributed database according to the set number.

[0157] Step 408: When the number of successfully applied physical partitions reaches the target, the scheduled task is terminated.

[0158] Step 409: Starting from the first batch of physical partition applications being successful, the real-time data is written into the corresponding physical partitions.

[0159] The description of the above steps 407 to 409 can refer to the relevant description in the above embodiment, which will not be repeated here.

[0160] Step 410: Establish a mapping relationship between the real-time data and the physical partition to which it is written. After deleting the data in the distributed database, delete the mapping relationship between the data and the physical partition to which it is written.

[0161] In distributed databases, deleting data is a common operation, especially when the data expires or is no longer needed. Deletion can be logical deletion (marked for deletion but not removed immediately) or physical deletion (actually removing the data from storage). Once the data is deleted, the mapping relationship must be updated in time to reflect this change, that is, remove the record of the deleted data in the mapping table to ensure that the system no longer associates the data with its physical location.

[0162] In the embodiment of the present application, a logical deletion method is adopted, that is, the record is not actually deleted from the data storage, but the mapping relationship or status identifier is updated to indicate that the record has been deleted. By retaining the actual data, it is helpful to temporarily deactivate or restore the data instead of completely losing it, and data rollback or recovery operations can be easily performed, especially in the case of accidental deletion. By only deleting the mapping relationship between real-time data and physical partitions without immediately deleting the actual data, the purpose of flexible data management can be achieved. Not only does it improve the efficiency of the operation, but it also provides convenience for future data recovery and management.

[0163] Step 411: After all the real-time data is successfully written into the corresponding physical partition, the real-time data is persisted from the middleware to the distributed database.

[0164] The description of the above step 411 can refer to the relevant description in the above embodiment, which will not be repeated here.

[0165] In the embodiments of the present application, the number of physical partitions is determined according to the actual demand for real-time data, thereby improving the system's ability to handle high-load real-time data; physical partitions are periodically applied for through scheduled tasks to ensure timely response to changing data demands and enhance the flexibility and response speed of the system; a mapping relationship between real-time data and physical partitions is established to facilitate data management and tracking, and ensure data accuracy and consistency; after all real-time data is successfully written, it is persisted to reduce the risk of data loss and improve the disaster recovery capability of the system; through scheduled tasks and dynamic allocation mechanisms, the distributed database can be expanded accordingly according to the growing data demand, supporting large-scale data processing, effectively combining the dynamic needs of real-time data processing and the resource management of distributed systems, and improving the flexibility, efficiency and data security of the system.

[0166] Figure 5 This is a block diagram of an embodiment of a real-time data processing device for a distributed database provided in an embodiment of the present application. Figure 5 As shown, the device comprises:

[0167] The real-time data acquisition module 51 is used to acquire the real-time data to be processed when the distributed database receives the real-time task;

[0168] The partition application and data writing module 52 is used to apply for physical partitions for the real-time data on the middleware of the distributed database in an asynchronous batch application manner, and to write the real-time data into the corresponding physical partitions starting from the successful application of the first batch of physical partitions;

[0169] The data persistence module 53 is used to persist the real-time data from the middleware to the distributed database after all the real-time data are successfully written into the corresponding physical partition.

[0170] In a possible implementation, the partition application and data writing module 52 includes:

[0171] A physical partition number initialization unit, used to determine the number of initialized physical partitions and the target number of physical partitions actually required for the real-time data according to the data volume of the real-time data, and apply for physical partitions for the real-time data on the middleware of the distributed database according to the number of initialized physical partitions;

[0172] A scheduled task starting unit, used for starting a scheduled task when the number of the initialized physical partitions is less than the target number of physical partitions, wherein the scheduled task is used for periodically applying for physical partitions for the real-time data on the middleware of the distributed database according to a set number;

[0173] The scheduled task ending unit is used to end the scheduled task when the number of successfully applied physical partitions reaches the target.

[0174] In a possible implementation manner, the physical partition number initialization unit includes:

[0175] A data volume threshold evaluation subunit, used to compare the data volume of the real-time data with a preset first threshold and a second threshold respectively, wherein the first threshold is smaller than the second threshold;

[0176] A small data volume processing subunit is used to determine the first threshold as the number of initialized physical partitions when comparing that the data volume of the real-time data is less than or equal to the first threshold;

[0177] A large data volume processing subunit, configured to determine the second threshold as the number of initialized physical partitions when the data volume of the real-time data is greater than or equal to the second threshold;

[0178] The threshold interval processing subunit is used to determine the initialization ratio value according to the data volume of the real-time data when it is compared that the data volume of the real-time data is greater than the first threshold and less than the second threshold, and multiply the initialization ratio value by the target physical partition number to obtain the initialized physical partition number.

[0179] In a possible implementation manner, the threshold interval processing subunit is specifically configured to:

[0180] The data volume of the real-time data is substituted into a pre-fitted initialization ratio value calculation formula to obtain an initialization ratio value.

[0181] In a possible implementation manner, the physical partition number initialization unit includes:

[0182] An idle physical partition detection subunit, used to determine the number of idle physical partitions on the middleware;

[0183] A physical partition number comparison subunit, used to compare the number of initialized physical partitions with the number of free physical partitions;

[0184] a target physical partition allocation subunit, configured to select a target physical partition from the free physical partitions according to the number of the initialized physical partitions and allocate it to the real-time data if the number of the free physical partitions is greater than or equal to the number of the initialized physical partitions;

[0185] The physical partition creation and allocation subunit is used to allocate all the free physical partitions to the real-time data when it is compared that the number of the free physical partitions is less than the number of the initialized physical partitions, and to create physical partitions for the real-time data on the middleware according to the difference between the number of the initialized physical partitions and the number of the free physical partitions.

[0186] In a possible implementation, the device further includes:

[0187] A mapping relationship establishing module, used to establish a mapping relationship between the real-time data and the physical partition to which it is written;

[0188] The device also includes:

[0189] A mapping relationship deletion module, used to delete the mapping relationship between the data and the physical partition where the data is written after deleting the data in the distributed database;

[0190] The idle physical partition detection subunit is specifically used for:

[0191] The physical partitions on the middleware where no data is written and the physical partitions where data is written but where the mapping relationship does not exist are determined as idle physical partitions, and the number of the idle physical partitions is determined.

[0192] In a possible implementation, the device further includes:

[0193] A mapping relationship establishing unit, used to establish a mapping relationship between the real-time data and the physical partition into which it is written;

[0194] The mapping relationship persistence unit is used to persist the mapping relationship between the real-time data and the physical partition into which it is written into the distributed database after all the real-time data are successfully written into the corresponding physical partition.

[0195] like Figure 6 As shown, an embodiment of the present application provides an electronic device, including a processor 111, a communication interface 112, a memory 113 and a communication bus 114, wherein the processor 111, the communication interface 112, and the memory 113 communicate with each other through the communication bus 114.

[0196] Memory 113, used for storing computer programs;

[0197] In one embodiment of the present application, the processor 111 is used to execute the program stored in the memory 113 to implement the real-time data processing method of the distributed database provided by any of the above method embodiments, including:

[0198] When the distributed database receives a real-time task, it obtains the real-time data to be processed;

[0199] Applying physical partitions for the real-time data on the middleware of the distributed database in an asynchronous batch application manner, and writing the real-time data into corresponding physical partitions starting from the successful application of the first batch of physical partitions;

[0200] After all the real-time data are successfully written into the corresponding physical partition, the real-time data is persisted from the middleware to the distributed database.

[0201] An embodiment of the present application also provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of the real-time data processing method of a distributed database provided in any of the aforementioned method embodiments are implemented.

[0202] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0203] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a general hardware platform, and of course, by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the relevant technology can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0204] It should be understood that the terms used herein are only for the purpose of describing specific example embodiments and are not intended to be limiting. Unless the context clearly indicates otherwise, the singular forms "one", "an" and "said" as used herein may also be meant to include plural forms. The terms "include", "comprise", "contain", and "have" are inclusive, and therefore specify the existence of stated features, steps, operations, elements and / or parts, but do not exclude the existence or addition of one or more other features, steps, operations, elements, parts, and / or combinations thereof. The method steps, processes, and operations described herein are not interpreted as necessarily requiring them to be performed in the specific order described or illustrated, unless the execution order is clearly indicated. It should also be understood that additional or alternative steps may be used.

[0205] The foregoing is merely a specific embodiment of the present invention, which enables those skilled in the art to understand or implement the present invention. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features claimed herein.

Claims

1. A real-time data processing method for a distributed database, It is characterized in that The method comprises: When the distributed database receives a real-time task, it obtains the real-time data to be processed; Applying physical partitions for the real-time data on the middleware of the distributed database in an asynchronous batch application manner, and writing the real-time data into corresponding physical partitions starting from the successful application of the first batch of physical partitions; After all the real-time data are successfully written into the corresponding physical partition, the real-time data is persisted from the middleware to the distributed database.

2. The method according to claim 1, It is characterized in that The method of applying for physical partitions for the real-time data on the middleware of the distributed database in an asynchronous batch application manner includes: Determine the number of initialized physical partitions and the target number of physical partitions actually required for the real-time data according to the data volume of the real-time data, and apply for physical partitions for the real-time data on the middleware of the distributed database according to the number of initialized physical partitions; In the case where the number of the initialized physical partitions is less than the target number of the physical partitions, starting a scheduled task, the scheduled task being used to periodically apply for physical partitions for the real-time data on the middleware of the distributed database according to a set number; When the number of successfully applied physical partitions reaches the target, the scheduled task is terminated.

3. The method according to claim 2, It is characterized in that Determining the number of physical partitions to be initialized according to the amount of real-time data includes: Comparing the data volume of the real-time data with a preset first threshold and a second threshold respectively, wherein the first threshold is smaller than the second threshold; In the case where it is found that the data volume of the real-time data is less than or equal to the first threshold, the first threshold is determined as the number of initialized physical partitions; In the case where it is found that the amount of the real-time data is greater than or equal to the second threshold, the second threshold is determined as the number of initialized physical partitions; When it is compared that the data volume of the real-time data is greater than the first threshold and less than the second threshold, an initialization ratio value is determined according to the data volume of the real-time data, and the initialization ratio value is multiplied by the target physical partition number to obtain the initialized physical partition number.

4. The method according to claim 3, It is characterized in that Determining an initialization ratio value according to the data volume of the real-time data includes: The data volume of the real-time data is substituted into a pre-fitted initialization ratio value calculation formula to obtain an initialization ratio value.

5. The method according to claim 2, It is characterized in that The applying for physical partitions for the real-time data on the middleware of the distributed database according to the number of initialized physical partitions includes: Determining the number of free physical partitions on the middleware; Comparing the number of initialized physical partitions with the number of free physical partitions; In the case that it is found that the number of the idle physical partitions is greater than or equal to the number of the initialized physical partitions, selecting a target physical partition from the idle physical partitions according to the number of the initialized physical partitions and allocating it to the real-time data; When it is found that the number of free physical partitions is smaller than the number of initialized physical partitions, all the free physical partitions are allocated to the real-time data, and physical partitions are created for the real-time data on the middleware according to the difference between the number of initialized physical partitions and the number of free physical partitions.

6. The method according to claim 5, It is characterized in that After writing the real-time data into the corresponding physical partition, the method further includes: Establishing a mapping relationship between the real-time data and the physical partition into which it is written; The method further comprises: After deleting the data in the distributed database, deleting the mapping relationship between the data and the physical partition where the data is written; The determining the number of idle physical partitions on the middleware includes: The physical partitions on the middleware where no data is written and the physical partitions where data is written but where the mapping relationship does not exist are determined as idle physical partitions, and the number of the idle physical partitions is determined.

7. The method according to claim 1, It is characterized in that After writing the real-time data into the corresponding physical partition, the method further includes: Establishing a mapping relationship between the real-time data and the physical partition into which it is written; After all the real-time data are successfully written into the corresponding physical partitions, the mapping relationship between the real-time data and the physical partitions into which the real-time data are written is persisted in the distributed database.

8. A real-time data processing device for a distributed database, It is characterized in that The device comprises: A real-time data acquisition module is used to acquire the real-time data to be processed when the distributed database receives a real-time task; A partition application and data writing module, used to apply for physical partitions for the real-time data on the middleware of the distributed database in an asynchronous batch application manner, and to write the real-time data into the corresponding physical partitions starting from the successful application of the first batch of physical partitions; The data persistence module is used to persist the real-time data from the middleware to the distributed database after all the real-time data are successfully written into the corresponding physical partition.

9. An electronic device, It is characterized in that It includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus; Memory, used to store computer programs; The processor is used to implement the real-time data processing method of the distributed database described in any one of claims 1 to 7 when executing the program stored in the memory.

10. A computer storage medium, It is characterized in that Computer executable instructions are stored, and the computer executable instructions are used to execute the real-time data processing method of the distributed database described in any one of claims 1 to 7 of the present application.