Wide table data synchronization method, related device and computer program product

Abstract

The invention aims to provide a wide table data synchronization method, electronic equipment, a storage medium and a computer program product, which are used for reducing delay in wide table data updating synchronization. When a piece of original data is obtained, the data is not synchronized to the wide table immediately, the piece of original data is stored, the value of the corresponding counter is increased, and when the value of the corresponding counter is equal to the preset threshold value, all the stored original data are synchronized to the wide table together. Compared with the mode that the wide table is operated once when one piece of original data is obtained in the prior art, the operation frequency of the wide table is reduced, and the system can have more system resources for completing the actual service request. On the whole, as the database can complete the actual service request more quickly, the generated wide table data synchronization request can be executed by the distributed system resources more quickly, and the average delay of the wide table data synchronization operation is reduced.

Description

Technical Field

[0001] This application relates to the field of database technology, specifically to a wide table data synchronization method, related equipment, and computer program products. Background Technology

[0002] Literally, a wide table is a database table with a large number of fields (columns). Wide tables are typically large tables obtained by merging and combining various business-related data tables through join fields. Using wide tables can reduce the number of cross-table join queries and improve data retrieval efficiency.

[0003] Wide tables need to aggregate large amounts of data. A common way to update wide table data is to first classify and store the data generated by the business in different sub-tables, and then aggregate and update the sub-table data into the wide table.

[0004] However, for a given business operation, when new data is generated, it's necessary to first update the sub-tables with the new data, and then synchronize the new data from each sub-table to the wide table. Secondly, in practical applications, the database also needs to handle a large number of data query requests. Even if it's determined that new data synchronization is required, it's not guaranteed that there will always be sufficient system resources to perform the operation of synchronizing new data to the wide table. Therefore, the update and synchronization of data in the wide table often has a certain delay compared to the actual implementation. Summary of the Invention

[0005] In view of this, this application aims to provide a wide table data synchronization method, electronic device, storage medium, and computer program product to reduce the latency in wide table data update synchronization.

[0006] In a first aspect, embodiments of this application provide a method for synchronizing wide table data, including:

[0007] Acquire the raw data and the operation type of the raw data;

[0008] Add the original data to the data to be synchronized corresponding to the operation type, and increase the value of the counter corresponding to the operation type by a preset value;

[0009] When the value of the counter corresponding to the operation type is equal to the preset threshold, the data to be synchronized corresponding to the operation type is synchronized to the wide table.

[0010] Optionally, the operation type of the original data includes data update;

[0011] When the operation type of the original data is data update, the step of adding the original data to the data to be synchronized corresponding to the operation type and increasing the value of the counter corresponding to the operation type by a preset value includes:

[0012] The original data is added to the data to be synchronized corresponding to the data update, and the value of the counter corresponding to the data update is incremented by one.

[0013] Optionally, the operation type of the original data includes data insertion;

[0014] When the operation type of the original data is data insertion, the step of adding the original data to the data to be synchronized corresponding to the operation type and increasing the value of the counter corresponding to the operation type by a preset value includes:

[0015] The original data is added to the data to be synchronized corresponding to the data insertion, and the value of the counter corresponding to the data insertion is incremented by one.

[0016] When the operation type of the original data is data insertion, the step of synchronizing the data to be synchronized corresponding to the operation type to the wide table when the value of the counter corresponding to the operation type is equal to a preset threshold includes:

[0017] When the value of the counter corresponding to the data insertion is equal to the number of sub-tables, the data to be synchronized corresponding to the operation type is synchronized to the wide table. The data in the wide table is obtained from the data synchronization of the sub-tables.

[0018] Optionally, the operation type of the original data includes data insertion;

[0019] The counters corresponding to the data insertion include counters corresponding to the sub-tables, and the data in the wide table is obtained from the sub-tables through data synchronization.

[0020] When the operation type of the original data is data insertion, the step of adding the original data to the data to be synchronized corresponding to the operation type and increasing the value of the counter corresponding to the operation type by a preset value includes:

[0021] Add the original data to the data to be synchronized corresponding to the data insertion;

[0022] Determine the source sub-table of the original data, select the counter corresponding to the source sub-table from the counters corresponding to the sub-table, and increment the value of the counter corresponding to the source sub-table by one;

[0023] When the operation type of the original data is data insertion, the step of synchronizing the data to be synchronized corresponding to the operation type to the wide table when the value of the counter corresponding to the operation type is equal to a preset threshold includes:

[0024] When the values ​​in the counters corresponding to all the sub-tables are equal to the preset sub-table threshold, the data to be synchronized corresponding to the operation type is synchronized to the wide table.

[0025] Optionally, the wide table is stored in a secondary database;

[0026] The types of operations for acquiring raw data and acquiring raw data include:

[0027] Obtain cached data and verify the cached data; use the cached data that passes the verification as the original data and store it in the first-level database.

[0028] The original data and the operation type of the original data are obtained from the primary database.

[0029] Optionally, before obtaining the cached data and validating the cached data, the method further includes:

[0030] Data from a relational database is retrieved and stored as cached data in the message middleware.

[0031] Optionally, the primary database is an HBase database;

[0032] The secondary database is the StarRocks database.

[0033] Secondly, embodiments of this application provide an electronic device, including a memory and a processor;

[0034] The memory is connected to the processor and is used to store programs;

[0035] The processor is used to implement the wide table data synchronization method as described in the first aspect by running a program in the memory.

[0036] Thirdly, embodiments of this application provide a storage medium storing a computer program, which, when executed by a processor, implements the wide table data synchronization method as described in the first aspect.

[0037] Fourthly, embodiments of this application provide a computer program product, including computer program instructions, which, when executed by a processor, cause the processor to perform the wide table data synchronization method as described in the first aspect.

[0038] This application provides a method for synchronizing wide table data. When a piece of raw data is obtained, it is not immediately synchronized to the wide table. Instead, the raw data is stored and a corresponding counter is incremented. When the counter value equals a preset threshold, all stored raw data is synchronized to the wide table. Compared to the existing method of performing an operation on the wide table every time a piece of raw data is obtained, this reduces the frequency of operations on the wide table, allowing the system to allocate more system resources to complete actual business requests. Overall, because the database can complete actual business requests faster, the resulting wide table data synchronization requests can be allocated system resources for execution more quickly, reducing the average latency of wide table data synchronization operations. Attached Figure Description

[0039] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of this application. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0040] Figure 1 A flowchart illustrating a wide table data synchronization method provided in an embodiment of this application;

[0041] Figure 2 This is a flowchart illustrating the specific implementation of the wide table data synchronization method described in this application, provided for an embodiment of the present application.

[0042] Figure 3 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0043] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0044] The first embodiment of this application provides a method for synchronizing wide table data, such as... Figure 1 As shown, the method may include the following steps:

[0045] Step 101: Obtain the raw data and the operation type of the raw data.

[0046] The original data comes from multiple data sources (usually from multiple sub-tables) and needs to be aggregated and synchronized to the same wide table.

[0047] Currently, operations on data in a database can be summarized as "CRUD" (Create, Read, Update, Query), namely, data insertion (adding new data to the database), data deletion, data update, and data query. Since the primary purpose of wide tables is to reduce resource-intensive and inefficient multi-table join operations and provide data query services that can be completed using a single wide table, the methods described in this embodiment are mainly applied to data insertion, data deletion, and data update, and generally not to data query operations, to avoid affecting the wide table's normal provision of data query services.

[0048] When the operation type is data update or data insertion, the original data is generally the data to be updated or inserted into the wide table. When the operation type is data deletion, the original data can be determined based on the deletion method. For example, if the source of the deletion request does not know the location of the data to be deleted in the wide table, the original data can be the data to be deleted itself. By searching the original data in the wide table, the data to be deleted can be found. If the source of the deletion request has determined the location of the data to be deleted in the wide table, the specific content of the original data can be data indicating the location of the data to be deleted. The data to be deleted can be found in the wide table by directly addressing the location.

[0049] Step 102: Add the original data to the data to be synchronized corresponding to the operation type, and increase the value of the counter corresponding to the operation type by a preset value.

[0050] This embodiment proposes that when raw data is acquired, it is not immediately synchronized to the wide table. Instead, the raw data is stored. Specifically, it is stored in the data to be synchronized corresponding to the operation type. For example, data update, data insertion, and data deletion each have corresponding data to be synchronized. When a piece of raw data needs to be inserted into the wide table, it is first stored in the data to be synchronized corresponding to the data insertion operation type. Then, a preset value is added to the counter corresponding to the operation type. The preset value can be set according to the actual situation. For example, if there is only one piece of raw data as the operation unit at the same time, the preset value is one; if there are three threads that can process three data synchronization requests including raw data in parallel at one time, the preset value is set to three.

[0051] Step 103: When the value of the counter corresponding to the operation type is equal to the preset threshold, synchronize the data to be synchronized corresponding to the operation type to the wide table.

[0052] In step 102, the acquired raw data has been added to the data to be synchronized. When the value of the corresponding counter equals the preset threshold, it is considered that a certain amount of raw data has accumulated in the data to be synchronized, and the data to be synchronized is synchronized to the wide table. For example, if the preset threshold for the data update operation type is set to five, whenever five pieces of raw data that need to be updated to the wide table are added to the data to be synchronized, the data to be synchronized is synchronized to the wide table.

[0053] After synchronizing data to a wide table, a counter for that operation type is typically reset. Alternatively, depending on the needs, the counter can be incremented based on the previous synchronization to count the number of synchronizations. For example, setting a preset threshold of five for the data update operation type allows you to easily determine that two data synchronization operations have been performed when the corresponding counter reaches ten. However, if using this method, it's crucial to set a sufficient upper limit for the counter to prevent overflow.

[0054] In existing technologies, the common practice is to perform an operation on the wide table immediately upon acquiring a single piece of raw data. In high-concurrency, high-traffic application environments, this method significantly increases the number of accesses to the wide table, straining system resources that provide data query services to the wide table, reducing database performance, and even causing system problems such as deadlocks. The method provided in this embodiment reduces the frequency of operations on the wide table by packaging multiple pieces of raw data into a single piece of data to be processed and synchronizing them uniformly. Although the waiting time for synchronizing a single piece of raw data into the wide table may increase, from a macro perspective, because the frequency of data synchronization requests operating on the wide table is reduced, the system can allocate more resources to complete actual business requests. This improves the overall performance of the database containing the wide table, allowing the system to complete actual business requests faster and thus freeing up sufficient system resources for wide table data synchronization requests more quickly. Therefore, for all data synchronization requests, the average latency of data synchronization to the wide table is actually reduced.

[0055] The second embodiment of this application further specifies the wide table data synchronization method in the first embodiment in a more detailed and specific way. Some or all of the technical features in the second embodiment can be combined with or replaced by the first embodiment, either individually or in combination, to obtain more feasible wide table data synchronization methods.

[0056] The wide table data synchronization method in the second embodiment of this application is described in detail below:

[0057] Optionally, the operation type of the original data includes data update; when the operation type of the original data is data update, the original data is added to the data to be synchronized corresponding to the operation type, and the value of the counter corresponding to the operation type is increased by a preset value, including: adding the original data to the data to be synchronized corresponding to the data update, and incrementing the value of the counter corresponding to the data update by one.

[0058] This embodiment defines the specific implementation method for wide table data synchronization when the operation type is data update. When the operation type is data update, data update requests, including original data, are sent and received one by one. Each data update request requests to update a specified portion of data in the wide table. Therefore, a preset value of one can be set, and each time an original data is received and added to the data to be synchronized, the counter of the data to be synchronized corresponding to the operation type of this original data is incremented by one.

[0059] The data update operation updates the existing data in the wide table. Since the wide table already has the old data to be replaced as a basis, no further judgment is needed. Therefore, this embodiment proposes a relatively simple and easy-to-implement implementation logic for the data update operation type, which is convenient for subsequent maintenance and expansion.

[0060] Optionally, the operation type of the original data includes data insertion; when the operation type of the original data is data insertion, the original data is added to the data to be synchronized corresponding to the operation type, and the value of the counter corresponding to the operation type is increased by a preset value, including: adding the original data to the data to be synchronized corresponding to data insertion, and incrementing the value of the counter corresponding to data insertion by one; when the operation type of the original data is data insertion, when the value of the counter corresponding to the operation type is equal to a preset threshold, the data to be synchronized corresponding to the operation type is synchronized to the wide table, including: when the value of the counter corresponding to data insertion is equal to the number of sub-tables, the data to be synchronized corresponding to the operation type is synchronized to the wide table, and the data in the wide table is obtained from the data synchronization of the sub-tables.

[0061] When the operation type is data insertion, the process of synchronizing the original data to the wide table is relatively complex. Specifically, the time at which the data to be updated was generated may vary. For example, in actual business scenarios, a customer requests to create an order, order number data is created in the order number sub-table, and a request is immediately made to synchronize the order to the wide table. However, the customer continues operating, and after a period of time, completes the order information, such as the address. Once the order information is complete, it is stored in the order information sub-table, and a request is made to synchronize the order information to the wide table.

[0062] For the above example, this embodiment proposes the following approach: Data synchronization requests initiated by the sub-table corresponding to the order number are not synchronized initially. Instead, all data is inserted into the wide table only after all sub-tables constituting the wide table have initiated data synchronization requests. In the above example, this specifically means that when a data synchronization request including order information is received, all data is then inserted into the wide table.

[0063] In practice, a preset threshold is set as the number of sub-tables. For example, if the wide table is aggregated from data from three sub-tables, then data synchronization will only be performed when the value of the counter is three, indicating that the data from the three sub-tables has been received.

[0064] Furthermore, considering that data synchronization requests from various sub-tables are asynchronous, multiple counters can be set up for the same operation type, using certain data as keys to distinguish between multiple counters. Data synchronization requests with the same key value can be merged and counted. For example, using the order number as the key, when a data synchronization request with the same order number is received, the counter corresponding to that order number is incremented by one. The above method can process data synchronization requests for multiple order numbers in parallel.

[0065] This embodiment proposes a method for data insertion operations that modifies the existing method of performing a data query in the wide table to determine whether the insertion conditions are met whenever a data update request from a sub-table is received. Instead, it increments a counter every time a data update request from a sub-table is received. This avoids the problem of multiple data queries consuming system resources during wide table data insertion due to data originating from multiple sub-tables. It further reduces the access frequency to the wide table.

[0066] Optionally, the operation type of the original data includes data insertion; the counter corresponding to data insertion includes the counter corresponding to the sub-table, and the data in the wide table is obtained from the sub-table through data synchronization; when the operation type of the original data is data insertion, the original data is added to the data to be synchronized corresponding to the operation type, and the value of the counter corresponding to the operation type is increased by a preset value, including: adding the original data to the data to be synchronized corresponding to data insertion; determining the source sub-table of the original data, selecting the counter corresponding to the source sub-table from the counters corresponding to the sub-table, and incrementing the value of the counter corresponding to the source sub-table; when the operation type of the original data is data insertion, when the value of the counter corresponding to the operation type is equal to the preset threshold, the data to be synchronized corresponding to the operation type is synchronized to the wide table, including: when the values ​​in the counters corresponding to all sub-tables are equal to the preset sub-table threshold, the data to be synchronized corresponding to the operation type is synchronized to the wide table.

[0067] This embodiment proposes an alternative processing method for data insertion operations. Specifically, considering that while determining the counter value equals the number of sub-tables is convenient, it is not necessarily accurate and may result in erroneous increments, this embodiment proposes using the counter as a flag bit. The counter corresponding to data insertion is further divided into counters corresponding to each sub-table. Whenever raw data from a sub-table is obtained, the value of the counter corresponding to that sub-table is incremented by one (in effect of marking that data from that sub-table has been received). When the counters corresponding to each sub-table are equal to a preset sub-table threshold (generally set to one), it indicates that raw data from all sub-tables has been received, and the data can be synchronized to the wide table.

[0068] This embodiment proposes an alternative wide-table data synchronization method for data insertion operations. By refining the counter to correspond to each sub-table, it can accurately determine which sub-table's data has been received and which sub-table's data has not been received, thereby improving the accuracy of data synchronization.

[0069] Optionally, the wide table is stored in the secondary database; the operation types for obtaining raw data include: obtaining cached data and validating the cached data, storing the cached data that passes the validation as raw data in the primary database; and the operation types for obtaining raw data from the primary database.

[0070] This embodiment proposes a multi-level database structure. Cache data is retrieved and validated. Validated cache data is used as the original data and stored in the first-level database. Here, validation refers to business logic validation, verifying the validity of the data, such as the existence of the order number and the normality of associated data. If a validation error occurs, abnormal data is identified and recorded in the abnormal backup, terminating subsequent processing.

[0071] Since data synchronization is typically asynchronous, existing wide table data synchronization techniques can lead to data consistency issues. For example, after an order number and order address are modified, the order number update is synchronized into the wide table first. Then, the database provides a data query service before receiving the request to synchronize the order address update into the wide table. This intermediate data query service will result in inconsistencies in the data provided, with inconsistent and mismatched data retrieval times. Therefore, this embodiment proposes a two-tier database structure. The first-tier database stores verified data first, and the data in the first-tier database is then synchronized into the wide table after steps such as adding data to be synchronized and checking counter values. The verification and buffering in the first-tier database ensures no data loss and maintains the consistency of the wide table data.

[0072] Optionally, before obtaining and validating the cached data, the process further includes: obtaining data from a relational database and storing the data from the relational database as cached data in the message middleware.

[0073] This embodiment proposes recording data from a relational database as cached data in a message middleware. This allows for peak smoothing through the message middleware. When the upstream data volume is too large, insufficient downstream system resources and delayed processing can cause backlogs. By caching data through the message middleware, a large backlog of requests can be avoided in the downstream system, allowing the downstream to process data at a relatively constant speed regardless of the amount of data generated upstream.

[0074] Optionally, the primary database is an HBase database; the secondary database is a StarRocks database.

[0075] This embodiment proposes that an HBase database with high scalability, high availability, and low latency can be selected as the primary database. This database, as an intermediate layer, can provide stable data buffering and verification services to ensure data consistency.

[0076] You can choose StarRocks, a high-performance and scalable analytical database, as a secondary database. This database supports complex SQL queries and is suitable for large-scale data analysis scenarios. It can quickly query and analyze massive amounts of data to complete tasks such as multidimensional analysis, data mining, and report generation.

[0077] like Figure 2 The diagram shows a flowchart illustrating the implementation of the wide table data synchronization method proposed in this embodiment. Starting from step 201, step 202 is executed, where data from the relational database is written to a message middleware for peak smoothing and used as cached data. Then, step 203 is executed to verify the cached data and write it to the secondary database—the HBase database. Step 204 continues, determining the operation type. If the operation type is data insertion, step 205 is executed; if the operation type is data update, step 207 is executed. In step 205, if the counter value is not equal to the number of sub-tables, the process returns to step 201; if the counter value is equal to the number of sub-tables, step 206 is executed, synchronizing the data to be synchronized to the wide table. In step 207, if the counter value is not equal to a preset threshold, the process returns to step 201; if the counter value is equal to the preset threshold, step 208 is executed, synchronizing the data to be synchronized to the wide table.

[0078] The third embodiment of this application also proposes an electronic device, see [link to relevant documentation] Figure 3 As shown, the device includes:

[0079] Memory 300 and processor 310;

[0080] The memory 300 is connected to the processor 310 and is used to store programs;

[0081] The processor 310 is used to implement the variable mounting method disclosed in any of the above embodiments by running the program stored in the memory 300.

[0082] Specifically, the aforementioned electronic device may also include: a bus, a communication interface 320, an input device 330, and an output device 340.

[0083] The processor 310, memory 300, communication interface 320, input device 330, and output device 340 are interconnected via a bus. Among them:

[0084] A bus can include a pathway for transmitting information between various components of a computer system.

[0085] The processor 310 can be a general-purpose processor, such as a general-purpose central processing unit (CPU), a microprocessor, etc., or an application-specific integrated circuit (ASIC), or one or more integrated circuits used to control the execution of the program of the present invention. It can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), an off-the-shelf programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0086] Processor 310 may include a main processor, as well as a baseband chip, modem, etc.

[0087] The memory 300 stores a program that executes the technical solution of this invention, and may also store an operating system and other key business functions. Specifically, the program may include program code, which includes computer operation instructions. More specifically, the memory 300 may include read-only memory (ROM), other types of static storage devices capable of storing static information and instructions, random access memory (RAM), other types of dynamic storage devices capable of storing information and instructions, disk storage, flash memory, etc.

[0088] Input device 330 may include a device for receiving user input data and information, such as a keyboard, mouse, camera, scanner, light pen, voice input device, touch screen, pedometer, or gravity sensor.

[0089] Output device 340 may include devices that allow information to be output to a user, such as a display screen, printer, speaker, etc.

[0090] The communication interface 320 may include a device that uses any transceiver to communicate with other devices or communication networks, such as Ethernet, Radio Access Network (RAN), Wireless Local Area Network (WLAN), etc.

[0091] The processor 310 executes the program stored in the memory 300 and calls other devices, which can be used to implement the various steps of any of the variable mounting methods provided in the above embodiments of this application.

[0092] The fourth embodiment of this application also provides a storage medium on which a computer program is stored. The computer program is executed by a processor using the steps of the variable mounting method described in the "Exemplary Method" section above, specifically implementing the following steps:

[0093] Step 101: Obtain the raw data and the operation type of the raw data.

[0094] Step 102: Add the original data to the data to be synchronized corresponding to the operation type, and increase the value of the counter corresponding to the operation type by a preset value.

[0095] Step 103: When the value of the counter corresponding to the operation type is equal to the preset threshold, synchronize the data to be synchronized corresponding to the operation type to the wide table.

[0096] The fifth embodiment of this application provides a computer program product including computer program instructions that, when executed by a processor, cause the processor to perform the steps of the variable mounting methods according to various embodiments of this application described in the "Exemplary Methods" section above.

[0097] The computer program product can be written in any combination of one or more programming languages ​​to perform the operations of the embodiments of this application. The programming languages ​​include object-oriented programming languages ​​such as Java and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on the user's computing device, partially on the user's computing device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0098] For the foregoing method embodiments, in order to simplify the description, they are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, because according to this application, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily essential to this application.

[0099] It should be noted that the various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For apparatus embodiments, since they are basically similar to method embodiments, the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.

[0100] The steps in the methods of the various embodiments of this application can be adjusted, merged, or deleted in order according to actual needs, and the technical features described in each embodiment can be replaced or combined.

[0101] The modules and sub-modules in the various embodiments of the present application's devices and terminals can be merged, divided, and deleted according to actual needs.

[0102] It should be understood that the disclosed terminals, devices, and methods can be implemented in other ways, given the several embodiments provided in this application. For example, the terminal embodiments described above are merely illustrative. For instance, the division of modules or sub-modules is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple sub-modules or modules may be combined or integrated into another module, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces, devices, or modules, and may be electrical, mechanical, or other forms.

[0103] The modules or submodules described as separate components may or may not be physically separate. The components that constitute a module or submodule may or may not be physical modules or submodules; that is, they may be located in one place or distributed across multiple network modules or submodules. Some or all of the modules or submodules can be selected to achieve the purpose of this embodiment's solution, depending on actual needs.

[0104] Furthermore, the functional modules or sub-modules in the various embodiments of this application can be integrated into one processing module, or each module or sub-module can exist physically separately, or two or more modules or sub-modules can be integrated into one module. The integrated modules or sub-modules described above can be implemented in hardware or in the form of software functional modules or sub-modules.

[0105] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0106] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented directly by hardware, a software unit executed by a processor, or a combination of both. The software unit can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0107] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0108] The above description of the disclosed embodiments enables those skilled in the art to make or use this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for synchronizing wide table data, characterized in that, include: Acquire the raw data and the operation type of the raw data; Add the original data to the data to be synchronized corresponding to the operation type, and increase the value of the counter corresponding to the operation type by a preset value; When the value of the counter corresponding to the operation type is equal to the preset threshold, the data to be synchronized corresponding to the operation type is synchronized to the wide table.

2. The wide table data synchronization method according to claim 1, characterized in that, The operation type of the raw data includes data update; When the operation type of the original data is data update, the step of adding the original data to the data to be synchronized corresponding to the operation type and increasing the value of the counter corresponding to the operation type by a preset value includes: The original data is added to the data to be synchronized corresponding to the data update, and the value of the counter corresponding to the data update is incremented by one.

3. The wide table data synchronization method according to claim 1, characterized in that, The operation types for the raw data include data insertion; When the operation type of the original data is data insertion, the step of adding the original data to the data to be synchronized corresponding to the operation type and increasing the value of the counter corresponding to the operation type by a preset value includes: The original data is added to the data to be synchronized corresponding to the data insertion, and the value of the counter corresponding to the data insertion is incremented by one. When the operation type of the original data is data insertion, the step of synchronizing the data to be synchronized corresponding to the operation type to the wide table when the value of the counter corresponding to the operation type is equal to a preset threshold includes: When the value of the counter corresponding to the data insertion is equal to the number of sub-tables, the data to be synchronized corresponding to the operation type is synchronized to the wide table. The data in the wide table is obtained from the data synchronization of the sub-tables.

4. The wide table data synchronization method according to claim 1, characterized in that, The operation types for the raw data include data insertion; The counters corresponding to the data insertion include counters corresponding to the sub-tables, and the data in the wide table is obtained from the sub-tables through data synchronization. When the operation type of the original data is data insertion, the step of adding the original data to the data to be synchronized corresponding to the operation type and increasing the value of the counter corresponding to the operation type by a preset value includes: Add the original data to the data to be synchronized corresponding to the data insertion; Determine the source sub-table of the original data, select the counter corresponding to the source sub-table from the counters corresponding to the sub-table, and increment the value of the counter corresponding to the source sub-table by one; When the operation type of the original data is data insertion, the step of synchronizing the data to be synchronized corresponding to the operation type to the wide table when the value of the counter corresponding to the operation type is equal to a preset threshold includes: When the values ​​in the counters corresponding to all the sub-tables are equal to the preset sub-table threshold, the data to be synchronized corresponding to the operation type is synchronized to the wide table.

5. The wide table data synchronization method according to claim 1, characterized in that, The wide table is stored in a secondary database; The types of operations for acquiring raw data and acquiring raw data include: Obtain cached data and verify the cached data; use the cached data that passes the verification as the original data and store it in the first-level database. The original data and the operation type of the original data are obtained from the primary database.

6. The wide table data synchronization method according to claim 5, characterized in that, Before obtaining and validating the cached data, the process further includes: Data from a relational database is retrieved and stored as cached data in the message middleware.

7. The wide table data synchronization method according to claim 5, characterized in that, The primary database is an HBase database; The secondary database is the StarRocks database.

8. An electronic device, characterized in that, Including memory and processor; The memory is connected to the processor and is used to store programs; The processor is used to implement the wide table data synchronization method as described in any one of claims 1 to 7 by running a program in the memory.

9. A storage medium, characterized in that, The storage medium stores a computer program, which, when executed by a processor, implements the wide table data synchronization method as described in any one of claims 1 to 7.

10. A computer program product, characterized in that, It includes computer program instructions that, when executed by a processor, cause the processor to perform the wide table data synchronization method as described in any one of claims 1 to 7.

Images