Data dynamic balancing method and device

Abstract

The invention provides a data dynamic balancing method and device. The method comprises the following steps: subscribing and pulling first detail data in a first theme queue and second detail data in a second theme queue in message middleware; performing specified data processing on the second detail data; identifying the current data accumulation piece number of the data partition to which the first detail data belongs in the first theme queue; if the current data accumulation number is greater than the number threshold, performing data marking on the first detail data and then storing the first detail data to a second theme queue; if the current data accumulation number is not greater than the number threshold, performing specified data processing on the first detail data; and on the basis of the first detail data and the second detail data subjected to the specified data processing, the index data result is obtained through calculation, data dynamic balance in the operation process is achieved on the premise that operation does not need to be stopped, and the data processing effect is improved.

Description

Technical Field

[0001] The present invention relates to the field of dynamic balancing technology, and in particular to a data dynamic balancing method and device. Background Art

[0002] When processing job data (such as credit card business data), banks often encounter problems with unbalanced consumption and processing of detailed data, which in turn causes the processing nodes to reach performance bottlenecks. Currently, dynamic capacity expansion is usually used to expand job resources to alleviate the aforementioned situation. However, this method involves stopping the original job and reissuing the new job, which causes a short-term processing interruption to the job data processing, and there is a certain probability of startup failure, resulting in poor job data processing results. Summary of the invention

[0003] In view of this, an embodiment of the present invention provides a data dynamic balancing method and device to solve the problem of poor operation data processing effect existing in the dynamic expansion method.

[0004] To achieve the above objectives, the embodiments of the present invention provide the following technical solutions:

[0005] A first aspect of an embodiment of the present invention discloses a method for dynamic data balancing, the method comprising:

[0006] Subscribe to and pull the first detailed data in the first topic queue and the second detailed data in the second topic queue in the message middleware, where the first detailed data is real-time data collected from a specified business page, and the second detailed data is any first detailed data after data marking;

[0007] performing specified data processing on the second detailed data;

[0008] Identify the current number of data accumulation entries in the data partition to which the first detailed data belongs in the first subject queue;

[0009] If the number of currently accumulated data is greater than the threshold, the first detailed data is marked and stored in the second topic queue;

[0010] If the number of accumulated items of the current data is not greater than the item number threshold, performing the specified data processing on the first detailed data;

[0011] Based on the first detailed data and the second detailed data after the designated data processing, an indicator data result is calculated.

[0012] Preferably, the indicator data result is calculated based on the first detailed data and the second detailed data after the designated data processing, including:

[0013] Merging the first detailed data and the second detailed data after the designated data processing to obtain merged detailed data;

[0014] The index calculations on multiple specified dimensions are performed on the merged detailed data to obtain the index data results.

[0015] Preferably, identifying the current number of data accumulation entries of the data partition to which the first detailed data belongs in the first subject queue includes:

[0016] Determine the data partition to which the first detailed data belongs in the first subject queue according to the partition information attached to the first detailed data;

[0017] From the partition accumulation information of the first subject queue of the in-memory database, identify the current number of data accumulation items of the data partition to which the first detailed data belongs, wherein the partition accumulation information includes the number of data accumulation items of each data partition in the first subject queue.

[0018] Preferably, after calculating the indicator data results, the following is further included:

[0019] The indicator data results are sent to an indicator repository for storage.

[0020] Preferably, it also includes:

[0021] Periodically obtain the number of data accumulation entries in each data partition in the first subject queue;

[0022] The number of data accumulation entries in each data partition in the first topic queue is updated into the in-memory database.

[0023] A second aspect of an embodiment of the present invention discloses a data dynamic balancing device, the device comprising:

[0024] A subscription unit, used to subscribe to and pull first detailed data in a first topic queue and second detailed data in a second topic queue in the message middleware, wherein the first detailed data is real-time data collected from a specified business page, and the second detailed data is any first detailed data after data marking;

[0025] A first processing unit, used for performing specified data processing on the second detailed data;

[0026] an identification unit, used to identify the number of current data accumulation entries in the data partition to which the first detailed data belongs in the first subject queue;

[0027] A marking unit, used for marking the first detailed data and storing it in the second subject queue if the number of the current data accumulation is greater than the number threshold;

[0028] A second processing unit, configured to perform the specified data processing on the first detailed data if the number of accumulated items of the current data is not greater than the item number threshold;

[0029] A calculation unit is used to calculate the indicator data result based on the first detailed data and the second detailed data after the specified data processing.

[0030] Preferably, the calculation unit is specifically used to: merge the first detailed data and the second detailed data after the specified data processing to obtain merged detailed data; perform indicator calculations on multiple specified dimensions on the merged detailed data to obtain indicator data results.

[0031] Preferably, the identification unit is specifically used to: determine the data partition to which the first detailed data belongs in the first subject queue based on the partition information attached to the first detailed data; identify the current number of data accumulation items of the data partition to which the first detailed data belongs from the partition accumulation information of the first subject queue in the in-memory database, and the partition accumulation information includes the number of data accumulation items of each data partition in the first subject queue.

[0032] Preferably, it also includes:

[0033] The storage unit is used to send the indicator data result to the indicator storage repository for storage.

[0034] Preferably, it also includes:

[0035] An updating unit is used to periodically obtain the number of data accumulation items in each data partition in the first subject queue; and update the number of data accumulation items in each data partition in the first subject queue to the memory database.

[0036] Based on the above-mentioned embodiment of the present invention, a data dynamic balancing method and device are provided, the method is: subscribe to and pull the first detailed data in the first subject queue and the second detailed data in the second subject queue in the message middleware; perform specified data processing on the second detailed data; identify the current number of data accumulation items in the data partition to which the first detailed data belongs in the first subject queue; if the current number of data accumulation items is greater than the number threshold, the first detailed data is data-marked and stored in the second subject queue; if the current number of data accumulation items is not greater than the number threshold, the first detailed data is subjected to specified data processing; based on the first detailed data and the second detailed data after the specified data processing, the indicator data result is calculated. According to the accumulation of data partitions in the subject queue, this scheme resends the data that has accumulated consumption to another subject queue to be consumed and processed again, realizes the dynamic balancing of data in the operation process without stopping the operation, and improves the data processing effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0038] Figure 1 A flow chart of a data dynamic balancing method provided by an embodiment of the present invention;

[0039] Figure 2 A dynamic balance diagram of credit card real-time streaming operation data provided by an embodiment of the present invention;

[0040] Figure 3 A structural block diagram of a data dynamic balancing device provided by an embodiment of the present invention;

[0041] Figure 4 Another structural block diagram of a data dynamic balancing device provided by an embodiment of the present invention;

[0042] Figure 5 Another structural block diagram of a data dynamic balancing device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0043] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0044] In this application, the terms "comprises", "comprising" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprising a ..." does not exclude the presence of other identical elements in the process, method, article or device comprising the element.

[0045] In order to better understand the following content, some terms are explained here:

[0046] Stream computing: Analyze large-scale flow data in real time during its ever-changing motion process, capture potentially useful information, and send the results to the next computing node.

[0047] Online transaction processing: Using computer networks to connect business processing computer equipment (or networks) distributed in different geographical locations to the business management center network, so that unified, real-time business processing activities or customer service can be carried out at any network node.

[0048] Message queue: a method of communication between processes or between different threads of the same process.

[0049] Kafka partitioning: Partitioning refers to dividing each topic into multiple partitions.

[0050] Data accumulation: Message middleware accumulation usually means that in a message middleware system, the production speed of messages exceeds the consumption speed, resulting in an increasing number of messages stored in the queue.

[0051] When using stream computing technology to process job data (such as credit card business data), we often encounter the problem of unbalanced consumption and processing of detailed data, which in turn causes the processing node to reach a performance bottleneck. For example, when processing real-time streaming data of a credit card browsing page, each piece of data needs to be processed online with the marketing library to add additional information for use in subsequent real-time stream processing windows. Due to the imbalance of consumption data and the uncertainty of the time required for online query of the marketing library, this will cause uneven consumption speeds among various consumption tasks. Once a consumption node reaches a consumption bottleneck, it will also cause data back pressure, affecting the accuracy of downstream data indicator processing.

[0052] When encountering consumption imbalance or data back pressure, the current method of dynamic expansion is to expand job resources and re-publish job tasks to solve the resource bottleneck problem. However, this method will stop the original job and re-publish the new job, which will cause a short interruption in business data processing and there is a certain probability of startup failure.

[0053] The inventor has found through research that message middleware is usually used as a real-time data source for stream computing. Real-time stream computing jobs usually start multiple message middleware consumer instances in parallel to consume message queues and perform logical processing on the acquired message data. When message middleware and stream computing are applied to the processing of credit card business data, the existing processing steps are as follows:

[0054] 1. Collect data on the credit card application browsing page and send it to the message queue;

[0055] 2. Start a stream computing job in the stream computing platform. The job generates multiple consumer instances to consume the message queue.

[0056] 3. Use the credit card browsing page information to access the external marketing database online to check whether it is on the marketing list, and attach relevant information to the credit card browsing page information.

[0057] 4. Carry out data processing in subsequent steps.

[0058] However, since the logic processing in step 3 above often requires online access to external data or involves complex logic calls, the processing time of a single piece of data is uncertain, and the number of data processed per unit time is uncertain; when a large amount of data takes a long time to process, it is easy to cause data consumption imbalance and data back pressure, which in turn causes increased consumer consumption delays.

[0059] Due to data imbalance, the downstream data processing link is prone to lack of data, resulting in reduced accuracy of data indicator processing. The use of traditional operations to expand resources or dynamically expand capacity to restart and stop operations will cause temporary suspension of data processing and reduced accuracy of processed data. There is also a certain probability of failure when re-publishing the operation. Real-time streaming operations usually require high throughput, low latency, data consistency and other characteristics. The suspension of the operation will cause service unavailability and data processing to stop for a period of time, which has a certain impact on data low latency and data consistency. In some cases, the operation may not be able to start, resulting in interruption of data service, and the stability and service availability requirements of the business (such as bank credit card marketing data processing operations) cannot be met.

[0060] To solve the above problems, this solution proposes a data dynamic balancing method and device. According to the accumulation of data partitions in the subject queue, the accumulated data will be resent to another subject queue to be consumed and processed again, thereby achieving dynamic balancing of data in the operation process without stopping the operation, thereby improving the data processing effect.

[0061] See also Figure 1 , shows a flow chart of a data dynamic balancing method provided by an embodiment of the present invention, the data dynamic balancing method comprising:

[0062] Step S101: subscribe to and pull the first detailed data in the first topic queue and the second detailed data in the second topic queue in the message middleware.

[0063] It should be noted that real-time data is collected through a designated business page (such as a credit card application browsing page), and the collected real-time data is sent to the message middleware; wherein a first subject queue (message queue) and a second subject queue (message queue) are created in advance in the message middleware, the first subject queue is used to receive the real-time data collected above, and the second subject queue is used to receive the real-time data after data marking;

[0064] Each topic queue in the message middleware will have multiple data partitions. Each data partition is a separate data queue (which can support high-concurrency writing and reading). When reading data, each data can obtain partition information (such as the data partition and offset, etc.) through configuration.

[0065] In the process of implementing step S101, multiple consumer instances are generated in a distributed manner in the real-time stream job. Through these consumer instances, the first topic queue is subscribed to in the message middleware to pull the first detail data from the first topic queue. In the process of pulling the first detail data, the partition information of the first detail data is attached. The partition information is used to characterize the data partition (such as a kafka partition) to which the first detail data belongs in the first topic queue, wherein the first detail data is real-time data collected from a specified business page.

[0066] Similarly, multiple consumer instances are generated in a distributed manner in the real-time stream job, and the second topic queue is subscribed to in the message middleware through these consumer instances to pull the second detail data from the second topic queue, wherein the second detail data is any first detail data after data marking.

[0067] Step S102: performing designated data processing on the second detailed data.

[0068] In the specific implementation of step S102, after the second detailed data is pulled, the second detailed data is transmitted to the data processing unit for designated data processing and online query of additional information.

[0069] For example: the second detailed data is transferred to the data processing unit for designated data processing, and additional marketing related information is queried online based on the marketing library.

[0070] After performing designated data processing on the second detailed data, the process proceeds to step S106, and then continues to execute subsequent steps S103 to S105.

[0071] It should be noted that the designated data processing mentioned above can be determined according to actual business needs and is not specifically limited here.

[0072] Step S103: Identify the current number of data accumulation items in the data partition to which the first detailed data belongs in the first subject queue.

[0073] It should be noted that when the first detailed data is pulled from the first topic queue, the partition information of the first detailed data is attached.

[0074] In the process of implementing step S103, the data partition to which the first detailed data belongs in the first subject queue is determined according to the partition information attached to the first detailed data; the current number of data accumulation items of the data partition to which the first detailed data belongs is identified from the partition accumulation information of the first subject queue of the memory database (such as a memory server), and the partition accumulation information contains the number of data accumulation items of each data partition in the first subject queue.

[0075] Specifically, the partition accumulation information of each subject queue is stored in the memory database, the partition accumulation information of the first subject queue is obtained from the memory database, and then the current data accumulation number of the data partition to which the first detailed data belongs is obtained from the partition accumulation information (that is, the accumulation data is obtained).

[0076] After obtaining the current number of data accumulation entries of the data partition to which the first detailed data belongs, the current number of data accumulation entries is compared with the entry threshold;

[0077] If the number of currently accumulated data items is greater than the item number threshold, execute step S104; if the number of currently accumulated data items is not greater than the item number threshold, execute step S105.

[0078] Step S104: If the number of currently accumulated data is greater than the threshold, the first detailed data is marked and stored in the second topic queue.

[0079] In the process of implementing step S104, if the current number of accumulated data is greater than the threshold, it indicates that data accumulation has occurred in the data partition to which the first detailed data belongs. At this time, the first detailed data is marked and stored in the second subject queue. At this time, the "first detailed data" after data marking and stored in the second subject queue is used as the "second detailed data".

[0080] Step S105: If the number of currently accumulated data items is not greater than the item threshold, perform designated data processing on the first detailed data and execute step S106.

[0081] In the specific implementation of step S105, if the current number of accumulated data is not greater than the threshold, indicating that no data accumulation has occurred in the data partition to which the first detailed data belongs, the first detailed data is transferred to the data processing unit for designated data processing and online query of additional information, and step S106 is executed.

[0082] Step S106: Calculate the index data result based on the first detailed data and the second detailed data after the designated data processing.

[0083] In the specific implementation of step S106, after obtaining the first detailed data after the designated data processing and the second detailed data after the designated data processing, the first detailed data after the designated data processing and the second detailed data are merged to obtain merged detailed data;

[0084] The index calculations on multiple specified dimensions are performed on the merged detailed data to obtain the index data results, wherein the index calculations on multiple specified dimensions include but are not limited to: aggregation calculations such as summing and averaging.

[0085] For example: assuming that the merged detail data is the browsing data of the credit card application page, statistics are performed on the number of visitors who browsed the credit card application page on that day, and statistics on the browsing data of each type of card on that day.

[0086] In some embodiments, after the indicator data result is calculated, the indicator data result is sent to an indicator repository for storage.

[0087] In other embodiments, the number of data accumulation items in each data partition in the first subject queue is periodically obtained through the data accumulation interface of the message middleware; and the number of data accumulation items in each data partition in the first subject queue is updated to the memory database.

[0088] In an embodiment of the present invention, according to the accumulation of data partitions in the subject queue, the accumulated data is resent to another subject queue to be consumed and processed again, thereby achieving dynamic balancing of data in the operation process without stopping the operation, thereby improving the data processing effect.

[0089] To better understand the above embodiments of the present invention Figure 1 The following is a practical scenario of dynamic balance of credit card data, combined with Figure 2 The credit card real-time streaming data dynamic balance diagram is shown as an example. Figure 2 The Topic 1 queue in is the first topic queue mentioned above. Figure 2 The Topic 2 queue in the above is the second topic queue mentioned above; Figure 2 It includes the following steps 1 to 13:

[0090] Link 1, credit card application browsing page: collect real-time data of the credit card application page (real-time browsing data, i.e., the first detailed data) through the credit card application browsing page, and write the collected real-time data into the Topic 1 queue.

[0091] Link 2, Message Middleware: The message middleware stores the information of the credit card application browsing page and provides message subscription services. It creates Topic 1 queue and Topic 2 queue in advance before data collection.

[0092] The subject 1 queue receives the real-time data (i.e., the first detailed data) collected from the credit card application browsing page, and the subject 2 queue receives the first detailed data pushed after data marking in the following "step 7", and the "first detailed data pushed after data marking" is called the second detailed data;

[0093] Topic 1 queue is used for data processing and consumption by the main process of the job, and Topic 2 queue is used for data processing and consumption by the accumulation data bypass. The sum of the data in Topic 1 queue and Topic 2 queue is a complete set of credit card application page browsing data.

[0094] Step 3, consumer data pulling: Multiple consumer instances are generated in a distributed manner in the real-time stream job. These consumer instances are used to subscribe to the Topic 1 queue in the message middleware and pull the first detail data. During the pulling process, the partition information of each piece of the first detail data in the Topic 1 queue is attached.

[0095] Step 4, obtaining accumulated data: obtaining the partition accumulation information about the topic 1 queue from the memory database, and obtaining the current number of data accumulation items of the data partition to which the first detailed data belongs from the partition accumulation information (that is, obtaining the accumulated data).

[0096] Step 5, accumulation judgment: judge whether the current number of accumulated data is greater than the number threshold n; if the current number of accumulated data is greater than the number threshold n, the first detailed data enters step 7; if the current number of accumulated data is not greater than the number threshold n, the first detailed data enters step 6.

[0097] Link 6, data processing unit: The first detailed data enters the input data processing unit for designated data processing, and performs online query based on the marketing database for additional marketing related information.

[0098] Step 7, data marking: If the current number of accumulated data is greater than the threshold value n, the first detailed data will be marked, and the marked first detailed data will be pushed to the Topic 2 queue, and the marked first detailed data pushed to the Topic 2 queue will be used as the second detailed data.

[0099] Step 8: Pulling consumption data: Generate multiple consumer instances in a distributed manner in the real-time stream job, and use these consumer instances to subscribe to the second detailed data in the queue of Topic 2 in the message middleware.

[0100] Link 9, detailed data processing branch: The second detailed data is transferred to the data processing unit for designated data processing, and additional marketing related information is queried online based on the marketing database.

[0101] Step 10, data union: The first detailed data and the second detailed data that have completed the specified data processing in steps 6 and 9 are combined into a detailed data stream. The synthesized detailed data stream is called merged detailed data. Specifically, the stream computing platform is used to merge the first detailed data and the second detailed data that have completed the specified data processing.

[0102] Step 11, Aggregate Calculation: Perform indicator calculations on various dimensions of the merged detailed data (such as sum, average, and other aggregate calculations) to obtain indicator data results; specifically, the merged detailed data is the credit card application page browsing data, and the credit card application page browsing data is counted for the number of visitors who browsed the page on that day, the browsing data of each type of card on that day, and the number of visitors who browsed the page on that day who belong to the marketing target library.

[0103] Step 12: Result output: Send the calculated indicator data results to various indicator repositories for indicator data storage.

[0104] Step 13, data delay indicator detection: Use the data accumulation interface of the message middleware to obtain the number of data accumulation items for each data partition of the topic 1 queue every second, and update the obtained data accumulation number to the in-memory database.

[0105] From the contents of the above embodiments, it can be seen that this solution, according to the data accumulation situation of each data partition of the message middleware, will mark the data of the data partition where consumption accumulation occurs and then resend it to another topic queue of the message middleware, so that this part of the accumulated data can be consumed and processed again in the bypass, and can achieve dynamic data balancing in seconds during the entire job operation, so that the backlog of data nodes where hot spots are concentrated can be processed, ensuring the stable operation of the entire job. At the same time, the number of processing nodes of the bypass branch is generally one order of magnitude smaller than that of the main data processing nodes, focusing on the balanced use of data for performance bottlenecks of certain nodes during the operation process. Under the premise of taking into account resource overhead, it can dynamically solve the temporary performance bottleneck problem caused by data hot spots and aggregation during data processing.

[0106] Corresponding to a data dynamic balancing method provided by the above embodiment of the present invention, see Figure 3 , the embodiment of the present invention also provides a structural block diagram of a data dynamic balancing device, the data dynamic balancing device includes: a subscription unit 100, a first processing unit 200, an identification unit 300, a marking unit 400, a second processing unit 500 and a calculation unit 600;

[0107] The subscription unit 100 is used to subscribe to and pull the first detailed data in the first topic queue and the second detailed data in the second topic queue in the message middleware. The first detailed data is real-time data collected from a specified business page, and the second detailed data is any first detailed data after data marking.

[0108] The first processing unit 200 is used to perform specified data processing on the second detailed data.

[0109] The identification unit 300 is used to identify the current number of data accumulation items in the data partition to which the first detailed data belongs in the first subject queue.

[0110] In the specific implementation, the identification unit 300 is specifically used to: determine the data partition to which the first detailed data belongs in the first subject queue according to the partition information attached to the first detailed data; identify the current number of data accumulation items of the data partition to which the first detailed data belongs from the partition accumulation information of the first subject queue of the in-memory database, and the partition accumulation information includes the number of data accumulation items of each data partition in the first subject queue.

[0111] The marking unit 400 is used to mark the first detailed data and store it in the second topic queue if the number of current data accumulation items is greater than the item number threshold.

[0112] The second processing unit 500 is used to perform designated data processing on the first detailed data if the number of currently accumulated data items is not greater than the item number threshold.

[0113] The calculation unit 600 is used to calculate the indicator data result based on the first detailed data and the second detailed data after the specified data processing.

[0114] In a specific implementation, the calculation unit 600 is specifically used to: merge the first detailed data and the second detailed data after the specified data processing to obtain merged detailed data; perform indicator calculations on multiple specified dimensions on the merged detailed data to obtain indicator data results.

[0115] In an embodiment of the present invention, according to the accumulation of data partitions in the subject queue, the accumulated data is resent to another subject queue to be consumed and processed again, thereby achieving dynamic balancing of data in the operation process without stopping the operation, thereby improving the data processing effect.

[0116] Preferably, combined Figure 3 , see Figure 4 , shows another structural block diagram of a data dynamic balancing device provided by an embodiment of the present invention, the data dynamic balancing device also includes:

[0117] The storage unit 700 is used to send the indicator data results to the indicator storage repository for storage.

[0118] Preferably, combined Figure 3 , see Figure 5 , shows another structural block diagram of a data dynamic balancing device provided by an embodiment of the present invention, the data dynamic balancing device also includes:

[0119] The updating unit 800 is used to periodically obtain the number of data accumulation items in each data partition in the first subject queue; and update the number of data accumulation items in each data partition in the first subject queue to the memory database.

[0120] In summary, the embodiments of the present invention provide a method and device for dynamic data balancing. According to the accumulation of data partitions in a subject queue, the accumulated data is resent to another subject queue to be consumed and processed again, thereby achieving dynamic data balancing of the operation process without stopping the operation, thereby improving the data processing effect.

[0121] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can refer to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system or system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can refer to the partial description of the method embodiment. The system and system embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without creative work.

[0122] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0123] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A data dynamic balancing method, characterized in that: The method comprises: Subscribe to and pull the first detailed data in the first topic queue and the second detailed data in the second topic queue in the message middleware, where the first detailed data is real-time data collected from a specified business page, and the second detailed data is any first detailed data after data marking; performing specified data processing on the second detailed data; Identify the current number of data accumulation entries in the data partition to which the first detailed data belongs in the first subject queue; If the number of currently accumulated data is greater than the threshold, the first detailed data is marked and stored in the second topic queue; If the number of accumulated items of the current data is not greater than the item number threshold, performing the specified data processing on the first detailed data; Based on the first detailed data and the second detailed data after the designated data processing, an indicator data result is calculated.

2. The method according to claim 1, characterized in that The index data result is calculated based on the first detailed data and the second detailed data after the designated data processing, including: Merging the first detailed data and the second detailed data after the designated data processing to obtain merged detailed data; The index calculations on multiple specified dimensions are performed on the merged detailed data to obtain the index data results.

3. The method according to claim 1, characterized in that Identifying the current number of data accumulation entries of the data partition to which the first detailed data belongs in the first subject queue includes: Determine the data partition to which the first detailed data belongs in the first subject queue according to the partition information attached to the first detailed data; From the partition accumulation information of the first subject queue of the in-memory database, identify the current number of data accumulation entries of the data partition to which the first detailed data belongs, wherein the partition accumulation information includes the number of data accumulation entries of each data partition in the first subject queue.

4. The method according to claim 1, characterized in that: After the indicator data results are calculated, it also includes: The indicator data results are sent to an indicator repository for storage.

5. The method according to any one of claims 1 to 4, characterized in that: Also includes: Periodically obtain the number of data accumulation entries in each data partition in the first subject queue; The number of data accumulation entries in each data partition in the first topic queue is updated into the in-memory database.

6. A data dynamic balancing device, characterized in that: The device comprises: A subscription unit, used to subscribe to and pull first detailed data in a first topic queue and second detailed data in a second topic queue in the message middleware, wherein the first detailed data is real-time data collected from a specified business page, and the second detailed data is any first detailed data after data marking; A first processing unit, used for performing specified data processing on the second detailed data; an identification unit, used to identify the number of current data accumulation entries in the data partition to which the first detailed data belongs in the first subject queue; A marking unit, used for marking the first detailed data and storing it in the second subject queue if the number of the current data accumulation is greater than the number threshold; A second processing unit, configured to perform the specified data processing on the first detailed data if the number of accumulated items of the current data is not greater than the item number threshold; A calculation unit is used to calculate the indicator data result based on the first detailed data and the second detailed data after the specified data processing.

7. The device according to claim 6, characterized in that The calculation unit is specifically used for: merging the first detailed data and the second detailed data after the specified data processing to obtain merged detailed data; performing indicator calculation on multiple specified dimensions on the merged detailed data to obtain indicator data results.

8. The device according to claim 6, characterized in that The identification unit is specifically used to: determine the data partition to which the first detailed data belongs in the first subject queue according to the partition information attached to the first detailed data; and identify the current number of data accumulation items of the data partition to which the first detailed data belongs from the partition accumulation information of the first subject queue in the memory database, wherein the partition accumulation information includes the number of data accumulation items of each data partition in the first subject queue.

9. The device according to claim 6, characterized in that Also includes: The storage unit is used to send the indicator data result to the indicator storage repository for storage.

10. The device according to any one of claims 6 to 9, characterized in that: Also includes: An updating unit is used to periodically obtain the number of data accumulation items in each data partition in the first subject queue; and update the number of data accumulation items in each data partition in the first subject queue to the memory database.

Images