Real-time data processing methods, devices and electronic equipment

By calculating and comparing the actual and predicted values ​​of real-time data, machine learning algorithms are used to handle anomalies in real-time data stream computation, achieving self-recovery and smooth display. This solves the problem of low efficiency in anomaly handling in real-time data stream computation and improves the user experience.

CN113590705BActive Publication Date: 2025-10-31BEIJING KINGSOFT CLOUD NETWORK TECH CO LTD
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
CN202010369828.X
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2020-04-30
Publication Date
2025-10-31
Estimated Expiration
2040-04-30

AI Technical Summary

Technical Problem

Existing technologies have low efficiency in handling anomalies during real-time data stream computation, resulting in poor user experience and a lack of systematic solutions.

Method used

By acquiring the real and estimated values ​​of the real-time data for the current time slice, calculating the difference between the two, and pushing the estimated value to the display unit when the difference exceeds a preset threshold, the estimated value is calculated using a machine learning regression algorithm, and data recovery is performed by combining the weight values ​​of historical data.

Benefits of technology

It achieves self-recovery and smooth display of abnormal situations, improves data processing efficiency, avoids the exposure of abnormal data, and enhances user experience.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN113590705B_ABST
    Figure CN113590705B_ABST
Patent Text Reader

Abstract

This application provides a real-time data processing method, apparatus, and electronic device, relating to the field of data processing technology, and solves the technical problem of low processing efficiency in methods for handling abnormal situations in real-time data stream computation. The method includes: obtaining the true value of real-time data for the current time slice; calculating a predicted value of the real-time data for the current time slice based on historical data; comparing the true value with the predicted value to obtain the difference between the true value and the predicted value; and if the difference exceeds a preset threshold, pushing the predicted value to a display unit.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a real-time data processing method, apparatus, and electronic device. Background Technology

[0002] Currently, there are many scenarios for real-time data flow computing in big data contexts. For example, through the monitoring screen of an online trading platform, one can continuously see the real-time increase in transaction volume, transaction location, and the flow of express logistics, etc.

[0003] However, this real-time calculation process involves many risks and uncontrollable scenarios. For example, if an error occurs in any real-time step, the error will be reflected on the monitoring screen in real time. This could cause a smooth curve to become an incorrect jagged shape, or a curve showing an increase in transaction amount to appear as an incorrect drop. Existing methods for handling such anomalies generally rely on data repair, but this method is inefficient. Summary of the Invention

[0004] The purpose of this invention is to provide a real-time data processing method, apparatus, and electronic device to solve the technical problem of low processing efficiency in the method for handling abnormal situations in real-time calculation of data streams.

[0005] In a first aspect, embodiments of this application provide a real-time data processing method, including:

[0006] Obtain the actual value of the real-time data for the current time slice;

[0007] Based on historical data, calculate the estimated value of the real-time data for the current time slice;

[0008] The actual value is compared with the predicted value to obtain the difference between the actual value and the predicted value.

[0009] If the difference exceeds a preset threshold, the predicted value is pushed to the display unit.

[0010] In one possible implementation, it also includes:

[0011] If the difference does not exceed the preset threshold, the true value is pushed to the display unit.

[0012] In one possible implementation, the step of obtaining the true value of the real-time data for the current time slice includes:

[0013] Input the collected values ​​with time snapshots into multiple real-time systems;

[0014] Each of the real-time systems acquires multiple preliminary true values ​​calculated based on its respective collected values;

[0015] Based on the time snapshot, the multiple preliminary real values ​​are written into multiple time buckets in the database; wherein, the multiple time buckets correspond to the multiple real-time systems.

[0016] By superimposing the preliminary true values ​​with time snapshots from different data points within the same time bucket, the true value of the real-time data for the current time slice is obtained.

[0017] In one possible implementation, the database is an HBase database.

[0018] In one possible implementation, the step of calculating the estimated value of the real-time data for the current time slice based on historical data includes:

[0019] Acquire historical data from multiple time slices;

[0020] Determine the weight value of the historical data for each time slice;

[0021] Based on multiple historical data points and their weight values, the estimated value of the real-time data for the current time slice is calculated.

[0022] In one possible implementation, the historical data for the multiple time slices includes: year-on-year data, month-on-month data, week-on-week data, or day-on-day data.

[0023] Secondly, a real-time data processing device is provided, comprising:

[0024] The acquisition module is used to obtain the actual value of the real-time data for the current time slice;

[0025] The calculation module is used to calculate the estimated value of the real-time data of the current time slice based on historical data;

[0026] The comparison module is used to compare the actual value with the predicted value and obtain the difference between the actual value and the predicted value;

[0027] The first push module is used to push the predicted value to the display unit if the difference exceeds a preset threshold.

[0028] Thirdly, embodiments of this application provide an electronic device, including a memory and a processor, wherein the memory stores a computer program that can run on the processor, and the processor executes the computer program to implement the method described in the first aspect above.

[0029] Fourthly, embodiments of this application also provide a computer-readable storage medium storing machine-executable instructions, which, when invoked and executed by a processor, cause the processor to perform the method described in the first aspect above.

[0030] The embodiments of this application bring the following beneficial effects:

[0031] This application provides a real-time data processing method, apparatus, and electronic device that can acquire the true value of real-time data for the current time slice, then calculate the estimated value of the real-time data for the current time slice based on the historical data, compare the true value with the estimated value to obtain the difference between the true value and the estimated value, and push the estimated value to the display unit when the difference exceeds a preset threshold. Through the process of calculating and comparing the true value and the estimated value, the estimated value is displayed instead of the true value when the difference between the true value and the actual value is large, such as due to error or other abnormality. This achieves anomaly isolation, avoids the need for subsequent data resending and other post-event repair processes, prevents the exposure of abnormal problems to users, improves the efficiency of handling abnormal situations, and solves the technical problem of low processing efficiency of abnormal situation handling methods for real-time data stream calculation.

[0032] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0033] To more clearly illustrate the technical solutions in the specific embodiments of this application or the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0034] Figure 1 A flowchart illustrating the real-time data processing method provided in the embodiments of this application;

[0035] Figure 2 A schematic diagram illustrating the isolation effect of anomalies in the real-time data processing method provided in the embodiments of this application;

[0036] Figure 3 A schematic diagram of the self-recovering real-time computing model provided in the embodiments of this application;

[0037] Figure 4 A schematic diagram of the system framework provided in the embodiments of this application;

[0038] Figure 5This is a schematic diagram of the structure of a real-time data processing device provided in an embodiment of this application;

[0039] Figure 6 This is a schematic diagram illustrating the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0040] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0041] The terms "comprising" and "having," and any variations thereof, used in the embodiments of this application, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or units is not limited to the steps or units listed, but may optionally include other steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or devices.

[0042] Currently, real-time data stream computing has been applied in many fields, such as monitoring transaction volume on online trading platforms and monitoring and analyzing user access on online service platforms. Through real-time data stream computing and analysis, the collected data can be dynamically displayed in real time. However, if anomalies occur in real-time data stream computing, the dynamically displayed data may also be abnormal, resulting in a poor user experience. In such cases, emergency plans are typically activated to handle the situation through data repair. However, real-time data repair requires a long response time; repairing after the problem is exposed to the user is considered reactive. Users prefer to see normal data and do not want abnormal data to affect the normal trend of the data curve. In other words, abnormal data or problems should not be displayed to users during dynamic data display; the data curve should be displayed accurately and smoothly at all times, and any abnormal data should be able to recover on its own. Furthermore, most current methods for handling these anomalies lack effective standards and universal models; they are non-standard business implementations without systematic solutions, resulting in generally low efficiency in resolving these anomalies.

[0043] Based on this, embodiments of this application provide a real-time data processing method, apparatus, and electronic device. This method can solve the technical problem of low processing efficiency in handling abnormal situations during real-time data stream computation.

[0044] The embodiments of the present invention will be further described below with reference to the accompanying drawings.

[0045] Figure 1 This is a flowchart illustrating a real-time data processing method provided in an embodiment of this application. Figure 1 As shown, the method includes:

[0046] S110, obtain the real value of the real-time data for the current time slice.

[0047] In this step, the Content Delivery Network (CDN) can cache real-time traffic monitoring statistics from the monitoring dashboard provided to customers. This process can be implemented in real-time computing within a big data scenario. The principle is to stream large-scale real-time data to computing nodes and perform calculations using specific algorithms.

[0048] For example, the transaction data at each moment can be accumulated, and the accumulated result at a certain point in time can be stored in a storage space with a time attribute. For instance, at 00:00:01 on November 11, 2019, the transaction amount at that moment is 300 million; at 00:00:02 on November 11, 2019, the transaction amount at that moment is 500 million, and so on. The total transaction amount of 300 million was completed in the first minute of November 11, 2019, and the total transaction amount of 800 million (300 million + 500 million) was completed in the second minute. The transaction amount is the real value of the real-time data of the current time slice in this step.

[0049] S120, calculate the estimated value of the real-time data for the current time slice based on historical data.

[0050] In this step, the CDN can calculate the estimated value of the real-time data for the current time slice based on historical data from the same trading platform.

[0051] S130, compare the actual value with the predicted value to obtain the difference between the actual value and the predicted value.

[0052] In this step, the CDN can compare and calculate the actual value obtained in step S110 and the predicted value obtained in step S120 to obtain the difference between the actual value and the predicted value.

[0053] S140, determine whether the difference exceeds a preset threshold. If so, proceed to step S150.

[0054] In this step, the CDN can compare the difference obtained in step S130 with a preset threshold. If the difference exceeds the preset threshold, the next step is executed.

[0055] S150 pushes the predicted value to the display unit.

[0056] In this step, the CDN can push the predicted value obtained in step S120 to the display unit for display by the user.

[0057] In this embodiment of the application, fault isolation is achieved by calculating and comparing the actual value and the estimated value, such as... Figure 2 As shown, problems can be isolated through predictive segments during the abnormal phase, enabling self-recovery backfilling and thus improving the user experience.

[0058] By using predictive segments to isolate problems during the abnormal phase and then self-recovering and backfilling them, data delays, errors, and other anomalies are avoided, and there is no need to resort to emergency plans to handle data repair. This ensures that data is presented to users in an accurate and smooth state at all times.

[0059] In practical applications, the real-time data processing method provided in this application embodiment can also be combined with a standard real-time computing model to obtain a fault isolation real-time model through machine learning. The fault isolation real-time model includes two parts: the predicted value and the actual value. Then, the predicted data determines the final display value, thereby avoiding the exposure of abnormal situations. At the same time, it is convenient to automatically repair the historical curves of the data obtained from the same real-time channel through the backfilling algorithm, so that the data curves are output on the screen efficiently and stably.

[0060] The steps described above will be explained in detail below.

[0061] In some embodiments, the method may further include the following steps:

[0062] S160, if the difference does not exceed the preset threshold, the real value is pushed to the display unit.

[0063] In this embodiment of the application, if the difference calculated in step S130 does not exceed the preset threshold, it indicates that the error of the true value is small. The CDN can push the true value obtained in step S110 to the display unit to display it for the user to view the true value with the error or small error.

[0064] In some embodiments, step S110 above may include the following steps:

[0065] S1102 inputs the collected values ​​with time snapshots to multiple real-time systems.

[0066] S1104, acquire multiple preliminary true values ​​calculated by each real-time system based on its own collected values.

[0067] S1106, based on the time snapshot, write multiple preliminary true values ​​into multiple time buckets in the database respectively.

[0068] S1108: Superimpose the preliminary real values ​​with time snapshots from different time buckets in the same time segment to obtain the real value of the real-time data for the current time slice.

[0069] Among them, multiple time buckets correspond to multiple real-time systems.

[0070] In practical applications, the process of calculating the aforementioned predicted values ​​can be performed using a computational model to improve the efficiency of the calculation. For example, historical data can be input into a self-healing real-time computational model, which can then be used to calculate the predicted values ​​of the real-time data for the current time slice. Figure 3 As shown, after obtaining the predicted values ​​(displayed values, i.e., the inferred values ​​mentioned above) for each time slice range through the self-recovering real-time calculation model, they can be listed in a table along with the actual values ​​so that the actual values ​​can be compared with the predicted values ​​to determine whether to use the actual values.

[0071] For example, a set of defined data models with time snapshots can be used. In a specific computational scenario, the calculation of some values ​​depends on specific data from that scenario. For example... Figure 4 As shown, snapshots can differentiate between versions of intermediate or final results at different times, which is beneficial for data recovery in real-time calculations. In this embodiment, by using input data with time attributes, a standard time attribute field can be added to each batch of massive data requiring real-time calculations.

[0072] For steps S1102, S1104, S1106, and S1108 above, as follows: Figure 4As shown, real-time data with time attributes (i.e., the collected values ​​with time snapshots in S1102) can first be sent as a stream to a general real-time computing framework (i.e., a real-time system, for example, STORM Realtime (processing each data point in real time), Flink Timewindows, Spark Streaming (primarily batch processing offline data), and other real-time implementations). The general real-time computing framework calculates the true value of the real-time data (i.e., the preliminary true value in S1104). Then, the snapshot manager (i.e., the time snapshot in S1106) writes multiple true values ​​into the Time Bucket (i.e., multiple time buckets in the database in S1106) of the database (e.g., a distributed database HBase). Afterwards, the regression prediction module adjusts the displayed value by superimposing different preliminary true values ​​with time snapshots from the same time bucket (i.e., S1108), obtaining usable regression data (the true value of the real-time data of the current time slice in S1108), thereby restoring the actual value at the current moment.

[0073] In some embodiments, the database is an HBase database. Of course, the above model can also be implemented using many other different storage methods, such as DBMS, MongoDB, etc.

[0074] In some embodiments, step S120 above may include the following steps:

[0075] S122, retrieve historical data from multiple time slices.

[0076] S124, determine the weight value of the historical data for each time slice.

[0077] S126, based on multiple historical data and their weight values, calculate the estimated value of the real-time data for the current time slice.

[0078] In practical applications, the inference process involves calculating the growth rate using machine learning regression algorithms, multiplying it by certain weights, and then comparing it with the actual value to arrive at the final inference. This process not only relies on historical data for each time slice but also considers the weight values ​​of various historical data points to calculate the inferred value. These multiple considerations make the inferred value closer to the actual situation, thereby improving the accuracy of the inference.

[0079] In this embodiment, by inputting real-time data with time attributes, adding time buckets, and performing sharding calculations on KV storage with time attribute snapshots, it is possible to recover data in a weighted cumulative summation manner.

[0080] In some embodiments, the historical data for multiple time slices includes: year-on-year data, month-on-month data, week-on-week data, or day-on-day data.

[0081] The inference process in this application embodiment can be derived by using a machine learning regression algorithm to calculate the growth rate, and then calculate it by combining the year-on-year, month-on-month, week-on-week, and day-on-day comparisons with certain weights. The result is then compared with the actual value to arrive at the final inference, thereby improving the accuracy of the data.

[0082] Figure 5 A schematic diagram of a real-time data processing device is provided. Figure 5 As shown, the real-time data processing device 500 includes:

[0083] The acquisition module 501 is used to acquire the real value of the real-time data of the current time slice;

[0084] The calculation module 502 is used to calculate the estimated value of the real-time data of the current time slice based on historical data;

[0085] The comparison module 503 is used to calculate the estimated value of the real-time data of the current time slice based on historical data;

[0086] The first push module 504 is used to push the predicted value to the display unit if the difference exceeds a preset threshold.

[0087] In some embodiments, the device further includes:

[0088] The second push module is used to push the real value to the display unit if the difference does not exceed the preset threshold.

[0089] In some embodiments, the acquisition module 501 is specifically used for:

[0090] Input the collected values ​​with time snapshots into multiple real-time systems;

[0091] Each real-time system separately acquires multiple preliminary true values ​​calculated based on its own collected values;

[0092] Based on the time snapshot, multiple preliminary real values ​​are written into multiple time buckets in the database; each time bucket corresponds to a different real-time system.

[0093] By overlaying the preliminary real values ​​with time snapshots from different data points in the same time bucket, the real value of the real-time data for the current time slice is obtained.

[0094] In some embodiments, the database is an HBase database.

[0095] In some embodiments, the calculation module 502 is specifically used for:

[0096] Acquire historical data from multiple time slices;

[0097] Determine the weight value of the historical data for each time slice;

[0098] Based on multiple historical data points and their weights, the estimated value of the real-time data for the current time slice is calculated.

[0099] In some embodiments, the historical data for multiple time slices includes: year-on-year data, month-on-month data, week-on-week data, or day-on-day data.

[0100] The real-time data processing apparatus provided in this application embodiment has the same technical features as the real-time data processing method provided in the above embodiments, so it can also solve the same technical problems and achieve the same technical effects.

[0101] An electronic device provided in this application embodiment, such as Figure 6 As shown, the electronic device 600 includes a memory 601 and a processor 602. The memory stores a computer program that can run on the processor. When the processor executes the computer program, it implements the steps of the method provided in the above embodiments.

[0102] See Figure 6 The electronic device also includes a bus 603 and a communication interface 604. The processor 602, the communication interface 604 and the memory 601 are connected through the bus 603. The processor 602 is used to execute executable modules, such as computer programs, stored in the memory 601.

[0103] The memory 601 may include high-speed random access memory (RAM) or non-volatile memory, such as at least one disk storage device. Communication between this system network element and at least one other network element is achieved through at least one communication interface 604 (which can be wired or wireless), such as the Internet, wide area network, local area network, metropolitan area network, etc.

[0104] Bus 603 can be an ISA bus, PCI bus, or EISA bus, etc. The bus can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 6 The symbol is represented by a single double-headed arrow, but this does not mean that there is only one bus or one type of bus.

[0105] The memory 601 is used to store programs. After receiving an execution instruction, the processor 602 executes the program. The method executed by the apparatus defined by the process disclosed in any of the preceding embodiments of this application can be applied to the processor 602 or implemented by the processor 602.

[0106] Processor 602 may be an integrated circuit chip with signal processing capabilities. In implementation, each step of the above method can be completed by the integrated logic circuitry in the hardware of processor 602 or by instructions in software form. The processor 602 can be a general-purpose processor, including a Central Processing Unit (CPU), a Network Processor (NP), etc.; it can also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field-Programmable Gate Array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly manifested as execution by a hardware decoding processor, or execution by a combination of hardware and software modules in the decoding processor. The software module can reside in a mature storage medium in the field, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, or registers. This storage medium is located in memory 601, and processor 602 reads the information from memory 601 and, in conjunction with its hardware, completes the steps of the above method.

[0107] Corresponding to the above-described real-time data processing method, this application embodiment also provides a computer-readable storage medium storing machine-executable instructions. When the machine-executable instructions are invoked and executed by a processor, the machine-executable instructions cause the processor to perform the steps of the above-described real-time data processing method.

[0108] The real-time data processing apparatus provided in this application embodiment can be specific hardware on a device or software or firmware installed on the device. The implementation principle and technical effects of the apparatus provided in this application embodiment are the same as those in the foregoing method embodiments. For the sake of brevity, any parts not mentioned in the apparatus embodiment can be referred to the corresponding content in the foregoing method embodiments. Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, apparatuses, and units described above can all be referred to the corresponding processes in the above method embodiments, and will not be repeated here.

[0109] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can also be implemented in other ways. The apparatus embodiments described above are merely illustrative. For example, the flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of apparatus, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram and / or flowchart, and combinations of blocks in block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0110] For example, the division of units is merely a logical functional division; in actual implementation, there may be other division methods. Furthermore, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Additionally, the displayed or discussed mutual couplings, direct couplings, or communication connections may be indirect couplings or communication connections through some communication interfaces, devices, or units, and may be electrical, mechanical, or other forms.

[0111] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0112] In addition, the functional units in the embodiments provided in this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0113] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the real-time data processing method described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0114] It should be noted that similar labels and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. In addition, the terms "first", "second", "third", etc. are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0115] Finally, it should be noted that the above-described embodiments are merely specific implementations of this application, used to illustrate the technical solutions of this application, and not to limit them. The protection scope of this application is not limited thereto. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can still modify or easily conceive of changes to the technical solutions described in the foregoing embodiments, or make equivalent substitutions for some of the technical features, within the scope of the technology disclosed in this application; and these modifications, changes, or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application. All should be covered within the protection scope of this application. Therefore, the protection scope of this application should be determined by the protection scope of the claims.

Claims

1. A real-time data processing method, characterized in that, include: Obtain the actual value of the real-time data for the current time slice; Based on historical data, calculate the estimated value of the real-time data for the current time slice; The actual value is compared with the predicted value to obtain the difference between the actual value and the predicted value. If the difference exceeds a preset threshold, the predicted value is pushed to the display unit; The step of obtaining the true value of the real-time data for the current time slice includes: Input the collected values ​​with time snapshots into multiple real-time systems; Each of the real-time systems acquires multiple preliminary true values ​​calculated based on its respective collected values; Based on the time snapshot, the multiple preliminary real values ​​are written into multiple time buckets in the database; wherein, the multiple time buckets correspond to the multiple real-time systems. By superimposing the preliminary true values ​​with time snapshots from different data points within the same time bucket, the true value of the real-time data for the current time slice is obtained. The step of calculating the estimated value of the real-time data for the current time slice based on historical data includes: Acquire historical data from multiple time slices; The historical data is input into a self-healing real-time computing model, which is obtained through machine learning. The weight value of the historical data for each time slice is determined using the self-recovering real-time computing model, and the predicted value of the real-time data for the current time slice is calculated based on the multiple historical data and their weight values.

2. The method according to claim 1, characterized in that, Also includes: If the difference does not exceed the preset threshold, the true value is pushed to the display unit.

3. The method according to claim 1, characterized in that, The database in question is HBase.

4. The method according to claim 1, characterized in that, The historical data for the multiple time slices includes: year-on-year data, month-on-month data, week-on-week data, or day-on-day data.

5. A real-time data processing device, characterized in that, include: The acquisition module is used to obtain the actual value of the real-time data for the current time slice; The calculation module is used to calculate the estimated value of the real-time data of the current time slice based on historical data; The comparison module is used to compare the actual value with the predicted value and obtain the difference between the actual value and the predicted value; The first push module is used to push the predicted value to the display unit if the difference exceeds a preset threshold. The acquisition module is specifically used for: Input the collected values ​​with time snapshots into multiple real-time systems; Each of the real-time systems acquires multiple preliminary true values ​​calculated based on its respective collected values. Based on the time snapshot, the multiple preliminary real values ​​are written into multiple time buckets in the database; wherein, the multiple time buckets correspond to the multiple real-time systems. By superimposing the preliminary true values ​​with time snapshots from different data points within the same time bucket, the true value of the real-time data for the current time slice is obtained. The calculation module is specifically used for: acquiring historical data from multiple time slices; inputting the historical data into a self-recovering real-time calculation model, which is obtained through machine learning; using the self-recovering real-time calculation model to determine the weight value of the historical data for each time slice; and calculating the predicted value of the real-time data for the current time slice based on the multiple historical data and their weight values.

6. The apparatus according to claim 5, characterized in that, Also includes: The second push module is used to push the real value to the display unit if the difference does not exceed the preset threshold.

7. An electronic device comprising a memory and a processor, wherein the memory stores a computer program executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method described in any one of claims 1 to 4.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores machine-executable instructions that, when invoked and executed by a processor, cause the processor to perform the method according to any one of claims 1 to 4.

Citation Information

Patent Citations

  • Order forecasting method and system

    CN104766144A

  • Data processing method and device

    CN108829718A