Data display system and method based on big data stream processing

Abstract

The invention discloses a data display system and method based on big data stream processing, and relates to the technical field of data processing.The display method comprises the following steps that a data management system is constructed to record a processing process and a data display process every other unit period, and data display records are generated, carrying out anomaly identification on the data display record; any abnormal display record is extracted, the change process of the abnormal display result is analyzed, abnormal behaviors are confirmed, and abnormal features are extracted; randomly selecting two adjacent data display records, and formulating a data adjustment strategy for correction based on the abnormal identification condition of the two data display records; performing anomaly identification on the real-time display record generated in real time; if the real-time display record is abnormal, abnormal reminding is sent, and a corresponding data adjustment strategy is called for real-time correction; the requirement for manual intervention is effectively reduced, and the operation and maintenance cost is reduced.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a data display system and method based on large data stream processing. Background Art

[0002] Big data stream processing refers to the technology and methods for processing and analyzing real-time data streams. Different from traditional batch processing, stream processing emphasizes the real-time and continuity of data, and can process data at the moment it is generated, so as to quickly obtain valuable information. Traditional data display usually relies on batch processing, and the data update frequency is low. If data stream processing is added to data display, it can help users obtain the latest news and insights.

[0003] However, after adding data stream processing, a more complex system architecture is required in the data presentation system, and the debugging and monitoring of real-time data streams are also more complicated. When an exception occurs during the processing process, how to effectively handle the fault and recover the data is also an important challenge. Summary of the invention

[0004] The purpose of the present invention is to provide a data display system and method based on large data stream processing to solve the problems raised in the prior art.

[0005] To achieve the above object, the present invention provides the following technical solution: a data display method based on big data stream processing, the display method comprising the following steps:

[0006] Step S100: real-time monitoring of the data processing process within the data display platform, building a data management system to record the monitoring process, and generating corresponding data display records; based on the differences in the data processing process in any data display record, anomaly identification is performed on the data display record;

[0007] Step S200: extracting data display results from any abnormal display record, analyzing the change process of the abnormal display results, and confirming the abnormal behavior in the abnormal display record; based on the abnormal situation presented by the abnormal behavior, extracting abnormal features of the abnormal display record;

[0008] Step S300: analyzing the inclusion of abnormal features in any data display record, analyzing the impact of the included abnormal features based on the abnormal identification differences between any adjacent data display records, and formulating a data adjustment strategy to correct any abnormal display record;

[0009] Step S400: performing abnormal identification on the data processing process recorded in the real-time display record generated in real time; in case of abnormal identification results, sending abnormal reminders and calling corresponding data adjustment strategies for real-time correction.

[0010] Furthermore, step S100 includes the following steps:

[0011] Step S101: Data from various channels are collected by data collection equipment and transmitted to the data display platform in real time. The transmitted data is preprocessed in the data display platform to generate valid data. The data display platform retrieves the valid data in real time and uses data visualization tools to display the data.

[0012] Step S102: randomly selecting a unit period, extracting the valid data recorded in the unit period and the displayed data, and generating a data display record corresponding to the unit period in the data management system; performing a same data comparison between any display data and any valid data in the data display record to obtain the same data set of the data display record; constructing the same data set can distinguish the difference between the valid data and the displayed data, and provide a data basis for subsequent error analysis;

[0013] Step S103: Preset a display time delay T Delay , obtain the amount of data D in the same data set recorded in the data display sim , according to the formula:

[0014]

[0015] Among them, T is the unit cycle length, D eff is the amount of valid data in the data display record, D show is the amount of display data in the data display record; calculate the error degree η of the data display record; set an allowable error η th , if η>η th , then the data display record is set as an abnormal display record; there will be a normal time delay in the process of displaying the valid data after processing, but if the delay is too high, it is an abnormal situation. Similarly, another abnormal situation is the omission of display data. The error situation in a unit cycle is reflected based on the comprehensive situation of the delay degree and the proportion of omitted data, which can effectively identify the abnormal situation; by first identifying the abnormality of each data display record, the abnormal type can be identified in the subsequent steps.

[0016] Further, step S200 includes the following steps:

[0017] Step S201: randomly select an abnormal display record, obtain the same data set in the abnormal display record, randomly select a piece of data from the same data set, and obtain the time point when the data is processed into valid data in the corresponding unit period as t eff The time point when the data is displayed is t show , the actual display delay of the data is obtained as (T delay ) ’ =t show -t eff ; respectively obtain the actual display delay of each data in the same data set, and calculate the average value to obtain the average display delay (T delay ) ac ;

[0018] Step S202: Acquire all valid data of the abnormal display record, and sort each valid data in the order of processing time to obtain a valid data set; sort each identical data in the identical data set in the order of display time, arbitrarily select two identical data adjacent to each other from the re-sorted identical data set, and respectively obtain the position sequence of the two identical data selected in the valid data set; if the two identical data are not adjacent to each other in the valid data set, perform a first abnormal mark on the abnormal display record; the first abnormal mark reflects data omission, which is a problem occurring during the data display process;

[0019] Step S203: randomly select a piece of identical data from the reordered identical data set, and obtain the actual display delay (T delay ) ’ , if (T delay ) ’ >(T delay ) ac , then continue to select the next piece of the same data, and obtain the actual display delay (T delay )";

[0020] Step S204: When (T delay )”>(T delay ) ac When , the actual display delay of each identical data behind the identical data in the reordered identical data set is obtained. If the actual display delay of consecutive identical data is higher than the average display delay, the data volume D1 contained in the consecutive identical data is counted. yc , get the abnormal data proportion of the abnormal display record τ1=D1 yc / D1, where D1 is the amount of data behind the same data, and a percentage threshold τ is set. ’, if τ1>τ ’ , then the abnormal display record is marked as a second abnormality; when (T delay )”<(T delay ) ac When , the amount of abnormal data D2 in which the actual display delay of the same record set is higher than the average display delay is counted yc , get the abnormal data proportion of the abnormal display record τ2=D2 yc / D2 sim , where D2 sim is the amount of data in the same data set, if τ2>τ ’ , then the abnormal display record is marked as a second abnormality; the second abnormality mark is an abnormal delay problem, which is a problem that occurs during data transmission, and the abnormal delay can be subdivided into two situations, one is a fluctuating delay, and the other is a continuous high-amplitude delay;

[0021] Step S205: setting the data display records without abnormal marks as normal display records, respectively presetting characteristic information of several dimensions for the data flow processing process and the data display process of the data display platform, and extracting the operation data corresponding to the preset characteristic information of the data display platform from each normal display record; acquiring the operation data corresponding to any characteristic information in each normal display record, and obtaining the normal value range corresponding to the characteristic information;

[0022] Step S206: Obtain the operating data of any characteristic information in the abnormal display record. If the operating data of the obtained characteristic information is not within the normal value range, set the characteristic information as an abnormal feature of the abnormal display record; extract corresponding abnormal features for different abnormal situations, so that the cause of the abnormality can be directly obtained in the subsequent abnormality identification process, so that the corresponding correction strategy can be called in the subsequent process, reducing manual intervention and improving processing efficiency.

[0023] Further, step S300 includes the following steps:

[0024] Step S301: obtaining a plurality of abnormal features of two data display records respectively and generating two abnormal feature sets, performing feature comparison on the two abnormal feature sets, and obtaining a difference abnormal feature set of any one data display record relative to the other data display record respectively;

[0025] Step S302: if the first data display record of the two data display records is an abnormal display record, the difference abnormal feature set of the first data display record is set as the target adjustment set; if the second data display record is an abnormal display record, the difference abnormal feature set of the second data display record is set as the target adjustment set; because the data display record records the data display situation in each unit period, the operating conditions presented by the two adjacent display records are the most similar, and the difference in abnormal features between the two adjacent display records is the most important feature that causes the abnormal situation to change, so it is most reasonable to use the difference abnormal features as the adjustment target;

[0026] Step S303: Obtain an abnormal mark of the abnormal display record. If the abnormal display record has a first abnormal mark, obtain several feature information preset in the data display process. If the abnormal display record has a second abnormal mark, obtain several feature information preset in the data stream processing process. Extract each abnormal feature in the target adjustment set and compare it with the obtained several feature information, remove the abnormal features that are different from the feature information, and obtain a corrected target adjustment set. Here, some abnormal features are checked because not all abnormal features are the cause of the abnormality.

[0027] Step S304: Obtain the target adjustment set after correction of each data display record, perform intersection processing on any two target adjustment sets, and if one target adjustment set contains another target adjustment set, remove the target adjustment set containing more abnormal features to obtain all target adjustment sets, and correct each abnormal feature in each target adjustment set, record the correction strategy of each abnormal feature, and generate a data adjustment strategy for each target adjustment set.

[0028] Furthermore, step S400 includes the following steps:

[0029] Step S401: selecting a newly generated real-time display record from the data management system, obtaining valid data recorded in the real-time display record and the displayed display data, calculating the error degree of the real-time display record, and if the calculated error degree is greater than the allowable error, issuing an abnormal reminder for the real-time display record;

[0030] Step S402: extract feature information from the data flow processing process and the data display process in the real-time display record to obtain an abnormal feature set; compare the abnormal feature set with each target adjustment set, and if the abnormal feature set contains a target adjustment set, call the data adjustment strategy of the target adjustment set, and then correct each abnormal feature.

[0031] In order to better implement the above method, a data display system based on big data stream processing is also proposed. The display system includes a historical record analysis module, an abnormal feature processing module, a display data correction module and a real-time display analysis module;

[0032] The historical record analysis module is used to monitor the data processing process inside the data display platform in real time, build a data management system to record the monitoring process, and generate corresponding data display records; based on the differences in the data processing process in any data display record, the data display record is identified as abnormal;

[0033] The abnormal feature processing module is used to extract the data display results in any abnormal display record, analyze the change process of the abnormal display results, confirm the abnormal behavior in the abnormal display record; based on the abnormal situation presented by the abnormal behavior, extract the abnormal features of the abnormal display record;

[0034] The display data correction module is used to analyze the inclusion of abnormal features in any data display record, analyze the impact of the included abnormal features based on the abnormal identification differences between any adjacent data display records, and formulate a data adjustment strategy to correct any abnormal display record;

[0035] The real-time display analysis module is used to identify anomalies in the data processing process recorded by the real-time display records generated in real time; in the case of abnormal identification results, an abnormal reminder is sent and the corresponding data adjustment strategy is called for real-time correction.

[0036] Further, the historical record analysis module includes a display data acquisition unit and an abnormal data identification unit;

[0037] The display data acquisition unit is used to monitor the data processing process inside the data display platform in real time, build a data management system to record the monitoring process, and generate corresponding data display records; the abnormal data identification unit is used to identify abnormalities in the data display records based on the differences in the data processing process in any data display record.

[0038] Further, the abnormal feature processing module includes an abnormal behavior analysis unit and an abnormal feature extraction unit;

[0039] The abnormal behavior analysis unit is used to extract the data display results in any abnormal display record, analyze the change process of the abnormal display results, and confirm the abnormal behavior in the abnormal display record; the abnormal feature extraction unit is used to extract the abnormal features of the abnormal display record based on the abnormal situation presented by the abnormal behavior.

[0040] Further, the display data correction module includes a feature inclusion analysis unit and an adjustment strategy processing unit;

[0041] The feature inclusion analysis unit is used to analyze the inclusion of abnormal features in any data display record; the adjustment strategy processing unit is used to analyze the impact of the included abnormal features based on the abnormal identification differences between any adjacent data display records, and formulate a data adjustment strategy to correct any abnormal display record.

[0042] Furthermore, the real-time display analysis module includes a real-time abnormality judgment unit and a real-time abnormality correction unit;

[0043] The real-time anomaly judgment unit is used to identify anomalies in the data processing process recorded by the real-time display record generated in real time; the real-time anomaly correction unit is used to send anomaly reminders and call corresponding data adjustment strategies for real-time correction when the identification results are abnormal.

[0044] Compared with the prior art, the present invention has the following beneficial effects:

[0045] 1. The present invention integrates a real-time data stream processing mode into the data display platform, thereby ensuring the real-time nature of the data display process and helping to respond to business needs and market changes more quickly;

[0046] 2. The present invention monitors and processes abnormal situations in the data display process of the data display platform in real time, thereby improving the reliability of the system; by analyzing different abnormal situations occurring in each unit cycle, various abnormal situations are identified, helping staff to accurately identify problems and solve them, thereby improving processing efficiency;

[0047] 3. The present invention formulates different correction strategies for different abnormal situations, helping staff to actively make adjustments after identifying the abnormalities; it can effectively reduce the need for manual intervention and reduce operation and maintenance costs; users will not feel confused or dissatisfied due to interruptions or errors in data display, thereby improving user satisfaction. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 A schematic diagram of the steps of a data display method based on large data stream processing;

[0049] Figure 2 The figure is a structural diagram of a data display system based on big data stream processing. DETAILED DESCRIPTION

[0050] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0051] Example: Figure 1 to Figure 2 As shown, the present invention provides a data display method based on large data stream processing, and the display method comprises the following steps:

[0052] Step S100: real-time monitoring of the data processing process within the data display platform, building a data management system to record the monitoring process, and generating corresponding data display records; based on the differences in the data processing process in any data display record, anomaly identification is performed on the data display record;

[0053] Wherein, step S100 includes the following steps:

[0054] Step S101: Data from various channels are collected by data collection equipment and transmitted to the data display platform in real time. The transmitted data is preprocessed in the data display platform to generate valid data. The data display platform retrieves the valid data in real time and uses data visualization tools to display the data.

[0055] Step S102: randomly selecting a unit period, extracting the valid data recorded in the unit period and the displayed data presented, and generating a data display record corresponding to the unit period in the data management system; performing a data comparison between any display data and any valid data in the data display record to obtain the same data set of the data display record;

[0056] Step S103: Preset a display time delay T Delay , obtain the amount of data D in the same data set recorded in the data display sim , according to the formula:

[0057]

[0058] Among them, T is the unit cycle length, D eff is the amount of valid data in the data display record, D show is the amount of display data in the data display record; calculate the error degree η of the data display record; set an allowable error η th , if η>η th , then the data display record is set as an abnormal display record;

[0059] Embodiment 1: Preset a display time delay of 1s, and the unit cycle length is 10s, and obtain the data volume of the same data set in the data display record as 1M; set the valid data volume in the data display record to 5M, and the display data volume to 3M, and obtain the error degree η=|0.1-6 / 8| / 0.1=6.5; set the allowable error to 5, because 6.5>5, so judge the data display record as an abnormal display record.

[0060] Step S200: extracting data display results from any abnormal display record, analyzing the change process of the abnormal display results, and confirming the abnormal behavior in the abnormal display record; based on the abnormal situation presented by the abnormal behavior, extracting abnormal features of the abnormal display record;

[0061] Wherein, step S200 includes the following steps:

[0062] Step S201: randomly select an abnormal display record, obtain the same data set in the abnormal display record, randomly select a piece of data from the same data set, and obtain the time point when the data is processed into valid data in the corresponding unit period as t eff The time point when the data is displayed is t show , the actual display delay of the data is obtained as (T delay ) ’ =t show -t eff ; respectively obtain the actual display delay of each data in the same data set, and calculate the average value to obtain the average display delay (T delay ) ac ;

[0063] Step S202: Acquire all valid data of the abnormal display record, and sort each valid data in the order of processing time to obtain a valid data set; sort each identical data in the identical data set in the order of display time, arbitrarily select two identical data adjacent to each other from the re-sorted identical data set, and respectively obtain the position sequence of the two selected identical data in the valid data set; if the two identical data are not adjacent to each other in the valid data set, perform a first abnormal mark on the abnormal display record;

[0064] Step S203: randomly select a piece of identical data from the reordered identical data set, and obtain the actual display delay (T delay ) ’ , if (T delay ) ’ >(T delay ) ac, then continue to select the next piece of the same data, and obtain the actual display delay (T delay )";

[0065] Step S204: When (T delay )”>(T delay ) ac When , the actual display delay of each identical data behind the identical data in the reordered identical data set is obtained. If the actual display delay of consecutive identical data is higher than the average display delay, the data volume D1 contained in the consecutive identical data is counted. yc , get the abnormal data proportion of the abnormal display record τ1=D1 yc / D1, where D1 is the amount of data behind the same data, and a percentage threshold τ is set. ’ , if τ1>τ ’ , then the abnormal display record is marked as a second abnormality; when (T delay )”<(T delay ) ac When , the amount of abnormal data D2 in which the actual display delay of the same record set is higher than the average display delay is counted yc , get the abnormal data proportion of the abnormal display record τ2=D2 yc / D2 sim , where D2 sim is the amount of data in the same data set, if τ2>τ ’ , then a second abnormality mark is performed on the abnormal display record;

[0066] Embodiment 2: randomly select an abnormal display record, where the actual display delay of one data is 1.5s, obtain the actual display delay of the adjacent and next data is 1.3s, set the average display delay to 1s, so both data are abnormally delayed, then continue to obtain the delay of subsequent data, obtain the actual display delay of continuous data is higher than the average display delay, and the continuous data volume is 1M, set the subsequent data volume to 2M, obtain the abnormal data ratio is 50%, set the ratio threshold to 30%, because 50%>30%, then the abnormal display record is marked as a second abnormality;

[0067] Step S205: setting the data display records without abnormal marks as normal display records, respectively presetting characteristic information of several dimensions for the data flow processing process and the data display process of the data display platform, and extracting the operation data corresponding to the preset characteristic information of the data display platform from each normal display record; acquiring the operation data corresponding to any characteristic information in each normal display record, and obtaining the normal value range corresponding to the characteristic information;

[0068] Step S206: Obtaining the operating data of any characteristic information in the abnormal display record. If the operating data of the obtained characteristic information is not within the normal value range, setting the characteristic information as an abnormal characteristic of the abnormal display record.

[0069] Step S300: analyzing the inclusion of abnormal features in any data display record, analyzing the impact of the included abnormal features based on the abnormal identification differences between any adjacent data display records, and formulating a data adjustment strategy to correct any abnormal display record;

[0070] Wherein, step S300 includes the following steps:

[0071] Step S301: obtaining a plurality of abnormal features of two data display records respectively and generating two abnormal feature sets, performing feature comparison on the two abnormal feature sets, and obtaining a difference abnormal feature set of any one data display record relative to the other data display record respectively;

[0072] Step S302: if the first data display record of the two data display records is an abnormal display record, the difference abnormal feature set of the first data display record is set as the target adjustment set; if the second data display record is an abnormal display record, the difference abnormal feature set of the second data display record is set as the target adjustment set;

[0073] Step S303: obtaining an abnormal mark of the abnormal display record, if the abnormal display record has a first abnormal mark, obtaining a number of feature information preset in the data display process, if the abnormal display record has a second abnormal mark, obtaining a number of feature information preset in the data stream processing process; extracting each abnormal feature in the target adjustment set and comparing it with the obtained number of feature information, removing the abnormal features different from the feature information, and obtaining a corrected target adjustment set;

[0074] Step S304: Obtain the target adjustment set after correction of each data display record, perform intersection processing on any two target adjustment sets, and if one target adjustment set contains another target adjustment set, remove the target adjustment set containing more abnormal features to obtain all target adjustment sets, and correct each abnormal feature in each target adjustment set, record the correction strategy of each abnormal feature, and generate a data adjustment strategy for each target adjustment set.

[0075] Step S400: performing abnormal identification on the data processing process recorded in the real-time display record generated in real time; in case of abnormal identification results, sending abnormal reminders and calling corresponding data adjustment strategies for real-time correction;

[0076] Wherein, step S400 includes the following steps:

[0077] Step S401: selecting a newly generated real-time display record from the data management system, obtaining valid data recorded in the real-time display record and the displayed display data, calculating the error degree of the real-time display record, and if the calculated error degree is greater than the allowable error, issuing an abnormal reminder for the real-time display record;

[0078] Step S402: extract feature information from the data flow processing process and the data display process in the real-time display record to obtain an abnormal feature set; compare the abnormal feature set with each target adjustment set, and if the abnormal feature set contains a target adjustment set, call the data adjustment strategy of the target adjustment set, and then correct each abnormal feature.

[0079] A data display system based on big data stream processing, the display system includes a historical record analysis module, an abnormal feature processing module, a display data correction module and a real-time display analysis module;

[0080] The historical record analysis module is used to monitor the data processing process inside the data display platform in real time, build a data management system to record the monitoring process, and generate corresponding data display records; based on the differences in the data processing process in any data display record, the data display record is identified as abnormal;

[0081] The abnormal feature processing module is used to extract the data display results in any abnormal display record, analyze the change process of the abnormal display results, confirm the abnormal behavior in the abnormal display record; based on the abnormal situation presented by the abnormal behavior, extract the abnormal features of the abnormal display record;

[0082] The display data correction module is used to analyze the inclusion of abnormal features in any data display record, analyze the impact of the included abnormal features based on the abnormal identification differences between any adjacent data display records, and formulate a data adjustment strategy to correct any abnormal display record;

[0083] The real-time display analysis module is used to identify anomalies in the data processing process recorded by the real-time display records generated in real time; in the case of abnormal identification results, an abnormal reminder is sent and the corresponding data adjustment strategy is called for real-time correction.

[0084] Among them, the historical record analysis module includes a display data collection unit and an abnormal data identification unit;

[0085] The display data acquisition unit is used to monitor the data processing process inside the data display platform in real time, build a data management system to record the monitoring process, and generate corresponding data display records; the abnormal data identification unit is used to identify abnormalities in the data display records based on the differences in the data processing process in any data display record.

[0086] Among them, the abnormal feature processing module includes an abnormal behavior analysis unit and an abnormal feature extraction unit;

[0087] The abnormal behavior analysis unit is used to extract the data display results in any abnormal display record, analyze the change process of the abnormal display results, and confirm the abnormal behavior in the abnormal display record; the abnormal feature extraction unit is used to extract the abnormal features of the abnormal display record based on the abnormal situation presented by the abnormal behavior.

[0088] Among them, the display data correction module includes a feature inclusion analysis unit and an adjustment strategy processing unit;

[0089] The feature inclusion analysis unit is used to analyze the inclusion of abnormal features in any data display record; the adjustment strategy processing unit is used to analyze the impact of the included abnormal features based on the abnormal identification differences between any adjacent data display records, and formulate a data adjustment strategy to correct any abnormal display record.

[0090] Among them, the real-time display analysis module includes a real-time abnormality judgment unit and a real-time abnormality correction unit;

[0091] The real-time anomaly judgment unit is used to identify anomalies in the data processing process recorded by the real-time display record generated in real time; the real-time anomaly correction unit is used to send anomaly reminders and call corresponding data adjustment strategies for real-time correction when the identification results are abnormal.

[0092] It will be apparent to those skilled in the art that the invention is not limited to the details of the exemplary embodiments described above and that the invention can be implemented in other specific forms without departing from the spirit or essential features of the invention. Therefore, the embodiments should be considered exemplary and non-limiting in all respects, and the scope of the invention is defined by the appended claims rather than the foregoing description, and it is intended that all variations falling within the meaning and scope of the equivalent elements of the claims be included in the invention. Any reference numeral in a claim should not be considered as limiting the claim to which it relates.

Claims

1. A data display method based on big data stream processing, characterized in that: The display method comprises the following steps: Step S100: real-time monitoring of the data processing process within the data display platform, building a data management system to record the monitoring process, and generating corresponding data display records; based on the differences in the data processing process in any data display record, anomaly identification is performed on the data display record; Step S200: extracting data display results from any abnormal display record, analyzing the change process of the abnormal display results, and confirming the abnormal behavior in the abnormal display record; based on the abnormal situation presented by the abnormal behavior, extracting abnormal features of the abnormal display record; Step S300: analyzing the inclusion of abnormal features in any data display record, analyzing the impact of the included abnormal features based on the abnormal identification differences between any adjacent data display records, and formulating a data adjustment strategy to correct any abnormal display record; Step S400: performing abnormal identification on the data processing process recorded in the real-time display record generated in real time; in case of abnormal identification results, sending abnormal reminders and calling corresponding data adjustment strategies for real-time correction.

2. A data display method based on large data stream processing according to claim 1, characterized in that: The step S100 includes the following steps: Step S101: Data from various channels are collected by data collection equipment and transmitted to the data display platform in real time. The transmitted data is preprocessed in the data display platform to generate valid data. The data display platform retrieves the valid data in real time and uses data visualization tools to display the data. Step S102: randomly selecting a unit period, extracting the valid data recorded in the unit period and the displayed data presented, and generating a data display record corresponding to the unit period in the data management system; performing a data comparison between any display data and any valid data in the data display record to obtain the same data set of the data display record; Step S103: Preset a display time delay T Delay , obtain the amount of data D in the same data set recorded in the data display sim , according to the formula: Among them, T is the unit cycle length, D eff is the amount of valid data in the data display record, D show is the amount of display data in the data display record; calculate the error degree η of the data display record; set an allowable error η th , if η>η th , then the data display record is set as an abnormal display record.

3. A data display method based on large data stream processing according to claim 2, characterized in that: The step S200 includes the following steps: Step S201: randomly select an abnormal display record, obtain the same data set in the abnormal display record, randomly select a piece of data from the same data set, and obtain the time point when the data is processed into valid data in the corresponding unit period as t eff The time point when the data is displayed is t show , the actual display delay of the data is obtained as (T delay ) ’ =t show -t eff ; respectively obtain the actual display delay of each data in the same data set, and calculate the average value to obtain the average display delay (T delay ) ac ; Step S202: Acquire all valid data of the abnormal display record, and sort each valid data in the order of processing time to obtain a valid data set; sort each identical data in the identical data set in the order of display time, arbitrarily select two identical data adjacent to each other from the re-sorted identical data set, and respectively obtain the position sequence of the two selected identical data in the valid data set; if the two identical data are not adjacent to each other in the valid data set, perform a first abnormal mark on the abnormal display record; Step S203: randomly select a piece of identical data from the reordered identical data set, and obtain the actual display delay (T delay ) ’ , if (T delay ) ’ >(T delay ) ac , then continue to select the next piece of the same data, and obtain the actual display delay (T delay )"; Step S204: When (T delay )”>(T delay ) ac When , the actual display delay of each identical data behind the identical data in the reordered identical data set is obtained. If the actual display delay of consecutive identical data is higher than the average display delay, the data volume D1 contained in the consecutive identical data is counted. yc , get the abnormal data proportion of the abnormal display record τ1=D1 yc / D1, where D1 is the amount of data behind the same data, and a percentage threshold τ is set. ’ , if τ1>τ ’ , then the abnormal display record is marked as a second abnormality; when (T delay )”<(T delay ) ac When , the amount of abnormal data D2 in which the actual display delay of the same record set is higher than the average display delay is counted yc , get the abnormal data proportion of the abnormal display record τ2=D2 yc / D2 sim , where D2 sim is the amount of data in the same data set, if τ2>τ ’ , then a second abnormality mark is performed on the abnormal display record; Step S205: setting the data display records without abnormal marks as normal display records, respectively presetting characteristic information of several dimensions for the data flow processing process and the data display process of the data display platform, and extracting the operation data corresponding to the preset characteristic information of the data display platform from each normal display record; acquiring the operation data corresponding to any characteristic information in each normal display record, and obtaining the normal value range corresponding to the characteristic information; Step S206: Obtaining the operating data of any characteristic information in the abnormal display record. If the operating data of the obtained characteristic information is not within the normal value range, setting the characteristic information as an abnormal characteristic of the abnormal display record.

4. The data display method based on large data stream processing according to claim 3 is characterized in that: The step S300 includes the following steps: Step S301: obtaining a plurality of abnormal features of two data display records respectively and generating two abnormal feature sets, performing feature comparison on the two abnormal feature sets, and obtaining a difference abnormal feature set of any one data display record relative to the other data display record respectively; Step S302: if the first data display record of the two data display records is an abnormal display record, the difference abnormal feature set of the first data display record is set as the target adjustment set; if the second data display record is an abnormal display record, the difference abnormal feature set of the second data display record is set as the target adjustment set; Step S303: obtaining an abnormal mark of the abnormal display record, if the abnormal display record has a first abnormal mark, obtaining a number of feature information preset in the data display process, if the abnormal display record has a second abnormal mark, obtaining a number of feature information preset in the data stream processing process; extracting each abnormal feature in the target adjustment set and comparing it with the obtained number of feature information, removing the abnormal features different from the feature information, and obtaining a corrected target adjustment set; Step S304: Obtain the target adjustment set after correction of each data display record, perform intersection processing on any two target adjustment sets, and if one target adjustment set contains another target adjustment set, remove the target adjustment set containing more abnormal features to obtain all target adjustment sets, and correct each abnormal feature in each target adjustment set, record the correction strategy of each abnormal feature, and generate a data adjustment strategy for each target adjustment set.

5. A data display method based on large data stream processing according to claim 4, characterized in that: The step S400 includes the following steps: Step S401: selecting a newly generated real-time display record from the data management system, obtaining valid data recorded in the real-time display record and the displayed display data, calculating the error degree of the real-time display record, and if the calculated error degree is greater than the allowable error, issuing an abnormal reminder for the real-time display record; Step S402: extract feature information from the data flow processing process and the data display process in the real-time display record to obtain an abnormal feature set; compare the abnormal feature set with each target adjustment set, and if the abnormal feature set contains a target adjustment set, call the data adjustment strategy of the target adjustment set, and then correct each abnormal feature.

6. A data display system based on big data stream processing, used to execute a data display method based on big data stream processing according to any one of claims 1 to 5, characterized in that: The display system includes a historical record analysis module, an abnormal feature processing module, a display data correction module and a real-time display analysis module; The historical record analysis module is used to monitor the data processing process inside the data display platform in real time, build a data management system to record the monitoring process, and generate corresponding data display records; based on the differences in the data processing process in any data display record, the data display record is identified as abnormal; The abnormal feature processing module is used to extract the data display results in any abnormal display record, analyze the change process of the abnormal display results, confirm the abnormal behavior in the abnormal display record; based on the abnormal situation presented by the abnormal behavior, extract the abnormal features of the abnormal display record; The display data correction module is used to analyze the inclusion of abnormal features in any data display record, analyze the impact of the included abnormal features based on the abnormal identification differences between any adjacent data display records, and formulate a data adjustment strategy to correct any abnormal display record; The real-time display analysis module is used to identify anomalies in the data processing process recorded by the real-time display record generated in real time; in the case of abnormal identification results, send an abnormal reminder and call the corresponding data adjustment strategy for real-time correction.

7. A data display system based on large data stream processing according to claim 6, characterized in that: The historical record analysis module includes a display data acquisition unit and an abnormal data identification unit; The display data acquisition unit is used to monitor the data processing process inside the data display platform in real time, build a data management system to record the monitoring process, and generate corresponding data display records; the abnormal data identification unit is used to identify abnormalities in the data display records based on the differences in the data processing process in any data display record.

8. The data display system based on large data stream processing according to claim 6, characterized in that: The abnormal feature processing module includes an abnormal behavior analysis unit and an abnormal feature extraction unit; The abnormal behavior analysis unit is used to extract the data display results in any abnormal display record, analyze the change process of the abnormal display results, and confirm the abnormal behavior in the abnormal display record; the abnormal feature extraction unit is used to extract the abnormal features of the abnormal display record based on the abnormal situation presented by the abnormal behavior.

9. The data display system based on large data stream processing according to claim 6, characterized in that: The display data correction module includes a feature inclusion analysis unit and an adjustment strategy processing unit; The feature inclusion analysis unit is used to analyze the inclusion of abnormal features in any data display record; the adjustment strategy processing unit is used to analyze the impact of the included abnormal features based on the abnormal identification differences between any adjacent data display records, and formulate a data adjustment strategy to correct any abnormal display record.

10. The data display system based on large data stream processing according to claim 6, characterized in that: The real-time display analysis module includes a real-time abnormality judgment unit and a real-time abnormality correction unit; The real-time anomaly determination unit is used to identify anomalies in the data processing process recorded by the real-time display record generated in real time; The abnormal real-time correction unit is used to send abnormal reminders and call corresponding data adjustment strategies to perform real-time corrections when the recognition results are abnormal.

Images