Real-time data analysis system based on big data

Abstract

The invention relates to the technical field of real-time data analysis, in particular to a real-time data analysis system based on big data, which comprises an event capture module, a trend identification module, a priority processing module and a data isolation module. According to the method, the data stream is monitored in real time, the timestamps, the change amplitude and the frequency of the data points are extracted, the timeliness and the accuracy of data processing are enhanced, real-time change analysis is carried out on the data, instant understanding of the data stream is enhanced through key change marks, the response speed of market changes is increased, and the market competitiveness is improved. By analyzing the detailed background and development trend of key events, the prediction accuracy and the operation foresight are improved, resources are dynamically reconfigured according to the priority of the events, the key events are ensured to be processed preferentially, the resource utilization efficiency and the operation response speed are greatly improved, data isolation guarantees the concentration and safety of high-priority event processing, and the method is suitable for popularization and application. And the data processing efficiency is obviously improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of real-time data analysis, and in particular to a real-time data analysis system based on big data. BACKGROUND

[0002] The technical field of real-time data analysis involves using advanced data processing techniques to process, analyze, and extract useful information from continuously generated data streams in real-time or near real-time, widely used in various industries, including financial services, telecommunications, retail, healthcare, and manufacturing. Core technologies include stream data processing, event-driven programming, complex event processing (CEP), and machine learning algorithms, aiming to identify patterns, predict trends, and trigger business processes in real-time, support enterprises to make decisions in real-time, optimize operational efficiency, enhance user experience, and effectively respond to rapidly changing market conditions.

[0003] Among them, the real-time data analysis system based on big data is a system for processing and analyzing large-scale data sets to provide immediate insights and decision support, usually integrating data collection, storage, processing, and analysis functions, capable of processing massive data streams from multiple data sources, and its main uses include but are not limited to market trend analysis, real-time monitoring, personalized recommendations, preventive maintenance, and real-time fraud detection, enabling enterprises to respond to market changes in real-time, improve business efficiency and customer satisfaction.

[0004] Although the existing technology is widely used in real-time data processing, it often faces problems such as uneven resource allocation and response delay when processing large-scale data, relies heavily on preset processes, and lacks flexible response capabilities to sudden events in data streams, especially when data traffic is large, processing efficiency is limited. Lack of effective data isolation mechanism, performance bottleneck under high resource load, affecting overall system performance and data security. For example, in the financial services industry, important events often cannot be processed in time during critical moments, resulting in the loss of important business opportunities or insufficient response to market dynamics, indicating its shortcomings in responding to rapidly changing market conditions. SUMMARY

[0005] The purpose of the present application is to solve the shortcomings in the prior art and propose a real-time data analysis system based on big data.

[0006] In order to achieve the above-mentioned purpose, the present application adopts the following technical scheme: a real-time data analysis system based on big data includes: The event capture module continuously monitors the data stream through the data interface, collects the timestamp, change amplitude and frequency information of the data points in real time, performs change analysis, identifies and marks the key change data points, and organizes and generates key event data; The trend identification module receives the key event data, performs preliminary screening on the data, extracts time points and variation amplitudes of the key events, analyzes backgrounds and prior and posterior relations of the key events, evaluates potential development directions of the key events according to an analysis result, determines an influence range and performs classification processing, and generates an event trend analysis result; The priority processing module utilizes the event trend analysis result, evaluates influence and urgency of the key events, and performs priority sorting of the key events, reconfigures resources according to a priority sorting result, allocates resources and processing time according to priority levels, matches processing requirements of different key events, and outputs a priority processing queue; The data isolation module performs data and resource isolation according to the priority processing queue, sets independent data channels and storage spaces for each high-priority event, performs physical and logical isolation, monitors resource utilization in real time according to an isolation result, dynamically adjusts resource allocation, updates data processing efficiency and response speed, and generates a data view after isolation.

[0007] As a further scheme of the present application, the key event data acquisition step is: Real-time monitoring of data flow through a data interface, extraction of time stamp, variation amplitude and frequency parameters in the data flow, verification of information integrity, and generation of an initial data point set; Based on the initial data point set, variation amplitude and frequency parameters are extracted, and a formula is used: ; Calculation of the comprehensive variation value of each data point , screening through setting of a variation threshold value, and obtaining of a data point set with significant variation, wherein, is the variation amplitude of the data point, is the frequency of the data point; Time series analysis on the data point set with significant variation, screening of key data points conforming to variation rules through judgment of continuity of adjacent time stamps, synchronous marking of the key data points as key event points, and arrangement to generate key event data.

[0008] As a further scheme of the present application, the background and prior and posterior relation analysis step is: Calling the key event data, ascending arrangement of the data with time points as a sorting reference, calculation of time differences between adjacent time points, and corresponding of variation amplitude information to the arranged time points, generation of associated data; Analysis on the associated data, screening of short-time interval events through setting of a time difference threshold value, and segment classification of the variation amplitude, association of significant variation time points with adjacent events, and generation of continuous key event data; According to the continuous key event data, background contact analysis is performed on each pair of events, the distribution mode and the relative change value of the variation amplitude at the previous and subsequent time points are quantified, and the formula is used: ; The composite contact value of each pair of events is calculated , and the event background contact analysis result is generated, wherein, is the time point interval, is the variation amplitude of the previous and subsequent events.

[0009] As a further scheme of the present application, the event trend analysis result acquisition step is: According to the event background contact analysis result, the starting time point, the ending time point and the internal variation amplitude sequence of each key event chain are extracted, the time span of the event chain is calculated, and statistical analysis is performed on the internal variation amplitude sequence to generate event chain statistical data; Based on the time span and variation amplitude characteristics in the event chain statistical data, the potential development direction of the event is evaluated, the relationship between the time span and the variation amplitude change trend is analyzed, and the formula is used: ; The variation direction value of the event chain is calculated , the event chain with a positive variation value is marked as positive development, the event chain with a negative variation value is marked as negative development, and the event chain with a zero variation value is marked as a static state, and the key event chain data with development direction marking is generated, wherein, and are the variation amplitudes at the end and the beginning of the event chain, and are the time points at the end and the beginning of the event chain; The key event chain data with development direction marking is called, the fluctuation type is divided according to the fluctuation range of the variation amplitude and the time span, combined with the development direction marking, and the event trend analysis result is generated.

[0010] As a further scheme of the present application, the priority sorting step is: Based on the event trend analysis result, the time span, the fluctuation range and the development direction marking of each event are extracted, the persistence of the event is measured, the influence amplitude of the event is reflected, the trend change characteristics of the event are revealed, and the preliminary event feature data is generated; Based on the preliminary event feature data, quantitative analysis is performed on the time span, the fluctuation range and the development direction marking, the influence of the event is evaluated, and the numerical value of the fluctuation range is analyzed, the event with a fluctuation range exceeding the fluctuation threshold is marked as a high urgency event, and the key data with urgency marking is generated; According to the key data marked with the emergency label, events marked as high emergency are prioritized according to the emergency label as the main sorting basis, and events with the same emergency label are arranged in order according to the influence from high to low, to generate a priority sorting result.

[0011] As a further scheme of the present application, the step of obtaining the priority processing queue is: Based on the priority sorting result, the priority of each event, the emergency label, and the comprehensive influence value are extracted, the number and distribution of high-priority events are counted, the processing demand and the expected processing time of each event are extracted, and event resource demand data is generated; According to the event resource demand data, in combination with the total amount of existing resources and processing capacity, a resource allocation scheme is planned, and it is checked whether the resource allocation meets the event processing demand, the processing time is matched one by one according to the emergency of the event, and a preliminary resource allocation scheme is generated; The preliminary resource allocation scheme is called to optimize the resource allocation result, the resource allocation of high-priority events is increased, the resource allocation of low-priority events is reduced, and all events are rearranged according to the priority to generate a priority processing queue.

[0012] As a further scheme of the present application, the step of physically and logically isolating is: The priority processing queue is called to extract the resource demand and data flow characteristics of each high-priority event, record the data source, target path, data type and data volume of the event, analyze the cross data path and resource usage, and generate a data flow and resource demand characteristic list; According to the data flow and resource demand characteristic list, each event is physically and logically isolated, an independent data transmission channel is established, an independent bandwidth, transmission rate and transmission direction are allocated to each channel, the storage demand of each event is checked, and an independent event data channel and storage configuration are generated; Based on the independent event data channel and storage configuration, a mapping relationship between event resource usage and isolation state is established, the transmission path of each event is logically bound to the storage operation, and the formula: ; The isolation degree of resource demand is calculated , and it is checked whether all high-priority events meet the independence requirement to generate a high-priority event isolation scheme, wherein, represents the data channel bandwidth demand of the event, represents the number of nodes on the transmission path, represents the allocated storage space demand.

[0013] As a further scheme of the present application, the step of obtaining the isolated data view is: The high-priority event isolation scheme is called to monitor the bandwidth usage, data transmission rate and storage space occupation of each independent data channel in real time, periodic statistics are performed to generate real-time resource utilization monitoring data; According to the resource utilization monitoring data, the resource usage state of each event is analyzed, the bandwidth resource of low-priority events is released and the storage space capacity is expanded to dynamically optimize resource allocation, and adjusted resource utilization configuration is generated; Based on the adjusted resource utilization configuration, the event transmission performance and storage state are re-compared with the real-time monitoring results, the change of resource utilization rate is analyzed, the data processing efficiency and response speed of each event are updated, and the isolated data view is integrated and generated.

[0014] Compared with the prior art, the present application has the advantages and positive effects that: In the present application, the time stamp, change amplitude and frequency of the data points are extracted by real-time monitoring of the data stream, the timeliness and accuracy of data processing are enhanced, real-time change analysis is performed on the data, and the immediate understanding of the data stream is enhanced through key change markers, the response speed to market changes is improved, the accuracy of prediction and the forward-looking of operation are improved through analysis of the detailed background and development trend of key events, resources are dynamically reconfigured according to event priority, key events are processed in priority, the resource utilization efficiency and operation response speed are greatly improved, data isolation ensures the focus and safety of high-priority event processing, and the data processing efficiency is significantly improved. BRIEF DESCRIPTION OF DRAWINGS

[0015] Figure 1 The system flowchart of the present application is shown in Figure 1; Figure 2 The key event data acquisition flowchart of the present application is shown in Figure 2; Figure 3 The background and contact analysis flowchart of the present application is shown in Figure 3; Figure 4 The event trend analysis result acquisition flowchart of the present application is shown in Figure 4; Figure 5 The priority sorting flowchart of the present application is shown in Figure 5; Figure 6 The priority processing queue acquisition flowchart of the present application is shown in Figure 6; Figure 7 The physical and logical isolation flowchart of the present application is shown in Figure 7; Figure 8 The isolated data view acquisition flowchart of the present application is shown in Figure 8. DETAILED DESCRIPTION

[0016] In order to make the objects, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and not to limit the present application.

[0017] In the description of the present application, it should be understood that the terms "length", "width", "upper", "lower", "front", "rear", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer" and the like indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings, and are only for the convenience of describing the present application and simplifying the description, and do not indicate or imply that the device or element referred to must have a particular orientation, be constructed and operated in a particular orientation, and therefore cannot be understood as a limitation on the present application. In addition, in the description of the present application, the meaning of "a plurality of" is two or more, unless otherwise explicitly and specifically limited.

[0018] Please refer to Figure 1 A real-time data analysis system based on big data includes: The event capture module continuously monitors the data stream through the data interface, collects the timestamp, change amplitude and frequency information of the data points in real time, performs change analysis, identifies the key change data points and marks them, and generates key event data by sorting and organizing; The trend identification module receives the key event data, performs preliminary screening on the data, extracts the time point and change amplitude of the key event, analyzes the background and the relationship before and after the key event, evaluates the potential development direction of the key event according to the analysis result, determines the influence range and performs classification processing, and generates the event trend analysis result; The priority processing module uses the event trend analysis result to evaluate the influence and urgency of the key event, and performs priority sorting on the key event. According to the priority sorting result, the resource is reconfigured, the resource and processing time are allocated according to the priority, the processing needs of different key events are matched, and the priority processing queue is output; The data isolation module performs data and resource isolation according to the priority processing queue, sets independent data channels and storage spaces for each high-priority event, performs physical and logical isolation, monitors the resource utilization in real time according to the isolation result, dynamically adjusts the resource allocation, updates the data processing efficiency and response speed, and generates the isolated data view.

[0019] The key event data includes timestamp, change amplitude and frequency information; the event trend analysis result includes the time point, change amplitude, development direction and influence range of the key event; the priority processing queue includes the priority sorting result, resource allocation result and processing time allocation record; the isolated data view includes independent data channels, storage spaces and resource utilization monitoring state records.

[0020] Referring to Figure 2 , the key event data acquisition step is: Real-time monitoring of data flow through the data interface, extracting the timestamp, change amplitude and frequency parameters in the data flow, checking the integrity of the information, and generating an initial data point set; Through the data interface, the real-time data stream is received, and the data stream is parsed one by one according to the arrival order. First, the original timestamp information in the data stream is extracted and used as the key sorting parameter. The original timestamp is processed to ensure that there is no duplicate record. Then the change amplitude information and frequency information corresponding to each data point are extracted. The change amplitude information is calculated by comparing the numerical difference between adjacent data points. The frequency information is obtained by calculating the number of repeated data points in a unit of time. For each data point, it needs to be checked whether it contains timestamp, change amplitude and frequency parameters. If any of them is missing, the data point will be marked as incomplete and excluded from the subsequent processing. After completing the integrity check, the remaining data points are sorted according to the timestamp to build the initial structure of the time series, and stored in the memory for further processing. The initial data point set with complete timestamp, change amplitude and frequency parameters is selected.

[0021] Based on the initial data point set, the change amplitude and frequency parameters are extracted, and the formula is: ; Calculate the comprehensive change value of each data point , filter by setting the change threshold to get the data point set with significant changes, where is the change amplitude of the data point, is the frequency of the data point; Suppose the change amplitude of a data point in the initial data point set is , and the frequency is .

[0022] Substitute the values of change amplitude and frequency into the formula:

[0023] Calculate the square of the change amplitude and the square of the frequency:

[0024] The result of the square sum is:

[0025] Take the square root to get:

[0026] The calculation result shows that the comprehensive change value of the data point is 6.71. The contribution of the change range and the frequency of the data point to the comprehensive change value is considered, wherein the contribution of the change range is greater, the contribution of the frequency is smaller, and according to the set change value threshold, for example, the threshold is set to 5.0, it is determined that Therefore, the data point is identified as a data point with significant change.

[0027] The time series analysis is performed on the set of data points with significant change, the key data points meeting the change rule are screened by judging the continuity of adjacent time stamps, the key data points are marked as key event points at the same time, and the key event data is generated by collating; The time series analysis is performed on the set of data points with significant change, the key data points meeting the change rule are screened by judging the continuity of adjacent time stamps, the key data points are marked as key event points at the same time, and the key event data is generated by collating;

[0028] Please refer to Figure 3 , the analysis steps of background and context are as follows: Call the key event data, arrange the data in ascending order with time point as the sorting reference, calculate the time difference between adjacent time points, and correspond the change range information with the sorted time points to generate associated data; After calling the key event data, the time point and variation amplitude information in the data are extracted, the time points are arranged in ascending order according to the order from small to large, all time points are compared with their adjacent time points in pairs, and the time interval data is generated by calculating the difference between the time points. The calculation method of the time interval is to directly subtract the previous time point from the next time point to form a time difference sequence. At the same time, the variation amplitude data corresponding to each time point is extracted, and a one-to-one correspondence is established between the variation amplitude data and the time point. The time interval sequence is checked to identify whether there are extreme values exceeding the normal range, and if abnormal values are found, they are marked as abnormal and removed from the data. For the variation amplitude data, the median and interquartile range of the data value are calculated point by point, and the abnormal values of the variation amplitude are identified by setting reasonable upper and lower limit ranges, for example, the values within 1.5 times the interquartile range of the median are considered normal, and the values exceeding the range are considered abnormal. For the abnormal variation amplitude values exceeding the range, the mean of the adjacent data points before and after them is used for correction processing to ensure data integrity. Finally, the time points, time intervals and variation amplitude data processed are arranged in a correlated form to form a complete correlated data set containing time points, time intervals and variation amplitudes.

[0029] The correlated data is analyzed, the short time interval events are screened by setting a time difference threshold, and the significant variation time points are associated with adjacent events to generate continuous key event data. The correlated data containing time points, time intervals and variation amplitudes is called, the time interval is used as the screening basis, and the time interval of each pair of adjacent time points is checked one by one to extract time point pairs with a time interval less than the set threshold. The data of the time point pairs are further processed, the corresponding variation amplitudes are extracted, the absolute difference of the variation amplitude of each pair of time points is calculated, and the absolute difference is compared with the set variation amplitude threshold. If the absolute difference exceeds the threshold, the time point pair is marked as a pair of significantly varying events. At the same time, all marked event pairs are checked for continuity to determine whether there are continuous significant variation time points, and if there are, these time points are classified into the same event chain to form a continuous key event chain. In the processing process, the basis for continuity judgment is whether the time interval between adjacent time points is less than the set time threshold and whether the variation amplitude is within the range of significant change. Finally, all time point chains that meet the continuity and significant variation conditions are arranged as continuous key event data, and the start time point, end time point and internal variation amplitude information of each event chain are summarized and recorded to generate a continuous key event data set for subsequent analysis.

[0030] According to the continuous key event data, the background contact analysis is performed on each pair of events, the distribution pattern of the time points before and after and the relative change value of the variation amplitude are quantified, and the formula is: ; Calculate the composite contact value of each pair of events , generate event background contact analysis results, wherein, is the time interval, is the variation range of the previous and subsequent events; Suppose in a continuous event data, the time point of event A is 10, the variation range is 6, the time point of event B is 13, and the variation range is 4.

[0031] Calculate the time interval of the time point:

[0032] Calculate the absolute difference of the variation range:

[0033] Calculate the square root of the sum of squares of the variation range:

[0034] Substitute the above calculation results into the formula:

[0035] The results show that the composite contact value between event A and event B is 0.693, indicating the contact strength of the time interval and the variation range between the two. The correlation between events can be further quantified by the composite contact value, so as to optimize the background analysis results and provide more accurate input for subsequent trend analysis.

[0036] Please refer to Figure 4 , the steps for obtaining event trend analysis results are: According to the event background contact analysis results, extract the starting time point, ending time point and internal variation range sequence of each key event chain, calculate the time span of the event chain, and perform statistical analysis on the internal variation range sequence to generate event chain statistical data; Call the continuous key event data generated before, extract the starting time point and ending time point of each key event chain, calculate the time span by subtracting the starting time point from the ending time point, and record the time span of each event chain. At the same time, statistical analysis is performed on the variation range sequence inside each event chain. First, calculate the average value of all variation range values, then calculate the fluctuation range by calculating the difference between the maximum value and the minimum value of the variation range values. In addition, check the abnormal data in the variation range values, eliminate the extreme values that exceed the upper and lower quartile range, and recalculate the statistical results. After completing these operations, combine the time span, average variation range and fluctuation range, and summarize and organize the results in units of event chains to generate event chain statistical data containing time span, average variation range and fluctuation range.

[0037] Based on the time span and variation amplitude characteristics in the event chain statistical data, the potential development direction of the event is evaluated, the relationship between the time span and the variation amplitude change trend is analyzed, and the formula is adopted: ; Calculate the variation direction value of the event chain , mark the event chain with positive variation value as positive development, negative as negative development, and zero as static state, generate key event chain data with development direction mark, wherein, and are the variation amplitudes of the end and start of the event chain, and are the time points of the end and start of the event chain; Suppose the start time point of an event chain is 10, the end time point is 15, the start variation amplitude is 6, and the end variation amplitude is 10.

[0038] Calculate the time span:

[0039] Calculate the variation amplitude change:

[0040] Substitute the above results into the formula:

[0041] The results show that the variation direction value of the event chain is 0.8, indicating that the event chain belongs to the positive development trend, and the variation amplitude shows the characteristics of continuous increase with time. This value can further provide clear trend marking information for subsequent classification and trend analysis, and lay the foundation for dynamic analysis of the development of the event chain.

[0042] Call the key event chain data with development direction mark, according to the fluctuation range of variation amplitude and time span, combined with the development direction mark, divide the fluctuation type, and generate the event trend analysis result; The key event chain data with development direction markers is called, and each event chain is classified according to the fluctuation range and time span of the variation amplitude. The fluctuation range is divided into high fluctuation and low fluctuation according to the preset threshold, and the event chain is further classified in combination with the value of the development direction marker. The event chain with a positive development direction marker is divided into positive high fluctuation and positive low fluctuation according to the fluctuation range, and the event chain with a negative development direction marker is also divided into negative high fluctuation and negative low fluctuation. In addition, the event chain with a development direction marker of zero is classified as a static state category. After classification, the event chains in each category are summarized to record the time span distribution, fluctuation range and development direction marker value, and finally the event trend analysis result containing the time span, fluctuation range, development direction marker and classification information is generated.

[0043] Please refer to Figure 5 The priority sorting step is: Based on the event trend analysis result, the time span, fluctuation range and development direction marker of each event are extracted to measure the persistence of the event, reflect the impact amplitude of the event, and reveal the trend change characteristics of the event, and preliminary event feature data is generated; The time span, fluctuation range and development direction marker of each event are extracted from the event trend analysis result. First, the starting time point and the ending time point of the event are extracted, and the time span is calculated by subtracting the starting time point from the ending time point. All time span data need to be checked to ensure the rationality of the value, for example, check whether the time span is a non-negative value, if an abnormal value is found, the starting and ending time points need to be manually corrected or data interpolated. For the fluctuation range, the maximum value and the minimum value of the event are extracted, and the difference between the two is calculated to obtain the fluctuation range. During the calculation process, extreme values need to be checked to avoid distortion of the fluctuation range calculation result due to data anomalies. After removing the extreme values, the fluctuation range value is recalculated. The development direction marker is directly extracted and classified according to the trend classification, and the marker value is divided into positive, negative and static three categories to ensure that the direction marker of each event is clear and the data is complete. Finally, the time span, fluctuation range and development direction marker are integrated into the event feature parameter set to provide complete data support for the subsequent evaluation step.

[0044] Based on the preliminary event feature data, the time span, fluctuation range and development direction marker are quantitatively analyzed to evaluate the influence of the event, and the numerical value of the fluctuation range is analyzed. Events with fluctuation range exceeding the fluctuation threshold are marked as high-emergency events, and key data with emergency markers are generated; On the basis of the event characteristic parameter set, the persistence and influence of the event are evaluated in detail. First, according to the time span parameter, the persistence of each event is quantitatively analyzed, and events with a larger time span value are marked as high-persistence events. According to the set persistence classification standard, the events are classified, for example, divided into three categories of high, medium and low according to the interval range of the time span, to ensure the continuity and rationality of the classification. For the fluctuation range parameter, further statistical analysis is performed, and the fluctuation range value is compared with the set influence threshold one by one, and events with a fluctuation range higher than the set threshold are selected and marked as high-influence events. At the same time, events with a fluctuation range lower than the threshold are marked as low-influence events, to ensure that each event has a clear influence classification mark. Finally, according to the value of the development direction mark, the trend of each event is classified, and events marked as positive are classified as positive development category, events marked as negative are classified as negative development category, and events with a mark value close to zero are classified as static category. Through the combination of these classification marks, for example, high-persistence and high-influence positive development events, basic data for priority ranking are provided.

[0045] According to the key data with emergency marks, the events marked as high emergency are prioritized according to the emergency mark as the main sorting basis, and the events with the same emergency mark are arranged in order according to the influence from high to low, to generate the priority ranking result; Through the key data with emergency marks, the events are sorted in detail according to the emergency and priority requirements. First, all events are grouped according to the emergency mark, and events marked as high emergency are prioritized, and the influence classification of these events is checked. High-emergency and high-influence events are extracted separately and sorted according to the priority. For events with the same emergency, the time span is used as the sorting basis, and events with longer time span are prioritized, because events with longer time span usually have higher influence requirements for subsequent processing resource allocation. Finally, in the case of the same emergency and time span, the value of the development direction mark is used to further refine the sorting, and the positive development events are prioritized, followed by the static events, and finally the negative development events. During the sorting process, the sorting basis of each step is verified to ensure that the sorting result meets the preset emergency and priority rules, and a key event list sorted according to the priority is generated to provide clear guidance basis for subsequent resource allocation and processing order.

[0046] Please refer to Figure 6 The acquisition step of the priority processing queue is: Based on the priority ranking result, the priority, emergency mark and comprehensive influence value of each event are extracted, the number and distribution of high-priority events are counted, the processing demand and expected processing time of each event are extracted, and event resource demand data is generated; The priority ranking result is called, from which the priority, urgency label and comprehensive influence value of the event are extracted, and these parameters are taken as the basis for resource configuration. First, events with high priority are screened, and the urgency label of the event is extracted, and grouped according to the high and low urgency. For each group of events, record the processing requirements of each event, such as the required resource type, resource quantity and expected processing time. In combination with the processing requirements, the total amount of processing resources is counted for all events in the event group, and the processing time requirement is integrated into a time period distribution to form a complete resource demand framework. In this way, a set of event resource demand data including priority, urgency, comprehensive influence and resource demand is generated to provide basic data support for resource allocation.

[0047] According to the event resource demand data, in combination with the total amount of existing resources and processing capacity, the resource allocation scheme is planned, and whether the resource allocation meets the event processing requirements is checked, the processing time is matched one by one according to the urgency of the event, and a preliminary resource configuration scheme is generated. According to the event resource demand data, in combination with the total amount of existing resources and processing capacity, the resource allocation scheme for the event is planned. First, check the total amount and type of the available resources, and compare the resource demand of each event with the existing resources one by one, and preferentially allocate the required resources for high-priority and high-urgency events. In the allocation process, record the type, quantity and expected occupation time of the allocated resources, and ensure that the resource demand of each high-priority event is met. If there is a shortage of resources, adjust the resource allocation strategy through priority ranking, give priority to the resource allocation of high-priority events, and appropriately reduce the resource allocation of medium and low-priority events, to ensure that the processing requirements of high-priority events can be completed on time, and generate a preliminary resource configuration scheme including the resource allocation of each event, the expected processing time and the adjusted resource demand.

[0048] The preliminary resource configuration scheme is called, and the resource allocation result is optimized and checked, the resource allocation of high-priority events is increased, the resource allocation of medium and low-priority events is reduced, and all events are rearranged according to the high and low priority to generate a priority processing queue. Call the preliminary resource allocation scheme, optimize the resource allocation result, and ensure that each high-priority event is allocated sufficient resources and time to meet its processing needs. During the optimization check, the types and quantities of allocated resources are checked one by one, and the allocated time period is checked to see if it matches the processing time required by the event. If unreasonable resource and time allocation is found, fine-tune it, for example, allocate more continuous processing time for high-priority events in the time period, and re-allocate resources for low-priority events to reduce the types and time periods of resources they occupy. After optimization, all events are rearranged according to priority, and the allocation results are integrated into a complete priority processing queue.

[0049] Please refer to Figure 7 The physical and logical isolation steps are: Call the priority processing queue, extract the resource requirements and data flow characteristics of each high-priority event, record the data source, target path, data type and data volume of the event, analyze the cross-data path and resource usage, and generate a list of data flow and resource requirement characteristics; Call the priority processing queue, extract the data flow characteristics of each high-priority event, including data source, target path, data type and data volume, and record the data flow characteristics one by one. First, analyze each event's source and target path pair by pair to confirm whether there is a path intersection, mark out potential conflict paths and record detailed characteristics by recording the transmission direction, bandwidth requirement and node number of each path. Then, process the data type and data volume, classify and aggregate the data according to size and structural characteristics (such as text, image, video, etc.), and associate them with the transmission path. In this way, a detailed characteristics list containing data flow direction, bandwidth requirement, node distribution and transmission content type is generated as the basis data for subsequent data channel setting.

[0050] According to the data flow and resource requirement characteristics list, perform physical and logical data isolation for each event, establish independent data transmission channels, allocate independent bandwidth, transmission rate and transmission direction for each channel, check the storage requirements of each event, allocate independent storage space, and generate independent event data channels and storage configuration; According to the data flow characteristics list, physical and logical isolation is performed for each high-priority event, including transmission channel division and storage space independence setting. First, independent physical transmission channels are divided one by one according to the transmission direction of the data flow and the node requirements, and through the setting of dedicated bandwidth, node mapping and routing path, the data transmission of each event is ensured to be mutually independent. After physical isolation, the storage requirements of each event are independently allocated, and through checking the data volume and type of the event, appropriate physical storage areas are allocated, and logical access permissions are set to ensure the logical isolation of the storage space between different events. An independent data channel and storage configuration scheme is generated, in which each event is equipped with a dedicated data channel and logical storage area to ensure the integrity and effectiveness of event resource isolation.

[0051] Based on the independent event data channel and storage configuration, a mapping relationship between event resource usage and isolation state is established, and the transmission path and storage operation of each event are logically bound. The formula is: ; Calculate the resource requirement isolation degree , and check whether all high-priority events meet the independence requirement to generate a high-priority event isolation scheme, wherein, represents the data channel bandwidth requirement of the event, represents the number of nodes on the transmission path, represents the allocated storage space requirement; The data channel bandwidth requirement of an event is 200 Mbps, there are 4 nodes on the transmission path, and the allocated storage space requirement is 50 GB: Calculate the bandwidth to node ratio:

[0052] Add the storage space requirement:

[0053] The results show that for the physical and logical isolation setting of this event, the resource requirement isolation degree is 100 units, indicating that the transmission channel and storage resource setting can meet the event independence requirement. This value can further verify the rationality of resource allocation and provide a quantitative basis for the next step of isolation optimization.

[0054] Please refer to Figure 8 , the steps for obtaining the isolated data view are: Call the high-priority event isolation scheme to monitor the bandwidth usage, data transmission rate and storage space occupation state of each independent data channel in real time, perform periodic statistics, and generate real-time resource utilization monitoring data; The high-priority event isolation scheme is called to monitor the resource utilization of each high-priority event in real time, specifically analyzing the bandwidth usage, transmission rate and storage space occupation status of the independent data channel. First, the bandwidth usage is recorded continuously and periodically. By counting the real-time data transmission volume of each data channel, the actual utilization rate of the bandwidth is calculated and compared with the preset bandwidth capacity of the data channel to find the bandwidth underutilization or overload. Subsequently, the transmission rate is analyzed. By analyzing the change of the data volume transmitted per second, combined with the point-by-point record of the transmission delay, the rate fluctuation amplitude in each time period is analyzed. For the monitoring of the storage space, the storage capacity occupation ratio of each event record is recorded, combined with the historical growth rate data, the storage saturation time node is predicted, and a resource utilization monitoring result including bandwidth, rate and storage status is generated by integrating parameters, providing accurate basis for subsequent adjustment of resource allocation.

[0055] According to the resource utilization monitoring data, the resource usage status of each event is analyzed, and the resource allocation is dynamically optimized by releasing the bandwidth resources of low-priority events and expanding the storage space capacity, to generate the adjusted resource utilization configuration; According to the monitoring result, the resource utilization status is identified, and the events with bandwidth utilization rate exceeding the preset load, transmission rate fluctuation being obvious or storage capacity being close to saturation are extracted and marked as resource high-load events. The resource allocation of these events is dynamically adjusted. The specific adjustment operation includes releasing part of the bandwidth and preferentially allocating it to the high-load events by reevaluating the resource distribution of low-priority events, and enabling additional storage units according to the storage capacity demand to reconfigure the data storage path of the high-load events and reduce the pressure on a single storage space. In addition, the adjusted data transmission rate needs to be rebalanced by reallocating the reserved resource bandwidth to optimize the channel pressure and smooth the rate fluctuation. The adjusted resource usage result is compared with the original monitoring data to form the adjusted resource utilization configuration.

[0056] Based on the adjusted resource utilization configuration, the event transmission performance and storage status are compared with the real-time monitoring result to analyze the change of the resource utilization rate, update the data processing efficiency and response speed of each event, and integrate to generate the isolated data view; In combination with the adjusted resource utilization configuration, the resource utilization before and after the adjustment of the high-load event is compared, and the resource utilization efficiency and data processing speed of the event are further updated. First, the resource utilization efficiency of each high-priority event is re-counted, and the performance improvement effect after the resource adjustment is analyzed through dynamic comparison of bandwidth utilization rate, transmission delay change and storage capacity occupancy rate. Secondly, combined with the processing time record after the adjustment of the event, the data processing time and response speed are segmented and counted, the completion time of each data processing task is averaged, and the overall efficiency after optimization is obtained. Finally, the adjusted resource usage state, transmission efficiency and storage condition of all high-priority events are integrated into a unified view, and an isolated data view containing resource usage distribution and processing performance state is output, forming a global resource utilization and processing effect chart, which provides an intuitive reference for subsequent resource configuration.

[0057] The above is only a preferred embodiment of the present application, and does not limit the present application in other forms. Any skilled person in the art can use the disclosed technical content to make changes or modifications to equivalent embodiments applied to other fields, but any simple modification, equivalent change and modification made to the above embodiments without departing from the technical solution content of the present application, according to the technical essence of the present application, still belongs to the protection scope of the technical solution of the present application.

Claims

1. A real-time data analysis system based on big data, characterized in that: The system comprises: The event capture module continuously monitors data streams through data interfaces, collects the timestamps, change amplitudes, and frequency information of data points in real time, performs change analysis, identifies and marks key change data points, and organizes and generates key event data. The trend identification module receives the key event data, performs preliminary screening on the data, extracts the time points and fluctuation ranges of key events, analyzes the background and connections of key events, evaluates the potential development direction of key events based on the analysis results, determines the scope of impact and classifies the data, and generates event trend analysis results; The priority processing module uses the event trend analysis results to evaluate the impact and urgency of key events, and prioritizes the key events. It reallocates resources based on the priority ranking results, allocates resources and processing time according to priority, matches the processing requirements of different key events, and outputs a priority processing queue; The data isolation module isolates data and resources according to the priority processing queue, sets up independent data channels and storage space for each high-priority event, performs physical and logical isolation, monitors resource utilization in real time according to the isolation results, dynamically adjusts resource allocation, updates data processing efficiency and response speed, and generates an isolated data view.

2. The real-time data analysis system based on big data according to claim 1, characterized in that: The steps for obtaining the key event data are: Monitor data streams in real time through data interfaces, extract timestamps, change amplitudes, and frequency parameters from the data streams, verify the integrity of the information, and generate an initial set of data points; Based on the initial data point set, the variation amplitude and frequency parameters are extracted using the formula: ； Calculate the comprehensive change value for each data point , by setting the change threshold for screening, we can get a set of data points with significant changes, where is the variation of the data point, is the frequency of the data points; Perform time series analysis on the set of data points with significant changes, screen key data points that conform to the change rules by judging the continuity of adjacent timestamps, mark the key data points as key event points simultaneously, and organize and generate key event data.

3. The real-time data analysis system based on big data according to claim 2, characterized in that: The analysis steps of the background and context are: Calling the key event data, arranging the data in ascending order based on the time point as the sorting basis, calculating the time difference between adjacent time points, and matching the change amplitude information with the sorted time points to generate related data; Analyze the associated data, filter out short time interval events by setting a time difference threshold, and classify the change amplitude into segments, associate the significant change time points with adjacent events, and generate continuous key event data; Based on the continuous key event data, a background connection analysis is performed on each pair of events, and the relative change values ​​of the distribution pattern and the amplitude of the change at the previous and next time points are quantified using the formula: ； Calculate the composite connection value for each pair of events , generate event background connection analysis results, where, is the time point interval, is the magnitude of change between the preceding and following events.

4. The real-time data analysis system based on big data according to claim 3, characterized in that: The steps for obtaining the event trend analysis results are: Based on the event background connection analysis results, extract the starting time point, ending time point and internal variation amplitude sequence of each key event chain, calculate the time span of the event chain, and perform statistical analysis on the internal variation amplitude sequence to generate event chain statistical data; Based on the time span and change amplitude characteristics in the event chain statistics, the potential development direction of the event is evaluated and the relationship between the time span and the change amplitude trend is analyzed. The formula is used: ； Calculate the change direction value of the event chain , mark the event chain with positive change value as positive development, negative as negative development, and zero as static state, generate key event chain data with development direction mark, where, and are the fluctuation ranges at the end and start of the event chain, and are the time points at which the event chain ends and starts, respectively; The key event chain data with development direction marks are called, and fluctuation types are divided according to the fluctuation range and time span of the change amplitude in combination with the development direction marks to generate event trend analysis results.

5. The real-time data analysis system based on big data according to claim 4, characterized in that: The priority sorting steps are: Based on the event trend analysis results, extract the time span, fluctuation range and development direction markers of each event, measure the persistence of the event, reflect the impact of the event, reveal the trend change characteristics of the event, and generate preliminary event feature data; Based on the preliminary event feature data, quantitative analysis is performed on the time span, fluctuation range, and development direction markers to assess the impact of the event. The numerical value of the fluctuation range is analyzed, and events with fluctuation ranges exceeding the fluctuation threshold are marked as high-urgency events, generating key data with urgency markers. According to the key data with urgency mark, the urgency mark is used as the main sorting basis, and events marked as high urgency are prioritized. At the same time, events with the same urgency mark are arranged in order from high to low according to their influence to generate a priority sorting result.

6. The real-time data analysis system based on big data according to claim 5, characterized in that: The steps for obtaining the priority processing queue are: Based on the priority sorting results, extract the priority, urgency mark and comprehensive impact value of each event, count the number and distribution of high-priority events, extract the processing requirements and estimated processing time of each event, and generate event resource demand data; Based on the event resource demand data, combined with the total available resources and processing capacity, a resource allocation plan is planned. At the same time, whether the resource allocation meets the event processing requirements is checked. The processing time is matched one by one according to the urgency of the event to generate a preliminary resource allocation plan. The preliminary resource allocation plan is called to optimize and check the resource allocation results, increase the resource allocation for high-priority events, reduce the resource allocation for medium and low-priority events, rearrange all events according to their priority, and organize and generate a priority processing queue.

7. The real-time data analysis system based on big data according to claim 6, characterized in that: The physical and logical isolation steps are: Invoke the priority processing queue, extract resource requirements and data flow characteristics of each high-priority event, record the data source, target path, data type and data volume of the event, analyze cross-data paths and resource usage, and generate a list of data flow and resource requirement characteristics; Based on the data flow and resource requirement characteristics, isolate data at the physical and logical levels for each event, establish independent data transmission channels, allocate independent bandwidth, transmission rate, and transmission direction for each channel, check the storage requirements of each event, allocate independent storage space, and generate independent event data channels and storage configurations; Based on the independent event data channel and storage configuration, a mapping relationship between event resource usage and isolation status is established, and the transmission path of each event is logically bound to the storage operation using the formula: ； Computing resource demand isolation , and check whether all high-priority events meet the independence requirements, and generate a high-priority event isolation plan, where, The data channel bandwidth requirement for the event, Indicates the number of nodes on the transmission path, Indicates the allocated storage space requirements.

8. The real-time data analysis system based on big data according to claim 7, characterized in that: The steps for obtaining the isolated data view are: Invoke the high-priority event isolation solution to monitor bandwidth usage, data transmission rate, and storage space occupancy of each independent data channel in real time, perform periodic statistics, and generate real-time resource utilization monitoring data; Analyzing the resource usage status of each event based on the resource utilization monitoring data, dynamically optimizing resource allocation by releasing bandwidth resources for low-priority events and expanding storage space capacity, and generating an adjusted resource utilization configuration; Based on the adjusted resource utilization configuration, the event transmission performance and storage status are re-compared with the real-time monitoring results, the changes in resource utilization are analyzed, the data processing efficiency and response speed of each event are updated, and the isolated data view is integrated and generated.

Images