A business big data stream processing system, method and medium based on feature analysis
By introducing feature analysis modules into the service big data stream processing system, the problem of low accuracy in the existing technology of service big data stream processing is solved, and more accurate data feature extraction and processing strategies are realized, thus improving network performance and service quality are improved. .
Patent Information
- Application Number
- CN202311315677.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-10-11
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2043-10-11
AI Technical Summary
The existing business big data stream processing technology analyzes data within a real-time time range, but fails to effectively process the characteristics of the data, resulting in low accuracy of business big data stream processing.
Adopt a service big data stream processing system based on feature analysis, including heterogeneous data fusion module, business feature extraction module, business data block generation module, business feature analysis diagram generation module and business processing status determination module, through real-time target business data classification , heterogeneous fusion, independent feature component analysis, flow density and bandwidth calculation, bidirectional sliding window and virtual transmission path generation, forming multi-dimensional features and processing strategies to improve processing accuracy.
By extracting more accurate business data features and generating multi-dimensional features, network performance and service quality can be improved, real-time business data can be more comprehensively analyzed and processed, timely discovering and solving potential problems, and improving data transmission efficiency and processing accuracy.
Smart Images

Figure CN117390069B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of data processing technology, and in particular to a business big data stream processing system, method and medium based on feature analysis. Background Art
[0002] With the continuous development and improvement of big data technology, including distributed computing, storage, data mining and machine learning technologies, processing large-scale data streams has become more efficient and feasible, providing a strong foundation for business big data stream processing. However, in order to improve the accuracy of business big data stream processing, it is necessary to analyze the multi-dimensional characteristics of business big data in order to accurately process business big data streams.
[0003] The existing business big data stream processing technology is based on distributed message queues and stream processing engines to receive data streams in real time and perform real-time calculations and processing on data streams. In actual applications, the stream processing engine only analyzes all data within the real-time time range, without analyzing and processing the characteristics of the data, resulting in low accuracy in business big data stream processing. Summary of the invention
[0004] The present invention provides a business big data stream processing system, method and medium based on feature analysis, the main purpose of which is to solve the problem of low accuracy when processing business big data streams.
[0005] To achieve the above-mentioned purpose, the present invention provides a business big data stream processing system based on feature analysis, the system comprises a heterogeneous data fusion module, a business feature extraction module, a business data block generation module, a business feature analysis diagram generation module and a business processing state determination module, wherein:
[0006] The heterogeneous data fusion module is used to obtain real-time target business data, classify the real-time target business data according to the preset business time sequence to obtain business time sequence data, and fuse the business time sequence data through the preset heterogeneous fusion algorithm to obtain business heterogeneous fusion data;
[0007] The service feature extraction module is used to calculate the service independent features of the heterogeneous fusion data by using a preset independent feature component analysis algorithm, calculate the flow density of the heterogeneous fusion data according to the service independent features, and calculate the flow bandwidth of the heterogeneous fusion data according to the service independent features. When calculating the flow density of the heterogeneous fusion data according to the service independent features, the service feature extraction module is specifically used to:
[0008] The service data flow corresponding to the heterogeneous fusion data in the service independence feature is counted according to a preset time interval; the central time point of the heterogeneous fusion data in the time interval is calculated according to the service data flow; the flow density of the heterogeneous fusion data is calculated according to the central time point, wherein the flow density calculation formula is:
[0009]
[0010] Among them, H t is the traffic density at the tth moment, μ is the data traffic correction factor, arctg is the inverse tangent function, t is the time point, and T0 is the central time point;
[0011] The service data block generation module is used to generate a target service multi-dimensional feature of the real-time target service data according to the traffic density and the traffic bandwidth, and generate a real-time target service data block from the real-time target service data according to the target service multi-dimensional feature through a preset bidirectional sliding window;
[0012] The service characteristic analysis diagram generation module is used to generate a virtual transmission path of the real-time target service data according to the real-time target service data block and a preset virtual stream channel, generate a parallel data chain of the real-time target service data block according to the virtual transmission path, and generate a service characteristic analysis diagram through the real-time target service data block in the parallel data chain;
[0013] The business processing status determination module is used to generate a business big data stream processing strategy for the real-time target business data according to the business feature analysis diagram, and perform data processing on the real-time target business data according to the business big data stream processing strategy to obtain the business processing status.
[0014] Optionally, when the heterogeneous data fusion module classifies the real-time target service data according to a preset service time sequence to obtain service time sequence data, it is specifically used to:
[0015] Generate a target service identifier sequence according to the time period and trigger identifier of the real-time target service data;
[0016] Generate a timing comparison matrix of the real-time target service data according to a preset service timing and the target service identification sequence;
[0017] The business timing data is determined according to the timing characteristics of the timing comparison matrix.
[0018] Optionally, when the heterogeneous data fusion module fuses the service time series data through a preset heterogeneous fusion algorithm to obtain service heterogeneous fusion data, it is specifically used to:
[0019] Performing standardization processing on the business time series data to obtain business standardized time series data;
[0020] Performing data alignment processing on the business standardized time series data to obtain business aligned time series data;
[0021] The business time series data is fused by the following heterogeneous fusion algorithm to obtain business heterogeneous fusion data:
[0022] P m ={δ m1 ,δ m2 ,…,δ mi ,…,δ mn}
[0023] Among them, P m is the mth type of heterogeneous fusion data of the service time series data, δ mi is the i-th service data in the m-th service heterogeneous fusion data, δ mn It is the nth service data in the mth service heterogeneous fusion data.
[0024] Optionally, when the business feature extraction module calculates the business independent features of the heterogeneous fusion data using a preset independent feature component analysis algorithm, it is specifically used to:
[0025] The independent matrix of the heterogeneous fusion data is calculated using the independent characteristic component analysis algorithm described below:
[0026] G(w)=E{[g(w T x)] 2}-κ(w)
[0027] Wherein, G(w) is the objective function of the independent matrix w, E is the expected function, g is the non-Gaussian metric function, x is the heterogeneous fusion data, κ(w) is the orthogonal constraint term of the independent matrix w, and T is the transposed symbol;
[0028] Generate a service mixing matrix according to the independent matrix and the heterogeneous fusion data;
[0029] The service independence characteristic is determined according to the service mixing matrix.
[0030] Optionally, when the service feature extraction module calculates the flow bandwidth of the heterogeneous fusion data according to the service independent feature, it is specifically used to:
[0031] Determining a data impact factor of the heterogeneous fusion data according to the service independence characteristics;
[0032] The traffic bandwidth of the heterogeneous fusion data is calculated according to the service data traffic and the data impact factor, wherein the traffic bandwidth calculation formula is:
[0033]
[0034] Wherein, Z is the traffic bandwidth, F is the service data traffic, R is the data traffic statistics time, χ is the traffic bandwidth control correction factor, λ k is the kth data impact factor, τ k is the weight of the kth influencing factor, and K is the number of factors that influence the data.
[0035] Optionally, when the service data block generation module generates the target service multi-dimensional features of the real-time target service data according to the traffic density and the traffic bandwidth, it is specifically used to:
[0036] Determining the flow characteristics of the real-time target service data according to the flow density;
[0037] Determining resource utilization characteristics according to the traffic density and the traffic bandwidth;
[0038] Determining the service priority characteristics of the real-time target service data according to the ratio of the traffic density and the traffic bandwidth;
[0039] The traffic characteristics, the resource utilization characteristics and the service priority characteristics are spliced into target service multi-dimensional characteristics of the real-time target service data.
[0040] Optionally, when the service data block generation module generates a real-time target service data block from the real-time target service data according to the multi-dimensional characteristics of the target service through a preset bidirectional sliding window, it is specifically used to:
[0041] Determine the initial window size of the real-time target service data according to the multi-dimensional characteristics of the target service;
[0042] Moving the bidirectional sliding window forward according to the initial window size to obtain a forward service data block;
[0043] Reversely moving the bidirectional sliding window according to the initial window size to obtain a reverse service data block;
[0044] The forward service data block and the reverse data block are collected as the real-time target service data block.
[0045] Optionally, when the service characteristic analysis diagram generation module generates a virtual transmission path of the real-time target service data according to the real-time target service data block and a preset virtual flow channel, it is specifically used to:
[0046] Determining the service characteristics of the real-time target service data according to the block identifier of the real-time target service data block;
[0047] Extracting channel characteristic parameters of the virtual flow channel, and determining a virtual transmission level of the virtual flow channel according to the channel characteristic parameters;
[0048] Constructing a level mapping between the service priority corresponding to the service feature and the virtual transmission level;
[0049] A virtual transmission path for the real-time target service data is generated according to the level mapping.
[0050] In order to solve the above problems, the present invention also provides a method for processing business big data stream based on feature analysis, the method comprising:
[0051] Acquire real-time target business data, classify the real-time target business data according to preset business time sequence to obtain business time sequence data, and fuse the business time sequence data through a preset heterogeneous fusion algorithm to obtain business heterogeneous fusion data;
[0052] Calculate the service-independent features of the heterogeneous fusion data using a preset independent feature component analysis algorithm, calculate the traffic density of the heterogeneous fusion data according to the service-independent features, and calculate the traffic bandwidth of the heterogeneous fusion data according to the service-independent features;
[0053] Generate a target service multi-dimensional feature of the real-time target service data according to the traffic density and the traffic bandwidth, and generate a real-time target service data block from the real-time target service data according to the target service multi-dimensional feature through a preset bidirectional sliding window;
[0054] Generate a virtual transmission path for the real-time target service data according to the real-time target service data block and a preset virtual stream channel, generate a parallel data chain for the real-time target service data block according to the virtual transmission path, and generate a service feature analysis diagram through the real-time target service data block in the parallel data chain;
[0055] A business big data stream processing strategy for the real-time target business data is generated according to the business characteristic analysis diagram, and data processing is performed on the real-time target business data according to the business big data stream processing strategy to obtain a business processing status.
[0056] In order to solve the above problems, the present invention also provides a medium, in which at least one computer program is stored, and the at least one computer program is executed by a processor in an electronic device to implement the above-mentioned business big data stream processing method based on feature analysis.
[0057] The embodiment of the present invention classifies the real-time target service data, and then performs data heterogeneous fusion on the classified service time series data, which is conducive to obtaining more comprehensive fusion data and facilitating the subsequent extraction of more accurate service data features; extracting the service independent features of the heterogeneous fusion data can provide more detailed and higher-level data analysis, and then calculate the traffic density and traffic bandwidth of the heterogeneous fusion data according to the service independent features, and then generate multi-dimensional service features, which can more comprehensively describe the characteristics and requirements of the real-time target service data to improve the performance and service quality of the network; generating real-time target service data blocks according to the multi-dimensional features of the target service, and using the data inside and outside the window for more comprehensive analysis and calculation, thereby improving the efficiency of generating real-time target service data blocks; generating virtual transmission paths according to the real-time target service data blocks and virtual flow channels, and then generating parallel data chains according to the virtual transmission paths, which is conducive to flexible adjustment of data transmission to meet the transmission requirements of the real-time target service data; generating a service big data stream processing strategy according to the service feature analysis diagram, and then determining the service processing status, can timely understand the progress of service data processing, timely discover and solve potential problems, and make corresponding decisions or adjustments according to the processing status. Therefore, the business big data stream processing system, method and medium based on feature analysis proposed in the present invention can solve the problem of low accuracy when processing business big data streams. BRIEF DESCRIPTION OF THE DRAWINGS
[0058] Figure 1 A schematic diagram of a process flow of a business big data stream processing system based on feature analysis provided by an embodiment of the present invention;
[0059] Figure 2 A schematic diagram of a flow chart of an operation method of a business big data stream processing system based on feature analysis provided by an embodiment of the present invention;
[0060] Figure 3 A schematic diagram of the structure of a device for implementing the operating method of the business big data stream processing system based on feature analysis provided in one embodiment of the present invention.
[0061] The realization of the purpose, functional features and advantages of the present invention will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION
[0062] It should be understood that the specific embodiments described herein are only used to explain the present invention, and are not used to limit the present invention.
[0063] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0064] The terms used in the embodiments of the present invention are only for the purpose of describing specific embodiments, and are not intended to limit the present invention. The singular forms "a", "said" and "the" used in the embodiments of the present invention and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings, and "multiple" generally includes at least two.
[0065] As used herein, the words "if" and "if" may be interpreted as "at the time of" or "when" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrases "if it is determined" or "if (stated condition or event) is detected" may be interpreted as "when it is determined" or "in response to determining" or "when detecting (stated condition or event)" or "in response to detecting (stated condition or event)", depending on the context.
[0066] In addition, the step sequence in the following method embodiments is only an example and not a strict limitation.
[0067] In fact, the server-side device deployed by the business big data stream processing system based on feature analysis may be composed of one or more devices. The above-mentioned business big data stream processing system based on feature analysis can be implemented as: business instance, virtual machine, hardware device. For example, the business big data stream processing system based on feature analysis can be implemented as a business instance deployed on one or more devices in the cloud node. In simple terms, the business big data stream processing system based on feature analysis can be understood as a software deployed on the cloud node, which is used to provide a business big data stream processing system based on feature analysis for each user terminal. Alternatively, the business big data stream processing system based on feature analysis can also be implemented as a virtual machine deployed on one or more devices in the cloud node. The virtual machine is installed with application software for managing each user terminal. Alternatively, the business big data stream processing system based on feature analysis can also be implemented as a server consisting of many hardware devices of the same or different types, and one or more hardware devices are set to provide a business big data stream processing system based on feature analysis for each user terminal.
[0068] In terms of implementation, the business big data stream processing system based on feature analysis and the user end are adapted to each other. That is, the business big data stream processing system based on feature analysis is an application installed on the cloud service platform, and the user end is a client that establishes a communication connection with the application; or the business big data stream processing system based on feature analysis is implemented as a website, and the user end is implemented as a web page; or the business big data stream processing system based on feature analysis is implemented as a cloud service platform, and the user end is implemented as a small program in an instant messaging application.
[0069] Reference Figure 1 1 is a functional module diagram of a business big data stream processing system based on feature analysis provided by an embodiment of the present invention.
[0070] The business big data stream processing system 100 based on feature analysis of the present invention can be set in a cloud server. In terms of implementation, it can be used as one or more service devices, or it can be installed as an application on the cloud (such as a server of a mobile service operator, a server cluster, etc.), or it can be developed as a website. According to the functions implemented, the business big data stream processing system 100 based on feature analysis can include a heterogeneous data fusion module 101, a business feature extraction module 102, a business data block generation module 103, a business feature analysis diagram generation module 104 and a business processing status determination module 105. The module described in the present invention can also be called a unit, which refers to a series of computer program segments that can be executed by a device processor and can complete fixed functions, which are stored in the memory of the device.
[0071] In the embodiment of the present invention, in the business big data stream processing system based on feature analysis, each of the above modules can be independently implemented and called with other modules. The call here can be understood as a module that can connect to multiple modules of another type and provide corresponding services to the multiple modules connected to it. For example, the business feature extraction module can call the heterogeneous data fusion module to obtain the information collected by the heterogeneous data fusion module. Based on the above characteristics, in the business big data stream processing system based on feature analysis provided by the embodiment of the present invention, the application scope of the business big data stream processing system architecture based on feature analysis can be adjusted by adding modules and directly calling them without modifying the program code, so as to achieve cluster-based horizontal expansion, so as to achieve the purpose of quickly and flexibly expanding the business big data stream processing system based on feature analysis. In practical applications, the above modules can be set in the same device or different devices, or they can be set in virtual devices, such as service instances in cloud servers.
[0072] The following describes the various components and specific workflows of the business big data stream processing system based on feature analysis in combination with specific embodiments:
[0073] The heterogeneous data fusion module 101 is used to obtain real-time target business data, classify the real-time target business data according to preset business timing to obtain business timing data, and fuse the business timing data through a preset heterogeneous fusion algorithm to obtain business heterogeneous fusion data.
[0074] In an embodiment of the present invention, the real-time target business data includes transaction data, customer data, financial data, log data, etc. generated in real time, wherein the transaction data may be sales orders, purchase orders, payment information, etc., transaction time, location, amount, product and other information; customer data includes basic customer information, purchase history, preferences, feedback and other information; financial data includes cash flow, tax status, investment return and other information; log data includes server logs, network equipment logs, application logs, etc., and business big data stream refers to a large amount of data generated in the daily operation of an enterprise, including structured and unstructured data. Business data can come from various business links, such as sales, procurement, supply chain, logistics, customer service, finance, etc., and are usually generated in a continuous form and involve data transmission and processing between multiple systems.
[0075] In detail, real-time target business data can be obtained from a pre-stored storage area through a computer statement with a data capture function (such as a Java statement, a Python statement, etc.), and the storage area includes but is not limited to a database and a blockchain.
[0076] Furthermore, in order to understand the data flow of each link, optimize business processes, and improve efficiency and quality, it is necessary to classify the real-time target business data to obtain the business data of each link.
[0077] In an embodiment of the present invention, the business timing data refers to the division of real-time target business data into periodic data, random data and burst data based on the timing of the business data, wherein periodic data refers to data streams triggered according to a fixed time period, such as periodic collection of transaction data, the transaction data including transaction time, location, amount, product, etc.; random data streams are data streams triggered by random events, such as business data transmission alarm information; burst data streams are data streams triggered by sudden events, which have high real-time requirements but occupy less traffic, such as fault information and business event record information.
[0078] In the embodiment of the present invention, when the heterogeneous data fusion module 101 classifies the real-time target service data according to the preset service time sequence to obtain the service time sequence data, it is specifically used to:
[0079] Generate a target service identifier sequence according to the time period and trigger identifier of the real-time target service data;
[0080] Generate a timing comparison matrix of the real-time target service data according to a preset service timing and the target service identification sequence;
[0081] The business timing data is determined according to the timing characteristics of the timing comparison matrix.
[0082] In detail, the business timing refers to recording and sorting business activities on a time axis in order to analyze and grasp the timing relationship between different business links and their impact on the entire business process. The time period and trigger identifier corresponding to the real-time target business data are used to generate a target business identifier sequence. For example, if the time period of the real-time target business data is 1 hour and the trigger identifier is none, the target business identifier sequence of the real-time target business data is {time interval, 1 hour, none}; if the time period of the real-time target business data is a time point, the time period is set to zero, and the trigger identifier is a random trigger, then the corresponding target business identifier sequence is {time point, 0, random trigger identifier}, and then according to the business timing. The time point corresponding to each business activity is compared with the time point or time interval in the target business identification sequence. According to the time point sequence in the business timing being ABCDE, the time interval in the target business identification sequence being AB or the time point being C, a 2×5 timing comparison matrix is constructed using ABCDE and AB. If the value corresponding to AB in the time point sequence is 1, it indicates that the data corresponding to this business activity is periodic data. A 1×5 timing comparison matrix is constructed using ABCDE and C. If the value corresponding to C in the time point sequence is 1, it indicates that the data corresponding to this business activity is random data. The timing features in the timing comparison matrix are represented by whether the value at the corresponding position in the matrix is 1.
[0083] Furthermore, when the real-time target business data is divided into periodic data, random data and burst data, business data from different sources will exist in different business time series data. Therefore, it is necessary to fuse the business data from different sources to obtain more comprehensive fused data, so as to facilitate the subsequent extraction of more accurate business data features.
[0084] In the embodiment of the present invention, the heterogeneous service fusion data is obtained by splicing and fusion of service data from different sources in the same service time series data through a series of processing to obtain service fusion data.
[0085] In the embodiment of the present invention, when the heterogeneous data fusion module 101 fuses the service time series data through a preset heterogeneous fusion algorithm to obtain service heterogeneous fusion data, it is specifically used to:
[0086] Performing standardization processing on the business time series data to obtain business standardized time series data;
[0087] Performing data alignment processing on the business standardized time series data to obtain business aligned time series data;
[0088] The business time series data is fused by the following heterogeneous fusion algorithm to obtain business heterogeneous fusion data:
[0089] P m ={δ m1 ,δ m2 ,…,δ mi ,…,δ mn}
[0090] Among them, P m is the mth type of heterogeneous fusion data of the service time series data, δ mi is the i-th service data in the m-th service heterogeneous fusion data, δ mn It is the nth service data in the mth service heterogeneous fusion data.
[0091] In detail, business time series data of different types and sources are standardized to ensure that the data has a consistent format and unit, including a unified format of timestamps, alignment of data granularity, consistency of data types, etc. Different data sources may use different timestamp formats, such as UNIX timestamps, date and time strings, etc., and the timestamp formats of all data need to be unified; different data sources may provide data at different time granularities (such as seconds, minutes, hours). In order to conduct effective analysis and comparison, the granularity of the data needs to be aligned. Data can be converted to a unified granularity by downsampling or interpolation to ensure that the time intervals of data points are consistent; different data sources may use different data types, such as integers, floating point numbers, strings, etc. Before analysis, the types of all data need to be consistent. Appropriate data conversion functions or tools can be used to convert data into corresponding unified types; in addition, since the timestamps of different data sources may differ, data alignment operations need to be performed to align the timestamps of different data sources so that they can be compared and merged at the same time point, and then the business data after data alignment is spliced to obtain the merged data, thereby providing a more comprehensive and accurate data foundation for subsequent business analysis and decision-making.
[0092] For example, business time series data is divided into periodic data, random data and burst data. For example, periodic data includes A, B, C; random data includes D, E, F; burst data includes G, H. However, different business time series data may have different data acquisition sources, which are heterogeneous data. It is necessary to perform heterogeneous fusion of data to obtain more comprehensive fused data and extract more accurate business features. For example, the periodic data in heterogeneous fused data is P = {δ1,δ2,…,δ i ,…,δn}, where P is the heterogeneous fusion data in the periodic data, δ1 is the transaction data, δ2 is the sales data, δ n For customer data, etc.
[0093] Furthermore, the original heterogeneous fusion data can be converted into a smaller number of independent features, thereby reducing the dimension of the data, simplifying the data analysis process, removing redundant information, and extracting the most meaningful features for business tasks, thereby better guiding business decisions and optimization.
[0094] The service feature extraction module 102 is used to calculate the service independent features of the heterogeneous fusion data using a preset independent feature component analysis algorithm, calculate the traffic density of the heterogeneous fusion data according to the service independent features, and calculate the traffic bandwidth of the heterogeneous fusion data according to the service independent features.
[0095] In the embodiment of the present invention, the business-independent feature refers to a feature that is independent in heterogeneous fused data and related to business activities. The business-independent features are independent of each other in the data, are not affected by other features, and can provide meaningful information for business activities.
[0096] In the embodiment of the present invention, when the service feature extraction module 102 calculates the service independent features of the heterogeneous fusion data using a preset independent feature component analysis algorithm, it is specifically used to:
[0097] The independent matrix of the heterogeneous fusion data is calculated using the independent characteristic component analysis algorithm described below:
[0098] G(w)=E{[g(w T x)] 2}-κ(w)
[0099] Wherein, G(w) is the objective function of the independent matrix w, E is the expected function, g is the non-Gaussian metric function, x is the heterogeneous fusion data, κ(w) is the orthogonal constraint term of the independent matrix w, and T is the transposed symbol;
[0100] Generate a service mixing matrix according to the independent matrix and the heterogeneous fusion data;
[0101] The service independence characteristic is determined according to the service mixing matrix.
[0102] In detail, the independent matrix of the heterogeneous fusion data is calculated by the independent feature component analysis algorithm. The independent matrix w contains the weight of each independent component in the original data, and the independent matrix w can be solved by the maximum likelihood estimation method. In each iteration, the independent feature component analysis algorithm first uses the first-order Taylor expansion to measure the non-Gaussianity, and then solves the independent matrix w through Newton iteration or gradient optimization method to maximize the objective function in the iterative process, and solves the maximum point of the objective function to obtain the independent matrix w, and then multiplies the independent matrix w with the heterogeneous fusion data corresponding to different time points to obtain the service mixing matrix, and each column in the service mixing matrix is used as a service independent feature to obtain the service independent feature of the heterogeneous fusion data.
[0103] Specifically, different heterogeneous fusion data are obtained at different time points, and the matrix generated according to the heterogeneous fusion data at different time points is: The independent matrix w is an n×n orthogonal matrix, so the business mixing matrix is an n×n matrix. Each column in the business mixing matrix represents an independent business feature. For example, heterogeneous fusion data includes transaction data, sales data, customer data, etc. corresponding to different time points. The business mixing matrix is obtained by multiplying the independent matrix with the heterogeneous fusion data. Each column in the business mixing matrix represents transaction data, sales data, customer data, etc., thereby obtaining the business independent features of the heterogeneous fusion data.
[0104] Furthermore, in order to provide more detailed and higher-level data analysis based on the independent characteristics of the services, it is necessary to calculate the traffic density characteristics and traffic bandwidth characteristics of the heterogeneous fused data to better reflect the data flow characteristics of the service data.
[0105] In the embodiment of the present invention, the traffic density refers to the data flow passing through a network node or link per unit time, indicating the amount of data in the process of business data flow transmission.
[0106] In the embodiment of the present invention, when the service feature extraction module 102 calculates the flow density of the heterogeneous fusion data according to the service independent feature, it is specifically used to:
[0107] Counting the service data flow corresponding to the heterogeneous fusion data in the service independent characteristics according to a preset time interval;
[0108] Calculate the central time point of the heterogeneous fusion data within the time interval according to the business data flow;
[0109] The flow density of the heterogeneous fusion data is calculated according to the central time point, wherein the flow density calculation formula is:
[0110]
[0111] Among them, H t is the flow density at the tth moment, μ is the data flow correction factor, arctg is the inverse tangent function, t is the time point, and T0 is the central time point.
[0112] Specifically, the service data flow in the service independent feature can be counted within a preset time interval, such as the time interval is t0-t n , then at t0-t n Statistics of business data traffic in each business independent feature, which can be viewed through the task manager at t0-t n The business data traffic within t0-t n The business data traffic generated by the transaction data and the n The business data traffic generated by the customer data in the network is calculated based on the business data traffic at t0-t n The central time point T0 within the n As a data collection cycle, F(t) is the data flow at the tth time, t0 is the start time of the data collection cycle, and t1 is the end time of the data collection cycle, so as to calculate the flow density of heterogeneous fusion data according to the central time point.
[0113] Specifically, the data flow correction factor μ in the flow density calculation formula is usually a self-defined real number less than 1, which is used to correct or adjust the original data flow to more accurately reflect the actual situation. Specifically, the correction factor μ is a value between 0 and 1, which represents the reliability or accuracy of the original data flow. The data flow recorded by the business system may experience a variety of interferences and errors, such as data transmission loss, network delays, data corruption, etc., and the data can be corrected and adjusted by introducing the correction factor μ to obtain a more accurate flow value, thereby obtaining a more accurate flow density.
[0114] Furthermore, it is necessary not only to calculate the traffic density of heterogeneous fused data to process the big data stream, but also to calculate the traffic bandwidth of heterogeneous fused data to obtain more accurate and comprehensive business data characteristics.
[0115] In the embodiment of the present invention, the traffic bandwidth refers to the amount of data transmitted in the network and the transmission rate, and is used to describe the transmission capacity and performance of the network.
[0116] In the embodiment of the present invention, when the service feature extraction module 102 calculates the flow bandwidth of the heterogeneous fusion data according to the service independent feature, it is specifically used to:
[0117] Determining a data impact factor of the heterogeneous fusion data according to the service independence characteristics;
[0118] The traffic bandwidth of the heterogeneous fusion data is calculated according to the service data traffic and the data impact factor, wherein the traffic bandwidth calculation formula is:
[0119]
[0120] Wherein, Z is the traffic bandwidth, F is the service data traffic, R is the data traffic statistics time, χ is the traffic bandwidth control correction factor, λ k is the kth data impact factor, τ k is the weight of the kth influencing factor, and K is the number of factors that influence the data.
[0121] In detail, the data impact factor includes factors such as data transmission frequency, transmission distance, and bandwidth limitation of the transmission channel. The data impact factor is a factor affecting the traffic bandwidth of the independent characteristics of the service, and the data transmission frequency, transmission distance, and transmission channel in the data impact factor are pre-set or known data. The traffic bandwidth of the heterogeneous fused data is calculated based on the previously counted business data traffic and data impact factors, which represents the maximum data traffic that the network can carry within a specific time period.
[0122] Specifically, the flow bandwidth calculation formula is the data impact factor λ k Including transmission frequency, transmission distance, and influencing factor weight τ k It is set based on the data influencing factors. The weight of the influencing factors of the data influencing factors can be determined by the hierarchical analysis method. For example, the hierarchical structure of the transmission distance can be determined based on the security of the transmission distance, signal strength and other factors, and a judgment matrix can be constructed based on the hierarchical structure. The weight of the transmission distance is calculated according to the judgment matrix; the weight of the transmission frequency can be determined according to the delay of the transmission frequency, the amount of data and other factors. In addition, the flow bandwidth control correction factor χ can better adjust the flow bandwidth control algorithm or strategy, and can more accurately grasp the network situation, such as network congestion, transmission delay, packet loss, etc.
[0123] Furthermore, in order to provide a comprehensive business perspective, better understand the operation of the business, and help the enterprise make corresponding decisions, it is necessary to comprehensively consider business indicators from different aspects through multi-dimensional features, and it is necessary to generate multi-dimensional business features based on traffic density and traffic bandwidth.
[0124] The business data block generation module 103 is used to generate target business multi-dimensional features of the real-time target business data according to the traffic density and the traffic bandwidth, and generate real-time target business data blocks from the real-time target business data according to the target business multi-dimensional features through a preset bidirectional sliding window.
[0125] In the embodiment of the present invention, the multi-dimensional characteristics of the target service refer to a quantitative data description of the target service from different dimensions, including traffic density and traffic bandwidth.
[0126] In the embodiment of the present invention, when the service data block generation module 103 generates the target service multi-dimensional features of the real-time target service data according to the traffic density and the traffic bandwidth, it is specifically used to:
[0127] Determining the flow characteristics of the real-time target service data according to the flow density;
[0128] Determining resource utilization characteristics according to the traffic density and the traffic bandwidth;
[0129] Determining the service priority characteristics of the real-time target service data according to the ratio of the traffic density and the traffic bandwidth;
[0130] The traffic characteristics, the resource utilization characteristics and the service priority characteristics are spliced into target service multi-dimensional characteristics of the real-time target service data.
[0131] In detail, traffic density represents the traffic passing through a certain network node or link in a unit time. Traffic density can be divided into three levels: low, medium, and high. Calculating the average value of traffic density can reflect the overall traffic status of real-time target business data. Selecting the maximum value of traffic density can represent the traffic peak when the network is busy. Therefore, the traffic status and traffic peak of real-time target business data can be determined according to traffic density, that is, the traffic characteristics of real-time target business data. In addition, traffic bandwidth represents the maximum traffic that the network can carry in a specific time period. Calculating the ratio of traffic bandwidth to traffic density can reflect the degree of network resource utilization; according to the relationship between traffic density and traffic bandwidth, evaluate whether the network is congested; according to the ratio of traffic density and traffic bandwidth, determine the priority order of the business. The larger the ratio, the higher the business priority, and the higher the order. The smaller the ratio, the lower the business priority, and the lower the order. Then, the traffic characteristics, resource utilization characteristics, and business priority characteristics are spliced into the multi-dimensional characteristics of the target business of the real-time target business data, so that the characteristics and requirements of the real-time target business data can be more comprehensively described, which can be used in business scheduling, resource allocation, network optimization, etc. to improve the performance and service quality of the network.
[0132] Furthermore, in order to ensure the real-time nature of the data, it is necessary to capture the latest real-time target business data in a timely manner. By generating real-time target business data blocks, the operating status of the target business can be monitored in real time, and predictions and trend analysis can be performed based on historical data, which will help companies to identify problems in a timely manner, make decisions, and predict future business trends.
[0133] In the embodiment of the present invention, the target service data block is a data set within a period of time generated based on a sliding window and multi-dimensional features of the target service, and can monitor the operating status of the target service in real time.
[0134] In the embodiment of the present invention, when the service data block generation module 103 generates a real-time target service data block from the real-time target service data according to the multi-dimensional characteristics of the target service through a preset bidirectional sliding window, it is specifically used to:
[0135] Determine the initial window size of the real-time target service data according to the multi-dimensional characteristics of the target service;
[0136] Moving the bidirectional sliding window forward according to the initial window size to obtain a forward service data block;
[0137] Reversely moving the bidirectional sliding window according to the initial window size to obtain a reverse service data block;
[0138] The forward service data block and the reverse data block are collected as the real-time target service data block.
[0139] In detail, the traffic characteristics and resource utilization characteristics in the target business multidimensional characteristics are used to determine the business multidimensional characteristic value, and the peak value of the traffic density in the traffic characteristics and the resource utilization rate are added and averaged to obtain the business multidimensional characteristic value, and then the business multidimensional characteristic value is compared with the number of target business multidimensional characteristics. If the business multidimensional characteristic value is greater than the number of target business multidimensional characteristics, the value obtained by rounding up the business multidimensional characteristic value is used as the initial window size; if the business multidimensional characteristic value is less than or equal to the number of target business multidimensional characteristics, the number of target business multidimensional characteristics is used as the initial window size. For example, if the number of target business multidimensional characteristics is 3, and the business multidimensional characteristic value is less than the number of target business multidimensional characteristics, the initial window size is 3.
[0140] Specifically, the center of the entire window corresponding to the real-time target business data is used as the sliding starting point. In the process of bidirectional sliding window, the data of the previous and subsequent windows can be processed simultaneously. Then, by moving forward (moving forward) with the window center as the moving center and the initial window size and moving backward (moving backward) with the window center as the moving center and the initial window size, the entire window data can be covered, and then all the real-time target business data can be covered. When new business data arrives, the bidirectional moving window will also cover the new business data in real time, thereby generating a real-time target business data block of the real-time target business data according to the window size. Then, the data inside and outside the window can be used for more comprehensive analysis and calculation, thereby improving the efficiency of generating real-time target business data blocks.
[0141] Furthermore, when the target service data block is transmitted, in order to more accurately evaluate the performance of the target service in the actual network, it is necessary to simulate the real network environment through the virtual transmission path, including different network topologies, routing strategies and link conditions, etc., to provide more fine-grained performance monitoring and control, and to monitor and alarm the real-time target service data in real time, and timely discover and handle potential problems. At the same time, traffic adjustment and service quality control can be carried out according to real-time data to meet business needs and optimize user experience.
[0142] The service characteristic analysis diagram generation module 104 is used to generate a virtual transmission path for the real-time target service data according to the real-time target service data block and a preset virtual stream channel, generate a parallel data chain for the real-time target service data block according to the virtual transmission path, and generate a service characteristic analysis diagram through the real-time target service data blocks in the parallel data chain.
[0143] In the embodiment of the present invention, the virtual transmission path simulates the data transmission between the source node and the target node through a preset virtual channel and routing mode, so as to evaluate and optimize the network performance, capacity and reliability. In the virtual transmission path, the data stream is transmitted through a series of logical links (including virtual links and physical links).
[0144] In the embodiment of the present invention, when the service characteristic analysis diagram generating module 104 generates the virtual transmission path of the real-time target service data according to the real-time target service data block and the preset virtual flow channel, it is specifically used to:
[0145] Determining the service characteristics of the real-time target service data according to the block identifier of the real-time target service data block;
[0146] Extracting channel characteristic parameters of the virtual flow channel, and determining a virtual transmission level of the virtual flow channel according to the channel characteristic parameters;
[0147] Constructing a level mapping between the service priority corresponding to the service feature and the virtual transmission level;
[0148] A virtual transmission path for the real-time target service data is generated according to the level mapping.
[0149] In detail, the block identifier corresponding to each block of real-time target service data is used to determine the service characteristics in the real-time target service data. For example, if the block identifier corresponding to the real-time target service data is 002, the service characteristics such as traffic density, traffic bandwidth, service priority and service resource utilization corresponding to the real-time target service data with block identifier 002 are extracted. Channel characteristic parameters are set for each virtual flow channel, such as transmission bandwidth, delay, reliability, etc., and then the virtual transmission level of each virtual flow channel is determined according to the channel characteristic parameters. If the channel characteristic parameters have a large transmission bandwidth, low delay and high reliability, the virtual transmission level of this virtual flow channel is high. If the channel characteristic parameters have a small transmission bandwidth, high delay and low reliability, the virtual transmission level of this virtual flow channel is low. For other channel characteristic parameters, the virtual transmission level of the virtual flow channel is determined to be medium.
[0150] Specifically, the service level corresponding to the real-time target service data block and the virtual transmission level are mapped to a level. If the service level is high, the corresponding virtual transmission level is high; if the service level is medium, the corresponding virtual transmission level is medium; if the service level is low, the corresponding virtual transmission level is low, thereby obtaining a level mapping, and then the real-time target service data block is routed through the corresponding virtual transmission path according to different level mappings. If the service level corresponding to the real-time target service data block is high, the real-time target service data block is transmitted through a high-level virtual flow channel. In addition, in the case where the service level does not have a high level, the real-time target service data can be mapped to a high-level virtual flow channel to increase the data transmission rate.
[0151] Furthermore, a parallel data chain of a real-time target data block can be generated according to the virtual transmission path, and the virtual transmission paths corresponding to different real-time target data blocks are transmitted in parallel, wherein the parallel data chain refers to a method of transmitting the real-time target business data block sent by the source node in parallel on the virtual transmission path, and the real-time target data block is transmitted simultaneously on different channels or paths according to the parallel channels and routing methods set in the virtual transmission path, and the real-time target data block can be transmitted on different links at the same time, thereby improving the transmission speed and efficiency. At the target node, after receiving all the real-time target data blocks transmitted in parallel, a data merging operation is performed, and each real-time target data block is merged in a predetermined order, and restored to a complete real-time target business data, which can make full use of network resources, reduce the delay of data transmission, and provide a better user experience. At the same time, it can be flexibly adjusted according to specific needs and network status to meet the transmission requirements of real-time target business data.
[0152] In the embodiment of the present invention, a business characteristic analysis diagram is generated through the real-time target business data block in the parallel data chain, that is, the business data characteristics in the real-time target business data block are displayed in a visual manner. For example, if the business data characteristics of the real-time target business data block are flow density, flow bandwidth and business resource utilization, a bar graph of business data characteristics can be generated according to different real-time target business data blocks, such as the bar graph of the business characteristic analysis diagram of the real-time target business data block 002 corresponds to the values of flow density, flow bandwidth and business resource utilization. By generating a business characteristic analysis diagram, the characteristics of the real-time target business data can be better understood and analyzed, helping users make decisions, optimize business processes or detect abnormal situations.
[0153] Furthermore, based on the business characteristic analysis chart, we can understand the changing trends and key characteristics of business data, so as to formulate corresponding processing strategies according to real-time needs, have strong real-time response capabilities, and quickly process and respond to changes in business data.
[0154] The business processing status determination module 105 is used to generate a business big data stream processing strategy for the real-time target business data according to the business feature analysis diagram, and perform data processing on the real-time target business data according to the business big data stream processing strategy to obtain a business processing status.
[0155] In the embodiment of the present invention, the business big data stream processing strategy is a measure for processing different business situations. By real-time monitoring of changes and trends in business data, abnormal situations are detected in a timely manner, and early warning notifications are sent to help users respond to problems quickly and take corresponding measures to reduce potential business risks.
[0156] In the embodiment of the present invention, when the business processing state determination module 105 generates the business big data stream processing strategy of the real-time target business data according to the business characteristic analysis diagram, it is specifically used to:
[0157] Generate a trigger factor according to the characteristic value in the business characteristic analysis diagram and a preset characteristic threshold;
[0158] Generate a service processing factor for the real-time target service data according to the trigger factor;
[0159] A business big data stream processing strategy for the real-time target business data is generated according to the business processing factor.
[0160] In detail, the trigger factor is expressed as a Boolean value (true if the condition is met, otherwise false). The result of comparing the characteristic value in the business characteristic analysis diagram with the characteristic threshold is used to determine whether the trigger condition is met, and a corresponding trigger factor is generated. If the characteristic value is greater than or equal to the characteristic threshold, it means that the trigger condition is met, and the trigger factor is true, represented by 1; if the characteristic value is less than the characteristic threshold, it means that the trigger condition is not met, and the trigger factor is false, represented by 0; and then according to the generated trigger factor, the corresponding business logic, alarm notification or other operations can be triggered. For example, when a trigger factor is true, it can trigger the early warning system to send an alarm notification, trigger related automated operations, record related data and generate corresponding reports and other business processing factors, thereby using the business processing factor as a business big data stream processing strategy for real-time target business data.
[0161] Specifically, after the real-time target business data is processed according to the business big data stream processing strategy, the business processing status can be obtained. The business processing status represents the result of data processing or the status information of the current business. The business processing status includes success, failure, in progress, blocking, and timeout. The successful status indicates that the business data processing is successful, and the required operation or calculation is completed as expected, indicating that the business data is accurately processed and produces meaningful results; the failed state indicates that the business data processing encounters an error or anomaly and cannot successfully complete the required operation or calculation, which may be due to data quality problems, calculation errors or other technical problems; the in progress state indicates that the business data is being processed but has not yet been completed, indicating that the processing is in progress but the final result has not yet been obtained; the blocked state indicates that the business data processing cannot be carried out temporarily because there are some obstacles or dependencies, which may be caused by insufficient resources, unsatisfied external dependencies or other waiting conditions; the timeout state indicates that the business data processing exceeds the predetermined time limit and cannot be completed within the specified time, which may be caused by excessive data volume, high calculation complexity or system performance problems. The business processing status can help users understand the progress of business data processing, discover and solve potential problems in a timely manner, and make corresponding decisions or adjustments based on the processing status.
[0162] The embodiment of the present invention classifies the real-time target service data, and then performs data heterogeneous fusion on the classified service time series data, which is conducive to obtaining more comprehensive fusion data and facilitating the subsequent extraction of more accurate service data features; extracting the service independent features of the heterogeneous fusion data can provide more detailed and higher-level data analysis, and then calculate the traffic density and traffic bandwidth of the heterogeneous fusion data according to the service independent features, and then generate multi-dimensional service features, which can more comprehensively describe the characteristics and requirements of the real-time target service data to improve the performance and service quality of the network; generating real-time target service data blocks according to the multi-dimensional features of the target service, and using the data inside and outside the window for more comprehensive analysis and calculation, thereby improving the efficiency of generating real-time target service data blocks; generating virtual transmission paths according to the real-time target service data blocks and virtual flow channels, and then generating parallel data chains according to the virtual transmission paths, which is conducive to flexible adjustment of data transmission to meet the transmission requirements of the real-time target service data; generating a service big data stream processing strategy according to the service feature analysis diagram, and then determining the service processing status, can timely understand the progress of service data processing, timely discover and solve potential problems, and make corresponding decisions or adjustments according to the processing status. Therefore, the business big data stream processing system, method and medium based on feature analysis proposed in the present invention can solve the problem of low accuracy when processing business big data streams.
[0163] Reference Figure 2 FIG. 1 is a flow chart of an operation method of a business big data stream processing system based on feature analysis provided by an embodiment of the present invention. In this embodiment, the operation method of the business big data stream processing system based on feature analysis includes:
[0164] S1. Acquire real-time target business data, classify the real-time target business data according to preset business time sequence to obtain business time sequence data, and fuse the business time sequence data through a preset heterogeneous fusion algorithm to obtain business heterogeneous fusion data;
[0165] S2. Calculate the service-independent features of the heterogeneous fusion data using a preset independent feature component analysis algorithm, calculate the traffic density of the heterogeneous fusion data according to the service-independent features, and calculate the traffic bandwidth of the heterogeneous fusion data according to the service-independent features;
[0166] S3, generating a target service multi-dimensional feature of the real-time target service data according to the traffic density and the traffic bandwidth, and generating a real-time target service data block from the real-time target service data according to the target service multi-dimensional feature through a preset bidirectional sliding window;
[0167] S4, generating a virtual transmission path for the real-time target service data according to the real-time target service data block and a preset virtual stream channel, generating a parallel data chain for the real-time target service data block according to the virtual transmission path, and generating a service feature analysis diagram through the real-time target service data block in the parallel data chain;
[0168] S5. Generate a business big data stream processing strategy for the real-time target business data according to the business feature analysis diagram, and perform data processing on the real-time target business data according to the business big data stream processing strategy to obtain a business processing status.
[0169] like Figure 3 1 is a schematic diagram of the structure of a device for implementing an operation method of a business big data stream processing system based on feature analysis provided by an embodiment of the present invention.
[0170] The device may include a processor 10, a memory 11, a communication bus 12, and a communication interface 13, and may also include a computer program stored in the memory 11 and executable on the processor 10, such as a business big data stream processing system program based on feature analysis.
[0171] In some embodiments, the processor 10 may be composed of an integrated circuit, for example, a single packaged integrated circuit, or a plurality of packaged integrated circuits with the same or different functions, including one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and combinations of various control chips. The processor 10 is the control core (Control Unit) of the device, and uses various interfaces and lines to connect various components of the entire device, and executes or executes programs or modules stored in the memory 11 (for example, a program for processing a business big data stream based on feature analysis, etc.), and calls data stored in the memory 11 to execute various functions of the device and process data.
[0172] The memory 11 includes at least one type of readable storage medium, and the readable storage medium includes a flash memory, a mobile hard disk, a multimedia card, a card-type memory (for example, an SD or DX memory, etc.), a magnetic memory, a disk, an optical disk, etc. In some embodiments, the memory 11 may be an internal storage unit of the device, such as a mobile hard disk of the device. In other embodiments, the memory 11 may also be an external storage device of the device, such as a plug-in mobile hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the device. Further, the memory 11 may also include both an internal storage unit of the device and an external storage device. The memory 11 can not only be used to store application software and various types of data installed on the device, such as the code of a business big data stream processing system program based on feature analysis, but also can be used to temporarily store data that has been output or is to be output.
[0173] The communication bus 12 may be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. The bus is configured to realize connection and communication between the memory 11 and at least one processor 10, etc.
[0174] The communication interface 13 is used for communication between the above-mentioned device and other devices, including a network interface and a user interface. Optionally, the network interface may include a wired interface and / or a wireless interface (such as a WI-FI interface, a Bluetooth interface, etc.), which is generally used to establish a communication connection between the device and other devices. The user interface may be a display (Display), an input unit (such as a keyboard (Keyboard)), and optionally, the user interface may also be a standard wired interface, a wireless interface. Optionally, in some embodiments, the display may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, and an OLED (Organic Light-Emitting Diode) touch device, etc. Among them, the display may also be appropriately referred to as a display screen or a display unit, which is used to display information processed in the device and to display a visual user interface.
[0175] Figure 3 Only the device with components is shown, and it can be understood by those skilled in the art that Figure 3The structure shown does not constitute a limitation of the device, and may include fewer or more components than shown, or combine certain components, or arrange the components differently.
[0176] For example, although not shown, the device may also include a power source (such as a battery) for supplying power to each component. Preferably, the power source may be logically connected to the at least one processor 10 through a power management device, so that the power management device can realize functions such as charging management, discharging management, and power consumption management. The power source may also include any components such as one or more DC or AC power sources, recharging devices, power failure detection circuits, power converters or inverters, power status indicators, etc. The device may also include a variety of sensors, Bluetooth modules, Wi-Fi modules, etc., which will not be repeated here.
[0177] It should be understood that the embodiment is for illustration only and the scope of the patent application is not limited to this structure.
[0178] The service big data stream processing system program based on feature analysis stored in the memory 11 in the device is a combination of multiple instructions. When running in the processor 10, it can achieve:
[0179] Acquire real-time target business data, classify the real-time target business data according to preset business time sequence to obtain business time sequence data, and fuse the business time sequence data through a preset heterogeneous fusion algorithm to obtain business heterogeneous fusion data;
[0180] Calculate the service-independent features of the heterogeneous fusion data using a preset independent feature component analysis algorithm, calculate the traffic density of the heterogeneous fusion data according to the service-independent features, and calculate the traffic bandwidth of the heterogeneous fusion data according to the service-independent features;
[0181] Generate a target service multi-dimensional feature of the real-time target service data according to the traffic density and the traffic bandwidth, and generate a real-time target service data block from the real-time target service data according to the target service multi-dimensional feature through a preset bidirectional sliding window;
[0182] Generate a virtual transmission path for the real-time target service data according to the real-time target service data block and a preset virtual stream channel, generate a parallel data chain for the real-time target service data block according to the virtual transmission path, and generate a service feature analysis diagram through the real-time target service data block in the parallel data chain;
[0183] A business big data stream processing strategy for the real-time target business data is generated according to the business characteristic analysis diagram, and data processing is performed on the real-time target business data according to the business big data stream processing strategy to obtain a business processing status.
[0184] Specifically, the specific implementation method of the processor 10 for the above instructions can refer to the description of the relevant steps in the corresponding embodiment of the accompanying drawings, which will not be repeated here.
[0185] Furthermore, if the module / unit integrated in the device is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. The computer-readable storage medium can be volatile or non-volatile. For example, the computer-readable medium may include: any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a mobile hard disk, a magnetic disk, an optical disk, a computer memory, and a read-only memory (ROM).
[0186] The present invention further provides a computer-readable storage medium, wherein the medium stores a computer program, and when the computer program is executed by a processor of an electronic device, the computer program can implement:
[0187] Acquire real-time target business data, classify the real-time target business data according to preset business time sequence to obtain business time sequence data, and fuse the business time sequence data through a preset heterogeneous fusion algorithm to obtain business heterogeneous fusion data;
[0188] Calculate the service-independent features of the heterogeneous fusion data using a preset independent feature component analysis algorithm, calculate the traffic density of the heterogeneous fusion data according to the service-independent features, and calculate the traffic bandwidth of the heterogeneous fusion data according to the service-independent features;
[0189] Generate a target service multi-dimensional feature of the real-time target service data according to the traffic density and the traffic bandwidth, and generate a real-time target service data block from the real-time target service data according to the target service multi-dimensional feature through a preset bidirectional sliding window;
[0190] Generate a virtual transmission path for the real-time target service data according to the real-time target service data block and a preset virtual stream channel, generate a parallel data chain for the real-time target service data block according to the virtual transmission path, and generate a service feature analysis diagram through the real-time target service data block in the parallel data chain;
[0191] A business big data stream processing strategy for the real-time target business data is generated according to the business characteristic analysis diagram, and data processing is performed on the real-time target business data according to the business big data stream processing strategy to obtain a business processing status.
[0192] In the several embodiments provided by the present invention, it should be understood that the disclosed devices, systems and methods can be implemented in other ways. For example, the system embodiments described above are only schematic, for example, the division of the modules is only a logical function division, and there may be other division methods in actual implementation.
[0193] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
[0194] In addition, each functional module in each embodiment of the present invention may be integrated into one processing unit, each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of hardware plus software functional modules.
[0195] It is obvious to those skilled in the art that the present invention is not limited to the details of the above exemplary embodiments, and that the present invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the present invention.
[0196] Therefore, no matter from which point of view, the embodiments should be regarded as illustrative and non-restrictive, and the scope of the present invention is not limited only according to the above description, and it is intended that all changes within the meaning and scope of equivalent elements within the scope of protection are included in the present invention.
[0197] In addition, it is clear that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units or systems stated in a system claim can also be implemented by one unit or system through software or hardware. The words first, second, etc. are used to indicate names, and do not indicate any particular order.
[0198] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solution of the present invention can be modified or replaced by equivalents without departing from the spirit and scope of the technical solution of the present invention.
Claims
1. A business big data stream processing system based on feature analysis, characterized in that: The system includes a heterogeneous data fusion module, a business feature extraction module, a business data block generation module, a business feature analysis diagram generation module and a business processing status determination module, wherein: The heterogeneous data fusion module is used to obtain real-time target business data, classify the real-time target business data according to the preset business time sequence to obtain business time sequence data, and fuse the business time sequence data through the preset heterogeneous fusion algorithm to obtain business heterogeneous fusion data; The service feature extraction module is used to calculate the service independent features of the heterogeneous fusion data using a preset independent feature component analysis algorithm, and is specifically used to calculate the independent matrix of the heterogeneous fusion data using the following independent feature component analysis algorithm: in, For independent matrices The objective function is is the expected function, is a non-Gaussian metric function, For the heterogeneous fusion data, For independent matrices The orthogonal constraints are is the transpose symbol; Generate a service mixing matrix according to the independent matrix and the heterogeneous fusion data; determine the service independence feature according to the service mixing matrix, wherein the independent matrix The heterogeneous fusion data corresponding to different time points are multiplied to obtain a service mixing matrix, and each column in the service mixing matrix is used as a service independent feature to obtain a service independent feature of the heterogeneous fusion data; the flow density of the heterogeneous fusion data is calculated according to the service independent feature, and the flow bandwidth of the heterogeneous fusion data is calculated according to the service independent feature, wherein the service feature extraction module is specifically used to calculate the flow density of the heterogeneous fusion data according to the service independent feature: The service data flow corresponding to the heterogeneous fusion data in the service independence feature is counted according to a preset time interval; the central time point of the heterogeneous fusion data in the time interval is calculated according to the service data flow; the flow density of the heterogeneous fusion data is calculated according to the central time point, wherein the flow density calculation formula is: in, For the The flow density at the time, is the data flow correction factor, is the inverse tangent function, For the time point, is the central time point; The service data block generation module is used to generate a target service multi-dimensional feature of the real-time target service data according to the traffic density and the traffic bandwidth, and generate a real-time target service data block from the real-time target service data according to the target service multi-dimensional feature through a preset bidirectional sliding window; The service characteristic analysis diagram generation module is used to generate a virtual transmission path of the real-time target service data according to the real-time target service data block and a preset virtual stream channel, generate a parallel data chain of the real-time target service data block according to the virtual transmission path, and generate a service characteristic analysis diagram through the real-time target service data block in the parallel data chain; The business processing status determination module is used to generate a business big data stream processing strategy for the real-time target business data according to the business feature analysis diagram, and perform data processing on the real-time target business data according to the business big data stream processing strategy to obtain the business processing status.
2. The business big data stream processing system based on feature analysis according to claim 1, characterized in that: When the heterogeneous data fusion module classifies the real-time target service data according to the preset service time sequence to obtain service time sequence data, it is specifically used to: Generate a target service identifier sequence according to the time period and trigger identifier of the real-time target service data; Generate a timing comparison matrix of the real-time target service data according to a preset service timing and the target service identification sequence; The business timing data is determined according to the timing characteristics of the timing comparison matrix.
3. The business big data stream processing system based on feature analysis as described in claim 1, characterized in that: When the heterogeneous data fusion module fuses the service time series data through a preset heterogeneous fusion algorithm to obtain service heterogeneous fusion data, it is specifically used to: Performing standardization processing on the business time series data to obtain business standardized time series data; Performing data alignment processing on the business standardized time series data to obtain business aligned time series data; The business time series data is fused by the following heterogeneous fusion algorithm to obtain business heterogeneous fusion data: in, is the first Heterogeneous fusion data of various businesses, For the The first type of heterogeneous fusion data Business data, For the The first type of heterogeneous fusion data Business data.
4. The business big data stream processing system based on feature analysis according to claim 1, characterized in that: When the service feature extraction module calculates the flow bandwidth of the heterogeneous fusion data according to the service independent feature, it is specifically used to: Determining a data impact factor of the heterogeneous fusion data according to the service independence characteristics; The traffic bandwidth of the heterogeneous fusion data is calculated according to the service data traffic and the data impact factor, wherein the traffic bandwidth calculation formula is: in, is the traffic bandwidth, is the business data traffic, The data traffic statistics time. is the flow bandwidth control correction factor, For the Data influencing factors, For the The weight of the impact factor, is the number of factors influencing the data.
5. The business big data stream processing system based on feature analysis according to claim 1, characterized in that: When the service data block generation module generates the target service multi-dimensional features of the real-time target service data according to the traffic density and the traffic bandwidth, it is specifically used to: Determining the flow characteristics of the real-time target service data according to the flow density; Determining resource utilization characteristics according to the traffic density and the traffic bandwidth; Determining the service priority characteristics of the real-time target service data according to the ratio of the traffic density and the traffic bandwidth; The traffic characteristics, the resource utilization characteristics and the service priority characteristics are spliced into target service multi-dimensional characteristics of the real-time target service data.
6. The business big data stream processing system based on feature analysis according to claim 1, characterized in that: When the service data block generation module generates a real-time target service data block from the real-time target service data according to the multi-dimensional characteristics of the target service through a preset bidirectional sliding window, it is specifically used to: Determine the initial window size of the real-time target service data according to the multi-dimensional characteristics of the target service; Moving the bidirectional sliding window forward according to the initial window size to obtain a forward service data block; Reversely moving the bidirectional sliding window according to the initial window size to obtain a reverse service data block; The forward service data block and the reverse service data block are collected as the real-time target service data block.
7. The business big data stream processing system based on feature analysis according to claim 1, characterized in that: When the service characteristic analysis diagram generation module generates a virtual transmission path of the real-time target service data according to the real-time target service data block and the preset virtual flow channel, it is specifically used to: Determining the service characteristics of the real-time target service data according to the block identifier of the real-time target service data block; Extracting channel characteristic parameters of the virtual flow channel, and determining a virtual transmission level of the virtual flow channel according to the channel characteristic parameters; Constructing a level mapping between the service priority corresponding to the service feature and the virtual transmission level; A virtual transmission path for the real-time target service data is generated according to the level mapping.
8. A method for operating a business big data stream processing system based on feature analysis, characterized in that: A business big data stream processing system for executing feature analysis according to any one of claims 1 to 7, the method comprising: Acquire real-time target business data, classify the real-time target business data according to preset business time sequence to obtain business time sequence data, and fuse the business time sequence data through a preset heterogeneous fusion algorithm to obtain business heterogeneous fusion data; The business independent features of the heterogeneous fusion data are calculated using a preset independent feature component analysis algorithm, specifically used for: calculating the independent matrix of the heterogeneous fusion data using the following independent feature component analysis algorithm: in, For independent matrices The objective function is is the expected function, is a non-Gaussian metric function, For the heterogeneous fusion data, For independent matrices The orthogonal constraints are is the transpose symbol; Generate a service mixing matrix according to the independent matrix and the heterogeneous fusion data; determine the service independence feature according to the service mixing matrix, wherein the independent matrix The heterogeneous fusion data corresponding to different time points are multiplied to obtain a service mixing matrix, and each column in the service mixing matrix is used as a service independent feature to obtain a service independent feature of the heterogeneous fusion data; the flow density of the heterogeneous fusion data is calculated according to the service independent feature, and the flow bandwidth of the heterogeneous fusion data is calculated according to the service independent feature, wherein the flow density of the heterogeneous fusion data is calculated according to the service independent feature, specifically used for: The service data flow corresponding to the heterogeneous fusion data in the service independence feature is counted according to a preset time interval; the central time point of the heterogeneous fusion data in the time interval is calculated according to the service data flow; the flow density of the heterogeneous fusion data is calculated according to the central time point, wherein the flow density calculation formula is: in, For the The flow density at the time, is the data flow correction factor, is the inverse tangent function, For the time point, is the central time point; Generate a target service multi-dimensional feature of the real-time target service data according to the traffic density and the traffic bandwidth, and generate a real-time target service data block from the real-time target service data according to the target service multi-dimensional feature through a preset bidirectional sliding window; Generate a virtual transmission path for the real-time target service data according to the real-time target service data block and a preset virtual stream channel, generate a parallel data chain for the real-time target service data block according to the virtual transmission path, and generate a service feature analysis diagram through the real-time target service data block in the parallel data chain; A business big data stream processing strategy for the real-time target business data is generated according to the business characteristic analysis diagram, and data processing is performed on the real-time target business data according to the business big data stream processing strategy to obtain a business processing status.
9. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by the processor, the method for processing business big data stream based on feature analysis as described in claim 8 is implemented.
