Data stream processing method and apparatus, electronic device, and storage medium

Abstract

The present disclosure relates to a data stream processing method and device, electronic equipment and storage medium. The method comprises: obtaining a source data stream; performing data structure conversion on the source data stream based on a preset data table template to obtain a source data table; the preset data table template corresponds to a target data format; processing data in the source data table to obtain a target data table; generating a target data stream of the target data format based on the target data table; and sending the target data stream to a target data storage module. The present disclosure improves the expansibility of data stream access and enhances the stability of the abnormal monitoring system.

Description

Technical Field

[0001] This disclosure relates to the field of big data technology, and in particular to a data stream processing method, apparatus, electronic device, and storage medium. Background Technology

[0002] During project operation, anomaly information that occurs during the operation can be monitored, recorded, and aggregated based on the anomaly monitoring platform. Specifically, the data stream generated during project operation can be connected to the anomaly monitoring platform so that the platform can analyze the anomaly information based on the connected data stream.

[0003] In related technologies, when connecting the data stream generated by a project to an anomaly monitoring platform, the platform needs to modify the backend code for each project. This data stream connection method has poor scalability, and as the number of connected projects increases, the number of times the backend code needs to be modified also increases, resulting in low stability of the anomaly monitoring system. Summary of the Invention

[0004] This disclosure provides a data stream processing method, apparatus, electronic device, and storage medium to at least solve the problems of low data stream access scalability and low stability of anomaly monitoring systems in related technologies. The technical solution of this disclosure is as follows:

[0005] According to a first aspect of the present disclosure, a data stream processing method is provided, comprising:

[0006] Obtain the source data stream;

[0007] The source data stream is transformed based on a preset data table template to obtain a source data table; the preset data table template corresponds to the target data format.

[0008] The data in the source data table is processed to obtain the target data table;

[0009] Based on the target data table, generate a target data stream in the target data format;

[0010] The target data stream is sent to the target data storage module.

[0011] In one exemplary embodiment, the preset data table template includes multiple field information;

[0012] The process of transforming the source data stream based on a preset data table template to obtain a source data table includes:

[0013] Extract first data information corresponding to the multiple field information from the source data stream;

[0014] The first data information is filled into their respective field information to obtain the source data table.

[0015] In one exemplary embodiment, the source data stream includes source data streams from multiple data sources;

[0016] The process of transforming the source data stream based on a preset data table template to obtain the source data table includes:

[0017] Based on the preset data table template, the data structure of the multiple source data streams is transformed to obtain the source data table corresponding to each of the multiple source data streams.

[0018] In one exemplary embodiment, the preset data table template includes multiple field information;

[0019] The step of performing data structure transformation on the source data streams from multiple sources based on the preset data table template to obtain source data tables corresponding to each of the multiple source data streams includes:

[0020] For each original data stream in the multiple source data streams, determine a preset data table template corresponding to the original data stream;

[0021] Extract second data information corresponding to the multiple field information from the original data stream;

[0022] The second data information is filled into the multiple fields of the preset data table template corresponding to the original data stream to obtain the source data table corresponding to the original data stream.

[0023] In one exemplary embodiment, the source data table includes multiple data items, each carrying an event type identifier;

[0024] The process of processing the data in the source data table to obtain the target data table includes:

[0025] Aggregate at least two data items with the same event type identifier to obtain aggregated data items;

[0026] The target data table is generated based on the aggregated data items.

[0027] In one exemplary embodiment, the method further includes:

[0028] Determine the first quantity of source data streams acquired per unit time;

[0029] Based on the first quantity, the memory space information of the target data storage module is determined.

[0030] In one exemplary embodiment, sending the target data stream to the target data storage module includes:

[0031] A second quantity of the target data stream is sent to the target data storage module within a unit of time; the second quantity is matched with the processing speed of the target data storage module.

[0032] According to a second aspect of the present disclosure, a data stream processing apparatus is provided, comprising:

[0033] The source data stream acquisition unit is configured to acquire the source data stream;

[0034] The first data structure conversion unit is configured to perform data structure conversion on the source data stream based on a preset data table template to obtain a source data table; the preset data table template corresponds to the target data format.

[0035] The target data table generation unit is configured to process the data in the source data table to obtain the target data table;

[0036] The target data stream generation unit is configured to generate a target data stream based on the target data table and the target data format.

[0037] The first sending unit is configured to send the target data stream to the target data storage module.

[0038] In one exemplary embodiment, the preset data table template includes multiple field information;

[0039] The first data structure conversion unit includes:

[0040] The first extraction unit is configured to extract first data information corresponding to the multiple field information from the source data stream;

[0041] The first filling unit is configured to fill the first data information into their respective corresponding field information to obtain the source data table.

[0042] In one exemplary embodiment, the source data stream includes source data streams from multiple data sources;

[0043] The first data structure conversion unit includes:

[0044] The second data structure conversion unit is configured to perform data structure conversion on the source data streams from the multiple sources based on the preset data table template, so as to obtain the source data table corresponding to each of the multiple source data streams.

[0045] In one exemplary embodiment, the preset data table template includes multiple field information;

[0046] The second data structure conversion unit includes:

[0047] The first determining unit is configured to perform, for each of the multiple source data streams, determine a preset data table template corresponding to the original data stream;

[0048] The second extraction unit is configured to extract second data information corresponding to the multiple field information from the original data stream;

[0049] The second filling unit is configured to fill the second data information into the multiple field information of the preset data table template corresponding to the original data stream, thereby obtaining the source data table corresponding to the original data stream.

[0050] In one exemplary embodiment, the source data table includes multiple data items, each carrying an event type identifier;

[0051] The target data table generation unit includes:

[0052] The aggregation processing unit is configured to perform aggregation processing on at least two data items with the same event type identifier to obtain aggregated data items;

[0053] The target data table determination unit is configured to generate the target data table based on the aggregated data items.

[0054] In one exemplary embodiment, the apparatus further includes:

[0055] The first quantity determination unit is configured to determine the first quantity of the source data stream acquired within a unit time period;

[0056] The second determining unit is configured to determine the memory space information of the target data storage module based on the first quantity.

[0057] In an exemplary embodiment, the first transmitting unit includes:

[0058] The second sending unit is configured to send a second number of the target data streams to the target data storage module within a unit time; the second number is matched with the processing speed of the target data storage module.

[0059] According to a third aspect of the present disclosure, an electronic device is provided, comprising: a processor; and a memory for storing processor-executable instructions; wherein the processor is configured to execute the instructions to implement the data stream processing method described above.

[0060] According to a fourth aspect of the present disclosure, a computer-readable storage medium is provided that, when instructions in the computer-readable storage medium are executed by a processor of a server, enables the server to perform the data stream processing method as described above.

[0061] According to a fifth aspect of the present disclosure, a computer program product is provided, the computer program product including a computer program stored in a readable storage medium, wherein at least one processor of a computer device reads from the readable storage medium and executes the computer program, causing the device to perform the data stream processing method described above.

[0062] The technical solutions provided by the embodiments of this disclosure bring at least the following beneficial effects:

[0063] This disclosure transforms the data structure of the acquired source data stream based on a preset data table template to obtain a source data table. The preset data table template corresponds to the target data format, and the data structure transformation can convert unstructured data into processable structured data. After processing the data in the source data table, a target data table is obtained. Based on the target data table, a target data stream corresponding to the target data format is generated and sent to the target data storage module. Thus, this disclosure can write the source data stream into the target data storage module by performing data structure transformation, processing, and further transformation on the source data stream. The target data storage module is used to store data in the target data format. This does not require modification of the backend code, and the same data stream access method can be used for data streams from different sources, thereby improving the scalability of data stream access and enhancing the stability of the anomaly monitoring system.

[0064] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0065] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure, and are not intended to unduly limit this disclosure.

[0066] Figure 1 This is a schematic diagram of an implementation environment according to an exemplary embodiment.

[0067] Figure 2 This is a flowchart illustrating a data stream processing method according to an exemplary embodiment.

[0068] Figure 3 This is a flowchart illustrating a source data table generation method according to an exemplary embodiment.

[0069] Figure 4 This is a flowchart illustrating another data stream processing method according to an exemplary embodiment.

[0070] Figure 5 This is a flowchart illustrating a method for generating a source data table corresponding to source data streams from multiple sources, according to an exemplary embodiment.

[0071] Figure 6 This is a flowchart illustrating a method for data processing based on a source data table, according to an exemplary embodiment.

[0072] Figure 7 This is a diagram illustrating a real-time data stream processing framework according to an exemplary embodiment.

[0073] Figure 8 This is a data stream processing apparatus illustrated according to an exemplary embodiment.

[0074] Figure 9 This is a schematic diagram of an electronic device structure according to an exemplary embodiment. Detailed Implementation

[0075] To enable those skilled in the art to better understand the technical solutions of this disclosure, the technical solutions in the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings.

[0076] It should be noted that the terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this disclosure are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this disclosure described herein can be implemented in orders other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0077] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for display, data used for analysis, etc.) involved in this disclosure are all information and data authorized by the user or fully authorized by all parties.

[0078] Please see Figure 1 The illustration shows an implementation environment provided by the present disclosure, which may include at least one data stream processing terminal 110 and an anomaly monitoring terminal 120, and the data stream processing terminal 110 and the anomaly monitoring terminal 120 may communicate with each other via a network.

[0079] Specifically, the data stream processing terminal 110 can process the data streams from various data sources to obtain a data stream that conforms to the target format, and then send the data stream that conforms to the target format to the anomaly monitoring terminal 120 so that the anomaly monitoring terminal 120 can store and analyze the data stream.

[0080] The data stream processing terminal 110 can communicate with the anomaly monitoring terminal 120 based on a browser / server (B / S) mode or a client / server (C / S) mode. The data stream processing terminal 110 may include physical devices such as smartphones, tablets, laptops, digital assistants, smart wearable devices, in-vehicle terminals, and servers, and may also include software running on the physical device, such as applications. The operating system running on the data stream processing terminal 110 in this embodiment may include, but is not limited to, Android, iOS, Linux, and Windows.

[0081] The anomaly monitoring terminal 120 and the data stream processing terminal 110 can establish a communication connection via wired or wireless means. The anomaly monitoring terminal 120 may include a stand-alone server, a distributed server, or a server cluster consisting of multiple servers, wherein the server may be a cloud server.

[0082] To address the issues of low scalability of data stream access and low stability of anomaly monitoring systems in related technologies, please refer to [link / reference]. Figure 2 It illustrates a data stream processing method, the execution entity of which can be the aforementioned Figure 1 In the data stream processing end of the process, the method may include:

[0083] S210. Obtain the source data stream.

[0084] In one specific embodiment, the source data stream can be a real-time data stream generated during the operation of a project or application; the source data stream can be obtained through reporting from collection points in the project, or it can be obtained through information collection tools in the project.

[0085] S220. The source data stream is transformed based on a preset data table template to obtain a source data table; the preset data table template corresponds to the target data format.

[0086] In one example, the source data stream can be unstructured data, data with irregular or incomplete data structures, no predefined data model, or data that is inconvenient to represent using a two-dimensional logical table in a database. The preset data table template can provide a basis for data structure transformation and can identify the two-dimensional logical representation of the data after transformation. Unstructured data can be converted into structured data through the preset data table template, and the data in the source data table becomes structured data.

[0087] The preset data template can be defined according to the data format that the anomaly monitoring platform can receive, and can be updated or redefined according to the specific implementation situation.

[0088] S230. Process the data in the source data table to obtain the target data table.

[0089] Specifically, the data in the source data table is structured data, which can be directly processed. Specific processing methods may include data filtering, data aggregation, and data renaming. After a series of processing steps, a target data table is generated based on the data in the processed source data table. Furthermore, since data filtering and other operations have been performed on the data in the source data table, noisy data in the source data table can be filtered out, and data that is helpful for subsequent analysis can be retained. As a result, the data size of the generated target data table is generally smaller than the data size of the corresponding source data table.

[0090] S240. Based on the target data table, generate a target data stream in the target data format.

[0091] In one optional embodiment, a data item in the target data table can correspond to a data stream. By combining the data corresponding to each field of each data item, the data stream corresponding to that data item can be obtained.

[0092] Specifically, in each target data stream, the data corresponding to each field information can be directly combined without being separated by delimiters, thereby saving the resources required for data transmission and network transmission resources.

[0093] S250. Send the target data stream to the target data storage module.

[0094] The target data storage module can be a module set up on the anomaly monitoring platform to store target data streams. When the target data storage module receives the target data stream, it can either write the target data stream directly into the target data storage module or store the target data stream in a preset data stream format.

[0095] In an optional embodiment, the data stream processing end may include a business stream processing end and a data table processing end, wherein the business stream processing end may correspond to each business party, and the above steps S210-S220 may be executed by the business stream processing end, and steps S230-S250 may be executed by the data table processing end; that is, each business stream processing end first performs data structure transformation on its own business data stream to obtain the source data table of each business party, and then processes the source data table based on the data table processing end to obtain the target data table; a target data stream is generated based on the target data table; and the target data stream is sent to the target data storage module.

[0096] In another optional embodiment, the business flow processing terminal can be set up at each business party, and the data table processing terminal can be set up in the background. In this way, each business party can perform data structure transformation on its own source data flow through the business flow processing terminal to obtain the source data table; and then send the source data table to the data table processing terminal; thereby improving the efficiency of business flow processing and reducing the data processing pressure on the data table processing terminal.

[0097] This disclosure enables the source data stream to be written into the target data storage module by performing data structure transformation, processing, and re-transformation on the source data stream. The target data storage module is used to store data in the target data format. This does not require modification of the backend code, and the same data stream access method can be used for data streams from different sources, thereby improving the scalability of data stream access and enhancing the stability of the anomaly monitoring system.

[0098] In one optional embodiment, the preset data table template includes multiple field information; accordingly, please refer to... Figure 3 It illustrates a method for generating a source data table, which may include:

[0099] S310. Extract first data information corresponding to the multiple field information from the source data stream.

[0100] S320. The first data information is filled into their respective field information to obtain the source data table.

[0101] Specifically, the preset data table template may include data identifier fields, event type fields, generation timestamp fields, etc., thereby extracting relevant data from the source data stream based on this field information, and filling the extracted data information into the corresponding field information to obtain the source data table. Thus, by extracting data from the source data stream by field and filling it into the preset data table template, the transformation from unstructured data to structured data is achieved, as well as a two-dimensional logical representation of the source data is realized.

[0102] In one optional embodiment, when the source data stream includes multiple data streams, the multiple data streams can be processed in batches to obtain a two-dimensional logical representation corresponding to the multiple data streams, thereby generating the corresponding source data table. Since the data streams are continuously generated, batch processing can improve the efficiency of data structure transformation.

[0103] In one example, a preset data table template can be a data structure conversion protocol, which can be a data structure conversion function. When data structure conversion is required, the preset data table template can be called, and the source data stream can be directly input into the preset data table template to obtain a two-dimensional logical expression corresponding to the source data stream, thereby generating the corresponding source data table.

[0104] In another optional embodiment, when the source data stream includes multiple data streams, the multiple data streams can be used as input parameters of a preset data table template. Each pair of data streams can be separated by a preset delimiter to avoid confusion between data streams, thereby obtaining a two-dimensional logical expression corresponding to each of the multiple data streams, and then generating the corresponding source data table.

[0105] In one specific embodiment, please refer to Figure 4 It illustrates another data stream processing method, which may include:

[0106] S410. Obtain the source data stream.

[0107] In one specific embodiment, the source data stream can be a real-time data stream generated during the operation of a project or application; the source data stream can be obtained through reporting from collection points in the project, or it can be obtained through information collection tools in the project.

[0108] S420. Based on the preset data table template, perform data structure transformation on the source data streams from the multiple sources respectively to obtain source data tables corresponding to each of the multiple source data streams; the preset data table template corresponds to the target data format.

[0109] S430. Process the data in the source data table to obtain the target data table.

[0110] S440. Based on the target data table, generate a target data stream in the target data format.

[0111] S450. Send the target data stream to the target data storage module.

[0112] For specific implementation details of steps S430-S450, please refer to steps S230-S250 above, which will not be repeated here.

[0113] Specifically, source data streams from different sources can refer to source data streams from different projects, source data streams from different data collection platforms, etc. This allows for separate processing of source data streams from different sources, generating source data tables corresponding to each source. Each source data table includes data information from one source, avoiding the problem of data information confusion between different sources and improving the efficiency and convenience of subsequent data processing.

[0114] In one example, the data stream processing end can receive data streams from multiple sources through a single data stream access channel. Each data stream can carry a corresponding source identifier. The data stream processing end can then classify the data streams based on the source identifiers and perform data structure transformation on each source's data stream to obtain a source data table corresponding to each source.

[0115] In another example, the data stream processing end can receive data streams from multiple sources through multiple data stream access channels. The number of data stream access channels can be the same as the number of data stream sources. Thus, each data stream access channel corresponds to a data stream source. Multiple data stream access channels can respectively perform data structure transformation on the source data stream of the corresponding source to obtain a source data table, process the source data table to obtain a target data table, and generate a target data stream based on the target data table. The various data stream access channels do not affect each other and can be implemented in parallel, thereby improving the processing efficiency of the data stream processing end.

[0116] In another specific embodiment, the preset data table template includes multiple field information; please refer to 5, which illustrates a method for generating source data tables corresponding to source data streams from multiple sources, the method may include:

[0117] S510. For each original data stream in the multiple source data streams, determine a preset data table template corresponding to the original data stream.

[0118] S520. Extract second data information corresponding to the multiple field information from the original data stream.

[0119] S530. The second data information is filled into the multiple field information of the preset data table template corresponding to the original data stream to obtain the source data table corresponding to the original data stream.

[0120] As described above, the preset data table template may include data identifier fields, event type fields, generation timestamp fields, etc. Based on this field information, relevant data in the source data stream is extracted, and the extracted data information is filled into the corresponding field information, resulting in a source data table corresponding to each source. Thus, by extracting data from the source data stream by field and filling it into the preset data table template, the transformation from unstructured data to structured data is achieved, as well as a two-dimensional logical representation of the source data is realized.

[0121] In one example, the source data table includes multiple data items, each carrying an event type identifier; please refer to [link to relevant documentation]. Figure 6 It illustrates a method for data processing based on a source data table, which may include:

[0122] S610. Aggregate at least two data items with the same event type identifier to obtain aggregated data items.

[0123] S620. Generate the target data table based on the aggregated data items.

[0124] Specifically, the event type identifier can be used as a field in the source data table. Data items with the same time type identifier in the source data table can be aggregated to obtain aggregated data items. Data streams of the same event type appear at different times. By aggregating them, it is easier to analyze the relationships between the corresponding data streams, thus improving data analysis efficiency. If the event type is an anomaly type, it also helps to locate the time of the anomaly event and determine the relationship between different anomalies.

[0125] In one optional embodiment, each data item may include a generation timestamp, thereby enabling aggregation processing of each data item based on the generation timestamp, such as aggregating data items generated within one hour or aggregating data items generated within one day. By aggregating each data item based on the timestamp, it is easier to analyze the data based on time periods, thereby determining the overall operating data of the corresponding project or application within a preset time period, and analyzing and evaluating its operating status based on the overall operating data.

[0126] In one specific embodiment, the spatial information of the target data storage module can be determined in advance based on the data flow of the data stream processing terminal. Specifically, this includes determining a first quantity of source data streams acquired per unit time; and determining the memory space information of the target data storage module based on the first quantity. This achieves consistency between the data flow and the memory space of the target data storage module, avoiding the waste of space resources caused by allocating too much memory space to the target data storage module, and avoiding the problem of insufficient space resources caused by allocating too little memory space to the target data storage module.

[0127] In one example, the number of data streams sent to the target data storage module can be set. Specifically, a second number of the target data streams can be sent to the target data storage module per unit time; the second number matches the processing speed of the target data storage module. This ensures that the data stream sending speed matches the data stream writing speed, avoiding problems such as excessive writing pressure on the target data storage module due to excessively fast data stream sending speed, which could lead to data stream writing errors or data loss, thus improving the accuracy of data stream writing.

[0128] The following is a specific example illustrating the implementation process of this disclosure, which includes:

[0129] 1. Establish a standard data protocol between the data stream processing end and the anomaly monitoring end.

[0130] 2. The anomaly monitoring terminal requests physical resources and a data storage module for storing data streams based on indicators such as traffic and throughput of the connected business terminal; the data storage module is uniquely defined by the anomaly monitoring terminal.

[0131] 3. The data storage module is a streaming data "channel," an unformatted data structure that cannot be processed by existing real-time data processing systems. Standard logical tables can be defined and bound to the data storage module, thereby enabling formatted data definition.

[0132] 4. The data flow processing end can import the standard logical table defined above to realize the definition of the logical table (the standard has been set, and only a copy-like operation is needed). The data flow of each business party is processed so that the data can be written from the data storage module of each business party to the data storage module of the abnormal monitoring end in step 2.

[0133] 5. Based on the logical tables in section 4, use the structured query language editing panel to write structured query logic to import data from the business side's data storage module to the exception monitoring end's data storage module.

[0134] 6. The data that falls into the data storage module of the anomaly monitoring terminal will eventually be processed by the anomaly monitoring terminal through the processing interface of the real-time data processing platform and sent to the database to provide a data source for subsequent use.

[0135] 7. Since there is already abnormal data in the data, the subsequent process only requires simple front-end and back-end interaction, and the relevant business abnormal data can be obtained through the monitoring interface.

[0136] Please refer to the details. Figure 7It shows a real-time data stream processing framework diagram, which imports "endpoint-type abnormal data" from the data streams of the respective business parties into the backend streaming system of the abnormal monitoring end. Figure 7 It is explicitly stated that the real-time data processing platform can handle logical tables. Therefore, in this disclosure, the anomaly monitoring end, which receives the data, needs to create a standard logical table according to its requirements, i.e., the operation in process 3. In this case, the data outflow end generally does not need to define an additional logical table (it likely already exists); if it does not exist, it needs to be created. In the real-time data processing platform, the relevant tools provided by the platform are used to perform routine structured query operations such as filtering, aggregating, and renaming of source data to destination data. Of course, to avoid a sudden increase in processing pressure on the anomaly monitoring end, rate limiting is also necessary.

[0137] This disclosure achieves the purpose of business streaming data aggregation by defining a unique data storage module, and achieves the purpose of being processed by a real-time data processing platform by binding a formatted data target logical table (similar to a database table structure) to the data storage module. Then, the real-time data processing system writes the data from each business line from different business party data storage modules to the abnormal monitoring end data storage module, thereby realizing data flow docking, thus achieving multi-party reuse of a single channel, and the system is stable, has a clear protocol, and low maintenance costs.

[0138] Figure 8 A data stream processing apparatus according to an exemplary embodiment includes:

[0139] The source data stream acquisition unit 810 is configured to acquire the source data stream.

[0140] The first data structure conversion unit 820 is configured to perform data structure conversion on the source data stream based on a preset data table template to obtain a source data table; the preset data table template corresponds to the target data format.

[0141] The target data table generation unit 830 is configured to process the data in the source data table to obtain the target data table;

[0142] The target data stream generation unit 840 is configured to generate a target data stream based on the target data table and the target data format.

[0143] The first sending unit 850 is configured to send the target data stream to the target data storage module.

[0144] In one exemplary embodiment, the preset data table template includes multiple field information;

[0145] The first data structure conversion unit 820 includes:

[0146] The first extraction unit is configured to extract first data information corresponding to the multiple field information from the source data stream;

[0147] The first filling unit is configured to fill the first data information into their respective corresponding field information to obtain the source data table.

[0148] In one exemplary embodiment, the source data stream includes source data streams from multiple data sources;

[0149] The first data structure conversion unit 820 includes:

[0150] The second data structure conversion unit is configured to perform data structure conversion on the source data streams from the multiple sources based on the preset data table template, so as to obtain the source data table corresponding to each of the multiple source data streams.

[0151] In one exemplary embodiment, the preset data table template includes multiple field information;

[0152] The second data structure conversion unit includes:

[0153] The first determining unit is configured to perform, for each of the multiple source data streams, determine a preset data table template corresponding to the original data stream;

[0154] The second extraction unit is configured to extract second data information corresponding to the multiple field information from the original data stream;

[0155] The second filling unit is configured to fill the second data information into the multiple field information of the preset data table template corresponding to the original data stream, thereby obtaining the source data table corresponding to the original data stream.

[0156] In one exemplary embodiment, the source data table includes multiple data items, each carrying an event type identifier;

[0157] The target data table generation unit 830 includes:

[0158] The aggregation processing unit is configured to perform aggregation processing on at least two data items with the same event type identifier to obtain aggregated data items;

[0159] The target data table determination unit is configured to generate the target data table based on the aggregated data items.

[0160] In one exemplary embodiment, the apparatus further includes:

[0161] The first quantity determination unit is configured to determine the first quantity of the source data stream acquired within a unit time period;

[0162] The second determining unit is configured to determine the memory space information of the target data storage module based on the first quantity.

[0163] In an exemplary embodiment, the first transmitting unit 850 includes:

[0164] The second sending unit is configured to send a second number of the target data streams to the target data storage module within a unit time; the second number is matched with the processing speed of the target data storage module.

[0165] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0166] In an exemplary embodiment, a computer-readable storage medium including instructions is also provided. Optionally, the computer-readable storage medium may be a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device, etc. When the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is able to perform any of the methods described above.

[0167] In an exemplary embodiment, a computer program product is also provided, the computer program product including a computer program stored in a readable storage medium, wherein at least one processor of a computer device reads from the readable storage medium and executes the computer program, causing the device to perform any of the methods described above.

[0168] This embodiment also provides an electronic device, the structural diagram of which can be found in [reference needed]. Figure 9The device 900 can vary significantly in configuration or performance, and may include one or more central processing units (CPUs) 922 (e.g., one or more processors) and memory 932, and one or more storage media 930 (e.g., one or more mass storage devices) storing application programs 942 or data 944. The memory 932 and storage media 930 may be temporary or persistent storage. The program stored in the storage media 930 may include one or more modules (not shown), each module including a series of instruction operations on the device. Furthermore, the CPU 922 may be configured to communicate with the storage media 930 and execute the series of instruction operations in the storage media 930 on the device 900. The device 900 may also include one or more power supplies 926, one or more wired or wireless network interfaces 950, one or more input / output interfaces 958, and / or one or more operating systems 941, such as Windows Server. TM Mac OS X TM Unix TM Linux TM FreeBSD TM Etc. Any of the methods described above in this embodiment can be based on... Figure 9 The equipment shown is used for implementation.

[0169] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the following claims.

[0170] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. A data stream processing method, characterized in that, include: Obtain the source data stream; The source data stream is transformed based on a preset data table template to obtain a source data table; The preset data table template corresponds to the target data format; The preset data table template is defined according to the data format that the anomaly monitoring platform can receive; the source data table includes multiple data items, each of which carries an event type identifier; Aggregate at least two data items with the same event type identifier to obtain aggregated data items; Generate a target data table based on the aggregated data items; Based on the target data table, generate a target data stream in the target data format; Send the target data stream to the target data storage module; The target data storage module is a module installed on the anomaly monitoring platform for storing the target data stream; When data streams with the same event type appear at different times in the aggregated data items, and the event type is an anomaly type, the anomaly monitoring platform locates the time of occurrence of the anomaly event and the relationship between each anomaly event. Determine the first quantity of source data streams acquired per unit time; Based on the first quantity, determine the memory space information of the target data storage module; Sending the target data stream to the target data storage module includes: Send a second number of the target data streams to the target data storage module within a unit of time; The second quantity is matched with the processing speed of the target data storage module.

2. The method according to claim 1, characterized in that, The preset data table template includes multiple fields; The process of transforming the source data stream based on a preset data table template to obtain a source data table includes: Extract first data information corresponding to the multiple field information from the source data stream; The first data information is filled into their respective field information to obtain the source data table.

3. The method according to claim 1, characterized in that, The source data stream includes source data streams from multiple data sources; The process of transforming the source data stream based on a preset data table template to obtain the source data table includes: Based on the preset data table template, the data structure of the multiple source data streams is transformed to obtain the source data table corresponding to each of the multiple source data streams.

4. The method according to claim 3, characterized in that, The preset data table template includes multiple fields; The step of performing data structure transformation on the source data streams from multiple sources based on the preset data table template to obtain source data tables corresponding to each of the multiple source data streams includes: For each original data stream in the multiple source data streams, determine a preset data table template corresponding to the original data stream; Extract second data information corresponding to the multiple field information from the original data stream; The second data information is filled into the multiple fields of the preset data table template corresponding to the original data stream to obtain the source data table corresponding to the original data stream.

5. A data stream processing apparatus, characterized in that, include: The source data stream acquisition unit is configured to acquire the source data stream; The first data structure conversion unit is configured to perform data structure conversion on the source data stream based on a preset data table template to obtain a source data table. The preset data table template corresponds to the target data format; The preset data table template is defined according to the data format that the anomaly monitoring platform can receive; the source data table includes multiple data items, each of which carries an event type identifier; The target data table generation unit is configured to process the data in the source data table to obtain the target data table; The target data table generation unit includes: an aggregation processing unit configured to perform aggregation processing on at least two data items with the same event type identifier to obtain aggregated data items; and a target data table determination unit configured to generate the target data table based on the aggregated data items. The target data stream generation unit is configured to generate a target data stream based on the target data table and the target data format. The first sending unit is configured to send the target data stream to the target data storage module; the target data storage module is a module set in the anomaly monitoring platform for storing the target data stream; when data streams with the same event type appear at different time points in the aggregated data items, and the event type is an anomaly type, the anomaly monitoring platform locates the time of occurrence of the anomaly event and the relationship between each anomaly event; The first quantity determination unit is configured to determine the first quantity of the source data stream acquired within a unit time period; The second determining unit is configured to determine the memory space information of the target data storage module based on the first quantity; The first transmitting unit includes: The second sending unit is configured to send a second number of the target data streams to the target data storage module within a unit time; the second number is matched with the processing speed of the target data storage module.

6. The apparatus according to claim 5, characterized in that, The preset data table template includes multiple fields; The first data structure conversion unit includes: The first extraction unit is configured to extract first data information corresponding to the multiple field information from the source data stream; The first filling unit is configured to fill the first data information into their respective corresponding field information to obtain the source data table.

7. The apparatus according to claim 5, characterized in that, The source data stream includes source data streams from multiple data sources; The first data structure conversion unit includes: The second data structure conversion unit is configured to perform data structure conversion on the source data streams from the multiple sources based on the preset data table template, so as to obtain the source data table corresponding to each of the multiple source data streams.

8. The apparatus according to claim 7, characterized in that, The preset data table template includes multiple fields; The second data structure conversion unit includes: The first determining unit is configured to perform, for each of the multiple source data streams, determine a preset data table template corresponding to the original data stream; The second extraction unit is configured to extract second data information corresponding to the multiple field information from the original data stream; The second filling unit is configured to fill the second data information into the multiple field information of the preset data table template corresponding to the original data stream, thereby obtaining the source data table corresponding to the original data stream.

9. An electronic device, characterized in that, include: processor; Memory used to store the processor's executable instructions; The processor is configured to execute the instructions to implement the data stream processing method as described in any one of claims 1 to 4.

10. A computer-readable storage medium, characterized in that, When the instructions in the computer-readable storage medium are executed by the processor of the electronic device, the electronic device is able to perform the data stream processing method as described in any one of claims 1 to 4.

11. A computer program product, comprising a computer program / instructions, characterized in that, When the computer program / instructions are executed by the processor, they implement the data stream processing method according to any one of claims 1 to 4.

Images