Statistical method and system for structured data in streaming processing and electronic equipment

Abstract

The invention provides a statistical method and system for structured data in streaming processing and electronic equipment, and relates to the technical field of data processing.The statistical method comprises the steps that the structured data of a streaming processing system are obtained, and a statistical strategy corresponding to the structured data is configured based on the format attribute structure of the structured data; performing statistical conversion on each piece of structured data according to the target statistical function to obtain a statistical conversion result; and on the basis of structures and statistical grading conditions of various types of structured data in the statistical strategy, performing classification and integration processing on each statistical conversion result to obtain a target statistical result. According to the method, the real-time performance of data processing in a streaming system is improved, and the statistical ability of the data is improved.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a statistical method, system and electronic device for structured data in streaming processing. Background Art

[0002] With the rapid development of the Internet and the Internet of Things, the speed and scale of data generation for communication traffic analysis are growing exponentially. Traditional batch statistical methods usually perform centralized analysis on accumulated data after a period of time, rather than processing each packet or stream in real time. As a result, they lack real-time performance and cannot respond to sudden security threats or performance issues in a timely manner. They may miss detection of instantaneous traffic peaks or short-lived attacks, and cannot meet the needs of real-time data processing. Summary of the invention

[0003] The purpose of this application is to provide a statistical method, system and electronic device for structured data in stream processing to alleviate the above-mentioned technical problems existing in the prior art.

[0004] In a first aspect, the present invention provides a statistical method for structured data in stream processing, comprising:

[0005] Acquire structured data of a streaming processing system, and configure a statistical strategy corresponding to the structured data based on a format attribute structure of the structured data;

[0006] Perform statistical transformation on each structured data according to the target statistical function to obtain a statistical transformation result;

[0007] Based on the structure of each type of structured data in the statistical strategy and the statistical classification conditions, each statistical conversion result is classified and integrated to obtain the target statistical result.

[0008] In an optional implementation, configuring a statistical strategy corresponding to the structured data based on the format attribute name of the structured data includes:

[0009] Based on the classification and format attribute structure of the structured data table, the statistical strategy corresponding to the structured data is configured using an XML file.

[0010] In an optional embodiment, the target statistical function includes a hierarchical conditional statistical function, a counting statistical function and a sum statistical function.

[0011] In an optional implementation, statistical transformation is performed on each piece of structured data according to a target statistical function to obtain a statistical transformation result, including:

[0012] Acquiring attribute content of the structured data based on the format attribute structure of the structured data;

[0013] The attribute content of the structured data is subjected to hierarchical conditional statistics according to the value of the key value determination function to obtain hierarchical statistical results; wherein the key value determination function is used to specify whether a certain value is to be processed as a key.

[0014] In an optional implementation, statistical transformation is performed on each piece of structured data according to a target statistical function to obtain a statistical transformation result, including:

[0015] Acquiring attribute content of the structured data based on the format attribute structure of the structured data;

[0016] Determining whether the attribute content of the structured data has a valid value;

[0017] If yes, counting processing is performed according to the counting statistics function to obtain a counting statistics result.

[0018] In an optional implementation, statistical transformation is performed on each piece of structured data according to a target statistical function to obtain a statistical transformation result, including:

[0019] Acquiring attribute content of the structured data based on the format attribute structure of the structured data;

[0020] The attribute content of the structured data is summed and counted according to the summation and statistics function, and numerical attributes are limited. When the numerical attributes are met, summation and statistics are performed to obtain a summation and statistics result.

[0021] In an optional embodiment, the method further comprises:

[0022] The logs stored in real time in each statistical interval are sorted into a Json structure according to the structured data classification and grading conditions;

[0023] The attribute name in the log file corresponding to the statistical strategy is used as the key value of the Json string corresponding to the Json structure, and the entire log file is output to the specified directory.

[0024] In a second aspect, the present invention provides a statistical system for structured data in stream processing, comprising:

[0025] A configuration module, used to obtain structured data of a streaming processing system, and configure a statistical strategy corresponding to the structured data based on a format attribute structure of the structured data;

[0026] A conversion module, used to perform statistical conversion on each structured data according to a target statistical function to obtain a statistical conversion result;

[0027] The statistical module is used to classify and integrate each of the statistical conversion results based on the structure of each type of structured data in the statistical strategy and the statistical classification conditions to obtain the target statistical result.

[0028] In a third aspect, the present invention provides an electronic device comprising a processor and a memory, wherein the memory stores computer-executable instructions that can be executed by the processor, and the processor executes the computer-executable instructions to implement the statistical method of structured data in streaming processing as described in any of the aforementioned embodiments.

[0029] In a fourth aspect, the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions. When the computer-executable instructions are called and executed by a processor, the computer-executable instructions prompt the processor to implement the statistical method for structured data in streaming processing as described in any one of the aforementioned embodiments.

[0030] The statistical method, system and electronic device for structured data in streaming processing provided by the present application have the following beneficial effects:

[0031] By configuring the corresponding statistical strategy and performing statistical transformation on each structured data according to the target statistical function, the statistical transformation results are obtained, and then the corresponding classification and integration processing is performed according to the statistical strategy, thereby improving the real-time performance of data processing in the streaming system and improving the statistical capabilities of the data. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] In order to more clearly illustrate the specific implementation methods of the present application or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0033] Figure 1 A flowchart of a statistical method for structured data in stream processing provided in an embodiment of the present application;

[0034] Figure 2 A flowchart of a data processing process provided by an embodiment of the present application;

[0035] Figure 3 A structural diagram of a statistical system for structured data in stream processing provided in an embodiment of the present application;

[0036] Figure 4 A structural diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0037] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. The components of the embodiments of the present application described and shown in the drawings here can be arranged and designed in various different configurations.

[0038] Therefore, the following detailed description of the embodiments of the present application provided in the accompanying drawings is not intended to limit the scope of the present application for which protection is sought, but merely represents selected embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without creative work are within the scope of protection of the present application.

[0039] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, further definition and explanation thereof is not required in subsequent drawings.

[0040] The present application embodiment provides a statistical method for structured data in stream processing, see Figure 1 As shown, the method mainly includes the following steps:

[0041] Step S110, acquiring structured data of the streaming processing system, and configuring a statistical strategy corresponding to the structured data based on the format attribute structure of the structured data.

[0042] In streaming systems, structured data structures refer to data with a well-defined format and organization so that the data can be quickly parsed, processed, and stored. Compared with unstructured data (such as text files or images), structured data usually follows a predefined data model or schema, which helps improve processing efficiency and simplify the process of data analysis.

[0043] Optionally, the above-mentioned structured data may include, for example, tabular data, lightweight data exchange format data (such as JSON data), row storage format data (such as Avro data), serialization mechanism data (such as Protocol Buffers data), big data in batch processing scenarios (such as Parquet / ORC file format data), and message format data in message queues.

[0044] In one implementation, the statistical strategy corresponding to the structured data is configured based on the format attribute name of the structured data. The statistical strategy corresponding to the structured data can be configured using an XML file based on the classification and format attribute structure of the structured data table.

[0045] In the specific implementation, you can first define an XML Schema (XSD) to specify the structure and content of the XML configuration file, so as to ensure that all configuration files follow the same format and can be correctly understood by the parsing tool. Then, create the actual configuration file according to the defined Schema. <field>The elements correspond to the fields in the table and their statistical strategies. Then, write code to read and parse the XML configuration file, and then perform the corresponding statistical operations according to the configuration. For example, in Python, you can use the xml.etree.ElementTree or lxml library to parse XML files and implement statistical methods for different field types (such as counting, summing, average calculation, etc.). Finally, after completing the above steps, the entire process needs to be fully tested, including but not limited to unit testing, integration testing, and performance testing, to ensure that the configuration file can be parsed correctly and the statistical strategy can work as expected.

[0046] Step S120, performing statistical transformation on each piece of structured data according to the target statistical function to obtain a statistical transformation result.

[0047] The above-mentioned target statistical functions include hierarchical conditional statistical functions, counting statistical functions and sum statistical functions.

[0048] Applied to the above target statistical function, for different statistical requirements, the following different methods are used for statistical conversion processing.

[0049] (1) Statistics of classification conditions

[0050] Obtaining attribute content of structured data based on the format attribute structure of the structured data;

[0051] The attribute content of the structured data is statistically graded according to the value of the key-value determination function to obtain a graded statistical result; wherein the key-value determination function is used to specify whether a certain value is treated as a key, and the key-value determination function is also the subsequent iskey.

[0052] (2) Counting statistics

[0053] Obtaining attribute content of structured data based on the format attribute structure of the structured data;

[0054] Determine whether the attribute content of structured data has a valid value;

[0055] If there is, the counting process is performed according to the counting statistics function to obtain the counting statistics result. The counting statistics function is also the subsequent getnum.

[0056] (3) Sum statistics

[0057] Obtaining attribute content of structured data based on the format attribute structure of the structured data;

[0058] The attribute content of the structured data is summed and counted according to the summation and statistics function, and the numerical attributes are limited. When the numerical attributes are met, the summation and statistics are performed to obtain the summation and statistics results. The summation and statistics function is also the subsequent getsum.

[0059] Step S130, based on the structure of each type of structured data in the statistical strategy and the statistical classification conditions, each statistical conversion result is classified and integrated to obtain the target statistical result.

[0060] In one implementation, the statistical results of each streaming process may be summarized and collated to obtain target statistical results, which are then stored according to the classification and statistical dimensions of structured data.

[0061] For ease of understanding, Figure 2 A schematic diagram of a data processing process is shown, which mainly includes the following parts:

[0062] 1. Structured Data

[0063] In the stream processing system, the structured data structure has corresponding naming and encoding for each attribute in the processing link.

[0064] 2. Statistical strategy

[0065] In one implementation, based on the format attribute name of the structured data, an XML file may be used to configure statistical requirements corresponding to the structured data attributes to implement a statistical strategy for the structured data.

[0066] In another implementation, the XML file configuration may be performed based on the classification and attribute structure of the structured data table.

[0067] In one example, a sample of a certain type of structured data statistics configuration is shown:

[0068] < / field> The element represents a data table, and

[0069]

[0070]

[0071]

[0072]

[0073]

[0074]

[0075]

[0076] <table name＝"table_0003"mid＝"102000012173"><ITEM action="getvalue"src_code="SRC0001"iskey="true"dst_code="DST0001"def="value1"desc="TEST_1" / ><ITEM action="getvalue"src_code="SRC0002"iskey="false"dst_code="DST0002"def="value2"desc="TEST_2" / ><ITEM action="getnum"src_code="SRC0003"iskey="false"dst_code="DST0003"def="0"desc="TEST_3" / ><ITEM action="getnum"src_code="SRC0004"iskey="false"dst_code="DST0004"def="0"desc="TEST_4" / ><ITEM action="getnum"src_code="SRC0005"iskey="false"dst_code="DST0005"def="0"desc="TEST_5" / ><ITEM action="getsum"src_code="SRC0006"iskey="false"dst_code="DST0006"def="0"desc="TEST_6" / ><ITEM action="getsum"src_code="SRC0007"iskey="false"dst_code="DST0007"def="0"desc="TEST 7" / >

[0077]

[0078]

[0079]

[0080]

[0081]

[0082]

[0083]

[0084]

[0085] <table name＝"table_0004"mid＝"102000012173"><ITEM action="getvalue"src_code="SRC0011"iskey="true"dst_code="DST0011"def="value3"desc="TEST_11" / ><ITEM action="getvalue"src_code="SRC0012"iskey="false"dst_code="DST0012"def="value4"desc="TEST_12" / ><ITEM action="getnum"src_code="SRC0013"iskey="false"dst_code="DST0013"def="0"desc="TEST_13" / ><ITEM action="getnum"src_code="SRC0014"iskey="false"dst_code="DST0014"def="0"desc="TEST_14" / ><ITEM action="getnum"src_code="SRC0015"iskey="false"dst_code="DST0015"def="0"desc="TEST_15" / ><ITEM action="getsum"src_code="SRC0016"iskey="false"dst_code="DST0016"def="0"desc="TEST_16" / ><ITEM action="getsum"src_code="SRC0017"iskey="false"dst_code="DST0017"def="0"desc="TEST_17" / >

[0086] Among them, name corresponds to the structured data table, src_code corresponds to the attribute of the structured data, mid is the assigned module ID, and dst_code is the attribute name recorded in the log file. The attribute content of the statistical business information can be hidden through mid and dst_code. In the case of only statistical log files, no information leakage will occur. desc describes the current statistical attribute, which is convenient for interpreting XML and has no practical meaning in code processing.

[0087] (III) Data Statistical Conversion

[0088] In the streaming processing system, a processing link is added for each piece of data, and the results are converted according to the statistical strategy, which only involves non-null judgment and value-taking operations, and has a very low impact on the performance of structured processing in the system.

[0089] Perform statistical conversion on a single structured data. The conversion result of each data will be sent to the data statistics process. Three statistical methods are supported: getvalue, getnum, and getsum.

[0090] getvalue: Get the content of the attribute and use it with iskey to achieve the effect of statistical grading conditions.

[0091] getnum: counts the attribute content. If the attribute content in the data has a value, it is recorded as 1; if it has no value, it is recorded as 0.

[0092] getsun: sums and counts the attribute content. This function is limited to numerical attributes. If the attribute has a value in the data, the content is used as the statistical value. If it has no value, it is recorded as 0.

[0093] iskey: only takes effect in getvalue and is used as a condition for grouping statistics.

[0094] def: Attribute default value, used when the attribute has no value in the data.

[0095] In one example, the example effect is as follows:

[0096] {Sample 1,Value2,0,1,1,3,5},{Sample 1,Value3,0,1,1,2,6},{Sample 1,Value2,1,1,1,5,5},{Sample 1,Value3,0,1,1,4,6},{Sample 1,Value1,1,0,1,7,3},{Sample 2,Value4,1,0,1,5,4},{Sample 2,Value5,1,0,0,1,6},{Sample 2,Value4,0,0,1,3,4},{Sample 2,Value4,0,0,1,6,2}. Converted into a table, it shows the following effect:

[0097] DST0001 DST0002 DST0003 DST0004 DST0005 DST0006 DST0007 Example 1 Value1 1 0 1 3 5 Example 1 Value2 0 1 1 2 6 Example 1 Value3 0 1 1 5 5 Example 1 Value2 1 1 1 4 6 Example 1 Value3 0 1 1 7 3 Example 2 Value4 1 0 1 5 4 Example 2 Value5 1 0 0 1 6 Example 2 Value4 0 0 1 3 4 Example 2 Value4 1 1 0 6 2

[0098] (IV) Data Statistics

[0099] Separate statistics process:

[0100] (1) The statistical results of each streaming process are summarized and organized, and stored according to the classification and statistical dimensions of structured data.

[0101] Based on the structure of various structured data in the statistical strategy and the statistical classification conditions, each result of the data statistical conversion is processed to achieve the effect of sub-library and sub-table storage. The processing method is: getvalue takes the value, the part with iskey="true" is used as the primary key, the part with iskey="false" takes the latest value, getnum and getsun are summed and incremental processing is performed based on the primary key. The example effect is as follows:

[0102] DST0001 DST0002 DST0003 DST0004 DST0005 DST0006 DST0007 Example 1 Value3 2 4 5 21 25 Example 2 Value4 3 1 2 15 16

[0103] (2) Statistical log printing, writing the stored statistical results to a file in JSON structure. In specific implementation, the logs stored in real time within each statistical interval can be sorted into a JSON structure according to the classification and grading conditions of structured data; the attribute names (i.e., dst_code) in the log file corresponding to the statistical policy are used as the key values of the JSON string corresponding to the JSON structure, and the entire log file is output to a specified directory.

[0104] An example of the content of the JSON log file is as follows:

[0105] {"MID":102000012173,data:[{"DST0001":"Example 1","DST0002":"Value3","DST0003":2,"DST0004":4,"DST0005":5,"DST0006":21,"DST0001":25},{"DST0001":"Example 2","DST0002":"Value4","DST0003":3,"DST0004":1,"DST0005":2,"DST0006":15,"DST0001":16}]}。

[0106] In summary, by configuring the corresponding statistical policy and performing statistical conversion on each piece of structured data according to the target statistical function to obtain the statistical conversion result, and then performing corresponding classification and integration processing according to the statistical policy, the real-time performance of data processing in the streaming system is improved, and the statistical ability of data is enhanced.

[0107] Based on the above method embodiments, the embodiments of the present application further provide a statistical system for structured data in stream processing. Refer to Figure 3 as shown, the system includes the following parts:

[0108] A configuration module 310, configured to obtain the structured data of the stream processing system and configure the statistical policy corresponding to the structured data based on the format attribute structure of the structured data;

[0109] A conversion module 320, configured to perform statistical conversion on each piece of structured data according to the target statistical function to obtain the statistical conversion result;

[0110] A statistical module 330, configured to perform classification and integration processing on each statistical conversion result based on the structure and statistical grading conditions of various structured data in the statistical policy to obtain the target statistical result.

[0111] In a feasible implementation manner, the above configuration module 310 is further configured to:

[0112] Based on the classification and format attribute structure of the structured data table, the statistical strategy corresponding to the structured data is configured using an XML file.

[0113] In a feasible implementation manner, the target statistical function includes a hierarchical conditional statistical function, a counting statistical function, and a sum statistical function.

[0114] In a feasible implementation manner, the conversion module 320 is further used for:

[0115] Obtaining attribute content of structured data based on the format attribute structure of the structured data;

[0116] The attribute content of the structured data is statistically graded according to the value of the key-value determination function to obtain a graded statistical result; wherein the key-value determination function is used to specify whether a certain value is to be processed as a key.

[0117] In a feasible implementation manner, the conversion module 320 is further used for:

[0118] Obtaining attribute content of structured data based on the format attribute structure of the structured data;

[0119] Determine whether the attribute content of structured data has a valid value;

[0120] If yes, counting processing is performed according to the counting statistics function to obtain the counting statistics result.

[0121] In a feasible implementation manner, the conversion module 320 is further used for:

[0122] Obtaining attribute content of structured data based on the format attribute structure of the structured data;

[0123] The attribute content of the structured data is summed and counted according to the summation and statistics function, and numerical attributes are limited. When the numerical attributes are met, summation and statistics are performed to obtain summation and statistics results.

[0124] In a feasible implementation manner, the system further includes: a log output module, which is used to:

[0125] The logs stored in real time in each statistical interval are sorted into a Json structure according to the structured data classification and grading conditions;

[0126] The attribute name in the log file corresponding to the statistical strategy is used as the key value of the Json string corresponding to the Json structure, and the entire log file is output to the specified directory.

[0127] The statistical system for structured data in streaming processing provided in the embodiments of the present application has the same implementation principle and technical effects as those of the aforementioned method embodiments. For the sake of brief description, for matters not mentioned in the embodiment of the statistical system for structured data in streaming processing, reference may be made to the corresponding contents in the aforementioned statistical method embodiment for structured data in streaming processing.

[0128] The present application also provides an electronic device, such as Figure 4 As shown, it is a schematic diagram of the structure of the electronic device, wherein the electronic device 100 includes a processor 41 and a memory 40, the memory 40 stores computer executable instructions that can be executed by the processor 41, and the processor 41 executes the computer executable instructions to implement any of the above-mentioned statistical methods for structured data in streaming processing.

[0129] exist Figure 4 In the illustrated embodiment, the electronic device further includes a bus 42 and a communication interface 43 , wherein the processor 41 , the communication interface 43 and the memory 40 are connected via the bus 42 .

[0130] Among them, the memory 40 may include a high-speed random access memory (RAM), and may also include a non-volatile memory (non-volatile memory), such as at least one disk storage. The communication connection between the system network element and at least one other network element is realized through at least one communication interface 43 (which can be wired or wireless), and the Internet, wide area network, local area network, metropolitan area network, etc. can be used. The bus 42 can be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. The bus 42 can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 4 Only one bidirectional arrow is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0131] The processor 41 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the hardware integrated logic circuit or software instructions in the processor 41. The above processor 41 can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor. The steps of the method disclosed in the embodiment of the present application can be directly embodied as a hardware decoding processor to execute, or the hardware and software modules in the decoding processor can be executed. The software module can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory, and the processor 41 reads the information in the memory and completes the steps of the statistical method for structured data in stream processing in the aforementioned embodiment in combination with its hardware.

[0132] An embodiment of the present application also provides a computer-readable storage medium, which stores computer-executable instructions. When the computer-executable instructions are called and executed by a processor, the computer-executable instructions prompt the processor to implement the statistical method of structured data in the above-mentioned streaming processing. The specific implementation can be found in the aforementioned method embodiment, which will not be repeated here.

[0133] The computer program product of the statistical method, system and electronic device for structured data in streaming processing provided in the embodiments of the present application includes a computer-readable storage medium storing program code. The instructions included in the program code can be used to execute the methods in the previous method embodiments. The specific implementation can be found in the method embodiments, which will not be repeated here.

[0134] Unless otherwise specifically stated, the relative steps, numerical expressions and values ​​of the components and steps set forth in these embodiments do not limit the scope of the present application.

[0135] If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a non-volatile computer-readable storage medium that is executable by a processor. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, and other media that can store program codes.

[0136] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A statistical method for structured data in stream processing, characterized in that: include: Acquire structured data of a streaming processing system, and configure a statistical strategy corresponding to the structured data based on a format attribute structure of the structured data; Perform statistical transformation on each structured data according to the target statistical function to obtain a statistical transformation result; Based on the structure of each type of structured data in the statistical strategy and the statistical classification conditions, each statistical conversion result is classified and integrated to obtain the target statistical result.

2. The statistical method for structured data in stream processing according to claim 1, characterized in that: Configuring a statistical strategy corresponding to the structured data based on the format attribute name of the structured data includes: Based on the classification and format attribute structure of the structured data table, the statistical strategy corresponding to the structured data is configured using an XML file.

3. The statistical method for structured data in stream processing according to claim 1, characterized in that: The target statistical functions include a hierarchical conditional statistical function, a counting statistical function and a sum statistical function.

4. The statistical method for structured data in stream processing according to claim 3, characterized in that: Perform statistical transformation on each structured data according to the target statistical function to obtain statistical transformation results, including: Acquiring attribute content of the structured data based on the format attribute structure of the structured data; The attribute content of the structured data is subjected to hierarchical conditional statistics according to the value of the key value determination function to obtain hierarchical statistical results; wherein the key value determination function is used to specify whether a certain value is to be processed as a key.

5. The statistical method for structured data in stream processing according to claim 3, characterized in that: Perform statistical transformation on each structured data according to the target statistical function to obtain statistical transformation results, including: Acquiring attribute content of the structured data based on the format attribute structure of the structured data; Determining whether the attribute content of the structured data has a valid value; If yes, counting processing is performed according to the counting statistics function to obtain a counting statistics result.

6. The statistical method for structured data in stream processing according to claim 3, characterized in that: Perform statistical transformation on each structured data according to the target statistical function to obtain statistical transformation results, including: Acquiring attribute content of the structured data based on the format attribute structure of the structured data; The attribute content of the structured data is summed and counted according to the summation and statistics function, and numerical attributes are limited. When the numerical attributes are met, summation and statistics are performed to obtain a summation and statistics result.

7. The statistical method for structured data in stream processing according to claim 1, characterized in that: The method further comprises: The logs stored in real time in each statistical interval are sorted into a Json structure according to the structured data classification and grading conditions; The attribute name in the log file corresponding to the statistical strategy is used as the key value of the Json string corresponding to the Json structure, and the entire log file is output to the specified directory.

8. A statistical system for structured data in stream processing, characterized in that: include: A configuration module, used to obtain structured data of a streaming processing system, and configure a statistical strategy corresponding to the structured data based on a format attribute structure of the structured data; A conversion module, used to perform statistical conversion on each structured data according to a target statistical function to obtain a statistical conversion result; The statistical module is used to classify and integrate each of the statistical conversion results based on the structure of each type of structured data in the statistical strategy and the statistical classification conditions to obtain the target statistical result.

9. An electronic device, characterized in that: It comprises a processor and a memory, wherein the memory stores computer executable instructions that can be executed by the processor, and the processor executes the computer executable instructions to implement the statistical method for structured data in streaming processing as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions. When the computer-executable instructions are called and executed by a processor, the computer-executable instructions prompt the processor to implement the statistical method for structured data in streaming processing according to any one of claims 1 to 7.

Images