Analytical methods, apparatus, electronic devices and storage media

Abstract

This application discloses an analysis method, apparatus, electronic device, and storage medium. The embodiments of this application, based on the processor source code of the application being analyzed, obtain a first information set corresponding to the analysis system and store it in a database; extract second information corresponding to the stream file to be analyzed from the first information set; determine the strategy mode corresponding to the stream file based on the second information; based on the strategy mode, parse and process the stream file, analyzing whether the header node corresponding to the processed stream file is a sharding processor; if the header node is a sharding processor, perform data association on the processed data obtained by at least one sharding processor through which the stream file passes, obtaining association information; based on the association information, determine the change information corresponding to the stream file through a recursive algorithm, and perform general analysis on the stream file based on the change information. The embodiments of this application provide analysis methods for various scenarios while also improving analysis efficiency.

Description

Technical Field

[0001] This application belongs to the field of analytical technology, specifically relating to an analytical method, apparatus, electronic device, and storage medium. Background Technology

[0002] With the development of the NiFi architecture, efficient processing of big data can be achieved. Its operation is based on workflows, which are composed of several processors. Each processor generates corresponding data source events during the data processing process. Currently, NiFi only persists the data source events generated during the workflow process and there is no universal method for analyzing NiFi data lineage source events.

[0003] Existing methods only support data source event analysis for processors in specific scenarios, and the supported analysis scenarios are relatively limited. Especially when the processing combinations are flexible and varied, the support is extremely poor or even non-existent. When different scenarios occur and workflows are composed of different processors with different combination structures, the analysis cannot be supported. The scalability is also poor, and it cannot provide users with analysis methods for multiple scenarios, thus reducing the efficiency of analysis. Summary of the Invention

[0004] The purpose of this application is to provide an analysis method, apparatus, electronic device, and storage medium that can solve the problem of low analysis efficiency in the analysis process of related technologies, and the inability to provide users with analysis methods in various scenarios.

[0005] In a first aspect, embodiments of this application provide an analysis method, the method comprising:

[0006] Based on the processor source code of the application being analyzed, a first information set corresponding to the analysis system is obtained and stored in a database; the first information set includes the type of each processor in the analysis system and the configuration information of the processor.

[0007] Extract the second information corresponding to the stream file to be analyzed from the first information set; the second information includes the type and configuration information of each processor corresponding to the stream file.

[0008] The policy mode corresponding to the stream file is determined based on the second information; the policy mode is used to instruct the processor to process the stream file according to the corresponding policy mode;

[0009] Based on the strategy pattern, the stream file is parsed and processed to analyze whether the header node corresponding to the stream file is a fragmentation processor;

[0010] When the head node is a sharding processor, data association is performed on the processed data obtained by at least one sharding processor through which the stream file passes to obtain association information.

[0011] Based on the associated information, a recursive algorithm is used to determine the change information corresponding to the stream file, and a general analysis is performed on the stream file based on the change information.

[0012] Secondly, embodiments of this application provide an analysis apparatus, the apparatus comprising:

[0013] The acquisition module is used to acquire a first information set corresponding to the analysis system based on the processor source code of the application being analyzed, and store it in a database; the first information set includes the type of each processor in the analysis system and the configuration information of the processor.

[0014] The extraction module is used to extract second information corresponding to the stream file to be analyzed from the first information set; the second information includes the type and configuration information of each processor corresponding to the stream file.

[0015] The first determining module is used to determine the policy mode corresponding to the stream file based on the second information; the policy mode is used to instruct the processor to process the stream file according to the corresponding policy mode;

[0016] The analysis module is used to parse the stream file based on the strategy pattern and analyze whether the header node corresponding to the stream file is a fragment processor.

[0017] The acquisition module is used to perform data association on the processed data obtained by at least one shard processor through which the stream file passes, when the head node is a shard processor, and obtain association information.

[0018] The first determining module is further configured to determine the change information corresponding to the stream file based on the association information through a recursive algorithm, and perform a general analysis on the stream file based on the change information.

[0019] Thirdly, embodiments of this application provide an electronic device, which includes a processor, a memory, a communication interface, and a communication bus. The processor, the memory, and the communication interface communicate with each other through the communication bus. The memory is used to store executable instructions, which cause the processor to execute the analysis method as described in any of the preceding claims.

[0020] Fourthly, embodiments of this application provide a readable storage medium on which a program or instructions are stored, which, when executed by a processor, implement the analysis method as described in any of the preceding claims.

[0021] In this embodiment, an analysis method is provided. Based on the processor source code of the application being analyzed, a first information set corresponding to the analysis system is obtained and stored in a database. The first information set includes the type and configuration information of each processor in the analysis system. Second information corresponding to the stream file to be analyzed is extracted from the first information set. The second information includes the type and configuration information of each processor used to process the stream file. A policy mode corresponding to the stream file is determined based on the second information. The policy mode instructs the processor to process the stream file according to the corresponding policy mode. Based on the policy mode, the stream file is parsed, and it is analyzed whether the header node corresponding to the stream file is a sharding processor. If the header node is a sharding processor, data association is performed on the processing data obtained by at least one sharding processor through which the stream file passes, to obtain an association. Information; Based on associated information, a recursive algorithm is used to determine the change information corresponding to the streaming file, and a general analysis of the streaming file is performed based on the change information. During the analysis process, a second information corresponding to the streaming file to be analyzed can be extracted from a pre-stored first information set; then, the strategy mode corresponding to the streaming file is determined based on the second information, and the streaming file is processed. This enables analysis and processing in various scenarios. Furthermore, the analysis system can first determine whether the header node corresponding to the streaming file is a sharding processor, and then, if the header node is a sharding processor, perform data association on the processed data obtained after the sharding processor to obtain association information; finally, based on the association information, a recursive algorithm is used to determine the change information corresponding to the streaming file, and a general analysis of the streaming file is performed based on the change information, improving the efficiency of streaming file analysis. Attached Figure Description

[0022] Figure 1 This is a flowchart of the steps of an analysis method provided in an embodiment of this application;

[0023] Figure 2 This is a flowchart illustrating a NiFi workflow execution example provided in this application embodiment;

[0024] Figure 3 This is a general NiFi data source event analysis flowchart provided in the embodiments of this application;

[0025] Figure 4 This is a flowchart illustrating the analysis of a running instance, provided in an embodiment of this application.

[0026] Figure 5 This is a logic block diagram of an analysis device provided in an embodiment of this application;

[0027] Figure 6This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0028] The technical solutions of the embodiments of this application will be clearly described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application are within the scope of protection of this application.

[0029] The terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such use of data can be interchanged where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class and the number of objects is not limited; for example, a first object can be one or more. Furthermore, in the specification and claims, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0030] Method Implementation Examples

[0031] The analysis method provided in this application will be described in detail below with reference to the accompanying drawings, through specific embodiments and application scenarios.

[0032] Reference Figure 1 The flowchart illustrates the steps of an analysis method provided in an embodiment of this application, as shown below. Figure 1 As shown, the method specifically includes steps S101 to S106:

[0033] Step S101: Based on the processor source code of the application being analyzed, obtain the first information set corresponding to the analysis system and store it in the database; the first information set includes the type and configuration information of each processor in the analysis system.

[0034] For example, in this step, the processor may include processors that enable batch generation of stream files, such as QueryDatabaseTable, QueryDatabaseTableRecord, SelectHive3QL, etc.; processors that can split a stream file into multiple new stream files, such as SpliJson, SplitAvro, etc.; processors that can merge multiple stream files into a new stream file, such as MergeRecord processor, etc.; and processors that can change the content or attributes of the stream file, such as JoltTransformRecord, etc.

[0035] In this step, the analysis system can first obtain the first information set corresponding to the analysis system based on the processor source code of the application being analyzed, and store it in the database. Then, in the subsequent analysis and processing, it can directly retrieve the type and configuration information of each processor in the first information set from the database, and perform analysis and processing based on the type and configuration information of each processor.

[0036] For example, the processor type may include, but is not limited to, sharded processor type, merged processor type, and UUID-varying processor type, and is not limited in the embodiments of this application.

[0037] In this step, the first set of information may include, but is not limited to, the type of the processor (e.g., QueryDatabaseTable), whether the processor is a sharded processor (some processors are sharded, and some processors depend on the value of a certain attribute configuration to determine whether they are sharded, for example, the QueryDatabaseTable processor depends on the value of the "Max RowsPer Flow File" attribute to be sharded, and not sharded if it does not have a value), and what type of data source event will be generated when it is a sharded processor (e.g., RECEIVE event type), whether the processor is a database reader, the number of stream files when reading the database, whether it is a processor that changes the stream file UUID, and the data source type corresponding to changing the stream file UUID, and the data source event type received when connecting to other processors, etc., which are not limited in the embodiments of this application.

[0038] The processor type can be any type, such as a sharding processor type, a merging processor type, a UUID-changing processor type, etc., and is not limited in the embodiments of this application.

[0039] For example, some processors depend on the value of a certain attribute configuration to determine whether they are sharded. For instance, the QueryDatabaseTable processor depends on the value of the "Max Rows PerFlow File" attribute to shard, and not to shard if the attribute is not present. This is not limited in the embodiments of this application.

[0040] In this step, the analysis system can obtain the first set of information corresponding to the analysis system based on the processor source code of the application being analyzed, and store it in the database. Then, through high abstraction, all information involved in the analysis can be configured, resulting in better scalability.

[0041] Furthermore, in the process of obtaining the first information set corresponding to the analysis system based on the processor source code of the application being analyzed, the analysis method may also include steps S1011 to S1013:

[0042] Step S1011: Based on the processor source code of the application being analyzed, obtain the data source event types generated by each processor in the analysis system.

[0043] In this step, during the process of obtaining the first set of information corresponding to the analysis system based on the processor source code of the application being analyzed, the analysis system can first obtain the data source event types generated by each processor in the analysis system based on the processor source code of the application being analyzed.

[0044] In this step, each processor in the analysis system can be any processor on the processor node link, such as the first processor, the last processor, or the third processor on the processor node link, etc., and is not limited in the embodiments of this application.

[0045] For example, each processor type can generate a data source event type corresponding to the processor type. For instance, in the case of a sharded processor type, the generated data source event type is the data source event corresponding to the sharded processor type.

[0046] In this step, during the analysis process, the analysis system can obtain the data source event type generated by any processor in the analysis system, thereby enabling support for more analysis scenarios and improving scalability.

[0047] Step S1012: Call the processor type analysis strategy corresponding to the data source event type to determine the type and configuration information of each processor in the analysis system.

[0048] In this step, after obtaining the data source event types generated by each processor in the analysis system based on the processor source code of the application being analyzed, the analysis system can call the processor type analysis strategy corresponding to the data source event type to determine the type and configuration information of each processor in the analysis system.

[0049] In this step, the processor type analysis strategy can be an analysis strategy for different processors. For example, it may include an analysis strategy for sliced ​​processors, or an analysis strategy for merged processors, or an analysis strategy for processors with changing UUIDs, etc., which are not limited in the embodiments of this application.

[0050] In this step, during the process of obtaining the first set of information, the data source event types generated by each processor in the analysis system can be obtained first, according to different processor analysis strategies.

[0051] In this step, during the process of obtaining the first set of information corresponding to the analysis system based on the processor source code of the application being analyzed, the analysis system can first obtain the data source event types generated by each processor in the analysis system based on the processor source code of the application being analyzed.

[0052] Step S1013: Determine the first information set corresponding to the analysis system based on the type of each processor, the configuration information of the processor, and the data source event type.

[0053] In this step, after calling the processor type analysis strategy corresponding to the data source event type to determine the type and configuration information of each processor in the analysis system, the analysis system can determine the first information set corresponding to the analysis system based on the type of each processor, the configuration information of the processor, and the data source event type.

[0054] Step S102: Extract the second information corresponding to the stream file to be analyzed from the first information set; the second information includes the type and configuration information of each processor used to process the stream file.

[0055] In this step, after obtaining the first information set corresponding to the analysis system based on the processor source code of the application being analyzed and storing it in the database, the analysis system can also extract the second information corresponding to the stream file to be analyzed from the first information set. Then, in the subsequent analysis and processing, the second information can be directly determined from the database and the analysis and processing can be performed based on the second information.

[0056] In this step, the stream file to be analyzed can be any stream file that needs to be analyzed and processed. For example, the stream file to be analyzed can be a stream file composed of temperature data or a stream file composed of humidity data, etc., and is not limited in the embodiments of this application.

[0057] In this step, each processor can be any processor on the processor node link. For example, the processors can be of the same type or different types, and this is not limited in the embodiments of this application.

[0058] In this step, the analysis system can extract the second information corresponding to the stream file to be analyzed from the first information set, thereby improving the efficiency of the analysis.

[0059] Step S103: Determine the policy mode corresponding to the stream file based on the second information; the policy mode is used to instruct the processor to process the stream file according to the corresponding policy mode.

[0060] The analysis method provided in this application, after extracting the second information corresponding to the stream file to be analyzed from the first information set, allows the analysis system to determine the policy mode corresponding to the stream file based on the second information; the policy mode is used to instruct the processor to process the stream file according to the corresponding policy mode.

[0061] In this step, the strategy mode corresponding to the processor may include, but is not limited to, the type of data source event (such as RECEIVE event type) generated when acting as a sharding processor, whether the processor is a database reader, the number of stream file attributes when reading the database, whether it is a processor that changes the stream file UUID, the data source type corresponding to changing the stream file UUID, and the data source event type received when connecting to other processors. These strategies are not limited in the embodiments of this application.

[0062] In this step, during the analysis process, the analysis system can first determine the policy mode corresponding to the stream file based on the second information, and then analyze and process the stream file based on the policy mode.

[0063] For example, if the first processor is a sharding processor and the third processor is also a sharding processor, the second information corresponding to the sharding processor can be directly determined from the database, and then the policy mode corresponding to the stream file can be determined based on the second information, and data processing can be performed according to the policy mode.

[0064] Furthermore, when determining the strategy mode corresponding to the stream file based on the first information, the analysis method may further include steps S1031 to S1034:

[0065] Step S1031: Based on the second information, determine the processor node link corresponding to the stream file to be analyzed.

[0066] In this step, when determining the policy mode corresponding to the stream file based on the second information, the analysis system can determine the processor node link corresponding to the stream file to be analyzed based on the second information.

[0067] In this step, the processor node link corresponding to the stream file to be analyzed may include multiple processor nodes and multiple connection queues that process the stream file to be analyzed. The ordered processor node link can be determined by the pointing relationship between the connection queues and the processors.

[0068] In this step, during the analysis process, the analysis system can first determine the processor node link corresponding to the stream file to be analyzed based on the second information, and then analyze and process the stream file that passes through any processor on the processor node link.

[0069] Step S1032: Sort the processors based on the processor node links.

[0070] In this step, after determining the processor node link corresponding to the stream file to be analyzed based on the second information, the analysis system can also sort the processors based on the processor node link.

[0071] In this step, the sorting process can be the process of sorting all processors on the processor link, which can ultimately form an ordered processor node link.

[0072] In this step, the sorting results on the processor node link can be obtained based on the processors arranged in order.

[0073] Step S1033: Determine the sub-policy mode corresponding to each processor in the processor node link according to the sorting result.

[0074] In this step, after sorting the processors based on the processor node link, the analysis system can determine the sub-policy mode corresponding to each processor in the processor node link according to the sorting result.

[0075] In this step, the policy mode corresponding to each processor in the processor node link is determined sequentially. It can be any policy mode, including but not limited to the type of data source event (such as RECEIVE event type) generated when acting as a sharding processor, whether the processor is a database read processor, the number of stream file attributes when reading the database, whether it is a processor that changes the stream file UUID, the data source type corresponding to changing the stream file UUID, and the data source event type received when connecting to other processors. In the embodiments of this application, no limitation is made.

[0076] In this step, the sub-policy mode corresponding to each processor is determined according to the processor sorting results, and then the analysis and processing are carried out according to the sub-policy mode.

[0077] Step S1034: Determine the policy mode corresponding to the stream file based on the sub-policy mode.

[0078] In this step, after determining the sub-policy patterns corresponding to each processor in the processor node link according to the sorting results, the analysis system can also determine the policy pattern corresponding to the stream file based on the sub-policy patterns.

[0079] Step S104: Based on the strategy pattern, parse and process the stream file, and analyze whether the header node corresponding to the stream file is a sharding processor.

[0080] The analysis method provided in this application, based on first information, determines the policy mode corresponding to the processor; the policy mode is used to instruct the processor to process the data of the stream file according to the corresponding policy mode. After that, the analysis system can parse the stream file based on the policy mode and analyze whether the header node corresponding to the stream file is a sharding processor.

[0081] In this step, the parsing process can be a process of processing the stream file, which may include, but is not limited to, determining whether the header node corresponding to the stream file is a fragment processor.

[0082] In this step, the head node can be the first processor on the processor node link.

[0083] In this step, the sharding processor can be a processor that enables the batch generation of stream files. That is, when the head node is a sharding processor, the stream files that pass through the sharding processor can generate N stream files. When it reaches the tail node processor, which is the last processor in the processor node link, the number of sharding processors it has passed through is M. Finally, the number of stream files reaching the tail node becomes N^M, which is N to the power of M stream files.

[0084] In this step, during the analysis process, the analysis system can parse and process the stream file based on the policy pattern to determine whether the header node corresponding to the stream file is a sharding processor.

[0085] Step S105: When the head node is a sharding processor, perform data association on the processing data obtained by at least one sharding processor through which the stream file passes, and obtain association information.

[0086] The analysis method provided in this application embodiment, after parsing and processing the stream file based on the strategy pattern and analyzing whether the header node corresponding to the stream file is a sharding processor, can also perform data association on the processing data obtained by at least one sharding processor through which the stream file passes, and obtain association information, if the header node is a sharding processor.

[0087] In this step, the processed data can be the data obtained after any shard processor processes the stream file. For example, the processed data can be multiple stream file data obtained after processing a single stream file through a shard processor.

[0088] In this step, at least one sharding processor can be a multi-layered sharding processor. After the stream file is processed by the sharding processor, there will be a process of 1 becoming many after each sharding.

[0089] This step involves the data association process, specifically, associating the attributes of the stream files after each sharding.

[0090] In this step, the associated information can be the information obtained after associating the attributes of the stream file.

[0091] In this step, when the head node is a sharding processor, the processed data obtained by at least one sharding processor through which the stream file passes is correlated to obtain correlation information. That is, during the analysis process, the analysis system can calculate the stream file at each sharding time by recursively going from the head node to the tail node. When calculating the results, the stream file execution results at each sharding level can be calculated by recursively going from the tail node to the head node. Thus, the correlation information can be used for analysis and processing.

[0092] Furthermore, when the head node is a sharding processor, and when performing data association on the processed data obtained by at least one sharding processor through which the stream file passes to obtain association information, the analysis method may further include steps S1051 to S1054:

[0093] Step S1051: When the head node is a sharding processor, determine the processor node link corresponding to the stream file to be analyzed.

[0094] In this step, when the head node is a sharding processor, the analysis system can first determine the processor node link corresponding to the stream file to be analyzed. The processor node link can include several processors and several connection queues, and the ordered processor node link can be determined by the pointing relationship between the connection queues and processors.

[0095] Step S1052: For each shard processor in the processor node link, obtain the exponential incremental stream file obtained after the shard processor processes the stream file.

[0096] In this step, when the head node is a sharding processor, after determining the processor node link corresponding to the stream file to be analyzed, the analysis system can also obtain the exponential incremental stream file obtained by the sharding processor after processing the stream file for each sharding processor in the processor node link.

[0097] In this step, each shard processor in the processor node link can be any shard processor in the processor node link.

[0098] In this step, after passing through the sharding processor, an exponentially incremental stream file can be obtained, meaning that each time it passes through the sharding processor, there will be a process of 1 becoming many.

[0099] In this step, during the data association process, the analysis system can obtain the exponentially incremental stream files obtained by each shard processor in the processor node link after the shard processor processes the stream file. For example, if the stream file that passes through the shard processor can generate N stream files, and the number of shard processors it passes through before reaching the tail node processor, which is the last processor in the processor node link, is M, then the number of stream files that finally reach the tail node becomes N^M, which is N to the power of M stream files.

[0100] Step S1053: Based on the exponential incremental stream file, determine the processing data corresponding to the sharding processor.

[0101] In this step, after obtaining the exponentially incremental stream file obtained by the shard processor after processing the stream file for each shard processor in the processor node link, the analysis system can also determine the processing data corresponding to the shard processor based on the exponentially incremental stream file.

[0102] In this step, the data processed by the sharding processor can be multiple stream files obtained by multiplying a single stream file by one.

[0103] In this step, after the stream file passes through the sharding processor, an exponentially incremental stream file can be obtained. The analysis system can then determine the processing data corresponding to the sharding processor based on the exponentially incremental stream file.

[0104] Step S1054: Perform data association on the processing data corresponding to each shard processor in the processor node link to obtain association information.

[0105] In this step, after determining the processing data corresponding to the shard processor based on the exponential incremental stream file, the analysis system can also perform data association on the processing data corresponding to each shard processor in the processor node link to obtain association information.

[0106] For example, the process of associating the processing data corresponding to each shard processor in the processor node link may include the process of associating stream file attributes.

[0107] In this step, the processing data corresponding to each shard processor in the processor node link is associated to obtain association information. Then, based on the association information, the stream file for each shard is calculated in a recursive manner from the head node to the tail node. When calculating the result, the stream file execution result of each shard level can be calculated by recursively using the tail node to the head node.

[0108] Step S106: Based on the association information, determine the change information corresponding to the stream file through a recursive algorithm, and perform a general analysis on the stream file based on the change information.

[0109] The analysis method provided in this application embodiment, when the head node is a sharding processor, performs data association on the processing data obtained by at least one sharding processor through which the stream file passes. After obtaining the association information, the analysis system can also determine the change information corresponding to the stream file based on the association information through a recursive algorithm, and perform general analysis on the stream file based on the change information.

[0110] In this step, a recursive algorithm is a programming technique where a function or method calls itself. It's typically used to solve problems that can be broken down into subproblems, a process that continues until a baseline condition (also known as a recursion termination condition) is reached. At this point, the algorithm stops recursively calling itself and returns the result. Recursive algorithms have two main characteristics: they call themselves, typically reducing the problem size with each call; and they must have a clear termination condition to prevent infinite recursion.

[0111] The change information corresponding to the stream file can be the change information obtained by the processor after executing the stream file.

[0112] It should be noted that general analysis can be applied to the processing of streaming files based on change information.

[0113] In this step, based on the association information, the process of determining the change information corresponding to the stream file through a recursive algorithm and performing a general analysis of the stream file based on the change information can include associating the stream file attributes that will change from 1 to multiple each time it is split. That is, the stream file at each split can be calculated recursively from the head node to the tail node. When calculating the result, the stream file at each higher split level can be calculated recursively from the tail node to the head node.

[0114] Furthermore, when determining the change information corresponding to the stream file based on the association information and through a recursive algorithm, and performing a general analysis of the stream file based on the change information, the analysis method may further include steps S1061 to S1064:

[0115] Step S1061: Based on the association information, determine the target input data corresponding to the tail node of the stream file through a recursive algorithm.

[0116] In this step, the target input data corresponding to the tail node of the stream file can be preset input data, that is, the baseline data of the target input.

[0117] In this step, during the analysis and processing of the stream file, the analysis system can determine the target input data corresponding to the tail node of the stream file based on the association information and through a recursive algorithm. That is, it first determines the preset input data, and then compares the actual input data with the preset input data to determine whether the tail node has finished processing.

[0118] Step S1062: Determine whether the input data received by the tail node is consistent with the target input data.

[0119] In this application, after the analysis system determines the target input data corresponding to the tail node of the stream file based on the association information and through a recursive algorithm, it can determine whether the input data received by the tail node is consistent with the target input data.

[0120] In this step, the input data received by the tail node can be the actual input data. This data may deviate from the preset input data, or it may be the same as the preset input data. This is not limited in the embodiments of this application.

[0121] In this step, the processing of the tail node is determined by whether the input data received by the tail node is consistent with the target input data.

[0122] Step S1063: If the input data received by the tail node is consistent with the target input data and a processing completion event corresponding to the input data is generated, determine that the tail node processing has ended, and obtain the change information generated by each processor in the processor node link corresponding to the stream file after processing the stream file.

[0123] In this step, after determining whether the input data received by the tail node is consistent with the target input data, the analysis system can determine that the tail node processing has ended if the input data received by the tail node is consistent with the target input data and a processing completion event corresponding to the input data is generated. The system then obtains the change information generated by each processor in the processor node link corresponding to the stream file after processing the stream file.

[0124] In this step, the processing completion event corresponding to the input data may include processing completion information and processing incomplete information, which are not limited in the embodiments of this application.

[0125] In this step, if the input data received by the tail node is consistent with the target input data and a processing completion event corresponding to the input data is generated, the analysis system can determine that the tail node processing has ended. If the input data received by the tail node is inconsistent with the target input data, or if the processing completion event corresponding to the input data is not completed, the system can determine that the tail node processing has not ended.

[0126] In this step, after determining that the tail node processing has ended, the analysis system can also obtain the change information generated by each processor in the processor node link corresponding to the stream file after processing the stream file.

[0127] Step S1064: Perform a general analysis of the stream file based on the change information.

[0128] In this step, if the input data received by the tail node is consistent with the target input data and a processing completion event corresponding to the input data is generated, the tail node processing is determined to be finished. After obtaining the change information generated by each processor in the processor node link corresponding to the stream file after processing the stream file, the analysis system can perform general analysis on the stream file based on the change information.

[0129] In this step, the process of the analysis system performing general analysis on the stream file based on the change information can include associating the stream file attributes that will generate multiple stream files after each sharding. That is, the stream file at each sharding can be calculated recursively from the head node to the tail node. When calculating the results, the stream file execution result at each sharding level can be calculated recursively from the tail node to the head node.

[0130] Furthermore, the analytical method may also include steps S1065 to S1069:

[0131] Step S1065: If the head node is not a fragmentation processor, determine the first processor on the processor node link that causes the change in the stream file attributes.

[0132] In this step, the attributes of the stream file will change when it passes through the first processor; that is, the content or attributes of the stream file will change when it passes through a certain type of processor.

[0133] In this step, if the head node is not a sharding processor, the analysis system can determine the first processor on the processor node link that causes the change in the stream file attributes. If the head node is not a sharding processor, the head node generates N stream files, and the number of stream files reaching the tail node is N^(M-1), which is N to the power of M-1 stream files.

[0134] Step S1066: Obtain the target processing result after each first processor processes the stream file in sequence.

[0135] In this step, if the head node is not a sharding processor, after determining the first processor on the processor node link that causes the change in the stream file attributes, the analysis system can obtain the target processing result obtained after each first processor processes the stream file in sequence.

[0136] In this step, the target processing result obtained by each first processor after processing the stream file in sequence can be the preset input data, that is, the baseline data of the target input, which includes the data of the stream file whose attributes have changed after passing through the first processor.

[0137] In this step, during the analysis and processing of the stream file, the analysis system can first obtain the target processing results obtained by each first processor processing the stream file in sequence, and then compare the actual input data with the preset input data to determine whether the tail node has finished processing.

[0138] Step S1067: Determine whether the input data received by the tail node is consistent with the target processing result.

[0139] In this step, after obtaining the target processing results obtained by each first processor processing the stream file in sequence, the analysis system can determine whether the input data received by the tail node is consistent with the target processing results.

[0140] In this step, the input data received by the tail node can be the actual input data. This data may deviate from the preset input data, or it may be the same as the preset input data. This is not limited in the embodiments of this application.

[0141] In this step, by determining whether the input data received by the tail node is consistent with the target processing result, the analysis system can further determine whether the tail node has finished processing based on this determination process.

[0142] Step S1068: If the input data received by the tail node is consistent with the target processing result and a processing completion event corresponding to the input data is generated, determine that the tail node processing has ended, and obtain the change information generated by each processor in the processor node link corresponding to the stream file after processing the stream file.

[0143] In this step, after determining whether the input data received by the tail node is consistent with the target processing result, the analysis system can determine that the tail node processing has ended if the input data received by the tail node is consistent with the target processing result and a processing completion event corresponding to the input data is generated. The system then obtains the change information generated by each processor in the processor node link corresponding to the stream file after processing the stream file.

[0144] In this step, after determining whether the input data received by the tail node is consistent with the target processing result, the analysis system can determine that the tail node processing has ended if the input data received by the tail node is consistent with the target processing result and a processing completion event corresponding to the input data is generated. The system then obtains the change information generated by each processor in the processor node link corresponding to the stream file after processing the stream file.

[0145] Step S1069: Perform a general analysis of the stream file based on the change information.

[0146] In this step, if the input data received by the tail node is consistent with the target processing result and a processing completion event corresponding to the input data is generated, the tail node processing is determined to be finished. After obtaining the change information generated by each processor in the processor node link corresponding to the stream file after processing the stream file, the analysis system can perform general analysis on the stream file based on the change information.

[0147] In this step, general analysis can be considered as the process of processing streaming files based on change information.

[0148] In summary, this application provides an analysis method that, based on the processor source code of the application being analyzed, obtains a first information set corresponding to the analysis system and stores it in a database. The first information set includes the type and configuration information of each processor in the analysis system. Second information corresponding to the stream file to be analyzed is extracted from the first information set. The second information includes the type and configuration information of each processor used to process the stream file. A policy mode corresponding to the stream file is determined based on the second information. The policy mode instructs the processor to process the stream file according to the corresponding policy mode. Based on the policy mode, the stream file is parsed, and it is analyzed whether the header node corresponding to the stream file is a sharding processor. If the header node is a sharding processor, the processed data obtained by at least one sharding processor through which the stream file passes is correlated to obtain... The system obtains correlation information; based on this correlation information, a recursive algorithm is used to determine the change information corresponding to the stream file, and a general analysis of the stream file is performed based on this change information. During the analysis process, a second information corresponding to the stream file to be analyzed can be extracted from a pre-stored first information set; then, the strategy mode corresponding to the stream file is determined based on the second information, and the stream file is processed. This enables analysis and processing in various scenarios. Furthermore, the analysis system can first determine whether the header node corresponding to the stream file is a sharding processor, and then, if the header node is a sharding processor, perform data correlation on the processed data obtained after the sharding processor to obtain correlation information. Finally, based on the correlation information, a recursive algorithm is used to determine the change information corresponding to the stream file, and a general analysis of the stream file is performed based on this change information, improving the efficiency of stream file analysis.

[0149] Reference Figure 2 This document illustrates a flowchart illustrating a NiFi workflow execution example provided in an embodiment of this application. Figure 2 As shown, the method specifically includes steps S201 to S212:

[0150] Step S201: NiFi workflow runs.

[0151] The NiFi workflow is now being parsed and processed.

[0152] Step S202: The head node processor generates a data source event.

[0153] After the NiFi workflow runs, the head node processor generates a data source event.

[0154] Step S203: Obtain the head node processor data source event.

[0155] After the head node processor generates a data source event, the analysis system can obtain the head node processor's data source event.

[0156] Step S204: Parse out the standard processor node link.

[0157] After acquiring the head node processor data source event, the analysis system can parse out the standard processor node link.

[0158] Step S205: Depend on the processor standard node link and parse the number of sharded processors in the link.

[0159] After resolving the standard processor node links, the analysis system can rely on these links to determine the number of fragmented processors within them.

[0160] Step S206: Determine the fragmentation level based on the fragmentation processor.

[0161] After resolving the number of sharded processors in the standard node link dependent on the processor, the analysis system can determine the sharding level based on the sharded processors.

[0162] Step S207: Determine whether the fragmentation level is greater than 0.

[0163] After determining the sharding level based on the sharding processor, the analysis system can determine whether the sharding level is greater than 0.

[0164] Step S208 is the scene segmentation.

[0165] If the fragmentation level is greater than 0, it is determined to be a fragmented scene.

[0166] Step S209: Not a segmented scene.

[0167] If the sharding level is less than or equal to 0, the analysis system can confirm that it is not a sharding scenario.

[0168] Step S210: Determine whether the head node is a sharding processor.

[0169] In a sharding scenario, the analysis system can continue to determine whether the head node is a sharding processor.

[0170] Step S211: Based on the fragmented stream file, count one instance of batch association.

[0171] When the head node is a sharding processor, the analysis system can count a large number of related instances based on the sharded stream files.

[0172] Step S212: Instance initialization complete.

[0173] After identifying a large number of associated instances based on the fragmented stream files, the analysis system can determine that instance initialization has been completed.

[0174] NiFi is a big data processing and distribution application framework based on a component-based architecture to achieve efficient big data processing. Its operation is based on workflows, each consisting of several processors. Each processor generates corresponding data source events during data processing. Currently, NiFi only persists the data source events generated during workflow execution; there is no universal method for analyzing NiFi data lineage source events. Stream files undergo four main changes during processor processing: First, they are generated in batches by a certain type of processor, such as QueryDatabaseTable, QueryDatabaseTableRecord, and SelectHive3QL. Second, they are split into multiple new stream files by a certain type of processor, such as SpliJson and SplitAvro. Third, multiple stream files are merged into a new stream file by a certain type of processor, such as the MergeRecord processor. Fourth, the content or attributes of a stream file change when it passes through a certain type of processor, such as JoltTransformRecord. The first two are called sharding processor types, the third is called merging processor type, and the fourth is called UUID change processor type. Therefore, as long as the configuration information of all these types of processors is not required, and indeed most processor types do not need to be considered, their processing logic for data source events can be determined. This can be directly configured in the database table. For processor types that require configuration information analysis, it is necessary to add a new processor type strategy to obtain some key analytical information and analyze the processing logic of data source events. This allows for general data source analysis, meaning that regardless of the number of processors and connection structure of a NiFi workflow, the change process of data source events can be analyzed. The core principle of general analysis is that for one or more stream files generated by the head node, the change process of the stream file during its journey to the tail node is determined. The UUID attribute of the stream file is unique. By confirming the sequence of UUID changes and the data source events generated during this process, general analysis can be achieved.

[0175] Current methods for analyzing NiFi data lineage events only support data source event analysis for processors in specific scenarios, and the supported analysis scenarios are relatively limited. In particular, when the processing combinations are more flexible and varied, the support is extremely poor or even non-existent, such as relatively simple processors like QueryDatabaseTable / QueryDatabaseTableRecord processors.

[0176] When workflows are composed of different processors and different combinations in different scenarios, they cannot support analysis and have poor scalability.

[0177] The main steps of the technical method implemented in this application are as follows:

[0178] The workflow of NiFi is typically parsed into several processors and several connection queues. Furthermore, the connection queues and processors can be linked together to form an ordered chain of processor nodes.

[0179] After parsing the NiFi workflow, we can obtain the type of each processor, the configuration details of each processor, and the configuration details of the connection queue.

[0180] By generating data source event types for each processor type and invoking its configured processor type analysis strategy, key information can be obtained, including: the type of the processor (e.g., QueryDatabaseTable), whether the processor is a sharded processor (some processors are sharded, while others depend on the value of a certain attribute configuration to determine whether they are sharded; for example, the QueryDatabaseTable processor depends on the 'Max Rows Per Flow File' attribute to be sharded if it has a value, otherwise it is not sharded), and what type of data source event it will generate when acting as a sharded processor (e.g., RECEIVE event type), whether the processor is a database reader, the number of records in the stream file when reading the database, whether it is a processor that changes the stream file UUID, the corresponding data source type when changing the stream file UUID, and the data source event type received when connecting to other processors. Through high abstraction, all information involved in the analysis is configured, resulting in better scalability.

[0181] Stream files always start from the head node processor, which is the first processor in the processor node chain. Assuming the head node is a sharding processor, the stream files generated by the sharding processors will produce N stream files. When the stream files reach the tail node processor, which is the last processor in the processor node chain, the number of sharding processors it has passed through is M, resulting in N^M (N to the power of M) stream files reaching the tail node. Similarly, if the head node is not a sharding processor, the number of stream files reaching the tail node is normally N^(M-1), which is N to the power of M-1. This means that each sharding operation involves a process of increasing the number of stream files, and this is associated with stream file attributes. Therefore, the number of stream files at each sharding level can be calculated recursively from the head node to the tail node. Conversely, the result can be calculated recursively from the tail node back to the head node to determine the result of each higher-level sharding operation.

[0182] Reference Figure 3 This illustrates a general NiFi data source event analysis flowchart provided in an embodiment of this application, such as... Figure 3 As shown, the method specifically includes steps S301 to S315:

[0183] Step S301: Analyze the initialized instance.

[0184] Step S302: Should it be fragmented?

[0185] During the analysis of the initialized instance, it can be determined first whether it is a fragmented scenario.

[0186] Step S303: Parse out all the shard processors on the node link and sort them in order.

[0187] In a sharding scenario, the analysis system can parse all the sharding processors on the node link and sort them in order.

[0188] Step S304: Parse out all processors on the processor node link that cause UUID changes, as well as the load balancing connection queue.

[0189] In non-sharded scenarios, the analysis system can identify all processors on the processor node link that cause UUID changes, as well as the load balancing connection queue.

[0190] Step S305: Is the head node a sharding processor?

[0191] After parsing out all the sharding processors on the node link and sorting them in order, the analysis system can determine whether the head node is a sharding processor.

[0192] Step S306: Based on the processing of the stream file by the splitter at each step, recursively calculate the stream file after the next split.

[0193] When the head node is a sharding processor, the analysis system can recursively calculate the stream file after the next sharding based on each processing of the stream file by the sharding processor.

[0194] Step S307: Each sorted N-th fragmented stream file corresponds to the previous N-1 received stream file.

[0195] After recursively calculating the next segmented stream file based on each processing of the stream file by the segmentation processor, the analysis system can determine that each sorted N segmented stream file corresponds to the previous N-1 received stream file.

[0196] Step S308: Recursively confirm the processing results of the segmented stream files from tail to head.

[0197] After determining that each of the N sorted stream fragments corresponds to the previous N-1 received stream fragment, the analysis system can recursively verify the processing results of the stream fragments from the end to the beginning.

[0198] Step S309: Check whether the fragmented stream files generated by all sorted N fragment processors are received by the tail node and issue a drag and drop (DROP) event for the stream files.

[0199] After recursively confirming the processing results of the fragmented stream files from tail to head, the analysis system can determine whether the fragmented stream files generated by all sorted N fragment processors have been received by the tail node and issue a DROP event for the stream files.

[0200] Step S310: Tail node processing ends.

[0201] After all the fragmented stream files generated by the N-ordered fragment processors are received by the tail node and a DROP event for the stream file is issued, the analysis system can determine that the tail node processing has ended.

[0202] Step S311, Analysis complete.

[0203] After the tail node processing is completed, the analysis system can determine that the analysis is finished.

[0204] Step S312: Parse out the head node processor, all processors that cause UUID changes before reaching the first shard processor, and the load balancing connection queue.

[0205] If the head node is not a sharding processor, the analysis system can parse out the head node processor, all processors that cause UUID changes before reaching the first sharding processor, and the load balancing connection queue.

[0206] Step S313: Determine the UUID of the original stream file of the head node that was received by the first sharding processor, which is the changed UUID.

[0207] After parsing the head node processor, all processors that cause UUID changes before reaching the first shard processor, and the load balancing connection queue, the analysis system can determine the UUID of the original stream file of the head node that was received by the first shard processor, which is the changed UUID.

[0208] Step S314: Calculate the UUID of the original stream file of the head node after each change.

[0209] After parsing out all processors on the processor node link that cause UUID changes, as well as the load balancing connection queue, the analysis system can calculate the UUID of the original stream file of the head node after each change.

[0210] Step S315: Check whether the changed UUID is received by the tail node and whether a DROP event is issued for the stream file.

[0211] After calculating the original UUID of the head node's stream file and then analyzing the UUID after each change, the system checks whether the changed UUID has been received by the tail node and issues a DROP event for the stream file.

[0212] In summary, by analyzing the data source event generation behavior of different NiFi processor types, a highly abstract and universal core method for data source analysis was designed. This method supports the operation and analysis of a wide variety of NiFi workflow scenarios. It has good scalability and strong universal adaptability. It can be quickly adapted by simply adding processor strategies without modifying the core analysis algorithm, making it very flexible. For the data source events associated with the analyzed instance information, disk space can be actively cleaned up and released.

[0213] The technical solution of this application also includes the following:

[0214] Obtain the data source events generated by the head node processor during each NiFi execution, and initialize them into each running instance of the workflow. The initialization process of the running instance is as follows: Figure 2 .

[0215] Reference Figure 4 This illustrates a flowchart of an embodiment of the present application for analyzing a running instance, such as... Figure 4 As shown, the method specifically includes steps S401 to S403:

[0216] Step S401: Obtain the data source event generated by the NiFi workflow head node processor.

[0217] Step S402: Initialize the running instance.

[0218] Step S403: Perform a general analysis of the running instance.

[0219] In summary, by highly abstracting processor types and configuring templates, strategy patterns, and database configurations, a core general-purpose data source event analysis algorithm can be implemented. This algorithm can analyze the rich and diverse workflow operations of NiFi. Through data source event analysis, information about each workflow operation can be analyzed and calculated, such as the start and end times of each operation, the amount of data processed, and the time consumed. Moreover, since each processor and connection queue generates a large number of data source events during workflow operation, which consume a significant amount of disk space, the analyzed data source events can be cleaned up in a timely manner.

[0220] Device Examples

[0221] like Figure 5 As shown, Figure 5 A logic block diagram of an analysis apparatus provided in an embodiment of this application is shown. The apparatus includes:

[0222] The acquisition module 1001 is used to acquire a first information set corresponding to the analysis system based on the processor source code of the application being analyzed, and store it in a database; the first information set includes the type of each processor in the analysis system and the configuration information of the processor.

[0223] Extraction module 1002 is used to extract second information corresponding to the stream file to be analyzed from the first information set; the second information includes type and configuration information corresponding to each processor for processing the stream file;

[0224] The first determining module 1003 is used to determine the policy mode corresponding to the stream file based on the second information; the policy mode is used to instruct the processor to process the stream file according to the corresponding policy mode;

[0225] Analysis module 1004 is used to parse the stream file based on the strategy pattern and analyze whether the header node corresponding to the stream file is a fragment processor.

[0226] The module 1005 is used to perform data association on the processed data obtained by at least one shard processor through which the stream file passes, when the head node is a shard processor, to obtain association information.

[0227] The first determining module is further configured to determine the change information corresponding to the stream file based on the association information through a recursive algorithm, and perform a general analysis on the stream file based on the change information.

[0228] Optionally, the device further includes:

[0229] The second determining module is used to determine the processor node link corresponding to the stream file to be analyzed based on the second information;

[0230] A sorting module is used to sort the processors based on the processor node links;

[0231] The second determining module is further configured to determine the sub-policy mode corresponding to each processor in the processor node link according to the sorting result; and determine the policy mode corresponding to the stream file based on the sub-policy mode.

[0232] Optionally, the acquisition module includes:

[0233] The acquisition submodule is used to acquire the data source event types generated by each processor in the analysis system based on the processor source code of the application being analyzed.

[0234] The determination submodule is used to invoke the processor type analysis strategy corresponding to the data source event type to determine the type and configuration information of each processor in the analysis system; and to determine the first information set corresponding to the analysis system based on the type, configuration information and data source event type of each processor.

[0235] Optionally, the obtaining module includes:

[0236] The determination submodule is used to determine the processor node link corresponding to the stream file to be analyzed when the head node is a fragment processor;

[0237] The acquisition submodule is used to acquire, for each shard processor in the processor node link, the exponential incremental stream file obtained by the shard processor after processing the stream file;

[0238] The determining submodule is further configured to determine the processing data corresponding to the sharding processor based on the exponential incremental stream file;

[0239] The acquisition submodule is used to perform data association on the processing data corresponding to each shard processor in the processor node link to obtain association information.

[0240] Optionally, the first determining module further includes:

[0241] The first determining submodule is used to determine the target input data corresponding to the tail node of the stream file based on the association information and through a recursive algorithm.

[0242] The judgment module is used to determine whether the input data received by the tail node is consistent with the target input data;

[0243] The first determining submodule is further configured to determine that the processing of the tail node has ended when the input data received by the tail node is consistent with the target input data and a processing completion event corresponding to the input data is generated.

[0244] The acquisition submodule is used to acquire change information generated by each processor in the processor node link corresponding to the stream file after processing the stream file;

[0245] The analysis submodule is used to perform general analysis on the stream file based on the change information.

[0246] Optionally, the device further includes:

[0247] The determination submodule is used to determine the first processor on the processor node link that causes the change in the stream file attributes when the head node is not a fragment processor;

[0248] The acquisition submodule is used to acquire the target processing result obtained by each of the first processors sequentially processing the stream file;

[0249] The judgment module is used to determine whether the input data received by the tail node is consistent with the target processing result;

[0250] The determination submodule is used to determine the end of the processing of the tail node when the input data received by the tail node is consistent with the target processing result and a processing completion event corresponding to the input data is generated.

[0251] The acquisition submodule is used to acquire change information generated by each processor in the processor node link corresponding to the stream file after processing the stream file;

[0252] The analysis submodule is used to perform general analysis on the stream file based on the change information.

[0253] As the device embodiment is basically similar to the method embodiment, the description is relatively simple, and relevant parts can be found in the description of the method embodiment.

[0254] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0255] The analysis device in this application embodiment can be an electronic device or a component within an electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal or other devices besides a terminal. For example, the electronic device can be a GPU BOX, mobile phone, tablet computer, laptop computer, PDA, in-vehicle electronic device, mobile internet device (MID), augmented reality (AR) / virtual reality (VR) device, robot, wearable device, ultra-mobile personal computer (UMPC), netbook, or personal digital assistant (PDA), etc. It can also be a server, network attached storage (NAS), personal computer (PC), television (TV), ATM, or self-service machine, etc. This application embodiment does not specifically limit the scope of the device.

[0256] The analysis device in this application embodiment can be a device with an operating system. This operating system can be Android, Linux, Windows, or other possible operating systems; this application embodiment does not specifically limit it.

[0257] The analytical device provided in this application embodiment can achieve... Figure 1 The various processes implemented in the method implementation examples will not be described again here to avoid repetition.

[0258] Optionally, such as Figure 6 As shown, this application embodiment also provides an electronic device 1100, including a processor 1101 and a memory 1102. The memory 1102 stores a program or instructions that can run on the processor 1101. When the program or instructions are executed by the processor 1101, they implement the various steps of the above-described analysis method embodiment and can achieve the same technical effect. To avoid repetition, they will not be described again here.

[0259] In embodiments of this application, the memory 1102 can be used to store software programs and various data. The memory 1102 may primarily include a first storage area for storing programs or instructions and a second storage area for storing data. The first storage area may store the operating system, applications or instructions required for at least one function (such as sound playback, image playback, etc.). Furthermore, the memory 1102 may include volatile memory or non-volatile memory, or both. The non-volatile memory may be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory can be random access memory (RAM), static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct memory bus RAM (DRRAM). The memory 1102 in this embodiment includes, but is not limited to, these and any other suitable types of memory.

[0260] Processor 1101 may include one or more processing units; optionally, processor 1101 integrates an application processor and a modem processor, wherein the application processor mainly handles operations involving the operating system, user interface, and applications, and the modem processor mainly handles wireless communication signals, such as a baseband processor. It is understood that the aforementioned modem processor may not be integrated into processor 1101.

[0261] This application also provides an electronic device, which includes a processor, a memory, a communication interface, and a communication bus. The processor, the memory, and the communication interface communicate with each other through the communication bus. The memory is used to store executable instructions, which cause the processor to execute the various processes of the analysis method embodiments described above and achieve the same technical effect. To avoid repetition, these will not be described again here.

[0262] This application also provides a readable storage medium storing a program or instructions. When the program or instructions are executed by a processor, they implement the various processes of the above-described analysis method embodiments and achieve the same technical effect. To avoid repetition, they will not be described again here.

[0263] The processor is the processor in the electronic device described in the above embodiments. The readable storage medium includes computer-readable storage media, such as computer read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.

[0264] This application embodiment also provides a chip, which includes a processor and a communication interface. The communication interface is coupled to the processor. The processor is used to run programs or instructions to implement the various processes of the above analysis method embodiments and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0265] It should be understood that the chip involved in the embodiments of this application may also be referred to as a system-on-a-chip, system chip, chip system, or system-on-a-chip, etc.

[0266] This application provides a computer program product, which is stored in a storage medium and executed by at least one processor to implement the various processes of the analysis method embodiments described above, and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0267] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, it should be noted that the scope of the methods and apparatuses in the embodiments of this application is not limited to performing functions in the order shown or discussed, but may also include performing functions substantially simultaneously or in the reverse order, depending on the functions involved. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.

[0268] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the related technology, can be embodied in the form of a computer software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0269] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

Claims

1. An analytical method, characterized in that, Applied to an analysis system, the method includes: Based on the processor source code of the application being analyzed, a first information set corresponding to the analysis system is obtained and stored in a database; the first information set includes the type of each processor in the analysis system and the configuration information of the processor. Extract the second information corresponding to the stream file to be analyzed from the first information set; the second information includes the type and configuration information of each processor corresponding to the stream file. The policy mode corresponding to the stream file is determined based on the second information; the policy mode is used to instruct the processor to process the stream file according to the corresponding policy mode; Based on the strategy pattern, the stream file is parsed and processed to analyze whether the header node corresponding to the stream file is a fragmentation processor; When the head node is a sharding processor, data association is performed on the processed data obtained by at least one sharding processor through which the stream file passes to obtain association information. Based on the association information, the change information corresponding to the stream file is determined by a recursive algorithm, and a general analysis is performed on the stream file based on the change information. The general analysis calculates the stream file running result for each higher segment level in a recursive manner from the tail node corresponding to the stream file to the head node corresponding to the stream file. Wherein, when the head node is a sharding processor, the process of associating the processed data obtained by at least one sharding processor through which the stream file passes to obtain association information includes: In the case that the head node is a fragment processor, determine the processor node link corresponding to the stream file to be analyzed; For each shard processor in the processor node link, obtain the exponentially incremental stream file obtained after the shard processor processes the stream file; Based on the exponential incremental stream file, determine the processing data corresponding to the sharding processor; Data association is performed on the processing data corresponding to each shard processor in the processor node link to obtain association information.

2. The method according to claim 1, characterized in that, Determining the policy mode corresponding to the stream file based on the second information includes: Based on the second information, the processor node link corresponding to the stream file to be analyzed is determined; Based on the processor node link, the processors are sorted. The sub-policy mode corresponding to each processor in the processor node link is determined sequentially according to the sorting results; Based on the sub-policy pattern, the policy pattern corresponding to the stream file is determined.

3. The method according to claim 1, characterized in that, The first information set corresponding to the analysis system, obtained based on the processor source code of the application being analyzed, includes: Based on the processor source code of the application being analyzed, the data source event types generated by each processor in the analysis system are obtained; The processor type analysis strategy corresponding to the data source event type is invoked to determine the type of each processor in the analysis system and the configuration information of the processor. Based on the type of each processor, the configuration information of the processor, and the type of the data source event, a first information set corresponding to the analysis system is determined.

4. The method according to claim 1, characterized in that, Based on the association information, a recursive algorithm is used to determine the change information corresponding to the stream file, and a general analysis of the stream file is performed based on the change information, including: Based on the aforementioned association information, the target input data corresponding to the tail node of the stream file is determined through a recursive algorithm. Determine whether the input data received by the tail node is consistent with the target input data; If the input data received by the tail node is consistent with the target input data and a processing completion event corresponding to the input data is generated, the processing of the tail node is determined to be completed, and the change information generated by each processor in the processor node link corresponding to the stream file after processing the stream file is obtained. A general analysis is performed on the stream file based on the change information.

5. The method according to claim 1, characterized in that, The method further includes: If the head node is not a fragmentation processor, determine the first processor on the processor node link that causes the change in the stream file attributes; Obtain the target processing result obtained by each of the first processors sequentially processing the stream file; Determine whether the input data received by the tail node is consistent with the target processing result; If the input data received by the tail node is consistent with the target processing result and a processing completion event corresponding to the input data is generated, the tail node processing is determined to be finished, and the change information generated by each processor in the processor node link corresponding to the stream file after processing the stream file is obtained. A general analysis is performed on the stream file based on the change information.

6. An analytical apparatus, characterized in that, The device includes: The acquisition module is used to acquire a first information set corresponding to the analysis system based on the processor source code of the application being analyzed, and store it in a database; the first information set includes the type of each processor in the analysis system and the configuration information of the processor. The extraction module is used to extract second information corresponding to the stream file to be analyzed from the first information set; the second information includes the type and configuration information of each processor corresponding to the stream file. The first determining module is used to determine the policy mode corresponding to the stream file based on the second information; the policy mode is used to instruct the processor to process the stream file according to the corresponding policy mode; The analysis module is used to parse the stream file based on the strategy pattern and analyze whether the header node corresponding to the stream file is a fragment processor. The acquisition module is used to perform data association on the processed data obtained by at least one shard processor through which the stream file passes, when the head node is a shard processor, and obtain association information. The first determining module is further configured to determine the change information corresponding to the stream file based on the association information through a recursive algorithm, and perform a general analysis on the stream file based on the change information. The general analysis calculates the stream file running result for each upper-level segment in a recursive manner from the tail node corresponding to the stream file to the head node corresponding to the stream file. The obtaining module includes: The determination submodule is used to determine the processor node link corresponding to the stream file to be analyzed when the head node is a fragment processor; The acquisition submodule is used to acquire, for each shard processor in the processor node link, the exponential incremental stream file obtained by the shard processor after processing the stream file; The determining submodule is further configured to determine the processing data corresponding to the sharding processor based on the exponential incremental stream file; The acquisition submodule is used to perform data association on the processing data corresponding to each shard processor in the processor node link to obtain association information.

7. The apparatus according to claim 6, characterized in that, The device further includes: The second determining module is used to determine the processor node link corresponding to the stream file to be analyzed based on the second information; A sorting module is used to sort the processors based on the processor node links; The second determining module is further configured to determine the sub-policy mode corresponding to each processor in the processor node link according to the sorting result; and determine the policy mode corresponding to the stream file based on the sub-policy mode.

8. An electronic device, characterized in that, The electronic device includes a processor, a memory, a communication interface, and a communication bus, wherein the processor, the memory, and the communication interface communicate with each other through the communication bus; The memory is used to store executable instructions that cause the processor to perform the analysis method as described in any one of claims 1 to 5.

9. A readable storage medium, characterized in that, The readable storage medium stores a program or instructions that, when executed by a processor, implement the analysis method according to any one of claims 1 to 5.

Images