Analysis method and device, electronic equipment and storage medium

Abstract

The invention discloses an analysis method and device, electronic equipment and a storage medium, and the method comprises the steps: obtaining a first information set corresponding to an analysis system based on a processor source code of an analyzed application, and storing the first information set in a database; extracting second information corresponding to the to-be-analyzed stream file from the first information set; determining a strategy mode corresponding to the stream file according to the second information; based on the strategy mode, analyzing and processing the stream file, and analyzing and processing whether a head node corresponding to the stream file is a fragment processor or not; under the condition that the head node is the fragment processor, data association is performed on processing data obtained by at least one fragment processor through which the stream file passes, and association information is obtained; and based on the associated information, determining change information corresponding to the stream file through a recursive algorithm, and performing general analysis on the stream file based on the change information. According to the embodiment of the invention, the analysis efficiency is improved while the analysis method in various scenes is provided.

Description

Technical Field

[0001] The present application belongs to the field of analysis technology, and specifically relates to an analysis method, device, electronic equipment and storage medium. Background Art

[0002] With the development of NiFi architecture, efficient processing of big data can be achieved. Its operation is based on workflows. Workflows are composed of several processors. Each processor generates corresponding data source events in the process of processing data. Currently, NiFi only persists the data source events generated during the operation of the workflow, and there is no universal method for analyzing NiFi data lineage source events.

[0003] Existing methods only support data source event analysis of processors in some specific scenarios, and the supported analysis scenarios are relatively limited. In particular, when the processing combinations are flexible and changeable, the support is extremely poor or even unavailable. When the scenarios are different and the workflows are composed of different processors with different combination structures, the analysis cannot be supported, and the scalability is also relatively poor. It is impossible to provide users with analysis methods in multiple scenarios, which reduces the efficiency of the analysis. Summary of the invention

[0004] The purpose of the embodiments of the present application is to provide an analysis method, device, electronic device and storage medium, which can solve the problem of low analysis efficiency in the process of analysis by related technologies and cannot provide users with analysis methods in multiple scenarios.

[0005] In a first aspect, an embodiment of the present application provides an analysis method, the method comprising:

[0006] Based on the processor source code of the analyzed application, a first information set corresponding to the analysis system is obtained and stored in a database; the first information set includes a type corresponding to each processor in the analysis system and configuration information corresponding to the processor;

[0007] Extracting second information corresponding to the stream file to be analyzed from the first information set; the second information includes type and configuration information corresponding to each processor corresponding to the stream file;

[0008] Determine the policy mode corresponding to the stream file according to the second information; the policy mode is used to instruct the processor to process data of the stream file according to the corresponding policy mode;

[0009] Based on the strategy mode, the stream file is parsed and processed to analyze whether the head node corresponding to the stream file is a shard processor;

[0010] In the case where the head node is a shard processor, data association is performed on the processed data obtained by at least one shard processor through which the stream file passes, to obtain association information;

[0011] Based on the association information, the change information corresponding to the stream file is determined through a recursive algorithm, and a general analysis is performed on the stream file based on the change information.

[0012] In a second aspect, an embodiment of the present application provides an analysis device, the device comprising:

[0013] An acquisition module, used to acquire a first information set corresponding to the analysis system based on the processor source code of the analyzed application, and store it in a database; the first information set includes a type corresponding to each processor in the analysis system and configuration information corresponding to the processor;

[0014] An extraction module, used to extract second information corresponding to the stream file to be analyzed from the first information set; the second information includes type and configuration information corresponding to each processor corresponding to the stream file;

[0015] A first determining module, configured to determine a policy mode corresponding to the stream file according to the second information; the policy mode is used to instruct the processor to perform data processing on the stream file according to the corresponding policy mode;

[0016] An analysis module, used for parsing the stream file based on the strategy mode, and analyzing whether the head node corresponding to the stream file is a shard processor;

[0017] an acquisition module, configured to, when the head node is a shard processor, perform data association on the processed data acquired by at least one shard processor through which the stream file passes, and obtain association information;

[0018] The first determination module is further configured to determine, based on the association information and through a recursive algorithm, the change information corresponding to the stream file, and perform a general analysis on the stream file based on the change information.

[0019] In a third aspect, an embodiment of the present application provides an electronic device, comprising a processor, a memory, a communication interface and a communication bus, wherein the processor, the memory and the communication interface communicate with each other through the communication bus; the memory is used to store executable instructions, and the executable instructions enable the processor to execute the analysis method described in any one of the above items.

[0020] In a fourth aspect, an embodiment of the present application provides a readable storage medium, on which a program or instruction is stored, and when the program or instruction is executed by a processor, an analysis method as described in any one of the above items is implemented.

[0021] In an embodiment of the present application, an analysis method is provided, which obtains a first information set corresponding to an analysis system based on a processor source code of an application to be analyzed and stores the information in a database; the first information set includes types corresponding to each processor in the analysis system and configuration information corresponding to the processor; second information corresponding to a stream file to be analyzed is extracted from the first information set; the second information includes types and configuration information corresponding to each processor for processing the stream file; a policy mode corresponding to the stream file is determined based on the second information; the policy mode is used to instruct the processor to perform data processing on the stream file according to the corresponding policy mode; based on the policy mode, the stream file is parsed and processed to analyze whether the head node corresponding to the processed stream file is a shard processor; in the case where the head node is a shard processor, data association is performed on the processed data obtained by at least one shard processor through which the stream file passes, and an association is obtained. Information; based on the associated information, a recursive algorithm is used to determine the change information corresponding to the stream file, and a general analysis is performed on the stream file based on the change information. During the analysis process, the second information corresponding to the stream file to be analyzed can be first extracted from the first information set according to the pre-stored first information set; then the policy mode corresponding to the stream file is determined according to the second information, and data processing is performed on the stream file, which can realize analysis and processing in various scenarios; further, the analysis system can first determine whether the head node corresponding to the stream file is a shard processor, and then, when the head node is a shard processor, data association is performed on the processed data obtained after the shard processor to obtain associated information; finally, based on the associated information, a recursive algorithm can be used to determine the change information corresponding to the stream file, and a general analysis is performed on the stream file based on the change information, thereby improving the efficiency of analyzing the stream file. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 is a flowchart of the steps of an analysis method provided in an embodiment of the present application;

[0023] Figure 2 It is a NiFi workflow running example analysis flow chart provided in an embodiment of the present application;

[0024] Figure 3 It is a general NiFi data source event analysis flow chart provided in an embodiment of the present application;

[0025] Figure 4 It is a flow chart for analyzing a running instance provided in an embodiment of the present application;

[0026] Figure 5 It is a logic block diagram of an analysis device provided in an embodiment of the present application;

[0027] Figure 6It is a structural schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0028] The following will be combined with the drawings in the embodiments of the present application to clearly describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments in the present application belong to the scope of protection of this application.

[0029] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described here, and the objects distinguished by "first", "second", etc. are generally of one type, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.

[0030] Method Embodiment

[0031] The analysis method provided in the embodiment of the present application is described in detail below through specific embodiments and their application scenarios in conjunction with the accompanying drawings.

[0032] Reference Figure 1 , shows a flow chart of the steps of an analysis method provided in an embodiment of the present application, such as Figure 1 As shown, the method specifically includes steps S101 to S106:

[0033] Step S101: Based on the processor source code of the analyzed application, a first information set corresponding to the analysis system is obtained and stored in a database; the first information set includes the type corresponding to each processor in the analysis system and the configuration information corresponding to the processor.

[0034] Exemplarily, in this step, the processor may include a processor that enables batch generation of stream files, such as: QueryDatabaseTable, QueryDatabaseTableRecord, SelectHive3QL, etc.; it also includes a processor that can split a stream file into multiple new stream files, such as: SpliJson, SplitAvro, etc.; it also includes a processor that can merge multiple stream files into a new stream file, such as: MergeRecord processor, etc.; it also includes a processor that can change the content or attributes of a stream file, such as: JoltTransformRecord, etc.

[0035] In this step, the analysis system can first obtain the first information set corresponding to the analysis system based on the processor source code of the analyzed application, and store it in a database. Then, in the subsequent analysis and processing process, the type corresponding to each processor in the first information set and the configuration information corresponding to the processor are directly retrieved from the database, and then analysis and processing are performed according to the type corresponding to each processor and the configuration information corresponding to the processor.

[0036] Exemplarily, the processor type may include, but is not limited to, a sharding processor type, a merged processor type, and a uuid change processor type, which are not limited in the embodiments of the present application.

[0037] In this step, the first information set may include but is not limited to the type of the processor (for example: QueryDatabaseTable), whether the processor is a sharded processor (some processors are sharded, and some processors rely on the value of a certain attribute configuration to determine whether they are sharded, for example: the QueryDatabaseTable processor is sharded when it depends on the "Max RowsPer Flow File" attribute and is not sharded when it does not have a value), and what type of data source event (such as RECEIVE event type) will be generated when it is a sharded processor, whether the processor is a processor that reads a database type, the number of flow file attributes when reading a database, whether it is a processor that will change the flow file uuid, and the corresponding data source type when changing the flow file uuid, the data source event type received when connecting to other processors, and other information are not limited in the embodiments of the present application.

[0038] Among them, the type corresponding to the processor can be any type, for example: sharding processor type, merged processor type, uuid change processor type, etc., which is not limited in the embodiments of the present application.

[0039] Exemplarily, some processors rely on the value of a certain attribute configuration to determine whether they are sharded. For example, the QueryDatabaseTable processor relies on the "Max Rows PerFlow File" attribute. If it has a value, it is sharded, but if it does not, it is not sharded. This is not limited in the embodiments of the present application.

[0040] In this step, the analysis system can obtain the first information set corresponding to the analysis system based on the processor source code of the analyzed application and store it in the database. Then, all the information involved in the analysis can be configured through a high degree of abstraction, which is more scalable.

[0041] Furthermore, in the process of acquiring the first information set corresponding to the analysis system based on the processor source code of the analyzed application, the analysis method may further include steps S1011 to S1013:

[0042] Step S1011: Based on the processor source code of the analyzed application, obtain the data source event type generated by each processor in the analysis system.

[0043] In this step, in the process of obtaining the first information set corresponding to the analysis system based on the processor source code of the analyzed application, the analysis system can first obtain the data source event type generated by each processor in the analysis system based on the processor source code of the analyzed application.

[0044] In this step, each processor in the analysis system can be any processor on the processor node link, and can be the first processor or the last processor, or the third processor, etc. on the processor node link, which is not limited in the embodiments of the present application.

[0045] Exemplarily, each processor type may generate a data source event type corresponding to the processor type. For example, when the processor type is a sharded processor type, the generated data source event type is a data source event corresponding to the sharded processor type.

[0046] In this step, during the analysis process, the analysis system can obtain the data source event type generated by any processor in the analysis system, thereby supporting more analysis scenarios and improving scalability.

[0047] Step S1012: Call the processor type analysis strategy corresponding to the data source event type to determine the type corresponding to each processor in the analysis system and the configuration information corresponding to the processor.

[0048] In this step, after obtaining the data source event type generated by each processor in the analysis system based on the processor source code of the application being analyzed, the analysis system can call the processor type analysis strategy corresponding to the data source event type to determine the type corresponding to each processor in the analysis system and the configuration information corresponding to the processor.

[0049] In this step, the processor type analysis strategy can be the analysis strategy of different processors, for example, it can include the analysis strategy corresponding to the sharding processor, or the analysis strategy corresponding to the merged processor, or the analysis strategy corresponding to the uuid change processor, etc., which is not limited in the embodiments of the present application.

[0050] In this step, in the process of acquiring the first information set, the data source event type generated by each processor in the analysis system can be first acquired according to different processor analysis strategies.

[0051] In this step, in the process of obtaining the first information set corresponding to the analysis system based on the processor source code of the analyzed application, the analysis system can first obtain the data source event type generated by each processor in the analysis system based on the processor source code of the analyzed application.

[0052] Step S1013: Determine a first information set corresponding to the analysis system according to the types corresponding to each processor, the configuration information corresponding to the processor, and the data source event type.

[0053] In this step, after calling the processor type analysis strategy corresponding to the data source event type to determine the types corresponding to each processor in the analysis system and the configuration information corresponding to the processor, the analysis system can determine the first information set corresponding to the analysis system based on the types corresponding to each processor, the configuration information corresponding to the processor and the data source event type.

[0054] Step S102: extract second information corresponding to the stream file to be analyzed from the first information set; the second information includes type and configuration information corresponding to each processor corresponding to the stream file.

[0055] In this step, after obtaining the first information set corresponding to the analysis system based on the processor source code of the application being analyzed and storing it in the database, the analysis system can also extract the second information corresponding to the stream file to be analyzed from the first information set, and then in the subsequent analysis and processing process, directly determine the second information from the database, and then perform analysis and processing based on the second information.

[0056] In this step, the stream file to be analyzed may be any stream file that needs to be analyzed and processed, for example, the stream file to be analyzed may be a stream file composed of temperature data, or a stream file composed of humidity data, etc., which is not limited in the embodiments of the present application.

[0057] In this step, each processor may be any processor on the processor node link. For example, the types of the processors may be the same or different, which is not limited in the embodiments of the present application.

[0058] In this step, the analysis system can extract the second information corresponding to the stream file to be analyzed from the first information set, thereby improving the efficiency of the analysis.

[0059] Step S103: determine the policy mode corresponding to the stream file according to the second information; the policy mode is used to instruct the processor to perform data processing on the stream file according to the corresponding policy mode.

[0060] According to the analysis method provided in the embodiment of the present application, after extracting the second information corresponding to the stream file to be analyzed from the first information set, the analysis system can determine the policy mode corresponding to the stream file based on the second information; the policy mode is used to instruct the processor to process data of the stream file according to the corresponding policy mode.

[0061] In this step, the strategy mode corresponding to the processor may include but is not limited to what type of data source event (such as RECEIVE event type) will be generated when acting as a shard processor, whether the processor is a processor that reads the database type, the number of stream file attributes when reading the database, whether it is a processor that will change the stream file uuid, and the corresponding data source type when changing the stream file uuid, the data source event type received when connecting to other processors, and other strategy modes, which are not limited in the embodiments of the present application.

[0062] In this step, during the analysis process, the analysis system may first determine the policy mode corresponding to the stream file according to the second information, and then analyze and process the stream file based on the policy mode.

[0063] Exemplarily, when the first processor is a shard processor and the third processor is also a shard processor, the second information corresponding to the shard processor can be directly determined from the database, and then the policy mode corresponding to the stream file can be determined based on the second information, and data processing can be performed according to the policy mode.

[0064] Furthermore, when determining the policy mode corresponding to the stream file based on the first information, the analysis method may further include steps S1031 to S1034:

[0065] Step S1031: Based on the second information, determine the processor node link corresponding to the stream file to be analyzed.

[0066] In this step, when determining the policy mode corresponding to the stream file according to the second information, the analysis system may determine the processor node link corresponding to the stream file to be analyzed based on the second information.

[0067] In this step, the processor node link corresponding to the stream file to be analyzed may include multiple processor nodes and multiple connection queues for processing the stream file to be analyzed, and the ordered processor node link may be determined through the pointing relationship between the connection queues and the processors.

[0068] In this step, during the analysis process, the analysis system can first determine the processor node link corresponding to the stream file to be analyzed based on the second information, and then analyze and process the stream file passing through any processor on the processor node link.

[0069] Step S1032: sort the processors based on the processor node links.

[0070] In this step, after determining the processor node link corresponding to the stream file to be analyzed based on the second information, the analysis system may also sort the processors based on the processor node link.

[0071] In this step, the sorting process may be a process of sorting all processors on the processor link, and eventually an ordered processor node link may be formed.

[0072] In this step, a sorting result on the processor node link can be obtained based on the processors arranged in sequence.

[0073] Step S1033: determine the sub-strategy mode corresponding to each processor in the processor node link in sequence according to the sorting result.

[0074] In this step, after the processors are sorted based on the processor node links, the analysis system may sequentially determine the sub-strategy mode corresponding to each processor in the processor node link according to the sorting result.

[0075] In this step, the strategy mode corresponding to each processor in the processor node link is determined in turn. It can be any strategy mode, which may include but is not limited to what type of data source event (such as RECEIVE event type) will be generated when it is a shard processor, whether the processor is a processor that reads the database type, the number of stream file attributes when reading the database, whether it is a processor that will change the stream file uuid, and the corresponding data source type when changing the stream file uuid, the data source event type received when connecting to other processors, etc. The strategy mode is not limited in the embodiments of the present application.

[0076] In this step, the sub-strategy mode corresponding to each processor is determined according to the ranking result of the processors, and then in the process of analysis, analysis and processing are performed according to the sub-strategy mode.

[0077] Step S1034: Determine the policy mode corresponding to the stream file based on the sub-policy mode.

[0078] In this step, after determining the sub-strategy mode corresponding to each processor in the processor node link in sequence according to the sorting result, the analysis system can also determine the strategy mode corresponding to the flow file based on the sub-strategy mode.

[0079] Step S104: Based on the strategy mode, the stream file is parsed and processed to analyze whether the head node corresponding to the stream file is a shard processor.

[0080] The analysis method provided in the embodiment of the present application determines the policy mode corresponding to the processor based on the first information; after the policy mode is used to instruct the processor to process the data of the stream file according to the corresponding policy mode, the analysis system can parse the stream file based on the policy mode to analyze whether the head node corresponding to the processed stream file is a shard processor.

[0081] In this step, the parsing process may be a process of processing the stream file, which may include but is not limited to determining whether the head node corresponding to the processed stream file is a shard processor.

[0082] In this step, the head node may be the first processor on the processor node link.

[0083] In this step, the shard processor can be a processor that enables batch generation of stream files, that is, when the head node is a shard processor, the stream files passing through the shard processor can generate N stream files. When it reaches the tail node processor, which is the last processor in the processor node link, the number of shard processors it passes through is M, and the number of stream files that finally reach the tail node becomes N^M, that is, N to the power of M stream files.

[0084] In this step, during the analysis process, the analysis system can parse the stream file based on the policy mode to determine whether the head node corresponding to the processed stream file is a shard processor.

[0085] Step S105: When the head node is a shard processor, data association is performed on the processed data obtained by at least one shard processor through which the stream file passes, to obtain association information.

[0086] The analysis method provided in the embodiment of the present application parses and processes the stream file based on the policy mode, and analyzes whether the head node corresponding to the processed stream file is a shard processor. The analysis system can also perform data association on the processed data obtained by at least one shard processor through which the stream file passes, when the head node is a shard processor, to obtain association information.

[0087] In this step, the processed data may be the processed data obtained after any shard processor processes the stream file. For example, the processed data may be multiple stream file data obtained after a stream file is processed by a shard processor.

[0088] In this step, at least one shard processor may be a multi-layer shard processor. After the stream file is processed by the shard processor, there will be a process of 1 becoming many each time it passes through sharding.

[0089] In this step, the data is processed for data association, that is, each time it passes through a shard, the stream file attributes are associated.

[0090] In this step, the associated information may be information obtained after the stream file attributes are associated.

[0091] In this step, when the head node is a shard processor, data association is performed on the processed data obtained by at least one shard processor through which the stream file passes, and associated information is obtained. That is, during the analysis process, the analysis system can recursively calculate the stream file for each sharding from the head node to the tail node. When calculating the result, the running result of the stream file at each sharding level can be calculated recursively from the tail node to the head node, and analysis and processing can be performed through associated information.

[0092] Furthermore, in the case where the head node is a shard processor, data association is performed on the processed data obtained by at least one shard processor through which the stream file passes, and when the association information is obtained, the analysis method may further include steps S1051 to S1054:

[0093] Step S1051: When the head node is a shard processor, determine the processor node link corresponding to the stream file to be analyzed.

[0094] In this step, when the head node is a shard processor, the analysis system can first determine the processor node link corresponding to the stream file to be analyzed. The processor node link may include several processors and several connection queues, and the ordered processor node link can be determined through the pointing relationship between the connection queues and the processors.

[0095] Step S1052: For each shard processor in the processor node link, obtain the exponential incremental stream file obtained after the shard processor processes the stream file.

[0096] In this step, when the head node is a shard processor, after determining the processor node link corresponding to the stream file to be analyzed, the analysis system can also obtain the exponential incremental stream file obtained after the shard processor processes the stream file for each shard processor in the processor node link.

[0097] In this step, each shard processor in the processor node link may be any shard processor in the processor node link.

[0098] In this step, after passing through the sharding processor, an exponential incremental stream file can be obtained, that is, each time it passes through the sharding processor, there will be a process of 1 becoming more.

[0099] In this step, during the data association process, the analysis system can obtain the exponential incremental flow file obtained after the shard processor processes the flow file for each shard processor in the processor node link. For example, the flow file passing through the shard processor can generate N flow files. When it reaches the tail node processor, which is the last processor in the processor node link, the number of shard processors it passes through is M, and the number of flow files that finally reach the tail node becomes N^M, that is, N to the power of M flow files.

[0100] Step S1053: Based on the index incremental stream file, determine the processing data corresponding to the shard processor.

[0101] In this step, after obtaining the exponential incremental stream file obtained after the shard processor processes the stream file for each shard processor in the processor node link, the analysis system can also determine the processing data corresponding to the shard processor based on the exponential incremental stream file.

[0102] In this step, the processing data corresponding to the shard processor may be multiple stream file data obtained after converting one stream file into multiple stream files.

[0103] In this step, after the stream file passes through the shard processor, an exponential incremental stream file can be obtained, and then the analysis system can determine the processing data corresponding to the shard processor based on the exponential incremental stream file.

[0104] Step S1054: perform data association on the processing data corresponding to each shard processor in the processor node link to obtain association information.

[0105] In this step, after determining the processing data corresponding to the shard processor based on the index incremental stream file, the analysis system can also perform data association on the processing data corresponding to each shard processor in the processor node link to obtain association information.

[0106] Exemplarily, the process of associating data with respect to the processing data corresponding to each shard processor in the processor node link may include the process of associating stream file attributes.

[0107] In this step, the processing data corresponding to each shard processor in the processor node link is associated with each other to obtain the associated information. Then, based on the associated information, the flow file for each shard can be calculated recursively from the head node to the tail node. When calculating the result, the flow file operation result of each shard level can be calculated recursively from the tail node to the head node.

[0108] Step S106: Based on the association information, the change information corresponding to the stream file is determined by a recursive algorithm, and a general analysis is performed on the stream file based on the change information.

[0109] The analysis method provided in the embodiment of the present application, when the head node is a shard processor, performs data association on the processed data obtained by at least one shard processor through which the stream file passes. After obtaining the association information, the analysis system can also determine the change information corresponding to the stream file based on the association information through a recursive algorithm, and perform general analysis on the stream file based on the change information.

[0110] In this step, a recursive algorithm is a programming technique in which a function or method calls itself, usually to solve problems that can be decomposed into similar subproblems. This decomposition process continues until a certain benchmark condition (also called the recursive termination condition) is reached, at which point the algorithm stops calling recursively and returns the result. Recursive algorithms have two main characteristics: the function calls itself, usually reducing the size of the problem with each call, and there must be a clear termination condition to prevent infinite recursion.

[0111] The change information corresponding to the stream file may be the change information obtained after the processor executes the stream file.

[0112] It should be noted that the general analysis may be a process of processing the stream file based on the change information.

[0113] In this step, based on the association information, a recursive algorithm is used to determine the change information corresponding to the stream file, and the process of performing a general analysis of the stream file based on the change information may include associating one or more stream file attributes generated each time through sharding, that is, the stream file for each sharding can be calculated recursively from the head node to the tail node, and when calculating the result, the running result of the stream file at each previous sharding level can be calculated recursively from the tail node to the head node.

[0114] Furthermore, when the change information corresponding to the stream file is determined by a recursive algorithm based on the association information and the stream file is generally analyzed based on the change information, the analysis method may further include steps S1061 to S1064:

[0115] Step S1061: Based on the association information, determine the target input data corresponding to the tail node corresponding to the stream file through a recursive algorithm.

[0116] In this step, the target input data corresponding to the tail node corresponding to the stream file may be preset input data, that is, reference data of the target input.

[0117] In this step, during the analysis and processing of the stream file, the analysis system can determine the target input data corresponding to the tail node corresponding to the stream file based on the associated information through a recursive algorithm, that is, first determine the preset input data, and then compare the actual input data with the preset input data to determine whether the tail node has been processed.

[0118] Step S1062: Determine whether the input data received by the tail node is consistent with the target input data.

[0119] In the present application, after the analysis system determines the target input data corresponding to the tail node corresponding to the stream file through a recursive algorithm based on the association information, it can determine whether the input data received by the tail node is consistent with the target input data.

[0120] In this step, the input data received by the tail node may be actual input data, which may deviate from the preset input data or may be the same as the preset input data, which is not limited in the embodiments of the present application.

[0121] In this step, it is determined whether the processing of the tail node is completed by judging whether the input data received by the tail node is consistent with the target input data.

[0122] Step S1063: When the input data received by the tail node is consistent with the target input data and a processing completion event corresponding to the input data is generated, it is determined that the tail node processing is completed, and the change information generated after each processor in the processor node link corresponding to the stream file processes the stream file is obtained.

[0123] In this step, after determining whether the input data received by the tail node is consistent with the target input data, the analysis system can determine that the tail node processing is completed if the input data received by the tail node is consistent with the target input data and a processing completion event corresponding to the input data is generated, and obtain the change information generated after each processor in the processor node link corresponding to the stream file processes the stream file.

[0124] In this step, the processing completion event corresponding to the input data may include processing completion information and processing incomplete information, which is not limited in the embodiments of the present application.

[0125] In this step, when the input data received by the tail node is consistent with the target input data and a processing completion event corresponding to the generated input data is generated, the analysis system can determine that the tail node processing is completed. When the input data received by the tail node is inconsistent with the target input data, or the processing completion event corresponding to the generated input data is incomplete processing, it can be determined that the tail node processing is not completed.

[0126] In this step, after determining that the tail node processing is completed, the analysis system may also obtain change information generated after each processor in the processor node link corresponding to the stream file processes the stream file.

[0127] Step S1064: Perform general analysis on the stream file based on the change information.

[0128] In this step, when the input data received by the tail node is consistent with the target input data and a processing completion event corresponding to the input data is generated, the tail node processing is determined to be completed. After obtaining the change information generated after each processor in the processor node link corresponding to the stream file processes the stream file, the analysis system can perform a general analysis of the stream file based on the change information.

[0129] In this step, the process of the analysis system performing a general analysis on the stream file based on the change information may include associating one or more stream file attributes generated each time after sharding, that is, the stream file for each sharding can be calculated recursively from the head node to the tail node. When calculating the result, the stream file operation result of each sharding level can be calculated recursively from the tail node to the head node.

[0130] Furthermore, the analysis method may further include steps S1065 to S1069:

[0131] Step S1065: When the head node is not a shard processor, determine the first processor on the processor node link that causes the stream file attribute to change.

[0132] In this step, when the stream file passes through the first processor, the attribute of the stream file will change, that is, when the stream file passes through a certain type of processor, the content or attribute of the stream file will change.

[0133] In this step, when the head node is not a shard processor, the analysis system can determine the first processor on the processor node link that causes the change in the stream file attributes. When the head node is not a shard processor, the head node generates N stream files, and the number of stream files reaching the tail node is N^(M-1), that is, N to the power of M-1 stream files.

[0134] Step S1066: Obtain target processing results obtained after each first processor processes the stream file in sequence.

[0135] In this step, when the head node is not a shard processor, after determining the first processor on the processor node link that causes the change in the stream file attributes, the analysis system can obtain the target processing results obtained after each first processor processes the stream file in sequence.

[0136] In this step, the target processing result obtained after each first processor processes the stream file in sequence may be preset input data, that is, benchmark data of the target input, which includes data on which the attributes of the stream file change after passing through the first processor.

[0137] In this step, during the analysis and processing of the stream file, the analysis system can first obtain the target processing results obtained after each first processor processes the stream file in sequence, and then compare the actual input data with the preset input data to determine whether the tail node has been processed.

[0138] Step S1067: Determine whether the input data received by the tail node is consistent with the target processing result.

[0139] In this step, after obtaining the target processing results obtained after each first processor processes the stream file in sequence, the analysis system can determine whether the input data received by the tail node is consistent with the target processing results.

[0140] In this step, the input data received by the tail node may be actual input data, which may deviate from the preset input data or may be the same as the preset input data, which is not limited in the embodiments of the present application.

[0141] In this step, by judging whether the input data received by the tail node is consistent with the target processing result, the analysis system can further determine whether the processing of the tail node is completed according to the judgment process.

[0142] Step S1068: When the input data received by the tail node is consistent with the target processing result and a processing completion event corresponding to the input data is generated, it is determined that the tail node processing is completed, and the change information generated after each processor in the processor node link corresponding to the stream file processes the stream file is obtained.

[0143] In this step, after determining whether the input data received by the tail node is consistent with the target processing result, the analysis system can determine that the tail node processing is completed if the input data received by the tail node is consistent with the target processing result and a processing completion event corresponding to the input data is generated, and obtain the change information generated after each processor in the processor node link corresponding to the stream file processes the stream file.

[0144] In this step, after determining whether the input data received by the tail node is consistent with the target processing result, the analysis system can determine that the tail node processing is completed if the input data received by the tail node is consistent with the target processing result and a processing completion event corresponding to the input data is generated, and obtain the change information generated after each processor in the processor node link corresponding to the stream file processes the stream file.

[0145] Step S1069: Perform general analysis on the stream file based on the change information.

[0146] In this step, when the input data received by the tail node is consistent with the target processing result and a processing completion event corresponding to the input data is generated, the tail node processing is determined to be completed. After obtaining the change information generated after each processor in the processor node link corresponding to the stream file processes the stream file, the analysis system can perform a general analysis of the stream file based on the change information.

[0147] In this step, the general analysis may be a process of processing the stream file based on the change information.

[0148] In summary, in an embodiment of the present application, an analysis method is provided, which obtains a first information set corresponding to the analysis system based on the processor source code of the analyzed application and stores it in a database; the first information set includes the types corresponding to each processor in the analysis system and the configuration information corresponding to the processor; the second information corresponding to the stream file to be analyzed is extracted from the first information set; the second information includes the types and configuration information corresponding to each processor corresponding to the processing stream file; the policy mode corresponding to the stream file is determined according to the second information; the policy mode is used to instruct the processor to process the data of the stream file according to the corresponding policy mode; based on the policy mode, the stream file is parsed and processed to analyze whether the head node corresponding to the processed stream file is a shard processor; in the case where the head node is a shard processor, data association is performed on the processed data obtained by at least one shard processor through which the stream file passes, to obtain Obtain associated information; based on the associated information, determine the change information corresponding to the stream file through a recursive algorithm, and perform a general analysis on the stream file based on the change information. During the analysis process, the second information corresponding to the stream file to be analyzed can be first extracted from the first information set according to the pre-stored first information set; then determine the policy mode corresponding to the stream file according to the second information, and perform data processing on the stream file, which can realize analysis and processing in various scenarios; further, the analysis system can first determine whether the head node corresponding to the stream file is a shard processor, and then, when the head node is a shard processor, perform data association on the processed data obtained after passing through the shard processor to obtain associated information; finally, based on the associated information, determine the change information corresponding to the stream file through a recursive algorithm, and perform a general analysis on the stream file based on the change information, thereby improving the efficiency of analyzing the stream file.

[0149] Reference Figure 2 , shows a NiFi workflow running example analysis flow chart provided by an embodiment of the present application, such as Figure 2 As shown, the method specifically includes steps S201 to S212:

[0150] Step S201, NiFi workflow runs.

[0151] Start parsing the NiFi workflow.

[0152] Step S202: The head node processor generates a data source event.

[0153] After a NiFi workflow runs, the head node processor generates data provenance events.

[0154] Step S203: Obtain the data source event of the head node processor.

[0155] After the head node processor generates a data source event, the analysis system may obtain the head node processor data source event.

[0156] Step S204: parse out the standard processor node link.

[0157] After obtaining the head node processor data source event, the analysis system can parse out the standard processor node link.

[0158] Step S205: Rely on the processor standard node link and parse the number of shard processors in the link.

[0159] After parsing out the standard processor node link, the analysis system can rely on the processor standard node link to parse the number of shard processors in the link.

[0160] Step S206: Determine the sharding level according to the sharding processor.

[0161] After relying on the processor standard node link and parsing the number of shard processors in the link, the analysis system can determine the shard level based on the shard processors.

[0162] Step S207: Determine whether the slice level is greater than 0.

[0163] After determining the shard level according to the shard processor, the analysis system can determine whether the shard level is greater than 0.

[0164] Step S208 is a fragmentation scenario.

[0165] When the fragmentation level is greater than 0, it is determined to be a fragmentation scenario.

[0166] Step S209: Not a fragmentation scenario.

[0167] When the sharding level is less than or equal to 0, the analysis system can confirm that it is not a sharding scenario.

[0168] Step S210: Determine whether the head node is a shard processor.

[0169] In the case of a sharding scenario, the analysis system may continue to determine whether the head node is a sharding processor.

[0170] Step S211: Count a batch of associated instances according to the shard stream file.

[0171] When the head node is a shard processor, the analysis system can count a large number of related instances based on the shard stream files.

[0172] Step S212: Instance initialization is completed.

[0173] After counting a large number of associated instances based on the shard stream file, the analysis system can determine that the instance initialization is complete.

[0174] NiFi is a big data processing and distribution application framework. It realizes efficient processing of big data based on componentized architecture. Its operation is based on workflows. Workflows are composed of several processors. Each processor generates corresponding data source events in the process of processing data. NiFi currently only persists the data source events generated during the operation of workflows. There is currently no universal method for analyzing NiFi data lineage source events. There are mainly four changes in the flow file during the processor process: the first is batch generation by a certain type of processor, such as: QueryDatabaseTable, QueryDatabaseTableRecord, SelectHive3QL, etc.; the second is that it is split into multiple new flow files when passing through a certain type of processor, such as: SpliJson, SplitAvro, etc.; the third is that multiple flow files are merged into a new flow file when passing through a certain type of processor, such as: MergeRecord processor, etc.; the fourth is that when the flow file passes through a certain type of processor, the content or attributes of the flow file change, such as: JoltTransformRecord, etc. The first two are called sharding processor types, the third is called merging processor type, and the fourth is called uuid change processor type. Therefore, as long as all these types of processors, or even most types of processors, do not need to pay attention to their configuration information, you can determine their processing logic for data source events. These can be directly configured in the database table. For processor types that need to analyze configuration information, it is necessary to add a new type of processor strategy to achieve the acquisition of some key analysis information. By analyzing the processing logic of data source events, you can do general data source analysis, that is, no matter what the number of processors and connection structure of a NiFi workflow are, you can analyze the change process of data source events. The core implementation principle of general analysis is to analyze the change process of one or more flow files generated by the head node when they arrive at the tail node for processing. The uuid attribute of the flow file is unique. By confirming the change process of uuid and the data source events generated in the process, general analysis can be achieved.

[0175] The current method of analyzing NiFi data lineage source events only supports data source event analysis of processors in some specific scenarios, and the supported analysis scenarios are relatively limited, especially when the processing combinations are flexible and changeable, the support is extremely poor or even unsupported, such as: QueryDatabaseTable / QueryDatabaseTableRecord processors and other relatively simple processors;

[0176] In different scenarios, when the workflow is composed of different processors in different combination structures, it cannot support analysis and has poor scalability.

[0177] The main process steps implemented by the technical method of this application are as follows:

[0178] The workflow of NiFi is usually parsed into several processors and several connection queues, and can be parsed into an ordered processor node link through the pointing relationship between the connection queue and the processor.

[0179] After parsing NiFi's workflow, we can determine the type of each processor, the configuration details of each processor, and the configuration details of the connection queue in turn.

[0180] By generating a data source event type for each processor type and calling its configured processor type analysis strategy, some key information can be obtained, including: the type of the processor (for example, QueryDatabaseTable), whether the processor is a sharded processor (some processors are sharded, and some processors rely on the value of a certain attribute configuration to determine whether they are sharded, for example, the QueryDatabaseTable processor relies on the 'Max Rows Per Flow File' attribute to be sharded when there is a value, and not sharded when there is no value), and what type of data source event will be generated when it is a sharded processor (for example, RECEIVE event type), whether the processor is a processor that reads databases, the number of flow file attributes when reading databases, whether it is a processor that changes the flow file uuid, and the corresponding data source type when changing the flow file uuid, the data source event type received when connecting to other processors, etc. Through high abstraction, all information involved in the analysis is configured, and the scalability is better;

[0181] The flow file always starts from the head node processor, that is, the first processor of the processor node link. Assuming that the head node is a shard processor, the flow file that passes through the shard processor can generate N flow files. When it reaches the tail node processor, that is, the last processor of the processor node link, the number of shard processors it passes through is M, and the number of flow files that finally reach the tail node becomes N^M, that is, N to the Mth power of flow files. Similarly, if the head node is not a shard processor, the number of flow files that normally reach the tail node is N^(M-1), that is, N to the M-1th power, that is, every time it passes through a shard, there will be a process of 1 becoming more, and it is associated with the flow file attributes, that is, the flow file for each shard can be calculated recursively from the head node to the tail node. When calculating the result, the flow file running result of each previous shard level can be calculated recursively from the tail node to the head node.

[0182] Reference Figure 3 , shows a general NiFi data source event analysis flow chart provided by an embodiment of the present application, such as Figure 3 As shown, the method specifically includes steps S301 to S315:

[0183] Step S301: Analyze the initialized instance.

[0184] Step S302: whether to fragment.

[0185] When analyzing the initialized instance, you can first determine whether it is a sharding scenario.

[0186] Step S303: parse out all the shard processors on the node link and sort them in order.

[0187] In the case of sharding scenarios, the analysis system can parse out all the sharding processors on the node link and sort them in order.

[0188] Step S304: parse out all processors on the processor node link that will cause the uuid to change, and the load balancing connection queue.

[0189] In the case of non-sharding scenarios, the analysis system can parse out all processors on the processor node link that will cause the UUID to change, as well as the load balancing connection queue.

[0190] Step S305: Whether the head node is a shard processor.

[0191] After parsing out all the shard processors on the node link and sorting them in order, the analysis system can determine whether the head node is a shard processor.

[0192] Step S306: recursively calculate the stream file after the next sharding according to the processing of the stream file by the sharding processor each time.

[0193] When the head node is a shard processor, the analysis system can recursively calculate the stream file after the next sharding based on each processing of the stream file by the shard processor.

[0194] Step S307: Each of the N-ordered fragment stream files corresponds to the previous N-1 received stream files.

[0195] After recursively calculating the stream file after the next sharding according to each sharding processor's processing of the stream file, the analysis system can determine that each sharded stream file of sorting N corresponds to the previous N-1 received stream files.

[0196] Step S308: recursively confirm the processing result of the segmented stream file from the end to the beginning.

[0197] After determining that each shard stream file of sort N corresponds to the previous N-1 receiving stream files, the analysis system can recursively confirm the processing results of the shard stream files from the end to the beginning.

[0198] Step S309: Check whether the shard stream files generated by all shard processors of sort N are received by the tail node and a drag and drop event (Drag and Drop, DROP) of the stream file is issued.

[0199] After recursively confirming the processing results of the shard stream files from the tail to the head, the analysis system can determine whether the shard stream files generated by all the shard processors of sort N are received by the tail node and issue a DROP event for the stream file.

[0200] Step S310: The tail node processing is completed.

[0201] After all shard stream files generated by the shard processors of sort N are received by the tail node and a DROP event of the stream file is issued, the analysis system can determine that the tail node processing is completed.

[0202] Step S311, analysis ends.

[0203] After the tail node processing is completed, the analysis system can determine that the analysis is completed.

[0204] Step S312: parse out the head node processor, all processors that will cause the uuid to change before reaching the first shard processor, and the load balancing connection queue.

[0205] In the case where the head node is not a shard processor, the analysis system can parse out the head node processor, all processors that will cause the uuid to change before reaching the first shard processor, and the load balancing connection queue.

[0206] Step S313: Determine the uuid of the original stream file of the head node received by the first shard processor, that is, the changed uuid.

[0207] After parsing the head node processor, all processors that will cause the UUID to change before reaching the first shard processor, and the load balancing connection queue, the analysis system can determine the UUID of the original flow file of the head node received by the first shard processor, which is the changed UUID.

[0208] Step S314: Calculate the uuid of the original stream file of the head node after each change.

[0209] After parsing out all processors on the processor node link that will cause the UUID to change, as well as the load balancing connection queue, the analysis system can calculate the UUID of the original flow file of the head node after each change.

[0210] Step S315: Check whether the changed uuid is received by the tail node and whether a DROP event of the stream file is issued.

[0211] After calculating the original stream file uuid of the head node, the system analyzes whether the changed uuid is received by the tail node and issues a DROP event for the stream file.

[0212] In summary, through the analysis of the data source event generation behavior of different NiFi processor types, a common data source analysis core method is designed with high abstraction; it supports the operation analysis of rich and diverse NiFi workflow scenarios; it has good scalability and strong general adaptability, and can be quickly adapted by simply adding processor strategies without changing the core analysis algorithm, which is very flexible; for the data source events associated with the instance information after the analysis is completed, it can be actively cleaned up to release disk space.

[0213] The technical solution of this application also includes the following:

[0214] Get the data source events generated by the head node processor each time NiFi is executed, and initialize them into each running instance of the workflow. The initialization process of the running instance is as follows Figure 2 .

[0215] Reference Figure 4 , shows a flow chart of analyzing a running instance provided by an embodiment of the present application, such as Figure 4 As shown, the method specifically includes steps S401 to S403:

[0216] Step S401: Obtain the data source event generated by the NiFi workflow head node processor.

[0217] Step S402: Initialize the running instance.

[0218] Step S403: perform general analysis of the running instance.

[0219] In summary, through the high abstraction of processor types, configuration templates, policy patterns and database configuration, the core general data source event analysis algorithm can be implemented, which can be used to analyze the rich and diverse workflow operations of NiFi; through data source event analysis, each operation information of the workflow can be analyzed and calculated, such as when each operation starts and ends, how much data is processed, and how much time is spent, all of which can be well calculated. Moreover, during the operation of the workflow, since each processor and connection queue will generate a large number of data source events, their volume is huge and they take up more disk space. Through the analysis of the completed data source events, they can also be cleaned up in time.

[0220] Device Embodiment

[0221] like Figure 5 As shown, Figure 5 A logic block diagram of an analysis device provided in an embodiment of the present application is shown, the device comprising:

[0222] The acquisition module 1001 is used to acquire a first information set corresponding to the analysis system based on the processor source code of the analyzed application, and store it in a database; the first information set includes the type corresponding to each processor in the analysis system and the configuration information corresponding to the processor;

[0223] An extraction module 1002 is used to extract second information corresponding to the stream file to be analyzed from the first information set; the second information includes type and configuration information corresponding to each processor corresponding to the stream file for processing;

[0224] A first determining module 1003 is used to determine a policy mode corresponding to the stream file according to the second information; the policy mode is used to instruct the processor to process data of the stream file according to the corresponding policy mode;

[0225] An analysis module 1004 is used to parse the stream file based on the strategy mode, and analyze whether the head node corresponding to the stream file is a shard processor;

[0226] The obtaining module 1005 is used for, when the head node is a shard processor, performing data association on the processed data obtained by at least one shard processor through which the stream file passes, to obtain association information;

[0227] The first determination module is further configured to determine, based on the association information and through a recursive algorithm, the change information corresponding to the stream file, and perform a general analysis on the stream file based on the change information.

[0228] Optionally, the device further comprises:

[0229] A second determination module, configured to determine a processor node link corresponding to the stream file to be analyzed based on the second information;

[0230] A sorting module, used for sorting the processors based on the processor node links;

[0231] The second determination module is further used to determine the sub-strategy mode corresponding to each processor in the processor node link in sequence according to the sorting result; and determine the strategy mode corresponding to the stream file based on the sub-strategy mode.

[0232] Optionally, the acquisition module includes:

[0233] An acquisition submodule, used to acquire data source event types generated by each processor in the analysis system based on the processor source code of the analyzed application;

[0234] A determination submodule is used to call a processor type analysis strategy corresponding to the data source event type, determine the types corresponding to each processor in the analysis system and the configuration information corresponding to the processor; and determine a first information set corresponding to the analysis system based on the types corresponding to each processor, the configuration information corresponding to the processor and the data source event type.

[0235] Optionally, the obtaining module includes:

[0236] A determination submodule, used for determining a processor node link corresponding to the stream file to be analyzed when the head node is a shard processor;

[0237] An acquisition submodule, used for acquiring, for each shard processor in the processor node link, an exponential incremental stream file obtained after the shard processor processes the stream file;

[0238] The determination submodule is further used to determine the processing data corresponding to the shard processor based on the index increment stream file;

[0239] The acquisition submodule is used to perform data association on the processing data corresponding to each shard processor in the processor node link to obtain association information.

[0240] Optionally, the first determining module further includes:

[0241] A first determination submodule, configured to determine, based on the association information, target input data corresponding to the tail node corresponding to the stream file through a recursive algorithm;

[0242] A judging module, used to judge whether the input data received by the tail node is consistent with the target input data;

[0243] The first determining submodule is further configured to determine that the processing of the tail node is completed when the input data received by the tail node is consistent with the target input data and a processing completion event corresponding to the input data is generated;

[0244] An acquisition submodule, used to acquire change information generated after each processor in the processor node link corresponding to the stream file processes the stream file;

[0245] The analysis submodule is used to perform general analysis on the stream file based on the change information.

[0246] Optionally, the device further comprises:

[0247] A determination submodule, used for determining a first processor on a processor node link that causes a change in the stream file attribute when the head node is not a shard processor;

[0248] An acquisition submodule, used for acquiring target processing results obtained after each of the first processors processes the stream files in sequence;

[0249] A judgment module, used to judge whether the input data received by the tail node is consistent with the target processing result;

[0250] a determination submodule, configured to determine that the processing of the tail node is completed when the input data received by the tail node is consistent with the target processing result and a processing completion event corresponding to the input data is generated;

[0251] An acquisition submodule, used to acquire change information generated after each processor in the processor node link corresponding to the stream file processes the stream file;

[0252] The analysis submodule is used to perform general analysis on the stream file based on the change information.

[0253] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0254] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0255] The analysis device in the embodiment of the present application can be an electronic device, or a component in an electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal, or other devices other than a terminal. Exemplary, the electronic device can be a GPU BOX, a mobile phone, a tablet computer, a laptop computer, a PDA, a vehicle-mounted electronic device, a mobile Internet device (Mobile Internet Device, MID), an augmented reality (augmentedreality, AR) / virtual reality (virtualreality, VR) device, a robot, a wearable device, an ultra-mobile personal computer (ultra-mobile personalcomputer, UMPC), a netbook or a personal digital assistant (personal digital assistant, PDA), etc., and can also be a server, a network attached storage (Network Attached Storage, NAS), a personal computer (personalcomputer, PC), a television (television, TV), a teller machine or a self-service machine, etc., and the embodiment of the present application is not specifically limited.

[0256] The analysis device in the embodiment of the present application may be a device having an operating system. The operating system may be an Android operating system, a Linux operating system, a Windows operating system, or other possible operating systems, which are not specifically limited in the embodiment of the present application.

[0257] The analysis device provided in the embodiment of the present application can achieve Figure 1 To avoid repetition, the various processes implemented by the method embodiment are not described here.

[0258] Alternatively, if Figure 6 As shown, an embodiment of the present application also provides an electronic device 1100, including a processor 1101 and a memory 1102, wherein the memory 1102 stores programs or instructions that can be executed on the processor 1101, and when the program or instructions are executed by the processor 1101, the various steps of the above-mentioned analysis method embodiment are implemented, and the same technical effect can be achieved. To avoid repetition, they are not described here.

[0259] In an embodiment of the present application, the memory 1102 may be used to store software programs and various data. The memory 1102 may mainly include a first storage area for storing programs or instructions and a second storage area for storing data, wherein the first storage area may store an operating system, an application program or instructions required for at least one function (such as a sound playback function, an image playback function, etc.), etc. In addition, the memory 1102 may include a volatile memory or a non-volatile memory, or the memory 1102 may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), a static random access memory (SRAM), a dynamic random access memory (DRAM), a synchronous dynamic random access memory (SDRAM), a double data rate synchronous dynamic random access memory (DDRSDRAM), an enhanced synchronous dynamic random access memory (ESDRAM), a synchronous link dynamic random access memory (SLDRAM) and a direct memory bus random access memory (DRRAM). The memory 1102 in the embodiment of the present application includes but is not limited to these and any other suitable types of memory.

[0260] The processor 1101 may include one or more processing units; optionally, the processor 1101 integrates an application processor and a modem processor, wherein the application processor mainly processes operations related to an operating system, a user interface, and application programs, and the modem processor mainly processes wireless communication signals, such as a baseband processor. It is understandable that the modem processor may not be integrated into the processor 1101.

[0261] An embodiment of the present application also provides an electronic device, which includes a processor, a memory, a communication interface and a communication bus, wherein the processor, the memory and the communication interface communicate with each other via the communication bus; the memory is used to store executable instructions, and the executable instructions enable the processor to execute the various processes of the analysis method embodiment as described above, and can achieve the same technical effect. To avoid repetition, they will not be repeated here.

[0262] An embodiment of the present application also provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, the various processes of the above-mentioned analysis method embodiment are implemented and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0263] The processor is the processor in the electronic device described in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory ROM, a random access memory RAM, a magnetic disk or an optical disk.

[0264] An embodiment of the present application further provides a chip, which includes a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the various processes of the above-mentioned analysis method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0265] It should be understood that the chip involved in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.

[0266] An embodiment of the present application provides a computer program product, which is stored in a storage medium. The program product is executed by at least one processor to implement the various processes of the above-mentioned analysis method embodiment and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0267] It should be noted that, in this article, the terms "comprise", "include" or any other variant thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises one..." does not exclude the presence of other identical elements in the process, method, article or device including the element. In addition, it should be pointed out that the scope of the method and device in the embodiment of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in reverse order according to the functions involved, for example, the described method may be performed in an order different from that described, and various steps may also be added, omitted or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0268] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the relevant technology, can be embodied in the form of a computer software product, which is stored in a storage medium (such as ROM / RAM, a disk, or an optical disk), and includes a number of instructions for a terminal (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0269] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

Claims

1. An analysis method, characterized in that Applied to an analysis system, the method comprises: Based on the processor source code of the analyzed application, a first information set corresponding to the analysis system is obtained and stored in a database; the first information set includes a type corresponding to each processor in the analysis system and configuration information corresponding to the processor; Extracting second information corresponding to the stream file to be analyzed from the first information set; the second information includes type and configuration information corresponding to each processor corresponding to the stream file; Determine the policy mode corresponding to the stream file according to the second information; the policy mode is used to instruct the processor to process data of the stream file according to the corresponding policy mode; Based on the strategy mode, the stream file is parsed and processed to analyze whether the head node corresponding to the stream file is a shard processor; In the case where the head node is a shard processor, data association is performed on the processed data obtained by at least one shard processor through which the stream file passes, to obtain association information; Based on the association information, the change information corresponding to the stream file is determined through a recursive algorithm, and a general analysis is performed on the stream file based on the change information.

2. The method according to claim 1, characterized in that The determining, according to the second information, a policy mode corresponding to the stream file includes: Based on the second information, determining a processor node link corresponding to the stream file to be analyzed; Based on the processor node links, the processors are sorted; Determine the sub-strategy mode corresponding to each processor in the processor node link in sequence according to the sorting result; Based on the sub-strategy mode, a policy mode corresponding to the stream file is determined.

3. The method according to claim 1, characterized in that The acquiring, based on the processor source code of the analyzed application, a first information set corresponding to the analysis system includes: Based on the processor source code of the analyzed application, obtaining the data source event type generated by each processor in the analysis system; Calling a processor type analysis strategy corresponding to the data source event type to determine the type corresponding to each processor in the analysis system and configuration information corresponding to the processor; A first information set corresponding to the analysis system is determined according to the types corresponding to the respective processors, the configuration information corresponding to the processors, and the data source event type.

4. The method according to claim 1, characterized in that: In the case where the head node is a shard processor, data association is performed on the processed data obtained by at least one shard processor through which the stream file passes to obtain association information, including: In the case where the head node is a shard processor, determining a processor node link corresponding to the stream file to be analyzed; For each shard processor in the processor node link, obtaining an exponential incremental stream file obtained after the shard processor processes the stream file; Based on the index incremental stream file, determining the processing data corresponding to the shard processor; Data association is performed on the processing data corresponding to each shard processor in the processor node link to obtain association information.

5. The method according to claim 1, characterized in that The determining, based on the association information, change information corresponding to the stream file by a recursive algorithm, and performing general analysis on the stream file based on the change information includes: Based on the association information, determine the target input data corresponding to the tail node corresponding to the stream file through a recursive algorithm; Determining whether the input data received by the tail node is consistent with the target input data; When the input data received by the tail node is consistent with the target input data and a processing completion event corresponding to the input data is generated, it is determined that the processing of the tail node is completed, and change information generated after each processor in the processor node link corresponding to the stream file processes the stream file is obtained; A general analysis is performed on the stream file based on the change information.

6. The method according to claim 1, characterized in that The method further comprises: In the case where the head node is not a shard processor, determining a first processor on a processor node link that causes a change in the stream file attribute; Obtaining target processing results obtained after each of the first processors processes the stream file in sequence; Determine whether the input data received by the tail node is consistent with the target processing result; When the input data received by the tail node is consistent with the target processing result and a processing completion event corresponding to the input data is generated, it is determined that the processing of the tail node is completed, and change information generated after each processor in the processor node link corresponding to the stream file processes the stream file is obtained; A general analysis is performed on the stream file based on the change information.

7. An analysis device, characterized in that: The device comprises: An acquisition module, used to acquire a first information set corresponding to the analysis system based on the processor source code of the analyzed application, and store it in a database; the first information set includes a type corresponding to each processor in the analysis system and configuration information corresponding to the processor; An extraction module, used to extract second information corresponding to the stream file to be analyzed from the first information set; the second information includes type and configuration information corresponding to each processor corresponding to the stream file; A first determining module, configured to determine a policy mode corresponding to the stream file according to the second information; the policy mode is used to instruct the processor to perform data processing on the stream file according to the corresponding policy mode; An analysis module, used for parsing the stream file based on the strategy mode, and analyzing whether the head node corresponding to the stream file is a shard processor; an acquisition module, configured to, when the head node is a shard processor, perform data association on the processed data acquired by at least one shard processor through which the stream file passes, and obtain association information; The first determination module is further configured to determine, based on the association information and through a recursive algorithm, the change information corresponding to the stream file, and perform a general analysis on the stream file based on the change information.

8. An analysis device, characterized in that: The device also includes: A second determination module, configured to determine a processor node link corresponding to the stream file to be analyzed based on the second information; A sorting module, used for sorting the processors based on the processor node links; The second determination module is further used to determine the sub-strategy mode corresponding to each processor in the processor node link in sequence according to the sorting result; and determine the strategy mode corresponding to the stream file based on the sub-strategy mode.

9. An electronic device, characterized in that: The electronic device comprises a processor, a memory, a communication interface and a communication bus, and the processor, the memory and the communication interface communicate with each other via the communication bus; The memory is used to store executable instructions, and the executable instructions enable the processor to execute the analysis method according to any one of claims 1 to 6.

10. A readable storage medium, characterized in that: The readable storage medium stores a program or instruction, and when the program or instruction is executed by a processor, the analysis method according to any one of claims 1 to 6 is implemented.

Images