Log-based data analysis method and device

Abstract

The invention relates to a log-based data analysis method and device, and belongs to the technical field of data management.The method comprises the steps that a log system is constructed, the log system is determined according to data structure layer information and statistical screening conditions, the data structure layer information is used for representing a storage form and a storage structure of log information, and the statistical screening conditions are used for statistical screening; the statistical screening conditions are determined according to data analysis requirements; multiple sets of initial data submitted by multiple different data submission nodes are obtained, and the multiple sets of initial data comprise data submission node storage time; writing the log information corresponding to the multiple groups of initial data and the data submission node storage time into a log system to obtain a classified storage result, and obtaining an initial analysis result according to the classified storage result; and performing statistical analysis on the initial analysis result to obtain a target analysis result. According to the method, real-time data analysis is achieved in a read-write separation mode, and the access pressure of data analysis on a database is synchronously relieved on the basis of rapid data analysis.

Description

Technical Field

[0001] The present invention relates to the technical field of public credit data management, and in particular to a log-based data analysis method and device. Background Art

[0002] At present, in the process of collecting (or managing, providing services for) credit data, the competent departments of public credit investigation business need to conduct statistical analysis of the public credit data submitted by the delisted units (or departments at the same level) according to the characteristic indicators such as the place of ownership in the process of collecting (or managing, providing services for) credit data, based on the needs of work management for public credit data governance. Up to now, the storage of public credit information is generally based on the management stages such as "collection, governance, management, and services", using multiple sets of relational database physical storage devices for storage, forming public credit investigation data with certain "stage governance characteristics", and the statistical analysis based on public credit data is the statistical analysis of various types of stage business data stored in various types of relational database physical storage devices.

[0003] like Figure 1 As shown in the figure, based on the current needs of local credit authorities for "staged storage" of credit data, the public credit investigation data storage is generally divided into multiple areas with typical "staged" characteristics, such as the front-end shared database area and the classified governance database area. The front-end shared database equipment area generally uses multiple sets of database equipment to store the original data of public credit information reported by various departments (including lower-level public credit authorities) (n≥1, n is the number of nodes), and the classified governance database governs all types of data collected from the front-end shared database in accordance with the requirements of public credit business management, and stores the governance results (including governance intermediate data) in the classified governance database. Figure 2As shown in the figure, in the data governance activities for public credit information, the corresponding data source and the physical table of data storage participating in classification statistics are searched based on the established metadata information (the data source refers to the relational database instance, and the number of instances is generally >1), and the relevant metadata information is loaded into the array queue variable for storage; the metadata information is obtained from the loop array queue variable, and the sum operation is performed according to the parameters in the physical table result specified by the metadata, and the sum operation is stored in the specified operation result storage table; the data in the operation result storage table of each node is stored in a unified comprehensive data statistics storage device by using the data exchange tool (or DBLink technology); the corresponding metadata relationship model is established to provide support services for data search. In the above process domain processing, since all credit reporting authorities use relational database management systems to store public credit data. Therefore, the corresponding data processing is directly implemented using the stored procedures, scheduled tasks and other mechanisms of the relational database management system. Among them, the stored procedures are mainly used to solve the contents of the search data domain, classification statistical analysis, and result storage process domain in the technical process domain, while the scheduled tasks are mainly used to implement scheduled execution. Since this processing method directly performs a large number of read and write operations on the core links of the storage business, it increases the pressure on the corresponding relational database management system. In specific implementation, it is generally adopted to execute stored procedures in multiple databases separately (or use the database's DBLINK technology) to realize multi-database cluster joint query.

[0004] In the process of using conventional comprehensive data statistical analysis methods to conduct statistical analysis on specific areas of public credit information, it has been verified that there are the following risks: (1) There are certain differences in the statistical results of comprehensive data. This is because the storage time of the same public credit information in the shared library and the data governance library is different (for example, data A is written into the shared library by the business unit at 23:00 on July 14, and the credit authority synchronizes it to the waiting library in the data governance library at 2:00 on July 15, and the result library is stored in the data governance library at 3:00 on July 16). Based on this phenomenon, it is difficult to ensure that the analysis and calculation are carried out at the same time during the comprehensive data analysis process, resulting in different calculation results for the same type of data in each library. (2) Increase the pressure on the database server and affect the normal operation of the credit information main platform. Based on the comprehensive data statistical analysis process, a large number of read and write operations are required for all business data, which leads to high pressure on the database. In severe cases, database deadlock may occur. Summary of the invention

[0005] In view of this, it is necessary to provide a log-based data analysis method and device to solve the technical problems in the prior art that it is difficult to ensure that multiple databases participate in analysis and calculation at the same time during the storage and joint query of multiple database clusters, and a large number of read and write operations on business data cause high database pressure.

[0006] In order to solve the above problems, the present invention provides a log-based data analysis method, comprising: Constructing a log system, wherein the log system is determined according to data structure layer information and statistical screening conditions, wherein the data structure layer information is used to characterize the storage form and storage structure of log information, and the statistical screening conditions are determined according to data analysis requirements; Acquire multiple groups of initial data respectively reported by multiple different data reporting nodes, wherein the multiple groups of initial data all include storage time of the data reporting nodes; Writing the log information corresponding to the multiple groups of initial data and the storage time of the data reporting node into the log system to obtain a classified storage result, and obtaining an initial analysis result according to the classified storage result; Perform statistical analysis on the initial analysis results to obtain target analysis results.

[0007] In a possible implementation, building the log system includes: Determine the hierarchical classification information of log information based on the data structure layer information and statistical screening conditions; Determining a storage path for the log information according to the hierarchical classification information; A multi-level folder group is constructed according to the storage path of the log information.

[0008] In a possible implementation, determining the hierarchical classification category information of the log information according to the data structure layer information and the statistical screening condition includes: According to the data structure layer information, determine the routing node information, data instance, table and mode information of the log information classification; Determine the dimensions for grading and classifying log information based on statistical screening conditions.

[0009] In a possible implementation, determining the storage path of the log information according to the hierarchical classification category information includes: Constructing a storage structure, wherein the storage structure includes a plurality of routing nodes; Determine a primary routing node according to the routing node information; Determining a secondary routing node according to the data instance, the table, and the mode information; Determine the filtering condition storage channel node according to the dimension of the log information classification; According to the hierarchical classification category information, the leaf nodes are determined.

[0010] In a possible implementation, determining the screening condition storage channel node according to the dimension of the hierarchical classification of the log information includes: Determine the third-level routing nodes based on the statistical rule dimensions of log classification; According to the time dimension of log classification, the fourth-level routing nodes and the fifth-level routing nodes are determined.

[0011] In a possible implementation, determining the fourth-level routing node and the fifth-level routing node according to the time dimension of the log classification includes: Classify the log information by year and determine the four-level routing nodes; Classify the log information based on the month dimension and determine the five-level routing nodes.

[0012] In a possible implementation, determining a leaf node according to the hierarchical classification category information includes: Determine the first leaf node according to the successful recording result of the log information; Determine the second leaf node according to the failure record result of the log information; Determine the third leaf node according to the statistical rule results of the log information; The fourth leaf node is determined according to the result of writing the data details of the log information.

[0013] In a possible implementation, constructing a multi-level folder group according to the storage path of the log information includes: According to the first-level routing nodes, second-level routing nodes, third-level routing nodes, fourth-level routing nodes, fifth-level routing nodes and leaf nodes of the storage structure, a multi-level folder group corresponding to each node is created, wherein each routing node is arranged in order from the root node, the first-level routing node to the leaf node.

[0014] In a possible implementation, the obtaining of multiple groups of initial data respectively reported by multiple different data reporting nodes includes: Based on the JDBC interface, multiple groups of original data reported by multiple different data reporting nodes are obtained; The multiple groups of original data are respectively annotated with data reporting node storage time to obtain multiple groups of initial data.

[0015] In a possible implementation, the step of marking the storage time of the data reporting node on the multiple groups of original data to obtain the initial data includes: The original data is labeled with batch number, submission time and serial number to obtain the initial data.

[0016] In a possible implementation, writing the log information corresponding to the multiple groups of initial data and the data reporting node storage time into the log system to obtain the classified storage result includes: According to the operation type of the initial data, the operation corresponding to the operation type is performed on the data written in the database, and the processing result status information and operation information are obtained; wherein the database is used to store multiple groups of initial data respectively reported by multiple different data reporting nodes; The data reporting node storage time, operation information and processing result status information corresponding to the initial data are written into the log system.

[0017] In a possible implementation, after obtaining the processing result status information and the operation information, the following is further included: According to the processing result status information, the operation corresponding to the execution operation type is verified and updated.

[0018] In a possible implementation, the processing result status information includes success information and failure information, and the verifying and updating of the operation corresponding to the execution operation type according to the processing result status information includes: If the processing result status information is failure information, the result of executing the operation corresponding to the corresponding type of operation is canceled, and the failure information is recorded; The failure information is written into the log system.

[0019] In a possible implementation, after obtaining the processing result status information and the operation information, the following is further included: The initial data is written into the database.

[0020] In a possible implementation, the step of writing the log information corresponding to the multiple groups of initial data and the data reporting node storage time into the log system to obtain the classified storage result further includes: Building multiple log system mirror storage areas based on a shared storage device, wherein the log system mirror storage areas correspond one-to-one to multiple different data reporting nodes; Write the log information corresponding to the initial data reported by each data reporting node into the corresponding log system mirror storage area; The log information in the mirror storage areas of the multiple log systems is stored in the log system according to the data structure layer information and statistical screening conditions of the log system to obtain a classified storage result.

[0021] In a possible implementation, performing statistical analysis on the initial analysis result to obtain a target analysis result includes: Obtaining query conditions, adding the query conditions to a preset query condition library, and reading corresponding log information from the log system according to the query conditions; Perform statistical analysis on the log information to obtain target analysis results, and store them in a result statistics library in a preset format; wherein the result statistics library is used to store the target analysis results.

[0022] In a possible implementation, performing statistical analysis on the log information to obtain a target analysis result includes: The log information is summed, arithmetic evaluated, sorted and classified to obtain the target analysis result.

[0023] In a possible implementation, after performing statistical analysis on the log information to obtain a target analysis result and storing it in a preset format, the method further includes: Get the query conditions; If the query condition obtained is the same as the query condition in the preset query condition library, the target analysis result stored in the preset format in the result statistics library is directly called.

[0024] In a second aspect, the present invention further provides a log-based data analysis device, comprising: A system construction module, used to construct a log system, wherein the log system is determined according to data structure layer information and statistical screening conditions, wherein the data structure layer information is used to characterize the storage form and storage structure of log information, and the statistical screening conditions are determined according to data analysis requirements; An acquisition module, used to acquire multiple groups of initial data respectively reported by multiple different data reporting nodes, wherein the multiple groups of initial data all include the storage time of the data reporting node; A writing module, used for writing the log information corresponding to the multiple groups of initial data and the storage time of the data reporting node into the log system to obtain a classified storage result, and obtaining an initial analysis result according to the classified storage result; The analysis module is used to perform statistical analysis on the initial analysis results to obtain target analysis results.

[0025] In a third aspect, the present invention further provides a method for deploying a data analysis device, which is applied to the log-based data analysis device described above, including: multi-point deployment and single-point deployment; The multi-point deployment includes multiple system building modules and multiple writing modules, each of the system building modules corresponds to each of the writing modules one by one, and each of the writing modules is used to write the log information corresponding to the multiple groups of initial data and the data reporting node storage time into the log system constructed by the corresponding system building module to obtain a classified storage result, and obtain an initial analysis result according to the classified storage result; The single-point deployment includes a system building module and a writing module, and the writing module is used to write the log information corresponding to the multiple groups of initial data and the data reporting node storage time into the log system constructed by the system building module to obtain a classified storage result, and obtain an initial analysis result based on the classified storage result.

[0026] In a fourth aspect, the present invention further provides an electronic device, comprising: a processor and a memory; The memory stores a computer-readable program executable by the processor; When the processor executes the computer-readable program, the steps in the log-based data analysis method described above are implemented.

[0027] In a fifth aspect, the present invention also provides a computer-readable storage medium, which stores one or more programs, and the one or more programs can be executed by one or more processors to implement the steps in the log-based data analysis method as described above.

[0028] The beneficial effects of the present invention are as follows: the log system determined based on the data structure layer information and statistical screening conditions can automatically classify the stored data according to the data structure layer information and statistical screening conditions when the data is written to the storage. When the data analysis is subsequently performed based on the log system provided by the present application, it can be called in a targeted manner according to the data classification storage results, thereby narrowing the query scope and improving the analysis efficiency and accuracy. In addition, the data structure layer information defines the storage form and storage structure of the log information. Through the flexible storage form and storage structure design, the log system can adapt to data of different types and structures, thereby realizing the effective classification and management of complex data sets. The statistical screening conditions are determined according to the data analysis requirements, so that the log system can filter and classify the data according to specific business needs, and the customized data processing capabilities enhance the practicality and value of the log system.

[0029] Furthermore, when the log information is written to the log system provided by the present application, the classified storage result of the log information can be determined according to the data structure layer information and the statistical screening conditions, achieving the purpose of real-time analysis while writing and analyzing, and further improving the efficiency of data analysis based on logs. And in the subsequent analysis process, the classified storage result of the log system is directly called, which, on the one hand, facilitates the analysis process and reduces the pressure on the analysis system, and on the other hand, by calling the classified storage result of the log system, the defect of high database pressure caused by direct access to the database is avoided. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 It is a schematic diagram of the data analysis storage and analysis environment structure in the prior art; Figure 2 It is a schematic diagram of the process domain structure of data analysis in the prior art; Figure 3 A method flow chart of an embodiment of a log-based data analysis method provided by the present invention; Figure 4 This is a method flow chart of an embodiment of step S301; Figure 5 A schematic diagram of the log system hierarchy structure constructed in step S301; Figure 6 This is a method flow chart of an embodiment of step S302; Figure 7 This is a schematic diagram of the application logic transaction unit writing log information in step S303; Figure 8 This is a schematic diagram of an embodiment of a log information generation relationship in step S303; Fig. 9 The schematic diagram of the underlying architecture for writing log information in step S303; Fig.10 This is a schematic diagram of another embodiment of the log information generation relationship in step S303; Fig.11 is a method flow chart of an embodiment of step S303; Fig.12 A schematic diagram of a route for writing log information into a shared storage device in step S303; Fig.13 A schematic diagram of a shared storage device route in step S303; Fig.14 is a method flow chart of an embodiment of step S304; Fig.15 This is a schematic diagram of the underlying architecture of data analysis in step S304; Fig.16 This is a schematic diagram of target data storage format conversion in step S304; Fig.17 is a schematic diagram of an embodiment of a log-based data analysis device provided by the present invention; Fig.18 It is a schematic diagram of the operating environment of an electronic device provided by the present invention. DETAILED DESCRIPTION

[0031] The preferred embodiments of the present invention are described in detail below in conjunction with the accompanying drawings, wherein the accompanying drawings constitute a part of this application and are used together with the embodiments of the present invention to illustrate the principles of the present invention, but are not used to limit the scope of the present invention.

[0032] A specific embodiment of the present invention discloses a data analysis method based on logs, see Figure 3 ,include: S301, constructing a log system, wherein the log system is determined according to data structure layer information and statistical screening conditions, wherein the data structure layer information is used to characterize the storage form and storage structure of log information, and the statistical screening conditions are determined according to data analysis requirements; S302, obtaining multiple groups of initial data respectively reported by multiple different data reporting nodes, wherein the multiple groups of initial data all include the storage time of the data reporting nodes; S303, writing the log information corresponding to the multiple groups of initial data and the storage time of the data reporting node into the log system to obtain a classified storage result, and obtaining an initial analysis result according to the classified storage result; S304: Perform statistical analysis on the initial analysis results to obtain target analysis results.

[0033] In this embodiment, the log system is a carrier that records and stores log information corresponding to business data according to certain rules, and can be a local file system, a network file system, a database, a cloud storage service, or a log management platform. In order to solve the problem that a large number of read and write operations are required to analyze business data in the traditional method, resulting in a large database pressure or even database lock, the present invention builds a log system to store log information corresponding to business data. When the business data needs to be analyzed, the log information in the log system is directly called and analyzed to achieve the purpose of data analysis, which greatly alleviates the read and write pressure of the database.

[0034] Although traditional log information storage carriers (or traditional log systems) can also store log information corresponding to business data, and directly read the log information in the log system for analysis when business data needs to be analyzed, thus alleviating the reading and writing pressure caused by directly reading the database, traditional log information storage carriers are merely containers for storing information, and the written data is placed in them without any rules or regulations. When the amount of written log information is large, the efficiency of reading log information for analysis is low, and the needle-in-a-haystack reading of data will affect the reading accuracy.

[0035] Therefore, a log system for recording and storing log information corresponding to business data according to certain rules is provided in the embodiment of the present application, specifically, a log system is constructed according to data structure layer information and statistical screening conditions. The log system determined based on the data structure layer information and statistical screening conditions can automatically classify the stored data according to the data structure layer information and statistical screening conditions when the data is written to the storage. When the log system provided by the present application is subsequently used for data analysis, it can be called in a targeted manner according to the data classification storage results, narrowing the query scope and improving the analysis efficiency and accuracy. In addition, the data structure layer information defines the storage form and storage structure of the log information. Through the flexible storage form and storage structure design, the log system can adapt to data of different types and structures, thereby realizing the effective classification and management of complex data sets. The statistical screening conditions are determined according to the data analysis requirements, so that the log system can screen and classify the data according to specific business needs, and the customized data processing capabilities enhance the practicality and value of the log system.

[0036] Furthermore, when the log information is written to the log system provided by the present application, the classified storage result of the log information can be determined according to the data structure layer information and the statistical screening conditions, achieving the purpose of real-time analysis while writing and analyzing, and further improving the efficiency of data analysis based on logs. And in the subsequent analysis process, the classified storage result of the log system is directly called, which, on the one hand, facilitates the analysis process and reduces the pressure on the analysis system, and on the other hand, by calling the classified storage result of the log system, the defect of high database pressure caused by direct access to the database is avoided.

[0037] In this embodiment, the log system is not limited to a specific one of a local file system, a network file system, a database, a cloud storage service or a log management platform, etc. It is understandable that as long as the log system can store log information according to the rules and storage paths provided in this application, it will be fine.

[0038] It should be noted that the data structure layer information is used to characterize the storage form and storage structure of the log information, and the data structure layer information includes the configuration information, routing node information, the framework, index and partition of the log system storage data, and different hierarchical structures for characterizing the data source. Based on the configuration information of the data source, routing node information and the framework, index and partition of the log system storage data, each level of the log system is responsible for information management and query of different granularities. Thus, based on the data layer structure information, a log system similar to the hierarchical routing management system is formed. It is understandable that the log system is designed into the mode of the routing management system, which can clearly define the path selection process of the log information from the source node to the bottom node. Through the transmission relationship and path selection relationship between the top node and the bottom node, as the storage path and hierarchical classification architecture of the log information, the purpose of real-time writing of the log information and hierarchical classification statistical analysis can be achieved, and the log information can be ensured to be stored and analyzed efficiently and reliably.

[0039] Furthermore, based on the data structure layer information, the log system is designed as a multi-level storage system. According to different business data and business data analysis requirements, the hierarchical structure, number of levels and the configuration of each level can be flexibly adjusted, and the granular information responsible for each level can be dynamically and in real time managed, thereby improving the flexibility of the log system and achieving the scalability and practicality of the log system.

[0040] Furthermore, the storage form and storage structure are determined based on the configuration information of the data source, the routing node information, the framework, index and partition of the log system storage data, and different hierarchical structures, so as to construct a log system similar to the routing node management system. In this embodiment, the log system is a tree-like hierarchical structure, one routing node corresponds to one level, and multiple routing nodes are set from the top to the bottom. Each log information is based on its own attributes, labels and characteristics, and at each storage node, according to the limiting conditions of the node entrance, selects a path that matches itself for storage or transmits it to the next node.

[0041] It is understandable that in other embodiments, the log system can also be other structural types, such as a single-line link, a star structure, a bus structure, a ring structure or a mesh structure. In order to ensure that each log information has the criterion of uniqueness of the classification standard during the hierarchical and classified storage process, improve the accuracy of the storage classification analysis, and improve the accuracy of the subsequent call log information, the embodiment of the present invention selects a tree hierarchy. It should be noted that the limiting conditions between each node of the same architectural level are single, and a log information can select a node that matches its own attributes, labels and features for storage or transmission to the next level, thereby improving the accuracy of the hierarchical and classified storage of log information, improving the accuracy of the real-time write analysis of log information, and further improving the accuracy of data analysis based on log information.

[0042] It should be noted that data analysis requirements refer to the data processing and analysis requirements proposed in specific business scenarios in order to achieve specific business goals or solve specific problems. These requirements usually come from business personnel, data scientists or analysts, and are the starting point and basis for data analysis work. Statistical screening conditions refer to a set of conditions set in the data analysis process to extract and process a specific subset of data. Statistical screening conditions are determined according to data analysis requirements. Through statistical screening conditions, log information is accurately graded and classified for storage, thereby achieving the real-time purpose of variable storage and variable analysis. In addition, through classified storage, the scope of calling log information for subsequent analysis can be significantly reduced, and relevant log information can be effectively extracted and processed, thereby improving the efficiency and accuracy of log information analysis.

[0043] In a specific embodiment, the business data is analyzed mainly based on statistical theory. Specifically, the statistical analysis conditions include at least but not limited to the following scenarios: If the business goal is to analyze user login behavior during a specific time period on a certain day, the statistical filter conditions may include timestamp (specific time period) and log type (login log); If you need to find the number of occurrences of a specific error code, the statistical filter criteria may include the error code and the time range; If you need to analyze the operation behavior of a specific user in the system, the statistical filtering conditions may include user ID, operation type and time range.

[0044] In other embodiments, other screening conditions may be used, including but not limited to qualitative analysis conditions, correlation analysis conditions, or regression and classification analysis conditions, etc. It is understandable that any conditions may be used as long as the business data analysis requirements and objectives are met.

[0045] In step S302, different data reporting nodes refer to data sources in different locations, including but not limited to data from different databases, data from different storage media, data from different network disks, or data from different shared devices. It is understandable that the log system can process multiple sets of initial data reported by multiple different data reporting nodes in real time, and has the ability to integrate multi-source data.

[0046] It can be understood that the log system can not only receive data from different data source locations, but also receive data from the same data source location, that is, multiple different data reporting nodes can transmit log information to the log system at the same time, or transmit data intermittently. At the same time, multiple data reporting nodes can access the network to write log information, or a single data reporting node can access the network to write log information.

[0047] Furthermore, in order to solve the problem in the prior art that data statistical results are different due to the storage time of the same data information in each link of the shared library and the data governance library (for example, data A is written into the shared library by the business unit at 23:00 on July 14, and the credit supervisory unit synchronizes it to the to-be-cleaned library in the data governance library at 2:00 on July 15, and the result library is processed and stored in the data governance library at 3:00 on July 16), it is difficult to ensure that the analysis and calculation are carried out at the same time during the comprehensive data analysis process, resulting in different calculation results of the same type of data in each library, thereby affecting the accuracy of data analysis. Therefore, the present application marks the storage time of the data reporting node for each initial data, so that the generation time of each log information is traceable, and ensures that when the log information in the log system is called for analysis in the later stage, the statistical analysis can be carried out based on the storage time of the data reporting node, thereby ensuring the consistency of the analysis results of different data reporting nodes.

[0048] The storage time of the data reporting node refers to the time when the data enters the data reporting node.

[0049] Furthermore, the business type of the initial data is not limited, and it can be data for campus security management scenarios or urban traffic analysis data. In this embodiment, the initial data is the public credit business data submitted by the public credit investigation business department. Among them, the structure and type of data are not limited in this embodiment, and can at least include configuration data, user data, basic data, reference data, template data, permission data, default data, sample data, historical data and metadata.

[0050] In step S303, the log information corresponding to the multiple groups of initial data is a form of data, usually operation records, event records, error records, etc. In database operations, log information can record the time of the operation, the executing user, the type of operation, etc. Through the above recorded information, data analysis and management are achieved. Therefore, managing and monitoring data based on log information can restore the authenticity of the data and ensure the integrity of the data. Storing log information and original data separately, and calling log information as the object of subsequent data, can not only ensure the integrity and traceability of the data and enhance the value of the data, but also avoid frequent read and write calls to access the database during data analysis, resulting in excessive database pressure and locking, thereby achieving the purpose of "read-write" separation of data.

[0051] Furthermore, a log system with a tree-like hierarchical structure formed according to data structure layer information and statistical screening conditions is used to store log information. The log information is classified and hierarchically managed while it is being written, thereby achieving the purpose of real-time preliminary analysis of the log information, thereby being able to obtain initial analysis results based on the classified storage results obtained from the log write storage.

[0052] In step S304, the classification storage results of the log system are directly called and the initial analysis results are statistically analyzed to obtain the target analysis results, thereby avoiding the problem of excessive database pressure caused by frequent database reading during the analysis process.

[0053] In some embodiments, to construct the log system, see Figure 4 ,include: S401, determining hierarchical classification information of log information according to data structure layer information and statistical screening conditions; S402, determining a storage path for log information according to the hierarchical classification information; S403: Construct a multi-level folder group according to the storage path of the log information.

[0054] In this embodiment, the ultimate goal of log information storage is to realize data analysis while writing, so as to obtain classified storage results. Therefore, the hierarchical classification information of the data is determined according to the data structure layer information and statistical screening conditions. The hierarchical classification information includes classifying the log information according to the source of the data, configuration information, statistical rules and characteristics, so as to achieve the purpose of statistical analysis.

[0055] The essence of the log system is to store log information. Therefore, the storage structure of log information is determined based on the hierarchical classification information. The essence is to determine the storage path of log information in the log system. It should be noted that each level and node in the tree-like hierarchical structure of the log system uses a folder to store log information. Therefore, a folder group of a multi-level tree-like hierarchical structure is determined based on the data structure layer information and statistical screening conditions.

[0056] In a specific embodiment, the routing node information, data instance, table and pattern information of the folder group of the multi-level tree hierarchy are determined by the data structure layer information; thereby, the first-level routing node is determined according to the routing node information; the second-level routing node is determined according to the data instance, the table and the pattern information; according to the statistical filtering conditions, the dimension of the hierarchical classification of the log information is determined, thereby determining the filtering condition storage channel node; finally, the leaf node is determined according to the hierarchical classification category information.

[0057] Further, the determining of the screening condition storage channel node according to the dimension of the hierarchical classification of the log information includes: Determine the third-level routing nodes based on the statistical rule dimensions of log classification; According to the time dimension of log classification, the fourth-level routing node and the fifth-level routing node are determined. Specifically, the log information is classified according to the dimension of year to determine the fourth-level routing node; the log information is classified according to the dimension of month to determine the fifth-level routing node.

[0058] In other embodiments, the screening condition storage channel nodes may also be determined by dimensions such as data size, data structure, log information category, etc. The number of nodes corresponding to the screening condition storage channel level is not limited and can be flexibly designed and expanded according to business data analysis requirements and goals.

[0059] In a specific embodiment, taking the unified storage "credit information log system" as an example, the complete log file group set is as follows: Figure 5 As shown in the figure, the entire log system consists of six levels of folders, in which each routing node is arranged in order from the root node, the first-level routing node to the leaf node. In the entire design, the bottom layer used to store specific files is called "leaf node", and the others are "routing nodes". The description of each level of folders is shown in the following table: The first-level routing nodes, second-level routing nodes and third-level routing nodes are all generated based on the configuration rule module configuration. Specifically: The first-level routing node is used to store the corresponding routing node information, such as the database IP address, the corresponding deployment address, port, etc., and quickly query according to the relevant information when searching. It consists of info.yml and a series of folder collections. Among them, info.yml is used to describe each folder, and the relevant data is generated when editing in the "Configuration Module". The node code generates the corresponding routing folder according to the node code configured in the "Configuration Module".

[0060] The secondary routing node is used to store the corresponding instance, table, mode and other information. It is configured by the "Configuration Module" and defaults to the corresponding selected mode-table name.

[0061] The third-level routing node is used to store channel screening conditions and quickly filter the corresponding statistical dimensions. For example, statistics of the license data corresponding to the decision date are calculated according to the decision date. It consists of two parts: the info folder and the [code] folder. The info folder stores statistical information, including total quantity, batch statistics and other data. [code] is the condition code for filtering logs according to the channel configured in the "Configuration Module". The number of folders is determined by the settings in the "Configuration Module".

[0062] Level 4 and 5 routing nodes: mainly generate corresponding folders based on year and month.

[0063] Level 6 is set as leaf nodes: it consists of total.dfs, log.yml, log.dfs, process.log main files, and some corresponding history files. Among them: ①log.dfs: Write all log information into the log.dfs file under the corresponding log according to the channel setting rules, that is, store detailed and specific data details.

[0064] Generate historical log files according to the log file generation cycle set in the "Configuration Module". The rules include: 6 hours, 12 hours, and one day. The main steps involved in generating historical log files are: 1. The cache stops writing logs to log.dfs. 2. Modify the log.dfs folder name according to the rules. The naming principle is [logyyyymmddh24mmssxx].dfs. The relevant content is formed by obtaining the server timestamp.

[0065] Regenerate a new log.dfs file.

[0066] The cache rewrites records to the new log.dfs file.

[0067] ②Total.dfs: used to store statistical information of written files. In this embodiment, Total.dfs is mainly used to count and store log information corresponding to successfully written data.

[0068] The file writing rules are as follows: total:xx--the total amount of logs written Batch:{code:xx,num:xx,time:xxx,mode:I / D}, --Write specific batch related information, where I in the model represents new logs and D represents data removal information.

[0069] ③Log.yml: mainly describes the files under the current leaf node according to the set statistical rules, where the statistical rules can be rules such as generation time, frequency, size, etc. In this embodiment, Log.yml is mainly used to record the data written according to a certain statistical frequency.

[0070] The specific parameters are as follows: log.time:0 / 1 / 2--0 is 24 hours--1 is 12 hours--2 is 6 hours ④Process.dfs: used to handle submission failures during data submission. The compilation rules are as follows: {code:xx,num:xx,time:xx}where code is the corresponding batch number.

[0071] It should be noted that the leaf node can include but is limited to the main files of total.dfs, log.yml, log.dfs, and process.log. The storage rules and storage categories of each main file can be dynamically adjusted according to the analysis requirements and goals of the business data, and the number and structure of nodes of the entire leaf node can be expanded.

[0072] In some embodiments, the acquisition of multiple sets of initial data respectively reported by multiple different data reporting nodes, wherein the multiple sets of initial data each include the storage time of the data reporting node, see Figure 6 ,include: S601, acquiring multiple groups of original data respectively reported by multiple different data reporting nodes based on the JDBC interface; S602 , respectively annotate the multiple groups of original data with the storage time of the data reporting nodes to obtain multiple groups of initial data.

[0073] In this embodiment, two channels, JDBC and Http interface services, are provided to report data to external businesses. In order to meet the needs of different application scenarios, a corresponding Jdbc driver is designed. In this driver, database connection, submission of new data, data removal (including objection data) and other JDBC drivers that meet the characteristics of public credit information management business data are completed and provided for third-party business calls.

[0074] Furthermore, Http interface services are provided synchronously. Based on the currently popular Http protocol, corresponding interface services are provided, and third parties can report public credit information through this coordination on an authorized basis.

[0075] It should be noted that in this embodiment, the JDBC interface is a custom interface, and the JDBC data reporting mode and data processing method are set to realize the writing of log information into the log system. Specifically, when reporting data through the above two channels, two data reporting modes are supported: one record / time and one batch / time, where the serial number corresponds to a single record, the batch number corresponds to a batch of data reported at a time, and the reporting time is accurate to seconds. The later statistical analysis is carried out based on the above three contents.

[0076] Further, the log information corresponding to the multiple groups of initial data and the data reporting node storage time are written into the log system to obtain a classified storage result, including: According to the operation type of the initial data, the operation corresponding to the operation type is performed on the data written in the database, and the processing result status information and operation information are obtained; wherein the database is used to store multiple groups of initial data respectively reported by multiple different data reporting nodes; The data reporting node storage time, operation information and processing result status information corresponding to the initial data are written into the log system.

[0077] It should be noted that in the process of writing data into the database, operation information corresponding to the operation type of the data and data processing result status information are generated synchronously, wherein the data processing result status information is the result of operating the data in the database according to the operation type. The operation information and data processing result status information are recorded in the form of log information and written into the log system. On the one hand, it can ensure that the writing of log information is synchronized and consistent with the data. On the other hand, the operation status of the data is reflected through the log information, and the log information and the data are stored separately. When the data needs to be analyzed later, only the log information in the log system needs to be called to obtain the complete "basic data information" and "data operation" information, which not only ensures the accuracy of data analysis, but also avoids the problem of high database pressure caused by frequent reading and writing.

[0078] In this embodiment, the log information includes the data reporting node storage time, operation information, processing result status information and basic data information.

[0079] In a specific embodiment, by configuring the logical transaction unit group and generating a corresponding logical object array based on the logical transaction unit group according to the operation information provided by the front end, the array is uniformly packaged and provided to the logical transaction unit for further processing. Figure 7 As shown, the core task of the logical transaction unit in this embodiment is to intercept and receive various types of operation contents with the help of the "operation interception unit", and judge the query, add, delete and other operations according to the operation type. If it is a query operation, then this type of operation is provided to the data processing unit to query the corresponding result. If it is an add, delete and other operations, then this type of operation is transferred to the "transaction factory" to establish a corresponding failure rollback mechanism. And it is submitted to the data processing unit and the log unit for processing under the "factory instance".

[0080] It should be noted that data operation types include but are not limited to query, add, delete and update operation types.

[0081] Feedback data for the "Transaction Factory", such as Figure 8 As shown, based on the following logical design: After obtaining the corresponding operation, the "transaction factory" writes the data to the database on the basis of establishing the corresponding transaction. After receiving the successful status information of the data being written to the database, it synchronously provides the operation information and processing result status information to the log system for writing. After completing all the content, it returns a successful status indicator to indicate that the operation is completed. If any link fails, the operation content is rolled back.

[0082] Among them, on the basis of establishing the corresponding transaction, writing data into the database and receiving the successful status information of the data writing feedback are set as follows Fig. 9 The data processing unit shown in the figure interacts with the status information. The data processing unit is responsible for writing the data provided by the "transaction factory" instance into the database, establishing the corresponding database transaction as required, and returning the processing result status to the transaction factory instance. Fig.10 As shown in the figure, after receiving the data to be submitted, the "data processing unit" writes the data into the database. If the submission is successful, the status is returned to the corresponding transaction factory instance. After receiving the corresponding write status, the instance writes it into the corresponding log system. If the submission is successful, the instance task ends. If it fails, the corresponding database write operation is rolled back and the error log is written into the log system.

[0083] In some embodiments, the log information corresponding to the multiple groups of initial data and the storage time of the data reporting node are written into the log system to obtain the classified storage result, see Fig.11 , also includes: S1101, constructing multiple log system mirror storage areas based on a shared storage device, wherein the log system mirror storage areas correspond one-to-one to multiple different data reporting nodes; S1102, writing the log information corresponding to the initial data reported by each data reporting node into the corresponding log system mirror storage area; S1103: Store the log information in the multiple log system mirror storage areas into the log system according to the data structure layer information and statistical screening conditions of the log system to obtain a classified storage result.

[0084] A shared storage device (Storage Area Network, SAN) is a high-speed network system specially designed to connect servers and storage devices. The main function of SAN is to provide multiple servers with high-bandwidth, low-latency access to shared storage resources, which can be disk arrays, tape libraries, optical disk towers, etc. SAN supports block-level data transmission by using technologies such as Fiber Channel or Ethernet (using iSCSI protocol).

[0085] In this embodiment, if Figure 12-13As shown, taking the storage of public credit data as an example, through SAN technology, the disk storage space of the log system is shared into multiple log system mirror storage areas to multiple data reporting nodes in the network. Each node is responsible for writing the generated log information into the log system mirror storage area corresponding to each data reporting node, and aggregating and storing it in the folder group of the log system to obtain classified storage results, forming a unified log file storage area. When the data is analyzed later, the corresponding log information is directly called from the unified log file storage area for analysis.

[0086] Therefore, based on SAN high-bandwidth, low-latency access technology, it can ensure that all servers access the same log data to avoid data inconsistency. For example, in a network with multiple different data reporting nodes, each node needs to record operation logs. Through SAN shared storage, these logs can be centrally managed and analyzed, achieving the purpose of synchronous reading and writing of stored log information.

[0087] In some embodiments, see Fig.14 , performing statistical analysis on the initial analysis results to obtain target analysis results, including: S1401, obtaining a query condition, adding the query condition to a preset query condition library, and reading corresponding log information from the log system according to the query condition; S1402: Perform statistical analysis on the log information to obtain target analysis results, and store them in a result statistics library in a preset format; wherein the result statistics library is used to store the target analysis results.

[0088] In this embodiment, at the implementation level, a data analysis module is constructed to analyze the log system and obtain the target analysis result. Fig.15 As shown: Loading unit: responsible for passing the received information to the log system, and packaging the data provided by the log system into usable objects based on the metadata information, and providing operations based on the objects, including (querying the total number of objects, log details, etc.).

[0089] When querying the corresponding log, search rules (including table information, channel, time, etc.), where the channel can be set to "decision unit", "region", etc. according to the corresponding relationship.

[0090] (2) Statistical analysis unit: This unit is responsible for analyzing and running the data provided by the "loading unit" and writing the analysis results into the corresponding "cache area". This unit can be divided into three parts: "service interface", "processing unit" and "statistical cache unit". The service interface is responsible for providing external units with calls through the interface. The processing unit is responsible for analyzing and calculating various types of data provided by the "loading unit", including summation, arithmetic evaluation value, sorting, classification summary, etc. The corresponding processing results are written into the "statistical cache area" to provide support services for later operations. The "statistical cache area" mainly caches the last processing results directly into the corresponding file in the "key->value" method. If the current statistical analysis service directly calls the last query condition, the statistical value can be directly obtained in the statistical cache area.

[0091] (3) Configuration cache unit: responsible for obtaining the corresponding configuration data, writing the data into the global object, and providing it to other units for calling.

[0092] (4) Result unit: converts all processing results into corresponding JSON classes and returns the call results. Fig.16 shown.

[0093] (4) Service interface: Based on summation, arithmetic evaluation value, sorting, classification aggregation and other methods, the "decision unit" and time range set in the channel are encapsulated into corresponding interface methods for external applications to call. For example: The interface for obtaining the total amount information reported by the corresponding unit according to the time range in the front-end database is: Public static H getDataOfTotle(String ChannleCode,Date beginDate,DateendDate); / / ChannleCode is written to JDDW Where H is the encapsulated JSON format return object, and its value is provided by the result unit.

[0094] The log-based data analysis device provided by the present invention adopts the "write-divide" separation form. On the basis of ensuring stability and reliability, it uses technical means to reduce the long-term cost investment caused by "personnel investment" and "data statistical analysis guarantee group", thereby improving the analysis efficiency.

[0095] Based on the above log-based data analysis method, the embodiment of the present invention further provides a log-based data analysis device 1700, see Fig.17 ,include: System construction module 1701, used to construct a log system, the log system is determined according to data structure layer information and statistical screening conditions, the data structure layer information is used to characterize the storage form and storage structure of log information, and the statistical screening conditions are determined according to data analysis requirements; The acquisition module 1702 is used to acquire multiple groups of initial data respectively reported by multiple different data reporting nodes, wherein the multiple groups of initial data all include the storage time of the data reporting node; A writing module 1703 is used to write the log information corresponding to the multiple groups of initial data and the storage time of the data reporting node into the log system to obtain a classified storage result, and obtain an initial analysis result according to the classified storage result; The analysis module 1704 is used to perform statistical analysis on the initial analysis results to obtain target analysis results.

[0096] Based on the above log-based data analysis device, an embodiment of the present invention further provides a method for deploying a data analysis device, including: multi-point deployment and single-point deployment; The multi-point deployment includes multiple system building modules and multiple writing modules, each of the system building modules corresponds to each of the writing modules one by one, and each of the writing modules is used to write the log information corresponding to the multiple groups of initial data and the data reporting node storage time into the log system constructed by the corresponding system building module to obtain a classified storage result, and obtain an initial analysis result according to the classified storage result; The single-point deployment includes a system building module and a writing module, and the writing module is used to write the log information corresponding to the multiple groups of initial data and the data reporting node storage time into the log system constructed by the system building module to obtain a classified storage result, and obtain an initial analysis result based on the classified storage result.

[0097] In this embodiment, single-point deployment means that the number of devices (or relational database instances) in the front-end shared database and the classification management database is 1, and multi-point deployment means that the number of devices (or relational database instances) in the front-end shared database and the classification management database is > 1.

[0098] like Fig.18 As shown, based on the above log-based data analysis method, the present invention also provides an electronic device, which can be a computing electronic device such as a mobile terminal, a desktop computer, a notebook, a palm computer, and a server. The electronic device includes a processor 1810, a memory 1820, and a display 1830. Fig.18 Only some components of the electronic device are shown, but it should be understood that it is not required to implement all of the components shown, and more or fewer components may be implemented instead.

[0099] In some embodiments, the memory 1820 may be an internal storage unit of the electronic device, such as a hard disk or memory of the electronic device. In other embodiments, the memory 1820 may also be an external storage electronic device of the electronic device, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (SecureDigital, SD) card, a flash card (Flash Card), etc. equipped on the electronic device. Further, the memory 1820 may also include both an internal storage unit of the electronic device and an external storage electronic device. The memory 1820 is used to store application software and various types of data installed in the electronic device, such as program codes for installing the electronic device. The memory 1820 may also be used to temporarily store data that has been output or is to be output. In one embodiment, a log-based data analysis program 1840 is stored on the memory 1820, and the log-based data analysis program 1840 can be executed by the processor 1810, thereby realizing the log-based data analysis method of each embodiment of the present application.

[0100] In some embodiments, the processor 1810 may be a central processing unit (CPU), a microprocessor or other data processing chip, used to run program codes stored in the memory 1820 or process data, such as executing a log-based data analysis method.

[0101] In some embodiments, the display 1830 may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, an OLED (Organic Light-Emitting Diode) touch device, etc. The display 1830 is used to display information on the log-based data analysis electronic device and to display a visual user interface. The components 1810-1830 of the electronic device communicate with each other via a system bus.

[0102] Those skilled in the art will appreciate that all or part of the processes of the above-mentioned embodiments can be implemented by instructing related hardware through a computer program, and the program can be stored in a computer-readable storage medium, wherein the computer-readable storage medium is a disk, an optical disk, a read-only storage memory, or a random access memory, etc.

[0103] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by any technician familiar with the technical field within the technical scope disclosed by the present invention should be covered within the protection scope of the present invention.

Claims

1. A log-based data analysis method, characterized in that: include: Constructing a log system, wherein the log system is determined according to data structure layer information and statistical screening conditions, wherein the data structure layer information is used to characterize the storage form and storage structure of log information, and the statistical screening conditions are determined according to data analysis requirements; Acquire multiple groups of initial data respectively reported by multiple different data reporting nodes, wherein the multiple groups of initial data all include storage time of the data reporting nodes; Writing the log information corresponding to the multiple groups of initial data and the storage time of the data reporting node into the log system to obtain a classified storage result, and obtaining an initial analysis result according to the classified storage result; Perform statistical analysis on the initial analysis results to obtain target analysis results.

2. The log-based data analysis method according to claim 1, characterized in that: Constructing the log system includes: Determine the hierarchical classification information of log information based on the data structure layer information and statistical screening conditions; Determining a storage path for the log information according to the hierarchical classification information; A multi-level folder group is constructed according to the storage path of the log information.

3. The log-based data analysis method according to claim 2, characterized in that: Determining the hierarchical classification information of the log information according to the data structure layer information and the statistical screening conditions includes: According to the data structure layer information, determine the routing node information, data instance, table and mode information of the log information classification; Determine the dimensions for grading and classifying log information based on statistical screening conditions.

4. The log-based data analysis method according to claim 3, characterized in that: Determining the storage path of the log information according to the hierarchical classification category information includes: Constructing a storage structure, wherein the storage structure includes a plurality of routing nodes; Determine a primary routing node according to the routing node information; Determining a secondary routing node according to the data instance, the table, and the mode information; Determine the filtering condition storage channel node according to the dimension of the log information classification; According to the hierarchical classification category information, the leaf nodes are determined.

5. The log-based data analysis method according to claim 4, characterized in that: The step of determining the filtering condition storage channel node according to the dimension of the hierarchical classification of the log information includes: Determine the third-level routing nodes based on the statistical rule dimensions of log classification; According to the time dimension of log classification, the fourth-level routing nodes and the fifth-level routing nodes are determined.

6. The log-based data analysis method according to claim 5, characterized in that: Determining the fourth-level routing node and the fifth-level routing node according to the time dimension of log classification includes: Classify the log information by year and determine the four-level routing nodes; Classify the log information based on the month dimension and determine the five-level routing nodes.

7. The log-based data analysis method according to claim 4, characterized in that: The step of determining the leaf node according to the hierarchical classification category information includes: Determine the first leaf node according to the successful recording result of the log information; Determine the second leaf node according to the failure record result of the log information; Determine the third leaf node according to the statistical rule results of the log information; The fourth leaf node is determined according to the result of writing the data details of the log information.

8. The log-based data analysis method according to claim 7, characterized in that: The step of constructing a multi-level folder group according to the storage path of the log information includes: According to the first-level routing nodes, second-level routing nodes, third-level routing nodes, fourth-level routing nodes, fifth-level routing nodes and leaf nodes of the storage structure, a multi-level folder group corresponding to each node is created, wherein each routing node is arranged in order from the root node, the first-level routing node to the leaf node.

9. The log-based data analysis method according to claim 1, characterized in that: The obtaining of multiple groups of initial data respectively reported by multiple different data reporting nodes includes: Based on the JDBC interface, multiple groups of original data reported by multiple different data reporting nodes are obtained; The multiple groups of original data are respectively annotated with data reporting node storage time to obtain multiple groups of initial data.

10. The log-based data analysis method according to claim 9, characterized in that: The step of marking the storage time of the data reporting node on the plurality of groups of original data to obtain initial data includes: The original data is labeled with batch number, submission time and serial number to obtain the initial data.

11. The log-based data analysis method according to claim 1, characterized in that: The step of writing the log information corresponding to the multiple groups of initial data and the storage time of the data reporting node into the log system to obtain the classified storage result includes: According to the operation type of the initial data, the operation corresponding to the operation type is performed on the data written in the database, and the processing result status information and operation information are obtained; wherein the database is used to store multiple groups of initial data respectively reported by multiple different data reporting nodes; The data reporting node storage time, operation information and processing result status information corresponding to the initial data are written into the log system.

12. The log-based data analysis method according to claim 11, characterized in that: After obtaining the processing result status information and operation information, it also includes: According to the processing result status information, the operation corresponding to the execution operation type is verified and updated.

13. The log-based data analysis method according to claim 12, characterized in that: The processing result status information includes success information and failure information, and the checking and updating of the operation corresponding to the execution operation type according to the processing result status information includes: If the processing result status information is failure information, the result of executing the operation corresponding to the corresponding type of operation is canceled, and the failure information is recorded; The failure information is written into the log system.

14. The log-based data analysis method according to claim 11, characterized in that: After obtaining the processing result status information and operation information, it also includes: The initial data is written into the database.

15. The log-based data analysis method according to claim 11, characterized in that: The step of writing the log information corresponding to the multiple groups of initial data and the data reporting node storage time into the log system to obtain the classified storage result also includes: Building multiple log system mirror storage areas based on a shared storage device, wherein the log system mirror storage areas correspond one-to-one to multiple different data reporting nodes; Write the log information corresponding to the initial data reported by each data reporting node into the corresponding log system mirror storage area; The log information in the mirror storage areas of the multiple log systems is stored in the log system according to the data structure layer information and statistical screening conditions of the log system to obtain a classified storage result.

16. The log-based data analysis method according to claim 1, characterized in that: The performing statistical analysis on the initial analysis results to obtain target analysis results includes: Obtaining query conditions, adding the query conditions to a preset query condition library, and reading corresponding log information from the log system according to the query conditions; Perform statistical analysis on the log information to obtain target analysis results, and store them in a result statistics library in a preset format; wherein the result statistics library is used to store the target analysis results.

17. The log-based data analysis method according to claim 16, characterized in that: The performing statistical analysis on the log information to obtain a target analysis result includes: The log information is summed, arithmetic evaluated, sorted and classified to obtain the target analysis result.

18. The log-based data analysis method according to claim 16, characterized in that: After the log information is statistically analyzed to obtain the target analysis result and stored in a preset format, the method further includes: Get the query conditions; If the query condition obtained is the same as the query condition in the preset query condition library, the target analysis result stored in the preset format in the result statistics library is directly called.

19. A log-based data analysis device, characterized in that: include: A system construction module, used to construct a log system, wherein the log system is determined according to data structure layer information and statistical screening conditions, wherein the data structure layer information is used to characterize the storage form and storage structure of log information, and the statistical screening conditions are determined according to data analysis requirements; An acquisition module, used to acquire multiple groups of initial data respectively reported by multiple different data reporting nodes, wherein the multiple groups of initial data all include the storage time of the data reporting node; A writing module, used for writing the log information corresponding to the multiple groups of initial data and the storage time of the data reporting node into the log system to obtain a classified storage result, and obtaining an initial analysis result according to the classified storage result; The analysis module is used to perform statistical analysis on the initial analysis results to obtain target analysis results.

20. A method for deploying a data analysis device, applied to the log-based data analysis device according to claim 19, characterized in that: include: Multi-point deployment and single-point deployment; The multi-point deployment includes multiple system building modules and multiple writing modules, each of the system building modules corresponds to each of the writing modules one by one, and each of the writing modules is used to write the log information corresponding to the multiple groups of initial data and the data reporting node storage time into the log system constructed by the corresponding system building module to obtain a classified storage result, and obtain an initial analysis result according to the classified storage result; The single-point deployment includes a system building module and a writing module, and the writing module is used to write the log information corresponding to the multiple groups of initial data and the data reporting node storage time into the log system constructed by the system building module to obtain a classified storage result, and obtain an initial analysis result based on the classified storage result.

21. An electronic device, characterized in that: include: Processor and memory; The memory stores a computer-readable program executable by the processor; When the processor executes the computer-readable program, the steps in the log-based data analysis method as described in any one of claims 1-18 are implemented.

22. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the steps in the log-based data analysis method as described in any one of claims 1-18.

Images