Data analysis method and device, equipment and storage medium

Abstract

The invention belongs to the technical field of big data, is applied to the field of medical health and the field of life insurance, and discloses a data analysis method, device and equipment and a storage medium. The method comprises the steps that Hive operation log information is obtained, a target library table is positioned, library table information in the Hive operation log information is extracted, and target library table list data is obtained; analyzing the target library table list data to obtain partition metadata and HDFS path metadata; judging each table in the target library table one by one, judging whether the table is a partition table or not, and if the table is the partition table, obtaining an HDFS path of a root directory and obtaining an HDFS path of a partition directory; if not, acquiring an HDFS path of a root directory; performing data analysis on the file metadata according to the HDFS path to obtain table storage data; and performing statistical analysis on the table storage data to obtain data storage characteristics and resource consumption indexes. The problem that an existing Hive task performance analysis result is not accurate is solved.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of big data, and particularly relates to a data analysis method, device, equipment and storage medium. BACKGROUND

[0002] With the rapid development of big data technology, the data analysis task based on Hive in the business management system of financial technology, medical health and old-age care has been widely applied in data processing. Hive is built on the MapReduce computing framework, and its underlying operation depends on mechanisms such as task splitting, resource scheduling and job execution. The performance of Hive task directly affects the timeliness of business data processing, including but not limited to the generation efficiency of business daily operation report, the timeliness of data decision-making of company management layer, and the quality of external data service. In the actual operation and development process of big data platform, task performance problems have always been the focus of development and operation personnel. At present, the monitoring and analysis of task performance mostly depend on the running time or simple resource consumption indicators of the whole task, and the developers usually preliminarily judge the performance trend according to the total running time of the task. However, in fact, the actual operation of the task is affected by many external factors, such as cluster resource load state, task priority configuration, network transmission condition and runtime system resource scheduling, etc. These factors may mask the performance bottleneck or abnormal resource usage of the task itself, resulting in inaccurate performance analysis results. Especially in the traditional monitoring method, there is a lack of fine tracking of each key stage in the running process of Hive task. Hive task is usually composed of multiple Jobs, each Job contains multiple Stages, and the Stage is further divided into several MapReduce (MR) task stages. The existing scheme cannot effectively collect performance and analyze resources from these finer dimensions, and it is difficult to locate the bottleneck of the specific stage. This causes great obstacles in task optimization, resource scheduling adjustment and fault troubleshooting. SUMMARY

[0003] The application provides a data analysis method, device, equipment and storage medium to solve the problem of inaccurate performance analysis results of existing Hive task.

[0004] In a first aspect, a data analysis method is provided, comprising:

[0005] Obtaining Hive running log information, locating a target library table and extracting library table information in the Hive running log information to obtain target library table list data;

[0006] Analyzing the target library table list data to obtain partition metadata and HDFS path metadata;

[0007] judging whether each table in the target library table is a partition table, if the target library table is a partition table, obtaining the HDFS path of the root directory of the target library table and obtaining the HDFS path of the partition directory of the target library table; if the target library table is not a partition table, obtaining the HDFS path of the root directory of the target library table;

[0008] performing data analysis on file metadata according to the HDFS path to obtain table storage data;

[0009] performing statistical analysis on the table storage data to obtain data storage characteristics and resource consumption indexes;

[0010] Further, the data analysis method further comprises: obtaining Hive running log information, extracting log key information, the log key information comprising task identification information, execution time information, computing resource usage, input and output data volume and task running state;

[0011] According to the log key information, recording the most fine-grained running resource information in the log;

[0012] processing the running resource information to obtain structured most fine-grained running data records;

[0013] performing statistical analysis on the running data records to obtain a diagnosis result of task performance.

[0014] In a second aspect, a data analysis device is provided, comprising:

[0015] a library table information extraction module for extracting library table information in the Hive running log information to obtain target library table list data and log key information;

[0016] an HDFS path acquisition module for analyzing the target library table list data to obtain partition metadata and HDFS path metadata, and judging whether each table in the target library table is a partition table, if the target library table is a partition table, obtaining the HDFS path of the root directory of the target library table and obtaining the HDFS path of the partition directory of the target library table; if the target library table is not a partition table, obtaining the HDFS path of the root directory of the target library table;

[0017] a storage data acquisition module for performing data analysis on file metadata according to the HDFS path to obtain table storage data;

[0018] a running data record module for recording the most fine-grained running resource information in the log according to the log key information to obtain structured most fine-grained running data records;

[0019] A statistical analysis module is configured to perform statistical analysis on the table storage data and the running data records to obtain data storage features, resource consumption indexes and fine-grained diagnosis results of task performance.

[0020] In a third aspect, a computer device is provided, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the steps of the data analysis method when executing the computer program.

[0021] In a fourth aspect, a computer readable storage medium is provided, which stores a computer program, and the computer program implements the steps of the data analysis method when executed by a processor.

[0022] In the above scheme based on the data analysis method, device, equipment and storage medium, the key log information including task identification, execution duration, resource usage, I / O data volume and the like is obtained by analyzing the Hive running log, and is further refined to the execution stages such as Job, Stage and MapReduce in the task, so that the resource consumption of each stage can be recorded in the finest granularity, and the structured tracking and analysis of the Hive task running performance is realized.

[0023] The data analysis method supports recording and analyzing the resource usage in multiple hierarchical granularities, and the developers can accurately locate the specific stage of performance anomaly in the task, so as to determine whether the performance problem is caused by code logic, data skew, Shuffle process, file reading and writing efficiency and the like. Compared with the overall index, it is more conducive to quickly locating the bottleneck point and improving the diagnosis efficiency.

[0024] By analyzing the table metadata and the HDFS path, the system can automatically count the total size, the number of partitions, the number of files, whether there is an empty partition or a small file, the latest update time and the like of the table. In combination with the storage analysis, the storage features such as large table, hot and cold data, data skew and file fragmentation can be identified, which is convenient for further use optimization, archiving and cleaning and resource planning. BRIEF DESCRIPTION OF DRAWINGS

[0025] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the description of the embodiments of the present application. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can also be obtained by those skilled in the art without creative labor.

[0026] Figure 1 is a first flowchart of the data analysis method in an embodiment of the present application;

[0027] Figure 2Figure 2 is a second flow diagram of a data analysis method according to an embodiment of the present application;

[0028] Figure 3 Figure 3 is a structural diagram of a data analysis device according to an embodiment of the present application;

[0029] Figure 4 Figure 4 is a structural diagram of a computer device according to an embodiment of the present application;

[0030] Figure 5 Figure 5 is another structural diagram of a computer device according to an embodiment of the present application. DETAILED DESCRIPTION

[0031] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the scope of the present application.

[0032] Figure 1 Figure 1 is a first flow diagram of a data analysis method according to an embodiment of the present application, including the following steps:

[0033] S1, obtaining Hive running log information, locating a target library table and extracting library table information in the Hive running log information, obtaining target library table list data.

[0034] In the embodiments of the present application, a Python program is used to load and analyze the running log content of Hive. The log sources can include the execution log of HiveServer2, the task log of YARN or the execution log of MR / Tez, etc. In the log analysis process, the program analyzes the SQL execution statements (such as SELECT, INSERT, CREATE, DROP, etc.) in the log, uses regular expressions, AST abstract syntax tree analysis or a parser based on lexical / syntactic rules to extract the library name and table name information appearing in the Hive script. Then, for each log file, the program can identify one or more Hive tables involved therein, and these tables constitute the target library table set involved in the task. Through this method, the originally unordered, loose and unstructured log text data is converted into a standardized "target table list" structure, which is convenient for subsequent unified table metadata analysis, HDFS path analysis, file resource statistics and other work. The generated target library table list data is used as the input for subsequent data analysis and performance analysis.

[0035] In an embodiment, in a medical health insurance data platform, a large number of Hive SQL tasks are run daily to support multi-dimensional report, risk assessment, claim review, medical insurance cost control and other business functions. The system deploys a set of Python programs to collect the log content of running Hive scripts in batches, and automatically analyzes the target tables involved in the log. For example, in the Hive log of processing the "medical insurance large reimbursement detail monthly report" task, the program can automatically identify a plurality of business tables involved therein, such as the claim detail table, the patient information table, the hospital cost table, and the insurance product table.

[0036] In another embodiment, in the field of life insurance, the system scans and analyzes the Hive job log run daily through an automatic log analysis program, extracts the database table information related to the life insurance business. For example, a certain customer portrait update task involves policy_base_info, premium_payment_detail, claim_history and other key business tables. The system locates the actual reference position of these target tables in the task by identifying the SQL statement structure, keywords and execution semantics in the Hive log, and summarizes them into target database table list data.

[0037] S2 analyzes the target database table list data to obtain partition metadata and HDFS path metadata.

[0038] After completing the Hive log analysis and extracting the target database table list, the system executes a metadata collection process for each target table. The Python program obtains the structure definition information of the target table through Hive SQL or direct connection to the Hive metadata database, including table type (external table / internal table), field structure, partition field list, storage format, compression method, and physical storage location of the table.

[0039] Subsequently, classification processing is performed according to whether it is a partition table. For a partition table, the program further queries all existing partitions under the table, extracts the specific path information, partition key value and creation time of each partition, and constructs an access list of the partition path. This list constitutes the partition metadata of the table, including: partition field and its enumerated value, full path of each partition path on HDFS, partition creation or latest update time, zone data size and other meta information.

[0040] The Python program constructs the HDFS root path of the table according to the location field in the Hive metadata, and establishes a one-to-one mapping relationship between the table, partition and HDFS path based on the path of each partition, forming the HDFS path metadata of the table. For non-partition tables, only the root path and the corresponding HDFS meta information are extracted to form a complete metadata link at the table level, partition level and path level.

[0041] In an embodiment, for example, for hospital cost breakdown, the system identifies its strategy of partitioning by time and hospital code. By querying the Hive metastore or using Hive SQL (SHOW PARTITIONS) to automatically obtain the list of all existing partitions, the meta information of each partition is constructed.

[0042] In another embodiment, in the field of life insurance, for each business table in the list (such as life_insurance_policy, insured_client_info, claim_settlement_log, etc.), the system determines whether the table is a partitioned table by connecting the Hive metastore or parsing the meta information in the SQL statement. If the table is a partitioned table, further identify the partition field (such as policy effective date, customer ID, region code, etc.), and query all valid partition paths to construct a list of partition access paths, and generate the corresponding partition metadata.

[0043] At the same time, according to the meta information of each table and its partition, the system automatically constructs the physical storage path in HDFS, for example

[0044] / warehouse / life_db.db / life_insurance_policy / issue_date=2024-01-01 / , and establishes the corresponding relationship between the library table and the path, and extracts the HDFS path metadata. These path data not only contain the root directory path of the table, but also include all active partition paths, which is convenient for further file-level resource analysis, storage usage statistics and data update frequency monitoring.

[0045] Through this process, the life insurance data platform can fully understand the data distribution, hot and cold partitions, path mapping and storage strategy of each business table.

[0046] S3 determines each table in the target library table one by one to determine whether it is a partitioned table, and if the target library table is a partitioned table, obtains the HDFS path of the root directory of the target library table and the HDFS path of the partition directory of the target library table; if the target library table is not a partitioned table, obtain the HDFS path of the root directory of the target library table.

[0047] Use the Hive metadata database or the Hive DESCRIBE FORMATTED command to check whether the table has partition fields. If partition fields exist, the table is considered a partitioned table; otherwise, it is considered a non-partitioned table. For partitioned tables, first obtain the table's root directory path. The root directory path is the storage root directory for table-level metadata on HDFS. Obtain all partition paths through SHOW PARTITIONS or a metadata database query. Each partition directory is the path where the actual data files are located. By batch accessing, counting, and analyzing these paths, you can obtain the storage characteristics, file count, and update time of each partition. For non-partitioned tables, only the HDFS path of the table's root directory needs to be obtained. All data files are stored in this directory, and there are no partition subdirectories. You can directly perform operations such as data reading and metadata parsing on the files in this directory.

[0048] In one embodiment, in the field of medical health insurance, if the table is a date-partitioned table, such as a claim record table, the system extracts the HDFS path of its root directory and all partition directory paths; if the table is a non-partitioned table, such as a policyholder information table, only the HDFS path of its root directory is extracted.

[0049] In another embodiment, in the life insurance field, the system first identifies each table related to core life insurance business (such as policy_base_info, premium_payment_record, customer_risk_profile, etc.) by accessing the Hive metadata database or parsing DDL metadata to determine whether the table has a partition field. For example, if a table is found to have issue_date or region_code as a partition field, the system determines that it is a partitioned table.

[0050] For partition tables, the system automatically extracts the HDFS path of its root directory (such as

[0051] / warehouse / life_db.db / policy_base_info / ), and enumerate all valid partitions (such as / issue_date=2024-06-01 / , / issue_date=2024-06-02 / , etc.), obtain the partition directory path corresponding to each partition, and build a complete path list; for non-partitioned tables, such as policy_type_dict (policy type dictionary table), the system directly extracts its root directory path.

[0052] S4 performs data analysis on the file metadata according to the HDFS path to obtain table storage data.

[0053] According to the path information recorded in the Hive metabase, enter the HDFS directory corresponding to the target table or partition, scan all files and subdirectories under the directory, identify data files (such as ORC, Parquet, Text, etc.) and invalid files (such as_SUCCESS identification file), and extract the following table data information for each valid file: the latest partition, partition file size, partition file number, whether the partition table, the number of copies, file update time, table owner, table partition number, table file size, and table file number.

[0054] In an embodiment, first, the HDFS path information of medical health insurance related business tables is obtained, such as medical insurance claim record table, chronic disease management tracking table, and customer physical examination report table, and then the corresponding directory is entered to perform file scanning and identification. Subsequently, the metadata of all valid data files under the path is extracted, including file size, update time, number of copies, and file number, and further aggregated to form storage feature data at the table level.

[0055] In another embodiment, in the life insurance field, in order to master the storage of policy, customer, and claim core business data on the distributed file system (HDFS), the system analyzes the file metadata under the corresponding directory based on the obtained HDFS path of the target library table, thereby obtaining the table storage data of each business table. Specifically, the system scans all data files under the root directory and partition path of each business table by programming (such as using Python to call HDFS API or Shell command), and extracts the following key file metadata information: including the size, last modification time, number of copies, owner, and total number of each data file; if the table is a partition table, further statistics of the number of files and file size of each partition are performed.

[0056] For example, in the analysis of policy_claim_record (policy claim record table), the system identifies that it is partitioned by month, and there may be a large number of small files or some empty directories under each partition path. Through the aggregation of file metadata, the system can calculate the total size, number of active partitions, average file size, and small file ratio of the table.

[0057] S5 performs statistical analysis on the table storage data to obtain data storage features and resource consumption indicators.

[0058] The table storage data is sequentially subjected to table granularity storage resource analysis, table file quantity analysis, storage resource compression cleaning, and storage resource early warning analysis. The table granularity storage resource analysis includes counting and evaluating the storage resource usage of the table on the HDFS to obtain the total storage size, the number of partitions, the total number of files, and the partition-level data distribution of the table. The table file quantity analysis analyzes the organization structure of the files in the target library table, identifies and optimizes small files, obtains the number and proportion of small files, the number and proportion of empty files, and the file distribution status: detects whether there are file fragmentation and uneven file size problems. The storage resource compression cleaning includes empty partition cleaning: identifies partition directories without data, judges whether the data no longer in use can be cleaned or archived according to the last data update time, analyzes whether there are data directories with abnormally high number of copies and abnormal file identifiers. The storage resource early warning analysis identifies potential resource risks and provides early warning. The storage space is close to the threshold alarm, the table data volume abnormally expands in a short time, and the short-term frequent repeated writing. Through the above analysis, the total size, the number of partitions, the total number of files, the hot partition and empty partition statistics, and the data skew degree of the target library table are calculated to obtain the data storage characteristics. According to the actual usage amount of the target library table in the table storage data, the data update time, and the active partition proportion, the resource consumption index is obtained. This provides support for subsequent data life cycle management, scheduling strategy optimization, and resource budget decision-making.

[0059] In one embodiment, in the medical health insurance field, by analyzing the total storage size, the number of partitions, and the total number of files of the table, it can be judged whether the table belongs to a table that occupies more storage resources (such as "outpatient record detail table" or "claim record table"), thereby identifying key tables with high-frequency access or high resource occupation. Statistical analysis of hot partitions, empty partition proportions, and data skew degrees in the table can find out whether the data is concentrated in a few partitions, some partitions are not used, or the data distribution is extremely uneven, prompting the need to optimize the partition design.

[0060] In another embodiment, in the field of life insurance, in order to support the stable operation and data optimization of business systems such as policy management, customer actuarial, and claim risk control, the system performs statistical analysis on the data after obtaining the table storage data, identifies the data storage characteristics and resource consumption indicators of each business table, in this embodiment, the system analyzes key business tables such as policy_base_info (policy base information table), customer_profile (customer profile table), and claim_history (historical claim record table), extracts and calculates the following storage characteristic indicators: storage size: statistics of total file size, partition total number and file total number of each table, used to judge whether the table is a "large table", whether it needs to be frequently accessed or archived; data distribution characteristics: identify empty partitions, hot partitions and data skew, for example, the data volume of some month partitions of the claim_history table is abnormally concentrated, which prompts that there may be a partition design problem; file health status: statistics of small file number and empty file proportion, if there are a large number of small files in the policy_renewal_detail table, it is recommended to merge the files to improve the subsequent query performance; storage resource consumption: analyze the real usage of HDFS (including the storage overhead of the replica), evaluate the disk resource occupation situation; data timeliness: judge whether a table is real-time, quasi-real-time or cold data through the last update time of the file and the active partition ratio, to provide support for life cycle management and storage layering; table owner and belonging information: mark the business department and responsible person to which the data belongs, to provide the basis for data cleaning, permission control and archiving.

[0061] Figure 2 The second flowchart of the data analysis method provided by the embodiment of the present application further includes the following steps:

[0062] S6 obtains Hive running log information and extracts log key information, wherein the log key information includes task identification information, execution time information, computing resource usage, input and output data volume, and task running state.

[0063] In the computing resource analysis process, the log scanning program reads the Hive log file in batches, and extracts the key information in the log. The log key information mainly includes the following types: task identification information, execution time information, computing resource usage, input and output data volume, and task running state. When extracting these information, keyword matching method is needed for positioning. For example, by scanning the log lines containing "CPU time spent", "vCore-seconds", "Map output records" and other keywords, the contents related to computing resources are identified and extracted. The extracted log information is unstructured information.

[0064] In an embodiment, in the field of medical health insurance, by obtaining the running log information of Hive in the medical health insurance system, the log content is scanned and analyzed, and information closely related to task execution is extracted, including query ID, Job ID, Stage ID generated for a certain claim data analysis script, total time consumption from task submission to execution end, and detailed time consumption of each Job, Stage and MapReduce stage, recording CPU usage time, vCore-seconds (virtual core seconds), memory resource consumption and other indicators during task execution, reflecting the amount of historical policy data read by the task, the amount of analysis result data output, and the data transmission amount in the intermediate process such as the Shuffle stage, indicating whether the task is successfully executed, and whether there is a failure retry condition.

[0065] In another embodiment, in the field of life insurance, the processing tasks related to massive policy information, customer behavior data, payment records, claim records and other data are executed frequently every day, and the performance and stability of the task execution directly affect the data acquisition efficiency and timeliness of decision-making of the business department. Therefore, in the present embodiment, the system obtains and analyzes the Hive running log information, extracts the log key information therefrom, and lays a foundation for subsequent task performance analysis and resource optimization. The system scans the completed Hive log files in batches every day through an automatic program, and identifies the following five types of log key information: task identification information, including task name, task ID, starting user, submission time, etc., which can accurately locate the business tasks related to life insurance, such as "customer risk scoring model calculation" or "quarterly policy claim report generation", etc.; execution time information, recording the overall running time of the task and the time consumption of each stage, for evaluating whether the data processing meets the business needs in time; calculation resource usage, including CPU usage time, memory occupation, garbage collection time consumption, container allocation, etc., reflecting the resource consumption level of task running; input and output data amount, extracting the data size processed by the task, including the read HDFS input byte number, the result data amount, the shuffle data transmission amount in the Map / Reduce stage, etc.; task running state, indicating whether the task is successfully completed, failed or abnormally terminated, for locating running abnormities and improving data processing stability.

[0066] S7 records the running resource information in the log with the finest granularity according to the log key information.

[0067] According to the above log key information, the running resource usage information recorded in the log is further structured, extracted and organized in a hierarchical manner. Specifically, since the Hive execution mechanism based on MapReduce has a clear hierarchical structure: one Hive query task usually corresponds to one or more Jobs, one Job further contains several Stages, each Stage may contain multiple MapReduce instances (MR), and each MR instance is executed concurrently by multiple TaskAttempts in the cluster. Therefore, when recording the most fine-grained running resource information in the log, the system will be organized and stored in a hierarchical manner of tasks, Jobs, Stages, MRs and TaskAttempts. For example, when parsing a certain Hive log record, the system can identify the total CPU usage time, the Shuffle data volume and the GC time in the Job hierarchy, and further parse the execution details of each of the multiple Stages it contains, such as: Map input data size, Reduce output data size, running node of each TaskAttempt, execution time-consuming, number of failure retries, etc. The information is recorded in a nested manner and the original hierarchical relationship is maintained, ensuring the integrity of the dependent structure between data.

[0068] In an embodiment, the medical health insurance analysis task is composed of multiple jobs, each job includes multiple execution stages, and each stage further contains multiple MapReduce subtasks, which are classified and organized according to the hierarchical structure in the log to form a hierarchical resource information model from tasks to Jobs, and then to Stages and MRs. For example, for a Hive query responsible for calculating the risk of medication for chronic disease users, the system can accurately record the long Map phase time-consuming and large input data volume of a certain Stage, thereby assisting in locating the performance bottleneck.

[0069] In another embodiment, in the field of life insurance, in the face of increasingly complex business scenarios and high-frequency data processing tasks, the overall task execution time alone cannot fully diagnose performance bottlenecks or resource waste problems. Therefore, in this embodiment, the system further records the most granular running resource information in the log based on the extraction of key information of the Hive log, to realize in-depth observation of the task execution process. Specifically, the system combines the Hive running log structure of life insurance related tasks, processes the log content in layers, and records resource information according to the following levels: task level (Task): records the overall resource usage of each task, including CPU total time, memory peak, garbage collection time, startup node, etc.; job level (Job): for multiple Jobs contained in a task, the system records the resource consumption of each Job, such as the number of containers used by a Job in “customer policy full correlation analysis” and the execution time; stage level (Stage): each Stage in a Job is subdivided, and its execution time, shuffle data volume, input and output record number are extracted one by one; MapReduce level (MR): further records the CPU usage, IO operation times, retry times, failure records, etc. of Map and Reduce sub-tasks, accurately reflecting the subtle resource differences in the execution process.

[0070] S8 processes the running resource information to obtain structured running data records of the most granular level.

[0071] By structuring the obtained running resource information, the original log line information is semantically extracted and field mapped to generate running data records with clear field definitions and hierarchical associations. By setting log parsing rules and regular expression libraries, the MR, Stage, Job, Task, etc. key paragraphs in the Hive log are scanned line by line, the parameters and corresponding values contained therein are identified, and field labels are established. For example, when identifying a log line such as “Stage-1:Reducer CPU time:13230ms”, the program parses it as: work stage: Stage-1, work role: Reducer, resource indicator: CPU time, indicator value: 13230ms, and maps the information to a structured data model to form a complete running data record. Each structured running data record contains at least the following fields: task name, task running time, task work stage, work MR parameter, work resource consumption, and file reading parameter. The generated structured records are not only stored in memory data structures (such as DataFrame), but also support export to standard Excel, CSV, etc. format files, facilitating developers to review and analyze, compare historical runs, and automatically generate performance reports.

[0072] In an embodiment, the Hive logs related to the processing of medical insurance data are parsed by a Python program to extract key fields such as the task name (e.g., "intelligent claim settlement preliminary screening calculation"), task running time, identification of each Job and Stage, MR parameter configuration (e.g., number of Mappers / Reducers), resource usage (CPU time, memory occupation, IO read / write volume), and HDFS file reading details (e.g., number of read records, input / output data volume). The parsed information is no longer saved in the form of original log lines, but is converted into structured data records, for example, stored as an Excel table or a database table, with each row representing an MR running unit and attached with its Stage and Job level information.

[0073] In another embodiment, in the life insurance field, to deeply understand the resource consumption of various data processing tasks, identify performance bottlenecks, and realize automatic monitoring and optimization, the system further performs structured processing operations after obtaining and recording the most fine-grained running resource information to generate standardized running data records. In this embodiment, for the core data tasks (e.g., "new single customer policy modeling", "claim fraud behavior detection", "renewal rate analysis", etc.) running every day, the system parses the original resource information lines in the Hive logs and extracts fields through a program to convert unstructured text information into structured table format data. This process includes the following steps: field identification and disassembly: using regular expressions or Hive log parsing rules, the fields related to CPU time consumption, execution stage, Shuffle data volume, container ID, etc. in the logs are identified; hierarchical classification and label tagging: the extracted information is classified according to different levels such as tasks, Jobs, Stages, MapReduce, and at the same time, task names, execution dates, business system labels (e.g., policy, customer, claim, etc.) are added; structured output: the processed data is arranged in table form, such as CSV, Excel or database table, and the fields include but are not limited to: task name (e.g., claim_risk_score_calculation), task running time (start and end time, total time consumption), work stage (Stage ID, Job ID, MR number), resource consumption (CPU time length, memory occupation, Shuffle byte number), IO operation information (read byte number, output record number), task execution node and failure retry situation.

[0074] S9 performs statistical analysis on the running data records to obtain a diagnosis result of the task performance.

[0075] Based on the structured finest granularity computing resource records, the resource consumption of each level in different task execution processes is aggregated and classified. By extracting core indexes such as CPU time consumption, virtual core seconds (vCore-seconds), GC time consumption, shuffle waiting and the like in each task, Job, Stage and MR stage, the CPU consumption and resource utilization of a single task are counted at different levels, which assists in determining whether the task has problems such as uneven overall resource utilization and excessive use. The resource proportion and execution time difference of each Job under the task are identified to find out whether there is an abnormally long execution time or a Job node with excessively high CPU consumption. The CPU time consumption distribution between Stages is analyzed in the DAG execution graph to diagnose performance bottlenecks in specific stages, which facilitates in-depth optimization by the optimizer or the developer. The resource usage proportion of the Map stage and the Reduce stage is calculated to identify typical performance bottleneck scenarios such as data skew, uneven shuffle and excessive Reduce. The execution unit of a specific MapTask / ReduceTask is further analyzed to identify abnormal behaviors of some execution threads, such as abnormal CPU occupation of a single MR node and excessively long deserialization time.

[0076] In an embodiment, the overall CPU usage of a certain insurance policy claim report task is counted, and the CPU time consumption of each Job is compared to further analyze whether a performance bottleneck occurs in the Map stage or the Reduce stage. If it is found that the CPU consumption of a certain Stage is abnormally high, it may indicate problems such as data skew, unreasonable Join operation or unpartitioned table. Through this fine-grained performance diagnosis method, the medical insurance institution can accurately locate the performance bottleneck of the Hive task, provide technical basis for ETL scheduling optimization, SQL statement rewriting and resource scheduling strategy adjustment, and thus improve the overall data processing efficiency and system stability.

[0077] In another embodiment, in the field of life insurance, data tasks usually involve complex calculations such as customer accurate portrait, policy risk pricing, claim abnormality monitoring, marketing crowd mining, etc. To ensure the stable operation of the business system and the timely delivery of decision-making data, the embodiment performs multi-level statistical analysis on the execution of data tasks based on structured operation data records, thereby obtaining fine-grained diagnostic results of task performance. Specifically, the system first performs aggregate analysis on structured operation resource records according to different levels such as task, job, stage, MapReduce subtask (MR), etc. Task overall performance evaluation: statistics of CPU total consumption time, average container utilization rate, task duration, etc. of each task to determine whether there is resource waste or execution timeout; Job level analysis: comparison of each Job within the task to identify which Job has abnormal time consumption and uneven resource allocation, such as a Job in the "customer claim scoring calculation" task that occupies a large amount of CPU for a long time, which may be due to data skew or unreasonable join design; Stage level bottleneck identification: analysis of the execution time and resource utilization distribution of the Stage phase, for example, in the "high net worth customer life cycle evaluation" task, it is found that the Shuffle data volume of Stage 4 is much higher than that of other stages, which may be a performance bottleneck; MR granularity problem positioning: further to the Map / Reduce task, check whether there are Container allocation failure, repeated execution, Shuffle phase retransmission, etc. to locate the specific resource abnormal point.

[0078] Through the above analysis, a complete task performance diagnostic report can be formed to find performance abnormalities and make tuning suggestions. For example, it is found that the "renewal customer credit scoring model" task repeatedly retries due to CPU saturation in the Map phase, which can be solved by adjusting the concurrency or optimizing the data skew problem. This helps insurance companies to realize the observability, quantifiability and traceability of task performance, greatly improving the efficiency and stability of the data platform.

[0079] It should be understood that the size of the serial number of each step in the above embodiment does not mean the order of execution, and the execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present application.

[0080] In an embodiment, a data parsing device 100 is provided, which corresponds to the data parsing method in the above embodiment. As shown in Figure 3 , Figure 3 is a structural schematic diagram of the data parsing device in an embodiment of the present application, and the detailed description of each functional module is as follows:

[0081] The library table information extraction module 101 is configured to extract library table information in the Hive operation log information, and obtain target library table list data and log key information.

[0082] The HDFS path acquisition module 102 is configured to parse the target library table list data, obtain partition metadata and HDFS path metadata, determine each table in the target library table one by one, determine whether the table is partitioned, if the target library table is a partitioned table, acquire the HDFS path of the root directory of the target library table and acquire the HDFS path of the partition directory of the target library table, and if the target library table is not a partitioned table, acquire the HDFS path of the root directory of the target library table.

[0083] The storage data acquisition module 103 is configured to analyze file metadata based on the HDFS path, and obtain table storage data.

[0084] The operation data recording module 104 is configured to record the most fine-grained operation resource information in the log based on the log key information, and obtain structured most fine-grained operation data records.

[0085] The statistical analysis module 105 is configured to statistically analyze the table storage data and the operation data records, and obtain fine-grained diagnosis results of data storage characteristics, resource consumption indexes and task performance.

[0086] In an embodiment, the library table information extraction module 101 is specifically configured to:

[0087] The log source can include an execution log of HiveServer2, a task log of YARN or an execution log of MR / Tez, etc. For storage resources, in the log analysis process, the library name and table name information appearing in the Hive script are extracted by analyzing SQL execution statements (such as SELECT, INSERT, CREATE, DROP, etc.) in the log, using regular expressions, AST (Abstract Syntax Tree) parsing or a parser based on lexical / syntactic rules. Then, for each log file, the program can identify one or more involved Hive tables from the log file, and these tables constitute a target library table set involved in the task. Through this method, originally unordered, loose and unstructured log text data is converted into a standardized "target table list" structure, which facilitates subsequent unified table metadata analysis, HDFS path analysis, file resource statistics and the like. The generated "target library table list data" serves as an input for subsequent data analysis and performance analysis.

[0088] For computing resources, the Hive log files are read in bulk by the log scanner program, and key information in the logs is extracted. The log key information mainly includes the following categories: task identification information, execution time information, computing resource usage, input and output data volume, and task running status. When extracting this information, keyword matching is required to locate it. For example, by scanning log lines containing "CPU time spent", "vCore-seconds", "Map output records", etc. The content related to computing resources is identified and extracted. The extracted log information is unstructured information.

[0089] The HDFS path acquisition module 102 is specifically configured to:

[0090] After completing the Hive log analysis and extracting the target library table list, the system performs a metadata collection process for each target table. The Python program obtains the structure definition information of the target table through Hive SQL or directly connects the Hive metadata database, including table type (external table / internal table), field structure, partition field list, storage format, compression method, and table physical storage location, etc.

[0091] Subsequently, classification processing is performed according to whether it is a partition table. For partition tables, the program further queries all existing partitions under the table, extracts the specific path information, partition key value, and creation time of each partition, and constructs an access list of the partition path. This list constitutes the partition metadata of the table, including: partition field and its enumeration value, full path of each partition path on HDFS, partition creation or latest update time, and zone data size, etc.

[0092] The storage data acquisition module 103 is specifically configured to:

[0093] The Hive metadata database or the Hive DESCRIBE FORMATTED command is used to check whether the table has a partition field; if there is a partition field, it is determined to be a partition table; otherwise, it is considered to be a non-partition table. For partition tables, the root directory path of the table is first obtained, which is the storage root directory of the table-level metadata on HDFS. All partition paths are obtained through SHOW PARTITIONS or metadata database queries, and each partition directory is the path of the actual data file. Batch access, statistics, and analysis of these paths can obtain the storage characteristics, file quantity, and update time of each partition. For non-partition tables, only the root directory HDFS path of the table needs to be obtained, all data files are stored in this directory, and there is no partition subdirectory. Directly read the data files in this directory, and perform metadata analysis and other operations.

[0094] The running data recording module 104 is specifically configured to:

[0095] According to the above log key information, the running resource usage information recorded in the log is further structured and organized in a hierarchical manner. Specifically, since the Hive execution mechanism based on MapReduce has a clear hierarchical structure: one Hive query task usually corresponds to one or more Jobs, one Job further contains several Stages, and each Stage may contain multiple MapReduce instances (MR), and each MR instance is executed concurrently by multiple TaskAttempts in the cluster. Therefore, when recording the most fine-grained running resource information in the log, the system will organize and store it in a hierarchical manner of tasks, Jobs, Stages, MRs, and TaskAttempts. For example, when parsing a certain Hive log record, the system can identify the total CPU usage time, Shuffle data volume, and GC time in the Job hierarchy, and further parse the execution details of each of the multiple Stages it contains, such as: Map input data size, Reduce output data size, running node of each TaskAttempt, execution time consumption, number of failure retries, etc. The information is recorded in a nested manner and maintains the original hierarchical relationship, ensuring the integrity of the dependency structure between data.

[0096] The statistical analysis module 105 is specifically configured to:

[0097] The table storage data is sequentially subjected to table granularity storage resource analysis, table file quantity analysis, storage resource compression cleaning, and storage resource early warning analysis. The table granularity storage resource analysis includes statistical and evaluation of the storage resource usage of the table on the HDFS to obtain the total storage size of the table, the number of partitions, the total number of files, and the partition-level data distribution. The table file quantity analysis analyzes the organization structure of the files in the target library table, identifies and optimizes small files, obtains the number and proportion of small files, the number and proportion of empty files, and the file distribution: detects whether there are file fragmentation, uneven file size, etc. The storage resource compression cleaning includes empty partition cleaning: identifies partition directories without data, judges whether the data no longer used can be cleaned or archived according to the last update time of the data, analyzes whether there are data directories with abnormally high number of copies and abnormal file identifiers. The storage resource early warning analysis identifies potential resource risks and gives early warning. The storage space is close to the threshold value alarm, the table data volume abnormally expands in a short time, and the short-term frequent repeated writing. Through the above analysis, the total size, the number of partitions, the total number of files, the hot partition and empty partition statistics, and the data skew degree of the target library table are calculated to obtain the data storage characteristics.

[0098] Based on the structured finest granularity computing resource record, the resource consumption of each level in different task execution processes is aggregated and classified, core indexes such as CPU time consumption, virtual core seconds (vCore-seconds), GC time consumption, shuffle waiting and the like in each task, Job, Stage, MR stage are extracted, the CPU consumption and resource utilization of the whole single task are counted at the level, and the problems such as uneven resource utilization and excessive use of the whole task are assisted to be judged; the resource proportion and execution time difference of each Job under the task are identified, and whether there is an abnormally long execution time or a Job node with excessively high CPU consumption is found; the CPU time consumption distribution between Stages in the DAG execution graph is analyzed, and the performance bottleneck in which Stage appears is diagnosed, so as to facilitate the optimizer or the developer to deeply optimize; the resource usage proportion of the Map stage and the Reduce stage is calculated, and typical performance bottleneck scenarios such as data skew, uneven shuffle and excessive Reduce are identified; the specific MapTask / ReduceTask execution unit is further reached, and the abnormal behavior of some execution threads is identified, such as abnormal CPU occupation of a single MR node and excessively long deserialization time and the like.

[0099] The data analysis device provided by the application can automatically identify and extract target library table information and task log key information in the Hive operation log through the "library table information extraction module", avoid the uncertainty of traditional manual positioning of log content, and improve the efficiency and accuracy of log information processing. Through the "HDFS path acquisition module", the system can accurately determine whether each table is a partition table, and further acquire the HDFS path of the root directory and the partition directory, so as to realize fine data statistics according to the table granularity and the partition granularity in subsequent analysis, which is beneficial to finding problems such as storage redundancy, data skew and empty partition, supporting compression and cleaning of storage resources and early warning control. The storage data acquisition module combines the HDFS path to automatically analyze file metadata such as file size, number and update time, generate storage characteristic indexes at the table level, and assist in formulating data life cycle management and cold-temperature-hot data layering strategies. Based on the log key information, the finest granularity level (such as Task-Job-Stage-MR structure) of task operation is parsed and reconstructed, so that the resource usage information at each level is structured and saved, facilitating multi-dimensional performance analysis and bottleneck positioning. The table storage data and operation resource record are systematically analyzed, and data storage characteristics (such as table size, file number and partition distribution), resource consumption indexes (such as HDFS usage rate and active partition proportion) and task performance diagnosis (such as Map / Reduce time consumption ratio and CPU bottleneck identification) are output, thereby providing data support basis for data development and platform operation and maintenance.

[0100] The specific definition of the data analysis device can refer to the definition of the data analysis method in the foregoing, and will not be described here. Each module in the data analysis device can be realized by software, hardware, and a combination thereof, in whole or in part. The above-mentioned modules can be embedded in or independent of the processor in the computer device in hardware form, or can be stored in the memory in the computer device in software form, so that the processor calls and executes the operations corresponding to each of the above-mentioned modules.

[0101] In one embodiment, a computer device is provided, which can be a server, and an internal structure diagram thereof can be as shown in Figure 4 Figure 4 is a structural schematic diagram of a computer device in an embodiment of the present application. The computer device includes a processor, a memory, a network interface, and a database connected through a system bus. The processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for running the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is configured to communicate with an external client through a network connection. The computer program, when executed by the processor, implements the functions or steps of a data analysis method on the server side.

[0102] In one embodiment, a computer device is provided, which can be a client, and an internal structure diagram thereof can be as shown in Figure 5 Figure 5 is another structural schematic diagram of a computer device in an embodiment of the present application. The computer device includes a processor, a memory, a network interface, a display screen, and an input device connected through a system bus. The processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for running the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is configured to communicate with an external server through a network connection. The computer program, when executed by the processor, implements the functions or steps of a data analysis method on the client side.

[0103] In one embodiment, a computer device is provided, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the following steps when executing the computer program:

[0104] obtaining Hive running log information, locating a target library table, and extracting library table information in the Hive running log information to obtain target library table list data; ​​

[0105] analyzing the target library table list data to obtain partition metadata and HDFS path metadata;

[0106] judging each table in the target library table one by one to determine whether the table is partitioned, if the target library table is partitioned, obtaining the HDFS path of the root directory of the target library table and obtaining the HDFS path of the partition directory of the target library table; if the target library table is not partitioned, obtaining the HDFS path of the root directory of the target library table;

[0107] performing data analysis on file metadata according to the HDFS path to obtain table storage data;

[0108] performing statistical analysis on the table storage data to obtain data storage characteristics and resource consumption indicators;

[0109] obtaining Hive running log information, extracting log key information, the log key information including task identification information, execution time information, computing resource usage, input and output data volume and task running state;

[0110] According to the log key information, record the most fine-grained running resource information in the log;

[0111] processing the running resource information to obtain structured most fine-grained running data records;

[0112] performing statistical analysis on the running data records to obtain a diagnosis result of task performance.

[0113] It should be noted that the above functions or steps that the computer readable storage medium or computer device can implement can correspond to the related description of the server side and the client side in the foregoing method embodiments, and to avoid repetition, they will not be described one by one here.

[0114] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer readable storage medium, and when executed, can include the processes of the above-mentioned embodiment methods. Any reference to memory, storage, database or other medium used in the embodiments provided by the present application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration but not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0115] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the above-mentioned division of functional units and modules is exemplified. In actual application, the above-mentioned functions can be completed by different functional units and modules according to needs, that is, the internal structure of the device is divided into different functional units or modules to complete all or part of the above-described functions.

[0116] The above-mentioned embodiments are only used to illustrate the technical solutions of the present application, but not limit it. Although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that they can modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for part of the technical features. The modification or replacement does not make the essence of the corresponding technical solution deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be included in the protection scope of the present application.

Claims

1. A data analysis method, characterized in that: include: Obtain Hive operation log information, locate the target library table and extract the library table information in the Hive operation log information to obtain the target library table list data; Analyze the target library table list data to obtain partition metadata and HDFS path metadata; Determine each table in the target library table one by one to determine whether it is a partitioned table. If the target library table is a partitioned table, obtain the HDFS path of the target library table's root directory and the HDFS path of the target library table's partition directory; if the target library table is not a partitioned table, obtain the HDFS path of the target library table's root directory; Perform data analysis on the file metadata according to the HDFS path to obtain table storage data; Statistical analysis is performed on the table storage data to obtain data storage characteristics and resource consumption indicators.

2. The data analysis method according to claim 1, characterized in that: Also includes: Obtain Hive operation log information and extract key log information, including task identification information, execution duration information, computing resource usage, input and output data volume, and task operation status; Record the most fine-grained operating resource information in the log according to the key log information; Processing the operation resource information to obtain structured and fine-grained operation data records; Statistical analysis is performed on the operation data records to obtain diagnostic results of task performance.

3. The data analysis method according to claim 1, characterized in that: The target library table list data is analyzed to obtain partition metadata and HDFS path metadata; including: Obtain the table structure of the target library table and perform partition field identification, query all partition paths of the target library table and build an access list to obtain partition metadata of the target library table; Construct an HDFS path for the target library table, establish a mapping relationship between the target library table and the HDFS path, and obtain the HDFS path metadata.

4. The data analysis method according to claim 3, characterized in that: The statistical analysis of the table storage data to obtain data storage characteristics and resource consumption indicators includes: The total size, number of partitions, total number of files, statistics of hot partitions and empty partitions, and degree of data skew of the target library table are calculated based on the table storage data to obtain data storage characteristics; the actual HDFS usage, data update time, and active partition ratio of the target library table are calculated based on the table storage data to obtain resource consumption indicators.

5. The data analysis method according to claim 2, characterized in that: The recording of the finest-grained operating resource information in the log according to the key log information includes: Based on the key log information, record the total CPU time, garbage collection time, virtual core seconds and container node, file parsing and deserialization overhead, file reading data scanning amount, CPU usage during the file reading process, and the number of retries for file processing failures for a single task in the Hive run log.

6. The data analysis method according to claim 5, characterized in that: The processing of the operation resource information to obtain structured and finest-grained operation data records includes: The extracted operation resource information is used to generate structured operation data records, and the operation data records are stored in a structured manner.

7. The data analysis method according to claim 6, characterized in that: The statistical analysis of the operation data records to obtain fine-grained diagnostic results of task performance includes: Record and analyze the CPU time consumption and resource utilization of individual tasks in the running data records, calculate the overall CPU consumption of each task, analyze the CPU time consumption distribution between DAG execution stages, compare the CPU computing resource ratio of the Map / Reduce stage, and obtain fine-grained diagnostic results of task performance.

8. A data analysis device, characterized in that: include: The library table information extraction module is used to extract the library table information in the Hive operation log information and obtain the target library table list data and log key information; An HDFS path acquisition module is used to parse the target library table list data to obtain partition metadata and HDFS path metadata, and to determine whether each table in the target library table is a partitioned table. If the target library table is a partitioned table, the HDFS path of the target library table's root directory and the HDFS path of the target library table's partition directory are obtained; if the target library table is not a partitioned table, the HDFS path of the target library table's root directory is obtained; The storage data acquisition module is used to perform data analysis on the file metadata in the HDFS path to obtain table storage data; An operation data recording module, configured to record the finest-grained operation resource information in the log according to the key information of the log, and obtain a structured finest-grained operation data record; The statistical analysis module is used to perform statistical analysis on the table storage data and the operation data records to obtain fine-grained diagnostic results of data storage characteristics, resource consumption indicators and task performance.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the steps of the data analysis method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the data analysis method according to any one of claims 1 to 7 are implemented.

Images