Automatic extraction method and device of test data set for data analysis problem

Abstract

The embodiment of the application provides a kind of automatic extraction method and device of test data set for data analysis problem, on the basis of establishing equipment life cycle data, formal definition is carried out to the analysis object of data analysis problem, problem characteristic, problem type with problem as orientation, the correlation degree of analysis object and problem characteristic is selected in different dimensions, according to the mapping of dimension feature and original data set, search the table, column of original data set, according to analysis data type, data analysis role definition, dimension definition is converted, with dimension feature as connection, with problem object as data scale fusion, generate preliminary data set;According to business definition, select different dimension range, select range to data, form analysis data set;Finally, from data set quality and data structure, check and evaluate, so that more comprehensive, more accurate data analysis result can be realized.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of automatic extraction of data sets, in particular to a method and device for automatic extraction of test data sets for data analysis problems. BACKGROUND

[0002] The current big data analysis application has the problem of separation of business field and analysis algorithm. The business field personnel lack data analysis means and corresponding knowledge, and lack of physical model, storage and data processing means of data. Algorithm personnel need to spend a lot of effort to learn field knowledge and understand data collection. Moreover, the original data collection and storage are distributed in each system and maintained by IT personnel. The existing analysis data set is generated by splicing the basic data table through the understanding and subjective judgment of business analysis personnel, and highly depends on expert experience. Therefore, there are problems such as unclear definition of data analysis problem, large amount of time and low efficiency of original data analysis and data processing, and poor algorithm effect.

[0003] The abstraction of analysis problem is the basis of problem analysis and the direction of solution, which needs to convert the business problem into formal description. However, the field personnel lack corresponding methods and means, and cannot realize formal definition and conversion of analysis problem to data set. The business problem is converted into data processing problem. SUMMARY

[0004] In order to solve one of the above technical defects, the present application provides a method and device for automatic extraction of test data sets for data analysis problems.

[0005] According to a first aspect of the present application, a method for automatic extraction of test data sets for data analysis problems is provided, which comprises:

[0006] Collecting the full life cycle data of equipment, determining the analysis object according to the full life cycle data, and establishing the original data set, the original data model and the index interrelation knowledge model;

[0007] Formalizing the definition of data analysis problem according to the analysis object, the original data set, the original data model and the index interrelation knowledge model, and forming the knowledge graph based on data analysis problem;

[0008] Selecting the dimension feature of test data set according to the correlation degree of data analysis problem and different dimension features and the knowledge graph;

[0009] Converting the data type and data role of original data set according to the dimension feature of test data set, and fusing the converted data type and data role to form the preliminary data set;

[0010] range selection is performed on the preliminary data set to form an analysis data set;

[0011] data set quality and data structure checking and evaluation are performed on the analysis data set to obtain an evaluation result.

[0012] According to a second aspect of the embodiments of the present application, an automatic extraction device for a test data set of a data analysis problem is provided, and the device comprises a processor configured with processor-executable operation instructions to perform the following operations:

[0013] Full life cycle data of equipment is collected, an analysis object is determined according to the full life cycle data, and an original data set, an original data model and an index interrelation knowledge model are established;

[0014] The data analysis problem is formally defined based on the analysis object, the original data set, the original data model and the index interrelation knowledge model to form a knowledge graph based on the data analysis problem;

[0015] Dimension feature selection of the test data set is performed according to the correlation degree of the data analysis problem and different dimension features and the knowledge graph;

[0016] Data type and data analysis role conversion of the original data set is performed according to the dimension features of the test data set, and the converted data type and data role are fused to form a preliminary data set;

[0017] Range selection is performed on the preliminary data set to form an analysis data set;

[0018] Data set quality and data structure checking and evaluation are performed on the analysis data set to obtain an evaluation result.

[0019] By using the automatic extraction method for a test data set of a data analysis problem provided in the embodiments of the present application, on the basis of establishing full life cycle data of equipment, the analysis object, problem characteristics and problem type of the data analysis problem are formally defined with the problem as a pointing direction, the dimension features are selected according to the correlation degree of the analysis object and the problem characteristics in different dimensions, the tables and columns of the original data set are searched according to the mapping of the dimension features and the original data set, the conversion is performed according to the analysis data type, data analysis role definition and dimension definition, the fusion is performed with the dimension features as a connection and the problem object as a data scale to generate a preliminary data set; according to the business definition, different dimension ranges are selected to perform range selection on the data to form an analysis data set; finally, the data set quality and data structure are checked and evaluated, so that more comprehensive and more accurate data analysis results can be achieved. BRIEF DESCRIPTION OF DRAWINGS

[0020] The accompanying drawings, which are included to provide a further understanding of the application and are incorporated in and constitute a part of this application, illustrate embodiments of the application and together with the description serve to explain the application. In the drawings:

[0021] Figure 1 A flowchart of an automatic extraction method of a test data set for a data analysis problem provided in Embodiment 1 of the present application;

[0022] Figure 2 A knowledge graph of a train traction braking system provided in Embodiment 1 of the present application. DETAILED DESCRIPTION

[0023] In order to make the technical solutions and advantages in the embodiments of the present application clearer, the exemplary embodiments of the present application are further described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present application, and are not exhaustive of all embodiments. It should be noted that the embodiments in the present application and the features in the embodiments can be combined with each other without conflict.

[0024] Embodiment 1

[0025] As shown in the present embodiment, an automatic extraction method of a test data set for a data analysis problem is provided, which comprises: Figure 1

[0026] S101, collecting full life cycle data of a device, determining an analysis object according to the full life cycle data, and establishing an original data set, an original data model and an index interrelation knowledge model.

[0027] Specifically, taking the analysis problem of energy signal transmission in the process of rail train traction braking as an example, the present embodiment first collects data of different life states of the rail train traction braking device, including different stages such as design, process processing, fault, maintenance, scrap, etc. Then, based on the collected data, the original data set, the original data model, the index interrelation knowledge model of the full life cycle of the rail train traction braking system and the determination of the analysis object are formed.

[0028] S102, formally defining the data analysis problem according to the analysis object, the original data set, the original data model and the index interrelation knowledge model, and forming a knowledge graph based on the data analysis problem.

[0029] ​Specifically, in this embodiment, the data analysis problem includes subject (analysis object), performance index, environmental feature, time and place range, other factors and the like. The business problem type mainly faces health assessment, fault diagnosis, prediction. The types of exploratory analysis for realizing business problem analysis include basic exploratory analysis types such as anomaly detection, correlation factor, trend analysis. The analysis object can be a system (system set), an alarm (multiple associated alarms) or a state; the analysis object should be derived from the analysis object interface, corresponding to the subject or action and the like in the whole life cycle of the equipment. From the business analysis of the train traction braking system, the analysis object can be all entities on the business process, such as traction system, brake, wheel set, wheel cake and the like, or can be actions in the business process, such as braking, traction, wear and the like.

[0030] This embodiment formulates the analysis problem based on the analysis object, including performance index, environmental feature, time and place range, other factors and the like. Based on the analysis problem of train traction braking transmission, the knowledge graph of the train traction braking system is established, as shown in Figure 2 .

[0031] S103, selecting the dimension feature of the test data set according to the correlation degree of the data analysis problem and different dimension features and the knowledge graph.

[0032] Specifically, according to the selected certain data analysis problem, the target feature is selected by combining mechanism analysis and graph retrieval. First, according to the data analysis problem, the data table column connection is carried out according to the consistency principle of the analysis object and the dimension feature, and the analysis object in the original data set is searched according to the analysis object. Then, according to the equipment performance, the fact table column is determined, and the factors associated with the equipment performance in the knowledge graph are searched. Then, the environment feature or the dimension feature is searched in the fact table column to obtain the position of the environment feature or the dimension feature in the fact table column and the data acquisition frequency. Finally, the dimension feature of the test data set is selected according to the position of the environment feature or the dimension feature in the fact table column and the data acquisition frequency. The dimension feature of the test data set should be able to explain the behavior or state change of the analysis object in the data analysis problem for a period of time, and reflect the environment or influencing factors of the equipment. For the process of traction and braking of the railway train, the wear of the wheel set and wheel cake in the train operation can be selected as the data analysis problem, and the selected indexes from the knowledge graph include time, line, train, car number, position number, wheel diameter value, wheel rim height, wheel rim thickness and maintenance cycle and the like.

[0033] S104, converting the data type and data analysis role of the original data set according to the dimension feature of the test data set, and fusing the converted data type and data role to form a preliminary data set.

[0034] Specifically, in the embodiment, the data types include continuous value (Continuous), discrete value (Nominal), string (String), and date time (Datetime); and the data analysis roles include features, targets, and meta. The original data set is identified by a label system, and the dimension features are identified. In wheel wear analysis, the line, train, car number, and position number are usually regarded as dimension features, and the data analysis roles are the analysis objects. The data roles of the flange height, flange thickness, and wheel diameter are features, and the data types are continuous values. The maintenance period is a target, and the data type is a discrete value.

[0035] The analysis objects and the dimension features of the test data set are used as the connection means, the data collection frequency is used as the base point, the data in each dimension or the environmental information table in the original data set is scaled or expanded according to the business meaning, the original data set, the original data model, and the index interrelation knowledge model are used to realize the fusion of multi-scale data. Corresponding to wheel wear analysis, the size information of the train wheel set is usually located in the maintenance record table, the wheel set configuration information is usually located in the train configuration table, and the maintenance period is recorded in the maintenance record table of the train. In the embodiment, different data need to be fused, the standard size value is used as the main parameter, and other features are scaled and expanded to form a preliminary data set for data analysis.

[0036] S105, range selection is performed on the preliminary data set to form an analysis data set.

[0037] Specifically, in the embodiment, the range selection of the test data set is performed on a certain dimension feature, and the working condition is segmented according to the working condition. For example, when analyzing the wear problem of the wheel set, the same car of different trains or the same side wheel set of different trains is analyzed, and a better analysis result is often obtained. Therefore, when the data set is screened, the position dimension is considered to be more beneficial to the establishment of the analysis data set of the wheel set. When the wheel set maintenance is analyzed, the time dimension of the wheel set is considered, the entire maintenance period of the wheel set system is used as the unit length of data segmentation, the data set after screening can reflect the behavior or state change of the wheel set size in a maintenance period, and the change trend is reflected, so that the data set can be directly applied to algorithm analysis.

[0038] S106, the data set quality and the data structure of the analysis data set are checked and evaluated to obtain an evaluation result.

[0039] Specifically, the embodiment forms a full assessment of the data set from two aspects of data set quality and data structure. The data structure is checked and described by data set evaluation indicators. According to the characteristics of the data set, it can be divided into single variable, multivariate and time series data set, among which the time series data set is the most complex. From different angles, such as the position characteristic evaluation indicators of the data set, such as median, mean, mode, trend characteristic evaluation indicators, such as slope, distribution characteristics, such as normal distribution, t distribution, randomness evaluation, dispersion evaluation, etc., the data set is evaluated from different angles. The data set quality check mainly evaluates the quality characteristics of the data set, and the main indicators are abnormal value, missing value, consistency, etc. If there are many abnormal values and missing values in the data set, the next step of data analysis can be processed from this aspect.

[0040] Embodiment 2

[0041] Corresponding to embodiment 1, the embodiment provides an automatic extraction device for test data set of data analysis problem, the device comprises a processor, the processor is configured with processor executable operation instruction to execute the following operations:

[0042] Collecting the full life cycle data of the equipment, determining the analysis object according to the full life cycle data, and establishing the original data set, the original data model and the index mutual relationship knowledge model;

[0043] Formal definition of data analysis problem based on the analysis object, the original data set, the original data model and the index mutual relationship knowledge model, forming a knowledge graph based on data analysis problem;

[0044] According to the correlation degree of data analysis problem and different dimension characteristics and the knowledge graph, the dimension characteristics of the test data set are selected;

[0045] According to the dimension characteristics of the test data set, the data type and the data analysis role of the original data set are converted, and the converted data type and data role are fused to form a preliminary data set;

[0046] Range selection is performed on the preliminary data set to form an analysis data set;

[0047] The analysis data set is checked and evaluated from the aspects of data set quality and data structure, and the evaluation result is obtained.

[0048] The principle of the automatic extraction device for test data set of data analysis problem provided in the embodiment can refer to the content described in embodiment 1, and the embodiment will not be described here.

[0049] On the basis of establishing the device life cycle data, the embodiment formalizes the analysis object, problem characteristics and problem type of data analysis problem by taking the problem as the orientation, selects the dimension characteristics through the correlation degree of the analysis object and the problem characteristics in different dimensions, searches the table and column of the original data set according to the mapping of the dimension characteristics and the original data set, converts according to the analysis data type, data analysis role definition and dimension definition, fuses with the dimension characteristics as the connection and the problem object as the data scale to generate the preliminary data set, selects the data range according to the business definition to form the analysis data set, and finally checks and evaluates from the data set quality and the data structure, so that more comprehensive and more accurate data analysis results can be realized.

[0050] Those skilled in the art should understand that the embodiments of the present application can be provided as a method, a system, or a computer program product. Therefore, the present application can take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROMs, optical storage, etc.) containing computer-usable program code. The solutions in the embodiments of the present application can be implemented in various computer languages, such as the object-oriented programming language Java and the interpreted scripting language JavaScript.

[0051] The present application is described with reference to flowcharts and / or block diagrams according to the methods, devices (systems), and computer program products of the embodiments of the present application. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, and the combination of flows and / or blocks in the flowcharts and / or block diagrams can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing apparatus to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing apparatus produce a device that implements the functions specified in the flowcharts and / or block diagrams. Figure 1 one or more flows and / or blocks Figure 1 an apparatus that carries out the functions specified in the flowcharts and / or block diagrams.

[0052] These computer program instructions can also be stored in a computer-readable memory that can direct the computer or other programmable data processing apparatus to work in a specific manner, so that the instructions stored in the computer-readable memory produce a product including instruction apparatus, which implements the functions specified in the flowcharts and / or block diagrams. Figure 1 one or more flows and / or blocks Figure 1 an apparatus that carries out the functions specified in the flowcharts and / or block diagrams.

[0053] These computer program instructions can also be loaded into a computer or other programmable data processing devices, so that a series of operation steps are performed on the computer or other programmable devices to generate computer-implemented processes, thus the instructions executed on the computer or other programmable devices provide the function of implementing the processes specified in the flowcharts Figure 1 one or more flows and / or blocks Figure 1 one or more flows and / or blocks

[0054] In the description of the present application, it needs to be understood that the terms "center", "longitudinal", "lateral", "length", "width", "thickness", "upper", "lower", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer" and the like indicate the orientation or positional relationship shown in the drawings, which are only for the purpose of facilitating the description of the present application and simplifying the description, and do not indicate or imply that the device or element referred to must have a particular orientation, be constructed and operated in a particular orientation, and therefore cannot be understood as a limitation to the present application.

[0055] In addition, the terms "first", "second" are only for descriptive purpose, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the technical features indicated. Therefore, the features defined with "first", "second" can explicitly or implicitly include one or more of the features. In the description of the present application, the meaning of "plurality" is at least two, such as two, three, etc., unless otherwise explicitly specified and limited.

[0056] In the present application, unless otherwise explicitly specified and limited, the terms "mounting", "connection", "connecting", "fixing" and the like should be understood in a broad sense, for example, can be fixed connection, can also be detachable connection, or integral; can be mechanical connection, can also be electrical connection or can communicate with each other; can be directly connected, can also be indirectly connected through intermediate medium, can be the internal communication of two elements or the interaction relationship between two elements. For those skilled in the art, the specific meaning of the above terms in the present application can be understood according to the specific circumstances.

[0057] Although the preferred embodiments of the present application have been described, those skilled in the art can make further changes and modifications to these embodiments once they know the basic inventive concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications falling within the scope of the present application.

[0058] Obviously, those skilled in the art can make various modifications and variations to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalent technologies, the present application also intends to include these modifications and variations.

Claims

1. A method for automatically extracting test data sets for data analysis problems, characterized in that: The method comprises: Collecting full life cycle data of the equipment, determining analysis objects based on the full life cycle data, and establishing original data sets, original data models, and indicator interrelationship knowledge models; Formalizing the data analysis problem based on the analysis object, original data set, original data model, and indicator relationship knowledge model to form a knowledge graph based on the data analysis problem; Selecting dimensional features of the test data set based on the degree of relevance between the data analysis problem and different dimensional features, including: connecting data table columns based on the consistency principle of the analysis object and the dimensional features according to the data analysis problem, and searching for the analysis object in the original data set based on the analysis object; determining fact table columns based on device performance, and searching the knowledge graph for factors associated with the device performance; searching the fact table columns based on environmental features or dimensional features, and obtaining the location and data collection frequency of the fact table columns where the environmental features or dimensional features are located; selecting dimensional features of the test data set based on the location and data collection frequency of the fact table columns where the environmental features or dimensional features are located; Converting the data type and data analysis role of the original data set according to the dimensional characteristics of the test data set, and fusing the converted data types and data roles to form a preliminary data set; performing range selection on the preliminary data set to form an analysis data set; The analysis data set is verified and evaluated for data set quality and data structure to obtain an evaluation result.

2. The method according to claim 1, characterized in that The process of converting the data type and data analysis role of the original data set according to the dimensional characteristics of the test data set, and fusing the converted data type and data role to form a preliminary data set includes: Convert the data type and data analysis role of the original dataset according to the dimensional characteristics of the test dataset; Using the dimensional characteristics of the analysis object and the test data set as the connection means, and the data collection frequency as the starting point, the data in the original data set is shrunk and expanded according to the business meaning to form a preliminary data set.

3. The method according to claim 1, characterized in that The process of performing range selection on the preliminary data set to form an analysis data set includes: selecting a screening dimension, and screening the data in the preliminary data set according to the screening dimension; The filtered data is segmented according to the working conditions to form an analysis data set.

4. The method according to claim 1, wherein The process of verifying and evaluating the quality and data structure of the analysis data set and obtaining the evaluation result includes: Verifying and evaluating the quality of the analysis dataset according to the dataset quality characteristic indicators to obtain a quality evaluation result; Verifying and evaluating the data structure of the analysis data set according to the data set evaluation index to obtain a data structure evaluation result; An evaluation result is obtained according to the quality evaluation result and the data structure evaluation result.

5. An automatic extraction device for test data sets for data analysis problems, characterized in that: The apparatus includes a processor configured with processor-executable operation instructions to perform the following operations: Collecting full life cycle data of the equipment, determining analysis objects based on the full life cycle data, and establishing original data sets, original data models, and indicator interrelationship knowledge models; Formalizing the data analysis problem based on the analysis object, original data set, original data model, and indicator relationship knowledge model to form a knowledge graph based on the data analysis problem; Selecting dimensional features of a test data set based on the degree of relevance between the data analysis problem and different dimensional features and the knowledge graph, including: connecting data table columns based on the consistency principle of the analysis object and the dimensional features according to the data analysis problem, and searching for the analysis object in the original data set based on the analysis object; determining fact table columns based on device performance, and searching the knowledge graph for factors associated with the device performance; searching the fact table columns based on environmental features or dimensional features, and obtaining the location and data collection frequency of the fact table columns where the environmental features or dimensional features are located; selecting dimensional features of the test data set based on the location and data collection frequency of the fact table columns where the environmental features or dimensional features are located; Converting the data type and data analysis role of the original data set according to the dimensional characteristics of the test data set, and fusing the converted data types and data roles to form a preliminary data set; performing range selection on the preliminary data set to form an analysis data set; The analysis data set is verified and evaluated for data set quality and data structure to obtain an evaluation result.

6. The device according to claim 5, characterized in that The processor is configured with processor-executable operation instructions to perform the following operations: Convert the data type and data analysis role of the original dataset according to the dimensional characteristics of the test dataset; Using the dimensional characteristics of the analysis object and the test data set as the connection means, and the data collection frequency as the starting point, the data in the original data set is shrunk and expanded according to the business meaning to form a preliminary data set.

7. The device according to claim 5, characterized in that The processor is configured with processor-executable operation instructions to perform the following operations: selecting a screening dimension, and screening the data in the preliminary data set according to the screening dimension; The filtered data is segmented according to the working conditions to form an analysis data set.

8. The device according to claim 5, characterized in that The processor is configured with processor-executable operation instructions to perform the following operations: Verifying and evaluating the quality of the analysis dataset according to the dataset quality characteristic indicators to obtain a quality evaluation result; Verifying and evaluating the data structure of the analysis data set according to the data set evaluation index to obtain a data structure evaluation result; An evaluation result is obtained according to the quality evaluation result and the data structure evaluation result.

Images