Test data standardization processing method and device and electronic equipment

Abstract

The invention relates to a test data standardization processing method and device and electronic equipment, and belongs to the technical field of data processing.The method comprises the steps that test data files are obtained, the test data files are sorted and positioned, and a preprocessing data set is determined; determining a standard variable name, a standard data header and a standard field of the data according to a consistency principle of the data and basic attributes of the data; according to a standard variable name, a standard data header and a standard field, carrying out standardization processing on the preprocessed data set, and carrying out reordering according to the standard field to obtain a standard data set; according to the standard fields, the test data files of the standard data sets are combined, and experiment analysis is carried out according to the combined test data files. The technical problems that in the prior art, data processing efficiency is low, standardization is insufficient, and abnormal data processing is incomplete are solved.

Description

Technical Field

[0001] The present invention relates to the technical field of data processing, and in particular to a method, device and electronic equipment for standardizing test data. Background Art

[0002] In the process of modern engine R&D and production, engine reliability and performance tests are key links to ensure engine quality and performance. These tests are usually conducted through various test platforms (such as test benches) to collect a large amount of test data. However, in the existing engine test data processing and analysis, the test data formats and languages ​​generated by different test benches are inconsistent, and even the same test bench may modify the collection fields during the test, resulting in inconsistent data formats. And usually each reliability test will generate more than 60 data files, each file has a data volume of about 1G, and the total data volume reaches about 4 million lines. Processing this massive data is a huge challenge.

[0003] At present, data processing and analysis mainly rely on manual operation and empirical judgment, which is not only time-consuming and error-prone, but also highly subjective, which is not conducive to the subsequent integration and comparative analysis of data. The Chinese invention patent with announcement number CN201310250677.6 discloses a method for importing and processing engine test data in a standardized specific format, including obtaining the engine test data and screening it by file name to screen out the data files that need to be processed; judging the internal format of the test data file obtained by step 1, and typeset the files that meet the specified header format, that is, deleting useless column data. However, the existing processing method cannot adapt to the headers and variables of different test benches, and lacks screening for the validity of the data, which is easy to misrecord the information files automatically generated by the test bench.

[0004] In summary, the existing engine test data processing and analysis methods have many shortcomings, and there is an urgent need for a complete set of integrated test data standardization, validation, screening, curve drawing, trend prediction and report generation methods and systems. Summary of the invention

[0005] In view of this, it is necessary to provide a test data standardization processing method, device and electronic equipment to solve the technical problems of low data processing efficiency, insufficient standardization and imperfect abnormal data processing in the prior art.

[0006] In order to solve the above problems, the present invention provides a test data standardization processing method, comprising: Acquire test data files, sort and locate the test data files, and determine a preprocessing data group; According to the consistency principle and basic attributes of data, determine the standard variable name, standard data header and standard field of data; The preprocessed data group is standardized according to the standard variable name, the standard data header and the standard field, and reordered according to the standard field to obtain a standard data group; According to the standard fields, the test data files of multiple standard data groups are merged, and the experimental analysis is performed based on the merged test data files.

[0007] In a possible implementation, the step of sorting and locating the test data files and determining the preprocessing data group includes: Sorting the test data files according to set rules to obtain a sorting result; Determine the data location conditions and data extraction conditions according to the working conditions and reading time requirements corresponding to the data; According to the data location condition and the data extraction condition, the data in the sorting result is extracted to obtain a preprocessed data group.

[0008] In a possible implementation manner, after obtaining the preprocessing data set, the method further includes: Determine the specific characters of the data group according to the working conditions; Judging whether the preprocessing data group is valid according to the inclusion relationship between the preprocessing data group and the specific character; If invalid, the invalid data is eliminated to obtain a valid preprocessed data set.

[0009] In a possible implementation, determining the standard variable name, standard data header, and standard field of the data according to the consistency principle of the data and the basic attributes of the data includes: Extract all variable names, data headers and fields from the preprocessed data set; Count the number of each variable name, each data header and each field; The variable name corresponding to the most-digit number of all variable names is determined as the standard variable name, the data header corresponding to the most-digit number of all data headers is determined as the standard data header, and the field corresponding to the most-digit number of fields is determined as the standard field.

[0010] In a possible implementation, the standardizing the preprocessed data set according to the standard variable name, the standard data header and the standard field includes: According to the standard fields, the existing fields in the preprocessed data group are mapped to the standard fields; According to the standard data header, the existing header of the preprocessed data group is mapped to the standard data header; Based on the standard variable names, map the existing variable names of the preprocessed data set to the standard variable names.

[0011] In a possible implementation, mapping the existing fields in the preprocessed data group to the standard fields according to the standard fields includes: Create a translation vocabulary and enter the field name and unit of the standard field into the translation vocabulary; Establishing a mapping relationship between existing fields and standard fields in the translation vocabulary; According to the mapping relationship, the existing fields in the preprocessed data group are mapped to the standard fields.

[0012] In a possible implementation, merging the test data files of multiple standard data groups according to the standard fields, and performing experimental analysis according to the merged test data files, includes: Based on consistent data headers and variable names, and taking standard fields as the merging basis, the folder data of multiple standard data groups are merged into the same folder to form a merged test data file; Construct an analysis trend graph based on the data in the experimental data file; According to the analysis trend curve chart, the analysis results are obtained.

[0013] In a second aspect, the present invention further provides a test data standardization processing device, comprising: A preprocessing module is used to obtain test data files, sort and locate the test data files, and determine a preprocessing data group; The standard determination module is used to determine the standard variable name, standard data header and standard field of the data according to the consistency principle of the data and the basic attributes of the data; A standardization processing module, used for standardizing the preprocessed data group according to the standard variable name, standard data header and standard field, and re-sorting according to the standard field to obtain a standard data group; The analysis module is used to merge the test data files of multiple standard data groups according to the standard fields, and perform experimental analysis according to the merged test data files.

[0014] In a third aspect, the present invention further provides an electronic device, comprising: a processor and a memory; The memory stores a computer-readable program executable by the processor; When the processor executes the computer-readable program, the steps in the test data standardization processing method described above are implemented.

[0015] In a fourth aspect, the present invention further provides a computer-readable storage medium storing one or more programs, wherein the one or more programs can be executed by one or more processors to implement the steps in the test data standardization processing method as described above.

[0016] The beneficial effects of the present invention are as follows: firstly, the test data file is obtained, and the test data file is sorted and located to determine the pre-processed data group; by sorting and pre-processing the data file, the order consistency of data processing can be guaranteed, thereby improving the efficiency and accuracy of data processing, and further, by positioning and pre-processing the data, the accuracy of data index retrieval can be improved, so as to quickly and accurately find the required data to be processed from the massive data. Then, according to the consistency principle of the data and the basic attributes of the data, the standard variable name, standard data header and standard field of the data are determined; through a series of standardized definitions, the pre-processed data group is standardized according to the standard variable name, standard data header and standard field, and re-sorted according to the standard field to obtain the standard data group; so that the format of the data is unified, and finally, according to the standard field, the test data files of multiple standard data groups are merged, and experimental analysis is performed according to the merged test data files. The present invention provides a universal data standardization processing method, which is applicable to various types of test data, improves the availability and accuracy of file data, and can realize multi-dimensional data screening through sorting and positioning, which is convenient for concise and clear multi-dimensional analysis and display of test data, and improves the efficiency of data analysis and processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 A method flow chart of an embodiment of a method for standardizing test data provided by the present invention; Figure 2 A method flow chart of an embodiment of step S101 in the test data standardization processing method provided by the present invention; Figure 3 A schematic diagram of an abnormal curve of original test data in the test data standardization processing method provided by the present invention; Figure 4 A schematic diagram of a curve after curve fitting and estimation in the test data standardization processing method provided by the present invention; Figure 5 is a schematic diagram of an embodiment of a test data standardization processing device provided by the present invention; Figure 6 It is a schematic diagram of the operating environment of an electronic device provided by the present invention. DETAILED DESCRIPTION

[0018] The preferred embodiments of the present invention are described in detail below in conjunction with the accompanying drawings, wherein the accompanying drawings constitute a part of this application and are used together with the embodiments of the present invention to illustrate the principles of the present invention, but are not used to limit the scope of the present invention.

[0019] A specific embodiment of the present invention discloses a method for standardizing test data. Figure 1,include: S101, obtaining test data files, sorting and locating the test data files, and determining a preprocessing data group; S102. Determine the standard variable name, standard data header and standard field of the data according to the consistency principle of the data and the basic attributes of the data; S103, performing standardization processing on the preprocessed data group according to the standard variable name, standard data header and standard field, and reordering according to the standard field to obtain a standard data group; S104. Merge the test data files of multiple standard data groups according to the standard fields, and perform experimental analysis according to the merged test data files.

[0020] In this embodiment, firstly, the test data file is obtained, and the test data file is sorted and located to determine the preprocessed data group; by sorting and preprocessing the data file, the order consistency of data processing can be guaranteed, thereby improving the efficiency and accuracy of data processing, and further, by positioning and preprocessing the data, the accuracy of data index retrieval can be improved, so as to quickly and accurately find the required data to be processed from the massive data. Then, according to the consistency principle of the data and the basic attributes of the data, the standard variable name, standard data header and standard field of the data are determined; through a series of standardized definitions, the preprocessed data group is standardized according to the standard variable name, standard data header and standard field, and re-sorted according to the standard field to obtain the standard data group; so that the format of the data is unified, and finally, according to the standard field, the test data files of multiple standard data groups are merged, and experimental analysis is performed according to the merged test data files. The present invention provides a universal data standardization processing method, which is applicable to various types of test data, improves the availability and accuracy of file data, and can realize multi-dimensional data screening through sorting and positioning, which is convenient for concise and clear multi-dimensional analysis and display of test data, and improves the efficiency of data analysis and processing.

[0021] It should be noted that the test data file can be various types of data obtained through bench tests, including but not limited to engine test data files, brake test data files, gearbox test data files or others. In this embodiment, the test data is engine bench test data, and the test data format includes common data formats: CSV, TXT, XLSX.

[0022] In a specific example, a 1000h thermal shock reliability test of a certain model generates 60 txt files and 600,000 lines of data. A line is taken every 1315 lines, and a total of about 4500 lines of data are read. First, all test data files of the specified test items are collected, sorted by name or time, and the file size and number of data lines are calculated. The recognizable file formats include common data formats: CSV, TXT, XLSX.

[0023] Furthermore, the file size and number of data rows are calculated to assess the data volume, check file integrity, prepare for data preprocessing, assist in sorting, and conduct preliminary data quality assessment, thereby ensuring the smooth progress of subsequent data processing steps and the accuracy of the results.

[0024] In some embodiments, the test data files are sorted and located to determine the preprocessing data set. Figure 2 ,include: S201, sorting the test data files according to set rules to obtain a sorting result; S202, determining data location conditions and data extraction conditions according to the working conditions and reading time requirements corresponding to the data; S203: extracting data from the sorting result according to the data location condition and the data extraction condition to obtain a preprocessed data group.

[0025] In this embodiment, the settings for data reading mainly include the specification of requirements related to the reading time and working conditions. These settings are intended to ensure that data under a specific time period and specific working conditions can be accurately and efficiently read and processed during the data processing process.

[0026] In step S201, the test data files are sorted according to a set rule, including sorting by name or time.

[0027] In step S202, the working condition corresponding to the data means to ensure that only data under specific working conditions, such as steady-state working conditions, transient working conditions, etc., are read to avoid reading irrelevant data. The working conditions, such as temperature, pressure, flow, etc., are set according to actual needs. Specify the type of working condition to be read.

[0028] Furthermore, the reading time requirement is set to ensure that data is read within a specific time period to avoid reading irrelevant data. The time resolution for reading data is set according to the requirements, such as every second, every minute, every hour, etc.

[0029] In step S203, according to the reading time setting, the data within a specific time period is located, and according to the requirements related to the working condition, the data under the specific working condition is filtered out, and according to the time resolution, part of the data is read on demand instead of reading all the data.

[0030] In some embodiments, after obtaining the preprocessing data set, the method further comprises: Determine the specific characters of the data group according to the working conditions; Judging whether the preprocessing data group is valid according to the inclusion relationship between the preprocessing data group and the specific character; If invalid, the invalid data is eliminated to obtain a valid preprocessed data set.

[0031] In this embodiment, whether the unit line data contains special characters (such as Nm, kg, kpa, etc.) is used to determine whether it is engine test data; by analyzing the number and proportion of digital characters in the line data, it is determined whether the data is valid. Non-engine test data or too few digital characters, such as invalid files automatically generated by document attributes or start-stop, are deleted. The attribute table files shown in Table 1 are deleted.

[0032] Table 1: Invalid attribute table file

[0033] In some embodiments, the step of determining the standard variable name, standard data header, and standard field of the data according to the consistency principle of the data and the basic attributes of the data includes: Extract all variable names, data headers and fields from the preprocessed data set; Count the number of each variable name, each data header and each field; The variable name corresponding to the most-digit number of all variable names is determined as the standard variable name, the data header corresponding to the most-digit number of all data headers is determined as the standard data header, and the field corresponding to the most-digit number of fields is determined as the standard field.

[0034] In this embodiment, by unifying the variable names, it is possible to ensure that the variable names in all data tables are consistent, avoiding the situation where the same variables in different data tables use different names. Determining the standard variable names, standard data headers, and standard fields of the data is a key step in the data standardization process. These steps ensure the consistency and readability of data in different data sources and different application scenarios. By reasonably defining and unifying these standards, the efficiency of data processing and data quality can be effectively improved, and the integrity and consistency of data can be ensured.

[0035] In a specific embodiment, some settings need to be made for data reading, mainly including test time period, reading frequency, main working conditions, steady-state length, downtime (blank) duration, etc. Through data positioning, jump reading from the complete test data can be specified, and the interval (resolution) can be specified. The resolution is specified as 10 this time, and after conversion, a total of 2400 lines are read, which is about 1 / 1300 shorter than the original length.

[0036] Furthermore, according to the specified steady-state length time, that is, the minimum time for speed stability. Since some reliability tests have several speed stable working conditions in a single cycle, and the durations are different, this setting ensures that the value of each working condition in a single cycle can be obtained. For example, in this test, the minimum stable time of a single cycle is 20s. The steady-state length is 20s, that is, the value is taken at intervals of 20s at most. Furthermore, according to the above specified standard shutdown (blank) time , the downtime is screened out in the following way: when the blank time is greater than 2 times the standard deviation S, it is screened out.

[0037]

[0038] According to the standard, the downtime is set to 120s. Usually, in the overall analysis of some tests, the working conditions that do not need to be paid attention to in this analysis can be eliminated according to the main working conditions specified above, further shortening the time for data processing and drawing. The main working condition this time is the rated point 2200rpm. According to this screening, the number of read lines is further shortened by at least 3 / 5.

[0039] It should be noted that other methods may be used to determine the standard variable name, standard data header and standard field, wherein the standard header, standard variable name and standard field are not unique.

[0040] In some embodiments, the standardizing the preprocessed data set according to standard variable names, standard data headers, and standard fields includes: According to the standard fields, the existing fields in the preprocessed data group are mapped to the standard fields; According to the standard data header, the existing header of the preprocessed data group is mapped to the standard data header; Based on the standard variable names, map the existing variable names of the preprocessed data set to the standard variable names.

[0041] In this embodiment, the field names and units of each rack are mapped to standard names through the translation word library function, and the translation word library is supported to be updated in real time to ensure the latest standards of field names and units. The field names and units of each rack are entered into the translation word library, and a mapping relationship is established in the translation word library. The existing field names and units are mapped to the standard field names and units. The mapping relationship in the translation word library is used to automatically update the field names and units in the data table.

[0042] In some embodiments, the step of merging the test data files of multiple standard data groups according to the standard fields, and performing experimental analysis according to the merged test data files includes: Based on consistent data headers and variable names, and with standard fields as the merging basis, multiple folder data are merged into the same folder to form a merged test data file; Construct an analysis trend graph based on the data in the experimental data file; According to the analysis trend curve chart, the analysis results are obtained.

[0043] In this embodiment, first, the data is filtered out and retained according to the test time interval required for analysis. After that, the data can be multi-dimensionally filtered according to the speed or other engine state parameters (such as oil pressure, water outlet temperature, etc.) and the curve can be drawn. Furthermore, the entire curve can be filtered by extreme values, average values, etc., which is convenient for analysis and can reduce data density. Taking extreme values ​​as an example, the extreme values ​​of the data in each cycle are extracted, and the extreme value points under the specified working conditions are connected to draw a curve, and the curve is normalized according to the specified interval.

[0044] Furthermore, the transition duration can be specified, and the data in the transition phase of the working condition can be filtered to retain only the steady-state data. According to the test data points, a trend line is constructed and fitted modeling is performed, and an inequality for eliminating abnormal data is constructed.

[0045] Furthermore, the operating data points are compared with the trend line data, and the maximum rate of change is calculated and displayed. When the rate of change exceeds the limit, a red alarm is used. The abnormal data is screened out according to the abnormal data elimination inequality.

[0046] Furthermore, the error calculation can specify the accuracy (preferably 3σ in this embodiment), and the value exceeding the specified accuracy is defined as abnormal data. In addition, the unmeasured or sensor fault data can be estimated and the curve trend can be predicted.

[0047] like Figure 3 As shown in the figure, the leakage of a certain engine piston is usually within the normal range of 100 at the rated point. The curve in the figure has three data distortions (caused by loose sensors or measurement leakage, etc.). First, remove the abnormal data as described above, and then estimate the missing data based on the trend line, such as Figure 4 shown.

[0048] Based on the above test data standardization processing method, the embodiment of the present invention also provides a test data standardization processing device, see Figure 5 ,include: The preprocessing module 510 is used to obtain test data files, sort and locate the test data files, and determine a preprocessing data group; A standard determination module 520 is used to determine the standard variable name, standard data header and standard field of the data according to the consistency principle of the data and the basic attributes of the data; A standardization processing module 530 is used to perform standardization processing on the pre-processed data group according to the standard variable name, standard data header and standard field, and re-sort them according to the standard field to obtain a standard data group; The analysis module 540 is used to merge the test data files according to the standard fields and perform experimental analysis based on the merged test data files.

[0049] like Figure 6 As shown, based on the above test data standardization processing method, the present invention also provides an electronic device, which can be a computing electronic device such as a mobile terminal, a desktop computer, a notebook, a palm computer, and a server. The electronic device includes a processor 610, a memory 620, and a display 630. Figure 6 Only some components of the electronic device are shown, but it should be understood that it is not required to implement all of the components shown, and more or fewer components may be implemented instead.

[0050] In some embodiments, the memory 620 may be an internal storage unit of the electronic device, such as a hard disk or memory of the electronic device. In other embodiments, the memory 620 may also be an external storage electronic device of the electronic device, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (SecureDigital, SD) card, a flash card (Flash Card), etc. equipped on the electronic device. Further, the memory 620 may also include both an internal storage unit of the electronic device and an external storage electronic device. The memory 620 is used to store application software and various types of data installed in the electronic device, such as program codes for installing the electronic device. The memory 620 may also be used to temporarily store data that has been output or is to be output. In one embodiment, a test data standardization processing program 640 is stored on the memory 620, and the test data standardization processing program 640 can be executed by the processor 610, thereby realizing the test data standardization processing method of each embodiment of the present application.

[0051] In some embodiments, the processor 610 may be a central processing unit (CPU), a microprocessor or other data processing chip, used to run program codes stored in the memory 620 or process data, such as executing a test data standardization processing method.

[0052] In some embodiments, the display 630 may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, an OLED (Organic Light-Emitting Diode) touch device, etc. The display 630 is used to display information in the test data standardization processing electronic device and to display a visual user interface. The components 610-630 of the electronic device communicate with each other via a system bus.

[0053] Those skilled in the art will appreciate that all or part of the processes of the above-mentioned embodiments can be implemented by instructing related hardware through a computer program, and the program can be stored in a computer-readable storage medium, wherein the computer-readable storage medium is a disk, an optical disk, a read-only storage memory, or a random access memory, etc.

[0054] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by any technician familiar with the technical field within the technical scope disclosed by the present invention should be covered within the protection scope of the present invention.

Claims

1. A method for standardizing test data, characterized in that: include: Acquire test data files, sort and locate the test data files, and determine a preprocessing data group; According to the consistency principle and basic attributes of data, determine the standard variable name, standard data header and standard field of data; The preprocessed data group is standardized according to the standard variable name, the standard data header and the standard field, and reordered according to the standard field to obtain a standard data group; According to the standard fields, the test data files of multiple standard data groups are merged, and the experimental analysis is performed based on the merged test data files.

2. The test data standardization processing method according to claim 1, characterized in that: The step of sorting and locating the test data files and determining the preprocessing data group comprises: Sorting the test data files according to set rules to obtain a sorting result; Determine the data location conditions and data extraction conditions according to the working conditions and reading time requirements corresponding to the data; According to the data location condition and the data extraction condition, the data in the sorting result is extracted to obtain a preprocessed data group.

3. The test data standardization processing method according to claim 1, characterized in that: After obtaining the preprocessing data set, the method further includes: Determine the specific characters of the data group according to the working conditions; Judging whether the preprocessing data group is valid according to the inclusion relationship between the preprocessing data group and the specific character; If invalid, the invalid data is eliminated to obtain a valid preprocessed data set.

4. The test data standardization processing method according to claim 1, characterized in that: The standard variable name, standard data header and standard field of the data are determined according to the consistency principle of the data and the basic attributes of the data, including: Extract all variable names, data headers and fields from the preprocessed data set; Count the number of each variable name, each data header and each field; The variable name corresponding to the most-digit number of all variable names is determined as the standard variable name, the data header corresponding to the most-digit number of all data headers is determined as the standard data header, and the field corresponding to the most-digit number of fields is determined as the standard field.

5. The test data standardization processing method according to claim 1, characterized in that: The step of standardizing the preprocessed data set according to the standard variable name, the standard data header and the standard field includes: According to the standard fields, the existing fields in the preprocessed data group are mapped to the standard fields; According to the standard data header, the existing header of the preprocessed data group is mapped to the standard data header; Based on the standard variable names, map the existing variable names of the preprocessed data set to the standard variable names.

6. The test data standardization processing method according to claim 5, characterized in that: The step of mapping the existing fields in the preprocessed data group to the standard fields according to the standard fields includes: Create a translation vocabulary and enter the field name and unit of the standard field into the translation vocabulary; Establishing a mapping relationship between existing fields and standard fields in the translation vocabulary; According to the mapping relationship, the existing fields in the preprocessed data group are mapped to the standard fields.

7. The test data standardization processing method according to claim 1, characterized in that: The step of merging the test data files of multiple standard data groups according to the standard fields and performing experimental analysis according to the merged test data files includes: Based on consistent data headers and variable names, and taking standard fields as the merging basis, the folder data of multiple standard data groups are merged into the same folder to form a merged test data file; Construct an analysis trend graph based on the data in the experimental data file; According to the analysis trend curve chart, the analysis results are obtained.

8. A test data standardization processing device, characterized in that: include: A preprocessing module is used to obtain test data files, sort and locate the test data files, and determine a preprocessing data group; The standard determination module is used to determine the standard variable name, standard data header and standard field of the data according to the consistency principle and basic attributes of the data; A standardization processing module, used for standardizing the preprocessed data group according to the standard variable name, standard data header and standard field, and re-sorting according to the standard field to obtain a standard data group; The analysis module is used to merge the test data files of multiple standard data groups according to the standard fields, and perform experimental analysis according to the merged test data files.

9. An electronic device, characterized in that: include: Processor and memory; The memory stores a computer-readable program executable by the processor; When the processor executes the computer-readable program, the steps in the test data standardization processing method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the steps in the test data standardization processing method according to any one of claims 1 to 7.

Images