Method and system for processing data

Abstract

This disclosure proposes a data processing method and a data processing system. The method includes: collecting multiple data files from multiple devices, wherein the format of one data file differs from that of another data file; performing field matching or filling in missing values on the data files, to generate a dataset; extracting multiple key attributes from the dataset; and obtaining test data associated with the key attributes from the dataset, presenting the test data and a specification associated with the test data in a graphical interface.

Description

[Technical Field]

[0001] This disclosure relates to a data processing method and data processing system capable of collecting and merging multiple different types of data. [Previous Technology]

[0002] In traditional production test data processing workflows, test fixtures generate a large amount of test data in different formats. Due to the inconsistent formats of the test data, operators usually need to manually observe the data structure first to confirm the data fields and data arrangement. Then, operators need to manually compile the test data from different sources into a single file for subsequent analysis.

[0003] After data collection, operators use built-in functions in common tools such as Microsoft Excel to calculate basic statistical data, such as average, maximum, and minimum values, and manually record the results for simple analysis. However, this method is highly dependent on manual operation, which is not only time-consuming but also susceptible to human error. In addition, the data processing results are limited to basic statistical analysis, making it difficult to deeply explore potential anomalies or trends in the data, thus limiting the decision-making efficiency of the production line.

[0004] Some modified traditional methods may employ partial database management systems (such as SQL) for basic data processing, such as data querying, filtering, and simple data aggregation. However, these methods still cannot effectively handle multi-format data from different test fixtures and typically lack the ability to automate data processing. Therefore, when the production line faces abnormal conditions, it cannot provide timely and accurate data analysis and diagnosis, leading to reduced production efficiency and potentially affecting product quality. [Summary of the Invention]

[0005] In order to solve the above problems, this disclosure proposes a data processing method and a data processing system.

[0006] This disclosure proposes a data processing method applicable to computing modules. This data processing method includes: collecting multiple data files from multiple devices, where one data file has a different format than another; performing field comparison or filling in missing values ​​on the data files to generate a dataset; extracting multiple key attributes from the dataset; and obtaining test data corresponding to the key attributes from the dataset, presenting the test data and the corresponding test data specifications in a graphical interface.

[0007] From another perspective, the embodiments disclosed herein provide a data processing system comprising multiple devices and a computing module. The devices are used to generate multiple data files. The computing module is communicatively connected to the devices to execute the data processing method described above.

[0008] To make the above-disclosed features and advantages more apparent and understandable, specific embodiments are described below in conjunction with the accompanying drawings.

Implementation Method

[0010] Some embodiments of this disclosure will now be described in detail with reference to the accompanying drawings. Component symbols used in the following description are considered the same or similar components when they appear in different drawings. These embodiments are only a part of this disclosure and do not disclose all possible implementations of this disclosure. More precisely, these embodiments are merely examples of the systems and methods within the scope of this patent application.

[0011] The terms "first" and "second" used in this document do not specifically refer to order or sequence, but are used only to distinguish elements or operations described with the same technical terms.

[0012] Figure 1 is a schematic diagram illustrating a data processing system according to an embodiment. Referring to Figure 1, the data processing system 100 includes a computing module 110 and devices 120-123. The computing module 110 is, for example, a personal computer, a laptop computer, a server, or other electronic devices with computing capabilities. Devices 120-123 may include any manufacturing equipment or testing fixtures. For example, one device is used to measure the volume of a speaker in various frequency bands, while other devices are used to measure the speaker's impedance, power, signal-to-noise ratio, water resistance, communication stability, etc. This data can be structured, semi-structured, or unstructured. Each device 120 can generate one or more data files based on its measured data, but since different devices 120-123 may be provided by different manufacturers or have different models, the formats of the data files generated by these devices 120-123 may be different. For example, the formats of these data files may include JSON (JavaScript Object Notation), XLSX, CSV (comma-separated values), etc., which are not limited to this disclosure. The computing module 110 can communicate with devices 120-123 via wired or wireless means to obtain these data files. In some embodiments, the computing module 110 can also standardize these test data, such as converting units or removing blanks. The computing module 110 executes a data processing method to merge these data files, automatically calculates multiple indicators, and provides a graphical interface to view these test data when the data type is appropriate.

[0013] Figure 2 is a flowchart illustrating a data processing method according to an embodiment. In step 201, multiple data files from multiple devices are collected. As mentioned above, at least two of the data files have different formats.

[0014] In step 202, field comparison, blanking, and logging are performed on these data files to generate a dataset. Here, the dataset can conform to any preset format; that is, all data files are converted to the same format and merged together. To merge data files, field comparison is first performed. For example, in a JSON file, there are multiple key-value pairs, where the keys are used to describe the data and can therefore be used as field names. In an XLSX file, the worksheet name can be analyzed, and then the field names can be obtained from specific cells. After identifying the field names from these files, an identifier or test number can be selected as the primary key, and then other test data can be associated with this primary key. For example, two data files each have a field recording the product identifier; one data file has a field recording volume, and the other data file has a field recording impedance. After comparing the fields of different data files, the product identifier can be obtained, and then the volume and impedance can be added to the product's attributes. In some embodiments, fuzzy matching can be performed when comparing fields. This fuzzy matching can be achieved using an algorithm or provided by any software. For example, the pandas software provides fuzzy matching functionality. When performing fuzzy matching, some meaningless characters (such as underscores) can be ignored, thus obtaining matching first and second field names, even if the first field name is different from the second field name. For example, if the first field name is "TEST1" and the second field name is "TEST_1", they will be paired after fuzzy matching. This also prevents data that should belong to the same attribute from becoming different attributes after merging.

[0015] Regarding filling null values, the first step is to search for and identify null values ​​in the data file. For example, in a JSON file, if there is no key, there is a lack of corresponding values; in an XLSX file, blank spaces, NaN, and NULL values ​​are also null values; in a CSV file, if the number of rows is too low, it also indicates the presence of null values. Next, multiple related values ​​corresponding to the null values ​​can be obtained. These related values ​​can be, for example, values ​​before or after in a time series, and then the weights of these related values ​​can be calculated. The weights used can be set using a Gaussian distribution, or all weights can be set to be equal. Next, the null values ​​can be replaced with the weighted sum. In some embodiments, the user can also set whether to delete outliers, which can be determined using the standard deviation range or the interquartile range. In some embodiments, if the data format is incorrect or inconsistent, the data format can also be modified, for example, by changing it to a uniform date format, unit, or numerical representation.

[0016] Regarding the logging process, if an unexpected error occurs during the merging of data files, the relevant information can be added to a log file. For example, if multiple data entries with the same identifier and attribute exist in different data files, and this attribute should only have one entry, this indicates an anomaly, and the data information (e.g., field name, value, which data file it belongs to, etc.) can be written to the log file. Alternatively, if the number of null values ​​is found to be greater than a threshold when filling null values, the null value information (e.g., field name, number, which data file it belongs to, etc.) can also be written to the log file. In some embodiments, if the null value belongs to an important field that is necessary, the information of the important field can also be written to the log file. In some embodiments, if the data format is abnormal or inconsistent, the relevant format information can also be written to the log file after format conversion. In other embodiments, merging-related information can also be written to the log file, such as the merged file name, merging conditions, field alignment, etc.

[0017] In some embodiments, only one or more of the above-described field comparison, null filling, and recording procedures may be performed. For example, in some embodiments, only field comparison and null filling may be performed, but the recording procedure may not be performed.

[0018] In some embodiments, step 202 may employ chunking technology, which can efficiently process large amounts of data, support batch merging operations on multiple data files, and shorten execution time.

[0019] In step 203, multiple key attributes are extracted from the dataset. These key attributes may include equipment serial number, product serial number, operator, test time and location, test items, test methods, parameter configurations, etc. These key attributes will be standardized (e.g., converted to a uniform format) and then compiled into the all raw data archive. In addition, the computing module 110 can also perform version control to ensure data traceability.

[0020] In step 204, multiple indicators and statistical values ​​are generated based on the aforementioned data set. These statistical values ​​may include count, maximum value, minimum value, average value, median, quartiles, standard deviation, etc. The aforementioned indicators may include Process Performance Lower Index (Ppl), Process Performance Upper Index (Ppu), repeatability, reproducibility, part-by-part variation, Process Performance Index (Ppk), Gauge repeatability and reproducibility (GRR), etc. In some embodiments, a table can be created based on GRR and Ppk to record the corresponding measurement quality, as shown in Table 1 below. GRR (%) <= 10 10 < GRR (%) < 30 GRR(%)>=30 Ppk >= 1.67 Acceptable Acceptable Needs improvement 1.67 > Ppk > 1.33 Acceptable Needs improvement Unacceptable Ppk <= 1.33 Needs improvement Unacceptable Unacceptable Table 1

[0021] The first dimension (vertical rows) of Table 1 above is Gauge Repeatability and Reproducibility (GRR), and the second dimension (horizontal columns) is the process performance indicator Ppk. This table contains multiple measurement quality categories, namely "Acceptable," "Needs Improvement," and "Unacceptable." Lower GRR is better; in this embodiment, 10% and 30% are used as two threshold values, resulting in three rows in Table 1. Higher Ppk is better; 1.33 and 1.67 are used as two threshold values, resulting in three columns in Table 1. Therefore, Table 1 has nine blanks. The upper left corner represents better measurement quality, and the lower right corner represents poorer measurement quality. After a measurement is performed, the test data can be paired with Table 1 to obtain the measurement quality in the corresponding blank. This approach allows for a comprehensive consideration of both GRR and Ppk to determine the appropriate measurement quality, providing a more complete indicator.

[0022] In step 205, it is determined whether there is continuously sampled data. If not, step 207 is executed to generate an analysis report. This analysis report may include the above-mentioned statistical values ​​and indicators, and this disclosure does not limit the content of the analysis report.

[0023] If there is continuous data, proceed to step 206 to obtain the test data corresponding to the key attributes from the dataset and present the test data and the corresponding test data specifications in the graphical interface. Figure 3 is a schematic diagram illustrating the graphical interface according to an embodiment. The horizontal axis of graph 400 is frequency, and the vertical axis is volume (in decibels). Each curve represents a piece of test data. In this example, the key attributes include serial number, operator, and time, so the test data associated with these key attributes will be presented in graph 400. In addition, the corresponding specifications are displayed on each frequency band. For example, when the horizontal axis is "800", the upper limit of the specification 401 and the lower limit of the specification 402 are drawn. These upper and lower limits can be displayed as dots. Each frequency band has a corresponding upper limit and lower limit of the specification. For simplicity, only two frequency bands are drawn here as an example.

[0024] In some embodiments, the standard deviation for each frequency band is also calculated, and then the average value is added to or subtracted by n times the standard deviation, where n is a positive integer. In this embodiment, n=6, and such upper and lower limits are called six standard deviations (6 Sigma). Developing specifications according to six standard deviations is an important strategy to ensure product quality and manufacturing stability. After calculating six standard deviations for the test data, the six standard deviations can also be displayed in the graphical interface. Each frequency band has its own upper and lower limits, which are connected together to form a dashed line 411, and the lower limits are connected together to form a dashed line 412. As can be seen from Figure 3, the original specifications may not necessarily conform to six standard deviations. For example, the specification upper limit 401 is much larger than the six standard deviation upper limit 421, and the specification lower limit 402 is also lower than the six standard deviation lower limit 422. This indicates that the specifications are too lenient, allowing most products to pass the specifications. In some embodiments, the average value 423 of the test data can also be displayed in the graphical interface. In some cases, the specifications may also be too strict. For example, when the horizontal axis is "6100", the upper limit of 6 times the standard deviation 441 is greater than the upper limit of the specification 431, while the lower limit of 6 times the standard deviation 442 is lower than the lower limit of the specification 432. The average value 443 is also displayed in the graphical interface. In this embodiment, both the specification and 6 times the standard deviation are displayed in the graphical interface, allowing users to quickly analyze whether the specification is appropriate.

[0025] In some embodiments, a table 450 may also be displayed in a graphical interface. This table 450 displays test data belonging to key attributes. When a user observes an anomaly in a certain test data, they can learn from the table 450 about the corresponding operator, time, test device, test data, etc. For example, in the graph 400, there are curves 451 and 452 (corresponding to multiple test data, drawn as thicker lines for simplicity). Curve 451 is clearly different from other curves 452. Therefore, from the table 450, it can be seen which operator or test fixture curve 451 belongs to, thereby finding the cause of the anomaly. In some embodiments, the user can also edit the test data in the table 450. In some embodiments, the user can also change, delete, switch, or add key attributes, so the rows of the table 450 can be dynamically adjusted or selected. For example, the first column of the table 450 also has a drop-down object 453. After the user clicks on this drop-down object 453 with the cursor, they can select a key attribute. For example, the user can select another operator, another time, or replace the time with another key attribute.

[0026] In some embodiments, table 460 may also be displayed in the graphical interface, showing statistical values, indicators, specifications, etc. Tables 450 and 460 can be displayed in any location or in a new window, and this disclosure is not limited thereto. Users can directly modify the values ​​in tables 450 and 460, and these modifications will be reflected in chart 400 in real time. For example, if a user finds that the specification upper limit is too high, after modifying the specification upper limit in table 460, the dotted line 411 in chart 400 will be adjusted accordingly. In other words, the calculation module 110 can receive instructions (including keyboard input, mouse clicks, etc.) through this graphical interface, modify specifications, test data, or key attributes according to these instructions, and update the graphical interface in real time. In the prior art, users observe these test data in a table format, so it is not possible to quickly see which test data is abnormal, and the prior art cannot quickly identify which specifications are inappropriate.

[0027] In the above-described data processing method and system, data from various devices can be collected and merged. After merging and filling in missing values, a more complete dataset is obtained. Furthermore, the aforementioned techniques can automatically calculate various statistical values ​​and indicators, saving considerable manual effort. Through the graphical interface, users can easily observe whether the formulation of various specifications is reasonable.

[0028] Although this disclosure has been disclosed above with reference to embodiments, it is not intended to limit this disclosure. Anyone skilled in the art may make some modifications and refinements without departing from the spirit and scope of this disclosure. Therefore, the scope of protection of this disclosure shall be determined by the appended claims. [Simplified Explanation of the Diagram]

[0009] Figure 1 is a schematic diagram illustrating a data processing system according to an embodiment. Figure 2 is a flowchart illustrating a data processing method according to an embodiment. Figure 3 is a schematic diagram illustrating a graphical interface according to an embodiment.

Claims

1. A data processing method applicable to a computing module, the data processing method comprising: collecting multiple data files from multiple devices, wherein one of the data files has a different format than another of the data files; performing a field comparison or a null filling operation on the data files to generate a dataset, wherein the field comparison includes performing a fuzzy comparison of multiple field names to obtain mutually matching first field name and second field name, wherein the first field name is different from the second field name; extracting multiple key attributes from the dataset; and obtaining multiple test data corresponding to the key attributes from the dataset, and presenting the test data and a specification corresponding to the test data in a graphical interface.

2. The data processing method as described in claim 1, wherein filling in blank values ​​includes: Search for a null value in these data files; retrieve multiple related values ​​corresponding to the null value; Calculate a weighted sum of these related values; and replace the null value with this weighted sum.

3. The data processing method as described in claim 1 further includes: if the data files contain multiple data entries corresponding to the same identification code and the same attributes, writing the information of the data into a record file; and if the number of multiple null values ​​in the data files is greater than a threshold value, writing the information of the null values ​​into the record file.

4. The data processing method as described in claim 1 further includes: A table is created based on the repeatability and reproducibility of multiple gauges and multiple process performance indicators to record the corresponding multiple measurement qualities; And one of these test data is paired with the table to obtain one of the measurement qualities.

5. The data processing method as described in claim 1 further includes: Calculate the n-fold standard deviation of the test data and display the n-fold standard deviation and the specification in the graphical interface, where n is a positive integer.

6. The data processing method as described in claim 5 further includes: The system receives an instruction through the graphical interface and modifies the specification, test data, or key attributes according to the instruction, and updates the graphical interface in real time.

7. A data processing system comprising: a plurality of devices, each generating a plurality of data files; and a computing module communicatively connected to the devices and configured to perform a plurality of steps: collecting the data files, wherein one of the data files has a different format than another of the data files; performing a field comparison or filling in missing values ​​on the data files to generate a dataset, the field comparison comprising performing a fuzzy comparison of a plurality of field names to obtain a first field name and a second field name that match each other, wherein the first field name is different from the second field name; extracting a plurality of key attributes from the dataset; and obtaining a plurality of test data corresponding to the key attributes from the dataset, and presenting the test data and a specification corresponding to the test data in a graphical interface.

8. The data processing system as described in claim 7, wherein the filling of blank values ​​includes: Search for a null value in these data files; retrieve multiple related values ​​corresponding to the null value; Calculate a weighted sum of these related values; and replace the null value with this weighted sum.

9. The data processing system as described in claim 7, wherein the computing module is further configured to perform: if the data files contain multiple data entries corresponding to the same identification code and the same attributes, write information of the data to a record file; and if the number of multiple null values ​​in the data files is greater than a threshold value, write information of the null values ​​to the record file.

10. The data processing system as claimed in claim 7, wherein the computing module is further configured to: establish a table based on multiple gauge repeatability and reproducibility and multiple process performance indicators to record corresponding multiple measurement qualities; and match one of the test data to the table to obtain one of the measurement qualities.

Images