Data processing method and data processing system

Abstract

The invention provides a data processing method, which comprises the following steps of: grouping a data set to generate a plurality of data groups by utilizing a grouping method, and enabling the data set to comprise a plurality of data points; selecting a data group with least data points from the plurality of data groups; for each data group with the least data points, judging a local most possible outlier value of the data group; and determining a most probable outlier from the local most probable outliers of the data group having the least data points according to the center of the data body. The processing method can provide a correct outlier which is closest to reality.

Description

Technical Field

[0001] The present invention relates to a data processing method and a data processing system, and in particular to a data processing method and a data processing system capable of determining the most likely outlier. Background Art

[0002] Data sets are prone to containing noisy data, meaning data that lacks important information or misleads the analysis. To obtain more accurate data analysis results, data preprocessing is required before analysis to remove insignificant data and correct misleading data, thereby enhancing the reliability of the results. Outliers are a type of noisy data. A common method for identifying outliers today is to use statistically-based P-values. Given a given acceptable P-value (or significance level), the corresponding threshold value is found and the outliers are identified based on this threshold value. However, this outlier identification method can only identify extremely large or extremely small values ​​within the outlier dataset. Therefore, existing methods are not widely applicable in many digital applications. For example, a current signal with a value of "1" can represent a power-on state, while a value of "0" can represent a power-off state. However, due to external interference, the data detected by the received current signal may contain values ​​between 0 and 1. In this case, the actual noise signal is judged to have a value between 0 and 1, but the correct signal values ​​of 0 and 1 are considered outliers because they are extreme values ​​in the data. In short, the existing method is prone to incorrect outlier detection. Therefore, the existing technology needs improvement. Summary of the Invention

[0003] In order to solve the above problems, the present invention provides a data processing method and a data processing system that can determine the most likely outlier to solve the above problems.

[0004] The present invention provides a data processing method, comprising: clustering a data set using a clustering method to generate a plurality of data groups, wherein the data set includes a plurality of data points; selecting a data group with the fewest data points from the plurality of data groups; determining a local most likely outlier value for each data group with the fewest data points; and determining a most likely outlier value from the local most likely outliers of the data group with the fewest data points based on the center of a data body.

[0005] The present invention further provides a data processing system comprising: a database for storing a data set, wherein the data set includes a plurality of data points; a processing circuit coupled to the database, for obtaining the data set and clustering the data set using a clustering method to generate a plurality of data groups; wherein the processing circuit selects a data group with the fewest data points from the plurality of data groups, determines a local most likely outlier value of each data group with the fewest data points, and determines a most likely outlier value from the local most likely outliers of the data group with the fewest data points based on the center of a data body. BRIEF DESCRIPTION OF THE DRAWINGS

[0006] Figure 1 FIG. 1 is a schematic diagram of a data processing system according to an embodiment of the present invention.

[0007] Figure 2 This is a flow chart of an embodiment of the present invention.

[0008] Figure 3 Schematic diagram of a data set in an embodiment of the present invention.

[0009] Figure 4 Schematic diagram of data grouping in an embodiment of the present invention.

[0010] Figure 5 Schematic diagram of a possible outlier distribution in an embodiment of the present invention.

[0011] Figure 6 Schematic diagram of a local most likely outlier distribution in an embodiment of the present invention.

[0012] Figure 7 Schematic diagram of a most likely outlier in an embodiment of the present invention.

[0013] Figure 8 Schematic diagram of a data set change embodiment in an embodiment of the present invention.

[0014] Figure 9 Schematic diagram of an embodiment of data clustering and possible outlier distribution changes in an embodiment of the present invention.

[0015] Figure 10 Schematic diagram of the distribution of local most likely outliers and the variation of the most likely outliers in an embodiment of the present invention.

[0016] Component number description

[0017] 1 Data processing system

[0018] 10 Processing circuit

[0019] 20 databases:

[0020] 2 Process

[0021] Steps S200, S202, S204, S206, S208, S210

[0022] C1, CL1, CL2 Center

[0023] DB Data Subject

[0024] G1, G2, G3 data groups

[0025] LMPO Local Probable Outlier

[0026] PO possible outlier

[0027] MPO Maximum Probable Outlier DETAILED DESCRIPTION

[0028] Please refer to Figure 1 , Figure 1 Figure 1 is a schematic diagram of a data processing system 1 according to an embodiment of the present invention. Data processing system 1 includes processing circuitry 10 and a database 20. Database 20 stores a data set. A data set may include multiple data points. Processing circuitry 10 can access data sets stored in database 2. Processing circuitry 10 can also receive and process data sets from external devices.

[0029] Please refer to Figure 2 , Figure 2 Schematic diagram of a process 2 in an embodiment of the present invention. Process 2 includes the following steps:

[0030] Step S200: Start.

[0031] Step S202: clustering the data set using a clustering method to generate a complex data group, wherein the data set includes complex data points.

[0032] Step S204: Selecting a data group with the least data points from the plurality of data groups.

[0033] Step S206 : For each data group with the least data points, determine a local most likely outlier value of the data group.

[0034] Step S208: Determine a most likely outlier from the local most likely outliers of the data group with the least data points according to the center of a data body.

[0035] Step S210: End.

[0036] According to process 2, in step S202, the processing circuit 10 can obtain a data set from the database 20 or an external device. The data set includes a plurality of data points. For example, please refer to Figure 3 , Figure 3 FIG. 1 is a schematic diagram of a data set according to an embodiment of the present invention. Figure 3 As shown in the figure, the solid dots represent data points. After obtaining the data set, the processing circuit 10 uses a clustering method to cluster the data set to generate a plurality of data groups. Each data group includes at least one data point. Please refer to Figure 4 , Figure 4 FIG. 1 is a schematic diagram of data grouping according to an embodiment of the present invention. Figure 4 As shown, the dataset is clustered and labeled, and the multiple data points in the dataset are divided into data groups G1, G2, and G3. The clustering method can be set according to needs and can be any clustering method, such as, but not limited to, DBSCAN clustering, Agglomerative Hierarchical Clustering in machine learning, and K-means partitioning clustering.

[0037] In step S204, the processing circuit 10 selects a data group with the least data points from the plurality of data groups. All data points in the data group with the least data points are possible outliers. For example, please refer to Figure 5 , Figure 5 FIG. 1 is a schematic diagram of a possible outlier distribution according to an embodiment of the present invention. Figure 5 As shown, data groups G1 and G2 each include 3 data points, and data group G3 includes 10 data points. The processing circuit 10 selects the data group G1 and data group G2 with the least data points. Figure 5 As shown, all data points of the data group G1 and the data group G2 are possible outliers PO.

[0038] In step S206, the processing circuit 10 determines a local most likely outlier for each data group with the fewest data points. For each data group with the fewest data points, the processing circuit 10 determines the center of each data group. The center of a data group can be the arithmetic mean, median, or mode of all the data points in the data group. The center of a data group can be the data point whose sum of distances from other points in the data group is the minimum. The center of a data group can be one of all the data points in the data group. For each data point in the data group with the fewest data points, the processing circuit 10 calculates the distance between each data point and the center of the data group and determines the data point with the farthest distance from the center of the data group. The processing circuit 10 determines the data point with the farthest distance from the center of the data group as a local most likely outlier LMPO.

[0039] Please refer to Figure 6 , Figure 6FIG. 1 is a schematic diagram of a local most likely outlier distribution according to an embodiment of the present invention. Figure 6 As shown, for the data group G1, the processing circuit 10 determines that the center of the data group G1 is CL1. The processing circuit 10 calculates each data point in the data group G1 (i.e. Figure 5 The processing circuit 10 determines the data point with the farthest distance from the center CL1 of the data group G1 as a local most likely outlier LMPO. Similarly, for the data group G2, the processing circuit 10 determines that the center of the data group G5 is CL2. The processing circuit 10 calculates the distance between each data point (i.e. Figure 5 The processing circuit 10 then determines the data point with the farthest distance from the center CL2 of the data group G2 as a local most likely outlier LMPO.

[0040] The embodiments of the present invention may use any distance metric to calculate the distance. For example, the distance metric may be Euclidean distance, a distance metric derived from the Lp norm (|xy|p, 1≤p≤∞), or other distance metrics, but is not limited thereto. For example, any distance metric may be used in the process of the embodiments of the present invention as long as it satisfies the following three conditions for the data set:

[0041] 1. For any two data points x, y, d(x, y) ≥ 0, and

[0042] 2. For any two data points x and y, d(x, y) = d(y, x);

[0043] 3. For any three data points x, y, z, d(x, y) + d(y, z) ≥ d(x, z) (1)

[0044] Where d() is the distance function, and x, y, and z are the data points of the dataset.

[0045] In step S208, the processing circuit 10 determines a most likely outlier from the local most likely outliers of the data group with the least data points based on the center of a data body. The processing circuit 10 can determine a data body, and the data body can include at least one data point in the data set. For example, the data body may include regional data points in the data set where there are more data points and the distribution is more concentrated. The data body may include all data points in the data set except possible outliers, that is, the data body may include all data points that are not possible outliers in the data set. The data body may include the data group with the largest amount of data. The processing circuit 10 can set the data body according to different requirements. Further, the processing circuit 10 determines the center of the data body. The center of the data body can be the arithmetic mean, median or mode of all data points in the data body. The center of the data group can be the data point whose sum of distances to other points in the data group is the minimum. The center of the data group can be one of all the data points in the data body. For example, please refer to Figure 7 , Figure 7 FIG. 1 is a schematic diagram of a most likely outlier according to an embodiment of the present invention. Figure 7 As shown, the processing circuit 10 processes the data group with the largest amount of data (e.g. Figure 4 The data group G3) is judged as a data subject DB and a center C of the data subject DB is determined.

[0046] In step S208, the processing circuit 10 calculates the distance between each local most likely outlier and the center of the data body. The processing circuit 10 determines the local most likely outlier with the farthest distance from the center of the data body as a most likely outlier of the current data set. For example, please continue to refer to Figure 6 and Figure 7 , the processing circuit 10 calculates the distance between the local most likely outlier LMPO in the data group G1 and the center C of the data body DB. The processing circuit 10 calculates the distance between the local most likely outlier LMPO in the data group G2 and the center C of the data body DB. Because the distance between the local most likely outlier LMPO in the data group G1 and the center C is greater than the distance between the local most likely outlier LMPO in the data group G2 and the center C, the processing circuit 10 determines the local most likely outlier LMPO in the data group G1 as a most likely outlier MPO of the current data set. In this way, the embodiment of the present invention can provide an accurate outlier that is closest to the truth.

[0047] Furthermore, the processing circuit 10 can delete the most likely outliers in the data set to perform a data cleaning process. For example, when applied to abnormal current signal or digital signal detection, the data value "1" (first logic level) of the current signal can represent a power-on state, and the value data "0" (second logic level) can represent a power-off state. However, due to external interference, the data value determined by the received signal may have a value between 0 and 1. Figure 8 As shown, the data set includes data points related to the current signal. Most of the data points of the received current signal fall between the value 1 or the value 0. In practice, due to the influence of unstable current intensity, circuit interference, interference from other signal sources, etc., some data points are between the value 1 and the value 0. Figure 9 As shown, the processing circuit 10 of the embodiment of the present invention can divide the complex data points of the data set into data groups G1, G2, and G3, and select the data group G2 with the least data points (steps S202 and S204). All data points of the data group G2 are possible outliers PO. The processing circuit 10 can set the data body as the union of the data groups containing data value 0 and data value 1. That is, the data body includes data groups G1 and G3. The processing circuit 10 can define the center of the data body as value 1 and value 0. Figure 10 As shown, the processing circuit 10 can determine a local most likely outlier LMPO of the data group G2 and a most likely outlier MPO of the data set through steps S204 and S206. The processing circuit 10 can delete the most likely outlier MPO in the data set to eliminate abnormal current signals and achieve the purpose of purifying the two-bit signal. The data distribution of the current signal usually presents a bimodal state on both sides of the value 0 and the value 1. The outliers determined by the embodiment of the present invention are mostly data far away from the value 0 or the value 1. The above method can be used to repeatedly execute process 2 to eliminate abnormal current signals so that the data points of the data set gradually approach the correct signal data, and the bimodal phenomenon of the signal data distribution is also more obvious.

[0048] For example, regarding product inspection on a production line, in order to introduce an artificial intelligence model for application in automated product inspection, it is often necessary to train the artificial intelligence model. Since production lines with superior yield performance produce mostly normal products and few defective products, when all product data are used for training, a model is often obtained that infers all products as normal products. When an embodiment of the present invention is applied to automated product inspection, the processing circuit 10 can mark all or part of the data points corresponding to normal products as data subjects to determine defective products, and after determining the defective products, execute the defective product marking program to achieve automated detection of defects using artificial intelligence. For example, the method of executing process 2 of an embodiment of the present invention can also be applied to identify erroneous data in a database.

[0049] Those skilled in the art may combine, modify, or vary the above-described embodiments in accordance with the spirit of the present invention, without limitation. All of the above statements, steps, and / or processes (including suggested steps) may be implemented in hardware, software, firmware (i.e., a combination of hardware devices and computer instructions, where the data in the hardware devices is read-only software data), electronic systems, or combinations thereof. The device may be a data processing system 1. The hardware may include analog, digital, and mixed circuits (i.e., microcircuits, microchips, or silicon chips). For example, the hardware may be an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), a programmable logic element, coupled hardware components, or a combination of the above. In other embodiments, the hardware may include a general-purpose processor, a microprocessor, a controller, a digital signal processor (DSP), or a combination of the above. Software may be a combination of program code, instructions, and / or functions stored in a storage device, such as a computer-readable recording medium or a non-transitory computer-readable medium. For example, computer-readable recording media may include read-only memory (ROM), flash memory, random-access memory (RAM), subscriber identity module (SIM), hard disk, floppy disk, or compact disc read-only memory (CD-ROM / DVD-ROM / BD-ROM), but are not limited thereto. The data center system 1 of the embodiment of the present invention may include a processing circuit 10 and a storage device. The process steps and embodiments of the present invention may be compiled into program code or instructions and stored in the storage device. The processing circuit 10 may be used to read and execute the program code or instructions stored in the storage device to implement all the aforementioned steps and functions.

[0050] In summary, the present invention clusters a data set to generate multiple data groups, determines the most likely local outlier for the data group with the fewest data points, and then determines the most likely outlier from the most likely local outliers in the data group with the fewest data points based on the center of the data body. In this way, the present invention can provide accurate and true outlier values.

[0051] The above description is only a preferred embodiment of the present invention. All equivalent changes and modifications made according to the scope of the patent application of the present invention should fall within the scope of the present invention.

Claims

1. A data processing method, characterized in that: include: clustering a data set using a clustering method to generate a plurality of data groups, wherein the data set includes a plurality of data points; Selecting a data group having the least data points from the plurality of data groups; For each data group with the fewest data points, determine a local most likely outlier in the data group; and A most likely outlier is determined from the local most likely outliers of the data group with the least data points according to the center of a data body.

2. The data processing method according to claim 1, wherein: Also includes: For each data group with the least number of data points, determining the center of the data group and calculating the distance between each data point in the data group and the center of the data group; as well as A data point that is farthest from the center of the data group is determined and the data point that is farthest from the center of the data group is determined as the local most likely outlier of the data group.

3. The data processing method according to claim 1, wherein: Also includes: For the local most likely outliers of the data group with the fewest data points, calculating the distance between each local most likely outlier and the center of the data body; as well as A local most likely outlier with the greatest distance from the center of the data body is determined and the local most likely outlier with the greatest distance from the center of the data body is determined as the most likely outlier of the data set.

4. The data processing method according to claim 1, wherein: The plurality of data points of the data set are data points related to a current signal, and the data body includes data points whose current signal values ​​correspond to a first logic level and data points whose current signal values ​​correspond to a second logic level.

5. The data processing method according to claim 1, wherein: Also includes: The most likely outliers in the data set are deleted to perform a data cleaning process.

6. A data processing system, characterized in that: include: a database for storing a data set, wherein the data set includes a plurality of data points; a processing circuit coupled to the database, configured to obtain the data set and group the data set using a grouping method to generate a plurality of data groups; The processing circuit selects a data group with the least data points from the plurality of data groups, determines a local most likely outlier value of the data group for each data group with the least data points, and determines a most likely outlier value from the local most likely outliers of the data group with the least data points based on the center of a data body.

7. The data processing system according to claim 6, wherein: The processing circuit determines the center of each data group with the least data points and calculates the distance between each data point in the data group and the center of the data group. The processing circuit also determines a data point that is farthest from the center of the data group and determines the data point that is farthest from the center of the data group as the local most likely outlier of the data group.

8. The data processing system according to claim 6, wherein: The processing circuit calculates the distance between each local most likely outlier and the center of the data body for the local most likely outlier of the data group with the least data points, and the processing circuit determines the local most likely outlier that is farthest from the center of the data body and determines the local most likely outlier that is farthest from the center of the data body as the most likely outlier of the data set.

9. The data processing system according to claim 6, wherein: The plurality of data points of the data set are data points related to a current signal, and the data body includes data points whose data values ​​of the current signal correspond to a first logic level and data points whose data values ​​of the current signal correspond to a second logic level.

10. The data processing system according to claim 6, wherein: The processing circuit deletes the most likely outliers in the data set to perform a data cleaning process.

Images