Numerical table type data preprocessing method for CNN (Convolutional Neural Network)

Abstract

The invention belongs to the technical field of data processing and deep learning, and discloses a numerical table type data preprocessing method for a CNN (Convolutional Neural Network), which comprises a data acquisition module, an early-stage error correction module, a core color assignment module, a pixel sorting module and an image output module. The data acquisition module acquires original data of a multi-source multi-time sequence numerical table; the early-stage error correction module marks'no data 'and'unreasonable value' as pure black RGB (0, 0, 0) and pure white RGB (255, 255, 255) respectively; the core color giving module calculates RGB colors for the reasonable values according to the formula R = (nmod254) + 1, G = (n / 254mod254) + 1 and B = (n / 254mod254) + 1; the pixel sorting module arranges pixels according to the condition that the same time sequence is taken as a row and the same data source is taken as a column; the image output module generates an 'N * Z' size image. The problems that numerical table data is not matched with CNN input and abnormal data interferes with training are solved, data precision is reserved, coding is unified, a network structure is adapted, and the method is suitable for deep learning preprocessing of industrial sensors, environment monitoring and other scenes.

Description

Technical Field

[0001] This invention belongs to the field of data processing and deep learning interdisciplinary technology, specifically involving a preprocessing method for numerical table-type data that meets the input requirements of convolutional neural networks (CNNs). It is particularly suitable for structured numerical table data from multiple sources and time series (such as industrial sensor monitoring data, environmental factor collection data, medical physiological indicator data, etc.). By using unified anomaly marking, color coding, and pixel arrangement, the numerical table is converted into an image format that can be directly input by CNNs, solving the problems of poor adaptability and accuracy loss in traditional preprocessing methods. Background Technology

[0002] With the widespread application of deep learning technology in industries, environments, and medicine, convolutional neural networks (CNNs) are widely used for tasks such as data classification and anomaly detection due to their powerful spatial feature extraction capabilities. However, the input format of CNNs is image data (multi-channel pixel matrix), while in real-world scenarios, a large amount of data exists in the form of "numerical tables" (such as each row in a table corresponding to time, each column corresponding to a sensor, and cells containing monitoring values). The two formats are incompatible and require preprocessing for conversion.

[0003] Existing numerical table data preprocessing methods have the following shortcomings:

[0004] Poor format adaptability: Traditional methods often directly reshape numerical tables into single-channel matrices, ignoring the CNN's need for multi-channel features, resulting in the model's feature extraction capabilities not being fully utilized;

[0005] Non-standard anomaly handling: "No data" and "unreasonable values ​​beyond the range" were not uniformly marked. Some methods directly deleted abnormal data, resulting in data integrity loss. Some methods filled in with 0, which is easy to be confused with normal values ​​and interferes with model training.

[0006] Data precision loss: Normalization is used to compress values ​​to the 0-1 range before mapping colors, ignoring the precision differences in the original data (such as a 0.01% change in humidity), which leads to the masking of fine-grained features;

[0007] Weak versatility: Coding rules need to be redesigned for different data types (such as temperature, humidity, and pressure), making it difficult to quickly adapt to the needs of multiple scenarios.

[0008] Therefore, there is an urgent need for a numerical table-type data preprocessing method that can achieve "unified anomaly labeling, accurate color coding, and universal format adaptation" to meet the input requirements of CNN networks and improve the training efficiency and prediction accuracy of deep learning models. Summary of the Invention

[0009] 3.1 Purpose of the Invention

[0010] To address the problems of incompatibility between numerical table data and CNN input, non-standard anomaly handling, and loss of precision in existing technologies, this invention provides a method for preprocessing numerical table data for CNN networks. Through a complete process design of "anomaly marking - precise color assignment - rule sorting", it achieves efficient conversion of numerical tables into CNN image input while preserving data precision and integrity.

[0011] 3.2 Technical Solution

[0012] The core of this invention lies in solving the problem of abnormal data interference through a "preliminary correction module," achieving accurate multi-channel encoding of reasonable values ​​through a "core color assignment module," and adapting the image input format of CNN through a "pixel sorting module." The specific technical solutions are as follows:

[0013] 3.2.1 The data acquisition module is responsible for acquiring raw numerical data, supporting multiple sources (such as sensors A, B, and C) and multiple time series (such as time 1-320).

[0014] The data input can be in Excel, CSV, or a database table format, and must include three core fields: "Data Source Code", "Time Series Number", and "Value".

[0015] 3.2.2 Preliminary Errata Module

[0016] Design a unified labeling rule for the two types of abnormal data:

[0017] No data: This means that no data was collected by the corresponding sensor in this time series (such as blank values ​​caused by sensor failure). It is assigned pure black RGB (0,0,0) to facilitate CNN to quickly identify the "data missing" feature.

[0018] Unreasonable values: Index values ​​that exceed the physical / measurement range (e.g., soil moisture of 150% and temperature of 200℃) are assigned a pure color RGB (255,255,255) to create a sharp contrast with the black of "no data" and avoid model misjudgment.

[0019] Meanwhile, a reasonable value range and accuracy standard are preset: for example, soil moisture data, considering that the actual monitoring humidity range is 0-100%, the accuracy of the value needs to be retained at 0.01% to capture subtle changes. Therefore, the original value is converted into an integer n by multiplying by 100 (e.g., 57.32% is converted into 5732). The reasonable range corresponds to n=1-10000, ensuring that the accuracy is not lost after the data is standardized.

[0020] 3.2.3 Core Color Assignment Module

[0021] Based on three-channel RGB color encoding, precise numerical mapping is achieved using modulo 254 operations:

[0022] The choice of 254 instead of 255 as the modulus is to ensure that the calculated R, G, and B values ​​are in the range of 1-255, avoiding repetition with the "no data" RGB (0,0,0);

[0023] By designing the formulas R=(n mod 254)+1, G=(⌊n / 254⌋ mod 254)+1, and B=(⌊n / 254²⌋ mod 254)+1, the numerical range that can be covered is 1-254³=16387064, which is sufficient to meet the numerical accuracy requirements in most scenarios (such as temperature -40℃-125℃, accuracy 0.01℃, corresponding to n=0-16500, which is much smaller than 16387064).

[0024] Example: Soil moisture reading 57.32% (standardized n=5732), the calculation process is as follows:

[0025] R=(5732 mod 254)+1=5732-(254×22)+1=144+1=145

[0026] G=(⌊5732 / 254⌋ mod 254)+1=22 mod 254+1=23

[0027] B=(⌊5732 / 254²⌋ mod 254)+1=⌊5732 / 64516⌋ mod 254+1=0+1=1

[0028] The final color output is RGB (145,23,1), which uniquely corresponds to the humidity value of 57.32%, ensuring that accuracy is not lost.

[0029] 3.2.4 Pixel Sorting Module

[0030] Pixels are arranged according to the rule of "time as rows, source as columns" to form a structured image:

[0031] For example: if data source N=3 (sensors A, B, C) and data entries Z=5 (times 1-5), then the pixel arrangement is as follows:

[0032] A 5x3 matrix:

[0033] Line 1: Value A of time 1, value B of time 1, value C of time 1 Line 2: Value A of time 2, value B of time 2, value C of time 2

[0034] ...

[0035] Line 5: Value A of time 5, value B of time 5, value C of time 5

[0036] This arrangement allows CNNs to simultaneously capture both "horizontal correlations between different sources at the same time" and "vertical trends between different times from the same source," which aligns with the spatial feature extraction logic of CNNs.

[0037] 3.2.5 Image Output Module

[0038] Generate PNG format images, with the image size strictly following the formula "Number of data sources N × Number of data entries Z". For example, when N=20 and Z=320, output an image with 20 columns × 320 rows. This image can be directly input into the feature extraction layer of a CNN without additional size adjustment, simplifying the model preprocessing process. Detailed Implementation

[0039] The following example, "72-hour monitoring data from three temperature sensors (A, B, and C) in an industrial workshop," illustrates the implementation process of this invention in detail:

[0040] 4.1 Implementation Preparation

[0041] Data source: Sensors A, B, and C (N=3);

[0042] Time series: 72 hours, 1 data point per hour (Z=72);

[0043] Data type: Temperature, physical range -20℃ to 80℃, accuracy 0.01℃;

[0044] Reasonable value range: After standardization, n = (temperature + 20) × 100, corresponding to n = 0-10000 (-20℃→0, 80℃→10000);

[0045] Abnormal data: Sensor A had no data in the 10th hour, Sensor B had a data of 95℃ in the 25th hour (unreasonable value), and Sensor C had no abnormalities.

[0046] 4.2 Implementation Steps

[0047] Step 1: Data acquisition, obtain a 3×72 numerical table containing the fields "sensor code (A / B / C), time (1-72), temperature value";

[0048] Step 2: Preliminary Errata:

[0049] A10 (Sensor A, 10th hour): No data → Assign RGB (0,0,0);

[0050] B25 (25th hour of sensor B): 95℃ → n = (95 + 20) × 100 = 11500 > 10000 (unreasonable value) → Assign

[0051] RGB (255,255,255);

[0052] All other data are within the range of -20℃ to 80℃ → Proceed to step 3;

[0053] Step 3: Core coloring (taking C30 as an example, temperature 23.45℃):

[0054] Standardized n = (23.45 + 20) × 100 = 4345;

[0055] R=(4345 mod 254)+1: 254×17=4318, 4345-4318=27→R=27+1=28; G=(⌊4345 / 254⌋mod 254)+1: ⌊4345 / 254⌋=17→17 mod 254=17→G=17+1=18;

[0056] B=(⌊4345 / 254²⌋ mod 254)+1: 254²=64516, 4345<64516→⌊·⌋=0→B=0+1=1;

[0057] The RGB value of C30 is (28, 18, 1);

[0058] Step 4: Pixel sorting:

[0059] Lines: Time 1-72 corresponds to lines 1-72;

[0060] Columns: Sensors A, B, and C correspond to columns 1-3;

[0061] This forms a pixel matrix of 72 rows × 3 columns;

[0062] Step 5: Image output, generate a PNG format image (3 columns × 72 rows), and directly input it into the CNN network.

[0063] The "temperature anomaly detection" model was trained. The model can identify missing data in A10 using RGB (0,0,0) data.

[0064] RGB (255,255,255) identifies over-range anomalies in B25 and learns the variation pattern of normal temperature by using the RGB differences of reasonable values. Attached Figure Description

[0065] Figure 1 The overall flowchart of the preprocessing method of this invention includes the logical relationship between five modules: data acquisition, preliminary correction, core color assignment, pixel sorting, and image output.

[0066] 6. Beneficial effects compared to existing technologies

[0067] Strong format adaptability: Directly generates image formats that can be input into CNNs without modifying the network structure, resolving the format conflict problem between traditional numerical tables and CNNs;

[0068] Clear anomaly labeling: Use pure black / pure white to clearly distinguish between "no data" and "unreasonable value" to prevent the model from misclassifying outliers as normal data and improve training accuracy;

[0069] Precision is fully preserved: By covering a numerical range of 254³ and normalizing integers, fine-grained differences in the original data (such as temperature changes of 0.01℃) are preserved, avoiding the loss of precision caused by normalization;

[0070] Wide applicability: It can adapt to various numerical data such as temperature, humidity, pressure, and pH value by simply adjusting the "reasonable value range" and "accuracy standard" without redesigning the coding logic;

[0071] Significant efficiency improvement: The entire process requires no manual intervention and can process large-scale numerical table data in batches (such as 100,000 data points with N=100 and Z=1000), reducing preprocessing time by more than 60% compared to traditional methods.

Claims

1. A method for preprocessing numerical table-type data for convolutional neural networks (CNNs), characterized in that, Includes the following steps: Step 1: Data Acquisition The raw data of numerical tables is acquired through the data acquisition module. The raw data includes at least one data source distinguished by letters or numbers and at least one time series distinguished by numbers, and each data source corresponds to the values ​​of at least one time series. Step 2: Preliminary Errata The preliminary errata module marks the original data for anomalies, with the following specific rules: 2.1 If the data is in a "no data" state, meaning there are no collected values ​​from the corresponding data source for this time series, then the pixel corresponding to the data is assigned pure black, with an RGB color value of (0,0,0); 2.2 If the data is an "unreasonable value," meaning the value is outside the preset reasonable value range, then the pixel corresponding to the data is assigned pure white, with an RGB color value of (255,255,255); 2.3 If the data is a "reasonable value," meaning the value is within the preset reasonable value range, then proceed to Step 3. Step 3: Core Color Assignment. The core color assignment module calculates the RGB colors for reasonable values. The specific process is as follows: 3.1 Perform precision standardization on reasonable values, converting the original values ​​into integers n. The conversion rule is n = original value × (1 / value precision). The value precision is preset according to the data type and must retain at least two decimal places. 3.2 Calculate the RGB three-channel color values ​​based on the following formulas, with the color range limited to 0-255: Red channel R = (n mod 254) + 1 (Formula 1); Green channel G = (⌊n / 254⌋ mod 254) + 1 (Formula 2); Blue channel B = (⌊n / 254²⌋ mod 254) + 1 (Formula 3); where "mod" represents the modulo operation and "⌊・⌋" represents the floor operation. Step 4: Pixel sorting. The pixel sorting module arranges the pixels according to preset rules, as follows: 4.1 Pixels corresponding to all data sources belonging to the same time series are arranged in the same row; 4.2 Pixels corresponding to all time series belonging to the same data source are arranged in the same column. Step 5: Image Output. The image output module generates a preprocessed image with a size of "Number of data sources N × Number of data entries Z", where N represents the total number of data sources and Z represents the total number of time series data. The image format is PNG, and the output image is used as input data for the CNN network.

2. The numerical table-type data preprocessing method for CNN networks according to claim 1, characterized in that, The "preset reasonable value range" mentioned in step 2.3 is set differently according to the data type: if the data is soil moisture, its reasonable value range is 0.01% to 100%, the accuracy is 0.01, and the reasonable range of the standardized integer n is 1 to 10000; if the data is temperature, its reasonable value range is -40℃ to 125℃, the accuracy is 0.01, and the reasonable range of the standardized integer n is 0 to 16500. This standardization process realizes the conversion of negative temperature to positive integer through the calculation method of n=(original temperature + 40)×100.

3. The numerical table-type data preprocessing method for CNN networks according to claim 1, characterized in that, In step 3.1, if the normalized integer n is greater than 254³-1 (i.e. 16387063), then n needs to be processed by "n mod 254³", and then substituted into Equations 1, 2 and 3 in step 3.2 to calculate the RGB value, so as to ensure that the three-channel color values ​​do not exceed the range of 0-255.

4. The numerical table-type data preprocessing method for CNN networks according to claim 1, characterized in that, The specific rules for pixel sorting in step 4 are as follows: the row index is arranged in ascending order of the time series number, where time 1 corresponds to the 1st row, time 2 corresponds to the 2nd row, and so on, until time Z corresponds to the Zth row; the column index is arranged in lexicographical order of the data source code, where code A corresponds to the 1st column, code B corresponds to the 2nd column, and so on, until code N corresponds to the Nth column.

5. The numerical table-type data preprocessing method for CNN networks according to claim 1, characterized in that, The preliminary correction module also integrates an "abnormal log recording" function. For data marked as "no data" and "unreasonable value", this function will record the corresponding data source code, time series number and specific abnormality type. The recorded information can be used for subsequent data tracing analysis and model performance optimization.

Images