Data preprocessing method and preprocessing system

Abstract

The invention relates to the technical field of data processing, and discloses a data preprocessing method and a preprocessing system. The preprocessing system comprises a data acquisition management module and a data calibration management module. By using a data recognition algorithm to perform abnormal value analysis and comparison on calibrated images, sounds and numerical data with stored image abnormality data, sound abnormality data and numerical abnormality data, intelligent recognition and screening of abnormal images, sounds and numerical data in the data acquisition stage are realized, thereby improving the scientificity and reliability of data preprocessing. Secondary data acquisition is performed on the recognized image abnormality data, sound abnormality data and numerical abnormality data according to data numbers using an acquisition device, and secondary abnormal data analysis and feedback are performed on the acquired images, sounds and numerical data, thereby realizing abnormal data self-checking and abnormal data feedback of data acquisition, thereby improving the intelligence and accuracy of data preprocessing.

Description

Technical Field

[0001] The present invention relates to the technical field of data processing, and in particular to a data preprocessing method and a preprocessing system. Background Art

[0002] Data preprocessing refers to the necessary processing such as review, screening, and sorting before classifying or grouping the collected data; for example, seismic data preprocessing includes vertical stacking, rearrangement, adding headers, editing, resampling, multi-channel editing, etc. The collected data in the real world is often messy, which is manifested in the following ways: Incompleteness: missing or uncertain data attribute values. Due to the different sources of original data, there is a lack of unified standards for data definition, resulting in inconsistent data connotations between systems; Redundancy: duplication of data records or attributes. This type of data cannot be directly mined, or the mining results are unsatisfactory. There are many methods for data preprocessing: data cleaning, data integration, data transformation, data reduction, etc. These data processing technologies are used before data mining, which greatly improves the quality of data mining models and reduces the time required for actual mining. With the development of technologies such as big data, the Internet of Things, artificial intelligence, and intelligent manufacturing, data has become a very important resource. Data use includes data collection, data organization, data analysis, data application, and data evaluation. Current data preprocessing mainly focuses on smoothing and denoising the collected data, deleting outliers, storing, transforming data formats, and reducing data. However, the accuracy and authenticity of data collection are what determine whether the data has use value. The current data collection method in production and life directly obtains data through sensor equipment or monitoring cameras. The acquired data undergoes conventional data preprocessing, and no calibration analysis of the data is performed during the data collection stage, resulting in large errors and low accuracy in the collected data. At the same time, there is a lack of identification and reprocessing of outliers in the collected data, which leads to the omission of important data and reduces the effect of data preprocessing.

[0003] A Chinese invention patent with publication number CN103077163B discloses a data preprocessing method, device and system, which divides text data into at least one data unit by using an anchor character in a preset anchor character set, each data unit does not include the anchor character, and the number of characters in each data unit does not exceed a preset unit character upper limit; combines the at least one data unit into at least one data block, each data block includes at least one data unit and the number of characters in each data block does not exceed a preset block character threshold, so as to perform fingerprint extraction processing on the at least one data block. By using anchor characters, that is, segmenting text data according to the grammatical and semantic information of natural language, removing characters with low semantic distinction, obtaining data units, and combining the data units into data blocks, the extracted fingerprint is used for DLP technology, which improves the accuracy of data leakage detection. At the same time, the data block is a two-level structure data of data unit-block, and the number of characters in each data block is basically consistent, which can improve the efficiency of fingerprint extraction. However, in the fingerprint collection process, there is a lack of feature calibration analysis of fingerprint data, which makes the collected fingerprint data prone to erroneous data; at the same time, there is a lack of recognition and reprocessing of outliers in the collected data, which leads to the omission of important data and reduces the accuracy and quality of data preprocessing. Summary of the invention

[0004] 1. Technical issues to be solved

[0005] In order to solve the above-mentioned problem that the current data collection method in production and life directly obtains data through sensor equipment or monitoring cameras, and the acquired data undergoes conventional data preprocessing, the data is not calibrated and analyzed during the data collection stage, resulting in large errors and low accuracy in the collected data; at the same time, there is a lack of identification and reprocessing of outliers in the collected data, resulting in the omission of important data and reducing the effect of data preprocessing, the purpose of achieving the above data collection calibration preprocessing, abnormal data reprocessing, and improving data collection accuracy and preprocessing accuracy.

[0006] (II) Technical solution

[0007] The present invention is implemented by the following technical solution: a data preprocessing method, the method comprising the following steps:

[0008] S1, collecting image type feature data, sound type feature data, and numerical type feature data;

[0009] S2, performing image and sound dirty data calibration analysis on the image type feature data and image type data calibration dirty data, and on the sound type feature data and sound type data calibration dirty data, respectively, clearing the image and sound dirty data in the image type feature data and the sound type feature data, collecting the remaining image and sound data in the image type feature data and the sound type feature data, and generating corresponding image type preprocessing feature data and sound type preprocessing feature data;

[0010] S3, performing numerical dirty data calibration analysis on the numerical type characteristic data and the numerical type data calibration dirty data, removing the numerical dirty data in the numerical type characteristic data, collecting the remaining numerical data in the numerical type characteristic data and generating numerical type preprocessing characteristic data;

[0011] S4, performing image and sound abnormal data analysis respectively according to the image type preprocessing feature data and abnormal image feature data, and the sound type preprocessing feature data and abnormal sound feature data, filtering out the image and sound abnormal data in the image type preprocessing feature data and the sound type preprocessing feature data, and generating image type abnormal feature data and sound type abnormal feature data;

[0012] S5, performing numerical abnormal data analysis based on the numerical type preprocessing feature data and the abnormal numerical feature data, filtering out the numerical abnormal data in the numerical type preprocessing feature data, and generating numerical type abnormal feature data;

[0013] S6, obtaining the image type abnormal feature data, the sound type abnormal feature data, and the numerical type abnormal feature data;

[0014] S7. Re-collect the acquired image type abnormal feature data, sound type abnormal feature data, and numerical type abnormal feature data according to the data numbers using the collection device to generate image reprocessing data, sound reprocessing data, and numerical reprocessing data. The image reprocessing data and sound reprocessing data are respectively subjected to step S4 to re-analyze the image and sound abnormal data, and the numerical reprocessing data is subjected to step S5 to re-analyze the numerical abnormal data, and the abnormal data processing operation is performed according to the analysis results.

[0015] Preferably, the steps of collecting image type feature data, sound type feature data, and numerical type feature data are as follows:

[0016] S11. Use cloud lens to collect image type feature data of specific application objects online and establish a collection where a m Represents the image type feature data collected by the mth cloud lens, Indicates the maximum number of cloud lenses for collecting image type feature data;

[0017] Use sound sensors to collect sound type feature data of specific application objects online and establish a set A'=(a'1,…,a' n ,…,a' θ ), n=1,2,3,...,θ; where a' n represents the sound type feature data collected by the nth sound sensor, and θ represents the maximum number of sound sensors collecting sound type feature data;

[0018] Use parameter sensors to collect numerical characteristic data of specific application objects online and establish a collection where a' p ' represents the numerical characteristic data collected by the pth parameter sensor, Indicates the maximum number of parameter sensors that collect numerical type feature data. The application objects include specific data collection and use fields in industry, agriculture and service industries. The parameter sensors include one or more of temperature, humidity, flow, rotation speed and vibration.

[0019] Preferably, the steps of performing image and sound dirty data calibration analysis on the image type feature data and image type data calibration dirty data, and on the sound type feature data and sound type data calibration dirty data, clearing the image and sound dirty data in the image type feature data and the sound type feature data, collecting the remaining image and sound data in the image type feature data and the sound type feature data, and generating corresponding image type preprocessing feature data and sound type preprocessing feature data are as follows:

[0020] S21, construct image type data calibration dirty data set B = (b1, ..., b u ,…,b ρ ), u=1,2,3,…,ρ; where b u represents the dirty data of the u-th image type data calibration, ρ represents the maximum number of dirty data of the image type data calibration;

[0021] Construct sound type data calibration dirty data set B'=(b'1,…,b' v ,…,b' σ ), v = 1, 2, 3, ..., σ; where b' v represents the vth dirty data of sound type data calibration, σ represents the maximum number of dirty data of sound type data calibration;

[0022] S22, using a data analysis algorithm to analyze the image type feature data a in the image type feature data set A mAlign dirty data b with image type data in dirty data set B u Perform image dirty data calibration traversal analysis and clear the image type data calibration dirty data b in the image type feature data set A u , collect the remaining image type feature data a in the image type feature data set A m And generate image type preprocessing feature data set where a m Represents the image type preprocessing feature data corresponding to the image type feature data collected by the m-th cloud lens after data calibration; Indicates the image type preprocessing feature data corresponding to the image type feature data collected by the m1-th cloud lens after data calibration; The image type feature data collected by the m2-th cloud lens is represented by the image type preprocessing feature data corresponding to the data calibration. The operation steps of the data analysis algorithm to perform image dirty data calibration traversal analysis are as follows:

[0023] S221, initialize algorithm parameters, population size N, maximum number of iterations T;

[0024] S222, initializing the population, calculating fitness, and determining image search pathfinders and image search followers;

[0025] S223, according to the formula Update the image search pathfinder position in the search space of the image type feature data set A, where t represents the current iteration number of the algorithm and T is the maximum iteration number of the algorithm; represents the position of the contemporary image search pathfinder in the search space of the image type feature data set A in the t-th iteration, represents the position of the previous generation image search pathfinder in the search space of the image type feature data set A in the t-1th iteration, represents the position of the image search pathfinder in the search space of the image type feature data set A after the update in the t+1th iteration, ε is the step size factor of the image search pathfinder, and ε follows a uniform distribution in [0,1];

[0026] S224, according to the formula Update the image search follower's position in the image type feature data set A search space, where represents the current position of the image search follower in the search space of the image type feature data set A in the tth iteration, represents its updated position in the search space of the image type feature data set A in the t+1th iteration; represents the current position of other image search followers in the search space of image type feature data set A in the tth iteration, Image Search Followers move not only with Image Search Pathfinder Related and followed by other image search followers , R1 represents the position distance parameter between image search followers in the search space of image type feature data set A, R2 represents the position distance parameter between image search pathfinder and image search follower in the search space of image type feature data set A, R1=α×r1, R2=β×r2; α represents the interaction coefficient between image search followers, β represents the attraction coefficient of image search pathfinder to image search follower, α and β are uniformly distributed in [1,2]; r1 is the step factor of the movement of image search follower and other image search followers, r2 is the step factor of the movement of image search follower and image search pathfinder, r1 and r2 are both random numbers in the range of [0,1];

[0027] S225, calculate the fitness value and update the global optimal value, that is, search for dirty data b that is calibrated with the image type data in the search space of the image type feature data set A u Matched image type feature data a m ;

[0028] S226, when the maximum number of iterations is met, the image type feature data set A and the image type data calibration dirty data b are combined. u The image type feature data a that matches successfully m Clear, collect the remaining image type feature data a in the image type feature data set A m And generate image type preprocessing feature data set Otherwise, repeat steps S221-S226;

[0029] S23, using step S22, the sound type feature data a′ in the sound type feature data set A′ n , and the sound type data calibration dirty data b' in the sound type data calibration dirty data set B v Perform sound dirty data calibration traversal analysis and clear the sound type data calibration dirty data b' in the sound type feature data set A' v , collect the sound type feature data to establish the remaining sound type feature data a′ in the set A′ n And generate sound type preprocessing feature data set 1≤n1≤n, n≤n2≤θ; where a′ n Indicates the sound type preprocessing feature data corresponding to the sound type feature data collected by the nth sound sensor after data calibration; Indicates the sound type preprocessing feature data corresponding to the sound type feature data collected by the n1th sound sensor after data calibration; It represents the sound type preprocessing feature data corresponding to the sound type feature data collected by the n2th sound sensor after data calibration.

[0030] Preferably, the steps of performing numerical dirty data calibration analysis on the numerical type characteristic data and the numerical type data calibration dirty data, removing the numerical dirty data in the numerical type characteristic data, collecting the remaining numerical data in the numerical type characteristic data and generating numerical type preprocessing characteristic data are as follows:

[0031] S31. Constructing a dirty data set for numerical data calibration Where b″ o Indicates the calibration dirty data of the oth numerical type data. Indicates the maximum number of dirty data for calibration of numerical type data, where the dirty data includes one or more of image missing values, image duplicate data, sound missing values, sound duplicate data, numerical missing values, and numerical duplicate data;

[0032] S32, using step S22, the numerical type feature data a″ in the numerical type feature data set A″ p , and the dirty data set B of the numerical type data calibration dirty data b″ o Perform numerical dirty data calibration traversal analysis and clear the numerical type data calibration dirty data b″ in the numerical type feature data set A″ o , collect the remaining numerical type feature data a″ in the numerical type feature data set A″ p And generate a numerical type preprocessing feature data set where a″ p Indicates the numerical type preprocessing characteristic data corresponding to the data calibration of the numerical type characteristic data collected by the p-th parameter sensor; Indicates the numerical type preprocessing characteristic data corresponding to the data calibration of the numerical type characteristic data collected by the p1th parameter sensor; It represents the numerical type preprocessing characteristic data corresponding to the numerical type characteristic data collected by the p2th parameter sensing after data calibration.

[0033] Preferably, the steps of performing image and sound abnormal data analysis respectively based on the image type preprocessing feature data and abnormal image feature data, and the sound type preprocessing feature data and abnormal sound feature data, filtering out the image and sound abnormal data in the image type preprocessing feature data and the sound type preprocessing feature data, and generating image type abnormal feature data and sound type abnormal feature data are as follows:

[0034] S41, establish abnormal image feature data set C = (c1, ..., c z ,…,c τ ), z=1,2,3,…,τ; where c z represents the zth abnormal image feature data, τ represents the maximum number of abnormal image feature data, and the abnormal image feature data represents the unconventional image feature data of the specific application object collected but does not belong to dirty data;

[0035] Establish abnormal sound feature data set C'=(c'1,…,c' y ,…,c' ω ), y=1,2,3,…,ω; where c′ y represents the yth abnormal sound feature data, ω represents the maximum number of abnormal sound feature data, and the abnormal sound feature data represents the unconventional sound feature data collected for a specific application object but does not belong to dirty data;

[0036] S42, using a depth-first search algorithm to preprocess the image type feature data set A yu Image type preprocessing feature data a m One by one with the abnormal image feature data c in the abnormal image feature data set C z Perform image abnormality data analysis to filter out image type preprocessing feature data set A yu Abnormal image feature data in and generate image type abnormal feature data when a m With c z If no analysis and matching is successful, the image type abnormal feature data analysis ends;

[0037] The sound type preprocessing feature data set A' is obtained by using a depth-first search algorithm. yu The sound type preprocessing feature data a′ n One by one, compare the abnormal sound feature data c' in the abnormal sound feature data set C y Perform sound abnormality data analysis to filter out sound type preprocessing feature data set A' yu Abnormal sound feature data in the sound type and generate abnormal feature data of the sound type When a′ n and c′y If no analysis or matching is successful, the analysis of abnormal sound type feature data is terminated.

[0038] Preferably, the steps of performing numerical abnormal data analysis based on the numerical type preprocessing feature data and the abnormal numerical feature data, filtering out the numerical abnormal data in the numerical type preprocessing feature data, and generating the numerical type abnormal feature data are as follows:

[0039] S51, establish abnormal numerical feature data set C" = (c'1', ..., c" q ,…,c” ψ ), q=1,2,3,…,ψ; where c″ q represents the qth abnormal numerical feature data, ψ represents the maximum number of abnormal numerical feature data, and the abnormal numerical feature data represents the unconventional numerical feature data of the specific application object collected but does not belong to dirty data;

[0040] S52, using a depth-first search algorithm to preprocess the feature data set A of the numerical type" yu Preprocessing feature data a" of numerical type p One by one, compare the abnormal numerical feature data c″ in the abnormal numerical feature data set C″ q Perform numerical abnormal data analysis to filter out numerical type preprocessing feature data set A" yu Abnormal numerical feature data in and generate numerical type abnormal feature data when a″ p With c″ q If no analysis or matching is successful, the analysis of abnormal characteristic data of numerical type ends.

[0041] Preferably, the steps of obtaining the image type abnormal feature data, the sound type abnormal feature data, and the numerical type abnormal feature data are as follows:

[0042] S61. Obtaining abnormal feature data of image type Abnormal sound type feature data Abnormal characteristic data of numerical type

[0043] Preferably, the acquired image type abnormal feature data, sound type abnormal feature data, and numerical type abnormal feature data are respectively collected again according to the data numbers using a collection device to generate image reprocessing data, sound reprocessing data, and numerical reprocessing data, and the image reprocessing data and sound reprocessing data are respectively subjected to step S4 to reanalyze the image and sound abnormal data, and the numerical reprocessing data is subjected to step S5 to reanalyze the numerical abnormal data, and the operation steps of performing the abnormal data processing operation according to the analysis results are as follows:

[0044] S71, image type abnormal feature data Abnormal sound type feature data Abnormal characteristic data of numerical type Re-collect the corresponding image type feature data, sound type feature data, and numerical type feature data according to the cloud lens collection number m, the sound sensor collection number n, and the parameter sensor collection number p, and generate the image type feature data, sound type feature data, and numerical type feature data as image reprocessing data Sound reprocessing data Numerical reprocessing of data

[0045] S72, reprocessing the image data Sound reprocessing data Execute step S42, when the image type abnormal feature data or the sound type abnormal feature data is generated again, the image type abnormal feature data Or abnormal characteristic data of sound type Push information and feedback through IoT communication;

[0046] The numerical reprocessing data Execute step S52, when the numerical type abnormal characteristic data is generated again, the numerical type abnormal characteristic data is Information push feedback is carried out through IoT communication.

[0047] A preprocessing system for implementing the data preprocessing method.

[0048] (III) Beneficial effects

[0049] The present invention provides a data preprocessing method and a preprocessing system, which have the following beneficial effects:

[0050] 1. Through the cooperation of image type data acquisition units, sound type data acquisition units, and numerical type data acquisition units, images, sounds, and numerical data of specific application objects in production and life are accurately collected online, thereby realizing scientific data collection in the data preprocessing process and improving the efficiency of data preprocessing. At the same time, it provides reliable data support for the calibration processing of images, sounds, and numerical data, and promotes the progress and timeliness of data preprocessing.

[0051] 2. The image data calibration analysis and collection unit, the sound data calibration analysis and collection unit, and the numerical data calibration analysis and collection unit cooperate with each other to perform data calibration analysis on the collected images, sounds, and numerical data with the set image dirty data, sound dirty data, and numerical dirty data, thereby intelligently filtering out useless data in the data collection stage, improving the quality of data collection, reducing the workload and errors of subsequent data processing, and improving the accuracy and efficiency of data preprocessing.

[0052] 3. The image type data anomaly analysis unit, the sound type data anomaly analysis unit, and the numerical type data anomaly analysis unit cooperate with each other to perform abnormal value analysis and comparison on the calibrated images, sounds, and numerical data with the stored image abnormality data, sound abnormality data, and numerical abnormal data using the data recognition algorithm, thereby realizing intelligent recognition and screening of abnormal images, sounds, and numerical data in the data collection stage, and improving the scientificity and reliability of data preprocessing; the abnormal data reprocessing execution unit uses the collection equipment to perform secondary data collection on the identified image abnormality data, sound abnormality data, and numerical abnormal data according to the data number, and performs secondary abnormal data analysis and feedback on the collected images, sounds, and numerical data, thereby realizing abnormal data self-inspection and abnormal data feedback in data collection, and improving the intelligence and accuracy of data preprocessing. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 A schematic diagram of a module of a data preprocessing system provided by the present invention;

[0054] Figure 2 The present invention provides a flow chart of a data preprocessing method. DETAILED DESCRIPTION

[0055] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0056] The embodiments of the data preprocessing method and preprocessing system are as follows:

[0057] See also Figure 1 - Figure 2 , a data preprocessing method, the method comprising the following steps:

[0058] S1, collecting image type feature data, sound type feature data, and numerical type feature data;

[0059] S2, performing image and sound dirty data calibration analysis on the image type feature data and the image type data calibration dirty data, and on the sound type feature data and the sound type data calibration dirty data, respectively, clearing the image and sound dirty data in the image type feature data and the sound type feature data, collecting the remaining image and sound data in the image type feature data and the sound type feature data, and generating corresponding image type preprocessing feature data and sound type preprocessing feature data;

[0060] S3, performing numerical dirty data calibration analysis on the numerical type feature data and the numerical type data calibration dirty data, removing the numerical dirty data in the numerical type feature data, collecting the remaining numerical data in the numerical type feature data and generating numerical type preprocessing feature data;

[0061] S4, performing image and sound abnormal data analysis respectively according to the image type preprocessing feature data and abnormal image feature data, as well as the sound type preprocessing feature data and abnormal sound feature data, filtering out the image and sound abnormal data in the image type preprocessing feature data and the sound type preprocessing feature data, and generating image type abnormal feature data and sound type abnormal feature data;

[0062] S5, performing numerical abnormal data analysis based on the numerical type preprocessing feature data and the abnormal numerical feature data, filtering out the numerical abnormal data in the numerical type preprocessing feature data, and generating numerical type abnormal feature data;

[0063] S6. Acquire image type abnormal feature data, sound type abnormal feature data, and numerical type abnormal feature data;

[0064] S7. Use the acquisition equipment to re-collect the acquired image type abnormal feature data, sound type abnormal feature data, and numerical type abnormal feature data according to the data number, to generate image reprocessing data, sound reprocessing data, and numerical reprocessing data. Perform step S4 on the image reprocessing data and sound reprocessing data to re-analyze the image and sound abnormal data, and perform step S5 on the numerical reprocessing data to re-analyze the numerical abnormal data, and perform abnormal data processing operations based on the analysis results.

[0065] For further information, see Figure 1 - Figure 2 The steps for collecting image type feature data, sound type feature data, and numerical type feature data are as follows:

[0066] S11. Use cloud lens to collect image type feature data of specific application objects online and establish a collection where a m Represents the image type feature data collected by the mth cloud lens, Indicates the maximum number of cloud lenses for collecting image type feature data;

[0067] Use sound sensors to collect sound type feature data of specific application objects online and establish a set A'=(a'1,…,a' n ,…,a' θ ), n=1,2,3,...,θ; where a' n represents the sound type feature data collected by the nth sound sensor, and θ represents the maximum number of sound sensors collecting sound type feature data;

[0068] Use parameter sensors to collect numerical characteristic data of specific application objects online and establish a collection where a″ p Indicates the numerical characteristic data collected by the pth parameter sensor, Indicates the maximum number of parameter sensors that collect numerical characteristic data. Application targets include specific data collection and use areas in industry, agriculture, and service industries. Parameter sensors include one or more of temperature, humidity, flow, speed, and vibration.

[0069] Through the cooperation of image type data acquisition units, sound type data acquisition units and numerical type data acquisition units, images, sounds and numerical data of specific application objects in production and life are accurately collected online, thereby realizing scientific data collection in the data preprocessing process and improving the efficiency of data preprocessing. At the same time, it provides reliable data support for data calibration processing in images, sounds and numerical values, and promotes the progress and timeliness of data preprocessing.

[0070] For further information, see Figure 1 - Figure 2 , image type feature data and image type data calibration dirty data, as well as sound type feature data and sound type data calibration dirty data are respectively subjected to image and sound dirty data calibration analysis, image type feature data and sound type feature data are cleared of image and sound dirty data, and the remaining image and sound data in the image type feature data and sound type feature data are collected to generate corresponding image type preprocessing feature data and sound type preprocessing feature data. The operation steps are as follows:

[0071] S21, construct image type data calibration dirty data set B = (b1, ..., b u ,…,b ρ ), u=1,2,3,…,ρ; where b u represents the dirty data of the u-th image type data calibration, ρ represents the maximum number of dirty data of the image type data calibration;

[0072] Construct sound type data calibration dirty data set B'=(b'1,…,b'v ,…,b' σ ), v = 1, 2, 3, ..., σ; where b' v represents the vth dirty data of sound type data calibration, σ represents the maximum number of dirty data of sound type data calibration;

[0073] S22, using a data analysis algorithm to analyze the image type feature data a in the image type feature data set A m Align dirty data b with image type data in dirty data set B u Perform image dirty data calibration traversal analysis and clear the image type data calibration dirty data b in the image type feature data set A u , collect the remaining image type feature data a in the image type feature data set A m And generate image type preprocessing feature data set where a m Represents the image type preprocessing feature data corresponding to the image type feature data collected by the m-th cloud lens after data calibration; Indicates the image type preprocessing feature data corresponding to the image type feature data collected by the m1-th cloud lens after data calibration; The image type feature data collected by the m2-th cloud lens is represented by the image type preprocessing feature data corresponding to the data calibration. The operation steps of the data analysis algorithm to perform image dirty data calibration traversal analysis are as follows:

[0074] S221, initialize algorithm parameters, population size N, maximum number of iterations T;

[0075] S222, initializing the population, calculating fitness, and determining image search pathfinders and image search followers;

[0076] S223, according to the formula Update the image search pathfinder position in the search space of the image type feature data set A, where t represents the current iteration number of the algorithm and T is the maximum iteration number of the algorithm; represents the position of the contemporary image search pathfinder in the search space of the image type feature data set A in the t-th iteration, represents the position of the previous generation image search pathfinder in the search space of the image type feature data set A in the t-1th iteration, represents the position of the image search pathfinder in the search space of the image type feature data set A after the update in the t+1th iteration, ε is the step size factor of the image search pathfinder, and ε follows a uniform distribution in [0,1];

[0077] S224, according to the formula Update the image search follower's position in the image type feature data set A search space, where represents the current position of the image search follower in the search space of the image type feature data set A in the tth iteration, represents its updated position in the search space of the image type feature data set A in the t+1th iteration; represents the current position of other image search followers in the search space of image type feature data set A in the tth iteration, Image Search Followers move not only with Image Search Pathfinder Related and followed by other image search followers , R1 represents the position distance parameter between image search followers in the search space of image type feature data set A, R2 represents the position distance parameter between image search pathfinder and image search follower in the search space of image type feature data set A, R1=α×r1, R2=β×r2; α represents the interaction coefficient between image search followers, β represents the attraction coefficient of image search pathfinder to image search follower, α and β are uniformly distributed in [1,2]; r1 is the step factor of the movement of image search follower and other image search followers, r2 is the step factor of the movement of image search follower and image search pathfinder, r1 and r2 are both random numbers in the range of [0,1];

[0078] S225, calculate the fitness value and update the global optimal value, that is, search for dirty data b that is calibrated with the image type data in the search space of the image type feature data set A u Matched image type feature data a m ;

[0079] S226, when the maximum number of iterations is met, the image type feature data set A and the image type data calibration dirty data b are combined. u The image type feature data a that matches successfully m Clear and collect the remaining image type feature data a in the image type feature data set A m And generate image type preprocessing feature data set Otherwise, repeat steps S221-S226;

[0080] S23, using step S22, the sound type feature data a′ in the sound type feature data set A′ n , and the sound type data calibration dirty data b' in the sound type data calibration dirty data set B' v Perform sound dirty data calibration traversal analysis and clear the sound type data calibration dirty data b' in the sound type feature data set A' v, collect the sound type feature data to establish the remaining sound type feature data a′ in the set A′ n And generate sound type preprocessing feature data set 1≤n1≤n, n≤n2≤θ; where a′ n Indicates the sound type preprocessing feature data corresponding to the sound type feature data collected by the nth sound sensor after data calibration; Indicates the sound type preprocessing feature data corresponding to the sound type feature data collected by the n1th sound sensor after data calibration; It represents the sound type preprocessing feature data corresponding to the sound type feature data collected by the n2th sound sensor after data calibration.

[0081] The steps of performing numerical dirty data calibration analysis on numerical type feature data and numerical type data calibration dirty data, removing numerical dirty data from numerical type feature data, collecting remaining numerical data from numerical type feature data and generating numerical type preprocessing feature data are as follows:

[0082] S31. Constructing a dirty data set for numerical data calibration Where b″ o Indicates the calibration dirty data of the oth numerical type data. Indicates the maximum number of dirty data for numerical data calibration, where dirty data includes one or more of image missing values, image duplicate data, sound missing values, sound duplicate data, numerical missing values, and numerical duplicate data;

[0083] S32, using step S22, the numerical type feature data a″ in the numerical type feature data set A″ p , and the dirty data set B″ in which the dirty data of the numerical type is aligned is the dirty data b″ of the numerical type. o Perform numerical dirty data calibration traversal analysis and clear the numerical type data calibration dirty data b″ in the numerical type feature data set A″ o , collect the remaining numerical type feature data a″ in the numerical type feature data set A″ p And generate a numerical type preprocessing feature data set 1≤p1≤p, where a″ p Indicates the numerical type preprocessing characteristic data corresponding to the data calibration of the numerical type characteristic data collected by the p-th parameter sensor; Indicates the numerical type preprocessing characteristic data corresponding to the data calibration of the numerical type characteristic data collected by the p1th parameter sensor; It represents the numerical type preprocessing characteristic data corresponding to the numerical type characteristic data collected by the p2th parameter sensing after data calibration.

[0084] The image data calibration analysis and collection unit, the sound data calibration analysis and collection unit, and the numerical data calibration analysis and collection unit cooperate with each other to perform data calibration analysis on the collected images, sounds, and numerical data with the set image dirty data, sound dirty data, and numerical dirty data, thereby realizing intelligent screening of useless data in the data collection stage, improving the quality of data collection, reducing the workload and errors of subsequent data processing, and improving the accuracy and efficiency of data preprocessing.

[0085] For further information, see Figure 1 - Figure 2 , according to the image type preprocessing feature data and abnormal image feature data, as well as the sound type preprocessing feature data and abnormal sound feature data, respectively, the image and sound abnormal data in the image type preprocessing feature data and the sound type preprocessing feature data are screened out, and the operation steps of generating the image type abnormal feature data and the sound type abnormal feature data are as follows:

[0086] S41, establish abnormal image feature data set C = (c1, ..., c z ,…,c τ ), z=1,2,3,…,τ; where c z represents the zth abnormal image feature data, τ represents the maximum number of abnormal image feature data, and the abnormal image feature data represents the unconventional image feature data collected for a specific application object but does not belong to dirty data;

[0087] Establish abnormal sound feature data set C'=(c'1,…,c' y ,…,c' ω ), y=1,2,3,…,ω; where c′ y represents the yth abnormal sound feature data, ω represents the maximum number of abnormal sound feature data, and the abnormal sound feature data represents the unconventional sound feature data collected for a specific application object but does not belong to dirty data;

[0088] S42, using a depth-first search algorithm to preprocess the image type feature data set A yu Image type preprocessing feature data a m One by one with the abnormal image feature data c in the abnormal image feature data set C z Perform image abnormality data analysis to filter out image type preprocessing feature data set A yu Abnormal image feature data in and generate image type abnormal feature data when a m With c zIf no analysis and matching is successful, the image type abnormal feature data analysis ends;

[0089] The sound type preprocessing feature data set A' is obtained by using a depth-first search algorithm. yu The sound type preprocessing feature data a′ n One by one, compare the abnormal sound feature data c' in the abnormal sound feature data set C' y Perform sound abnormality data analysis to filter out sound type preprocessing feature data set A' yu Abnormal sound feature data in the sound type and generate abnormal feature data of the sound type When a′ n and c′ y If no analysis or matching is successful, the analysis of abnormal sound type feature data is terminated.

[0090] The steps for performing numerical abnormal data analysis based on numerical type preprocessing feature data and abnormal numerical feature data, filtering out numerical abnormal data in numerical type preprocessing feature data, and generating numerical type abnormal feature data are as follows:

[0091] S51, establish abnormal numerical feature data set C" = (c"1, ..., c" q ,…,c″ ψ ), q=1,2,3,…,ψ; where c″ q represents the qth abnormal numerical feature data, ψ represents the maximum number of abnormal numerical feature data, and the abnormal numerical feature data represents the unconventional numerical feature data collected for a specific application object but does not belong to dirty data;

[0092] S52, using a depth-first search algorithm to preprocess the feature data set A of the numerical type" yu Preprocessing feature data a" of numerical type p One by one, compare the abnormal numerical feature data c″ in the abnormal numerical feature data set C″ q Perform numerical abnormal data analysis to filter out numerical type preprocessing feature data set A" yu Abnormal numerical feature data in and generate numerical type abnormal feature data when a″ p With c″ q If no analysis or matching is successful, the analysis of abnormal characteristic data of numerical type ends.

[0093] The steps for obtaining image type abnormal feature data, sound type abnormal feature data, and numerical type abnormal feature data are as follows:

[0094] S61. Obtaining image type abnormal feature data Abnormal sound type feature data Abnormal characteristic data of numerical type

[0095] The acquired image type abnormal feature data, sound type abnormal feature data, and numerical type abnormal feature data are respectively collected again using the collection device according to the data number to generate image reprocessing data, sound reprocessing data, and numerical reprocessing data. The image reprocessing data and the sound reprocessing data are respectively subjected to step S4 to reanalyze the image and sound abnormal data, and the numerical reprocessing data is subjected to step S5 to reanalyze the numerical abnormal data. The operation steps for performing the abnormal data processing operation according to the analysis results are as follows:

[0096] S71, image type abnormal feature data Abnormal sound type feature data Abnormal characteristic data of numerical type Re-collect the corresponding image type feature data, sound type feature data, and numerical type feature data according to the cloud lens collection number m, the sound sensor collection number n, and the parameter sensor collection number p, and generate the image type feature data, sound type feature data, and numerical type feature data identifiers as image reprocessing data Sound reprocessing data Numerical reprocessing of data

[0097] S72, reprocessing the image data Sound reprocessing data Execute step S42, when the image type abnormal feature data or the sound type abnormal feature data is generated again, the image type abnormal feature data Or abnormal characteristic data of sound type Push information and feedback through IoT communication;

[0098] Reprocess the data numerically Execute step S52, when the numerical type abnormal characteristic data is generated again, the numerical type abnormal characteristic data Information push feedback is carried out through IoT communication.

[0099] The image type data anomaly analysis unit, the sound type data anomaly analysis unit, and the numerical type data anomaly analysis unit cooperate with each other to perform abnormal value analysis and comparison on the calibrated images, sounds, and numerical data with the stored image abnormality data, sound abnormality data, and numerical abnormality data using a data recognition algorithm, thereby realizing intelligent recognition and screening of abnormal images, sounds, and numerical data in the data collection stage, and improving the scientificity and reliability of data preprocessing; the abnormal data reprocessing execution unit uses the collection equipment to perform secondary data collection on the identified image abnormality data, sound abnormality data, and numerical abnormal data according to the data number, and performs secondary abnormal data analysis and feedback on the collected images, sounds, and numerical data, thereby realizing abnormal data self-checking and abnormal data feedback in data collection, and improving the intelligence and accuracy of data preprocessing.

[0100] A preprocessing system for implementing a data preprocessing method, the preprocessing system includes a data acquisition management module, a data calibration management module, and a data anomaly analysis and reprocessing module;

[0101] The data acquisition management module includes an image type data acquisition unit, a sound type data acquisition unit, and a numerical type data acquisition unit;

[0102] The image type data acquisition unit uses the cloud lens to collect image type feature data; the sound type data acquisition unit uses the sound sensor to collect sound type feature data; the numerical type data acquisition unit uses the parameter sensor to collect numerical type feature data;

[0103] The data calibration management module includes an image calibration data storage unit, an image data calibration analysis and collection unit, a sound calibration data storage unit, a sound data calibration analysis and collection unit, a numerical calibration data storage unit, and a numerical data calibration analysis and collection unit;

[0104] An image calibration data storage unit, used to store image type data calibration dirty data; an image data calibration analysis and collection unit, which performs image dirty data calibration analysis on image type feature data and image type data calibration dirty data, removes image dirty data from the image type feature data, collects remaining image data from the image type feature data, and generates image type preprocessing feature data; a sound calibration data storage unit, used to store sound type data calibration dirty data; a sound data calibration analysis and collection unit, which performs sound dirty data calibration analysis on sound type feature data and sound type data calibration dirty data, removes sound dirty data from the sound type feature data, collects remaining sound data from the sound type feature data, and generates sound type preprocessing feature data; a numerical calibration data storage unit, used to store numerical type data calibration dirty data; a numerical data calibration analysis and collection unit, which performs numerical dirty data calibration analysis on numerical type feature data and numerical type data calibration dirty data, removes numerical dirty data from the numerical type feature data, collects remaining numerical data from the numerical type feature data, and generates numerical type preprocessing feature data;

[0105] The data anomaly analysis and reprocessing module includes an image type data anomaly analysis unit, an abnormal image data storage unit, a sound type data anomaly analysis unit, an abnormal sound data storage unit, a numerical type data anomaly analysis unit, an abnormal numerical data storage unit, an abnormal data collection information data acquisition unit, and an abnormal data reprocessing execution unit;

[0106] The image type data abnormality analysis unit performs image abnormality data analysis based on the image type preprocessing feature data and the abnormal image feature data, filters out the image abnormality data in the image type preprocessing feature data, and generates image type abnormality feature data; the abnormal image data storage unit is used to store the abnormal image feature data; the sound type data abnormality analysis unit performs sound abnormality data analysis based on the sound type preprocessing feature data and the abnormal sound feature data, filters out the sound abnormality data in the sound type preprocessing feature data, and generates sound type abnormality feature data; the abnormal sound data storage unit is used to store the abnormal sound feature data; the numerical type data abnormality analysis unit performs numerical abnormality data analysis based on the numerical type preprocessing feature data and the abnormal numerical feature data, The numerical abnormal data in the type preprocessing feature data is screened out, and the numerical type abnormal feature data is generated; the abnormal numerical data storage unit is used to store the abnormal numerical feature data; the abnormal data collection information data acquisition unit is used to acquire the image type abnormal feature data, the sound type abnormal feature data, and the numerical type abnormal feature data; the abnormal data reprocessing execution unit re-collects the acquired image type abnormal feature data, sound type abnormal feature data, and numerical type abnormal feature data according to the data number using the collection device to generate image reprocessing data, sound reprocessing data, and numerical reprocessing data, and re-analyzes the image reprocessing data, sound reprocessing data, and numerical reprocessing data for image, sound, and numerical abnormal data, and executes the abnormal data processing operation according to the analysis result.

[0107] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A data preprocessing method, characterized in that: The method comprises the following steps: S1, collecting image type feature data, sound type feature data, and numerical type feature data; S2, performing image and sound dirty data calibration analysis on the image type feature data and image type data calibration dirty data, and on the sound type feature data and sound type data calibration dirty data, respectively, clearing the image and sound dirty data in the image type feature data and the sound type feature data, collecting the remaining image and sound data in the image type feature data and the sound type feature data, and generating corresponding image type preprocessing feature data and sound type preprocessing feature data; S3, performing numerical dirty data calibration analysis on the numerical type characteristic data and the numerical type data calibration dirty data, removing the numerical dirty data in the numerical type characteristic data, collecting the remaining numerical data in the numerical type characteristic data and generating numerical type preprocessing characteristic data; S4, performing image and sound abnormal data analysis respectively according to the image type preprocessing feature data and abnormal image feature data, and the sound type preprocessing feature data and abnormal sound feature data, filtering out the image and sound abnormal data in the image type preprocessing feature data and the sound type preprocessing feature data, and generating image type abnormal feature data and sound type abnormal feature data, wherein S4 comprises the following steps: S41, establish abnormal image feature data set C = (c1, ..., c z ,…,c τ ), z=1,2,3,…,τ; where c z represents the zth abnormal image feature data, τ represents the maximum number of abnormal image feature data, and the abnormal image feature data represents the unconventional image feature data of the specific application object collected but does not belong to dirty data; Establish abnormal sound feature data set C'=(c'',...,c' y ,…,c′ ω ), y=1,2,3,…,ω; where c′ y represents the yth abnormal sound feature data, ω represents the maximum number of abnormal sound feature data, and the abnormal sound feature data represents the unconventional sound feature data collected for a specific application object but does not belong to dirty data; S42, using a depth-first search algorithm to preprocess the image type feature data set A yu Image type preprocessing feature data a m One by one with the abnormal image feature data c in the abnormal image feature data set C z Perform image abnormality data analysis to filter out image type preprocessing feature data set A yu Abnormal image feature data in and generate image type abnormal feature data when a m With c z If no analysis and matching is successful, the image type abnormal feature data analysis ends; The image type preprocessing feature data set A yu Includes a m , where a m Represents the image type preprocessing feature data corresponding to the image type feature data collected by the m-th cloud lens after data calibration; The sound type preprocessing feature data set A' is obtained by using a depth-first search algorithm. yu The sound type preprocessing feature data a′ n One by one with the abnormal sound feature data c' in the abnormal sound feature data set C' y Perform sound abnormality data analysis to filter out sound type preprocessing feature data set A' yu Abnormal sound feature data in the sound type and generate abnormal feature data of the sound type When a′ n and c′ y If no analysis and matching is successful, the analysis of abnormal sound type feature data ends; The sound type preprocessing feature data set A' yu Include a' n , where a' n Indicates the sound type preprocessing feature data corresponding to the sound type feature data collected by the nth sound sensor after data calibration; S5, performing numerical abnormal data analysis based on the numerical type preprocessing feature data and the abnormal numerical feature data, filtering out the numerical abnormal data in the numerical type preprocessing feature data, and generating numerical type abnormal feature data, wherein S5 includes the following steps: S51, establish abnormal numerical feature data set C" = (c"1, ..., c"q, ..., c" ψ ), q=1,2,3,…,ψ; where c″ q represents the qth abnormal numerical feature data, ψ represents the maximum number of abnormal numerical feature data, and the abnormal numerical feature data represents the unconventional numerical feature data of the specific application object but does not belong to dirty data; S52, using a depth-first search algorithm to preprocess the feature data set A of the numerical type" yu Preprocessing feature data a" of numerical type p One by one, compare the abnormal numerical feature data c″ in the abnormal numerical feature data set C″ q Perform numerical abnormal data analysis to filter out numerical type preprocessing feature data set A" yu Abnormal numerical feature data in and generate numerical type abnormal feature data when a″ p With c″ q If no analysis and matching is successful, the analysis of abnormal characteristic data of numerical type ends; Wherein the numerical type preprocessing feature data set A" yu Including a″ p , where a″ p Indicates the numerical type preprocessing characteristic data corresponding to the data calibration of the numerical type characteristic data collected by the p-th parameter sensor; S6, obtaining the image type abnormal feature data, the sound type abnormal feature data, and the numerical type abnormal feature data; S7. Re-collect the acquired image type abnormal feature data, sound type abnormal feature data, and numerical type abnormal feature data according to the data numbers using the collection device to generate image reprocessing data, sound reprocessing data, and numerical reprocessing data. The image reprocessing data and sound reprocessing data are respectively subjected to step S4 to re-analyze the image and sound abnormal data, and the numerical reprocessing data is subjected to step S5 to re-analyze the numerical abnormal data, and the abnormal data processing operation is performed according to the analysis results.

2. A data preprocessing method according to claim 1, characterized in that: The steps of collecting image type feature data, sound type feature data, and numerical type feature data are as follows: S11. Use cloud lens to collect image type feature data of specific application objects online and establish a collection where a m Represents the image type feature data collected by the mth cloud lens, Indicates the maximum number of cloud lenses for collecting image type feature data; Use sound sensors to collect sound type feature data of specific application objects online and establish a set A'=(a'1,…,a' n ,…,a' θ ), n=1,2,3,...,θ; where a' n represents the sound type feature data collected by the nth sound sensor, and θ represents the maximum number of sound sensors collecting sound type feature data; Use parameter sensors to collect numerical characteristic data of specific application objects online and establish a collection where a″ p Indicates the numerical characteristic data collected by the pth parameter sensor, Indicates the maximum number of parameter sensors that collect numerical type feature data. The application objects include specific data collection and use fields in industry, agriculture and service industries. The parameter sensors include one or more of temperature, humidity, flow, rotation speed and vibration.

3. A data preprocessing method according to claim 2, characterized in that: The steps of performing image and sound dirty data calibration analysis on the image type feature data and image type data calibration dirty data, and on the sound type feature data and sound type data calibration dirty data, respectively, clearing the image and sound dirty data in the image type feature data and the sound type feature data, collecting the remaining image and sound data in the image type feature data and the sound type feature data, and generating corresponding image type preprocessing feature data and sound type preprocessing feature data are as follows: S21, construct image type data calibration dirty data set B = (b1, ..., b u ,…,b ρ ), u=1,2,3,…,ρ; where b u represents the dirty data of the u-th image type data calibration, ρ represents the maximum number of dirty data of the image type data calibration; Construct sound type data calibration dirty data set B'=(b'1,…,b' v ,…,b' σ ), v = 1, 2, 3, ..., σ; where b' v represents the vth dirty data of sound type data calibration, σ represents the maximum number of dirty data of sound type data calibration; S22, using a data analysis algorithm to analyze the image type feature data a in the image type feature data set A m Align dirty data b with image type data in dirty data set B u Perform image dirty data calibration traversal analysis and clear the image type data calibration dirty data b in the image type feature data set A u , collect the remaining image type feature data a in the image type feature data set A m And generate image type preprocessing feature data set 1≤m1≤m, where a m Represents the image type preprocessing feature data corresponding to the image type feature data collected by the m-th cloud lens after data calibration; Indicates the image type preprocessing feature data corresponding to the image type feature data collected by the m1-th cloud lens after data calibration; The image type feature data collected by the m2-th cloud lens is represented by the image type preprocessing feature data corresponding to the data calibration. The operation steps of the data analysis algorithm to perform image dirty data calibration traversal analysis are as follows: S221, initialize algorithm parameters, population size N, maximum number of iterations T; S222, initializing the population, calculating fitness, and determining image search pathfinders and image search followers; S223, according to the formula Update the image search pathfinder position in the search space of the image type feature data set A, where t represents the current iteration number of the algorithm and T is the maximum iteration number of the algorithm; represents the position of the contemporary image search pathfinder in the search space of the image type feature data set A in the tth iteration, represents the position of the previous generation image search pathfinder in the search space of the image type feature data set A in the t-1th iteration, represents the position of the image search pathfinder in the search space of the image type feature data set A after the update in the t+1th iteration, ε is the step size factor of the image search pathfinder, and ε follows a uniform distribution in [0,1]; S224, according to the formula Update the image search follower's position in the image type feature data set A search space, where represents the current position of the image search follower in the search space of the image type feature data set A in the tth iteration, represents its updated position in the search space of the image type feature data set A in the t+1th iteration; represents the current position of other image search followers in the search space of image type feature data set A in the tth iteration, Image Search Followers move not only with Image Search Pathfinder Related and followed by other image search followers , R1 represents the position distance parameter between image search followers in the search space of image type feature data set A, R2 represents the position distance parameter between image search pathfinder and image search follower in the search space of image type feature data set A, R1=α×r1, R2=β×r2; α represents the interaction coefficient between image search followers, β represents the attraction coefficient of image search pathfinder to image search follower, α and β are uniformly distributed in [1,2]; r1 is the step factor of the movement of image search follower and other image search followers, r2 is the step factor of the movement of image search follower and image search pathfinder, r1 and r2 are both random numbers in the range of [0,1]; S225, calculate the fitness value and update the global optimal value, that is, search for dirty data b that is calibrated with the image type data in the search space of the image type feature data set A u Matched image type feature data a m ; S226, when the maximum number of iterations is met, the image type feature data set A and the image type data calibration dirty data b are combined. u The image type feature data a that matches successfully m Clear, collect the remaining image type feature data a in the image type feature data set A m And generate image type preprocessing feature data set Otherwise, repeat steps S221-S226; S23, using step S22, the sound type feature data a' in the sound type feature data set A' n , and the sound type data calibration dirty data b' in the sound type data calibration dirty data set B' v Perform sound dirty data calibration traversal analysis and clear the sound type data calibration dirty data b' existing in the sound type feature data set A' v , collect the sound type feature data to establish the remaining sound type feature data a′ in the set A′ n And generate sound type preprocessing feature data set 1≤n1≤n, n≤n2≤θ; where a′ n Indicates the sound type preprocessing feature data corresponding to the sound type feature data collected by the nth sound sensor after data calibration; Indicates the sound type preprocessing feature data corresponding to the sound type feature data collected by the n1th sound sensor after data calibration; It represents the sound type preprocessing feature data corresponding to the sound type feature data collected by the n2th sound sensor after data calibration.

4. A data preprocessing method according to claim 3, characterized in that: The steps of performing numerical dirty data calibration analysis on the numerical type characteristic data and the numerical type data calibration dirty data, removing the numerical dirty data in the numerical type characteristic data, collecting the remaining numerical data in the numerical type characteristic data and generating numerical type preprocessing characteristic data are as follows: S31. Constructing a dirty data set for numerical data calibration Where b″ o Indicates the calibration dirty data of the oth numerical type data. Indicates the maximum number of dirty data for calibration of numerical type data, where the dirty data includes one or more of image missing values, image duplicate data, sound missing values, sound duplicate data, numerical missing values, and numerical duplicate data; S32, using step S22, the numerical type feature data a″ in the numerical type feature data set A″ p , and the dirty data set B" in which the dirty data of the numerical type is aligned with the dirty data of the numerical type b" o Perform numerical dirty data calibration traversal analysis and clear the numerical type data calibration dirty data b″ in the numerical type feature data set A″ o , collect the remaining numerical type feature data a″ in the numerical type feature data set A″ p And generate a numerical type preprocessing feature data set 1≤p1≤p, where a″ p Indicates the numerical type preprocessing characteristic data corresponding to the data calibration of the numerical type characteristic data collected by the p-th parameter sensor; Indicates the numerical type preprocessing characteristic data corresponding to the data calibration of the numerical type characteristic data collected by the p1th parameter sensor; It represents the numerical type preprocessing characteristic data corresponding to the numerical type characteristic data collected by the p2th parameter sensing after data calibration.

5. A data preprocessing method according to claim 1, characterized in that: The steps of obtaining the image type abnormal feature data, the sound type abnormal feature data, and the numerical type abnormal feature data are as follows: S61. Obtaining abnormal feature data of image type Abnormal sound type feature data Abnormal characteristic data of numerical type 6. A data preprocessing method according to claim 5, characterized in that: The acquired image type abnormal feature data, sound type abnormal feature data, and numerical type abnormal feature data are respectively collected again according to the data number using the collection device to generate image reprocessing data, sound reprocessing data, and numerical reprocessing data, and the image reprocessing data and sound reprocessing data are respectively subjected to step S4 to reanalyze the image and sound abnormal data, and the numerical reprocessing data is subjected to step S5 to reanalyze the numerical abnormal data, and the operation steps of performing the abnormal data processing operation according to the analysis results are as follows: S71, image type abnormal feature data Abnormal sound type feature data Abnormal characteristic data of numerical type Re-collect the corresponding image type feature data, sound type feature data, and numerical type feature data according to the cloud lens collection number m, the sound sensor collection number n, and the parameter sensor collection number p, and generate the image type feature data, sound type feature data, and numerical type feature data as image reprocessing data Sound reprocessing data Numerical reprocessing of data S72, reprocessing the image data Sound reprocessing data Execute step S42, when the image type abnormal feature data or the sound type abnormal feature data is generated again, the image type abnormal feature data Or abnormal characteristic data of sound type Push information and feedback through IoT communication; The numerical reprocessing data Execute step S52, when the numerical type abnormal characteristic data is generated again, the numerical type abnormal characteristic data Information push feedback is carried out through IoT communication.

7. A preprocessing system for implementing the data preprocessing method according to any one of claims 1 to 6.

Images