High-dimensional data exception detection system and method

Abstract

The invention provides a high-dimensional data exception detection system and method, and the method comprises the following steps: carrying out the preprocessing of original high-dimensional data, soas to remove an interference value in the original high-dimensional data, and carrying out the filling of the data after the interference value is removed; performing normalization processing on thefilled data; carrying out dimension reduction on the normalized data; shaping the data after dimension reduction to obtain supervised data; analyzing the supervised data by using an LSTM network to obtain prediction data; and comparing the prediction data with the real data to judge whether the original high-dimensional data is abnormal or not. According to the high-dimensional data exception detection method, rapid and accurate exception detection can be carried out on the high-dimensional data, and when exceptions occur in equipment such as an automobile, the exceptions can be processed immediately, so that absolute safety of automobile driving is ensured.

Description

technical field [0001] The invention belongs to the technical field of big data processing, and in particular relates to a high-dimensional data anomaly detection system and method. Background technique [0002] In modern society, more and more electric vehicles have entered our lives. In the future, unmanned electric vehicles will definitely become the mainstream of automobiles in society. Therefore, how to understand the running status of the car has become a problem that we must care about. In recent years, due to the imperfection of the automatic driving algorithm, many unmanned vehicles have crashed and malfunctioned. Therefore, this objectively requires us to have the ability to use the data on the car body to detect abnormalities in the car. [0003] The data generated by cars has the characteristics of high dimensionality and large quantity. For this, whether it is a mathematical statistical method or a simple LSTM network (long short-term memory network) in deep...

AI Technical Summary

Problems solved by technology

[0004] Statistical methods in mathematics can only show the overall laws of the data, but cannot effectively analyze the contextual connections in the data and the relationship between different dimensions of the data

Moreover, in the case of a sharp increase in the amount of data, the analysis speed of the statistical method will slow down sharply, and there is no application value in the case of analyzing a large amount of data generated by the car

[0005] For the LSTM network, it is very good at analyzing time series data

However, the speed of the LSTM network will be significantly slowed down when analyzing high-dimensional data, and the data generated by the car is unsupervised data. If the unsupervised data generated by the car is converted into supervised data, the analysis time of the LSTM network for the data is still short. will greatly increase

This is obviously not enough for the abnormal detection of the car state when we are doing driverless driving

Images