Outlier detection method based on data acquisition and knowledge filtering and related device

Abstract

The invention belongs to the technical field of outlier detection / abnormal point detection of industrial production data, and discloses an outlier detection method based on data acquisition and knowledge filtering and a related device. The outlier detection method based on data acquisition and knowledge filtering comprises the following steps: acquiring industrial production data; based on the collected industrial production data, performing classification detection by using an outlier detection model based on knowledge filtering to obtain an outlier detection result; wherein the outlier detection model based on knowledge filtering comprises an encoder, a decoder and a knowledge filter. According to the technical scheme provided by the invention, the problem of low outlier detection precision caused by lack of priori knowledge utilization in the prior art can be solved, and compared with the prior art, the outlier detection precision can be remarkably improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of outlier detection / abnormal point detection of industrial production data, and in particular relates to an outlier detection method based on data collection and knowledge filtering and a related device. Background Art

[0002] A large amount of production data is generated during the industrial production process, which can characterize the quality of the product to a certain extent. Furthermore, in the mass production process, discovering outlier samples in products based on production data can help companies improve product quality and effectively reduce a series of safety and economic problems caused by product failures.

[0003] At present, the industrial production process is extremely complex. Even real-time signals acquired through intensive collection methods cannot fully characterize product quality. The outlier detection accuracy of existing detection schemes needs to be improved. In addition, experienced workers can screen outlier samples from products by combining their own knowledge with process documents, which has higher detection accuracy. However, the labor cost is high and the detection accuracy is limited by the workers' experience level.

[0004] In summary, in view of the problem that the existing technical solutions do not fully utilize prior knowledge such as workers' experience and process documents, resulting in low outlier detection accuracy, a new outlier detection solution is urgently needed. Summary of the invention

[0005] The purpose of the present invention is to provide an outlier detection method and related device based on data collection and knowledge filtering to solve one or more of the above-mentioned technical problems. The technical solution provided by the present invention can solve the problem of low outlier detection accuracy caused by the lack of prior knowledge in the existing technical solution, and can significantly improve the outlier detection accuracy compared with the existing technical solution.

[0006] In order to achieve the above object, the present invention adopts the following technical solutions: In a first aspect, the present invention provides an outlier detection method based on data collection and knowledge filtering, comprising the following steps: Collect industrial production data; Based on the collected industrial production data, the outlier detection model based on knowledge filtering is used for classification detection to obtain the outlier detection results; in, The structure of the outlier detection model based on knowledge filtering is a neural network model, including: Encoder, used to input raw industrial production data and perform dimension compression to obtain low-dimensional features; A decoder, used for mapping the low-dimensional features to the original industrial production data dimensions to obtain reconstructed data; Knowledge filter, which is used to input the original industrial production data and the reconstructed data and extract features respectively; When optimizing the outlier detection model based on knowledge filtering, gradient back propagation is performed according to the overall error, and the optimization goal is to reduce the overall error; wherein, the expression of the overall error is: ; In the formula, represents the overall error; Represents the reconstruction error, which is the mean square error between the input industrial production data and the reconstructed data; Represents the filtering error, which is the error between the features of the input industrial production data extracted by the knowledge filter and the reconstructed data.

[0007] A further improvement of the present invention is that The knowledge filter is obtained by summarizing industrial production process documents and converting the text in the process documents used to judge product quality into a mathematical formula; wherein the mathematical formula has an input and an output.

[0008] A further improvement of the present invention is that The text used to judge product quality includes one or both of threshold indicators and mechanism indicators.

[0009] A further improvement of the present invention is that The outlier detection result is industrial production data whose overall error exceeds a preset threshold; Alternatively, the outlier detection result is the one with the largest overall error. N Industrial production data; among them, N It is the product of the total amount of industrial production data collected and the preset percentage.

[0010] In a second aspect, the present invention provides an outlier detection system based on data collection and knowledge filtering, comprising: Data acquisition module, used to collect industrial production data; The classification detection module is used to perform classification detection based on the collected industrial production data using the outlier detection model based on knowledge filtering to obtain outlier detection results; in, The structure of the outlier detection model based on knowledge filtering is a neural network model, including: Encoder, used to input raw industrial production data and perform dimension compression to obtain low-dimensional features; A decoder, used for mapping the low-dimensional features to the original industrial production data dimensions to obtain reconstructed data; Knowledge filter, which is used to input the original industrial production data and the reconstructed data and extract features respectively; When optimizing the outlier detection model based on knowledge filtering, gradient back propagation is performed according to the overall error, and the optimization goal is to reduce the overall error; wherein, the expression of the overall error is: ; In the formula, represents the overall error; Represents the reconstruction error, which is the mean square error between the input industrial production data and the reconstructed data; Represents the filtering error, which is the error between the features of the input industrial production data extracted by the knowledge filter and the reconstructed data.

[0011] A further improvement of the present invention is that The knowledge filter is obtained by summarizing industrial production process documents and converting the text in the process documents used to judge product quality into a mathematical formula; wherein the mathematical formula has an input and an output.

[0012] A further improvement of the present invention is that The text used to judge product quality includes one or both of threshold indicators and mechanism indicators.

[0013] A further improvement of the present invention is that The outlier detection result is industrial production data whose overall error exceeds a preset threshold; Alternatively, the outlier detection result is the one with the largest overall error. N Industrial production data; among them, N It is the product of the total amount of industrial production data collected and the preset percentage.

[0014] In a third aspect of the present invention, an electronic device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, an outlier detection method based on data collection and knowledge filtering as described in any one of the first aspect of the present invention is implemented.

[0015] According to a fourth aspect of the present invention, a non-transitory computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the outlier detection method based on data collection and knowledge filtering as described in any one of the first aspects of the present invention is implemented.

[0016] Compared with the prior art, the present invention has the following beneficial effects: The outlier detection method disclosed in the present invention is specifically an outlier detection method based on data collection and knowledge filtering, which can solve the problem of low outlier detection accuracy caused by the lack of prior knowledge in the existing technical solutions, and can significantly improve the outlier detection accuracy compared with the existing technical solutions. Specifically and explanatoryally, the outlier detection model based on knowledge filtering adopted by the present invention is provided with a knowledge filter, which can make full use of prior knowledge such as worker experience and process documents. These prior knowledge can effectively improve the detection effect and significantly improve the level of outlier detection; further, the method of the present invention can accurately detect outlier individuals in a batch of products, which can improve the overall consistency and quality of the products, and show significant advantages and application potential in improving production efficiency, ensuring product quality, and promoting industrial upgrading, and has broad application prospects. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below; obviously, the drawings described below are some embodiments of the present invention, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0018] Figure 1 is a flow chart of an outlier detection method based on data collection and knowledge filtering in an embodiment of the present invention; Figure 2 yes Figure 1 A schematic diagram of the principle of the embodiment shown; Figure 3 It is a flowchart of an outlier detection method based on data collection and knowledge filtering in a specific embodiment of the present invention; Figure 4 It is a schematic diagram of an outlier detection system based on data collection and knowledge filtering in an embodiment of the present invention. DETAILED DESCRIPTION

[0019] In order to make the purpose, technical solutions and advantages of the present invention clearer, the technical solutions in the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention; it is obvious that the described embodiments and technical solutions are only part of the embodiments of the present invention, not all of the embodiments.

[0020] All other embodiments obtained by those of ordinary skill in the art without creative work based on the technical solutions disclosed in the embodiments of the present invention belong to the scope of protection of the present invention. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device including a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0021] See also Figure 1 and Figure 2 , an outlier detection method based on data collection and knowledge filtering disclosed in an embodiment of the present invention comprises the following steps: Step 1, collect industrial production data; Step 2: Based on the industrial production data collected in step 1, a knowledge-based outlier detection model is used to perform classification detection to obtain an outlier detection result; The structure of the outlier detection model based on knowledge filtering is a neural network model, including: Encoder, used to input raw industrial production data and perform dimension compression to obtain low-dimensional features; A decoder, used for mapping the low-dimensional features into the original industrial production data dimensions to obtain reconstructed data; Knowledge filter, which is used to input the original industrial production data and the reconstructed data and extract features respectively; When optimizing the outlier detection model based on knowledge filtering, gradient back propagation is performed according to the overall error; wherein the expression of the overall error is: ; In the formula, represents the overall error; Represents the reconstruction error, which is the mean square error between the input industrial production data and the reconstructed data; Represents the filtering error, which is the error between the features of the input industrial production data extracted by the knowledge filter and the reconstructed data.

[0022] The technical solution provided by the embodiment of the present invention is used to solve the problem of low detection accuracy caused by the lack of utilization of prior knowledge in the existing technical solutions, and can effectively improve the outlier detection accuracy.

[0023] See also Figure 3 In one embodiment of the present invention, the step of constructing an outlier detection model based on knowledge filtering may specifically include: Step 2.1, build an outlier detection model that does not contain knowledge, which can be called a basic model; the basic model includes an input and an output, and the input is multi-source, that is, it can accept multi-dimensional data; the basic model is a neural network model in structure, which can accept input data and compress the data through the encoder to obtain the low-dimensional features of the data; then, through the decoder, the low-dimensional features of the data are remapped to the original data dimension, and the data obtained at this time is called reconstructed data or reconstruction results; interpretatively, the mean square error between the reconstructed data and the input data is called the reconstruction error, and the parameter optimization direction of the model is to make the reconstructed data as similar as possible to the input data; interpretatively, the basis for the model to detect outliers in the basic model is that each sample data input into the model will generate a reconstructed data and a reconstruction error, and the reconstruction errors are arranged from large to small; there are two strategies to screen outlier samples, the first is to set a threshold, when the reconstruction error of a sample exceeds the threshold, the sample is judged as an outlier; the second is to set a percentage α%, and classify the total number of samples with the largest reconstruction error × α% samples as outliers; Step 2.2: embed the knowledge into the outlier detection model of step 2.1, and construct an outlier detection model based on knowledge filtering; the difference between the outlier detection model that does not contain knowledge (i.e., the basic model) is that the model constructed in step 2.2 is the model to be finally constructed in the technical solution of the embodiment of the present invention, and the model in step 2.1 is the basis, and its purpose is to serve as a framework and prepare for embedding knowledge; Step 2.3, summarize the process documents and convert the text used to judge product quality into a mathematical formula, which is also called a knowledge-based feature filter, or feature filter (knowledge filter) for short. The mathematical formula requires an input and an output. The dimension of the input data is consistent with the outlier detection model based on knowledge filtering, and the output is determined by the prior knowledge. At this time, consider the reconstructed data obtained in step 2.1. Since the reconstructed data and the input data have the same dimension, it can be used as the input of the formula constructed in this step. At the same time, the goal of the data being reconstructed is to be consistent with the input data. Therefore, the reconstruction result and the input are highly similar and retain certain differences. Step 2.4: In step 2.3, a knowledge-based feature filter is constructed. This feature filter extracts features from both the reconstruction result and the input data, which is equivalent to obtaining data features from another dimension. The pattern of the feature is determined by the knowledge from which the feature filter or the formula comes. In step 2.1, a basic model is constructed. When optimizing, the model performs gradient backpropagation based on the reconstruction error, that is, the mean square error between the reconstructed data and the input data. After extracting the two features of the reconstruction result and the input data, the error between the two features is called the filtering error. The filtering error is added to the loss function of the model in step 2.1. Then the final optimization goal of the model will be updated from "the parameter optimization direction of the model is to make the reconstructed data as similar as possible to the input data" as described in step 2.1 to "simultaneously reduce the reconstruction error and filtering error of the data". At this time, the outlier detection model based on knowledge filtering is referred to as the "knowledge filtering model", and the error obtained is named "total error".

[0024] In a specific embodiment of the present invention, the main process indicators can be summarized by analyzing the process documents, including threshold indicators and mechanism indicators, and summarized into a mathematical expression formula, which has an input and an output, and the input data is certain characteristic dimensions in the production data. For example, the highest voltage of a 18650 type battery in industrial production is 4.2V, which is converted and summarized into a mathematical expression to form a formula. When the input data contains voltage, if the voltage exceeds 4.2V, a signal 1 is issued, and if it does not exceed 4.2V, a signal 0 is issued.

[0025] In one embodiment of the present invention, a device for implementing an outlier detection method based on data collection and knowledge filtering is provided, which may include a data collector, an outlier detection model based on knowledge filtering, and a server; wherein the data collector is used to collect industrial production data, including real-time data and discrete data; the outlier detection model based on knowledge filtering is deployed on a server (central server) to receive data collected by the data collector, and then detect and classify the data to find outlier samples therein.

[0026] To further illustrate, the outlier detection model based on knowledge filtering can be implemented as code and deployed on a computer, which can be called a central server; wherein the source of the data is the data collector, which will be transmitted to the central server through the network, and then the server calls the model's interface to input the data into the model; the model obtains the overall error of the data, and ranks it according to the error, and uses the method specified in step 2.1 to perform detection and output a result, which is whether the sample in the data is an outlier data sample.

[0027] The embodiment of the present invention proposes an outlier detection solution based on knowledge filtering, which aims to solve the defect of low detection accuracy caused by insufficient utilization of prior knowledge in outlier detection in the prior art, and can significantly improve the accuracy and efficiency of detection. Furthermore, in the specific implementation system, the three core functions of data acquisition, intelligent analysis and feedback processing can be integrated, which is applicable to multiple fields such as industrial production, quality control, signal processing, etc., and realizes accurate outlier identification of various complex data; wherein, the data collector is not only responsible for collecting real-time data and discrete data in industrial production, but also has data preprocessing capabilities, such as data cleaning, format unification, etc., to ensure that the data input to the detection model is of high quality and high accuracy, and supports data collection from multiple sources (such as sensors, log files, databases, etc.), to meet the data acquisition requirements in different scenarios, and enhance the flexibility and applicability of the system; the core of the outlier detection model based on knowledge filtering is the introduction of the target knowledge embedding model, which can integrate the experience of workers, process documents, etc. , historical cases and other prior knowledge, provide strong knowledge support for outlier detection. After receiving the data transmitted by the data collector, the model first extracts features, and then intelligently classifies the data through the knowledge filtering algorithm, quickly and accurately identifying outlier samples that deviate significantly from the normal mode. The model has the ability of self-learning and updating, and can continuously optimize the knowledge base and algorithm parameters according to the newly collected feedback data and test results, so as to achieve continuous improvement of detection accuracy; the central server is responsible for carrying the outlier detection model, providing powerful computing power and storage space, ensuring the efficiency and stability of large-scale data processing, and the server can also display the test results to the user in the form of intuitive charts and reports, and can automatically trigger the alarm mechanism according to the preset threshold, and promptly notify relevant personnel to take measures. In summary, the technical solution of the present invention can effectively reduce false alarms and missed alarms and improve the accuracy of outlier detection by integrating prior knowledge; timely identification and processing of outlier individuals in the production process can help improve the quality and consistency of the entire batch of products and reduce the defective rate; the implementation of the solution of the present invention provides a powerful tool for the intelligence and automation of industrial production, and promotes the transformation of the manufacturing industry to intelligent manufacturing.

[0028] The following are device embodiments of the present invention, which can be used to implement the method embodiments of the present invention. For details not disclosed in the device embodiments, please refer to the method embodiments of the present invention.

[0029] See also Figure 4 In an embodiment of the present invention, a system for detecting outliers based on data collection and knowledge filtering is provided, comprising: Data acquisition module, used to collect industrial production data; The classification detection module is used to perform classification detection based on the collected industrial production data using the outlier detection model based on knowledge filtering to obtain outlier detection results; in, The structure of the outlier detection model based on knowledge filtering is a neural network model, including: Encoder, used to input raw industrial production data and perform dimension compression to obtain low-dimensional features; A decoder, used for mapping the low-dimensional features to the original industrial production data dimensions to obtain reconstructed data; Knowledge filter, which is used to input the original industrial production data and the reconstructed data and extract features respectively; When optimizing the outlier detection model based on knowledge filtering, gradient back propagation is performed according to the overall error, and the optimization goal is to reduce the overall error; wherein, the expression of the overall error is: ; In the formula, represents the overall error; Represents the reconstruction error, which is the mean square error between the input industrial production data and the reconstructed data; Represents the filtering error, which is the error between the features of the input industrial production data extracted by the knowledge filter and the reconstructed data.

[0030] In one embodiment of the present invention, a computer device is provided, the computer device comprising a processor and a memory, the memory being used to store a computer program, the computer program comprising program instructions, and the processor being used to execute the program instructions stored in the computer storage medium. The processor may be a central processing unit (CPU), or may be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. It is the computing core and control core of the terminal, which is suitable for implementing one or more instructions, and is specifically suitable for loading and executing one or more instructions in a computer storage medium to implement a corresponding method flow or corresponding function; the processor described in the embodiment of the present invention can be used to perform the operation of the outlier detection method based on data acquisition and knowledge filtering.

[0031] In one embodiment of the present invention, a storage medium is provided, specifically a computer-readable storage medium (Memory), which is a memory device in a computer device for storing programs and data. It is understandable that the computer-readable storage medium here can include both built-in storage media in the computer device and, of course, extended storage media supported by the computer device. The computer-readable storage medium provides a storage space, which stores the operating system of the terminal. In addition, one or more instructions suitable for being loaded and executed by the processor are also stored in the storage space, and these instructions can be one or more computer programs (including program codes). It should be noted that the computer-readable storage medium here can be a high-speed RAM (Random Access Memory) memory, or a non-volatile memory (non-volatile memory), such as at least one disk memory. The processor can load and execute one or more instructions stored in the computer-readable storage medium to implement the corresponding steps of the outlier detection method based on data acquisition and knowledge filtering in the above embodiment.

[0032] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, optical storage, etc.) containing computer-usable program codes.

[0033] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0034] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0035] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0036] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, ordinary technicians in the relevant field should understand that the specific implementation methods of the present invention can still be modified or replaced by equivalents. Any modification or equivalent replacement that does not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. A method for outlier detection based on data collection and knowledge filtering, characterized in that: The following steps are involved: Collect industrial production data; Based on the collected industrial production data, the outlier detection model based on knowledge filtering is used for classification detection to obtain the outlier detection results; in, The structure of the outlier detection model based on knowledge filtering is a neural network model, including: Encoder, used to input raw industrial production data and perform dimension compression to obtain low-dimensional features; A decoder, used for mapping the low-dimensional features to the original industrial production data dimensions to obtain reconstructed data; Knowledge filter, which is used to input the original industrial production data and the reconstructed data and extract features respectively; When optimizing the outlier detection model based on knowledge filtering, gradient back propagation is performed according to the overall error, and the optimization goal is to reduce the overall error; wherein, the expression of the overall error is: ； In the formula, represents the overall error; Represents the reconstruction error, which is the mean square error between the input industrial production data and the reconstructed data; Represents the filtering error, which is the error between the features of the input industrial production data extracted by the knowledge filter and the reconstructed data.

2. The outlier detection method based on data collection and knowledge filtering according to claim 1, characterized in that: The knowledge filter is obtained by summarizing industrial production process documents and converting the text in the process documents used to judge product quality into a mathematical formula; wherein the mathematical formula has an input and an output.

3. The outlier detection method based on data collection and knowledge filtering according to claim 2 is characterized in that: The text used to judge product quality includes one or both of threshold indicators and mechanism indicators.

4. The outlier detection method based on data collection and knowledge filtering according to claim 1 is characterized in that: The outlier detection result is industrial production data whose overall error exceeds a preset threshold; Alternatively, the outlier detection result is the one with the largest overall error. N Industrial production data; among them, N It is the product of the total amount of industrial production data collected and the preset percentage.

5. An outlier detection system based on data collection and knowledge filtering, characterized in that: include: Data acquisition module, used to collect industrial production data; The classification detection module is used to perform classification detection based on the collected industrial production data using the outlier detection model based on knowledge filtering to obtain outlier detection results; in, The structure of the outlier detection model based on knowledge filtering is a neural network model, including: Encoder, used to input raw industrial production data and perform dimension compression to obtain low-dimensional features; A decoder, used for mapping the low-dimensional features to the original industrial production data dimensions to obtain reconstructed data; Knowledge filter, which is used to input the original industrial production data and the reconstructed data and extract features respectively; When optimizing the outlier detection model based on knowledge filtering, gradient back propagation is performed according to the overall error, and the optimization goal is to reduce the overall error; wherein, the expression of the overall error is: ； In the formula, represents the overall error; Represents the reconstruction error, which is the mean square error between the input industrial production data and the reconstructed data; Represents the filtering error, which is the error between the features of the input industrial production data extracted by the knowledge filter and the reconstructed data.

6. The outlier detection system based on data collection and knowledge filtering according to claim 5, characterized in that: The knowledge filter is obtained by summarizing industrial production process documents and converting the text in the process documents used to judge product quality into a mathematical formula; wherein the mathematical formula has an input and an output.

7. The outlier detection system based on data collection and knowledge filtering according to claim 6, characterized in that: The text used to judge product quality includes one or both of threshold indicators and mechanism indicators.

8. The outlier detection system based on data collection and knowledge filtering according to claim 5, characterized in that: The outlier detection result is industrial production data whose overall error exceeds a preset threshold; Alternatively, the outlier detection result is the one with the largest overall error. N Industrial production data; among them, N It is the product of the total amount of industrial production data collected and the preset percentage.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the outlier detection method based on data collection and knowledge filtering as described in any one of claims 1 to 4 is implemented.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the outlier detection method based on data collection and knowledge filtering as claimed in any one of claims 1 to 4 is implemented.

Images