Classifying and filtering data from a data stream

Abstract

Methods, computer program products, devices, and / or systems are provided that perform the following: receiving data from an input data stream; separating the data into data blocks and storing the data blocks in a set of different matrix buffers according to a selection configuration matrix until the matrix buffers are filled; processing the data blocks by a plurality of processing units, each processing unit being associated with a matrix buffer of the matrix buffers via an I / O block; providing the processed data blocks as input to a neural network component; and classifying the data by the neural network component and filtering the data based on the classification of the data.

Description

Background Technology

[0001] This disclosure generally relates to data processing systems, and more specifically to apparatus, methods, computer program products, and data processing systems for classifying and filtering data from data streams.

[0002] With the dramatic increase in data volume caused by things like Internet of Things (IoT) devices or social media, the challenges of data storage are generally increasing. On the one hand, storing this massive amount of data is challenging in itself; on the other hand, machine learning algorithms are often needed to analyze data streams in real time and react with very high confidence. Summary of the Invention

[0003] According to an aspect of the invention, a computer-implemented method, computer program product, apparatus, and / or system is provided that performs the following operations (not necessarily in the following order): receiving data from an input data stream; dividing the data into data blocks and storing the data blocks in a set of different matrix buffers according to a selection configuration matrix until the matrix buffers are full; processing the data blocks by a plurality of processing units, each processing unit being associated with a matrix buffer in the matrix buffers via an I / O block; providing the processed data blocks as input to a neural network component; and classifying the data by the neural network component and filtering the data based on the classification of the data. Attached Figure Description

[0004] This disclosure, together with the foregoing and other objects and advantages, may be best understood from the following detailed description of the embodiments, but is not limited to the described embodiments.

[0005] Figure 1 A schematic diagram of a high-performance data analysis system with a device for filtering data from a data stream is shown according to an embodiment of the present invention.

[0006] Figure 2 A preprocessing component of a device according to an embodiment of the present invention is shown.

[0007] Figure 3 A post-processing component for a device's analog neural network component is shown according to an embodiment of the present invention.

[0008] Figure 4 A post-processing component for a digital neural network component of a device according to another embodiment of the present invention is shown.

[0009] Figure 5 A general flowchart of a method for filtering data from a data stream according to an embodiment of the present invention is shown.

[0010] Figure 6An exemplary embodiment of a data processing system for performing a method according to an embodiment of the present invention is shown. Detailed Implementation

[0011] According to various aspects of this disclosure, apparatus, systems, and methods can be provided that allow for high-performance classification and filtering of large data streams. Specifically, the apparatus, systems, and methods of this disclosure can provide a classification unit that receives an input data stream and performs classification of the input data in real time. Based on the classification, data from the data stream can be tagged (e.g., metadata) and sent for storage, such as to back-end parallel storage, or data can be discarded (e.g., data with little or no relevance). In some embodiments, for data to be stored, data can be moved to an appropriate storage tier of the storage system based on tags (e.g., metadata) from the classification. More specifically, embodiments of the systems and methods of this disclosure can provide for processing data from an input data stream to prepare data for analysis, and then providing for analyzing the data, for example via a neural network, to perform data classification and filtering to allow for appropriate data storage.

[0012] With the dramatic increase in data volume due to things like Internet of Things (IoT) devices or social media, the challenges of data storage are generally increasing. On the one hand, storing this massive amount of data is challenging in itself; on the other hand, machine learning algorithms are often needed to analyze data streams in real time and react with very high confidence.

[0013] Typically, some storage solutions offer the option to store and retrieve information from multiple clients in parallel to backend storage. Backend storage can be a mix of storage types, such as SSDs, HDDs, and tape. Tiering between different storage types can be automatically managed and may be invisible to multiple hosts.

[0014] With very large data volumes, such as those in the terabytes per second range, typical storage solutions, as outlined above, may reach their limitations. Storing the data first and then performing analysis may not be suitable, as this could consume too much storage capacity. Therefore, some solutions perform initial data classification within the sensor.

[0015] Parallel file system solutions may not be optimized for handling massive data streams, where only a small subset of the data may actually be worth preserving. Some storage technologies are operable so that all data is initially stored, for example, in a data pool. After storage, the host can, for example, leverage machine learning algorithms to scan for new (stored) data and then potentially trigger actions based on subsequent classification results.

[0016] Therefore, in many such cases, it is possible to initially store 100% of the data, even if only a very small subset of the data may be relevant. Furthermore, the order in which the data is stored and subsequently analyzed can be too slow for many real-time applications.

[0017] Furthermore, intra-sensor classification solutions may not scale effectively with the number of sensors. Multiple sensors can be difficult to manage, and real-time synchronous classification of a set of multiple sensors may be very difficult or even impossible.

[0018] The shared bus between program memory and data memory can lead to the so-called von Neumann bottleneck, where the throughput (data transfer rate) between the central processing unit (CPU) and memory is limited relative to the amount of memory. Because a single bus can only access one of the two types of memory at a time, the throughput is lower than the rate at which the CPU can operate. This can severely limit the effective processing speed when the CPU is required to perform minimal processing on large amounts of data. The CPU can be continuously forced to wait for the required data to move to or from memory. As CPU speed and memory size have increased much faster than the throughput between them, the bottleneck typically becomes more problematic, and its severity can increase with each new generation of CPUs.

[0019] In the accompanying drawings, the same elements are denoted by the same reference numerals. The drawings are merely schematic and are not intended to describe specific parameters of the invention. Furthermore, the drawings are intended to illustrate exemplary embodiments of the invention only and should therefore not be considered as limiting the scope of the invention.

[0020] Figure 1 A schematic diagram of a high-performance data analysis system with a device 100 for filtering data from a data stream 40, according to an embodiment of the present invention, is shown. According to various aspects of this disclosure, the device 100 includes a new data preprocessing unit and a subsequent neuromorphic classification unit.

[0021] A data acquisition unit 80 with multiple sensors 82 transmits a data stream 40 to a server 84. The server 84 includes a device 100 for receiving the data stream 40, classifying the data, and transmitting filtered data 70 to a tiered storage device 90. The storage device 90 may include different tiers of storage 92, 94, and 96 for short-term, medium-term, and long-term storage of the data, respectively. The storage device 90 is managed by a management node 98, which transfers data between the different tiers of storage 92, 94, and 96.

[0022] The filtered data 86 can be directly picked out from the data stream 40 (e.g., not stored in the tiered storage device 90). The filtered data 86 may include data from the data stream 40 that is determined to have little or no relevance, and therefore storage in the tiered storage device 90 is not guaranteed.

[0023] Data stream 40 includes data from multiple sensors 82 (e.g., associated with Internet of Things (IoT) devices). Data from each sensor 82 includes metadata and payload data, with the metadata relating to both the sensor 82 and the payload data. The metadata includes a sensor identifier, sensor category, and data identifier.

[0024] like Figure 2-4 As described herein, the device 100 for filtering data from data stream 40 may include a preprocessing component 10, a postprocessing component 50, and a neural network component 60.

[0025] To perform real-time classification of data stream 40, according to an embodiment of the invention, device 100 includes hardware based on a "classical" von Neumann data processing unit, which can modify the data in a manner that allows it to be analyzed by subsequent non-von Neumann hardware neural networks (NNs). The advantage of neural networks is that they can operate on parallel vector processes simultaneously to accelerate the classification task. However, neural networks may require a defined format for the input data vectors.

[0026] Figure 2 A preprocessing component 10 of a device 100 according to an embodiment of the present invention is shown. Figure 3 A post-processing component 50 for an analog neural network component 60 for a device 100 according to an embodiment of the present invention is shown.

[0027] The data pre-analyzer processing unit 14 of the preprocessing component 10 modifies the input data stream 40 into a correct vector format with dimension n, where n is the number of rows and columns of the selected configuration matrix. Subsequently, the data is post-processed as needed by a set of parallel processing elements (e.g., processing units) 54 of the post-processing component 50. The tasks of these processing units 54 may include, for example, normalization, pre-filtering, outlier detection, identifying focal regions, etc. The data vector is then fed to the neural network component 60 for final ultrafast classification.

[0028] In some embodiments, dynamically reconfigurable processing elements for data conditioning and NN processing can be used to dynamically adjust vector dimensions as an adaptation or flexibility process.

[0029] The data protocol from each sensor 82 includes the following information: sensor identifier, sensor category, data identifier, and payload data. Input data blocks from sensor 82 can be separated according to sensor category and sensor identifier. According to an embodiment of the invention, based on the data protocol, device 100 can use the payload data to construct matrices. These matrices can be fed into neural network component 60 for real-time processing and classification. Preprocessing component 10 performs data separation according to sensor category. Matrix can be constructed based on data identifiers in the data protocol. Postprocessing component 50 can perform operations such as normalization on the matrices. These operations make the data available for subsequent use by neural network component 60.

[0030] Preprocessing component 10 is configured to receive input data stream 40 and provide the data to postprocessing component 50. Postprocessing component 50 is configured to receive data from preprocessing component 10 and provide the data to neural network component 60. Neural network component 60 is configured to classify the data and filter the data based on one or more classifications.

[0031] The preprocessing component 10 is configured to separate data into blocks and store the data blocks in a set of different matrix buffers 18 (e.g., matrix buffer 0, matrix buffer 1… matrix buffer n) according to a selection configuration matrix. The postprocessing component 50 includes a plurality of processing units 54 (e.g., PS 0, PS 1… PS n), each processing unit 54 being associated with one of the matrix buffers 18. Each processing unit 54 is configured to receive data blocks from its associated matrix buffer 18, process the data blocks, and provide the processed data blocks as input 58 to the neural network component 60.

[0032] return Figure 2 The input data stream 40 is stored in the I / O buffer 12 of the preprocessing component 10 and then forwarded to the subsequent data preanalyzer processing unit 14. The I / O buffer 12 is connected to the I / O processor 22 via an internal bus 34. The I / O processor 22 includes processor software 20 for controlling the I / O processes of the preprocessing component 10 and processor memory 24. The preprocessing component 10 (e.g., the I / O processor 22) is connected to an external network 42 via a processor interface 26 to provide software variability for the system.

[0033] Data can be broken down into blocks in the data pre-analyzer processing unit 14, and these blocks are transferred to the matrix buffer 18 via a data bus 28 according to a selection configuration matrix. The data bus 28 can be a serial or parallel bus. The matrix buffer can be configured as, for example, a 64-byte or 128-byte buffer, but other data widths are also possible. Depending on the configuration of the device 100, the selection configuration matrix can be scalable in the row and column dimension n of the matrix buffer. Therefore, the number n of matrix buffers 18, as well as the number of I / O blocks 52, processor units 54, and digital-to-analog converters 56 of the post-processing unit 50, can vary accordingly.

[0034] Matrix buffer filling is enabled by matrix enable unit 16 via matrix enable signal 30. The selected configuration matrix is ​​based on metadata, and specifically, on sensor category and / or sensor identifier. Each matrix entry has a matrix buffer 18. Once all matrix buffers 18 are full, the complete set of matrix data is transmitted to post-processing component 50 for post-processing. Once all matrix buffers 18 are full via signal line 32, data blocks are synchronously transferred from pre-processing component 10 to post-processing component 50.

[0035] The post-processing component 50 includes a plurality of I / O blocks 52, each I / O block 52 (e.g., I / O block 0, I / O block 1...I / O block n) associated with a matrix buffer 18 (e.g., matrix buffer 0, matrix buffer 1...matrix buffer n), with one I / O block 52 for each matrix entry. The I / O blocks 52 are configured to receive data blocks from the matrix buffer 18 via signal lines 32 after the matrix buffer 18 is filled, and simultaneously transmit the data blocks to the processing unit 54 in a synchronous manner.

[0036] Each of the processing units 54 is configured to receive a data block from the associated I / O block 52 after the data block is populated. The processing unit 54 is configured to modify the received data block according to post-processing rules, such as performing operations on the data block, such as at least one of normalization, pre-filtering, outlier detection, region of interest identification, etc. The processing unit 54 is further configured to synchronously transmit the data blocks from all processing units 54 to the neural network component 60.

[0037] In some embodiments, the neural network component 60 may be configured as an analog neural network. In such an embodiment, the post-processing unit 50 may include a plurality of digital-to-analog converters 56, each of which (e.g., D / A0, D / A1…D / An) is associated with one of the processing units 54 (e.g., PS0, PS1…PSn). Furthermore, the processing units 54 may be configured to prepare blocks of data for input to the digital-to-analog converters 56. For example, the digital information vectors of the pre-processed and post-processed data may be converted into a set of analog signals as input to the non-von Neumann neural network component 60.

[0038] As a component of the analog neural network 60, for example, a chip developed for brain-inspired multi-scale computing in a neuromorphic hybrid system can be used.

[0039] After the data block is converted, the digital-to-analog converter 56 synchronously transmits the analog data 58 as a weighted input to the neural network component 60.

[0040] Figure 4 A post-processing component 50 for a digital neural network component 60 of a device 100 according to another embodiment of the present invention is shown. In this embodiment, the digital-to-analog converter (e.g., the one used in the previous embodiment of the post-processing component 50) is omitted. Figure 3 ). Figure 4 The remaining functions of the post-processing component 50 described herein are similar to those described in the section on... Figure 3 The described functionality.

[0041] Figure 5 It is shown that, according to, Figure 1-3 The above is a general flowchart of a method for filtering data from a data stream according to an embodiment of the present invention.

[0042] like Figure 5 As shown in the flowchart, the operation begins in operation S200, where a data stream (e.g., data stream 40) is received, for example, by the preprocessing component 10 of device 100. Data stream 40 may include data from multiple sensors (e.g., sensor 82), and the data from each sensor may include metadata and load data, the metadata being associated with the sensor and load data. In some embodiments, the metadata may include sensor identifiers, sensor categories, data identifiers, etc.

[0043] Processing continues to operation S202, where data in the data stream is separated into blocks and stored in a set of matrix buffers (e.g., matrix buffer 18) according to a selection configuration matrix (e.g., based on metadata provided in the data stream). In some embodiments, the selection configuration matrix may be based on sensor category and / or sensor identifier.

[0044] Processing continues to operation S204, where, once all matrix buffers are full, data blocks are synchronously transferred from matrix buffers (e.g., matrix buffer 18) to post-processing components (e.g., post-processing component 50). The processing unit can modify the received data blocks according to post-processing rules to perform operations on the data blocks.

[0045] Processing continues to operation S206, where the data block is processed by multiple processing units (e.g., processing unit 54) of the post-processing component according to post-processing rules, such as normalization, pre-filtering, outlier detection, region of interest identification, etc. Then, in embodiments with an analog neural network component, the processed data in the processed data block can be converted into analog data, for example, by a set of digital-to-analog converters associated with the multiple processing units. The converted analog data can be transferred to the analog neural network component (e.g., neural network component 60). In some embodiments, the analog data can be provided to the analog neural network component as a weighted input. Alternatively, in embodiments including a digital neural network component, the processed data can be transferred to the digital neural network component without conversion.

[0046] After the matrix buffer is filled, data blocks are provided from the matrix buffer (e.g., matrix buffer 18) to I / O blocks (e.g., I / O block 52). The data blocks are synchronously transmitted to multiple processing units (e.g., processing unit 54) of the post-processing component. After the associated I / O block is filled, each processing unit receives a data block from the associated I / O block. The processed data blocks are prepared by the multiple processing units (e.g., processing unit 54) to be input to a digital-to-analog converter (e.g., digital-to-analog converter 56) included in the post-processing component. Each digital-to-analog converter is associated with one of the processing units. The processed data blocks from all processing units are synchronously transmitted to the neural network component (e.g., neural network component 60). Analog data can be transmitted as weighted input from the digital-to-analog converter to the neural network component, and synchronously transmitted to the neural network component.

[0047] Processing continues to operation S208, where a neural network component (e.g., neural network component 60) determines the classification of data included in the processed data block and filters the data based on the classification in the neural network component. In some embodiments, the filtered data may be tagged (e.g., via metadata, etc.) to provide storage for the filtered data, for example, in a hierarchical storage unit. In some embodiments, based on the classification of the data, some data provided in the data stream may be discarded (e.g., not sent for storage, etc.) if the data has little or no relevance, etc.

[0048] See now Figure 6The diagram illustrates an example of a data processing system 210. Data processing system 210 is merely one example of a suitable data processing system and is not intended to impose any limitation on the scope of use or functionality of the embodiments of the invention described herein. In any case, data processing system 210 can be implemented and / or perform any of the functions set forth above herein.

[0049] In data processing system 210, there is a computer system / server 212 that can operate with many other general-purpose or special-purpose computing system environments or configurations. Examples of well-known computing systems, environments, and / or configurations that may be suitable for computer system / server 212 include, but are not limited to, personal computer systems, server computer systems, thin clients, thick clients, handheld or laptop devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputer systems, mainframe computer systems, and distributed cloud computing environments that include any of the above systems or devices.

[0050] Computer system / server 212 can be described in the general context of computer system executable instructions such as program modules executed by the computer system. Generally, program modules may include routines, programs, objects, components, logic, data structures, etc., that perform specific tasks or implement specific abstract data types. Computer system / server 212 can be implemented in a distributed cloud computing environment where tasks are performed by remote processing devices linked via a communication network. In a distributed cloud computing environment, program modules can reside in local and remote computer system storage media, including memory storage devices.

[0051] like Figure 6 As shown, the computer system / server 212 in the data processing system 210 is illustrated in the form of a general-purpose computing device. The components of the computer system / server 212 may include, but are not limited to, one or more processors or processing units 216, system memory 228, and a bus 218 that couples the various system components, including the system memory 228, to the processor 216.

[0052] Bus 218 represents any one or more of several types of bus architectures, including memory buses or memory controllers, peripheral buses, accelerated graphics ports, and processor or local buses using any of the various bus architectures. By way of example and not limitation, such architectures include Industry Standard Architecture (ISA) buses, Micro Channel Architecture (MCA) buses, Enhanced ISA (EISA) buses, Video Electronics Standards Association (VESA) local buses, and Peripheral Component Interconnect (PCI) buses.

[0053] Computer system / server 212 typically includes various computer system readable media. Such media can be any available media that can be accessed by computer system / server 212, and includes volatile and non-volatile media, removable and non-removable media.

[0054] System memory 228 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) 230 and / or cache memory 232. Computer system / server 212 may also include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, storage system 234 may be provided for reading and writing from non-removable non-volatile magnetic media (not shown, and generally referred to as a "hard disk drive"). Although not shown, disk drives for reading from or writing to removable non-volatile disks (e.g., "floppy disks") and optical disk drives for reading from or writing to removable non-volatile optical disks (such as CD-ROMs, DVD-ROMs, or other optical media) may be provided. In such cases, each may be connected to bus 218 via one or more data media interfaces. As will be further described and illustrated below, memory 228 may include at least one program product having at least one set of program modules configured to perform embodiments of the invention.

[0055] A program / utility 240 having a set (at least one) of program modules 242, along with an operating system, one or more applications, other program modules, and program data, may be stored in memory 228 in an exemplary, and not limited, manner. Each or some combination of the operating system, one or more applications, other program modules, and program data may include an implementation of a network environment. Program modules 242 typically perform functions and / or methods as described herein in embodiments of the invention.

[0056] Computer system / server 212 can also communicate with one or more external devices 214, such as a keyboard, pointing device, display 224, etc.; one or more devices that enable a user to interact with computer system / server 212; and / or any device that enables computer system / server 212 to communicate with one or more other computing devices (e.g., network interface card, modem, etc.). Such communication may occur via input / output (I / O) interface 222. Furthermore, computer system / server 212 can communicate with one or more networks, such as a local area network (LAN), a general wide area network (WAN), and / or a public network (e.g., the Internet), via network adapter 220. As depicted, network adapter 220 communicates with other components of computer system / server 212 via bus 218. It should be understood that, although not shown, other hardware and / or software components may be used in conjunction with computer system / server 212. Examples include, but are not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data archiving storage systems.

[0057] The present invention can be a system, method, and / or computer program product. A computer program product may include a computer-readable storage medium (or media) having computer-readable program instructions thereon for causing a processor to execute aspects of the present invention.

[0058] Computer-readable storage media can be tangible devices capable of retaining and storing instructions for use by an instruction execution device. Computer-readable storage media can be, for example, but not limited to, electronic storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination of the foregoing. A non-exhaustive list of more specific examples of computer-readable storage media includes: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disk read-only memory (CD-ROM), digital universal disk (DVD), memory sticks, floppy disks, mechanical encoding devices such as punch cards or protrusions in slots having instructions recorded thereon, and any suitable combination of the foregoing. As used herein, computer-readable storage media should not be construed as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses passing through fiber optic cables), or electrical signals transmitted through wires.

[0059] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to a suitable computing / processing device via a network (e.g., the Internet, a local area network, a wide area network, and / or a wireless network), or to an external computer or external storage device. The network may include copper cables, optical fibers, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to a computer-readable storage medium within the suitable computing / processing device.

[0060] Computer-readable program instructions used to perform the operations of this invention may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​(such as Smalltalk, C++, etc.) and conventional procedural programming languages ​​(such as the "C" programming language or similar programming languages). The computer-readable program instructions may be executed entirely on a user's computer, partially on a user's computer, as a standalone software package, partially on a user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter case, the remote computer may be connected to the user's computer via any type of network (including a local area network (LAN) or a wide area network (WAN)) or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuitry including, for example, programmable logic circuitry, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs) may be personalized to execute computer-readable program instructions by utilizing state information from the computer-readable program instructions in order to perform aspects of this invention.

[0061] The present invention will now be described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0062] These computer-readable program instructions may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions / actions specified in one or more blocks of a flowchart and / or block diagram. These computer-readable program instructions may also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner, such that the computer-readable storage medium storing the instructions includes an article of manufacture containing instructions that implement aspects of the functions / actions specified in one or more blocks of a flowchart and / or block diagram.

[0063] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus, or other device to produce computer-implemented processing, such that the instructions executed on the computer, other programmable apparatus, or other device perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0064] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. Each block in a flowchart or block diagram may represent a module, segment, or portion of instructions, including one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than indicated in the figures. For example, depending on the functions involved, two consecutively shown blocks may actually be executed substantially simultaneously, or these blocks may sometimes be executed in reverse order. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action or executes a combination of dedicated hardware and computer instructions.

[0065] The descriptions of various embodiments of the present invention are presented for illustrative purposes only and are not intended to be exhaustive or limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein has been chosen to best explain the principles of the embodiments, their practical application, or technical improvements to technologies found in the market, or to enable those skilled in the art to understand the embodiments disclosed herein.

Claims

1. An apparatus for filtering data from a data stream, comprising: Preprocessing components; Post-processing components; Neural network components; The preprocessing component is configured to receive data from the input data stream and provide the data to the postprocessing component; The post-processing component is configured to receive the data and provide the data to the neural network component; The neural network component is configured to classify the data, determine the relevance of the data based on the classification, and filter out data that is determined to have low or no relevance. The preprocessing component is configured to divide the data into data blocks and store the data blocks in a set of different matrix buffers according to a selection configuration matrix; and The post-processing component includes multiple processing units, each associated with one of the matrix buffers. Each processing unit is further configured to receive the data block from its associated matrix buffer, process the data block, and provide the processed data block as input to the neural network component. The data stream includes data from multiple sensors, and the data from each sensor includes metadata and load data, the metadata being associated with the sensor and the load data.

2. The device according to claim 1, wherein, The metadata includes sensor identifier, sensor category, and data identifier.

3. The device according to claim 1, wherein, Filtered data from the neural network component is provided to a storage device.

4. The device according to claim 1, wherein, Once all buffers are full, the data blocks are synchronously transferred from the preprocessing component to the postprocessing component.

5. The device according to claim 1, wherein, The processing unit is configured to modify the received data block according to post-processing rules to perform operations on the data block, wherein the operations include one or more of normalization, pre-filtering, outlier detection, and region of interest identification.

6. The device according to claim 1, wherein, The post-processing component includes multiple I / O blocks, each I / O block being associated with a matrix buffer, wherein the I / O block is configured to receive data blocks from the matrix buffer after the matrix buffer is filled and to transmit the data blocks to the processing unit synchronously.

7. The device according to claim 1, wherein, The processing unit is configured to receive the data block from the associated I / O block after the associated I / O block is filled, and to transmit the data block of the processing unit to the neural network component in a synchronous manner.

8. The device according to claim 1, wherein, The neural network component is configured to simulate a neural network.

9. The device according to claim 1, wherein, The post-processing component includes a plurality of digital-to-analog converters, each of which is associated with one of the processing units, wherein the processing unit is configured to prepare the data block for input to the digital-to-analog converter, and wherein the digital-to-analog converter is configured to synchronously transmit analog data as weighted input to the neural network component.

10. The device according to claim 2, wherein, The selection configuration matrix is ​​based on the metadata, specifically, on the sensor category and / or the sensor identifier.

11. The device according to claim 1, wherein, The selected configuration matrix is ​​scalable in both row and column dimensions.

12. A computer-implemented method, comprising: Receive data from the input data stream; The data is divided into data blocks, and the data blocks are stored in a set of different matrix buffers according to the selected configuration matrix until the matrix buffers are full. The data block is processed by multiple processing units, each processing unit being associated with a matrix buffer in the matrix buffer via an I / O block; The processed data blocks are provided as input to the neural network components; as well as The neural network component classifies the data and determines the relevance of the data based on the classification, filtering out data that is determined to have low or no relevance. The data stream includes data from multiple sensors, and the data from each sensor includes metadata and load data, the metadata being associated with the sensor and the load data.

13. The computer-implemented method according to claim 12, wherein, The metadata includes sensor identifier, sensor category, and data identifier.

14. The computer-implemented method of claim 12 further includes providing the filtered data to a storage device for storage.

15. The computer-implemented method of claim 12, further comprising, once all of the matrix buffers are filled, providing the data blocks in a synchronous manner for processing.

16. The computer-implemented method according to claim 12, wherein, Processing the data block by the plurality of processing units includes modifying the data block according to post-processing rules to perform one or more of the following operations: normalization, pre-filtering, outlier detection, and region of interest identification.

17. The computer-implemented method according to claim 12, further comprising: After the I / O blocks are filled, data blocks are provided from each I / O block for processing; as well as The processed data blocks are provided to the neural network component in a synchronous manner.

18. The computer-implemented method according to claim 12, wherein, The neural network component is configured to simulate a neural network; as well as The method further includes: The processed data blocks are converted into analog data via a set of digital-to-analog converters, each of which is associated with one of the plurality of processing units; and The simulated data is provided as a weighted input to the neural network component in a synchronous manner.

19. A computer program product comprising a computer-readable storage medium having program instructions executable by a computer system to cause the computer system to perform operations, the operations including: Receive data from the input data stream; The data is divided into data blocks, and the data blocks are stored in a set of different matrix buffers according to the selected configuration matrix until the matrix buffers are full. The data block is processed by multiple processing units, each processing unit being associated with a matrix buffer in the matrix buffer via an I / O block; The processed data blocks are provided as input to the neural network components; as well as The neural network component classifies the data and determines the relevance of the data based on the classification, filtering out data that is determined to have low or no relevance. The data stream includes data from multiple sensors, and the data from each sensor includes metadata and load data, the metadata being associated with the sensor and the load data.

Images