Multi-scene data intelligent processing system

Abstract

The invention discloses a multi-scene data intelligent processing system, which relates to the technical field of feature extraction, and comprises the following steps: collecting real-time data from a plurality of sources, and preprocessing the real-time data; converting the preprocessed data into time series data, and constructing a converter model for feature extraction; carrying out standardization processing on the extracted feature vector data; fusing the standardized feature vector data through a converter model to generate a comprehensive data set; a CDP is used for storing the fused comprehensive data set in a database in a distributed mode, and data are backed up regularly through billion cloud zero-trust SASE; according to the method, the complex mode and the dependency relationship in the data are captured through the self-attention mechanism, and the local window is defined through the sparse attention mechanism, so that the calculation complexity and the memory consumption are reduced.

Description

Technical Field

[0001] The present invention relates to the technical field of feature extraction, and in particular to a multi-scenario data intelligent processing system. Background Art

[0002] In recent years, with the rapid development of Internet of Things technology, multi-scenario data intelligent processing systems have been widely used in many fields such as industry, agriculture, and urban management; traditional data processing methods usually rely on a single data source and simple statistical analysis, which is difficult to meet the needs of today's complex application scenarios; in order to improve the efficiency and accuracy of data processing, researchers have proposed a variety of data processing frameworks and technologies; among them, the transformer model based on deep learning has attracted much attention due to its powerful feature extraction ability and self-attention mechanism; these models can extract key features from multi-source heterogeneous data and realize effective data processing through time series analysis; however, the existing multi-scenario data intelligent processing systems still have many shortcomings.

[0003] Existing technologies often use simple filtering and unit conversion methods in the data preprocessing stage, which cannot effectively remove noise and outliers, resulting in limited subsequent feature extraction; in addition, although traditional feature extraction methods such as convolutional neural networks (CNN) and recurrent neural networks (RNN) perform well on certain tasks, they have problems with high computational complexity and large memory consumption when processing long sequence data; especially in multimodal data fusion, existing methods usually use a simple weighted average method, but fail to fully utilize the correlation between the data of each modality; in addition, the security and reliability of data storage and backup are also issues that need to be addressed urgently. Summary of the invention

[0004] In view of the above existing problems, the present invention is proposed.

[0005] Therefore, the present invention provides a multi-scenario data intelligent processing system to solve the problems of high computational complexity and large memory consumption when processing long sequence data.

[0006] In order to solve the above technical problems, the present invention provides the following technical solutions:

[0007] In the first aspect, the present invention provides a multi-scenario data intelligent processing system, which includes: a data acquisition and preprocessing module, which collects real-time data from multiple sources and performs preprocessing; a feature extraction module, which converts the preprocessed data into time series data, and constructs a transformer model for feature extraction; a cleaning module, which standardizes the extracted feature vector data; a fusion module, which fuses the standardized feature vector data through a transformer model to generate a comprehensive data set; a storage module, which uses CDP to distribute and store the fused comprehensive data set in a database, and regularly backs up the data through Yige Cloud Zero Trust SASE.

[0008] As a preferred solution of the multi-scenario data intelligent processing system of the present invention, wherein: the real-time data is collected from multiple sources, and the specific steps are:

[0009] Use cameras to collect video streams, temperature sensors to collect temperature data, humidity sensors to collect humidity data, and RFID readers to collect location data;

[0010] The data is transmitted to the central server via the MQTT network protocol.

[0011] As a preferred solution of the multi-scenario data intelligent processing system of the present invention, the preprocessing comprises the following specific steps:

[0012] Use a median filter to remove noise from the video stream data, convert the color image into a grayscale image, and adjust the video frame rate and video resolution;

[0013] Use a sliding average filter to remove noise from the temperature and humidity data, use the IQR method to detect and remove data outliers, and convert the Celsius unit of the temperature data into the Fahrenheit unit, and convert the humidity data unit into absolute humidity;

[0014] The location data is cleaned to remove invalid location data, and the location data is converted from GPS coordinates to a local coordinate system, and Kalman filtering is used to remove location data noise.

[0015] As a preferred solution of the multi-scenario data intelligent processing system of the present invention, the conversion into time series data comprises the following specific steps:

[0016] Arrange each frame of the preprocessed video stream data in order according to the acquisition timestamp, adjust the video frame rate and resolution to form time series data;

[0017] Arrange the preprocessed temperature data in the order of the collected timestamps and arrange the temperature readings at each time point in chronological order to form time series data;

[0018] Arrange the preprocessed humidity data in the order of the collected timestamps and arrange the humidity readings at each time point in chronological order to form time series data;

[0019] The preprocessed position data are arranged in the order of the collected timestamps, and the position coordinates of each time point are arranged in chronological order to form time series data.

[0020] As a preferred solution of the multi-scenario data intelligent processing system of the present invention, wherein: the transformer model is constructed to extract features, and the specific steps are:

[0021] Choose BERT as the model architecture of the transformer;

[0022] Define the input layer as time series data and the output layer as processed feature vector data;

[0023] Based on the self-attention mechanism, the input time series data is linearly transformed to obtain the query feature vector matrix Q, the key feature vector matrix K and the value feature vector matrix V;

[0024] By querying the feature vector matrix, the key feature vector matrix and the value feature vector matrix, the attention score is calculated and the feature vector data O is output. The expression is:

[0025]

[0026] Among them, softmax is the normalization function, T is the transpose, and d k is the dimension of the eigenvector data matrix;

[0027] A sparse attention mechanism is introduced and the output is new feature vector data.

[0028] As a preferred solution of the multi-scenario data intelligent processing system of the present invention, wherein: the sparse attention mechanism is introduced and the output is new feature vector data, and the specific steps are as follows:

[0029] The sparse attention mechanism is introduced, and the local window of the feature vector data is defined as w, and the position is t. The attention score of the local range feature vector data is calculated and output as the new feature vector data O'. The expression is:

[0030]

[0031] Among them, O t is the feature vector data at position t, O [t-w,t+w] is the feature vector data near position t.

[0032] As a preferred solution of the multi-scenario data intelligent processing system of the present invention, wherein: the extracted feature vector data is subjected to standardization processing, and the specific steps are as follows:

[0033] The feature vector data is standardized by Z-score;

[0034] Use the quartile method to detect eigenvector data and remove outliers;

[0035] The DBSCAN clustering algorithm is used to remove duplicate feature vector data.

[0036] As a preferred solution of the multi-scenario data intelligent processing system of the present invention, wherein: the feature vector data is fused through the transformer model, and the specific steps are:

[0037] The feature vector data of the video stream, temperature data, humidity data and location data are fused through the feature vector data output by the transformer model to generate a comprehensive data set O' f , the expression is:

[0038] O' f =[O' a ,O' h ,O' n ,O' l ];

[0039] Among them, O' a is the feature vector data of the video stream, O' h is the characteristic vector data of temperature, O' n is the characteristic vector data of humidity, O' l is the feature vector data of the position.

[0040] As a preferred solution of the multi-scenario data intelligent processing system of the present invention, wherein: the use of CDP to store the fused comprehensive data set in a distributed manner in a database, the specific steps are:

[0041] Download and install Cloudera Manager.

[0042] Access the Cloudera Manager web interface through a browser and install CDP, configure cluster basic settings, and install Hadoop HDFS and other components in the web interface;

[0043] Use the Hadoop command-line tool to create an HDFS directory to store the comprehensive dataset.

[0044] Upload the synthetic dataset from a local file to the HDFS storage directory.

[0045] As a preferred solution of the multi-scenario data intelligent processing system of the present invention, the specific steps of regularly backing up data through Yige Cloud Zero Trust SASE are as follows:

[0046] Visit the official website of Yige Cloud, register and create an account;

[0047] Log in to the Yige Cloud Management Console;

[0048] In the security management of Yige Cloud Management Console, create security policies, set access control rules and configure network access policies;

[0049] In the backup management of the Yige Cloud Management Console, create a backup task and set the backup source to the storage directory of HDFS;

[0050] Select Yige Cloud's object storage service as the storage location and set the backup frequency and backup retention policy.

[0051] The beneficial effects of the present invention are: complex patterns and dependencies in data are captured through the self-attention mechanism, and local windows are defined through the sparse attention mechanism, which reduces computational complexity and memory consumption, and is particularly suitable for processing large-scale time series data; the combination of these two mechanisms not only improves the ability of feature representation, but also reduces the demand for computing resources, making it more efficient and reliable. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0053] Figure 1 This is a system diagram of the multi-scenario data intelligent processing system in Example 1.

[0054] Figure 2 This is a schematic diagram of using CDP in Example 1 to store comprehensive data in a database. DETAILED DESCRIPTION

[0055] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are described in detail below in conjunction with the accompanying drawings.

[0056] In the following description, many specific details are set forth to facilitate a full understanding of the present invention, but the present invention may also be implemented in other ways different from those described herein, and those skilled in the art may make similar generalizations without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0057] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The term "in one embodiment" that appears in different places in this specification does not necessarily refer to the same embodiment, nor does it refer to a separate or selective embodiment that is mutually exclusive with other embodiments.

[0058] Example 1, reference Figure 1 and Figure 2 , which is the first embodiment of the present invention, and provides a multi-scenario data intelligent processing system, comprising the following steps:

[0059] Data acquisition and preprocessing module: used to collect real-time data from multiple sources and perform preprocessing.

[0060] The specific steps for collecting real-time data from multiple sources are:

[0061] Use cameras to collect video streams, temperature sensors to collect temperature data, humidity sensors to collect humidity data, and RFID readers to collect location data;

[0062] The data is transmitted to the central server via the MQTT network protocol.

[0063] The specific steps for preprocessing are:

[0064] Use a median filter to remove noise from the video stream data, convert color images into grayscale images, adjust the video frame rate and video resolution, and reduce the amount of video stream data;

[0065] Use a sliding average filter to remove noise from the temperature and humidity data, use the IQR method to detect and remove data outliers, and convert the Celsius unit of the temperature data into the Fahrenheit unit, and convert the humidity data unit into absolute humidity;

[0066] The location data is cleaned to remove invalid location data, and the location data is converted from GPS coordinates to a local coordinate system, and Kalman filtering is used to remove location data noise.

[0067] It should be noted that real-time data is collected from multiple sources through cameras, temperature sensors, humidity sensors and RFID readers, and transmitted to the central server through the MQTT protocol; preprocessing includes: median filtering, grayscale conversion, frame rate and resolution adjustment of the video stream to reduce the amount of data; sliding average filtering and IQR method are applied to the temperature and humidity data to remove noise and outliers, and unify the units; the location data is cleaned, coordinate converted and Kalman filtered for denoising; this step improves data quality and consistency.

[0068] By using cameras to collect video streams, temperature sensors to collect temperature data, humidity sensors to collect humidity data, and RFID readers to collect location data, and transmitting the data to the central server through the MQTT network protocol, real-time collection and efficient transmission of multi-source data are achieved; the low bandwidth occupancy, low latency and high reliability characteristics of the MQTT protocol ensure that data can be transmitted to the central server quickly and accurately, reducing the risk of data loss and transmission errors; in addition, the high efficiency of the MQTT protocol reduces the demand for network resources, which is particularly suitable for resource-constrained IoT environments; the collection and transmission of real-time data supports real-time monitoring and rapid response of the system, improving the system's response speed and decision-making efficiency.

[0069] Feature extraction module: used to convert the preprocessed data into time series data and build a transformer model for feature extraction.

[0070] Convert to time series data. The specific steps are:

[0071] Arrange each frame of the preprocessed video stream data in order according to the acquisition timestamp, adjust the video frame rate and resolution to form time series data;

[0072] Arrange the preprocessed temperature data in the order of the collected timestamps and arrange the temperature readings at each time point in chronological order to form time series data;

[0073] Arrange the preprocessed humidity data in the order of the collected timestamps and arrange the humidity readings at each time point in chronological order to form time series data;

[0074] The preprocessed position data are arranged in the order of the collected timestamps, and the position coordinates of each time point are arranged in chronological order to form time series data.

[0075] Construct a transformer model to extract data features. The specific steps are:

[0076] Furthermore, BERT is selected as the model architecture of the transformer;

[0077] Define the input layer as time series data and the output layer as processed feature vector data;

[0078] Based on the self-attention mechanism, the input time series data is linearly transformed to obtain the query feature vector matrix Q, the key feature vector matrix K and the value feature vector matrix V;

[0079] By querying the feature vector matrix, the key feature vector matrix and the value feature vector matrix, the attention score is calculated and the feature vector data O is output. The expression is:

[0080]

[0081] Among them, softmax is the normalization function, T is the transpose, and d k is the dimension of the eigenvector data matrix;

[0082] The sparse attention mechanism is introduced, and the local window of the feature vector data is defined as w, and the position is t. The attention score of the local range feature vector data is calculated and output as the new feature vector data O'. The expression is:

[0083]

[0084] Among them, O tis the feature vector data at position t, O [t-w,t+w] is the feature vector data near position t;

[0085] It should be noted that in order to capture more complex patterns and relationships, the feature vector data processed by the sparse attention mechanism is nonlinearly transformed through a two-layer fully connected network to extract more complex feature vector data.

[0086] It should be noted that more complex feature vector data is extracted, specifically,

[0087] Input the feature vector data after sparse attention mechanism processing to the first layer of the fully connected network, perform linear transformation, and apply the ReLU activation function to obtain the feature vector data after the ReLU activation function;

[0088] The feature vector data after the ReLU activation function in the first layer is input into the fully connected layer of the second layer, and after linear transformation, the final complex feature vector data is obtained.

[0089] For example:

[0090] Define the time series data as X, with a shape of [N, d], where N is the length of the time series data and d is the feature dimension of the time series data;

[0091] Define the processed feature vector data as O;

[0092] Assuming that the length of the time series data is N = 5 and the feature dimension of the time series data is d = 3, then X is:

[0093]

[0094] Among them, x 1 、x 2 and x 3 Represents the three features of the first time step (i.e., time point), x 4 、x 5 and x 6 Represents the three features of the second time step, and so on, until the last row of the matrix;

[0095] Based on the self-attention mechanism, the input time series data is linearly transformed to obtain the query feature vector matrix Q, the key feature vector matrix K and the value feature vector matrix V, which are expressed as:

[0096] Q=XW Q ; K = XW K ; V = XW V ;

[0097] Among them, W Q is the query matrix, with shape [d, d k],W K is the key matrix, with shape [d, d k ],W V is a value matrix with shape [d, d v ];

[0098] Query the eigenvector matrix Q:

[0099] Q=XW Q ;

[0100] Assume the query matrix W Q for:

[0101]

[0102] but, Key eigenvector matrix K:

[0103] K=XW K ;

[0104] Assume that the bond matrix W K for:

[0105]

[0106] but, Value eigenvector matrix V:

[0107] V=XW V ;

[0108] Assumption value matrix W V for:

[0109]

[0110] but, Calculate the attention score, the expression is:

[0111]

[0112] Applying softmax normalization, it is expressed as:

[0113] Attention Weights=softmax(Attention Scores);

[0114] The weighted summation obtains the feature vector data O, which is expressed as:

[0115] O = Attention Weights V;

[0116] Introduce a sparse attention mechanism, assuming a local window size w;

[0117] Calculate the attention score in the local range, the expression is:

[0118]

[0119] Applying softmax normalization, it is expressed as:

[0120] Local Attention Weights=softmax(Local Attention Scores);

[0121] The weighted summation obtains the feature vector data O, which is expressed as:

[0122] O'=Local Attention Weights·O [t-w,t+w] ;

[0123] Assume that the time series data is X, the length of the time series data is N = 5, and the feature dimension of the time series data is d = 3, which can be expressed as:

[0124]

[0125] Assumption d k =d v = 2, the query matrix, key matrix and value matrix are in shape [3, 2];

[0126]

[0127]

[0128] Calculate the attention score, set the position to 2, and express it as:

[0129]

[0130] Attention Scores 2 =[83.1 125.7166.3209.9252.5];

[0131] Through softmax normalization, we get the feature vector data O, which is expressed as:

[0132] Attention Weights=softmax([83.1 125.7166.3209.9252.5]);

[0133] Attention Weights≈[0.00010.00030.00100.00330.9953];

[0134]

[0135] O≈[69.8 78.2];

[0136] Assume that the local window size w=3, the position t=2, and the local range is [1, 5];

[0137] Calculate the attention score within the local range, expressed as:

[0138]

[0139] Local Attention Scores=[493.51237.51975.52712.53449.5];

[0140] Through softmax normalization, the feature vector data O' is obtained, which is expressed as:

[0141] Local Attention Weights=softmax([493.51237.51975.52712.53449.5]);

[0142] Local Attention Weights≈[0.00000.00000.00000.00010.9999];

[0143]

[0144] O'≈[70.0 78.4];

[0145] Through a series of assumed values, the final new feature vector data O' is calculated.

[0146] It should be noted that the original formula is split and explained in detail in the above examples in order to step by step explain the specific meaning and calculation method of each matrix operation, help understand the working principles of the self-attention mechanism and the sparse attention mechanism, and ensure that each step is transparent and easy to understand.

[0147] It should be noted that by arranging the preprocessed video stream, temperature, humidity and location data by timestamp and adjusting the video frame rate and resolution to form time series data, the time alignment and consistency of the data are achieved; this content not only improves the efficiency of data processing, but also enhances the timing analysis capability, making it easier to capture dynamic changes and trends; BERT is selected as the transformer model architecture, and feature extraction is performed based on the self-attention mechanism and the sparse attention mechanism to generate richer and more expressive feature vectors; the self-attention mechanism can automatically focus on the key parts of the data, capture long-distance dependencies, and improve the quality of feature representation; the sparse attention mechanism reduces the computational complexity and speeds up the processing by defining local windows, which is particularly suitable for large-scale data sets.

[0148] Cleaning module: standardize the extracted feature vector data.

[0149] The feature vector data is standardized by Z-score;

[0150] Use the quartile method to detect eigenvector data and remove outliers;

[0151] The DBSCAN clustering algorithm is used to remove duplicate feature vector data.

[0152] It should be noted that by performing Z-score standardization on the feature vector data, a unified scale of the data is achieved, the dimensional influence between different features is eliminated, the convergence speed and prediction accuracy of the model are improved, and the comparability of the data is enhanced; the quartile method is used to detect and remove outliers, which ensures the quality of the data, reduces the interference of noise on subsequent analysis, and enhances the robustness and generalization ability of the model; the DBSCAN clustering algorithm is used to remove duplicate feature vector data, which reduces data redundancy, reduces the demand for storage and computing resources, speeds up data processing and analysis, and enhances the representativeness of the data, so that it more accurately reflects the actual data distribution.

[0153] Fusion module: The standardized feature vector data is fused through the transformer model to generate a comprehensive data set.

[0154] Furthermore, the feature vector data of the video stream, temperature data, humidity data and location data are fused through the feature vector data output by the transformer model to generate a comprehensive data set O' f , the expression is:

[0155] O' f =[O' a ,O' h ,O' n ,O' l ];

[0156] Among them, O' a is the feature vector data of the video stream, O' h is the characteristic vector data of temperature, O' n is the characteristic vector data of humidity, O' l is the feature vector data of the position.

[0157] It should be noted that the feature vector data of the video stream, temperature data, humidity data and location data are fused through the feature vector data output by the transformer model to generate a comprehensive data set; this step realizes the fusion of multimodal data and provides a unified data representation; this fusion not only enhances the correlation between data, but also improves the accuracy and depth of data analysis, supporting more advanced applications; at the same time, it simplifies the subsequent data processing process and improves the overall processing efficiency.

[0158] Storage module: Use CDP to store the integrated data set in a distributed manner in the database, and back up the data regularly through Yige Cloud Zero Trust SASE.

[0159] Use CDP to store the fused feature vector data in a distributed manner in the database. The specific steps are:

[0160] Going further, download and install Cloudera Manager;

[0161] Access the Cloudera Manager web interface through a browser and install CDP, configure cluster basic settings, and install Hadoop HDFS and other components in the web interface;

[0162] Use the Hadoop command-line tool to create an HDFS directory to store the comprehensive dataset.

[0163] Upload the comprehensive data set from the local file to the HDFS storage directory and confirm that the data has been uploaded successfully.

[0164] The basic settings of the cluster include: name, host list, storage directory and other basic settings;

[0165] The key parameters to be configured when installing Hadoop HDFS are: NameNode, Secondary NameNode, DataNodes, and replication factor;

[0166] NameNode is the location of the master node;

[0167] Secondary NameNode is the location of the secondary node;

[0168] DataNodes is the location of data nodes;

[0169] The replication factor is an important configuration of HDFS, which is used to specify the number of copies of each file block. The default value is usually 3. The replication factor determines the redundancy and fault tolerance of the data.

[0170] Other components include: YARN, Hive, HBase, Spark, ZooKeeper, Kafka, Flume, and Solr;

[0171] YARN (Yet Another Resource Negotiator) is used for resource management and task scheduling;

[0172] Hive is a data warehouse tool used to query and manage large data sets;

[0173] HBase is a distributed, scalable big data storage system that supports real-time read and write access;

[0174] Spark is a fast general-purpose computing engine for large-scale data processing;

[0175] ZooKeeper is a distributed coordination service that provides functions such as configuration management, naming services, and distributed synchronization;

[0176] Kafka is a high-throughput distributed messaging system for building real-time data pipelines and streaming applications.

[0177] Flume is used to efficiently collect, aggregate, and move large amounts of log data;

[0178] Solr is an enterprise-level search platform based on Apache Lucene.

[0179] It should be noted that Cloudera Data Platform (CDP) is an enterprise-level data management and analysis platform provided by Cloudera; it integrates a variety of big data processing and analysis tools, provides a unified data management interface, and supports multiple functions such as data storage, processing, analysis, and machine learning; CDP can run in multi-cloud and hybrid cloud environments to help enterprises manage and analyze large-scale data.

[0180] Back up data regularly through Yige Cloud Zero Trust SASE. The specific steps are as follows:

[0181] Visit the official website of Yige Cloud, register and create an account;

[0182] Log in to the Yige Cloud Management Console;

[0183] In the security management of Yige Cloud Management Console, create security policies, set access control rules and configure network access policies;

[0184] In the backup management of the Yige Cloud Management Console, create a backup task and set the backup source to the storage directory of HDFS;

[0185] Select Yige Cloud's object storage service as the storage location, and set the backup frequency and backup retention policy;

[0186] Manually trigger the backup task, confirm that the backup process is running normally, and check the backup files in the Yige Cloud storage.

[0187] It should be noted that Yige Cloud SASE is a security solution based on zero-trust network architecture, combining wide area network (WAN) optimization, network security services (such as firewalls, intrusion detection / prevention, etc.) and cloud security functions; SASE aims to provide consistent security policies and performance optimization at the edge, which is suitable for the distributed office environment of modern enterprises; the SASE solution provided by Yige Cloud can help enterprises achieve integrated management of office security.

[0188] It should be noted that Hadoop HDFS is a core component of the Apache Hadoop project. It is a distributed file system designed to store large amounts of data and provide high-throughput data access. HDFS achieves data reliability and fault tolerance by distributing data across multiple nodes.

[0189] In summary, the present invention captures complex patterns and dependencies in the data through the self-attention mechanism, and defines local windows through the sparse attention mechanism, which reduces the computational complexity and memory consumption, and is particularly suitable for processing large-scale time series data; further, nonlinear transformation is performed through a two-layer fully connected network to extract more complex feature vector data, thereby enhancing the data representation ability and the generalization ability of the model.

[0190] It should be noted that by downloading and installing Cloudera Manager, effective management and monitoring of the Hadoop cluster is achieved, improving management efficiency and system stability; installing CDP, configuring cluster basic settings, and installing Hadoop HDFS and other components through the browser to access the Cloudera Manager's Web interface simplifies the deployment process, provides a unified management and monitoring interface, and enhances the flexibility of the system; using the Hadoop command line tool to create an HDFS directory ensures data security and efficient storage and optimizes storage performance; uploading comprehensive data sets from local files to the HDFS storage directory improves data availability and processing efficiency and supports large-scale data processing.

[0191] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present invention, which should all be included in the scope of the claims of the present invention.

Claims

1. A multi-scenario data intelligent processing system, characterized by: include, Data collection and preprocessing module collects real-time data from multiple sources and performs preprocessing; Feature extraction module, which converts the preprocessed data into time series data and builds a transformer model for feature extraction; A cleaning module standardizes the extracted feature vector data; The fusion module fuses the standardized feature vector data through the transformer model to generate a comprehensive data set; The storage module uses CDP to store the integrated data set in a distributed manner in the database, and regularly backs up the data through Yige Cloud Zero Trust SASE.

2. The multi-scenario data intelligent processing system according to claim 1, characterized in that: The specific steps of collecting real-time data from multiple sources are as follows: Use cameras to collect video streams, temperature sensors to collect temperature data, humidity sensors to collect humidity data, and RFID readers to collect location data; The data is transmitted to the central server via the MQTT network protocol.

3. The multi-scenario data intelligent processing system according to claim 2, characterized in that: The pretreatment comprises the following specific steps: Use a median filter to remove noise from the video stream data, convert the color image into a grayscale image, and adjust the video frame rate and video resolution; Use a sliding average filter to remove noise from the temperature and humidity data, use the IQR method to detect and remove data outliers, and convert the Celsius unit of the temperature data into the Fahrenheit unit, and convert the humidity data unit into absolute humidity; The location data is cleaned to remove invalid location data, and the location data is converted from GPS coordinates to a local coordinate system, and Kalman filtering is used to remove location data noise.

4. The multi-scenario data intelligent processing system according to claim 3, characterized in that: The conversion into time series data includes the following specific steps: Arrange each frame of the preprocessed video stream data in order according to the acquisition timestamp, adjust the video frame rate and resolution to form time series data; Arrange the preprocessed temperature data in the order of the collected timestamps and arrange the temperature readings at each time point in chronological order to form time series data; Arrange the preprocessed humidity data in the order of the collected timestamps and arrange the humidity readings at each time point in chronological order to form time series data; The preprocessed position data are arranged in the order of the collected timestamps, and the position coordinates of each time point are arranged in chronological order to form time series data.

5. The multi-scenario data intelligent processing system according to claim 4, characterized in that: The transformer model is constructed to extract features, and the specific steps are: Choose BERT as the model architecture of the transformer; Define the input layer as time series data and the output layer as processed feature vector data; Based on the self-attention mechanism, the input time series data is linearly transformed to obtain the query feature vector matrix Q, the key feature vector matrix K and the value feature vector matrix V; By querying the feature vector matrix, the key feature vector matrix and the value feature vector matrix, the attention score is calculated and the feature vector data O is output. The expression is: Among them, softmax is the normalization function, T is the transpose, and d k is the dimension of the eigenvector data matrix; A sparse attention mechanism is introduced and the output is new feature vector data.

6. The multi-scenario data intelligent processing system according to claim 5, characterized in that: The sparse attention mechanism is introduced and the output is new feature vector data. The specific steps are: The sparse attention mechanism is introduced, and the local window of the feature vector data is defined as w, and the position is t. The attention score of the local range feature vector data is calculated and output as the new feature vector data O'. The expression is: Among them, O t is the feature vector data at position t, O [t-w,t+w] is the feature vector data near position t.

7. The multi-scenario data intelligent processing system according to claim 6, characterized in that: The specific steps of standardizing the extracted feature vector data are as follows: The feature vector data is standardized by Z-score; Use the quartile method to detect eigenvector data and remove outliers; The DBSCAN clustering algorithm is used to remove duplicate feature vector data.

8. The multi-scenario data intelligent processing system according to claim 7, characterized in that: The feature vector data is fused through the transformer model, and the specific steps are as follows: The feature vector data of the video stream, temperature data, humidity data and location data are fused through the feature vector data output by the transformer model to generate a comprehensive data set O' f , the expression is: Oh f =[O' a ,Oh' h ,Oh' n ,Oh' l ]; Among them, O' a is the feature vector data of the video stream, O' h is the characteristic vector data of temperature, O' n is the characteristic vector data of humidity, O' l is the feature vector data of the position.

9. The multi-scenario data intelligent processing system according to claim 8, characterized in that: The method uses CDP to store the fused comprehensive data set in a distributed manner in a database, and the specific steps are: Download and install Cloudera Manager. Access the Cloudera Manager web interface through a browser and install CDP, configure cluster basic settings, and install Hadoop HDFS and other components in the web interface. Use the Hadoop command-line tool to create an HDFS directory to store the comprehensive dataset. Upload the synthetic dataset from a local file to the HDFS storage directory.

10. The multi-scenario data intelligent processing system according to claim 9, characterized in that: The specific steps of regularly backing up data through Yige Cloud Zero Trust SASE are as follows: Visit the official website of Yige Cloud, register and create an account; Log in to the Yige Cloud Management Console; In the security management of Yige Cloud Management Console, create security policies, set access control rules and configure network access policies; In the backup management of the Yige Cloud Management Console, create a backup task and set the backup source to the storage directory of HDFS; Select Yige Cloud's object storage service as the storage location and set the backup frequency and backup retention policy.

Images