Artificial intelligence system and method based on big data training application

Abstract

The invention discloses an artificial intelligence system based on big data training application. The artificial intelligence system comprises a distributed data acquisition module, a data preprocessing module, a training cluster construction module and a storage module, the distributed data acquisition module is used for acquiring multi-source heterogeneous data in real time, and the multi-source heterogeneous data comprises a structured database, an unstructured text, time sequence sensor data and network flow data; the data preprocessing module is used for acquiring output information of the distributed data acquisition module and carrying out parallel data cleaning, feature dimension reduction and dynamic noise filtering to generate a standardized data set; the training cluster construction module is used for obtaining a standardized data set and carrying out sample quantification to construct a sample set for training; the system further comprises a training model used based on the training sample set, and the training model outputs classification data to the storage module. Compared with the prior art, the artificial intelligence system based on the big data training application has the advantages that data arrangement is facilitated, and subsequent data information extraction is facilitated.

Description

Technical Field

[0001] The present invention relates to the field of data analysis technology, and specifically to an artificial intelligence system and method based on big data training applications. Background Art

[0002] The wave of digitalization is sweeping in. Mobile Internet, Internet of Things, artificial intelligence, big data, cloud architecture, blockchain... These digital technologies are subverting the traditional industry structure and market environment and leading to changes in user consumption habits.

[0003] Data has become a vital resource in modern society. Faced with this massive amount of data, effectively collecting, processing, analyzing, and utilizing it has become a major challenge for businesses. Artificial intelligence and big data, two of the most popular technologies in the current information technology field, provide powerful technical support for intelligent data management.

[0004] In existing data management systems, although some systems have attempted to apply artificial intelligence and big data technologies, their data processing effects are poor, and the processed data is difficult to quickly retrieve and apply, making it inconvenient to use in actual production. Summary of the Invention

[0005] The technical problem to be solved by the present invention is to overcome the above technical defects and provide an artificial intelligence system based on big data training applications that is convenient for data organization and subsequent extraction of data information.

[0006] To solve the above technical problems, the present invention provides a technical solution: an artificial intelligence system based on big data training applications, including a distributed data acquisition module, a data preprocessing module, a training cluster construction module, and a storage module;

[0007] The distributed data acquisition module includes real-time acquisition of multi-source heterogeneous data, including structured databases, unstructured text, time-series sensor data, and network flow data;

[0008] The data preprocessing module obtains the output information of the distributed data acquisition module and performs parallel data cleaning, feature dimension reduction and dynamic noise filtering to generate a standardized data set;

[0009] The training cluster construction module obtains a standardized data set and performs sample quantitative construction to form a training sample set;

[0010] It also includes a training model used based on the training sample set, and the training model outputs classification data to the storage module.

[0011] Preferably, the training model is a hybrid neural network architecture that includes an attention mechanism, including a spatiotemporal attention subnetwork and a graph attention subnetwork.

[0012] Preferably, the distributed data acquisition module further includes generating data metadata with a timestamp and source identifier.

[0013] Preferably, the dynamic noise filtering in the data preprocessing module includes an adversarial sample detection subunit.

[0014] Preferably, the training cluster construction module further includes assigning adversarial samples based on sample weights of cluster distribution.

[0015] Preferably, it further includes an adjustment feedback module, which monitors the feature space evolution of the training model and dynamically adjusts the data enhancement parameters and the model regularization coefficient.

[0016] Preferably, a retrieval construction unit is provided in the storage module to generate a graph database index that supports multi-dimensional retrieval; and a feature hash inverted index is established to accelerate neighbor queries.

[0017] Another aspect of the present invention further discloses an artificial intelligence method based on big data training application, comprising the following steps:

[0018] S1: Multi-source heterogeneous data collection;

[0019] S2: dynamic preprocessing to generate standardized data sets;

[0020] S3: Use cluster distribution to allocate sample weights and construct a training sample set;

[0021] S4: Deploy a hybrid architecture consisting of a spatiotemporal attention subnetwork and a graph attention subnetwork to perform mixed precision training;

[0022] S5: Graph database index construction;

[0023] S6: Dynamic feedback regulation.

[0024] Preferably, said S5 includes heterogeneous graph embedding to perform feature alignment on multimodal data nodes.

[0025] The advantages of the present invention compared with the existing technology are: by integrating distributed data acquisition, dynamic noise filtering and adversarial sample detection technology, real-time fusion of multi-source heterogeneous data is achieved. At the same time, the present invention adopts a hybrid architecture of spatiotemporal attention and graph attention to enhance model stability. In the present invention, based on the processing results, multi-dimensional retrieval is possible based on feature hash inverted index and heterogeneous graph embedding technology, which greatly improves the subsequent retrieval effect and meets application needs. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 It is a structural diagram of an artificial intelligence system based on big data training applications.

[0027] Figure 2 It is a flowchart of an artificial intelligence method based on big data training application. DETAILED DESCRIPTION

[0028] The present invention will be described in further detail below with reference to the accompanying drawings.

[0029] Combined with attachment Figure 1-2 As shown, an artificial intelligence system based on big data training application includes a distributed data acquisition module, a data preprocessing module, a training cluster construction module, and a storage module; the distributed data acquisition module includes real-time acquisition of multi-source heterogeneous data, including structured databases, unstructured text, time series sensor data, and network flow data; the data preprocessing module obtains the output information of the distributed data acquisition module and performs parallel data cleaning, feature dimension reduction, and dynamic noise filtering to generate a standardized data set; the training cluster construction module obtains the standardized data set and performs sample quantitative construction as a training sample set; and also includes a training model used based on the training sample set, and the training model outputs classified data to the storage module.

[0030] In one embodiment, the training model is a hybrid neural network architecture that includes an attention mechanism, including a spatiotemporal attention subnetwork and a graph attention subnetwork, and the distributed data acquisition module also includes generating data metadata with timestamps and source identifiers. At the same time, the dynamic noise filtering in the data preprocessing module includes an adversarial sample detection subunit, and the training cluster construction module also includes sample weight distribution adversarial samples based on cluster distribution.

[0031] In one embodiment:

[0032] It also includes an adjustment feedback module, which monitors the feature space evolution of the training model and dynamically adjusts the data enhancement parameters and model regularization coefficients. A retrieval construction unit is provided in the storage module to generate a graph database index that supports multi-dimensional retrieval; and a feature hash inverted index is established to accelerate the use of neighbor queries.

[0033] The present invention, when implemented, comprises the following steps:

[0034] S1: Multi-source heterogeneous data acquisition: Using a distributed data acquisition module, we pull structured databases, unstructured text, time-series sensor data, and network flow data in real time while ensuring data consistency. S2: Dynamic preprocessing: Performing data cleaning, feature dimensionality reduction, and dynamic noise filtering to generate a standardized dataset. S3: Using clustered sample weight distribution to construct training sample sets. S4: Deploying a hybrid architecture consisting of spatiotemporal attention subnetworks and graph attention subnetworks to perform mixed-precision training. S5: Graph database index construction. S6: Dynamic feedback regulation: Real-time monitoring of feature space evolution and dynamic adjustment of data augmentation intensity. S5 also includes heterogeneous graph embedding to perform feature alignment on multimodal data nodes.

[0035] The contents not described in detail in this specification belong to the prior art known to those skilled in the art.

[0036] The basic principles, main features and advantages of the present invention are shown and described above. It should be understood by those skilled in the art that the present invention is not limited to the above embodiments. The above embodiments and descriptions are merely illustrative of the principles of the present invention. Various changes and modifications may be made to the present invention without departing from the spirit and scope of the present invention, and such changes and modifications fall within the scope of the invention as claimed.

[0037] The scope of protection of the present invention is defined by the appended claims and their equivalents, and therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention but is merely representative of selected embodiments of the present invention.

[0038] Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making any creative work shall fall within the scope of protection of the present invention.

[0039] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, it does not need to be further defined or explained in subsequent drawings.

[0040] The present invention and its embodiments are described above. This description is not restrictive. The drawings show only one embodiment of the present invention, and the actual structure is not limited thereto. In short, if a person skilled in the art is inspired by this and, without departing from the purpose of the present invention, designs structures and embodiments similar to this technical solution without inventiveness, they shall fall within the scope of protection of the present invention.

Claims

1. An artificial intelligence system based on big data training applications, characterized by: Includes distributed data acquisition module, data preprocessing module, training cluster construction module, and storage module; The distributed data acquisition module includes real-time acquisition of multi-source heterogeneous data, including structured databases, unstructured text, time-series sensor data, and network flow data; The data preprocessing module obtains the output information of the distributed data acquisition module and performs parallel data cleaning, feature dimension reduction and dynamic noise filtering to generate a standardized data set; The training cluster construction module obtains a standardized data set and performs sample quantitative construction to form a training sample set; It also includes a training model used based on the training sample set, and the training model outputs classification data to the storage module.

2. The artificial intelligence system based on big data training application according to claim 1, characterized in that: The training model is a hybrid neural network architecture that includes an attention mechanism, including a spatiotemporal attention subnetwork and a graph attention subnetwork.

3. The artificial intelligence system based on big data training application according to claim 1, characterized in that: The distributed data acquisition module also includes generating data metadata with a timestamp and a source identifier.

4. The artificial intelligence system based on big data training application according to claim 1, characterized in that: The dynamic noise filtering in the data preprocessing module includes an adversarial sample detection subunit.

5. The artificial intelligence system based on big data training application according to claim 1, characterized in that: The training cluster construction module also includes assigning adversarial samples based on sample weights of cluster distribution.

6. The artificial intelligence system based on big data training application according to claim 1, characterized in that: It also includes an adjustment feedback module, which monitors the feature space evolution of the training model and dynamically adjusts the data enhancement parameters and the model regularization coefficient.

7. The artificial intelligence system based on big data training application according to claim 1, characterized in that: The storage module is provided with a retrieval construction unit to generate a graph database index that supports multi-dimensional retrieval; Establish a feature hash inverted index to speed up neighbor queries.

8. An artificial intelligence method based on big data training applications, applied to the artificial intelligence system according to any one of claims 1 to 7, characterized in that: The steps include: S1: Multi-source heterogeneous data collection; S2: dynamic preprocessing to generate standardized data sets; S3: Use cluster distribution to allocate sample weights and construct a training sample set; S4: Deploy a hybrid architecture consisting of a spatiotemporal attention subnetwork and a graph attention subnetwork to perform mixed precision training; S5: Graph database index construction; S6: Dynamic feedback regulation.

9. The artificial intelligence method based on big data training application according to claim 8, characterized in that: The S5 includes heterogeneous graph embedding and performs feature alignment on multimodal data nodes.

Images