Big data intelligent modeling system and method based on dynamic metadata

A metadata and big data technology, applied in the field of big data intelligent modeling system based on dynamic metadata, can solve the problems of lowering the threshold of big data mining modeling technology, limited applicable business scenarios, low modeling efficiency, etc. Technical threshold, good real-time performance, and the effect of avoiding time consumption

Active Publication Date: 2020-05-08
BEIJING HUARU TECH
View PDF10 Cites 7 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0003] In order to obtain potential knowledge from massive amounts of information, a variety of systems or related practical tools for big data mining have emerged as the times require, realizing the integration of various technologies for big data from storage, calculation to visualization, etc., reducing the impact of big data to a certain extent. The technical threshold of mining modeling, but the following problems still exist in the existing technology: First, the modeling strategy is single, and the existing tools usua...

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Big data intelligent modeling system and method based on dynamic metadata
  • Big data intelligent modeling system and method based on dynamic metadata
  • Big data intelligent modeling system and method based on dynamic metadata

Examples

Experimental program
Comparison scheme
Effect test

Embodiment 1

[0046] see image 3 , this implementation takes the principal component analysis of big data as an example to describe the intelligent modeling method of this system and the interaction relationship of each module.

[0047] (1) Add start and finish to identify the start and end of the modeling process. Each link is called a process node, and the data conversion node is also called an operator node.

[0048] (2) Add a data source, automatically trigger the metadata collection module, generate data source metadata, and mark the metadata type as "exampleSet". Some metadata are shown in the table below.

[0049] name Types of Role scope level string label [low,middle,high] factor_1 float regular [-∞,+∞] factor_2 float regular [-∞,+∞] factor_3 float regular [-∞,+∞] factor_4 float regular [-∞,+∞]

[0050] It can be seen from the table that the "name" in the metadata corresponds to the field name in the real data...

Embodiment 2

[0058] see Figure 4 , this implementation takes big data regression prediction based on neural network as an example to describe the complete intelligent data modeling process.

[0059] (1) Add "Start", "Complete", and "Data Source" nodes, and generate data source metadata through the metadata collection module as follows:

[0060] name Types of Role scope value float regular [-∞,+∞] factor_1 float regular [-∞,+∞] factor_2 float regular [-∞,+∞] factor_3 float regular [-∞,+∞]

[0061] (2) Add the "Data Selection" node, and select the factor_1~factor_2 fields as the modeling features. The Data Selection node outputs metadata as follows:

[0062] name Types of Role scope value float regular [-∞,+∞] factor_1 float regular [-∞,+∞] factor_2 float regular [-∞,+∞]

[0063] (3) Add a "Normalization" node to perform normalization operations on the factor_1~factor_3 fields. I...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention discloses a big data intelligent modeling system and a method based on dynamic metadata. The system comprises a metadata acquisition module, a metadata algorithm library module, an abnormity detection module, a visualization module and a big data engine module. The method has the beneficial effects that (1) an intelligent modeling strategy based on metadata is provided, the whole data mining conversion process is simulated on the premise that actual data does not need to be loaded and even a big data mining engine does not need to be started, an output result of a current computing node is estimated in real time, modeling operation of an operator is assisted, and the technical threshold of big data modeling is reduced; and (2) an abnormity detection mechanism is further developed based on metadata, real data is replaced by the metadata, and matching detection is performed on data transmission forms and contents between upper and lower nodes, so that the method has the advantages of good real-time performance and high reliability compared with inspection of a big data set, thereby realizing real-time early warning of wrong process connection and improving the modelingefficiency.

Description

technical field [0001] The invention belongs to the technical field of big data processing, and in particular relates to a big data intelligent modeling system and method based on dynamic metadata. Background technique [0002] With the development of information technology and the Internet, all kinds of information have shown explosive growth, covering politics, economy, entertainment, military affairs, culture and other aspects. Huge data information contains rich knowledge and has become an important force to promote the development of various fields. [0003] In order to obtain potential knowledge from massive amounts of information, a variety of systems or related practical tools for big data mining have emerged as the times require, realizing the integration of various technologies for big data from storage, calculation to visualization, etc., reducing the impact of big data to a certain extent. The technical threshold of mining modeling, but the following problems sti...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
IPC IPC(8): G06F16/21G06F16/215G06F16/2458
CPCG06F16/212G06F16/215G06F16/2465
Inventor 王智永王文晋张可新
Owner BEIJING HUARU TECH
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap