An apparatus and method for intelligently analyzing extracted streaming data

Abstract

The present application relates to a kind of intelligent analysis extraction stream data device and method, the device includes: multiple heterogeneous stream data input interface;Intelligent docking DDL engine, for adaptive various upstream data interface;AI intelligent extraction template engine, for cooperating DDL engine initialization default extraction rule;Data timing sample analysis engine;User-defined correction rule engine, for user correction rule for the extraction data quality detection;Big screen intelligent display monitoring platform, for visualized scrolling play and monitor real-time stream data extraction effect.The present application can greatly reduce data governance analyst manual data extraction operation, save data governance analyst cost for large-scale big data enterprise.Can be matched and nested in each business governance system, realize real-time fast governance and extract effective information, complete accurate governance before preoperation, greatly improve the process of data governance intelligentization.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of big data intelligent analysis and management, and in particular to a device and method for intelligent analysis and extraction of stream data. BACKGROUND

[0002] With the rapid development of society and the promotion of digitalization in various industries, big data management and analysis is particularly important. However, it is extremely important for each big data management industry to quickly and efficiently analyze and extract useful data from the vast amount of business data every day.

[0003] In various industries, there is generally a set of or even multiple sets of overall data management and analysis processes. For traditional big data analysis industries, when facing various types of multi-source heterogeneous data, it is usually necessary to rely on a large amount of manual intervention for management and analysis, and the possibility of rework is extremely high. The operation is extremely complex, and if the customer customizes the adjustment at any time, it is necessary to modify the online code more frequently and redeploy it online. The manual cost is extremely high and inefficient, and it is often impossible to respond to customer dynamic needs in a timely manner. SUMMARY

[0004] To solve the above technical problems, the present application provides a device and method for intelligent analysis and extraction of stream data.

[0005] In a first aspect, the present application provides a device for intelligent analysis and extraction of stream data, comprising:

[0006] A multi-element heterogeneous stream data input interface is configured to receive and read real-time stream data.

[0007] An intelligent DDL engine is configured to adapt to a data stream interface and read database table creation statement semantic content.

[0008] An AI intelligent extraction and analysis template engine is configured to intelligently match and analyze the database table creation statement semantic content obtained by the intelligent DDL engine according to the specified extraction rule template, and to output the comparison result to the specified node according to the configuration.

[0009] A data timing sampling analysis engine is configured to extract sample data at regular intervals and analyze and extract the data according to its own rules and verify the data.

[0010] A user-defined correction rule engine is configured to visually dynamically correct the extraction template rule and real-time approve and take effect.

[0011] A large-screen intelligent display monitoring platform is configured to display the data analysis and extraction effect in real time and scroll to display for user customization adjustment and monitoring.

[0012] By adopting the technical scheme, the device for intelligently analyzing and extracting stream data provided by the application can well avoid the cumbersome manual analysis and management work, greatly reduce the development cost and manpower investment, and can realize intelligent online analysis and extraction and dynamically customized data rapid analysis and management extraction demand, and quickly realize zero coding of various different key business functions in the system business and function level to complete the rapid analysis and management extraction function of various different key business functions.

[0013] Preferably, the intelligent docking DDL engine is further configured to initialize data access rules according to table building statements of each source resource, resource English and Chinese names, and field English and Chinese names, and uniformly output the data access rules to the lower engine.

[0014] Preferably, the AI intelligent extraction and analysis template engine is provided with a basic analysis template, a target component type, and an address for comparing the target real-time stream data.

[0015] Preferably, the basic analysis template is provided with a plurality of analysis comparison dimensions, and the plurality of analysis comparison dimensions include a data item type, a data item format, a data item length, a resource English and Chinese name, a resource type, a resource analysis accuracy, an analysis and extraction start and end time, a number of resource threads used, and an extracted data threshold.

[0016] Preferably, the AI intelligent extraction and analysis template engine includes a garbage data filter, a regular matching rule device, and a unified output rule device, which are used to analyze the stream data and the preset extraction template engine and output the results.

[0017] Preferably, the AI intelligent extraction and analysis template engine is further configured to analyze a default analysis and extraction rule expression by calling each customized background AI intelligent extraction and analysis template engine, and the default analysis and extraction rule expression includes IDcard, MobilePhone, Name, Height, Age, and Hobbies.

[0018] Preferably, the user-defined correction rule engine is a user-operable window.

[0019] Preferably, the large-screen intelligent display monitoring platform is a computer display control.

[0020] In a second aspect, the application further provides a method for rapidly and intelligently analyzing and extracting stream data, which is applied to the device for rapidly and intelligently analyzing and extracting stream data as described in the first aspect, and the method comprises the following steps:

[0021] S1: embedding the code of the device for rapidly and intelligently analyzing and extracting stream data which needs to be accessed and analyzed into a data access analysis and management intermediate process of a business system;

[0022] S2: embedding an intelligent docking DDL engine to uniformly extract an upstream field template and output to a lower engine;

[0023] S3: embedding an AI intelligent extraction analysis template engine according to the field description uniformly given by the upper layer, and performing data extraction operation;

[0024] S4: using a user-defined correction rule engine to pass the rules that the user wants to customize to the AI intelligent extraction analysis template engine;

[0025] S5: using an AI intelligent extraction analysis template engine to refresh algorithm rules and modify extraction rules in real time;

[0026] S6: rolling and displaying data analysis extraction results, and correcting at any time until the entire governance process is completed.

[0027] In summary, the present application at least includes the following beneficial technical effects:

[0028] 1. The device and method for quickly and intelligently analyzing and extracting stream data disclosed in the present application can well avoid the tedious manual analysis and governance work, greatly reduce the development cost and manpower investment, and can realize a set of device to complete various big data governance and analysis data intelligent analysis and extraction fields, realize intelligent online analysis and extraction, and dynamically customize the demand for fast data analysis and governance extraction, and quickly realize zero coding in system business and function level to complete the fast analysis and governance extraction function of various different key business functions;

[0029] 2. The present application provides an AI intelligent dynamic data analysis and extraction template device with high intelligence and low coupling, which can greatly reduce the data analysis personnel and even eliminate the need for data analysis personnel, and reduce the development cost; the device of the present application can be seamlessly nested in various business governance analysis and extraction systems, realize real-time fast analysis and extraction, quickly respond to the real-time dynamic changes of customers, and greatly improve the quality of real-time data use;

[0030] 3. The present application can greatly reduce the manual data extraction operation of data governance analysis personnel, greatly save the cost of data governance analysis personnel for large data enterprises. It can be nested in various business governance systems, realize real-time fast governance and extraction of effective information, quickly complete the pre-operation before accurate governance, and greatly improve the process of data governance intelligence. BRIEF DESCRIPTION OF DRAWINGS

[0031] The accompanying drawings are included to provide a further understanding of embodiments and are incorporated in and constitute a part of this specification. The drawings illustrate embodiments and, together with the description, serve to explain principles of the present application. Other embodiments and many of the intended advantages of the present application will be readily appreciated as the same becomes better understood by reference to the following detailed description. The elements of the drawings are not necessarily to scale relative to each other. Like reference numerals designate corresponding similar parts.

[0032] Figure 1 is a schematic diagram of a device for intelligent analysis and extraction of streaming data according to an embodiment of the present application.

[0033] Figure 2 is a schematic diagram of a method for intelligent analysis and extraction of streaming data according to an embodiment of the present application.

[0034] Figure 3 is a schematic diagram of a method for intelligent analysis and extraction of streaming data according to an embodiment of the present application.

[0035] Figure 4 is a schematic diagram of a computer system of an electronic device suitable for implementing embodiments of the present application. DETAILED DESCRIPTION

[0036] The present application will be further described by reference to the following drawings and embodiments. It is to be understood that the specific embodiments described herein are merely illustrative of the application and are not to be taken in a limiting sense. It is further noted that, for the sake of brevity, the figures of the drawing are not necessarily to scale and that some of the features of the present application can be shown in generalized form and not according to precise dimensions.

[0037] It is to be understood that the embodiments and features of the present application can be combined with each other, if not contradictory. The present application will be described in further detail with reference to the drawings and embodiments.

[0038] Figure 1 is a schematic diagram of a device for intelligent analysis and extraction of streaming data according to an embodiment of the present application. Figure 1 The device specifically comprises a multi-heterogeneous streaming data input interface 101, an intelligent docking DDL engine 102, an AI intelligent extraction and analysis template engine 103, a data timing sampling analysis engine 104, a user-defined correction rule engine 105, and a large-screen intelligent display monitoring platform 106.

[0039] In one specific embodiment, the device for intelligent analysis and extraction of streaming data according to the present application is a computer program in the form of an embedded plug-in.

[0040] In one specific embodiment, the multi-heterogeneous streaming data input interface is used to receive and read real-time streaming data.

[0041] In a specific embodiment, the intelligent docking DDL engine is used to adapt the data stream interface and read the database table creation statement semantic content thereof; the intelligent docking DDL engine will initialize the data access rules according to the table creation statements of each source resource upstream, the English and Chinese names of the resources and the English and Chinese names of the fields, and uniformly output the rules to the lower engine.

[0042] In a specific embodiment, the AI intelligent extraction analysis template engine extracts and analyzes the DDL table creation statement content obtained by the intelligent docking DDL engine, and matches and analyzes the specified extraction rule template, so as to perform the first-level data intelligent analysis and extraction.

[0043] In a specific embodiment, the AI intelligent extraction analysis template engine is provided with a basic analysis template having multiple analysis comparison dimensions, so as to adapt to various business systems from various aspects, and finally give the comparison result.

[0044] In a specific embodiment, the AI intelligent extraction analysis template engine includes a garbage data filter, a regular matching rule device, and a unified output rule device, which are used to analyze the flow data and the preset extraction template engine and output the result.

[0045] In a specific embodiment, the analysis dimensions of the AI intelligent extraction template engine include data item remarks, data item length, data item format, data item rule, data item null rate, data item bit number, data item type, data item format, and data item length.

[0046] In a specific embodiment, the analysis dimensions of the AI intelligent extraction template engine further include: resource Chinese and English names, resource type, resource analysis precision, analysis extraction start and end time, number of resource threads used, and extraction data threshold.

[0047] In a specific embodiment, a setting example of a basic unified extraction DDL rule extraction dimension is given, and the analysis extraction dimension includes:

[0048] 1:resourceType:

[0049] Configure the resource type that needs to be compared

[0050] 2:resourceName:

[0051] Configure the resource table name that needs to be compared

[0052] 3:fieldNames:

[0053] Configure the field set that needs to be compared, which can be customized according to business needs (default full field)

[0054] 4: startTime / endTime:

[0055] Configure the start and end time of governance analysis

[0056] 5: threshold:

[0057] Configure the number of resource threads required for parsing, and optimize usage

[0058] 6: EngineType:

[0059] Initialize the intelligent template engine type

[0060] 7: sinkType / sinkAddress:

[0061] Configure the target component type and corresponding address that can be pushed after matching.

[0062] In a specific embodiment, the settings of the comparison dimensions of the basic analysis extraction resource at least include the default analysis template, target component type, and address to realize comparison of target real-time stream data. The remaining parameters are optional and are used for further comparison, screening, and filtering of real-time stream data to perfect the extraction content and rules.

[0063] In a specific embodiment, the default analysis extraction engine, such as PersonTypeEngine, analyzes the default analysis extraction rule expressions, such as IDcard, MobilePhone, Name, Height, Age, Hobbies, and other multi-dimensional extraction rule expressions, by calling each customized background AI intelligent extraction analysis template engine, and combines the upper-layer DDL analysis engine for field-level intelligent matching, which includes a field semantic filter, a garbage data filter, a field unified template, and a field matching regular expression, for real-time stream data comparison analysis extraction, and outputs the comparison results to the specified nodes, such as MessageQueue-01:2181:MessageQueue-02:2181:MessageQueue-03:2181, according to the configuration. The output node is usually a certain log system, and the comparison results are stored in the form of logs. In this computer code example, the specified node is kafka, which is a distributed, partitioned, multi-replica, multi-subscriber, and zookeeper-coordinated distributed log system (also can be used as an MQ system), which can be used for web / nginx logs, access logs, message services, and the like.

[0064] In a specific embodiment, the data timing sampling analysis engine is used to extract sample data at a certain time and analyze and verify it in combination with the data itself rules.

[0065] In one specific embodiment, the data timing sampling analysis engine is a plug-in for assisting intelligent analysis of data.

[0066] In one specific embodiment, the user self-defined correction rule engine is used for intelligent matching priority setting interface according to the output result of the upper DDL engine, and is also used for visualizing dynamic correction of the extraction template rule and real-time approval and validation.

[0067] In one specific embodiment, the user self-defined correction rule engine is a user operable window.

[0068] In one specific embodiment, the large-screen intelligent display monitoring platform can display the data analysis extraction effect in real time and scroll display the user customized adjustment and monitoring.

[0069] In one specific embodiment, the large-screen intelligent display monitoring platform is a computer display control.

[0070] In one specific embodiment, the input interface, the output interface and the access interface in the above-mentioned rapid intelligent analysis and extraction flow data device are parameter calling interfaces between programs.

[0071] The device of the present application provides an AI intelligent dynamic data analysis and extraction template device with high intelligence and low coupling, which can greatly reduce the need for data analysis personnel and reduce development costs. The device of the present application can be seamlessly nested in various business governance analysis and extraction systems, realize real-time rapid analysis and extraction, quickly respond to the real-time dynamic changes of customers, and greatly improve the quality of real-time data use.

[0072] On the other hand, with reference to Figure 2 and Figure 3 The present application also discloses a method for rapid intelligent analysis and extraction of flow data, which is applied to the rapid intelligent analysis and extraction flow data device in the above-mentioned embodiments, and the method comprises the following steps:

[0073] S1: embedding the code of the rapid intelligent analysis and extraction flow data device which needs to be accessed and analyzed into the data access and analysis governance intermediate process of the business system;

[0074] S2: embedding the intelligent interface DDL engine to uniformly extract the template of the upper field and output to the lower engine;

[0075] S3: embedding the AI intelligent extraction and analysis template engine according to the field description uniformly given by the upper layer, and performing data extraction operation;

[0076] S4: using the user self-defined correction rule engine to pass the rules which the user wants to self-correct to the AI intelligent extraction and analysis template engine;

[0077] S5: using AI intelligent extraction analysis template engine timing refresh algorithm rules and real-time modification extraction rules;

[0078] S6: scrolling and showing data analysis extraction results, and correcting at any time until the entire governance process is completed.

[0079] The disclosed device and method for quickly and intelligently analyzing and extracting stream data can well avoid the above-mentioned tedious manual analysis and governance work, greatly reduce development costs and manpower investment, and can practice a device to complete various big data governance and analysis data intelligent analysis and extraction fields, realize intelligent online analysis and extraction, and dynamically customize the demand for fast data analysis and governance extraction, and quickly realize zero coding in the system business and function level to complete the fast analysis and governance extraction function of various different key business functions.

[0080] The present application can realize dynamic analysis and extraction, AI intelligent template recommendation, and real-time push of analysis successful data to any system platform without affecting the original business system in the case of multi-source heterogeneity, solving the problem of single and high coupling data analysis in the past, and quickly realizing the data analysis and extraction of a large number of different table structures. Through the dynamic template, the previous single table structure analysis process is changed into a general analysis process of different table structures, and only the table field needs to be configured to analyze the resource data, avoiding repeated development and saving productivity.

[0081] Reference is made below to Figure 4 which shows a structural schematic diagram of a computer system 200 of an electronic device suitable for implementing the embodiments of the present application. Figure 4 The electronic device shown is only an example and should not bring any limitation to the function and use range of the embodiments of the present application.

[0082] As shown in Figure 4 , the computer system 200 includes a central processing unit (CPU) 201 which can perform various appropriate actions and processes according to programs stored in a read-only memory (ROM) 202 or programs loaded from a storage portion 208 into a random access memory (RAM) 203. In the RAM 203, various programs and data required for the operation of the system 200 are also stored. The CPU 201, the ROM 202, and the RAM 203 are connected to each other through a bus 204. An input / output (I / O) interface 205 is also connected to the bus 204.

[0083] The following components are connected to the I / O interface 205: an input part 206 including a keyboard, a mouse, etc.; an output part 207 including a display such as a liquid crystal display (LCD), a speaker, etc.; a storage part 208 including a hard disk, etc.; and a communication part 209 including a network interface card such as a LAN card, a modem, etc. The communication part 209 performs communication processing via a network such as the Internet. A drive 220 is also connected to the I / O interface 205 as necessary. A removable medium 211 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc. is attached to the drive 220 as necessary, so that a computer program read therefrom is installed in the storage part 208 as necessary.

[0084] In particular, according to embodiments of the present disclosure, the processes described above with reference to the flowcharts can be implemented as a computer software program. For example, embodiments of the present disclosure include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for executing the methods illustrated by the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via the communication part 209, and / or installed from the removable medium 211. When the computer program is executed by the central processing unit (CPU) 201, the above-described functions defined in the methods of the present application are performed.

[0085] As another aspect, the present application also provides a computer-readable storage medium, which can be included in the electronic device described in the above embodiments, or can exist separately without being assembled into the electronic device. The above computer-readable storage medium carries one or more programs, which, when executed by the electronic device, cause the electronic device to perform the methods as shown in Figure 2

[0086] ​It should be noted that the computer-readable storage medium described in the present application can be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may, for example, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or component, or any combination of the above. More specific examples of computer-readable storage media can include, but are not limited to, an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, the computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, device or component. In the present application, the computer-readable signal medium can include a data signal carried in a baseband or as a part of a carrier wave, which carries computer-readable program code. Such a propagated data signal can take many forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. The computer-readable signal medium can also be any computer-readable storage medium other than the computer-readable storage medium, which can send, propagate or transmit a program for use by or in conjunction with an instruction execution system, device or component. The program code contained on the computer-readable storage medium can be transmitted by any suitable medium, including but not limited to wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0087] Computer program code for carrying out operations of the present application can be written in one or more programming languages or combinations of languages including object oriented programming languages such as Java, Smalltalk, C++ or conventional procedural programming languages such as the "C" programming language or similar programming languages. The program code can execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider).

[0088] The computer program product of the present application can be a computer program embodied on a non-transitory computer readable medium. The body of computer readable program code can be any suitable set of instructions stored in a storage medium, which can be run on a system using an operating system. The system can be a mainframe computer, a desktop computer, a laptop computer, or other programmable computer components that can store program code and execute it. The system can also be a mobile device, such as a smart phone, a tablet, or other mobile device that can store program code and execute it. The system can also be a server, a cloud computing system, or other system that can store program code and execute it. The system can also be a combination of the above systems.

[0089] The specific implementations of the present application described above are not intended to limit the scope of the present application, and any skilled person in the art can easily think of changes or replacements within the technical scope disclosed in the present application, which should be covered by the scope of the present application. Therefore, the scope of the present application should be subject to the scope of protection of the claims.

[0090] In the description of the present application, it should be understood that the terms "upper", "lower", "inner", "outer", and the like indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings, and are only for the convenience of describing the present application and simplifying the description, and do not indicate or imply that the device or element referred to must have a particular orientation, be constructed and operated in a particular orientation, and therefore should not be understood as limiting the present application. The word 'comprising' does not exclude the existence of elements or steps not listed in the claims. The word 'a' or 'an' in front of an element does not exclude the existence of multiple such elements. The mere fact that certain measures are recited in mutually different dependent claims does not indicate that the combination of these measures cannot be used to improve. Any reference signs in the claims should not be interpreted as limiting the scope.

Claims

1. An apparatus for intelligently analyzing extracted streaming data, the apparatus comprising: Comprise: A multi-element heterogeneous flow data input interface for receiving and reading real-time flow data; An intelligent docking DDL engine for adapting the data flow interface and reading the database table creation statement semantic content; According to the table creation statement of each source resource upstream, the resource English name and the field English name are initialized to extract data access rules and uniformly output to the lower engine; An AI intelligent extraction analysis template engine for intelligent matching analysis according to the database table creation statement semantic content obtained by the intelligent docking DDL engine and corresponding specified extraction rule template, and asynchronously outputting the analysis comparison result to the specified node according to the configuration; including a garbage data filter, a regular matching rule, and a unified output rule, for analyzing the flow data and the preset extraction template engine and outputting the result; For calling each customized background AI intelligent extraction analysis template engine to analyze the default analysis extraction rule expression, the default analysis extraction rule expression includes IDcard, MobilePhone, Name, Height, Age, and Hobbies; A data timing sampling analysis engine for periodically extracting sample data and analyzing and extracting according to the data itself rules and verifying; A user-defined correction rule engine for visualizing dynamic correction of extraction template rules and real-time approval and taking effect; A large-screen intelligent display monitoring platform for real-time display of data analysis extraction effect and rolling display for user customization adjustment and monitoring. 2.The device of claim 1, wherein: The AI intelligent extraction analysis template engine is provided with a basic analysis template for comparing target real-time flow data, a target component type and an address. 3.The device of claim 2, wherein: The basic analysis template is provided with a plurality of analysis comparison dimensions, including data item type, data item format, data item length, data item note, data item rule, data item null rate, data item bit number, resource English name, resource type, resource analysis precision, analysis extraction start and end time, resource thread number, and extraction data threshold.

4. The apparatus for intelligently analyzing and extracting streaming data of claim 1, wherein: The user-defined correction rule engine is a user-operable window.

5. The apparatus for intelligently analyzing and extracting streaming data of claim 1, wherein: The large-screen intelligent display monitoring platform is a computer display control.

6. The apparatus for intelligently analyzing and extracting streaming data of claim 1, wherein: The specified node is a message queue service stored in a distributed log system in the form of a log.

7. A method of intelligently analyzing extracted streaming data, the method comprising: The method applied to the intelligent analysis extraction flow data device of any one of claims 1-6, the method comprises: S1: embedding the code of the intelligent analysis extraction flow data device that needs to access analysis into the data access analysis governance intermediate process of the business system; S2: embedding the intelligent docking DDL engine to uniformly extract the field template and output to the lower engine; S3: embedding the AI intelligent extraction analysis template engine according to the field description uniformly given by the upper layer to perform data extraction operation; S4: using the user-defined correction rule engine to pass the rule that the user wants to customize to the AI intelligent extraction analysis template engine; S5: using the AI intelligent extraction analysis template engine to refresh the algorithm rule and modify the extraction rule in real time; S6: rolling and playing and showing the data analysis extraction effect, and correcting at any time until the whole governance process is completed.

Images