A data analysis method, device and system

Abstract

The present invention discloses a data analysis method, device and system, and relates to the field of computer technology. A specific implementation of the method includes: obtaining a business logic script pre-written for a data analysis task; wherein the business logic script includes data source parameter information and business logic code; executing the data extraction logic of the business logic code to obtain business data from the data source specified by the data source parameter information; executing the data processing logic of the business logic code to call a calculator set set in the data processing logic to process the business data. The method obtains business data from a specified data source by obtaining a business logic script containing business logic code, and then calls a calculator set to process the business data, so that the system code can be reused and the development efficiency is high.

Description

Technical Field

[0001] The present invention relates to the field of computers, and in particular to a data analysis method, device and system. Background Art

[0002] With the rapid development of information technologies such as mobile Internet, cloud computing, and the Internet of Things, more and more data is being produced, and the entire information industry has accelerated into the "big data" era. At the same time, the value of data has attracted the attention of more and more companies and has become the wealth of more and more companies. In general business scenarios, after data is generated from the OLTP (On-Line Transaction Processing) system, the OLAP (Online Analytical Processing) system further analyzes the data and mines its value.

[0003] The principle of data analysis in OLAP system is: read data of various structures from different data sources (Extract), then process, process and summarize the data (Transform), store the processing results in different databases (Load), and provide query interface. In the above process, OLAP system needs to customize and develop various functional modules such as data source reading, data processing and result storage for different analysis and processing tasks, and it uses regular iteration to process incremental data.

[0004] In the process of implementing the present invention, the inventors found that there are at least the following problems in the prior art:

[0005] The customized development approach results in the system code being non-reusable and there is repeated development, low development efficiency, and waste of human resources. The regular iteration approach to processing incremental data has poor flexibility and high latency, and cannot meet business scenarios that require real-time performance. Summary of the invention

[0006] In view of this, an embodiment of the present invention provides a data analysis method, device and system, which obtains business data from a specified data source by obtaining a business logic script containing business logic code, and then calls a calculator set to process the business data, so that the system code can be reused and the development efficiency is high.

[0007] To achieve the above objective, according to one aspect of an embodiment of the present invention, a data analysis method is provided.

[0008] A data analysis method according to an embodiment of the present invention comprises: obtaining a business logic script pre-written for a data analysis task; wherein the business logic script comprises data source parameter information and business logic code; executing the data extraction logic of the business logic code to obtain business data from a data source specified by the data source parameter information; and executing the data processing logic of the business logic code to call a calculator set set in the data processing logic to perform data processing on the business data.

[0009] Optionally, the data source parameter information includes a data source parameter tuple, which includes basic data source information and data reading instructions; obtaining business data from the data source specified by the data source parameter information includes: establishing a connection with a corresponding data source according to the basic data source information; and executing the data reading instructions to obtain business data from the data source.

[0010] Optionally, the calculator set includes a calculator with a single responsibility for data processing, which takes the business data or the processing result of the business data as an input parameter, processes the input parameter, and returns the processing result of the input parameter.

[0011] Optionally, the data analysis task is a clustering task of item question and answer data, the business data is the item question and answer data, and the calculator set includes a word segmentation calculator, a similarity calculator and a clustering calculator; calling the calculator set set in the data processing logic to perform data processing on the business data, including: calling the word segmentation calculator to perform word segmentation on the item question and answer data to obtain a word segmentation result; calling the similarity calculator to perform similarity calculation on the word segmentation records of the same item in the word segmentation result to obtain a similarity calculation result; calling the clustering calculator to cluster the item question and answer data based on the similarity calculation result.

[0012] Optionally, the data source parameter information includes a parameter identifier; obtaining business data from a data source specified by the data source parameter information includes: obtaining a corresponding data source parameter tuple according to the parameter identifier; wherein the data source parameter tuple includes basic data source information and a data reading instruction; establishing a connection with a corresponding data source according to the basic data source information; and executing the data reading instruction to obtain business data from the data source.

[0013] Optionally, the method further includes: receiving data source configuration information, and generating a corresponding parameter identifier for a data source parameter tuple of the data source configuration information; wherein the data source configuration information includes a data source type and a data source parameter tuple corresponding to the data source type.

[0014] Optionally, the business data includes stock data and / or incremental data, and the incremental data is stored in the data source component by subscribing to a message queue of the business system.

[0015] To achieve the above objective, according to another aspect of an embodiment of the present invention, a data analysis device is provided.

[0016] A data analysis device according to an embodiment of the present invention comprises: a script acquisition module, used to acquire a business logic script pre-written for a data analysis task; wherein the business logic script comprises data source parameter information and business logic code; a data acquisition module, used to execute the data extraction logic of the business logic code to acquire business data from a data source specified by the data source parameter information; and a data processing module, used to execute the data processing logic of the business logic code to call a calculator set set in the data processing logic to perform data processing on the business data.

[0017] Optionally, the data source parameter information includes a data source parameter tuple, and the data source parameter tuple includes basic data source information and data reading instructions; the data acquisition module is also used to: establish a connection with a corresponding data source according to the basic data source information; and execute the data reading instructions to obtain business data from the data source.

[0018] Optionally, the calculator set includes a calculator with a single responsibility for data processing, which takes the business data or the processing result of the business data as an input parameter, processes the input parameter, and returns the processing result of the input parameter.

[0019] Optionally, the data analysis task is a clustering task of item question and answer data, the business data is the item question and answer data, and the calculator set includes a word segmentation calculator, a similarity calculator and a clustering calculator; the data processing module is further used to: call the word segmentation calculator to perform word segmentation on the item question and answer data to obtain a word segmentation result; call the similarity calculator to perform similarity calculation on the word segmentation records of the same item in the word segmentation result to obtain a similarity calculation result; call the clustering calculator to cluster the item question and answer data based on the similarity calculation result.

[0020] Optionally, the data source parameter information includes a parameter identifier; the data acquisition module is further used to: obtain a corresponding data source parameter tuple according to the parameter identifier; wherein the data source parameter tuple includes basic data source information and data reading instructions; establish a connection with a corresponding data source according to the basic data source information; and execute the data reading instructions to obtain business data from the data source.

[0021] Optionally, the device also includes: a receiving and generating module, used to receive data source configuration information, and generate a corresponding parameter identifier for a data source parameter tuple of the data source configuration information; wherein the data source configuration information includes a data source type and a data source parameter tuple corresponding to the data source type.

[0022] Optionally, the business data includes stock data and / or incremental data, and the incremental data is stored in the data source component by subscribing to a message queue of the business system.

[0023] To achieve the above objective, according to another aspect of an embodiment of the present invention, a data analysis system is provided.

[0024] A data analysis system according to an embodiment of the present invention comprises: a management background, which is used to send a business logic script written for a data analysis task to an assembler component; wherein the business logic script comprises data source parameter information and business logic code; the assembler component is used to execute the data extraction logic of the business logic code to send a data acquisition request to the data source component; and is also used to execute the data processing logic of the business logic code to send a calculator call request to the calculator component; the data source component is used to receive the data acquisition request, obtain business data from the data source specified in the data acquisition request, and send the business data to the assembler component; and the calculator component is used to receive the calculator call request, call the calculator set specified in the calculator call request, and perform data processing on the business data.

[0025] Optionally, the calculator component is also used to: send the processing results to the assembler component, so that the assembler component stores the processing results in the data source component; the system also includes: a retrieval system, used to retrieve the processing results from the data source component according to data query conditions.

[0026] To achieve the above objective, according to another aspect of an embodiment of the present invention, an electronic device is provided.

[0027] An electronic device according to an embodiment of the present invention includes: one or more processors; a storage device for storing one or more programs, and when the one or more programs are executed by the one or more processors, the one or more processors implement a data analysis method according to an embodiment of the present invention.

[0028] To achieve the above objective, according to another aspect of the embodiments of the present invention, a computer readable medium is provided.

[0029] A computer-readable medium according to an embodiment of the present invention stores a computer program, and when the program is executed by a processor, a data analysis method according to an embodiment of the present invention is implemented.

[0030] An embodiment of the above invention has the following advantages or beneficial effects: by obtaining a business logic script containing business logic code, business data is obtained from a specified data source, and then a calculator set is called to process the business data, so that the system code can be reused and the development efficiency is high; the parameters required for obtaining data are pre-sorted out, and then the data extraction logic is implemented, and the reuse of the data extraction logic is realized; a calculator with a single responsibility for data processing is pre-built, and then the data processing logic is implemented, and the reuse of the data processing logic is realized; the clustering task of the item question and answer data is realized by the pre-built word segmentation calculator, similarity calculator and clustering calculator, and the development efficiency is high and human resources are saved; a parameter identifier is generated for the data source parameter tuple, so that the acquisition of business data can be realized by specifying the parameter identifier later; by subscribing to the message queue of the business system, real-time processing of incremental data is realized; by abstracting the assembler component, the data source component and the calculator component to realize system reuse, and by submitting the business logic script in the management background, zero development on the platform side is realized; the data user queries the data through the retrieval system without directly accessing the database, making the system more user-friendly and easy to use for the data user.

[0031] The further effects of the above-mentioned non-conventional optional manner will be described below in conjunction with the specific implementation manner. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] The accompanying drawings are used to better understand the present invention and do not constitute an improper limitation of the present invention.

[0033] Figure 1 It is a schematic diagram of the implementation principle of data analysis by the OLAP system of the prior art;

[0034] Figure 2 It is a schematic diagram of the implementation process of data analysis by the OLAP system of the prior art;

[0035] Figure 3 is a schematic diagram of the main steps of the data analysis method according to an embodiment of the present invention;

[0036] Figure 4 is a schematic diagram of the system structure of a data analysis method according to an embodiment of the present invention;

[0037] Figure 5 2 is a schematic diagram of the implementation principle of the data analysis method of an embodiment of the present invention for clustering item question and answer data;

[0038] Figure 6is a schematic diagram of the main process of the data analysis method according to an embodiment of the present invention;

[0039] Figure 7 A timing diagram of configuring a data source according to a data analysis method according to an embodiment of the present invention;

[0040] Figure 8 A timing diagram of processing stock data by a data analysis method according to an embodiment of the present invention;

[0041] Fig. 9 This is a diagram showing the implementation architecture of the data analysis method according to an embodiment of the present invention for an OLAP system;

[0042] Fig.10 is a schematic diagram of main modules of a data analysis device according to an embodiment of the present invention;

[0043] Fig.11 is an exemplary system architecture diagram to which embodiments of the present invention may be applied;

[0044] Fig.12 It is a schematic diagram of the structure of a computer device suitable for implementing the electronic device of the embodiment of the present invention. DETAILED DESCRIPTION

[0045] The following is a description of exemplary embodiments of the present invention in conjunction with the accompanying drawings, including various details of the embodiments of the present invention to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present invention. Similarly, for clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0046] Figure 1 This is a schematic diagram of the implementation principle of data analysis in the OLAP system of the prior art. Figure 1 As shown, in an enterprise, different business systems generate data structures of various kinds, such as item data, user data, and user-generated content (UGC), which are stored in different databases.

[0047] OLAP system reads data of various structures from different data sources (Extract), and then processes, processes, and summarizes (Transform) the data, and stores the processing results in different databases (Load). The data stored in the database can then be used for data mining (DATA MINING), data visualization (DATAVISUALIZATION), report generation (REPORTS), dashboard generation (DASHBOARDS), alarms (ALERTS), etc.

[0048] Figure 2 This is a schematic diagram of the implementation process of data analysis by the OLAP system of the prior art. Figure 2 As shown, the OLAP system reads data from databases, cloud services, application systems, and AIRFLOW, performs data quality detection, data conversion, and data cleaning on data from different data sources, and then stores the processing results in message queues, data warehouses, databases, etc. Figure 2 In the example, the application system can be Android or Apple. AIRFLOW is a workload scheduler. KAFKA is a high-throughput distributed publish-subscribe messaging system. Data backup can be achieved through S3 (Simple Storage Service).

[0049] In the above process, script reading, data quality detection, data conversion & data cleaning, routing & schema detection (ROUTER & SCHEMA DETECTER), script writing and other links all need to customize the development scripts according to the specific analysis tasks. For incremental data, a regular iteration method is adopted to restart the analysis task for processing. This "customization + incremental iteration" processing method, on the one hand, the system code cannot be reused, and repeated development leads to inefficient delivery and waste of manpower; on the other hand, the regular iteration method for processing incremental data has poor flexibility and high latency, and cannot meet business scenarios that require real-time performance.

[0050] In order to solve the above problems in the prior art, the embodiments of the present invention optimize the existing OLAP system design scheme, and maximize the reuse of the OLAP system and the real-time processing of incremental data by abstracting three components with single responsibilities and the management background.

[0051] Figure 3 FIG. 1 is a schematic diagram of the main steps of the data analysis method according to an embodiment of the present invention. Figure 3 As shown, the data analysis method of the embodiment of the present invention mainly includes the following steps:

[0052] Step S301: Obtain a business logic script pre-written for a data analysis task. The business logic script includes: data source parameter information and business logic code. The data source parameter information may be a data source parameter tuple or a parameter identifier corresponding to the data source parameter tuple. The business logic code includes: data extraction logic and data processing logic. The data extraction logic is used to obtain business data from the corresponding data source according to the data source parameter information. The data processing logic is used to set the calculator set required for the data analysis task to call the calculator set to process the business data.

[0053] Step S302: Execute the data extraction logic of the business logic code to obtain business data from the data source specified by the data source parameter information. If the data source parameter information is a data source parameter tuple, which includes basic data source information and data reading instructions, then establish a connection with the corresponding data source according to the basic data source information; then execute the data reading instructions to obtain business data from the data source. If the data source parameter information is a parameter identifier, first obtain the corresponding data source parameter tuple according to the parameter identifier; then obtain the business data in the above manner.

[0054] Step S303: Execute the data processing logic of the business logic code to call the calculator set set in the data processing logic to process the business data. The calculator set includes a calculator with a single responsibility for data processing, which takes the business data or the processing result of the business data as an input parameter, processes the input parameter, and returns the processing result of the input parameter. Execute the data processing logic to call the calculator in the calculator set, and then perform data processing such as data cleaning, data conversion, and data verification on the business data.

[0055] Figure 4 Schematic diagram of the system structure of the data analysis method according to an embodiment of the present invention. Figure 4 As shown, the data analysis system of the embodiment of the present invention mainly includes: a data source component (DataSource), a calculator component (Calculator), an assembler component (Assembler) and an administration backend (Admin). Among them, DataSource and Calculator are reusable components. Figure 4 BDP is a big data storage and query platform.

[0056] DataSource: Multiple data analysis tasks can be reused, encapsulate access to the database, and provide a standard interface for Assembler to obtain business data. Business data of different business systems are stored in various databases, such as mysql, redis, hbase, elasticsearch, mongodb, MQ, etc. Although there are various types of databases, the access method is fixed for each specific database. Therefore, the data source parameter tuple can be defined in advance according to the data source type to realize business data reading. For example, to access mysql, specify the data source parameter tuple (ip, user, password, database, sql) to read business data. For another example, to access redis, specify the data source parameter tuple (ip, port, user, password, key, command) to read business data.

[0057] In the embodiment, the business data includes stock data and / or incremental data. The stock data and incremental data here are divided by reference time, the data generated before the reference time is stock data, and the data generated after the reference time is incremental data. For example, the launch time of the OLAP system is T. Before T, the business system (such as the product evaluation system) has generated a lot of data. These existing data are stock data. The new data generated after T is incremental data.

[0058] In the embodiment, DataSource provides two ways to pass the data source parameter tuple. One is to provide it as a method parameter when Assembler calls DataSource; the other is to enter it in advance in the Admin background and specify the parameter identifier when Assembler calls it. In addition, DataSource subscribes to the message queue (MQ) of the business system, so that the business system can send incremental data to the OLAP system in real time through MQ, realizing real-time processing of incremental data.

[0059] Calculator: Multiple data analysis tasks can be reused, corresponding to the data cleaning and processing functions of the existing OLAP system, for Assembler to call, completely independent of the business. For a new data analysis task, if the existing Calculator cannot meet the needs, it is necessary to develop a reusable Calculator that is not coupled with the business in advance according to the task requirements. Each Calculator instance is a fully reusable data processing plant with a single responsibility, such as word segmentation, sentiment analysis, face recognition, ASR (Automatic Speech Recognition), TTS (Text To Speech), etc.

[0060] The Calculator method accepts data as input parameters (not related to the business) and returns the result after processing. For example, the input parameter of the word segmentation calculator is text, and the output parameter is a term. Another example is the input parameter of the ASR calculator is audio, and the output is a text string. The word segmentation calculator and the ASR calculator can be used in multiple data analysis tasks, such as the word segmentation calculator can be used in the evaluation text processing task and the question-and-answer task. Assembler calls multiple Calculators to complete data processing.

[0061] Assembler: It is the scheduling center of the entire data analysis system. It reads business data through DataSource, then calls Calculator to clean and process the data, and finally calls DataSource to store the result data. Assembler can be understood as the business logic layer of data analysis tasks. Therefore, for each task, it is necessary to customize and develop its own Assembler processing logic.

[0062] For a new data analysis task, Assembler is the only part that needs to be developed (if the existing Calculator does not meet the requirements, a new Calculator needs to be developed). For a data analysis task, there are two ways to implement its business logic: one is to customize the business logic script of the task in the Assembler project. The other is to submit the Assembler script of the task in the Admin background.

[0063] Figure 5 FIG. 2 is a schematic diagram showing the implementation principle of the data analysis method of an embodiment of the present invention for clustering item question and answer data. Figure 5 As shown in the figure, Assembler calls DataSource to read the item question and answer data of the question and answer system, and then combines the word segmentation, similarity calculation, and clustering calculators to complete the clustering task. The specific implementation is as follows:

[0064] (1) Word segmentation: Assembler calls Datasource to obtain all item question and answer data from the data mart (such as the bdp database), then calls the word segmentation calculator to segment the text, and saves the entry records to the hbase database through Datasource. The record structure is: {"sku", "original text", "entry result"}.

[0065] (2) Similarity calculation: Assembler calls Datasource to obtain term data from hbase, and then calls Similarity Calculator to pair all term records under each item (sku) to calculate similarity scores. Each pair of terms obtains a similarity result (such as a score); finally, Datasource is called to save the similarity results to the ES database.

[0066] (3) Clustering: Assembler calls Datasource to read the similarity results, and calls Cluster Calculator to cluster the question and answer data under each sku. For example, sku=123 is a washing machine, and the following questions are ["Is this washing machine noisy?", "Does it wash cleanly?", "Is it noisy?", "How is the washing effect?"], and id is assumed to be 1, 2, 3, 4, and the clustering results are [{1,3},{2,4}].

[0067] In a preferred embodiment, before using the word segmentation calculator to process the item question and answer data, the embedding calculator can also be used to process the item question and answer data. The specific implementation is: Assembler obtains all item question and answer data from the data mart through Datasource, calls the embedding calculator for model training, generates the embedding model and background data, and calls Datasource to save it to the text database.

[0068] Admin: It is the key to achieve zero development on the data analysis system platform side. As a visual interface for users to interact with the data analysis system, Admin background provides rich functions. Users can submit data source parameter tuples, view currently supported database types, view existing calculators, submit task Assembler program scripts, manage task start / stop, view task status (such as progress), etc. through Admin.

[0069] In a preferred embodiment of the present invention, the data analysis system of the embodiment of the present invention also includes: a retrieval system (Search). The system provides a query entry for result data and relies on DataSource. The Search system includes two modules: SOA (Service-Oriented Architecture) and Admin. SOA is a server-side application that encapsulates a query interface, which exposes the interface to the outside through http and other methods, and is mainly used by R&D personnel of the user. Admin serves as a data query background interface and provides simple query conditions. Data users can use it to retrieve data processing results of the data analysis system. All data users query data through Search instead of directly accessing the database. Search can not only play the role of a database, but also make the system more user-friendly and easy to use.

[0070] Figure 6 Schematic diagram of the main process of the data analysis method according to an embodiment of the present invention. Figure 6 As shown, the data analysis method of the embodiment of the present invention mainly includes the following steps:

[0071] Step S601: The management backend receives the data source parameter information submitted by the user and configures the data source component. When the user configures the data source, he needs to specify the database type and provide the data source parameter tuple, and then submit and save it to the management backend. The management backend generates a parameter identifier (Id) for the data source parameter tuple, and the user can view the parameter identifier and data source parameter tuple in the management backend.

[0072] Step S602: The management backend receives the business logic script submitted by the user for the data analysis task. The business logic script can be customized and developed in the Assembler project, or submitted by the management backend.

[0073] Step S603: The management background starts the data analysis task.

[0074] Step S604: the assembler component obtains business data from the data source specified by the data source parameter information, establishes a connection with the corresponding data source according to the data source basic information, and then executes the data read instruction to obtain business data from the data source.

[0075] Step S605: The assembler component calls the calculator set set in the data processing logic to process the business data, and calls the calculators in the calculator set to perform data cleaning, data conversion, data verification and other data processing on the business data.

[0076] Step S606: The assembler component calls the data source component to save the data processing result. The assembler component stores the data processing result in the data source component.

[0077] Figure 7 FIG. 1 is a timing diagram of configuring a data source according to a data analysis method according to an embodiment of the present invention. Figure 7 As shown, after the user configures the data source, it is sent to the management backend. After the management backend generates the data source instance, it is sent to the data source component. The data source component verifies whether the parameters provided by the configured data source can connect to the database normally and whether the data can be queried normally. The dotted line in the timing diagram indicates that the process processing ends and returns. The following example illustrates:

[0078] The data source parameter tuple is as follows:

[0079] {"type":"mysql","ip":"192.168.100.49","port":3358,"user":"root","password":"pw123","database":"db_comment","sql":"select content from user_comment where created between'2019-10-01'and'2019-10-02'"}

[0080] After submitting and saving, you get:

[0081] {"dataSourceId":1024,"type":"mysql","ip":"192.168.100.49","port":3358,"user":"root","password":"pw123","database":"db_comment","sq l":"selectcontent from user_comment where created between'2019-10-01'and'2019-10-02'"}

[0082] The Assembler specifies the data source parameter tuple with Id = 1024, which means that MySQL is called using the above parameters.

[0083] In addition, for the processing of incremental data MQ: if a data analysis task needs to subscribe to the incremental data MQ of the business system, the user needs to additionally specify the MQ data source parameter tuple (ip, user, topic, etc.) when configuring the data source in the management background. After DataSource receives the incremental data, it calls back Assembler to process the incremental data according to the business script logic.

[0084] Figure 8 FIG. 1 is a timing diagram of processing stock data according to a data analysis method according to an embodiment of the present invention. Figure 8 As shown in the figure, after the user configures the stock data task, it is sent to the background management; after the background management verifies the calculator, it is sent to the assembler component; the assembler component verifies the data source instance. After the verification passes, the management background starts the stock data task. The assembler component pulls data from the data source component, calls the calculator component to process the data, saves the processing results, and returns the task status.

[0085] Among them, the task of configuring stock data corresponds to step S601 and step S602. The specific implementation is: users can view their saved data source parameter tuples and parameter identifiers, currently supported calculators and input and output parameter descriptions in the management background. Then, users write their own Assembler program scripts. The scripts are the core of the data analysis task, responsible for combining Datasource and Calculator for use, and adding the business processing logic of the task.

[0086] The Assembler script contains information such as parameter identifier Id (or directly provide data source parameter tuple), task logic code, etc. Submit and save, then you can start the task. If the Assembler script specifies the parameter identifier Id, the data source parameter tuple will be found from the saved records when the task is running, and then the data source will be read. In addition, the verification calculator inputs the input parameters and observes the output parameters.

[0087] Fig. 9 This is a diagram showing the implementation architecture of the data analysis method for an OLAP system according to an embodiment of the present invention. Fig. 9 As shown in the figure, the OLAP system includes OLAP-Datasource, OLAP-Assembler, OLAP-Calculator and OLAP-Admin. OLAP-Datasource obtains the stock quantity and data processing results from the database. OLAP-Calculator develops a new calculator. OLAP-Admin starts the data analysis task and configures the data source. OLAP-Assembler develops the task logic script, pulls the stock data or incremental data from OLAP-Datasource, and calls the calculator in OLAP-Calculator to process the pulled data.

[0088] In addition, the Search system includes Search-SOA and Search-Admin. Search-SOA receives data query requests from Client and queries data from OLAP-Datasource.

[0089] It can be seen from the data analysis method of the embodiment of the present invention that by obtaining the business logic script containing the business logic code, the business data is obtained from the specified data source, and then the calculator set is called to realize the processing of the business data, so that the system code can be reused, and the development efficiency is high; the parameters required for obtaining the data are sorted out in advance, and then the data extraction logic is realized, and the data extraction logic is reused; a calculator with a single responsibility for data processing is pre-built, and then the data processing logic is realized, and the data processing logic is reused; through the pre-built word segmentation calculator, similarity calculator and clustering calculator, the clustering task of the item question and answer data is realized, the development efficiency is high, and human resources are saved.

[0090] Generate parameter identifiers for data source parameter tuples, so that business data can be acquired later by simply specifying the parameter identifiers; real-time processing of incremental data is achieved by subscribing to the message queue of the business system; system reuse is achieved by abstracting assembler components, data source components, and calculator components, and zero development on the platform side is achieved by submitting business logic scripts in the management background; data users query data through the retrieval system without directly accessing the database, making the system more user-friendly and easy to use for data users.

[0091] Fig.10 Schematic diagram of the main modules of the data analysis device according to an embodiment of the present invention. Fig.10 As shown, the data analysis device 1000 of the embodiment of the present invention mainly includes:

[0092] The script acquisition module 1001 is used to acquire the business logic script pre-written for the data analysis task. The business logic script includes data source parameter information and business logic code. The data source parameter information can be a data source parameter tuple or a parameter identifier corresponding to the data source parameter tuple. The business logic code includes: data extraction logic and data processing logic. The data extraction logic is used to obtain business data from the corresponding data source according to the data source parameter information. The data processing logic is used to set the calculator set required for the data analysis task to call the calculator set to process the business data.

[0093] The data acquisition module 1002 is used to execute the data extraction logic of the business logic code to obtain business data from the data source specified by the data source parameter information. If the data source parameter information is a data source parameter tuple, which includes basic data source information and data reading instructions, a connection is established with the corresponding data source according to the basic data source information; then the data reading instructions are executed to obtain business data from the data source. If the data source parameter information is a parameter identifier, firstly the corresponding data source parameter tuple is obtained according to the parameter identifier; then the business data is obtained in the above manner.

[0094] The data processing module 1003 is used to execute the data processing logic of the business logic code to call the calculator set set in the data processing logic to process the business data. The calculator set includes a calculator with a single responsibility for data processing, which takes the business data or the processing result of the business data as an input parameter, processes the input parameter, and returns the processing result of the input parameter. The data processing logic is executed to call the calculator in the calculator set, and then perform data processing such as data cleaning, data conversion, and data verification on the business data.

[0095] In addition, the data analysis device 1000 of the embodiment of the present invention may further include: a receiving generation module ( Fig.10 The module is used to receive data source configuration information and generate corresponding parameter identifiers for the data source parameter tuples of the data source configuration information; wherein the data source configuration information includes a data source type and a data source parameter tuple corresponding to the data source type.

[0096] From the above description, it can be seen that by receiving the business logic script containing the business logic code, obtaining the business data from the specified data source, and then calling the calculator set to process the business data, the system code can be reused and the development efficiency is high.

[0097] Fig.11 An exemplary system architecture 1100 is shown to which the data analysis method or data analysis device according to the embodiment of the present invention can be applied.

[0098] like Fig.11 As shown, system architecture 1100 may include terminal devices 1101, 1102, 1103, a network 1104, and a server 1105. Network 1104 is used to provide a medium for communication links between terminal devices 1101, 1102, 1103 and server 1105. Network 1104 may include various connection types, such as wired, wireless communication links, or optical fiber cables, etc.

[0099] Users can use terminal devices 1101, 1102, 1103 to interact with server 1105 through network 1104 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 1101, 1102, 1103, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social platform software, etc.

[0100] The terminal devices 1101 , 1102 , and 1103 may be various electronic devices having a display screen and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, and desktop computers, etc.

[0101] The server 1105 may be a server that provides various services, such as a background management server that processes the business logic scripts sent by the administrator using the terminal devices 1101, 1102, and 1103. The background management server may execute the business logic scripts and feed back the processing results (such as data processing results) to the terminal devices.

[0102] It should be noted that the data analysis method provided in the embodiment of the present application is generally executed by the server 1105 , and accordingly, the data analysis device is generally set in the server 1105 .

[0103] It should be understood that Fig.11 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0104] According to an embodiment of the present invention, the present invention also provides an electronic device and a computer-readable medium.

[0105] The electronic device of the present invention comprises: one or more processors; a storage device for storing one or more programs, and when the one or more programs are executed by the one or more processors, the one or more processors implement a data analysis method of an embodiment of the present invention.

[0106] The computer-readable medium of the present invention stores a computer program thereon, and when the program is executed by a processor, a data analysis method of an embodiment of the present invention is implemented.

[0107] Reference below Fig.12 , which shows a schematic diagram of the structure of a computer system 1200 suitable for implementing an electronic device of an embodiment of the present invention. Fig.12 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0108] like Fig.12 As shown, the computer system 1200 includes a central processing unit (CPU) 1201, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1202 or a program loaded from a storage part 1208 into a random access memory (RAM) 1203. In the RAM 1203, various programs and data required for the operation of the computer system 1200 are also stored. The CPU 1201, the ROM 1202, and the RAM 1203 are connected to each other via a bus 1204. An input / output (I / O) interface 1205 is also connected to the bus 1204.

[0109] The following components are connected to the I / O interface 1205: an input section 1206 including a keyboard, a mouse, etc.; an output section 1207 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 1208 including a hard disk, etc.; and a communication section 1209 including a network interface card such as a LAN card, a modem, etc. The communication section 1209 performs communication processing via a network such as the Internet. A drive 1210 is also connected to the I / O interface 1205 as needed. A removable medium 1211, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 1210 as needed, so that a computer program read therefrom is installed into the storage section 1208 as needed.

[0110] In particular, according to the embodiments disclosed in the present invention, the process described in the main step diagram above can be implemented as a computer software program. For example, the embodiments disclosed in the present invention include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the main step diagram. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 1209, and / or installed from the removable medium 1211. When the computer program is executed by the central processing unit (CPU) 1201, the above-mentioned functions defined in the system of the present invention are executed.

[0111] It should be noted that the computer-readable medium shown in the present invention may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present invention, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In the present invention, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0112] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present invention. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0113] The modules involved in the embodiments of the present invention may be implemented in software or hardware. The modules described may also be set in a processor, for example, it may be described as: a processor includes a script acquisition module, a data acquisition module and a data processing module. The names of these modules do not constitute a limitation on the modules themselves in some cases, for example, the script acquisition module may also be described as "a module for acquiring business logic scripts pre-written for data analysis tasks".

[0114] As another aspect, the present invention also provides a computer-readable medium, which may be included in the device described in the above embodiment; or may exist independently without being assembled into the device. The above computer-readable medium carries one or more programs, and when the above one or more programs are executed by a device, the device includes: obtaining a business logic script pre-written for a data analysis task; wherein the business logic script includes data source parameter information and business logic code; executing the data extraction logic of the business logic code to obtain business data from the data source specified by the data source parameter information; executing the data processing logic of the business logic code to call the calculator set set in the data processing logic to process the business data. .

[0115] From the above description, it can be seen that by obtaining the business logic script containing the business logic code, obtaining the business data from the specified data source, and then calling the calculator set to process the business data, the system code can be reused and the development efficiency is high; the parameters required to obtain data are sorted out in advance, and then the data extraction logic is implemented, thereby realizing the reuse of the data extraction logic.

[0116] The above product can execute the method provided by the embodiment of the present invention, and has the functional modules and beneficial effects corresponding to the execution method. For technical details not described in detail in this embodiment, please refer to the method provided by the embodiment of the present invention.

[0117] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions may occur depending on design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A data analysis method, characterized in that: include: Receive data source configuration information, and generate corresponding parameter identifiers for data source parameter tuples of the data source configuration information; Obtaining a business logic script pre-written for a data analysis task; wherein the business logic script includes data source parameter information and business logic code, and the data source parameter information is a data source parameter tuple or a parameter identifier; Execute the data extraction logic of the business logic code to obtain business data from the data source specified by the data source parameter information; The data processing logic of the business logic code is executed to call the calculator set set in the data processing logic to perform data processing on the business data.

2. The method according to claim 1, characterized in that The data source parameter information includes a data source parameter tuple, and the data source parameter tuple includes basic data source information and a data reading instruction; Acquiring business data from a data source specified by the data source parameter information includes: Establishing a connection with a corresponding data source according to the basic information of the data source; The data reading instruction is executed to obtain business data from the data source.

3. The method according to claim 1, characterized in that The calculator set includes a calculator with a single responsibility for data processing. The calculator takes the business data or the processing result of the business data as an input parameter, processes the input parameter, and returns the processing result of the input parameter.

4. The method according to claim 2, characterized in that: The data analysis task is a clustering task of item question and answer data, the business data is the item question and answer data, and the calculator set includes a word segmentation calculator, a similarity calculator, and a clustering calculator; Calling the calculator set set in the data processing logic to process the business data includes: Calling the word segmentation calculator to segment the item question and answer data to obtain a word segmentation result; Calling the similarity calculator to perform similarity calculation on the segmentation records of the same item in the segmentation results to obtain a similarity calculation result; The clustering calculator is called to cluster the item question and answer data based on the similarity calculation result.

5. The method according to claim 1, characterized in that The data source parameter information includes a parameter identifier; Acquiring business data from a data source specified by the data source parameter information includes: According to the parameter identifier, obtain a corresponding data source parameter tuple; wherein the data source parameter tuple includes basic information of the data source and a data reading instruction; Establishing a connection with a corresponding data source according to the basic information of the data source; The data reading instruction is executed to obtain business data from the data source.

6. The method according to claim 5, characterized in that The data source configuration information includes a data source type and a data source parameter tuple corresponding to the data source type.

7. The method according to claim 1, characterized in that The business data includes stock data and / or incremental data, and the incremental data is stored in the data source component by subscribing to the message queue of the business system.

8. A data analysis device, characterized in that: include: A receiving and generating module, used for receiving data source configuration information and generating a corresponding parameter identifier for a data source parameter tuple of the data source configuration information; A script acquisition module is used to acquire a business logic script pre-written for a data analysis task; wherein the business logic script includes data source parameter information and business logic code, and the data source parameter information is a data source parameter tuple or a parameter identifier; A data acquisition module, used to execute the data extraction logic of the business logic code to obtain business data from the data source specified by the data source parameter information; The data processing module is used to execute the data processing logic of the business logic code to call the calculator set set in the data processing logic to process the business data.

9. A data analysis system, characterized in that: include: A management backend, used to receive data source configuration information and generate corresponding parameter identifiers for data source parameter tuples of the data source configuration information; Sending the business logic script written for the data analysis task to the assembler component; wherein the business logic script includes data source parameter information and business logic code, and the data source parameter information is a data source parameter tuple or a parameter identifier; The assembler component is used to execute the data extraction logic of the business logic code to send a data acquisition request to the data source component; and is also used to execute the data processing logic of the business logic code to send a calculator call request to the calculator component; The data source component is used to receive the data acquisition request, obtain business data from the data source specified in the data acquisition request, and send the business data to the assembler component; The calculator component is used to receive the calculator call request, call the calculator set specified in the calculator call request, and perform data processing on the business data.

10. The system according to claim 9, characterized in that The calculator component is further used to: send the processing result to the assembler component, so that the assembler component stores the processing result to the data source component; The system also includes: a retrieval system for retrieving the processing result from the data source component according to data query conditions.

11. An electronic device, characterized in that: include: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 7.

12. A computer readable medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images