Data analysis method and device, electronic equipment, storage medium and program product

By selecting the appropriate storage location and storage method on the stream processing platform, supporting the configuration of multiple data sources and analysis components, and using the orchestration canvas interface to generate data analysis tasks, solving the problem of only a single type of data analysis in the existing technology. , realizes multi-dimensional analysis of multiple data types to meet users' multi-dimensional analysis needs.

CN118964898APending Publication Date: 2024-11-15HANGZHOU DBAPPSECURITY CO LTD
View PDF 0 Cites 2 Cited by

Patent Information

Application Number
CN202411211277.9
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-30
Publication Date
2024-11-15

AI Technical Summary

Technical Problem

The existing data analysis management platform can only analyze a single type of data and cannot meet the multi-dimensional analysis needs of multiple data types. Especially in the context of the rapid development of big data and artificial intelligence, users analyze different data types. The demand is getting bigger.

Method used

Provide a data analysis method, by selecting appropriate storage locations and storage methods on the stream processing platform, performing multi-dimensional analysis of user data, supporting the configuration of multiple data sources and analysis components, and using the orchestration canvas interface to generate data analysis tasks. And conduct multi-dimensional analysis through data description information and rule models, correlation models, and statistical models.

Benefits of technology

It realizes multi-dimensional analysis of user data of multiple data types to meet users' actual needs for data analysis, supports storage and processing of different data types, and meets the needs of multi-dimensional analysis scenarios.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118964898A_ABST
    Figure CN118964898A_ABST
Patent Text Reader

Abstract

The invention discloses a data analysis method and device, electronic equipment, a storage medium and a program product, and is applied to the technical field of data processing. The method comprises the steps that the storage mode and the storage position of data are selected based on the data type of received user data, and the storage position comprises a message theme of a data analysis management platform, a temporary message theme of a stream processing platform and a user message theme. And displaying an arrangement canvas interface supporting the configuration of multiple data sources and multiple analysis components to the user, generating a data analysis task according to a target data source and a target analysis component selected by the user on the arrangement canvas, and performing corresponding analysis on each piece of user data in the user message theme according to the data description information and the data analysis task. According to the method and the device, the problem of data analysis of a single data type in related technologies can be solved, and data analysis requirements of different data types of users can be met.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a data analysis method, device, electronic device, non-volatile storage medium and computer program product. Background Art

[0002] Currently, the data analysis management platform can only analyze a single type of data, such as only supporting the analysis of standardized JSON data, and the data analysis dimension is relatively single. With the rapid development of big data and artificial intelligence, the types of user data are increasing, and the user's demand for analysis of different data types is also increasing. Single-dimensional data analysis cannot meet the needs of some specific application scenarios.

[0003] In view of this, realizing multi-dimensional data analysis of user data of various data types is a technical problem that needs to be solved by technical personnel in this field.

[0004] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present application, and therefore may include information that does not constitute the prior art known to ordinary technicians in the field. Summary of the invention

[0005] The present application provides a data analysis method, device, electronic device, non-volatile storage medium and computer program product, which can realize multi-dimensional data analysis of user data of various data types.

[0006] In order to solve the above technical problems, this application provides the following technical solutions:

[0007] On one hand, the present application provides a data analysis method, which is applied to a data analysis management platform for installing a stream processing platform, including:

[0008] When user data is received, a storage location and a corresponding storage method are selected based on the data type to which the user data belongs; the storage location includes a temporary message topic pre-created by the user on the stream processing platform, a user message topic customized by the user, and a local message topic of the data analysis management platform; the storage method includes a direct storage method and a storage method after converting the data type;

[0009] Displaying the orchestration canvas interface to the user, generating a data analysis task according to the scenario analysis configuration parameters input by the user, and performing corresponding analysis on each user data in the user message topic according to the data description information and the data analysis task;

[0010] Among them, the orchestration canvas interface includes at least data source configuration options and analysis component configuration options to support the configuration of multiple data sources and multiple analysis components. The scenario analysis configuration parameters are the target data source and target analysis component selected by the user; the data description information is the definition of data elements corresponding to different user data types.

[0011] Exemplarily, the arrangement canvas interface further includes a destination configuration option, and after the corresponding analysis of each user data in the user message topic according to the data description information and the data analysis task, it also includes:

[0012] Generate early warning information for corresponding user data based on the analysis results;

[0013] According to the destination configuration parameters input by the user, the warning information is output to the index of the stream processing platform or the search engine or to the data table of the database.

[0014] Exemplarily, the arrangement canvas interface further includes a data processing option, after the corresponding analysis of each user data in the user message topic according to the data description information and the data analysis task, further includes:

[0015] When receiving the data processing configuration parameters, generating a data reprocessing task according to the data configuration parameters;

[0016] Based on the data reprocessing task, the analysis results and / or user data are processed.

[0017] Exemplarily, the performing corresponding analysis on each user data in the user message topic according to the data description information and the data analysis task includes:

[0018] Pre-build a data dictionary based on different data types to serve as data description information;

[0019] According to the data type of the current user data to be analyzed in the user message subject, the corresponding data description information is matched in the data dictionary, and the current user data to be analyzed is analyzed in multiple dimensions based on one or any combination of rule models, association models, statistical models and statistical indicators.

[0020] Exemplarily, the selecting a storage location and a corresponding storage method based on the data type to which the user data belongs includes:

[0021] When the user data belongs to the data exchange format type, it is directly sent to the corresponding user message topic pre-created by the stream processing platform;

[0022] When the user data does not belong to the data exchange format type, a data aggregation task is created and a data sending port is configured; the user data is sent to the data sending port, and the user data is written to the corresponding temporary message topic through the data aggregation task;

[0023] The user data in the temporary message topic are enriched. If the enriched current user data is suitable for data analysis by the data analysis management platform, the enriched current user data will be sent to the local message topic of the data analysis management platform; if the enriched current user data is not suitable for data analysis by the data analysis management platform, the enriched current user data will be sent to the corresponding user message topic.

[0024] Exemplarily, the enriching of each user data in the temporary message topic includes:

[0025] When receiving a data processing task configuration request, a pre-created task canvas interface is displayed to the user, and a data processing task is automatically generated according to the data processing task configuration parameters; the task canvas interface includes operator configuration options and output configuration options, and the operator configuration options include operators corresponding to different data types;

[0026] If the current user data belongs to the log message protocol format type, the data processing task calls the regular parsing operator to process the current user data into the data exchange format type;

[0027] If the current user data is in the storage data format of a search engine, a database or a network server, the data processing task calls a json encoding operator to process the current user data into a data exchange format type;

[0028] If the converted current user data meets the standardized data requirements of the data analysis management platform, the converted current user data will be sent to the corresponding local message topic; if the converted current user data does not meet the standardized data requirements of the data analysis management platform, the converted current user data will be sent to the corresponding user message topic.

[0029] On the other hand, the present application provides a data analysis device, which is applied to a data analysis management platform for installing a stream processing platform, including:

[0030] A data storage module, configured to select a storage location and a corresponding storage method based on the data type of the user data when receiving the user data; the storage location includes a temporary message topic pre-created by the user on the stream processing platform, a user message topic customized by the user, and a local message topic of the data analysis management platform;

[0031] The data analysis module is used to display the orchestration canvas interface to the user, generate data analysis tasks according to the scenario analysis configuration parameters input by the user, and perform corresponding analysis on each user data in the user message topic according to the data description information and the data analysis tasks; wherein the orchestration canvas interface includes at least data source configuration options and analysis component configuration options to support the configuration of multiple data sources and multiple analysis components, and the scenario analysis configuration parameters are the target data source and target analysis component selected by the user; the data description information is the definition of data elements corresponding to different user data types.

[0032] The present application also provides an electronic device, comprising a processor, wherein the processor is configured to implement the steps of the data analysis method as described in any of the preceding items when executing a computer program stored in a memory.

[0033] The present application also provides a non-volatile storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the data analysis method as described in any of the preceding items are implemented.

[0034] Finally, the present application also provides a computer program product, including a computer program / instruction, which implements the steps of the data analysis method as described in any of the preceding items when executed by a processor.

[0035] The advantage of the technical solution provided by the present application is that it supports user data of any data type input by the user through the storage method and its corresponding storage location data, and the data that meets the original log standard is sent to the data analysis management platform for data analysis. Other types of data support customizing multiple data sources through the model orchestration canvas, and processing data in analysis scenarios of different dimensions, thereby realizing multi-dimensional data analysis of user data of various data types and meeting the actual needs of users for data analysis.

[0036] In addition, the present application also provides corresponding implementation devices, electronic devices, non-volatile storage media and computer program products for the data analysis method, which further makes the method more practical, and the devices, electronic devices, non-volatile storage media and computer program products have corresponding advantages.

[0037] The technical features mentioned above, the technical features to be mentioned below, and the technical features shown separately in the drawings can be combined with each other arbitrarily, as long as the combined technical features are not contradictory. All feasible feature combinations are technical contents clearly recorded in this application. Any of the multiple sub-features contained in the same sentence can be applied independently, and does not have to be applied together with other sub-features.

[0038] It should be understood that the foregoing general description and the following detailed description are exemplary only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the technical solutions of the present application or related technologies, the following briefly introduces the drawings required for use in the embodiments or related technical descriptions. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0040] Figure 1 A flowchart of a data analysis method provided in this application;

[0041] Figure 2 A schematic diagram of another data analysis method provided for this application;

[0042] Figure 3 A structural diagram of a specific implementation of the data analysis device provided in this application;

[0043] Figure 4 A structural diagram of a specific implementation of the electronic device provided in this application. DETAILED DESCRIPTION

[0044] In order to enable those skilled in the art to better understand the present application, the present application is further described in detail below in conjunction with the accompanying drawings and specific embodiments. The terms "including" and "having" and any variations thereof in the specification and the above-mentioned drawings are intended to cover non-exclusive inclusions. The term "exemplary" means "used as an example, embodiment or illustrative". Any embodiment described herein as "exemplary" is not necessarily to be interpreted as being superior or better than other embodiments.

[0045] Various non-limiting embodiments of the present application are described in detail below. In order to better illustrate the present application, numerous specific details are provided in the specific embodiments below. It should be understood by those skilled in the art that the present application can also be implemented without these specific details. In other examples, methods, means, components and circuits well known to those skilled in the art are not described in detail in order to highlight the subject matter of the present application.

[0046] First see Figure 1 , Figure 1A flow chart of a data analysis method provided in this application. The data analysis management platform only supports data analysis and processing in a single data format. For example, Ailpha (security analysis and management platform) can only analyze standardized json data at present. The data analysis management platform of the present invention installs any stream processing platform. By executing the data analysis method of this embodiment, multi-dimensional analysis of data of any type and format can be achieved, such as Kafka (distributed stream processing platform), which is a high-throughput distributed publish-subscribe message system that can process all action stream data of consumers on the website, unify online and offline message processing through Hadoop's parallel loading mechanism, and provide real-time messages through clusters. The process of supporting multi-type data and multi-dimensional analysis by the data analysis management platform may include the following:

[0047] S101: When user data is received, a storage location and a corresponding storage method are selected based on the data type to which the user data belongs.

[0048] The user data in this step is the data that the user wants to use the data analysis management platform for data analysis. The data analysis management platform supports access to any type of user data, that is, the data type to which the user data belongs can be any data format or data type, including but not limited to data in json format, syslog type data, Elasticsearch (open source search engine), mysql (database), and data stored in http Server (network server). There are at least three types of storage locations. One is the temporary Topic (topic) created in advance by the user on the stream processing platform. Topic is a message classification unit. Each Topic includes multiple Partitions, and each Partition includes multiple Replicas. The producer sends user data, the Broker receives the user data, and stores the user data in the Topic. The storage format of user data is Topic name-Partition name-serial number. For the sake of distinction, it is defined as a temporary message topic. One is the custom topic created in advance by the user on the stream processing platform. For the sake of distinction, it is defined as a user message topic. Another is the topic of the data analysis management platform itself. For the sake of distinction, it is defined as a local message topic. Since the data analysis management platform does not originally support multiple types of data, accordingly, it will not receive data that is not in the standard format allowed by it. In order to support the analysis of multi-format data, this embodiment adopts different storage methods for user data of different data types and stores them in different locations. Correspondingly, the storage methods are direct storage method and storage method after converting the data type. In other words, if the user data type input by the user belongs to the standard format data originally supported by the data analysis management platform, it can be stored directly, that is, according to the original data analysis process. If it is user data in other formats, data format conversion is required, and then storage is performed after conversion.

[0049] S102: Displaying the orchestration canvas interface to the user, generating a data analysis task according to the scenario analysis configuration parameters input by the user, and performing corresponding analysis on each user data in the user message topic according to the data description information and the data analysis task.

[0050] For standard format data that is not originally supported by the data analysis management platform, in order to implement data analysis for these types of user data, this embodiment uses Canvas to implement user-defined application scenarios to analyze data. For ease of description, the Canvas used in this step is defined as an orchestration canvas. The visual interface of the orchestration canvas includes at least data source configuration options and analysis component configuration options. The data source configuration options may include multiple selectable data sources, and may also include an input box to support users to input data sources that are not displayed. The analysis component configuration options may include multiple selectable analysis components. The analysis components include but are not limited to rule models, association models, statistical models, and statistical indicators. They may also include analysis component calling interfaces and / or adding interfaces to support users to add analysis components that are not currently included in the system, and to call third-party analysis components through calling interfaces to meet users' needs for more analysis dimensions. Through data source configuration options and analysis component configuration options, users are supported to access multiple data sources in the data analysis management platform, and multiple dimensions of data analysis are implemented through multiple analysis components. The scenario analysis configuration parameters are the target data source and target analysis component selected by the user. The target analysis component can be one analysis component or a combination of multiple analysis components. The target data source is the user data in the user message topic. In the process of analyzing user data, when you need to find the detailed definition of the relevant name in the data flow diagram, you can call the data description information. The data description information is the definition of the data elements corresponding to different user data types, and provides detailed data definitions and instructions for structured analysis. Of course, if the user data in the local message topic also requires customized data analysis, it can also support corresponding analysis of each user data in the local message topic according to the data description information and data analysis tasks. The corresponding data source can select the user data in the local message topic.

[0051] In the technical solution provided in the present application, user data of any data type input by the user is supported through the storage method and its corresponding storage location data. Data that meets the original log standard is sent to the data analysis management platform for data analysis. Other types of data support customizing multiple data sources through the model orchestration canvas, and analysis scenarios of different dimensions process data, thereby realizing multi-dimensional data analysis of user data of various data types and meeting the user's actual needs for data analysis.

[0052] In order to further meet the user's data analysis needs, based on the above embodiment, the above-mentioned arrangement canvas interface may also include destination configuration options and / or data processing options. The user configures the final analysis data output location based on the destination configuration options, and the data processing options can support the user to perform secondary processing on the data. According to the analysis results obtained by processing the data analysis task, warning information is generated for the corresponding user data, and the warning information can also be processed in a graded manner. According to the destination configuration parameters input by the user, the warning information is output to the index of the stream processing platform or search engine or the data table of the database. When the data processing configuration parameters are received, a data reprocessing task is generated according to the data configuration parameters; based on the data reprocessing task, the analysis results and / or user data are processed.

[0053] As can be seen from the above, this embodiment supports secondary data processing by improving the output configuration and analyzing the data after analysis by the analysis component. After the source data generates an early warning through a series of data analysis, it supports output to the specified Kafka or Elasticsearch index or Mysql table, thereby meeting the special needs of users in various application scenarios to the greatest extent.

[0054] In the above embodiment, there is no limitation on how to perform the steps of analyzing the user data in the user message subject according to the data description information and the data analysis task. In this embodiment, an exemplary implementation of how to perform the corresponding analysis on the user data in the user message subject is provided, which may include the following steps:

[0055] A data dictionary is pre-built based on different data types as data description information; the corresponding data description information is matched in the data dictionary according to the data type of the current user data to be analyzed in the user message subject, and a multi-dimensional analysis is performed on the current user data to be analyzed based on one or any combination of rule models, association models, statistical models and statistical indicators.

[0056] In this embodiment, data description information is generated in the form of a data dictionary. The data dictionary is a collection of information that describes the data and is a collection of definitions of all data elements used in the system. Standardize data content. Create a data dictionary of the corresponding type according to different types of data, configure the fields in the data dictionary, and try to improve all fields in this type of data. In the subsequent analysis component data standardization process, provide detailed data definitions and instructions for structured analysis. The analysis component configuration options of this embodiment include rule models, association models, statistical models, and statistical indicators. Correspondingly, the data analysis task is to call one or any combination of rule models, association models, statistical models, and statistical indicators to process user data, which can achieve multi-dimensional data analysis.

[0057] In the above embodiment, there is no limitation on how to store different types of user data. In this embodiment, an exemplary implementation of selecting a storage method and a storage location for user data is provided, which may include the following steps:

[0058] When user data belongs to the data exchange format type, it is directly sent to the corresponding user message topic pre-created by the stream processing platform; when user data does not belong to the data exchange format type, a data aggregation task is created and the data sending port is configured; the user data is sent to the data sending port, and the user data is written to the corresponding temporary message topic through the data aggregation task; each user data in the temporary message topic is enriched, and if the enriched current user data is suitable for data analysis by the data analysis management platform, the enriched current user data is sent to the local message topic of the data analysis management platform; if the enriched current user data is not suitable for data analysis by the data analysis management platform, the enriched current user data is sent to the corresponding user message topic.

[0059] In this embodiment, the data exchange format type may be data in json format, and data not in the data exchange format type may be data in syslog format, Elasticsearch (open source search engine), mysql (database), and data stored in http Server (network server). Enrichment processing may be implemented by data processing operators, which are functions or operations used in data processing and analysis to perform specific conversion or analysis tasks on data, including but not limited to regular parsing operators and json encoding operators.

[0060] Exemplarily, in order to facilitate user operation, enrichment processing can also be implemented by configuring data processing tasks based on the visual interface of the canvas, and the enriched data is stored in the corresponding destination Topic. The process of enrichment processing can be, for example: when a data processing task configuration request is received, a pre-created task canvas interface is displayed to the user, and a data processing task is automatically generated according to the data processing task configuration parameters; the task canvas interface includes operator configuration options and output configuration options, and the operator configuration options include operators corresponding to different data types; if the current user data belongs to the log message protocol format type, the data processing task calls the regular parsing operator to process the current user data into a data exchange format type; if the current user data is in the storage data format of a search engine, database, or network server, the data processing task calls the json encoding operator to process the current user data into a data exchange format type; if the converted current user data meets the standardized data requirements of the data analysis management platform, the converted current user data is sent to the corresponding local message topic; if the converted current user data does not meet the standardized data requirements of the data analysis management platform, the converted current user data is sent to the corresponding user message topic. In this embodiment, syslog type data is processed into json format using regular parsing operators, and data stored in Elasticsearch, mysql, and http Server can be processed into json format through json encoding operators. Among them, the json encoding operator converts different data types into formats supported by json. For example, integers, floating point numbers, Boolean values, strings, etc. are converted to json format. For data that conforms to the standardization of the main product, that is, the data security management platform, after the data enrichment operator, it supports output to the topic of the main product for data analysis by the main product. Data that is not applicable to the main product is output to the user-defined topic through the data processing operator.

[0061] In order to make those skilled in the art more clearly understand that the data analysis management platform of the present invention realizes multi-dimensional analysis of multi-type user data, the present invention also takes the stream processing platform as Kafka as an example to explain the data analysis and processing process of the data analysis management platform on the input user data, such as Figure 2 As shown, it may include the following:

[0062] 1. Users customize and create different topics in Kafka to receive different types of data, and pre-configure corresponding data dictionaries for different data types. The fields in the data dictionary include all fields of the corresponding type of data. At the same time, create task canvases and orchestration canvases.

[0063] 2. When receiving user input data, determine whether the user data is in JSON format. If the user data is in JSON format, send it directly to the corresponding topic of Kafka.

[0064] 3. If the user data is not in JSON format, configure an aggregation task to receive this type of data. After starting the task, the data is sent directly to the specified port, and the aggregation task writes the data to the corresponding temporary Topic.

[0065] 4. Task canvas uses a visual interface to configure data processing tasks, enrich user data in temporary topics, and support regular parsing operators to process syslog type data into json format. Data stored in Elasticsearch, mysql, and http Server can be processed into json format through json encoding operators. Data that conforms to the standardization of the main product can be output to the topic of the main product through the data enrichment operator for data analysis by the main product. Data that is not applicable to the main product can be output to a user-defined topic through the data processing operator.

[0066] 5. The model orchestration canvas uses a visual interface to analyze data and generate warnings through user-defined application scenarios. The canvas specifies data sources and supports analysis of multiple data sources, i.e., in step 4: outputting json data to a custom topic. Configure analysis components to support rule models, association models, statistical models, and statistical indicators. The analysis components support multi-dimensional analysis of data in any combination. After a series of data analysis to generate warnings, the source data can be output to the specified Kafka or Elasticsearch index or Mysql table. Improve the output configuration, and support secondary data processing after analysis by the analysis component.

[0067] 6. Start the model orchestration task flow to ensure the normal operation of the entire data processing and analysis process.

[0068] As can be seen from the above, this embodiment supports processing data access of different data types, configures the accessed data into standard json format data through the task canvas, and sends the data that meets the original log standard to the main product for data analysis. Other types of data support customizing multiple data sources through the orchestration canvas, processing data in analysis scenarios of different dimensions, and generating warning data to push to a third party. It can realize multi-dimensional data analysis of user data of various data types and meet the actual needs of users for data analysis.

[0069] It should be noted that there is no strict order of execution between the steps in this application. As long as they comply with the logical order, these steps can be executed simultaneously or in a certain preset order. Figure 1-Figure 2 This is just a schematic and does not mean that this is the only execution order.

[0070] The present application also provides a corresponding device for the data analysis method, which further makes the method more practical. Among them, the device can be described from the perspective of functional modules and hardware. The data analysis device provided by the present application is introduced below. The device is used to implement the data analysis method provided by the present application. In this embodiment, the data analysis device may include or be divided into one or more program modules, which are stored in a storage medium and executed by one or more processors to complete the data analysis method disclosed in Example 1. The program module referred to in this application refers to a series of computer program instruction segments that can perform specific functions, which is more suitable for describing the execution process of the data analysis device in the storage medium than the program itself. The following description will specifically introduce the functions of each program module in this embodiment. The data analysis device described below and the data analysis method described above can be referenced to each other.

[0071] From the perspective of functional modules, see Figure 3 , Figure 3 The structural diagram of the data analysis device provided in this application in a specific implementation mode is applied to a data analysis management platform for installing a stream processing platform. The device may include:

[0072] The data storage module 301 is used to select a storage location and a corresponding storage method based on the data type of the user data when receiving the user data; the storage location includes a temporary message topic pre-created by the user on the stream processing platform, a user-defined user message topic, and a local message topic of the data analysis management platform;

[0073] The data analysis module 302 is used to display the orchestration canvas interface to the user, generate data analysis tasks according to the scenario analysis configuration parameters input by the user, and perform corresponding analysis on each user data in the user message subject according to the data description information and the data analysis tasks; wherein the orchestration canvas interface includes at least data source configuration options and analysis component configuration options to support the configuration of multiple data sources and multiple analysis components, and the scenario analysis configuration parameters are the target data source and target analysis component selected by the user; the data description information is the definition of data elements corresponding to different user data types.

[0074] Optionally, in some implementations of the present embodiment, the above-mentioned device may also include a warning information push module, which is used to generate warning information for corresponding user data based on the analysis results; and output the warning information to the index of the stream processing platform or the search engine or the data table of the database according to the destination configuration parameters input by the user through the destination configuration options of the orchestration canvas interface.

[0075] Optionally, in some other implementations of the present embodiment, the above-mentioned device may also include a secondary processing module, for example, for generating a data reprocessing task according to the data configuration parameters when receiving the data processing configuration parameters sent by the user through the data processing option of the orchestration canvas interface; based on the data reprocessing task, processing the analysis results and / or user data.

[0076] Optionally, in some other implementations of the present embodiment, the above-mentioned data analysis module 302 can also be used to: pre-build a data dictionary based on different data types as data description information; match the corresponding data description information in the data dictionary according to the data type of the current user data to be analyzed in the user message subject, and perform multi-dimensional analysis on the current user data to be analyzed based on one or any combination of rule models, association models, statistical models and statistical indicators.

[0077] Optionally, in some other implementations of the present embodiment, the data storage module 301 may also be used for: when the user data belongs to the data exchange format type, it is directly sent to the corresponding user message topic pre-created by the stream processing platform; when the user data does not belong to the data exchange format type, a data aggregation task is created, and a data sending port is configured; the user data is sent to the data sending port, and the user data is written to the corresponding temporary message topic through the data aggregation task; each user data in the temporary message topic is enriched, and if the enriched current user data is suitable for data analysis by the data analysis management platform, the enriched current user data is sent to the local message topic of the data analysis management platform; if the enriched current user data is not suitable for data analysis by the data analysis management platform, the enriched current user data is sent to the corresponding user message topic.

[0078] As an exemplary implementation of the above embodiment, the above data storage module 301 can also be further used for: when receiving a data processing task configuration request, displaying a pre-created task canvas interface to the user, and automatically generating a data processing task according to the data processing task configuration parameters; the task canvas interface includes operator configuration options and output configuration options, and the operator configuration options include operators corresponding to different data types; if the current user data belongs to the log message protocol format type, the data processing task calls the regular parsing operator to process the current user data into a data exchange format type; if the current user data is in the storage data format of a search engine, database, or network server, the data processing task calls the json encoding operator to process the current user data into a data exchange format type; if the converted current user data meets the standardized data requirements of the data analysis management platform, the converted current user data is sent to the corresponding local message topic; if the converted current user data does not meet the standardized data requirements of the data analysis management platform, the converted current user data is sent to the corresponding user message topic.

[0079] The functions of each functional module of the data analysis device of the present application can be specifically implemented according to the method in the above method embodiment. The specific implementation process can refer to the relevant description of the above method embodiment, which will not be repeated here.

[0080] It can be seen from the above that this embodiment can realize multi-dimensional data analysis of user data of various data types, meeting the actual needs of users for data analysis.

[0081] The data analysis device mentioned above is described from the perspective of functional modules. Furthermore, the present application also provides an electronic device, which is described from the perspective of hardware. Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application in one implementation manner. Figure 4 As shown, the electronic device includes a memory 40 for storing a computer program; and a processor 41 for implementing the steps of the data analysis method mentioned in any of the above embodiments when executing the computer program.

[0082] Among them, the processor 41 may include one or more processing cores, such as a 4-core processor, an 8-core processor, and the processor 41 may also be a controller, a microcontroller, a microprocessor or other data processing chip. The processor 41 may be implemented in at least one hardware form of DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). The processor 41 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also known as a CPU (Central Processing Unit); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor 41 may be integrated with a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 41 may also include an AI (Artificial Intelligence) processor, which is used to process computing operations related to machine learning.

[0083] The memory 40 may include one or more computer-readable storage media, which may be non-transitory. The memory 40 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices and flash storage devices. In some embodiments, the memory 40 may be an internal storage unit of an electronic device, such as a hard disk of a server. In other embodiments, the memory 40 may also be an external storage device of an electronic device, such as a plug-in hard disk equipped on a server, a smart memory card (Smart Media Card, SMC), a secure digital (SecureDigital, SD) card, a flash card (Flash Card), etc. Further, the memory 40 may also include both an internal storage unit of an electronic device and an external storage device. The memory 40 may not only be used to store application software and various types of data installed in the electronic device, such as: the code of the program in the process of executing the data analysis method, etc., but also be used to temporarily store data that has been output or is to be output. In this embodiment, the memory 40 is at least used to store the following computer program 401, wherein, after the computer program is loaded and executed by the processor 41, the relevant steps of the data analysis method disclosed in any of the aforementioned embodiments can be implemented. In addition, the resources stored in the memory 40 may also include an operating system 402 and data 403, and the storage method may be temporary storage or permanent storage. The operating system 402 may include Windows, Unix, Linux, etc. The data 403 may include, but is not limited to, data corresponding to the data analysis results.

[0084] In some embodiments, the electronic device may further include a display screen 42, an input / output interface 43, a communication interface 44 or a network interface, a power supply 45 and a communication bus 46. Among them, the display screen 42 and the input / output interface 43 such as a keyboard belong to the user interface, and the optional user interface may also include a standard wired interface, a wireless interface, etc. Optionally, in some embodiments, the display may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, and an OLED (Organic Light-Emitting Diode) touch device, etc. The display may also be appropriately referred to as a display screen or a display unit, which is used to display information processed in the electronic device and to display a visual user interface. The communication interface 44 may optionally include a wired interface and / or a wireless interface, such as a WI-FI interface, a Bluetooth interface, etc., which is usually used to establish a communication connection between the electronic device and other electronic devices. The communication bus 46 may be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 4 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0085] Those skilled in the art will understand that Figure 4 The structure shown in the figure does not constitute a limitation on the electronic device, and may include more or fewer components than those shown in the figure, for example, it may also include a sensor 47 for realizing various functions.

[0086] It is understandable that if the data analysis method in the above embodiment is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a non-volatile storage medium. Based on such an understanding, the technical solution of the present invention is essentially or the part that contributes to the relevant technology or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium to perform all or part of the steps of the methods of each embodiment of the present invention. The aforementioned storage medium includes but is not limited to: U disk, mobile hard disk, read-only memory (Read-Only Memory, ROM), random access memory (Random Access Memory, RAM), electrically erasable programmable ROM, register, hard disk, multimedia card, card-type memory (such as SD or DX memory, etc.), magnetic memory, removable disk, CD-ROM, magnetic disk or optical disk, etc. Various media that can store program codes. Based on this, the present invention also provides a non-volatile storage medium on which a computer program is stored, and when the computer program is executed by a processor, the steps of data analysis recorded in any of the above embodiments are performed.

[0087] It is understandable that if the data analysis method in the above-mentioned embodiment is implemented in the form of a software functional unit and sold or used as an independent product, the computer software product does not need to be stored in a physical storage medium. For example, it can be directly transmitted to a computer or other device with information processing capabilities through a wired network or a wireless network to execute all or part of the steps of the methods of each embodiment of the present invention. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the relevant technology or all or part of the technical solution can be embodied in the form of a software product. Based on this, the present invention also provides a computer program product storing a computer program, which, when executed by a processor, performs the steps of data analysis as described in any of the above embodiments.

[0088] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the hardware disclosed in the embodiments, including devices and electronic devices, since they correspond to the methods disclosed in the embodiments, the description is relatively simple, and the relevant parts can be referred to the method part.

[0089] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0090] The above is a detailed introduction to a data analysis method, device, electronic device, non-volatile storage medium and computer program product provided by the present application. Specific examples are used herein to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method and core idea of ​​the present application. It should be pointed out that based on the embodiments in the present application, for ordinary technicians in the technical field, all other embodiments obtained without creative work are within the scope of protection of the present application. Without departing from the principles of the present application, the present application can also be improved and modified, and these improvements and modifications also fall within the scope of protection of the present application.

Claims

1. A data analysis method, characterized in that: The data analysis management platform used to install the stream processing platform includes: When user data is received, a storage location and a corresponding storage method are selected based on the data type to which the user data belongs; the storage location includes a temporary message topic pre-created by the user on the stream processing platform, a user message topic customized by the user, and a local message topic of the data analysis management platform; the storage method includes a direct storage method and a storage method after converting the data type; Displaying the orchestration canvas interface to the user, generating a data analysis task according to the scenario analysis configuration parameters input by the user, and performing corresponding analysis on each user data in the user message topic according to the data description information and the data analysis task; Among them, the orchestration canvas interface includes at least data source configuration options and analysis component configuration options to support the configuration of multiple data sources and multiple analysis components. The scenario analysis configuration parameters are the target data source and target analysis component selected by the user; the data description information is the definition of data elements corresponding to different user data types.

2. The data analysis method according to claim 1, characterized in that: The arrangement canvas interface also includes a destination configuration option, and after the corresponding analysis of each user data in the user message topic according to the data description information and the data analysis task, it also includes: Generate early warning information for corresponding user data based on the analysis results; According to the destination configuration parameters input by the user, the warning information is output to the index of the stream processing platform or the search engine or to the data table of the database.

3. The data analysis method according to claim 1, characterized in that: The arrangement canvas interface also includes a data processing option, after the corresponding analysis of each user data in the user message topic according to the data description information and the data analysis task, it also includes: When receiving the data processing configuration parameters, generating a data reprocessing task according to the data configuration parameters; Based on the data reprocessing task, the analysis results and / or user data are processed.

4. The data analysis method according to claim 1, characterized in that: The performing corresponding analysis on each user data in the user message topic according to the data description information and the data analysis task includes: Pre-build a data dictionary based on different data types to serve as data description information; According to the data type of the current user data to be analyzed in the user message subject, the corresponding data description information is matched in the data dictionary, and the current user data to be analyzed is analyzed in multiple dimensions based on one or any combination of rule models, association models, statistical models and statistical indicators.

5. The data analysis method according to any one of claims 1 to 4, characterized in that: The selecting a storage location and a corresponding storage method based on the data type of the user data includes: When the user data belongs to the data exchange format type, it is directly sent to the corresponding user message topic pre-created by the stream processing platform; When the user data does not belong to the data exchange format type, a data aggregation task is created and a data sending port is configured; the user data is sent to the data sending port, and the user data is written to the corresponding temporary message topic through the data aggregation task; The user data in the temporary message topic are enriched. If the enriched current user data is suitable for data analysis by the data analysis management platform, the enriched current user data will be sent to the local message topic of the data analysis management platform; if the enriched current user data is not suitable for data analysis by the data analysis management platform, the enriched current user data will be sent to the corresponding user message topic.

6. The data analysis method according to claim 5, characterized in that: The enriching of each user data in the temporary message topic includes: When receiving a data processing task configuration request, a pre-created task canvas interface is displayed to the user, and a data processing task is automatically generated according to the data processing task configuration parameters; the task canvas interface includes operator configuration options and output configuration options, and the operator configuration options include operators corresponding to different data types; If the current user data belongs to the log message protocol format type, the data processing task calls the regular parsing operator to process the current user data into the data exchange format type; If the current user data is in the storage data format of a search engine, a database or a network server, the data processing task calls a json encoding operator to process the current user data into a data exchange format type; If the converted current user data meets the standardized data requirements of the data analysis management platform, the converted current user data will be sent to the corresponding local message topic; if the converted current user data does not meet the standardized data requirements of the data analysis management platform, the converted current user data will be sent to the corresponding user message topic.

7. A data analysis device, characterized in that: The data analysis management platform used to install the stream processing platform includes: A data storage module, configured to select a storage location and a corresponding storage method based on the data type of the user data when receiving the user data; the storage location includes a temporary message topic pre-created by the user on the stream processing platform, a user message topic customized by the user, and a local message topic of the data analysis management platform; The data analysis module is used to display the orchestration canvas interface to the user, generate data analysis tasks according to the scenario analysis configuration parameters input by the user, and perform corresponding analysis on each user data in the user message topic according to the data description information and the data analysis tasks; wherein the orchestration canvas interface includes at least data source configuration options and analysis component configuration options to support the configuration of multiple data sources and multiple analysis components, and the scenario analysis configuration parameters are the target data source and target analysis component selected by the user; the data description information is the definition of data elements corresponding to different user data types.

8. An electronic device, characterized in that: The method comprises a processor and a memory, wherein the processor is used to implement the steps of the data analysis method according to any one of claims 1 to 6 when executing a computer program stored in the memory.

9. A non-volatile storage medium, characterized in that: The non-volatile storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the data analysis method according to any one of claims 1 to 6 are implemented.

10. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the steps of the data analysis method according to any one of claims 1 to 6 are implemented.

Citation Information

Cited By

  • Intelligent data analysis result online sharing method based on big data

    CN122471474A

  • Method for online sharing of intelligent data analysis results based on big data

    CN122471474B