Data analysis method, device and equipment, readable storage medium and program product

Abstract

The embodiment of the invention provides a data analysis method, device and equipment, a readable storage medium and a program product, and the method comprises the steps: generating a data association relation graph according to an editing instruction triggered by a first user on a visual human-computer interaction interface; wherein the data association relation graph comprises at least one data table, and the data tables are mutually connected according to the association relation; for any data table, the data table comprises at least one metadata field, and each metadata field corresponds to at least one database element of the target data source; generating a target physical model according to the data association graph and exporting the target physical model to a target database; a data collection task is generated according to a data collection task configuration instruction of the first user, the data collection task is executed when an execution condition is met, and the data collection task is used for collecting metadata corresponding to each metadata field from a target data source, processing is carried out according to the target physical model to obtain target data, and then the target data is stored in a target database; and analyzing the target data.

Description

Technical Field

[0001] The present invention relates to the field of data analysis technology, and in particular to a data analysis method, device, equipment, readable storage medium and program product. Background Art

[0002] As different organizations continue to advance their digitalization process, more and more data is stored digitally. Since important information may be hidden in the data stored in the data sources of different organizations, analyzing data across data sources to discover hidden information is helpful for business or management decision-making.

[0003] When performing data analysis for different data analysis needs, the software used to implement data analysis still remains in the traditional "requirements analysis - design - implementation - testing - delivery" software development process, requiring analysts to design a set of data analysis software for each data analysis need, resulting in low data analysis efficiency. Summary of the invention

[0004] Embodiments of the present invention provide a data analysis method, apparatus, device, readable storage medium, and program product to solve the problem of low efficiency in developing data analysis logic when performing data analysis for different data analysis requirements in the prior art.

[0005] An embodiment of the present invention provides a data analysis method, including:

[0006] Generate a data association relationship diagram according to an editing instruction triggered by a first user on a visual human-computer interaction interface; wherein the data association relationship diagram includes at least one data table, and each of the data tables is interconnected according to an association relationship; for any data table, the data table includes at least one metadata field, and each metadata field corresponds to at least one database element of a target data source;

[0007] Generate a target physical model according to the data association relationship diagram, and export the target physical model to a target database;

[0008] Generate a data collection task according to the data collection task configuration instruction of the first user, and execute the data collection task when the execution condition of the data collection task is met, wherein the data collection task is used to collect metadata corresponding to each metadata field from the target data source, and process the metadata according to the target physical model to obtain target data and then store it in the target database;

[0009] The target data in the target database is extracted and analyzed according to the target physical model.

[0010] Optionally, the editing instruction includes a field configuration instruction for configuring the metadata fields contained in the data table from the metadata fields contained in the data dictionary;

[0011] The data dictionary is obtained in the following way:

[0012] Determine at least one database element corresponding to at least one data source;

[0013] In response to a second user's instruction to edit the data standard of the database element, generate and publish a metadata standard;

[0014] Metadata fields are generated by referencing field attributes included in the published metadata standard, and the metadata fields are added to the data dictionary.

[0015] Optionally, after generating the target physical model according to the data association relationship graph, the method further includes:

[0016] If the metadata standard corresponding to the target metadata field included in any data table in the data association relationship diagram changes, a modification prompt message indicating that the data association relationship diagram needs to be modified is sent to the first user.

[0017] Optionally, after generating the target physical model according to the data association relationship graph, the method further includes:

[0018] The version number of the target physical model is updated, and the target physical model and / or the editing instruction are backed up.

[0019] Optionally, exporting the target physical model to a target database includes:

[0020] The target physical model is converted into an SQL script, the SQL script is sent to the target database, and the target database is controlled to execute the SQL script to import the target physical model.

[0021] Optionally, before executing the data collection task when the execution condition of the data collection task is met, the method further includes:

[0022] Acquire data source configuration information corresponding to the target data source, and establish a connection with the target data source according to the data source configuration information;

[0023] If the target data source is a database, the data source configuration information includes at least one of the following: database address, database port, database account, and account password;

[0024] If the target data source is a static file, the data source configuration information includes at least one of the following: a file storage location and a file password.

[0025] Based on the same inventive concept, an embodiment of the present invention further provides a data analysis device, including:

[0026] A relationship diagram design module, used to generate a data association relationship diagram according to an editing instruction triggered by a first user on a visual human-computer interaction interface; wherein the data association relationship diagram includes at least one data table, and each of the data tables is interconnected according to an association relationship; for any data table, the data table includes at least one metadata field, and each metadata field corresponds to at least one database element of a target data source;

[0027] A model generation module, used to generate a target physical model according to the data association relationship diagram, and export the target physical model to a target database;

[0028] A metadata collection module, used to generate a data collection task according to the data collection task configuration instruction of the first user, and execute the data collection task when the execution condition of the data collection task is met, wherein the data collection task is used to collect metadata corresponding to each metadata field from the target data source, and process the metadata according to the target physical model to obtain target data and then store it in the target database;

[0029] The target data analysis module is used to extract and analyze the target data in the target database according to the target physical model.

[0030] Based on the same inventive concept, an embodiment of the present invention further provides an electronic device, comprising: a processor and a memory for storing instructions executable by the processor;

[0031] Wherein, the processor is configured to execute the instructions to implement the data analysis method.

[0032] Based on the same inventive concept, an embodiment of the present invention further provides a readable storage medium, wherein the readable storage medium stores computer program code, and when the computer program code runs on a computer, the computer executes the data analysis method.

[0033] Based on the same inventive concept, an embodiment of the present invention further provides a computer program product, which includes: a computer program code, and when the computer program code runs on a computer, the computer executes the data analysis method.

[0034] The beneficial effects of the present invention are as follows:

[0035] The data analysis method, apparatus, device, readable storage medium and program product provided by the embodiments of the present invention generate a target physical model for collecting, processing and storing metadata of a data source through a visual interface, and then analyze the stored target data based on the target physical model, which greatly simplifies the process of building data analysis logic, greatly reduces the technical requirements for users to build data analysis logic, reduces the development cost of data analysis logic, and thus effectively improves the efficiency of data analysis. In addition, by encapsulating the data in the data source as metadata fields and then selecting and configuring them in the data table in the data association diagram, the same data can be reused in different data analysis logics, further reducing the development cost of data analysis logic, thereby effectively improving the efficiency of data analysis. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 A flow chart of a data analysis method provided by an embodiment of the present invention;

[0037] Figure 2 One of the schematic diagrams of the human-computer visualization interface in the data analysis method provided in the embodiment of the present invention;

[0038] Figure 3 The second schematic diagram of the human-computer visualization interface in the data analysis method provided in the embodiment of the present invention;

[0039] Figure 4 A partial flow chart of a data analysis method provided by an embodiment of the present invention;

[0040] Figure 5 A schematic diagram of the structure of a data analysis device provided by an embodiment of the present invention;

[0041] Figure 6 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0042] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and understandable, the present invention will be further described below with reference to the accompanying drawings and examples. However, the exemplary embodiments can be implemented in various forms and should not be understood as being limited to the embodiments described herein; on the contrary, these embodiments are provided to make the present invention more comprehensive and complete, and to fully convey the concepts of the exemplary embodiments to those skilled in the art. The same figure marks in the figures represent the same or similar structures, and thus their repeated descriptions will be omitted. The words expressing position and direction described in the present invention are all explained using the accompanying drawings as examples, but changes can be made as needed, and the changes made are all included in the scope of protection of the present invention. The drawings of the present invention are only used to illustrate the relative position relationship and do not represent the true proportions.

[0043] It should be noted that specific details are described in the following description to facilitate a full understanding of the present invention. However, the present invention can be implemented in a variety of other ways different from those described herein, and those skilled in the art can make similar generalizations without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below. The subsequent description of the specification is a preferred embodiment of the present application, but the description is for the purpose of illustrating the general principles of the present application and is not intended to limit the scope of the present application. The scope of protection of the present application shall be determined by the definition of the attached claims.

[0044] It should be stated that the acquisition, transmission, storage, and use of data in the technical solution of this application are in compliance with the requirements of relevant national laws and regulations.

[0045] The data analysis method, device, equipment, readable storage medium and program product provided by the embodiments of the present invention are described in detail below in conjunction with the accompanying drawings.

[0046] The embodiment of the present invention provides a data analysis method, such as Figure 1 As shown, including:

[0047] S110: Generate a data association relationship diagram according to the editing instruction triggered by the first user on the visual human-computer interaction interface. The data association relationship diagram includes at least one data table, and each of the data tables is connected to each other according to an association relationship. For any data table, the data table includes at least one metadata field, and each metadata field corresponds to at least one database element of the target data source.

[0048] In a specific implementation process, the database elements include at least one of the following:

[0049] Primary key, index, original field, foreign key, library, stored procedure parameter, table partition, table space, trigger.

[0050] In the specific implementation process, the editing instructions may specifically include a new data table instruction, a data table connection instruction, a field configuration instruction for configuring a data table containing metadata fields from metadata fields contained in a data dictionary, a data table deletion instruction, etc. For example, Figure 2 As shown, the first user can trigger a new data table instruction by moving the preset data table element in the candidate area to the drawing area on the visual human-computer interaction interface. The first user can trigger a data table connection instruction by drawing a connection line on the data tables with an associated relationship on the visual human-computer interaction interface. Figure 3 As shown, the first user can click on the data table, and after the data table expands the data dictionary, trigger the field configuration instruction by clicking on the selected metadata field in the metadata fields provided by the data dictionary.

[0051] Furthermore, if Figure 4 As shown, the data dictionary is obtained as follows:

[0052] S210: Determine at least one database element corresponding to at least one data source.

[0053] In the specific implementation process, the third user may manually configure the corresponding database element for the target data source. The third user may be the same user as the first user or a different user, and the embodiment of the present invention does not make too many restrictions here.

[0054] S220. In response to the second user's instruction to edit the data standard of the database element, generate and publish a metadata standard.

[0055] In the specific implementation process, metadata standards include at least one of the following:

[0056] Data name, data type, data length, data description information, data constraints, and data dependencies.

[0057] In the specific implementation process, the data standard editing instructions may include new metadata standard instructions, pick-up instructions for selectively importing database elements into metadata standards, modification instructions for modifying metadata standards, etc. By selectively importing database elements into metadata standards and modifying them, metadata standards can be quickly established based on existing database elements of the target data source, thereby simplifying the establishment of metadata standards and improving efficiency. In the process of generating metadata standards, the comparison information between the published metadata standards and the metadata standards being edited and / or the suggestion information generated for the metadata standards being edited based on the published metadata standards (such as suggested data names, suggested data types, etc.) can also be displayed to the second user for reference when editing the metadata standards.

[0058] Furthermore, before publishing the metadata standard, the generated metadata standard can be submitted to the fourth user, and the metadata field can be published after receiving the review information returned by the fourth user; if the fourth user fails the review, the review failure information and the suggested modification notes can be returned to the first user, so that the first user can modify the metadata standard according to the requirements of the fourth user. In this way, the fourth user reviews each metadata standard and decides whether the metadata standard can be published based on the data specifications formulated by the country, industry, enterprise, etc., which can ensure that the metadata standard has unified norms and standardize the metadata standard format.

[0059] Furthermore, after the metadata standard is published, the database elements corresponding to the metadata standard can also be evaluated. If the attribute value of the published metadata standard does not match the attribute value of the same attribute of the corresponding database element (for example, the data name of the original field of the existing or newly added data source does not match the data name of the metadata standard), rectification information is sent to the third user (the rectification information may include the attribute value of the unmatched metadata standard) so that the third user can adjust the corresponding database element of the corresponding data source to enable the data source to set the attribute value of the database element based on the metadata standard.

[0060] S230. Generate metadata fields by referencing field attributes included in published metadata standards.

[0061] In the specific implementation process, since the metadata standard already includes the field attributes of the metadata field, such as the data name, data type, data length, data constraints, data dependencies, etc., the metadata field can be generated by referencing the field attributes included in the metadata standard.

[0062] S240. Add metadata fields to the data dictionary.

[0063] In a specific implementation process, the third user may be the same user as the second user, or the third user may be a different user from the second user, and the embodiment of the present invention does not make too many limitations here.

[0064] In the specific implementation process, Figure 4 The steps shown can be independent of Figure 1 The steps shown are implemented, then the second user first generates a data dictionary, and then the first user edits the data association diagram to configure the first metadata field contained in the data table, so that the second user can be the same user as the first user, or the second user can be a different user from the first user. Or, Figure 4 At least some of the steps shown are implemented in the process of implementing the step S110, that is, the first user and the second user are the same user, and in the process of the first user generating the data association relationship diagram in the step S110, when the first user configures the first metadata field included in the data table included in the data association relationship diagram, the first user generates a new metadata standard through the steps S220-S240 and generates a new metadata field to add to the data dictionary. The embodiment of the present invention is not limited in detail here.

[0065] By constructing metadata fields and organizing them into a data dictionary for user configuration and selection, metadata from different data sources can be managed uniformly, and compatible processing of metadata in different formats from different data sources can be achieved.

[0066] S120: Generate a target physical model according to the data association relationship diagram, and export the target physical model to a target database.

[0067] In a specific implementation process, a data connection can be established with a target database, and then the target physical model can be directly exported to the target database; or, the target physical model can be converted into a structured query language (SQL) script, and the SQL script can be sent to the target database, and the target database can be controlled to execute the SQL script to import the target physical model. The embodiments of the present invention are not limited in detail here.

[0068] In the specific implementation process, the target physical model can be a physical model generated by the programming language of a database for a specific version (e.g., the version with the highest usage rate). Then, when the target physical model is exported to the target database, it is also necessary to convert the target physical model according to the target programming language dialect corresponding to the target database, and export the target physical model converted into the target programming language dialect format to the target database. If the target physical model is exported to the target database by converting the target physical model into an SQL script and importing it into the target database for execution, the target physical model can be converted into an SQL script in the target programming language dialect format corresponding to the target database, and then the SQL script in the target programming language dialect format is sent to the target database, and the target database is controlled to execute the SQL script to import the target physical model. Alternatively, the target physical model can be a physical model generated by a universal programming language for all versions of databases, so there is no need to convert the target physical model into the format of the programming language dialect of the target database, and the target physical model can be adapted to databases of different versions to achieve the reuse of the same physical model for different databases. The embodiments of the present invention are not limited too much here.

[0069] S130: Generate a data collection task according to the data collection task configuration instruction of the first user.

[0070] S140, executing the data collection task when the execution conditions of the data collection task are met. The data collection task is used to collect metadata corresponding to each metadata field from the target data source, and process the metadata according to the target physical model to obtain target data and then store it in the target database.

[0071] During the specific implementation process, the execution conditions of the data collection task may include reaching a preset data collection cycle, the amount of metadata stored in the target data source reaching a preset data amount, receiving a user-triggered data collection instruction, etc., which can be specifically determined based on the instructions of the data collection task configuration instruction of the first user, and the embodiments of the present invention do not make too many limitations here.

[0072] During the specific implementation process, the data collection task may include a full collection task for collecting metadata in the target data source, and / or an incremental collection task for collecting new metadata for all metadata currently stored in the target data source relative to all metadata stored in the target data source when the data collection task was last executed. The specific task can be determined based on the instructions of the data collection task configuration instruction of the first user, and the embodiments of the present invention do not make excessive limitations here.

[0073] Before executing the data collection task, it is also necessary to obtain the data source configuration information corresponding to the target data source, and establish a connection with the target data source according to the data source configuration information. If the target data source is a database, the data source configuration information includes at least one of the following: database address, database port, database account, account password; if the target data source is a static file (such as an Excel format file, a character-separated value (CSV) format file), the data source configuration information includes at least one of the following: file storage location, file password.

[0074] S150, extracting and analyzing target data in the target database according to the target physical model.

[0075] In the specific implementation process, the analysis method can be determined according to the specific content of the target data. For example, for online shopping scenarios, analysis can be performed based on consumer feature data (such as user age, user gender, etc.), commodity feature data (such as commodity category, commodity model, commodity price, brand, etc.), and order feature data (such as order generation time, order generation quantity, etc.) contained in the target data to obtain information such as shopping preferences of different types of users, sales of different types of commodities, and the impact of different sales strategies of online shopping platforms on consumer shopping intentions, thereby providing assistance for the business decisions of commodity sellers and online shopping platforms. For another example, for service scenarios, analysis can be performed based on service category data, business data for each service, and feedback data for each service (such as complaints, suggestions, satisfaction, etc.) contained in the target data to obtain service process information that needs to be optimized, thereby providing assistance for the management decision-making of the service provider. The embodiments of the present invention are not overly limited herein.

[0076] In this way, through the data analysis method provided by the embodiment of the present invention, a target physical model for collecting, processing and storing metadata of a data source is generated through a visual interface, and then the stored target data is analyzed based on the target physical model, which greatly simplifies the construction process of the data analysis logic, greatly reduces the technical requirements for users to build data analysis logic, and reduces the development cost of data analysis logic, thereby effectively improving the efficiency of data analysis. In addition, by encapsulating the data in the data source as metadata fields and then selecting and configuring them in the data table in the data association diagram, the same data can be reused in different data analysis logics, further reducing the development cost of data analysis logic, thereby effectively improving the efficiency of data analysis.

[0077] Further optionally, after generating the target physical model according to the data association relationship diagram, the method further includes:

[0078] If the metadata standard corresponding to the target metadata field included in any data table in the data association relationship diagram changes, a modification prompt message indicating that the data association relationship diagram needs to be modified is sent to the first user.

[0079] In the specific implementation process, the modification prompt information can be sent to the first user through an interactive element (such as a pop-up window, etc.) on the visual human-computer interaction interface that implements the data analysis method, or the modification prompt information can be sent to the first user through email, telephone, instant messaging software (such as QQ, WeChat, DingTalk, Feishu, etc.) message, short message service (Short Message Service, SMS), etc., and the embodiments of the present invention are not further limited here.

[0080] In the specific implementation process, the modification prompt information may also indicate the target metadata field that has been specifically changed in the data association relationship diagram and the data table corresponding to the target metadata field, so that the first user can quickly locate the content that needs to be modified.

[0081] In this way, after the metadata standards and metadata fields change, the first user can be promptly prompted to modify the data association diagram to avoid errors in the target physical model caused by changes in the metadata standards and metadata fields.

[0082] Further optionally, after generating the target physical model according to the data association relationship diagram, the method further includes:

[0083] Update the version number of the target physical model and back up the target physical model and / or editing instructions.

[0084] In this way, the target physical model and the content of each modification can be recorded, so as to find out the cause of the problem when the newly generated target physical model has a problem; and to facilitate rolling back to the old version of the target physical model to ensure the stability of the data analysis process.

[0085] Based on the same inventive concept, the embodiment of the present invention further provides a data analysis device U, such as Figure 5 As shown, including:

[0086] The relationship diagram design module M103 is used to generate a data association relationship diagram according to the editing instruction triggered by the first user on the visual human-computer interaction interface; wherein the data association relationship diagram includes at least one data table, and each of the data tables is interconnected according to the association relationship; for any data table, the data table includes at least one metadata field, and each metadata field corresponds to at least one database element of the target data source;

[0087] A model generation module M104 is used to generate a target physical model according to the data association relationship diagram, and export the target physical model to a target database;

[0088] The metadata collection module M302 is used to generate a data collection task according to the data collection task configuration instruction of the first user, and execute the data collection task when the execution condition of the data collection task is met. The data collection task is used to collect metadata corresponding to each metadata field from the target data source, and process the metadata according to the target physical model to obtain target data and then store it in the target database;

[0089] The target data analysis module M400 is used to extract and analyze the target data in the target database according to the target physical model.

[0090] Optionally, the data analysis device further includes:

[0091] The data table design module M102 is used to configure the metadata fields contained in the data table from the metadata fields contained in the data dictionary;

[0092] The data source management module M301 is used to determine at least one database element corresponding to at least one data source;

[0093] The metadata standard module M200 is used to generate and publish a metadata standard in response to a second user's data standard editing instruction for the database element;

[0094] The data dictionary module M101 is used to generate metadata fields by referring to field attributes included in the published metadata standard, and add the metadata fields to the data dictionary.

[0095] Optionally, the data analysis device U further includes:

[0096] The metadata management module M303 is used to send a modification prompt message to the first user indicating that the data association diagram needs to be modified if the metadata standard corresponding to the target metadata field included in any data table in the data association diagram changes.

[0097] Optionally, the data analysis device U further includes:

[0098] The model version management module M105 is used to update the version number of the target physical model and back up the target physical model and / or the editing instruction.

[0099] Optionally, the model generation module M104 is specifically used for:

[0100] The target physical model is converted into an SQL script, the SQL script is sent to the target database, and the target database is controlled to execute the SQL script to import the target physical model.

[0101] Optionally, the data source management module M301 is further used to:

[0102] Acquire data source configuration information corresponding to the target data source, and establish a connection with the target data source according to the data source configuration information;

[0103] If the target data source is a database, the data source configuration information includes at least one of the following: database address, database port, database account, and account password;

[0104] If the target data source is a static file, the data source configuration information includes at least one of the following: a file storage location and a file password.

[0105] Optionally, the data analysis device U further includes:

[0106] The metadata analysis module is used to analyze the metadata's blood relationship, influence relationship, attribute difference, etc.

[0107] It should be understood that the data analysis device embodiment provided in the embodiment of the present invention is only illustrative. For example, the division of the modules is only a logical function division, and there may be other division methods in actual implementation, such as multiple modules or components can be combined or integrated into another system (for example, the data dictionary module M101, the data table design module M102, the relationship diagram design module M103, the model generation module M104, and the model version management module M105 are integrated into the model module M100, and the data source management module M301, the metadata acquisition module M302, and the metadata management module M303 are integrated into the metadata module M300), or some features can be ignored or not executed. Another point is that the coupling or direct coupling or communication connection between each other shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or modules, which can be electrical, mechanical or other forms. The modules described as separate components may or may not be physically separated, and the components displayed as modules may or may not be physical modules, that is, they may be located in one place, or they may be distributed on multiple network modules. Some or all of the modules can be selected according to actual needs to achieve the purpose of the scheme of this embodiment. In addition, each functional module in the embodiment of the present application can be integrated in a processing module, or each module can exist physically alone, or two or more modules can be integrated in one module. The above-mentioned integrated module can be implemented in the form of hardware or in the form of software functional modules. If the integrated module is implemented in the form of software functional modules and sold or used as an independent product, it can be stored in a readable storage medium.

[0108] Since the working principle of the data analysis device is consistent with the data analysis method described above, the specific implementation process of the device can refer to the implementation of the corresponding method and will not be repeated here.

[0109] Based on the same inventive concept, the present application embodiment also provides an electronic device, such as Figure 6 As shown, a processor 110 and a memory 120 for storing instructions executable by the processor 110 .

[0110] The processor 110 is configured to execute the instructions to implement the data analysis method.

[0111] In the specific implementation process, the device may have relatively large differences due to different configurations or performances, and may include one or more processors 110, memory 120, and readable storage medium 130. The memory 120 and / or readable storage medium 130 include one or more applications 131 or data 132. The memory 120 and / or readable storage medium 130 may also include one or more operating systems 133, such as Windows, Mac OS, Linux, IOS, Android, Unix, FreeBSD, etc. Among them, the memory 120 and the readable storage medium 130 may be temporary storage or permanent storage. The application 131 may include one or more modules ( Figure 6 ), each module may include a series of instruction operations. Further, the processor 110 may be configured to communicate with the readable storage medium 130, and execute a series of instruction operations in the readable storage medium 130 on the device. The device may also include one or more power supplies ( Figure 6 one or more network interfaces 140, wherein the network interface 140 includes a wired network interface 141 and / or a wireless network interface 142; and one or more input / output interfaces 143.

[0112] Based on the same inventive concept, an embodiment of the present application provides a readable storage medium, wherein the readable storage medium stores a computer program code, and the computer program code is used to implement the data analysis method.

[0113] In the specific implementation process, any combination of one or more programming languages ​​can be used to write computer program codes for implementing the data analysis method as described above. The programming languages ​​include object-oriented programming languages ​​such as Java, C++, C#, etc., and conventional procedural programming languages ​​such as "C" language, assembly language or similar programming languages.

[0114] In the specific implementation process, the readable storage medium can be any available medium that can be stored by a computer or a data storage device such as a configuration server, data center, etc. that includes one or more available media integrated. The available medium can be a magnetic medium (such as a floppy disk, a hard disk, a magnetic tape), an optical medium (such as a high-density digital video disc (Digital Video Disc, DVD), a video compact disc (Video Compact Disc, VCD)), or a semiconductor medium (such as a solid state disk (Solid State Disk, SSD)), etc.

[0115] Since the principle of solving the problem by the above-mentioned readable storage medium corresponds to the data analysis method described above, the implementation of the above-mentioned readable storage medium can refer to the implementation of the corresponding method, and the repeated parts will not be repeated.

[0116] Based on the same inventive concept, an embodiment of the present application further provides a computer program product, which includes: a computer program code, and when the computer program code runs on a computer, the computer implements the data analysis method described above.

[0117] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions may be stored in a readable storage medium, or transmitted from a readable storage medium to another readable storage medium, for example, the computer instructions may be transmitted from a website site, a computer, a configuration server or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (Digital Subscriber Line, DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, configuration server or data center.

[0118] Since the principle of solving the problem by the above-mentioned computer program product corresponds to the data analysis method described above, the implementation of the above-mentioned computer program product can refer to the implementation of the corresponding method, and the repeated parts will not be repeated.

[0119] The data analysis method, apparatus, device, readable storage medium and program product provided by the embodiments of the present invention generate a target physical model for collecting, processing and storing metadata of a data source through a visual interface, and then analyze the stored target data based on the target physical model, which greatly simplifies the process of building data analysis logic, greatly reduces the technical requirements for users to build data analysis logic, reduces the development cost of data analysis logic, and thus effectively improves the efficiency of data analysis. In addition, by encapsulating the data in the data source as metadata fields and then selecting and configuring them in the data table in the data association diagram, the same data can be reused in different data analysis logics, further reducing the development cost of data analysis logic, thereby effectively improving the efficiency of data analysis.

[0120] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0121] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0122] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0123] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0124] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A data analysis method, characterized in that: include: Generate a data association relationship diagram according to an editing instruction triggered by a first user on a visual human-computer interaction interface; wherein the data association relationship diagram includes at least one data table, and each of the data tables is interconnected according to an association relationship; for any data table, the data table includes at least one metadata field, and each metadata field corresponds to at least one database element of a target data source; Generate a target physical model according to the data association relationship diagram, and export the target physical model to a target database; Generate a data collection task according to the data collection task configuration instruction of the first user, and execute the data collection task when the execution condition of the data collection task is met, wherein the data collection task is used to collect metadata corresponding to each metadata field from the target data source, and process the metadata according to the target physical model to obtain target data and then store it in the target database; The target data in the target database is extracted and analyzed according to the target physical model.

2. The method according to claim 1, characterized in that The editing instructions include field configuration instructions for configuring the metadata fields contained in the data table from the metadata fields contained in the data dictionary; The data dictionary is obtained in the following way: Determine at least one database element corresponding to at least one data source; In response to a second user's instruction to edit the data standard of the database element, generate and publish a metadata standard; Metadata fields are generated by referencing field attributes included in the published metadata standard, and the metadata fields are added to the data dictionary.

3. The method according to claim 2, characterized in that After generating the target physical model according to the data association relationship diagram, the method further includes: If the metadata standard corresponding to the target metadata field included in any data table in the data association relationship diagram changes, a modification prompt message indicating that the data association relationship diagram needs to be modified is sent to the first user.

4. The method according to claim 1, characterized in that After generating the target physical model according to the data association relationship diagram, the method further includes: The version number of the target physical model is updated, and the target physical model and / or the editing instruction are backed up.

5. The method according to claim 1, characterized in that The exporting the target physical model to a target database comprises: The target physical model is converted into an SQL script, the SQL script is sent to the target database, and the target database is controlled to execute the SQL script to import the target physical model.

6. The method according to claim 1, characterized in that Before executing the data collection task when the execution condition of the data collection task is met, the method further includes: Acquire data source configuration information corresponding to the target data source, and establish a connection with the target data source according to the data source configuration information; If the target data source is a database, the data source configuration information includes at least one of the following: database address, database port, database account, and account password; If the target data source is a static file, the data source configuration information includes at least one of the following: a file storage location and a file password.

7. A data analysis device, characterized in that: include: A relationship diagram design module, used to generate a data association relationship diagram according to an editing instruction triggered by a first user on a visual human-computer interaction interface; wherein the data association relationship diagram includes at least one data table, and each of the data tables is interconnected according to an association relationship; for any data table, the data table includes at least one metadata field, and each metadata field corresponds to at least one database element of a target data source; A model generation module, used to generate a target physical model according to the data association relationship diagram, and export the target physical model to a target database; A metadata collection module, used to generate a data collection task according to the data collection task configuration instruction of the first user, and execute the data collection task when the execution condition of the data collection task is met, wherein the data collection task is used to collect metadata corresponding to each metadata field from the target data source, and process the metadata according to the target physical model to obtain target data and then store it in the target database; The target data analysis module is used to extract and analyze the target data in the target database according to the target physical model.

8. An electronic device, characterized in that: include: a processor and a memory for storing instructions executable by the processor; Wherein, the processor is configured to execute the instructions to implement the data analysis method as described in any one of claims 1-6.

9. A readable storage medium, characterized in that: The readable storage medium stores computer program codes, and when the computer program codes are executed on a computer, the computer executes the data analysis method according to any one of claims 1 to 6.

10. A computer program product, characterized in that The computer program product comprises: a computer program code, and when the computer program code is run on a computer, the computer is enabled to execute the data analysis method according to any one of claims 1 to 6.

Images