Metadata processing method and system, computer equipment and storage medium

Abstract

The invention relates to the technical field of data processing, and discloses a metadata processing method and system, computer equipment and a storage medium. The metadata processing method is applied to the metadata processing system. The method comprises the following steps: acquiring a plurality of original data from a plurality of different types of data sources through an acquisition module; storing a plurality of original data through a storage module based on a preset data storage technology; metadata extraction processing is carried out on the multiple pieces of original data through a metadata extraction module, and multiple pieces of metadata are obtained; constructing an index relationship between each piece of metadata and original data through a metadata management module; performing association analysis on the plurality of metadata through a data analysis module to obtain a metadata analysis result; and displaying the plurality of metadata and the metadata analysis result through a visualization module. By implementing the method provided by the embodiment of the invention, the comprehensiveness of metadata extraction can be improved.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a metadata processing method, system, computer device and storage medium. Background Art

[0002] Metadata is data that describes data, such as the name, type, format, source, timestamp, etc. There are already many products on the market that can extract metadata. However, metadata extraction in a big data environment is still challenging because of the huge amount of data and complex structure. It is difficult to completely extract all metadata, resulting in a lack of comprehensiveness in metadata extraction, which affects users' understanding of the data. Summary of the invention

[0003] The embodiments of the present application provide a metadata processing method, system, computer device and storage medium, which can improve the comprehensiveness of metadata extraction.

[0004] In a first aspect, an embodiment of the present application provides a metadata processing method, which is applied to a metadata processing system, wherein the metadata processing system includes a collection module, a storage module, a metadata extraction module, a metadata management module, a data analysis module, and a visualization module, and the method includes:

[0005] Collecting a plurality of raw data from a plurality of different types of data sources through the collection module;

[0006] Storing the plurality of original data by the storage module based on a preset data storage technology;

[0007] Performing metadata extraction processing from the plurality of original data by the metadata extraction module to obtain a plurality of metadata;

[0008] Constructing an index relationship between each metadata and the original data through the metadata management module;

[0009] Performing correlation analysis on the plurality of metadata by the data analysis module to obtain metadata analysis results;

[0010] The plurality of metadata and the metadata analysis results are displayed through the visualization module.

[0011] In a second aspect, an embodiment of the present application further provides a metadata processing system, the metadata processing system comprising a collection module, a storage module, a metadata extraction module, a metadata management module, a data analysis module and a visualization module, wherein:

[0012] The acquisition module is used to acquire a plurality of raw data from a plurality of different types of data sources;

[0013] The storage module is used to store the plurality of original data based on a preset data storage technology;

[0014] The metadata extraction module is used to perform metadata extraction processing from the plurality of original data to obtain a plurality of metadata;

[0015] The metadata management module is used to construct an index relationship between each metadata and the original data;

[0016] The data analysis module is used to perform correlation analysis on the plurality of metadata to obtain metadata analysis results;

[0017] The visualization module is used to display the plurality of metadata and the metadata analysis results.

[0018] In a third aspect, an embodiment of the present application further provides a computer device, which includes a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the above method when executing the computer program.

[0019] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium, wherein the storage medium stores a computer program, wherein the computer program includes program instructions, and wherein the program instructions can implement the above method when executed by a processor.

[0020] The embodiment of the present application provides a metadata processing method, system, computer device and storage medium. The metadata processing method is applied to a metadata processing system, and the metadata processing system includes a collection module, a storage module, a metadata extraction module, a metadata management module, a data analysis module and a visualization module. The method includes: collecting multiple raw data from multiple different types of data sources through the collection module; storing multiple raw data based on a preset data storage technology through the storage module; performing metadata extraction processing from multiple raw data through the metadata extraction module to obtain multiple metadata; constructing an index relationship between each metadata and the raw data through the metadata management module; performing association analysis on multiple metadata through the data analysis module to obtain metadata analysis results; and displaying multiple metadata and the metadata analysis results through the visualization module. The metadata processing system in the embodiment of the present application can collect raw data from multiple different types of data sources, thereby improving the comprehensiveness of metadata extraction, and after the metadata is extracted from the raw data, the embodiment will also perform association analysis on the metadata, thereby improving the depth of metadata analysis. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0022] Figure 1 A schematic diagram of an application scenario of the metadata processing method provided in an embodiment of the present application;

[0023] Figure 2 A flowchart of a metadata processing method provided in an embodiment of the present application;

[0024] Figure 3 A schematic diagram of a sub-process of the metadata processing method provided in an embodiment of the present application;

[0025] Figure 4 A schematic block diagram of a metadata processing system provided in another embodiment of the present application;

[0026] Figure 5 A schematic block diagram of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0027] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0028] It should be understood that when used in this specification and the appended claims, the terms "include" and "comprises" indicate the presence of described features, integers, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or combinations thereof.

[0029] It should also be understood that the terms used in this application specification are only for the purpose of describing specific embodiments and are not intended to limit the application. As used in this application specification and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms.

[0030] It should be further understood that the term “and / or” used in the specification and appended claims refers to any combination and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0031] Embodiments of the present application provide a metadata processing method, system, computer device, and storage medium.

[0032] The executor of the metadata processing method may be the metadata processing system provided in the embodiment of the present application, or a computer device integrated with the metadata processing system, wherein the metadata processing system may be implemented in hardware or software, and the computer device may be a terminal or a server.

[0033] See also Figure 1 , Figure 1 Schematic diagram of an application scenario of the metadata processing method provided in the embodiment of the present application. The metadata processing method is applied to Figure 1 In the metadata processing system in the embodiment, the metadata processing system can collect multiple raw data from multiple different types of data sources through a collection module ( Figure 1 N is an integer greater than 1); storing multiple raw data based on a preset data storage technology through a storage module; performing metadata extraction and processing from multiple raw data through a metadata extraction module to obtain multiple metadata; constructing an index relationship between each metadata and the raw data through a metadata management module; performing association analysis on multiple metadata through a data analysis module to obtain metadata analysis results; and displaying multiple metadata and metadata analysis results through a visualization module. The metadata processing system in the embodiment of the present application can collect raw data from a variety of different types of data sources, thereby improving the comprehensiveness of metadata extraction, and after performing metadata extraction on the raw data, this embodiment will also perform association analysis on the metadata, thereby improving the depth of metadata analysis.

[0034] Figure 2 is a flowchart of a metadata processing method provided in an embodiment of the present application. The metadata processing method is applied to a metadata processing system, which includes a collection module, a storage module, a metadata extraction module, a metadata management module, a data analysis module, and a visualization module. Figure 2 As shown, the method includes the following steps S110-S160.

[0035] S110 , collecting a plurality of original data from a plurality of different types of data sources through the collection module.

[0036] Specifically, it includes accessing and storing data of different types and formats to ensure that the system can process data from various data sources.

[0037] There are many different types of data sources including:

[0038] Structured data sources: such as relational databases, non-relational databases, Hive data warehouses, Redshift data warehouses, etc.

[0039] Semi-structured data sources: such as Extensible Markup Language (XML) files, JSON files, Comma-Separated Values ​​(CSV) files, etc.

[0040] Unstructured data sources: such as text files, pictures, audio, video, emails, social media, etc.

[0041] Real-time data sources: such as stock market indices, network logs, transaction data, etc.

[0042] External data sources: such as regulatory information, news data, public data sets, etc.

[0043] S120. Storing the plurality of original data by the storage module based on a preset data storage technology.

[0044] In this embodiment, the data storage technology is a big data storage technology, including: distributed file system storage technology, column storage system storage technology, document database storage technology and graph database storage technology.

[0045] in:

[0046] Distributed file system storage technology is used to distribute data across multiple nodes, communicate and coordinate through the network, and provide high availability and scalability.

[0047] Column storage system storage technology: Organize and store data by columns, suitable for processing large-scale sparse data, such as Internet logs, social media data, etc. Usually uses a distributed architecture to support horizontal expansion and high concurrent access.

[0048] Document database storage technology: Document databases store data in document formats such as JSON and XML, and support flexible data models and query languages. They usually use a distributed architecture and support high availability and scalability.

[0049] Graph database storage technology: Models data as nodes, edges, and attributes, and processes complex relational data, such as social networks, knowledge graphs, etc. It usually uses a distributed architecture to support high-concurrency access and high-performance queries.

[0050] Among them, the data storage technology provided by this embodiment has distributed storage function, data compression function, data partitioning function, data caching function and data optimization function, thereby realizing efficient data storage.

[0051] S130: Perform metadata extraction processing from the plurality of original data by the metadata extraction module to obtain a plurality of metadata.

[0052] Specifically, metadata is extracted from raw data through data collection tools (such as streaming data (Extract Transform Load, ETL) tools, data crawlers, etc.). Specifically, metadata can be extracted when raw data is collected.

[0053] S140: construct an index relationship between each metadata and the original data through the metadata management module.

[0054] The subsequent system can perform metadata association analysis based on the index relationship and allow users to view the original data of different metadata.

[0055] You can use the existing metadata management system to build a metadata catalog, manage data assets, and provide data lineage and data quality functions. In addition, the metadata management module also provides functions such as collecting, organizing, storing, updating, and deleting metadata.

[0056] S150: Perform correlation analysis on the plurality of metadata by the data analysis module to obtain metadata analysis results.

[0057] In some embodiments, the data analysis module can automatically perform various types of association analysis on the acquired metadata. Specifically, the data analysis module includes a statistical analysis submodule, a machine learning analysis submodule, a time series analysis submodule, a social network analysis submodule, and a semantic analysis submodule; see Figure 3 , the data analysis module performs correlation analysis on the plurality of metadata to obtain metadata analysis results, including:

[0058] S1501. Perform statistical description on the plurality of metadata through the statistical analysis submodule to obtain statistical analysis results.

[0059] The statistical analysis results include metadata quantity, metadata distribution, metadata frequency and metadata relevance;

[0060] S1502, performing classification processing and association analysis processing on the plurality of metadata by the machine learning analysis submodule to obtain metadata classification results and metadata association results;

[0061] S1503, predicting the time series of the plurality of metadata by the time series analysis submodule to obtain a metadata change trend prediction result;

[0062] S1504, mining the social network structures of the plurality of metadata through the social network analysis submodule to obtain social network features and rules of the plurality of metadata;

[0063] S1505, performing sentiment analysis on each metadata according to the original data corresponding to each metadata by the semantic analysis submodule to obtain a sentiment analysis result;

[0064] S1506: Determine the metadata analysis result according to the statistical analysis result, the metadata classification result, the metadata association result, the metadata change trend prediction result, the social network characteristics and rules, and the sentiment analysis result.

[0065] Among them, the statistical analysis submodule is specifically used to perform basic statistical descriptions of metadata, such as calculating the quantity, distribution, frequency, and relevance of metadata. These statistical information can help users understand the overall characteristics and distribution of metadata, and provide a basis for further analysis and mining.

[0066] Machine learning analysis submodule: It is used to analyze and mine metadata using machine learning algorithms to gain deeper insights and discoveries. For example, clustering algorithms can be used to classify metadata, or association rule algorithms can be used to discover the association between metadata.

[0067] Time series analysis submodule: used to analyze and predict the metadata time series to obtain the data change trend and future prediction. For example, the Autoregressive Integrated Moving Average Model (ARIMA) can be used to model and predict the metadata time series.

[0068] Social network analysis submodule: used to analyze and mine the social network structure of metadata to obtain the characteristics and rules of social networks. For example, a network clustering algorithm can be used to group the social network of metadata, or a community detection algorithm can be used to discover the community structure of social networks.

[0069] Semantic analysis submodule: It is used to analyze and understand the semantic content of metadata to obtain the meaning and context of the data. For example, natural language processing technology can be used to perform sentiment analysis and topic modeling on the text content of metadata.

[0070] In other embodiments, after the metadata is acquired, further metadata analysis is performed only after receiving a metadata analysis request triggered by a user. Specifically, the data analysis module includes a plurality of analysis submodules; the metadata analysis results are obtained by performing association analysis on the plurality of metadata through the data analysis module, including:

[0071] Obtain a metadata analysis request; determine a target analysis submodule corresponding to the metadata analysis request from the plurality of analysis submodules; and perform correlation analysis on the plurality of metadata through the target analysis submodule to obtain the metadata analysis result.

[0072] Specifically, the metadata analysis request carries a corresponding analysis type, and then the system selects at least one submodule corresponding to the analysis type from a plurality of analysis submodules as a target analysis submodule.

[0073] S160: Display the plurality of metadata and the metadata analysis results through the visualization module.

[0074] Furthermore, the metadata processing system further includes a user authority module; before displaying the plurality of metadata and the metadata analysis results through the visualization module, the method further includes:

[0075] Get the target user's data viewing request;

[0076] Determine the current user authority corresponding to the target user through the user authority module;

[0077] The displaying of the plurality of metadata and the metadata analysis results by the visualization module includes:

[0078] The metadata corresponding to the current user authority and the metadata analysis result corresponding to the current user authority are displayed through the visualization module.

[0079] Specifically, the present embodiment can also perform data access control through the user authority module. For example, the system administrator can set user roles and authorities to allow only specific users or roles to access specific data sets or data tables. This ensures that only authorized users can access and operate data, preventing data leakage and misoperation.

[0080] Furthermore, the metadata processing system also includes a security module, through which encryption operations of designated data can be implemented to protect the security and privacy of the data. For example, the system can use an encryption algorithm to encrypt sensitive data, and only users with decryption keys can access and decrypt the data. This can effectively prevent data from being accessed and stolen by unauthorized users.

[0081] The security module also provides access log records to monitor user access and operations on data. For example, the system uses access log records to record user login time, data access time, accessed data sets or data tables, etc., so that administrators can track and audit user operations.

[0082] The security module also provides security audit functions to ensure the security and compliance of data. For example, the system can conduct security audits regularly to check the security and compliance of data and discover and fix potential security vulnerabilities and risks.

[0083] The security module also provides data backup and recovery functions to prevent data loss and damage. For example, the system can perform regular data backup and save a copy of the data so that it can be restored when the data is lost or damaged. At the same time, the system can also implement data recovery to restore the data to the state at a certain point in time so that users can restore the historical version of the data.

[0084] Furthermore, the metadata processing system further includes a performance optimization module; after the data analysis module performs correlation analysis on the plurality of metadata and obtains the metadata analysis result, the method further includes:

[0085] Determine the query frequency of each metadata analysis result within a preset time period through a performance optimization module;

[0086] The metadata analysis results whose query frequency is greater than a preset frequency threshold are stored in the local memory, and the metadata analysis results whose query frequency is less than or equal to the preset frequency threshold are stored in the cloud database.

[0087] It can be seen that this embodiment can store commonly used data or results in the memory so that users can quickly access and use them, which can reduce the calculation and I / O load of the system and improve the performance and response speed of the system.

[0088] Furthermore, the visualization module is preset with a plurality of candidate visualization components; before displaying the plurality of metadata and the metadata analysis results through the visualization module, the method further comprises:

[0089] A user's orchestration operation on the multiple candidate visualization components is received.

[0090] Specifically, the visualization module provided in this embodiment is designed with a user-friendly visualization interface that supports various data visualization components and interactive operations. The system can utilize existing data visualization libraries and technologies to implement various visualization components such as charts, graphs, and maps. Users can freely select and combine candidate visualization components as needed to meet different data display requirements.

[0091] In summary, the metadata processing system in the embodiment of the present application can collect original data from a variety of different types of data sources, thereby improving the comprehensiveness of metadata extraction. After extracting metadata from the original data, this embodiment will also perform correlation analysis on the metadata, thereby improving the depth of metadata analysis.

[0092] Among them, the software tools or components not provided by our company that appear in the embodiments of this application are merely examples and do not represent actual use.

[0093] Figure 4 4 is a schematic block diagram of a metadata processing system 400 provided in an embodiment of the present application. Figure 4 As shown, corresponding to the above metadata processing method, the present application also provides a metadata processing system 400. The metadata processing system 400 includes a unit for executing the above metadata processing method, and the metadata processing system 400 can be configured in a terminal or a server. Figure 4 The metadata processing system 400 includes a collection module 401, a storage module 402, a metadata extraction module 403, a metadata management module 404, a data analysis module 405 and a visualization module 406:

[0094] The acquisition module 401 is used to acquire a plurality of raw data from a plurality of different types of data sources;

[0095] The storage module 402 is used to store the plurality of original data based on a preset data storage technology;

[0096] The metadata extraction module 403 is used to extract metadata from the plurality of original data to obtain a plurality of metadata;

[0097] The metadata management module 404 is used to construct an index relationship between each metadata and the original data;

[0098] The data analysis module 405 is used to perform correlation analysis on the plurality of metadata to obtain metadata analysis results;

[0099] The visualization module 406 is used to display the plurality of metadata and the metadata analysis results.

[0100] In some embodiments, the data analysis module 405 includes a statistical analysis submodule, a machine learning analysis submodule, a time series analysis submodule, a social network analysis submodule, and a semantic analysis submodule; the data analysis module 405 is specifically used to:

[0101] Performing statistical description on the plurality of metadata by the statistical analysis submodule to obtain statistical analysis results, wherein the statistical analysis results include the quantity of metadata, metadata distribution, metadata frequency and metadata relevance;

[0102] The machine learning analysis submodule performs classification processing and association analysis processing on the plurality of metadata to obtain metadata classification results and metadata association results;

[0103] The time series analysis submodule is used to perform prediction processing on the time series of the plurality of metadata to obtain a metadata change trend prediction result;

[0104] Mining the social network structures of the plurality of metadata through the social network analysis submodule to obtain the social network features and rules of the plurality of metadata;

[0105] Perform sentiment analysis on each metadata according to the original data corresponding to each metadata by the semantic analysis submodule to obtain a sentiment analysis result;

[0106] The metadata analysis result is determined according to the statistical analysis result, the metadata classification result, the metadata association result, the metadata change trend prediction result, the social network characteristics and rules, and the sentiment analysis result.

[0107] In some embodiments, the data analysis module includes multiple analysis submodules; the data analysis module 405 is specifically used to:

[0108] Get metadata analysis request;

[0109] Determining a target analysis submodule corresponding to the metadata analysis request from the plurality of analysis submodules;

[0110] The target analysis submodule performs correlation analysis on the plurality of metadata to obtain the metadata analysis result.

[0111] In some embodiments, the metadata processing system 400 further includes a user authority module; the user authority module is used to:

[0112] Obtain the data viewing request of the target user; determine the current user permissions corresponding to the target user;

[0113] The visualization module 406 is specifically used for:

[0114] The metadata corresponding to the current user authority and the metadata analysis result corresponding to the current user authority are displayed.

[0115] In some embodiments, the metadata processing system 400 further includes a performance optimization module; the performance optimization module is specifically used to:

[0116] Determine the query frequency of each metadata analysis result within a preset time period through a performance optimization module;

[0117] The metadata analysis results whose query frequency is greater than a preset frequency threshold are stored in the local memory, and the metadata analysis results whose query frequency is less than or equal to the preset frequency threshold are stored in the cloud database.

[0118] In some embodiments, the visualization module is preset with a plurality of candidate visualization components; the visualization module 406 is further used to:

[0119] A user's orchestration operation on the multiple candidate visualization components is received.

[0120] In some embodiments, the data storage technology includes: distributed file system storage technology, column storage system storage technology, document database storage technology, and graph database storage technology.

[0121] In summary, the metadata processing system 400 in the embodiment of the present application can collect original data from a variety of different types of data sources, thereby improving the comprehensiveness of metadata extraction. In addition, after extracting metadata from the original data, the present embodiment also performs correlation analysis on the metadata, thereby improving the depth of metadata analysis.

[0122] It should be noted that those skilled in the art can clearly understand that the specific implementation process of the above-mentioned metadata processing system and each unit can refer to the corresponding description in the aforementioned method embodiment, and for the convenience and brevity of description, it will not be repeated here.

[0123] The metadata processing system can be implemented in the form of a computer program. Figure 5 Runs on the computer device shown.

[0124] See also Figure 5 , Figure 5 1 is a schematic block diagram of a computer device provided in an embodiment of the present application. The computer device 500 may be a terminal or a server. Specifically, a metadata processing system is deployed in the computer device 500, and the metadata processing system includes a collection module, a storage module, a metadata extraction module, a metadata management module, a data analysis module, and a visualization module.

[0125] See also Figure 5 The computer device 500 includes a processor 502 , a memory and a network interface 505 connected via a system bus 501 , wherein the memory may include a non-volatile storage medium 503 and an internal memory 504 .

[0126] The non-volatile storage medium 503 can store an operating system 5031 and a computer program 5032. The computer program 5032 includes program instructions, and when the program instructions are executed, the processor 502 can execute a metadata processing method.

[0127] The processor 502 is used to provide computing and control capabilities to support the operation of the entire computer device 500 .

[0128] The internal memory 504 provides an environment for the operation of the computer program 5032 in the non-volatile storage medium 503. When the computer program 5032 is executed by the processor 502, the processor 502 can execute a metadata processing method.

[0129] The network interface 505 is used to communicate with other devices over the network. Figure 5 The structure shown in the figure is merely a block diagram of a portion of the structure related to the solution of the present application, and does not constitute a limitation on the computer device 500 to which the solution of the present application is applied. The specific computer device 500 may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0130] The processor 502 is used to run the computer program 5032 stored in the memory to implement the following steps:

[0131] Collecting a plurality of raw data from a plurality of different types of data sources through the collection module;

[0132] Storing the plurality of original data by the storage module based on a preset data storage technology;

[0133] Performing metadata extraction processing from the plurality of original data by the metadata extraction module to obtain a plurality of metadata;

[0134] Constructing an index relationship between each metadata and the original data through the metadata management module;

[0135] Performing correlation analysis on the plurality of metadata by the data analysis module to obtain metadata analysis results;

[0136] The plurality of metadata and the metadata analysis results are displayed through the visualization module.

[0137] It should be understood that in the embodiment of the present application, the processor 502 may be a central processing unit (CPU), and the processor 502 may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0138] It can be understood by those skilled in the art that all or part of the processes in the method for implementing the above embodiment can be completed by instructing the relevant hardware through a computer program. The computer program includes program instructions, and the computer program can be stored in a storage medium, which is a computer-readable storage medium. The program instructions are executed by at least one processor in the computer system to implement the process steps of the embodiment of the above method.

[0139] Therefore, the present application also provides a storage medium. The storage medium may be a computer-readable storage medium. The storage medium stores a computer program, wherein the computer program includes program instructions. When the program instructions are executed by a processor, the processor executes the following steps:

[0140] Collecting a plurality of raw data from a plurality of different types of data sources through the collection module;

[0141] Storing the plurality of original data by the storage module based on a preset data storage technology;

[0142] Performing metadata extraction processing from the plurality of original data by the metadata extraction module to obtain a plurality of metadata;

[0143] Constructing an index relationship between each metadata and the original data through the metadata management module;

[0144] Performing correlation analysis on the plurality of metadata by the data analysis module to obtain metadata analysis results;

[0145] The plurality of metadata and the metadata analysis results are displayed through the visualization module.

[0146] The storage medium may be a USB flash drive, a mobile hard disk, a read-only memory (ROM), a magnetic disk, or an optical disk, etc., which are computer-readable storage media that can store program codes.

[0147] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in terms of function in the above description. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0148] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of each unit is only a logical function division, and there may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed.

[0149] The steps in the method of the embodiment of the present application can be adjusted in order, combined and deleted according to actual needs. The units in the device of the embodiment of the present application can be combined, divided and deleted according to actual needs. In addition, the functional units in the various embodiments of the present application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0150] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, terminal, or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application.

[0151] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed in the present application, and these modifications or replacements should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. A metadata processing method, characterized in that: The metadata processing method is applied to a metadata processing system, which includes a collection module, a storage module, a metadata extraction module, a metadata management module, a data analysis module and a visualization module. The method includes: Collecting a plurality of raw data from a plurality of different types of data sources through the collection module; Storing the plurality of original data by the storage module based on a preset data storage technology; Performing metadata extraction processing from the plurality of original data by the metadata extraction module to obtain a plurality of metadata; Constructing an index relationship between each metadata and the original data through the metadata management module; Performing correlation analysis on the plurality of metadata by the data analysis module to obtain metadata analysis results; The plurality of metadata and the metadata analysis results are displayed through the visualization module.

2. The method according to claim 1, characterized in that: The data analysis module includes a statistical analysis submodule, a machine learning analysis submodule, a time series analysis submodule, a social network analysis submodule and a semantic analysis submodule; The step of performing association analysis on the plurality of metadata by the data analysis module to obtain metadata analysis results includes: Performing statistical description on the plurality of metadata by the statistical analysis submodule to obtain statistical analysis results, wherein the statistical analysis results include the quantity of metadata, metadata distribution, metadata frequency and metadata relevance; The machine learning analysis submodule performs classification processing and association analysis processing on the plurality of metadata to obtain metadata classification results and metadata association results; The time series analysis submodule is used to perform prediction processing on the time series of the plurality of metadata to obtain a metadata change trend prediction result; Mining the social network structures of the plurality of metadata through the social network analysis submodule to obtain the social network features and rules of the plurality of metadata; Perform sentiment analysis on each metadata according to the original data corresponding to each metadata by the semantic analysis submodule to obtain a sentiment analysis result; The metadata analysis result is determined according to the statistical analysis result, the metadata classification result, the metadata association result, the metadata change trend prediction result, the social network characteristics and rules, and the sentiment analysis result.

3. The method according to claim 1, characterized in that The data analysis module includes a plurality of analysis submodules; the data analysis module performs correlation analysis on the plurality of metadata to obtain metadata analysis results, including: Get metadata analysis request; Determining a target analysis submodule corresponding to the metadata analysis request from the plurality of analysis submodules; The target analysis submodule performs correlation analysis on the plurality of metadata to obtain the metadata analysis result.

4. The method according to claim 1, characterized in that: The metadata processing system further includes a user authority module; before displaying the plurality of metadata and the metadata analysis results through the visualization module, the method further includes: Get the target user's data viewing request; Determine the current user authority corresponding to the target user through the user authority module; The displaying of the plurality of metadata and the metadata analysis results by the visualization module includes: The metadata corresponding to the current user authority and the metadata analysis result corresponding to the current user authority are displayed through the visualization module.

5. The method according to claim 1, characterized in that The metadata processing system also includes a performance optimization module; After performing association analysis on the plurality of metadata by the data analysis module to obtain metadata analysis results, the method further includes: Determine the query frequency of each metadata analysis result within a preset time period through a performance optimization module; The metadata analysis results whose query frequency is greater than a preset frequency threshold are stored in the local memory, and the metadata analysis results whose query frequency is less than or equal to the preset frequency threshold are stored in the cloud database.

6. The method according to claim 1, characterized in that The visualization module is preset with a plurality of candidate visualization components; before displaying the plurality of metadata and the metadata analysis results through the visualization module, the method further includes: A user's orchestration operation on the multiple candidate visualization components is received.

7. The method according to any one of claims 1 to 6, characterized in that The data storage technology includes: distributed file system storage technology, column storage system storage technology, document database storage technology and graph database storage technology.

8. A metadata processing system, characterized in that: The metadata processing system includes a collection module, a storage module, a metadata extraction module, a metadata management module, a data analysis module and a visualization module, wherein: The acquisition module is used to acquire a plurality of raw data from a plurality of different types of data sources; The storage module is used to store the plurality of original data based on a preset data storage technology; The metadata extraction module is used to perform metadata extraction processing from the plurality of original data to obtain a plurality of metadata; The metadata management module is used to construct an index relationship between each metadata and the original data; The data analysis module is used to perform correlation analysis on the plurality of metadata to obtain metadata analysis results; The visualization module is used to display the plurality of metadata and the metadata analysis results.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the metadata processing method according to any one of claims 1 to 7 is implemented.

10. A storage medium, characterized in that: The storage medium stores a computer program, wherein the computer program includes program instructions, and when the program instructions are executed by a processor, the processor executes the metadata processing method according to any one of claims 1 to 7.

Images