Implementation method and device of enterprise-level data middle platform system that aggregates multi-source data

By building an enterprise-level data middle platform system, the problem of enterprise data silos is solved, the unified management and efficient circulation of multi-source data is realized, time-sensitive and efficient data services are provided, and diversified data applications and business needs are supported.

CN112580914BActive Publication Date: 2024-12-06BEIJING GRIDSUM TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN201910942541.9
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2019-09-30
Publication Date
2024-12-06
Estimated Expiration
2039-09-30

AI Technical Summary

Technical Problem

It is difficult for enterprises to connect and circulate massive data scattered in various systems, resulting in the inability to effectively explore the value of data, affecting business decisions and market insights.

Method used

Build an enterprise-level data middle platform system, and call appropriate computing modes and storage modes through pre-set functional modules and business scenario mapping to collect, calculate, store and analyze multi-source data to form a unified data service system.

Benefits of technology

It realizes unified management and efficient circulation of massive data, provides timely and efficient data services, and supports diversified data applications and business needs.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN112580914B_ABST
    Figure CN112580914B_ABST
Patent Text Reader

Abstract

The present invention discloses an implementation method and device of an enterprise-level data middle-end system that collects multi-source data, relates to the field of data processing technology, pre-builds an enterprise-level data middle-end system, forms a data service system with unified standards and calibers for massive data, and at the same time, the platform guarantees the timeliness and high efficiency of providing data services. The main technical scheme of the present invention is: constructing data assets, data assets include a variety of different types of functional modules with data management capabilities, and the functional modules are used to provide front-end data services; receiving data processing requirements of business scenarios, and the data processing requirements include processing structured data, semi-structured data, and unstructured data; by finding the mapping relationship between pre-set business scenarios and functional modules, obtaining the corresponding functional modules for executing the data processing requirements; controlling the functional modules to call the calculation mode, perform calculation operations, output data processing results and store them.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to an implementation method and device for an enterprise-level data middle platform system that aggregates multi-source data. Background Art

[0002] In the past process of informatization, enterprises have generated a large amount of production and operation data and application data in professional fields. With the development of the Internet of Things, mobile technology, and sensor technology in recent years, enterprises have also accumulated a large amount of Internet of Things (IoT) data, image data, audio and video data, etc.

[0003] However, this massive amount of data is scattered across various enterprise systems, unable to connect or circulate. Furthermore, due to limitations in traditional technologies, it's difficult for companies to globally mine the value of this scattered data. While data value drives operational decisions, corporate innovation, and market insights, it currently remains largely promotional, with companies unable to truly reap the value of their data assets. Summary of the Invention

[0004] In view of this, the present invention provides an implementation method and device for an enterprise-level data middle-office system that aggregates multi-source data. The main purpose is to pre-build an enterprise-level data middle-office system, so as to rely on big data technology and data governance system to collect, calculate, store, model, analyze, and mine massive data to form a data service system with unified standards and caliber. At the same time, the platform ensures the timeliness and high efficiency of data services.

[0005] In order to achieve the above object, the present invention mainly provides the following technical solutions:

[0006] In one aspect, the present invention provides a method for implementing an enterprise-level data middle platform system that aggregates multi-source data, the method comprising:

[0007] Build data assets, which include various types of functional modules with data management capabilities, and are used to provide front-end data services;

[0008] Receiving data processing requirements of a business scenario, wherein the data processing requirements include processing structured data, semi-structured data, and unstructured data;

[0009] By searching for the mapping relationship between the pre-set business scenarios and the functional modules, the corresponding functional modules for executing the data processing requirements are obtained;

[0010] The calculation mode is called by controlling the functional modules, performing calculations and storing data processing results.

[0011] Optionally, controlling the functional module to call a corresponding calculation mode, outputting and storing data processing results includes:

[0012] Parsing the amount and type of data to be processed from the data processing requirements, where the data type is at least structured data, semi-structured data, or unstructured data;

[0013] Obtaining timeliness requirements included in the data processing requirements;

[0014] Select a matching computing mode based on the data volume, data type, and timeliness requirements, where the computing mode includes at least: in-memory computing, data pre-aggregation, offline computing, and streaming computing;

[0015] According to the calculation mode, the model is used to perform calculation operations and output data processing results;

[0016] Analyzing the amount and type of data contained in the data processing result;

[0017] Select a matching data storage mode based on the data volume and data type, the storage mode including at least: Mpp data storage, object storage, distributed data storage and non-relational database storage;

[0018] The data processing result is stored according to the data storage mode.

[0019] Optionally, after constructing the data asset, the method further includes:

[0020] Add a functional overview to each functional module;

[0021] Pre-customize multiple business scenarios;

[0022] By matching the data processing requirements corresponding to each business scenario with the functional overview corresponding to each functional module, one or more functional modules matching the business scenario are obtained;

[0023] A mapping relationship between the business scenario and a functional module or a combination of multiple functional modules is established.

[0024] Optionally, the method further includes:

[0025] During system operation, monitor whether components are functioning properly; and / or

[0026] Monitor the system's operation logs; and / or

[0027] Monitoring the system for alerts; and / or

[0028] A service gateway that monitors the system.

[0029] Optionally, the functional modules are at least the following: data governance standard module, data quality management module, data warehouse technology module, data asset catalog module, metadata management module, master data management module, data security management module, data life cycle management module, enterprise knowledge graph construction module and enterprise search engine module.

[0030] Optionally, the data services include at least the following: data report services, business intelligence services, enterprise large screen services, API services, message bus services, and mobile application services.

[0031] On the other hand, the present invention also provides an implementation device for an enterprise-level data middle platform system that aggregates multi-source data, the device comprising:

[0032] A construction unit, configured to construct data assets, wherein the data assets include various types of functional modules with data management capabilities, and the functional modules are configured to provide front-end data services;

[0033] A receiving unit, configured to receive data processing requirements of a business scenario, wherein the data processing requirements include processing structured data, semi-structured data, and unstructured data;

[0034] An acquiring unit, configured to acquire a corresponding functional module for executing the data processing requirement received by the receiving unit by searching a mapping relationship between a preset business scenario and a functional module;

[0035] The execution unit is used to call the calculation mode by controlling the functional module, perform calculations and store data processing results.

[0036] Optionally, the execution unit includes:

[0037] a parsing module, configured to parse the amount and type of data to be processed from the data processing requirement, wherein the data type is at least structured data, semi-structured data or unstructured data;

[0038] An acquisition module, configured to acquire the timeliness requirement included in the data processing requirement;

[0039] A selection module is configured to select a matching computing mode based on the amount of data parsed by the parsing module, the data type, and the timeliness requirement obtained by the obtaining module, wherein the computing mode includes at least: in-memory computing, data pre-aggregation, offline computing, and streaming computing;

[0040] An execution module, configured to execute a calculation operation using a model according to the calculation mode and output a data processing result;

[0041] The parsing module is further used to parse the data volume and data type contained in the data processing result;

[0042] The selection module is further configured to select a matching data storage mode according to the data volume and data type, wherein the storage mode includes at least: Mpp data storage, object storage, distributed data storage, and non-relational database storage;

[0043] A storage module is used to store the data processing result according to the data storage mode selected by the selection module.

[0044] Optionally, the device further includes:

[0045] An adding unit, configured to add a function overview to each function module after the data asset is constructed;

[0046] Customization unit, used to pre-customize multiple business scenarios;

[0047] A matching unit, configured to obtain one or more function modules matching the business scenario by matching the data processing requirements corresponding to the business scenario customized by each of the customization units with the function overview corresponding to the function module added by each of the adding units;

[0048] An establishing unit is used to establish a mapping relationship between the business scenario and a functional module or a combination of multiple functional modules.

[0049] Optionally, the device further includes:

[0050] The monitoring unit is used to monitor whether the components are operating normally during system operation;

[0051] The monitoring unit is further configured to monitor the operation log of the system;

[0052] The monitoring unit is further used to monitor the message warning of the system;

[0053] The monitoring unit is also used to monitor the service gateway of the system.

[0054] Optionally, the functional modules are at least the following: data governance standard module, data quality management module, data warehouse technology module, data asset catalog module, metadata management module, master data management module, data security management module, data life cycle management module, enterprise knowledge graph construction module and enterprise search engine module.

[0055] Optionally, the data services include at least the following: data report services, business intelligence services, enterprise large screen services, API services, message bus services, and mobile application services.

[0056] On the other hand, the present invention also provides a storage medium, which includes a stored program, wherein when the program is running, the device where the storage medium is located is controlled to execute the implementation method of the enterprise-level data middle-office system that aggregates multi-source data as described above.

[0057] In another aspect, the present invention further provides an electronic device, comprising at least one processor, and at least one memory and a bus connected to the processor;

[0058] The processor and the memory communicate with each other via the bus.

[0059] The processor is used to call the program instructions in the memory to execute the implementation method of the enterprise-level data middle platform system that aggregates multi-source data as described above.

[0060] By means of the above technical solution, the technical solution provided by the present invention has at least the following advantages:

[0061] The present invention provides an implementation method and device for an enterprise-level data middle-office system that collects multi-source data. The present invention pre-constructs data assets, which include various types of functional modules with data management capabilities. When receiving data processing requirements for a business scenario, it finds the functional modules that meet the requirements and calls the corresponding calculation mode to perform calculation operations and output data processing results. These data processing results can be used to provide a variety of data services to the front end. Compared with the existing technology, the present invention solves the problem of being unable to connect and circulate massive amounts of data scattered across various systems in an enterprise. The enterprise-level data middle-office system constructed by the present invention relies on big data technology and data governance systems to collect, calculate, store, model, analyze, and mine massive amounts of data to form a data service system with unified standards and calibers. At the same time, the platform ensures the timeliness and efficiency of data services.

[0062] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are specifically listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] Various other advantages and benefits will become apparent to those skilled in the art upon reading the detailed description of the preferred embodiment below. The accompanying drawings are for illustration purposes only and are not to be considered as limiting the present invention. The same reference symbols are used throughout the drawings to represent the same components. In the drawings:

[0064] Figure 1A flowchart of a method for implementing an enterprise-level data middle platform system that aggregates multi-source data provided by an embodiment of the present invention;

[0065] Figure 2 A flowchart of another implementation method of an enterprise-level data middle platform system that aggregates multi-source data provided by an embodiment of the present invention;

[0066] Figure 3 This is a diagram of the architecture of the enterprise-level data middle platform system provided by an embodiment of the present invention;

[0067] Figure 4 A block diagram of a device for implementing an enterprise-level data middle platform system that aggregates multi-source data, provided by an embodiment of the present invention;

[0068] Figure 5 A block diagram of another implementation device of an enterprise-level data middle platform system that aggregates multi-source data provided by an embodiment of the present invention;

[0069] Figure 6 An electronic device for implementing an enterprise-level data middle platform system that aggregates multi-source data provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0070] Exemplary embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present invention are shown in the accompanying drawings, it should be understood that the present invention can be implemented in various forms and should not be limited by the embodiments set forth herein. Rather, these embodiments are provided to enable a more thorough understanding of the present invention and to fully convey the scope of the present invention to those skilled in the art.

[0071] The embodiment of the present invention provides a method for implementing an enterprise-level data middle platform system that aggregates multi-source data. Figure 1 As shown, the method is to pre-build an enterprise-level data middle platform system, thereby relying on big data technology and data governance system to collect, calculate, store, model, analyze, and mine massive data to form a data service system with unified standards and caliber. For this, the embodiment of the present invention provides the following specific steps:

[0072] 101. Build data assets. Data assets include various types of functional modules with data management capabilities. Functional modules are used to provide front-end data services.

[0073] It should be noted that in the embodiment of the present invention, an enterprise-level data middle-end system that collects multi-source data is pre-built. Data assets are pre-built in this system to achieve: integrated metadata management, data standard management, data quality management, master data management, data asset catalog, data lifecycle management, enterprise knowledge graph, and enterprise search engine, to open up all aspects of data asset governance and quickly meet the data asset application scenarios of various business scenarios of the enterprise and upstream and downstream supply chains.

[0074] Therefore, building data assets involves various functional modules with data management capabilities, including at least the following: data governance standards module, data quality management module, data warehouse technology module, data asset catalog module, metadata management module, master data management module, data security management module, data lifecycle management module, enterprise knowledge graph construction module, and enterprise search engine module. The following detailed explanations of these functional modules include:

[0075] Data Governance Standards Module: Data governance refers to the process of transitioning from using fragmented data to using unified master data, from having little or no organizational and process governance to comprehensive enterprise-wide data governance, and from attempting to deal with chaotic master data to achieving a well-organized master data system. During this process, standards are established in advance to complete data governance. The Data Governance Standards Module provides the tools to edit and establish these governance standards. For example, it specifies when and in what form certain data should be delivered, and which data formats correspond to different processing requirements.

[0076] Data Quality Management Module: Data quality management involves identifying, measuring, monitoring, and providing early warnings for various data quality issues that may arise at each stage of the data lifecycle, including planning, acquisition, storage, sharing, maintenance, application, and extinction. This module further enhances data quality by improving and enhancing the organization's management capabilities. This module implements data quality management functions, such as recording the data's characteristics (e.g., whether the data is monotonically increasing).

[0077] Data Warehouse Technology Module: Data warehouse technology (Extract-Transform-Load, ETL) describes the process of extracting, transforming, and loading data from a source to a destination. For example, to calculate financial indicators, such as corporate profits, the data source collected is the revenue of each project. Through the use of multiple tables, the data source is processed layer by layer through logical relationships to ultimately obtain the company's profit.

[0078] Data asset catalog module: The data asset catalog is equivalent to a summary of data assets. Data assets can be opened in the form of this catalog. By querying this data asset catalog, it is convenient to query / call the corresponding data assets.

[0079] Metadata Management Module: Metadata management includes the development of business vocabularies, the definition of data elements and entities, business rules and algorithms, and data characteristics. At its most fundamental level, it manages the collection, organization, and maintenance of business metadata. The application of technical metadata is crucial to the success of master data management and data governance projects. Simply put, metadata is the data about data, used to explain it. For example, for a data report, its metadata includes at least the field names in the report, an explanation of each field name, and the format requirements for data stored in each field.

[0080] Master Data Management Module: Master Data Management describes a set of procedures, technologies, and solutions used to create and maintain the consistency, integrity, relevance, and accuracy of business data for all stakeholders, such as users, applications, data warehouses, processes, and trading partners. For example, for an enterprise, this involves managing customer data, supplier data, self-produced product data, and the company's external transaction data.

[0081] Data security management module: performs encryption and desensitization processing on data, limits the use rights of different users, etc.

[0082] Data life cycle management module: Data life cycle management (DLM) is a policy-based approach to managing the flow of data in an information system throughout its entire life cycle: from creation and initial storage to its obsolescence and deletion.

[0083] Enterprise knowledge graph construction module: This module is used to provide functional applications for building knowledge graphs.

[0084] Enterprise search engine module: This module is used to provide the functional application of enterprise search engine.

[0085] It should be noted that this data asset is equivalent to explaining the functions of the enterprise-level data middle-end system. Based on these functions, data services can be output to the front end accordingly.

[0086] 102. Receive data processing requirements for business scenarios, which include processing structured data, semi-structured data, and unstructured data.

[0087] In the embodiment of the present invention, the data processing requirements of the business scenario can be realized by processing the collected multi-source data, and the multi-source data can specifically be: structured data, semi-structured data, and unstructured data.

[0088] 103. By searching for a mapping relationship between a pre-set business scenario and a functional module, a corresponding functional module for executing data processing requirements is obtained.

[0089] In the embodiment of the present invention, the data processing requirements of different business scenarios are different. For different data processing requirements, different functional modules included in the data assets can be selected to implement data processing.

[0090] 104. Call the calculation mode through the control function module, perform calculations and store data processing results.

[0091] After determining the computing mode that corresponds to the data processing requirement, the calculation is executed, the data processing results are output, and the data processing results are stored. Using the embodiments of the present invention, corresponding to different business scenarios, matching functional modules can be selected for data processing. For the output data processing results, matching data services can be provided, thus forming a data service system with unified standards and caliber after collecting, calculating, storing, modeling, analyzing, and mining massive amounts of data.

[0092] An embodiment of the present invention provides an implementation method for an enterprise-level data middle-office system that aggregates multi-source data. The embodiment of the present invention pre-builds data assets, which include various types of functional modules with data management capabilities. When receiving data processing requirements for a business scenario, it finds the functional modules that meet the requirements and calls the corresponding calculation mode to perform calculation operations and output data processing results. These data processing results can be used to provide a variety of data services to the front end. Compared with the existing technology, the embodiment of the present invention solves the problem of being unable to connect and circulate massive amounts of data scattered across various systems in an enterprise. The enterprise-level data middle-office system constructed by the embodiment of the present invention relies on big data technology and data governance systems to collect, calculate, store, model, analyze, and mine massive amounts of data to form a data service system with unified standards and calibers. At the same time, the platform ensures the timeliness and efficiency of data services.

[0093] In order to explain the above embodiment in more detail, the embodiment of the present invention also provides another implementation method of an enterprise-level data middle platform system that aggregates multi-source data, such as Figure 2 As shown, this method is a refinement and supplementary statement of the above embodiment, and this embodiment of the present invention provides the following specific steps:

[0094] It should be noted that, in the embodiment of the present invention, the enterprise-level data center system architecture can be as follows: Figure 3 As shown, the enterprise data middle platform system primarily consists of four components: data technology, data assets, data services, and middle platform operations and maintenance. The middle platform also connects to multiple heterogeneous data sources and supports user access to the data middle platform through a unified portal. Building on this enterprise-level data middle platform system, the following steps 201-208 are detailed.

[0095] 201. Build data assets. Data assets include various types of functional modules with data management capabilities. Functional modules are used to provide front-end data services.

[0096] In the embodiment of the present invention, the specific explanation of the functional module is as described in step 101. In addition, Figure 3 For clarification, data services include at least the following: data reporting services, business intelligence services, enterprise large screen services, API services, message bus services, and mobile application services. These data services refer to the outward service capabilities of the enterprise-level data middle platform system, allowing users authorized to access the enterprise-level data middle platform system to enjoy / experience these data services. The following is a detailed explanation of these data services, which may include the following:

[0097] Data Reporting Service: This service combines computer technology with accounting report preparation methods to create a system specifically designed for report data processing, such as outputting data reports.

[0098] Business Intelligence Services: Business Intelligence (BI), also known as business wisdom or business intelligence, refers to the use of modern data warehouse technology, online analytical processing technology, data mining, and data presentation technology to analyze data to realize business value. For example, chart analysis.

[0099] Enterprise big screen service: It is equivalent to providing a multi-dimensional display of data, such as in the form of charts and reports, and providing visual display. Specifically, it can be displayed in a variety of ways depending on the business scenario.

[0100] API service: directly outputs data services through the Application Programming Interface (API), for example, allowing users authorized to access the enterprise-level data middle-end system to directly query and obtain data.

[0101] Message bus service: The data results processed by the enterprise-level data middle-end system are pushed directly to enterprise customers via the message bus.

[0102] Mobile application service: supports mobile terminals and uses enterprise-level data middle-end system.

[0103] Furthermore, in the embodiment of the present invention, the following steps 202-203 provide in detail the specific implementation steps for establishing the mapping relationship between different business scenarios and functional modules:

[0104] 202. Add a functional overview for each functional module.

[0105] In an embodiment of the present invention, according to the detailed explanation of each functional module in step 101, it can be understood that the data processing capabilities of each functional module are different, and a single module or a combination of multiple modules can process data. Here, a functional overview can be added to each functional module, which is equivalent to the identification information of the functional module, so as to facilitate data processing requirements according to different business scenarios. Through the identification information, it can be known which functional module or modules need to be called to complete the data processing operation.

[0106] 203. Pre-customize multiple business scenarios.

[0107] In an embodiment of the present invention, different business scenarios can be pre-customized and pre-configured according to the business scenario, so that when a data processing requirement matching the pre-customized business scenario is received, the pre-configuration can be directly obtained, and the technical effect of accelerating processing efficiency can be achieved according to the pre-configuration.

[0108] 204. One or more function modules matching the business scenario are obtained by matching the data processing requirements corresponding to each business scenario with the function overview corresponding to each function module.

[0109] 205. Establish a mapping relationship between a business scenario and a functional module or a combination of multiple functional modules.

[0110] In the embodiment of the present invention, Figure 3 As shown, data assets include multiple functional modules, and a single functional module or a combination of multiple functional modules can assist in completing data processing. Therefore, for the embodiment of the present invention, different functional modules or functional module combinations can be matched to multiple pre-customized business scenarios to achieve pre-processing of establishing mapping relationships, so that when data processing requirements for different business scenarios are received, the matching functional modules can be directly called to achieve data processing.

[0111] 206. Receive data processing requirements of business scenarios, where the data processing requirements include processing structured data, semi-structured data, and unstructured data.

[0112] 207. Obtain corresponding functional modules for executing data processing requirements by searching for a mapping relationship between pre-set business scenarios and functional modules.

[0113] In the embodiment of the present invention, for the description of this step, please refer to step 103, which will not be repeated here.

[0114] 208. Call the calculation mode through the control function module, perform calculations and store data processing results.

[0115] In the embodiment of the present invention, this step is further refined and may specifically include the following:

[0116] The first step is to analyze the amount and data type of data to be processed from the data processing requirements. The data type is at least structured data, semi-structured data or unstructured data.

[0117] Combine Figure 3 To illustrate, the enterprise-level data middle platform system can process multi-source heterogeneous data, which includes at least: structured data, semi-structured data or unstructured data.

[0118] The second step is to obtain the timeliness requirements included in the data processing requirements.

[0119] The third step is to select a matching computing mode based on the data volume, data type, and timeliness requirements. The computing modes include at least: in-memory computing, data pre-aggregation, offline computing, and streaming computing.

[0120] In the embodiment of the present invention, Figure 3 As shown, the enterprise-level data middle platform system includes: data technology for collecting, calculating, storing, modeling, analyzing, and mining massive amounts of data, supporting HDFS distributed storage, MPP data storage, object storage, NoSql database, RDMS storage, supporting in-memory computing, streaming computing, offline computing, and data pre-aggregation processing, integrating machine learning and deep learning models, and relying on data assets to form data with unified standards and caliber.

[0121] Specifically, in-memory computing, data pre-aggregation, offline computing, and streaming computing are explained:

[0122] In-memory computing: When the amount of data is relatively small and timeliness is required, you can choose to perform computing operations in-memory.

[0123] Data pre-aggregation: Aggregation refers to the selection, analysis, and classification of relevant data, and finally the analysis to obtain the desired results. It is more inclined to be a calculation method, so it is based on data processing needs and combined with other calculation modes to realize calculation operations.

[0124] Offline computing: For data with large amounts of data and low timeliness requirements, offline computing is the lowest cost option.

[0125] Streaming computing: For data such as order volume and transaction amount, which require higher real-time performance than in-memory computing, streaming computing is usually chosen to perform computing operations.

[0126] It should be noted that, during the data processing process, one is not limited to selecting a single computing mode, but one or more matching computing modes may be selected according to the data processing requirements.

[0127] In addition, combined Figure 3 As shown, machine learning or deep learning can be used in the calculation process to optimize the calculation process and improve calculation efficiency.

[0128] The fourth step is to use the model to perform calculation operations according to the calculation mode, output data processing results, analyze the data volume and data type contained in the data processing results, and select a matching data storage mode based on the data volume and data type. The storage mode includes at least: Mpp data storage, object storage, distributed data storage and non-relational database storage. According to the data storage mode, the data processing results are stored.

[0129] Combine Figure 3 As shown, the enterprise-level data middle-end system also provides a variety of data storage methods in terms of data technology. Specifically, it directly matches the storage mode and directly executes storage operations based on the amount and type of data contained in the data processing results.

[0130] Further, combined Figure 3 As shown in the figure, the enterprise-level data middle platform system also includes middle platform operations and maintenance, which is used to centrally register, deploy, manage, schedule, monitor, and issue warnings for each component of the data middle platform. It also provides DevOps functions for R&D personnel and provides unified configuration and control of data sharing services. Specifically, it includes: component operations and maintenance functions, log monitoring functions, message warning functions, service registration functions, service gateway functions, and DevOps functions.

[0131] Below, the functions of the middle platform operation and maintenance are explained in detail, which may include the following:

[0132] Component operation and maintenance function: Maintain components during system operation, such as components in data technology, to ensure that each component runs effectively to avoid downtime and inability to perform calculations.

[0133] Log monitoring function: obtain operation log data during system operation, analyze the system operation status by monitoring log data, etc.

[0134] Message warning function: Analyze log data to obtain message warnings, such as: disk storage files reaching the threshold, being almost full, or writing failure, directly execute message warnings, etc.

[0135] Service registration function: manages user access rights, including backend R&D and operation and maintenance users, as well as customers who provide data services to the frontend.

[0136] Service gateway function: Monitor the service gateway in the process of providing data services to the outside world from the enterprise-level data center. For example, monitor the call usage and access volume of asset functions to avoid high concurrency or blocking, etc.

[0137] DevOps capabilities: DevOps (a combination of Development and Operations) is a collective term for a set of processes, methods, and systems that promote communication, collaboration, and integration between development (application / software engineering), technical operations, and quality assurance (QA) departments.

[0138] In the embodiment of the present invention, it is also possible to implement the following operations: during the system operation, monitoring whether the components are running normally, monitoring the system's operation logs, monitoring the system's message warnings, monitoring the system's service gateway, and the like.

[0139] Furthermore, as a response to the above Figure 1 、 Figure 2 The embodiment of the present invention provides an implementation device for an enterprise-level data middle platform system that collects multi-source data. This device embodiment corresponds to the aforementioned method embodiment. For ease of reading, this device embodiment will no longer describe the details of the aforementioned method embodiment one by one, but it should be clear that the device in this embodiment can correspond to all the contents of the aforementioned method embodiment. This device is used to process multi-source data using an enterprise-level data middle platform system, specifically as follows Figure 4 As shown, the device includes:

[0140] A construction unit 31 is used to construct data assets, wherein the data assets include multiple different types of functional modules with data management capabilities, and the functional modules are used to provide front-end data services;

[0141] A receiving unit 32 is configured to receive data processing requirements of a business scenario, wherein the data processing requirements include processing structured data, semi-structured data, and unstructured data;

[0142] An acquiring unit 33 is configured to acquire a corresponding functional module for executing the data processing requirement received by the receiving unit 32 by searching for a mapping relationship between a preset business scenario and a functional module;

[0143] The execution unit 34 is used to call the calculation mode by controlling the functional modules, perform calculations and store data processing results.

[0144] Further, such as Figure 5As shown, the execution unit 34 includes:

[0145] A parsing module 341 is configured to parse the amount and type of data to be processed from the data processing requirement, where the data type is at least structured data, semi-structured data, or unstructured data;

[0146] An acquisition module 342 is configured to acquire the timeliness requirement included in the data processing requirement;

[0147] A selection module 343 is configured to select a matching computing mode based on the amount of data parsed by the parsing module 341, the data type, and the timeliness requirement obtained by the obtaining module 342. The computing modes include at least: in-memory computing, data pre-aggregation, offline computing, and streaming computing.

[0148] An execution module 344 is configured to execute a calculation operation using the model according to the calculation mode and output a data processing result;

[0149] The parsing module 341 is further used to parse the data volume and data type contained in the data processing result;

[0150] The selection module 343 is further configured to select a matching data storage mode according to the data volume and data type, wherein the storage mode includes at least: Mpp data storage, object storage, distributed data storage, and non-relational database storage;

[0151] The storage module 345 is configured to store the data processing result according to the data storage mode selected by the selection module 343 .

[0152] Further, such as Figure 5 As shown, the device also includes:

[0153] An adding unit 35, configured to add a function overview to each function module after the data asset is constructed;

[0154] A customization unit 36, configured to pre-customize multiple business scenarios;

[0155] A matching unit 37 is configured to match the data processing requirements corresponding to the business scenarios customized by each of the customization units 36 with the functional overviews corresponding to the functional modules added by each of the adding units 35 to obtain one or more functional modules matching the business scenarios;

[0156] The establishing unit 38 is configured to establish a mapping relationship between the business scenario and a functional module or a combination of multiple functional modules.

[0157] Further, such as Figure 5 As shown, the device also includes:

[0158] The monitoring unit 39 is used to monitor whether the components are operating normally during the system operation;

[0159] The monitoring unit 39 is also used to monitor the operation log of the system;

[0160] The monitoring unit 39 is also used to monitor the message warning of the system;

[0161] The monitoring unit 39 is also used to monitor the service gateway of the system.

[0162] Further, such as Figure 5 As shown, the functional modules are at least the following: data governance standard module, data quality management module, data warehouse technology module, data asset catalog module, metadata management module, master data management module, data security management module, data life cycle management module, enterprise knowledge graph construction module and enterprise search engine module.

[0163] Further, such as Figure 5 As shown, the data services are at least the following: data report services, business intelligence services, enterprise large screen services, API services, message bus services and mobile application services.

[0164] The implementation device of the enterprise-level data middle-office system that aggregates multi-source data includes a processor and a memory. The above-mentioned construction unit, receiving unit, acquisition unit and execution unit are all stored in the memory as program units, and the processor executes the above-mentioned program units stored in the memory to implement corresponding functions.

[0165] The processor contains a kernel, which retrieves the corresponding program unit from memory. One or more kernels can be configured, and kernel parameters can be adjusted to ensure timely detection of device anomalies, avoid false alarms, and improve detection accuracy.

[0166] An embodiment of the present invention provides a storage medium having a program stored thereon. When the program is executed by a processor, the method for performing device anomaly analysis based on real-time acquisition of sensor data is implemented.

[0167] An embodiment of the present invention provides a processor, which is used to run a program, wherein when the program is run, the method for analyzing device anomalies based on real-time acquisition of sensor data is executed.

[0168] An embodiment of the present invention provides an electronic device 40, such as Figure 6As shown, the device includes at least one processor 401, and at least one memory 402 and a bus 403 connected to the processor 401; wherein the processor 401 and the memory 402 communicate with each other through the bus 403; the processor 401 is used to call the program instructions in the memory 402 to execute the above-mentioned method for device anomaly analysis based on real-time acquisition of sensor data.

[0169] The devices in this article can be servers, PCs, PADs, mobile phones, etc.

[0170] The present application also provides a computer program product which, when executed on a data processing device, is adapted to execute a program for initializing the following method steps:

[0171] A method for implementing an enterprise-level data middle-end system that aggregates multi-source data, the method comprising: constructing data assets, the data assets comprising a plurality of different types of functional modules with data management capabilities, the functional modules being used to provide front-end data services; receiving data processing requirements of business scenarios, the data processing requirements comprising processing structured data, semi-structured data, and unstructured data; obtaining corresponding functional modules for executing the data processing requirements by searching for mapping relationships between pre-defined business scenarios and functional modules; and controlling the functional modules to call calculation modes, execute calculations, and store data processing results.

[0172] Furthermore, the control function module calls the corresponding computing mode, outputs the data processing results and stores them, including: parsing the amount and data type of data to be processed from the data processing requirements, the data type is at least structured data, semi-structured data or unstructured data; obtaining the timeliness requirements contained in the data processing requirements; selecting a matching computing mode according to the data amount, the data type and the timeliness requirements, the computing mode at least includes: memory computing, data pre-aggregation, offline computing and streaming computing; according to the computing mode, using the model to perform computing operations and output data processing results; parsing the data amount and data type contained in the data processing results; selecting a matching data storage mode according to the data amount and data type, the storage mode at least includes: Mpp data storage, object storage, distributed data storage and non-relational database storage; storing the data processing results according to the data storage mode.

[0173] Furthermore, after constructing the data assets, the method also includes: adding a functional overview to each functional module; pre-customizing multiple business scenarios; obtaining one or more functional modules matching the business scenario by matching the data processing requirements corresponding to each business scenario with the functional overview corresponding to each functional module; and establishing a mapping relationship between the business scenario and a functional module or a combination of multiple functional modules.

[0174] Furthermore, the method also includes: during the operation of the system, monitoring whether the components are operating normally; and / or, monitoring the operation log of the system; and / or, monitoring the message warning of the system; and / or, monitoring the service gateway of the system.

[0175] Furthermore, the functional modules are at least the following: data governance standard module, data quality management module, data warehouse technology module, data asset catalog module, metadata management module, master data management module, data security management module, data life cycle management module, enterprise knowledge graph construction module and enterprise search engine module.

[0176] Furthermore, the data services include at least the following: data report services, business intelligence services, enterprise large screen services, API services, message bus services, and mobile application services.

[0177] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the steps in the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0178] In a typical configuration, the device includes one or more processors (CPUs), memory, and a bus. The device may also include input / output interfaces, network interfaces, and the like.

[0179] Memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM, and includes at least one memory chip. Memory is an example of a computer-readable medium.

[0180] Computer-readable media includes permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. The information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include transitory computer-readable media (transitory media), such as modulated data signals and carrier waves.

[0181] It should also be noted that the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, commodity, or apparatus that includes a series of elements includes not only those elements but also other elements not explicitly listed, or includes elements inherent to such process, method, commodity, or apparatus. In the absence of further limitations, an element defined by the phrase "comprises a ..." does not exclude the presence of other identical elements in the process, method, commodity, or apparatus that includes the element.

[0182] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the present application may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0183] The above are merely embodiments of the present application and are not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application should all be included within the scope of the claims of the present application.

Claims

1. A method for implementing an enterprise-level data middle platform system that aggregates multi-source data, characterized in that: The method comprises: Building data assets, wherein the data assets include a variety of different types of functional modules with data management capabilities, and the functional modules are used to provide front-end data services; Add a functional overview to each functional module; Pre-customize multiple business scenarios; By matching the data processing requirements corresponding to each of the business scenarios with the functional overviews corresponding to each of the functional modules, one or more functional modules matching the business scenarios are obtained; Establishing a mapping relationship between the business scenario and a functional module or a combination of multiple functional modules; Receiving data processing requirements of a business scenario, wherein the data processing requirements include processing structured data, semi-structured data, and unstructured data; By searching for a mapping relationship between a preset business scenario and a functional module, a corresponding functional module for executing the data processing requirement is obtained; The function module is controlled to call the calculation mode, perform calculation and store data processing results, including: parsing the amount and data type of data to be processed from the data processing requirements, the data type is at least structured data, semi-structured data or unstructured data; obtaining the timeliness requirements contained in the data processing requirements; selecting a matching calculation mode according to the data amount, the data type and the timeliness requirements, the calculation mode at least includes: memory calculation, data pre-aggregation, offline calculation and streaming calculation; according to the calculation mode, using the model to perform calculation operations and output data processing results; parsing the data amount and data type contained in the data processing results; according to the data amount and data type, selecting a matching data storage mode, the storage mode at least includes: Mpp data storage, object storage, distributed data storage and non-relational database storage; according to the data storage mode, storing the data processing results.

2. The method according to claim 1, characterized in that The method further comprises: During system operation, monitor whether components are operating normally; and / or Monitor the system's operation logs; and / or Monitoring the system for message alerts; and / or A service gateway that monitors the system.

3. The method according to any one of claims 1 to 2, characterized in that The functional modules are at least the following: data governance standard module, data quality management module, data warehouse technology module, data asset catalog module, metadata management module, master data management module, data security management module, data life cycle management module, enterprise knowledge graph construction module and enterprise search engine module.

4. The method according to any one of claims 1 to 2, characterized in that The data services include at least the following: data report services, business intelligence services, enterprise large screen services, API services, message bus services and mobile application services.

5. An implementation device of an enterprise-level data middle platform system that aggregates multi-source data, characterized in that: The device comprises: A construction unit, used to construct data assets, wherein the data assets include a plurality of different types of functional modules with data management capabilities, and the functional modules are used to provide front-end data services; An adding unit, used for adding a function overview to each function module after said building of data assets; Customization unit, used to pre-customize multiple business scenarios; A matching unit, configured to obtain one or more function modules matching the business scenario by matching the data processing requirements corresponding to the business scenario customized by each of the customization units with the function overview corresponding to the function module added by each of the adding units; An establishing unit, used to establish a mapping relationship between the business scenario and a functional module or a combination of multiple functional modules; A receiving unit, configured to receive data processing requirements of a business scenario, wherein the data processing requirements include processing structured data, semi-structured data, and unstructured data; An acquisition unit, configured to acquire a corresponding functional module for executing the data processing requirement received by the receiving unit by searching for a mapping relationship between a preset business scenario and a functional module; An execution unit is used to call a calculation mode by controlling the functional module, perform calculations and store data processing results, and the execution unit includes: a parsing module, used to parse the amount and data type of data to be processed from the data processing requirements, and the data type is at least structured data, semi-structured data or unstructured data; an acquisition module, used to obtain the timeliness requirements contained in the data processing requirements; a selection module, used to select a matching calculation mode according to the amount of data parsed by the parsing module, the data type and the timeliness requirements obtained by the acquisition module, and the calculation mode includes at least: memory calculation, data pre-aggregation, offline calculation and streaming calculation; an execution module, used to perform calculation operations using a model according to the calculation mode and output data processing results; the parsing module is also used to parse the amount and data type contained in the data processing results; the selection module is also used to select a matching data storage mode according to the amount and data type contained in the data processing results, and the storage mode includes at least: Mpp data storage, object storage, distributed data storage and non-relational database storage; a storage module, used to store the data processing results according to the data storage mode selected by the selection module.

6. A storage medium, characterized in that: The storage medium includes a stored program, wherein when the program is running, the device where the storage medium is located is controlled to execute the implementation method of the enterprise-level data middle platform system that aggregates multi-source data as described in any one of claims 1-4.

7. An electronic device, characterized in that: The device includes at least one processor, and at least one memory and a bus connected to the processor; Wherein, the processor and the memory communicate with each other through the bus; The processor is used to call the program instructions in the memory to execute the implementation method of the enterprise-level data middle platform system that aggregates multi-source data as described in any one of claims 1-4.