Data service implementation method, device, equipment and storage medium

Through the method based on metadata and data modeling, the process and access of original data data components are integrated, and the problem of poor data processes in the existing technology is solved, and the establishment of the data full life cycle management system and data is realized. Coordination of the processing.

CN113434481BActive Publication Date: 2024-11-01PINGAN INT SMART CITY TECH CO LTD
View PDF 5 Cites 0 Cited by

Patent Information

Application Number
CN202110703331.1
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-06-24
Publication Date
2024-11-01
Estimated Expiration
2041-06-24

AI Technical Summary

Technical Problem

The existing technology has failed to integrate various big data component technologies, resulting in the poor flow of the entire process from the original data to the data services provided by users, and it is difficult to establish a full life cycle management system for data.

Method used

By modeling the original data based on preset metadata and data modeling, connecting to the stage data layer, and performing preset data processing in the data warehouse, ultimately providing data services to users to realize data modeling and data Integration of access, data processing and data services.

Benefits of technology

It realizes the full process integration from original data to data services provided by users, supports the establishment of a full life cycle management system for data, and improves the efficiency of data processing and the overall management.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN113434481B_ABST
    Figure CN113434481B_ABST
Patent Text Reader

Abstract

The present invention discloses a method for implementing data services, including: performing preset data modeling operations on raw data based on preset metadata and preset data modeling methods to obtain modeling data; accessing the modeling data to the stage data layer according to a preset data access method; inputting the data of the stage data layer into a preset data warehouse for preset data processing operations to obtain data warehouse data; and providing data services to users based on the data warehouse data. It can be seen that the present invention can realize the integration of big data component technologies for data modeling, data access, data processing, and data services, and realize the whole process from raw data to data processing to provide data services to users, which is conducive to the establishment of a data life cycle management system. The present invention also relates to the field of blockchain technology.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of data visualization, and in particular to a method, device, computer equipment and storage medium for implementing data services. Background Art

[0002] With the advent of the information society, information technology has gradually penetrated into people's daily lives, bringing great convenience to people's daily lives. For example, current communication technology, artificial intelligence technology, Internet technology, Internet of Things technology and other information technologies have created better living conditions for humans. As information technology is widely used, a large amount of data is generated. For example, when using Internet of Things technology to monitor the natural environment, a large amount of data will be generated (such as data collected by various monitoring sensors, work data of staff on the Internet of Things monitoring platform, etc.). These data usually need to be organized in a reasonable, easy-to-use, safe and easy-to-understand way to be used as valuable data assets.

[0003] At present, more and more companies have noticed the value of digital assets and gradually realized that the premise for data to become data assets is to have a complete data standard management, data quality management, data security management, easy-to-use metadata management and continuous data value management from data generation to destruction. The data life cycle management system. Although the current big data component technologies are relatively mature, different data processing processes can be completed by different big data components (for example, big data component technology a can be used for data cleaning processes, and big data component technology b can be used for data modeling processes), however, the big data component technologies used for data processing are varied and usually lack overall management, which is not conducive to the establishment of a data life cycle management system. Summary of the invention

[0004] The technical problem to be solved by the present invention is that the existing technology has not been able to integrate various big data component technologies, and has not been able to realize the entire process of data processing from raw data to providing data services to users, which is not conducive to the establishment of a data full life cycle management system.

[0005] In order to solve the above technical problem, the first aspect of the present invention discloses a method for implementing a data service, the method comprising:

[0006] Based on the preset metadata and the preset data modeling method, a preset data modeling operation is performed on the original data to obtain modeling data;

[0007] Access the modeling data to the stage data layer according to the preset data access method;

[0008] Input the data of the stage data layer into a preset data warehouse for a preset data processing operation to obtain data warehouse data;

[0009] Data services are provided to users based on the data warehouse data.

[0010] A second aspect of the present invention discloses a device for implementing a data service, the device comprising:

[0011] A data modeling module, used to perform a preset data modeling operation on the original data based on preset metadata and a preset data modeling method to obtain modeling data;

[0012] A data access module is used to access the modeling data to the stage data layer according to a preset data access method;

[0013] A data processing module is used to input the data of the stage data layer into a preset data warehouse for performing preset data processing operations to obtain data warehouse data;

[0014] The data service module is used to provide data services to users based on the data in the data warehouse.

[0015] A third aspect of the present invention discloses a computer device, the computer device comprising:

[0016] A memory storing executable program code;

[0017] a processor connected to the memory;

[0018] The processor calls the executable program code stored in the memory to execute part or all of the steps in the method for implementing the data service disclosed in the first aspect of the present invention.

[0019] The fourth aspect of the present invention discloses a computer storage medium, which stores computer instructions. When the computer instructions are called, they are used to execute part or all of the steps in the method for implementing the data service disclosed in the first aspect of the present invention.

[0020] In the embodiment of the present invention, data modeling is first performed on the original data based on metadata and data modeling methods to obtain modeling data, and then the modeling data is accessed to the stage data layer according to a preset data access method, and then the data of the stage data layer is input into the data warehouse for processing to obtain data warehouse data, and finally data services are provided to users based on the data warehouse data, thereby realizing the integration of big data component technologies for data modeling, data access, data processing and data services, realizing the whole process from original data to data processing to provide data services to users, and facilitating the establishment of a data life cycle management system. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0022] Figure 1 It is a flowchart of a method for implementing a data service disclosed in an embodiment of the present invention;

[0023] Figure 2 It is a structural diagram of a device for implementing a data service disclosed in an embodiment of the present invention;

[0024] Figure 3 It is a structural schematic diagram of a computer device disclosed in an embodiment of the present invention;

[0025] Figure 4 It is a structural schematic diagram of a computer storage medium disclosed in an embodiment of the present invention. DETAILED DESCRIPTION

[0026] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0027] The terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish different objects rather than to describe a specific order. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, device, product or end including a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units that are not listed, or may optionally include other steps or units inherent to these processes, methods, products or ends.

[0028] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present invention. The appearance of the phrase in various places in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0029] The present invention discloses a method, device, computer equipment and storage medium for implementing data services. First, data modeling is performed on the original data based on metadata and data modeling methods to obtain modeling data, and then the modeling data is accessed to the stage data layer according to a preset data access method, and then the data of the stage data layer is input into the data warehouse for processing to obtain data warehouse data, and finally data services are provided to users based on the data warehouse data, thereby realizing the integration of big data component technologies for data modeling, data access, data processing and data services, realizing the whole process from original data to data processing for providing data services to users, and facilitating the establishment of a data life cycle management system. The following are detailed descriptions.

[0030] Embodiment 1

[0031] See also Figure 1 , Figure 1 FIG. 1 is a flow chart of a method for implementing a data service disclosed in an embodiment of the present invention. Figure 1 As shown, the implementation method of the data service may include the following operations:

[0032] 101. Perform a preset data modeling operation on the original data based on the preset metadata and the preset data modeling method to obtain modeling data.

[0033] In the above step 101, metadata is a kind of data used to describe data, which records descriptive information about the data, such as the attributes and types of the data. Effective management of metadata is an important basis for data management. The data generated when the computer system is running (i.e., raw data) is usually scattered in the computer system. For example, when a user logs in to a system at a certain terminal, a raw data record of the user's login time, IP address, account used, etc. may be generated in the computer system. These raw data usually lack unified specifications and management, and have not been directly used to generate value. For example, 192.168.1.1 is the IP address of the user's login, but in the raw data, it usually only exists in the form of a string "192.168.1.1". The computer often fails to distinguish whether this is an ordinary string or a valid IP address, so it is difficult to process it appropriately to generate value. For raw data, it is usually necessary to perform data modeling on it and convert the raw data into standardized modeling data before it can be used for subsequent data processing. Specifically, data modeling for the original data can be achieved by establishing an association relationship between metadata and original data, associating the original data through the common attributes of the original data, etc. (i.e., data modeling operations) to obtain modeled data. For example, if the metadata is "IP address" and the original data is "192.168.1.1", then an association relationship between the metadata and the original data can be established, indicating that "192.168.1.1" is an IP address. For another example, the original data includes table a and table b, wherein the fields of table a include user ID, user's mobile phone number, and user's email address, and the fields of table b include user ID and user's address. In this way, the two tables can be associated through the field user ID shared between table a and table b to obtain a total table that includes user ID, user's mobile phone number, user's email address, user's address, and fields. Among them, the metadata and data modeling methods can be pre-set by the user according to his own business experience. For example, in the environmental protection business, the preset metadata can include common indicators of environmental protection business such as COD, BOD, ammonia nitrogen, and pH value.

[0034] 102. Access the modeling data to the stage data layer according to a preset data access method.

[0035] In the above step 102, the stage data layer is a data layer set as a temporary buffer before the data warehouse, also known as the interface layer. Before the modeling data is connected to the data warehouse, it needs to pass through the stage data layer. The stage data layer has a certain degree of shielding the interference of the modeling data on the data warehouse. In practical applications, the sources of modeling data are various. For example, the modeling data can be data generated by the Internet of Things, and the modeling data can be data accessed by the database. For modeling data from different sources, different data access methods can be set to access the data, which will be described later.

[0036] 103. Input the data of the stage data layer into a preset data warehouse for a preset data processing operation to obtain data warehouse data.

[0037] In the above step 103, the data warehouse can generally include an ods layer, a dwd layer, a dws layer, and an ads layer. The flow of data can be expressed as stage layer-ods layer-dwd layer-dws layer-ads layer, wherein each layer is provided with a unique data processing flow, which is based on the data of the previous one or more data layers and then the data of the layer (i.e., data warehouse data) is obtained. Then the user can use the data of each layer according to the actual application needs. Through such a hierarchical design, the reuse of data at different levels can be achieved, and the data processing speed can be improved. For example, when the ads layer needs data, it is no longer necessary to extract data from the bottom layer, but can directly use the data of the ods layer, dwd layer, and dws layer. At the same time, through such a hierarchical design, data redundancy can be achieved, which is conducive to preventing data loss. For example, even if the data of the ods layer is lost, the data of other data layers are still saved. Among them, the ods layer is also called the source layer, which is the layer closest to the data source in the data warehouse. The amount of data that needs to be stored is the largest, and the stored data is also the most original and has not been processed too much. In the ods layer, the most basic data processing can be performed, such as distinguishing data sources, storing data by time partitions, filtering data with incorrect formats, filtering data with missing key information, etc. The dwd layer is also called the data detail layer, which can perform further data processing based on the data in the ods layer, such as removing null values, dirty data, data exceeding the limit range, changing row storage to column storage, changing the compression format, etc. The dws layer is also called the data aggregation layer, which can perform light aggregation on data, such as aggregating data by day or month. The ads layer is also called the application service layer, which can usually be directly connected to the OLAP analysis or business layer data interface. It is the highest layer in the data warehouse, and the stored data is generally result-type data that can be directly used or displayed. In the data warehouse, it is also possible to track the lineage of data, which makes it easier to quickly and accurately locate data problems and understand the scope of their harm.

[0038] 104. Provide data services to users based on the data warehouse data.

[0039] In the above step 104, after the modeling data is input into the data warehouse for data processing to obtain the data warehouse data, the user can apply for the use of various data services provided by the data warehouse according to actual business needs (e.g., providing a data sharing interface for calling a part of the data in the application service layer, exporting a part of the data in the application service layer into an Excel file, etc.).

[0040] It can be seen that the implementation Figure 1 The implementation method of the described data service first performs data modeling on the original data based on metadata and data modeling methods to obtain modeling data, then accesses the modeling data to the stage data layer according to a preset data access method, and then inputs the data of the stage data layer into the data warehouse for processing to obtain data warehouse data, and finally provides data services to users based on the data warehouse data, thereby realizing the integration of big data component technologies for data modeling, data access, data processing and data services, and realizing the whole process from original data to data processing to provide data services to users, which is conducive to the establishment of a data life cycle management system.

[0041] In an optional embodiment, the step of accessing the modeling data to the stage data layer according to a preset data access method includes:

[0042] The modeling data is accessed to the stage data layer according to the data access method corresponding to the data type of the original data.

[0043] In an optional embodiment, accessing the modeling data to the stage data layer according to the data access method corresponding to the data type of the original data includes:

[0044] When the original data is data generated by docking with a preset interface, the request URL information, request header information, request method information, and return result information of the interface are extracted from the modeling data to access the stage data layer;

[0045] When the original data is data generated by docking with a preset crawler website, the request information of the crawler website, the user name information and password information for verifying and logging in to the crawler website, and the parsing path information of the crawler website are extracted from the modeling data to access the stage data layer;

[0046] When the original data is data generated by docking with FTP, the IP address information, port information, user name information, password information, file path information, parsed file type information, and parsed class information of the FTP link are extracted from the modeling data to access the stage data layer.

[0047] In the actual use scenarios of big data, data generated by connecting with interfaces, data generated by connecting with crawler websites, and data generated by connecting with FTP are all important sources of raw data. This type of raw data usually has a common feature, that is, it contains a lot of data of low value. Therefore, for this type of raw data, when it is connected to the stage data layer, high-value data (such as public attribute information) can be extracted from it and connected to the stage data layer. Among them, when the raw data is data generated by connecting with interfaces, the public attribute information may include the request URL information, request header information, request method information, return result information, etc. of the interface. When the raw data is data generated by connecting with crawler websites, the public attribute information may include the request information of the crawler website, the user name information and password information used for verification login of the crawler website, the parsing path information of the crawler website, etc. When the raw data is data generated by connecting with FTP, the public attribute information may include the IP address information, port information, user name information, password information, file path information, parsing file type information, parsing class information, etc. of the FTP link.

[0048] It can be seen that by implementing this optional embodiment, when the original data is data generated by docking with an interface, or data generated by docking with a crawler website, or data generated by docking with FTP, public attribute information is extracted therefrom and accessed to the stage data layer, so as to better adapt to the characteristics of this type of data and help ensure the quality of the data services provided.

[0049] In an optional embodiment, the step of accessing the modeling data to the stage data layer according to the data access method corresponding to the data type of the original data further includes:

[0050] When the raw data is data generated by the Internet of Things, the modeling data is accessed to the stage data layer through a preset target communication protocol, wherein the target communication protocol includes the 212 communication protocol, the 808 communication protocol, and the 809 communication protocol. In the actual use scenario of big data, the data generated by the Internet of Things can be an important source of raw data. For example, in the environmental protection big data business, environmental protection data is usually collected through Internet of Things devices set in the environment, such as collecting the pH value of the river through a pH monitor set in the river to monitor the pollution of the river. However, the communication protocols used by the Internet of Things devices in the Internet of Things are often not consistent. For example, Internet of Things device a uses the 212 communication protocol, while Internet of Things device b may use the 808 communication protocol. Therefore, in order to ensure that the data generated by the Internet of Things can be smoothly accessed to the stage data layer, the data management platform can be set to support the access of multiple communication protocols, wherein the supported communication protocols (i.e., the target communication protocols) can include the 212 communication protocol, the 808 communication protocol, and the 809 communication protocol commonly used in the Internet of Things.

[0051] It can be seen that the implementation of this optional embodiment, by using the 212 communication protocol, the 808 communication protocol and the 809 communication protocol to access the modeling data to the stage data layer, can better adapt to the characteristics of IoT data in practical applications, which is conducive to ensuring the smooth access of IoT data and ensuring the quality and stability of the data services provided.

[0052] In an optional embodiment, the step of accessing the modeling data to the stage data layer according to the data access method corresponding to the data type of the original data further includes:

[0053] When the raw data is data generated by docking with a preset database, the modeling data is connected to the stage data layer based on the target thread, wherein the number of threads of the target thread is dynamically adjusted according to the data access speed of the modeling data connected to the stage data layer. In the actual use scenario of big data, the data generated by docking with the database is also an important source of raw data. The amount of raw data generated by docking with the database is usually large, so when these raw data are connected to the stage data layer, ensuring sufficient data access speed is conducive to ensuring that these raw data can be smoothly connected to the stage data layer. Among them, when these raw data are connected to the stage data layer, the data access speed of the data access to the stage data layer can be monitored in real time, and then the number of threads (i.e., target threads) used to perform data access tasks can be dynamically adjusted according to the data access speed to ensure sufficient data access speed. Specifically, you can use Kafka's API to monitor the data volume increment of Kafka's topic, then calculate the data access speed by calculating the data volume increment of the topic within a unit time interval, and finally compare the data access speed with the consumption capacity preset by the current consumer service. When the data access speed reaches more than 90% of the consumption capacity preset by the current consumer service, you can increase the number of target threads. When the data access speed is less than 10% of the consumption capacity preset by the current consumer service, you can reduce the number of target threads.

[0054] It can be seen that by implementing this optional embodiment, when the original data is generated by docking with a database, when the original data is accessed to the stage data layer, the data access speed of the data access to the stage data layer is monitored in real time, and the number of threads used to execute data access tasks is dynamically adjusted according to the data access speed, thereby ensuring a suitable data access speed, facilitating smooth access to database data, and improving the stability of the data services provided.

[0055] In an optional embodiment, providing data services to users based on the data warehouse data includes:

[0056] When a database export request is received, exporting the data warehouse data to a preset database;

[0057] When receiving an Excel file generation request, generating a corresponding Excel file according to the data warehouse data;

[0058] When receiving a data sharing interface generation request, generating a corresponding data sharing interface according to the data warehouse data;

[0059] When an online query request is received, an online query function for the data warehouse data is provided to the user.

[0060] In actual applications, the data services required by users are in various forms. Therefore, in order to meet the diverse needs of users, the data management platform can provide users with functions such as exporting data warehouse data to a specified database, generating corresponding excel files, generating corresponding data sharing interfaces, and online query, so as to meet the diverse needs of users. Specifically, when the user performs a first operation (such as clicking button A), a database export request can be triggered; when the user performs a second operation (such as clicking button B), an excel file generation request can be triggered; when the user performs a third operation (such as clicking button C), a data sharing interface generation request can be triggered; and when the user performs a fourth operation (such as clicking button D), an online query request can be triggered.

[0061] It can be seen that the implementation of this optional embodiment can better adapt to various practical application scenarios and meet users' diverse data needs by providing users with various forms of data services such as exporting data warehouse data to a specified database, generating corresponding Excel files, generating corresponding data sharing interfaces, and online queries.

[0062] In an optional embodiment, providing data services to users based on the data warehouse data includes:

[0063] detecting whether a data use request for target data in the data warehouse data is received;

[0064] When the data use request is received, a preset data approval authority table is queried to determine the target approval terminal identifier corresponding to the data use request, wherein the data approval authority table stores the approval terminal identifier corresponding to the data in the data warehouse data;

[0065] Sending a data approval request to the target approval terminal corresponding to the target approval terminal identifier, wherein the data approval request is used to prompt a user of the target approval terminal to approve the data use request;

[0066] detecting whether an approval instruction transmitted back by the target approval terminal is received, wherein the approval instruction is used to indicate that the data use request has been approved by a user of the target approval terminal;

[0067] When the approval instruction is received, an operation of providing data services to users based on the data warehouse data is triggered.

[0068] In this optional embodiment, the amount of data stored in the data warehouse is usually large. In order to ensure the data security of the data warehouse, before providing data services to users, an approval process for the data services to be provided can be introduced to ensure the security of the data in the data warehouse. Specifically, when a user wants to use a certain part of the data in the data warehouse (i.e., the target data), a corresponding data use request can be initiated. After receiving the data use request, the data management platform can query the data approval authority table to obtain the target approval terminal identifier corresponding to the part of the data, and then send the data approval request to the target approval terminal, and only after receiving the approval instruction returned by the target approval terminal, the data service of the part of the data is provided to the user. For example, if a user wants to use the data of table a in the data warehouse data, a data use request for table a can be initiated. After the data management platform queries the data approval authority table and obtains that the target approval terminal of table a is terminal 1, a data approval request is sent to terminal 1, and then after the approval personnel of terminal 1 approve table a, the data service of table a is provided to the user.

[0069] It can be seen that by implementing this optional embodiment, before providing data services to users based on data warehouse data, an approval process for the data services to be provided is introduced, which is conducive to ensuring the security of data in the data warehouse.

[0070] In an optional embodiment, the data processing operation is performed based on a preset target computing engine, wherein the target computing engine includes a kettle computing engine, a hive computing engine, a flink computing engine, and a spark computing engine.

[0071] In a data warehouse, when processing data, there are many choices for the computing engine (i.e., the target computing engine), such as the kettle computing engine, the hive computing engine, the flink computing engine, and the spark computing engine. Different computing engines have different characteristics, and users can choose the appropriate computing engine based on the actual application scenario. The data management platform can encapsulate these computing engines and then provide a unified visual interface for users to develop and use.

[0072] It can be seen that the implementation of this optional embodiment provides a variety of computing engines that can perform data processing for users to choose from when performing data processing in the data warehouse, thereby being able to better adapt to various practical application scenarios and provide users with better data services.

[0073] It should be noted that the embodiment of the present invention can be a data management platform for providing a unified visualization platform for data processing, and users can perform the data processing methods they want to use on the platform, such as selecting the form of data service (generating an Excel file for data, generating an interface for data, etc.), selecting a computing engine for data processing, selecting metadata, etc. In the prior art, these data processing flows are all implemented by calling big data components in the developed code interface, which is complicated to operate, and the development interface is mostly unfriendly, which is not conducive to the operation of data processing personnel, and these big data components are relatively scattered, which is not conducive to centralized management.

[0074] Optionally, it is also possible to: upload implementation information of the data service of the data service implementation method to the blockchain.

[0075] Specifically, the implementation information of the data service is obtained by running the implementation method of the data service, and is used to record the implementation of the data service, for example, data modeling time, modeling data, modeling data access time, data warehouse data, etc. Uploading the implementation information of the data service to the blockchain can ensure its security and fairness and transparency to users. Users can download the implementation information of the data service from the blockchain to verify whether the implementation information of the data service of the implementation method of the data service has been tampered with. The blockchain referred to in this example is a new application mode of computer technologies such as distributed data storage, point-to-point transmission, consensus mechanism, and encryption algorithm. Blockchain is essentially a decentralized database, a string of data blocks generated by cryptographic methods, each of which contains a batch of network transaction information, which is used to verify the validity of its information (anti-counterfeiting) and generate the next block. The blockchain can include the underlying blockchain platform, the platform product service layer, and the application service layer.

[0076] Embodiment 2

[0077] See also Figure 2 , Figure 2 Schematic diagram of a data service implementation device disclosed in an embodiment of the present invention. Figure 2 As shown, the data service implementation device may include:

[0078] The data modeling module 201 is used to perform a preset data modeling operation on the original data based on the preset metadata and the preset data modeling method to obtain modeling data;

[0079] The data access module 202 is used to access the modeling data to the stage data layer according to a preset data access method;

[0080] The data processing module 203 is used to input the data of the stage data layer into a preset data warehouse for performing preset data processing operations to obtain data warehouse data;

[0081] The data service module 204 is used to provide data services to users based on the data warehouse data.

[0082] In an optional embodiment, the data access module 202 accesses the modeling data to the stage data layer in a specific manner according to a preset data access manner:

[0083] The modeling data is accessed to the stage data layer according to the data access method corresponding to the data type of the original data.

[0084] In an optional embodiment, the data access module 202 accesses the modeling data to the stage data layer in a specific manner according to the data access method corresponding to the data type of the original data, including:

[0085] When the original data is data generated by docking with a preset interface, the request URL information, request header information, request method information, and return result information of the interface are extracted from the modeling data to access the stage data layer;

[0086] When the original data is data generated by docking with a preset crawler website, the request information of the crawler website, the user name information and password information for verifying and logging in to the crawler website, and the parsing path information of the crawler website are extracted from the modeling data to access the stage data layer;

[0087] When the original data is data generated by docking with FTP, the IP address information, port information, user name information, password information, file path information, parsed file type information, and parsed class information of the FTP link are extracted from the modeling data to access the stage data layer.

[0088] In an optional embodiment, the data access module 202 accesses the modeling data to the stage data layer in a specific manner according to the data access method corresponding to the data type of the original data, and further includes:

[0089] When the original data is data generated by the Internet of Things, the modeling data is connected to the stage data layer through a preset target communication protocol, wherein the target communication protocol includes a 212 communication protocol, an 808 communication protocol, and an 809 communication protocol.

[0090] In an optional embodiment, the data access module 202 accesses the modeling data to the stage data layer in a specific manner according to the data access method corresponding to the data type of the original data, and further includes:

[0091] When the original data is data generated by docking with a preset database, the modeling data is accessed to the stage data layer based on the target thread, wherein the number of threads of the target thread is dynamically adjusted according to the data access speed of the modeling data accessed to the stage data layer.

[0092] In an optional embodiment, the data service module 204 provides data services to users based on the data warehouse data in the following specific manners:

[0093] When a database export request is received, exporting the data warehouse data to a preset database;

[0094] When receiving an Excel file generation request, generating a corresponding Excel file according to the data warehouse data;

[0095] When receiving a data sharing interface generation request, generating a corresponding data sharing interface according to the data warehouse data;

[0096] When an online query request is received, an online query function for the data warehouse data is provided to the user.

[0097] In an optional embodiment, the data service module 204 provides data services to users based on the data warehouse data in the following specific manners:

[0098] detecting whether a data use request for target data in the data warehouse data is received;

[0099] When the data use request is received, a preset data approval authority table is queried to determine the target approval terminal identifier corresponding to the data use request, wherein the data approval authority table stores the approval terminal identifier corresponding to the data in the data warehouse data;

[0100] Sending a data approval request to the target approval terminal corresponding to the target approval terminal identifier, wherein the data approval request is used to prompt a user of the target approval terminal to approve the data use request;

[0101] detecting whether an approval instruction transmitted back by the target approval terminal is received, wherein the approval instruction is used to indicate that the data use request has been approved by a user of the target approval terminal;

[0102] When the approval instruction is received, an operation of providing data services to users based on the data warehouse data is triggered.

[0103] In an optional embodiment, the data processing operation is performed based on a preset target computing engine, wherein the target computing engine includes a kettle computing engine, a hive computing engine, a flink computing engine, and a spark computing engine.

[0104] For the specific description of the implementation device of the above data service, reference can be made to the specific description of the implementation method of the above data service. To avoid repetition, they will not be described one by one here.

[0105] Embodiment 3

[0106] See also Figure 3 , Figure 3 Schematic diagram of the structure of a computer device disclosed in an embodiment of the present invention. Figure 3 As shown, the computer device may include:

[0107] A memory 301 storing executable program codes;

[0108] A processor 302 connected to the memory 301;

[0109] The processor 302 calls the executable program code stored in the memory 301 to execute the steps in the method for implementing the data service disclosed in the first embodiment of the present invention.

[0110] Embodiment 4

[0111] See also Figure 4 The embodiment of the present invention discloses a computer storage medium 401, which stores computer instructions. When the computer instructions are called, they are used to execute the steps in the method for implementing the data service disclosed in the first embodiment of the present invention.

[0112] The device embodiments described above are only illustrative, wherein the modules described as separate components may or may not be physically separated, and the components displayed as modules may or may not be physical modules, i.e., they may be located in one place, or they may be distributed on multiple network modules. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Those of ordinary skill in the art may understand and implement it without creative work.

[0113] Through the specific description of the above embodiments, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on such an understanding, the above technical solution can be essentially or partly contributed to the prior art in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, and the storage medium includes a read-only memory (ROM), a random access memory (RAM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), a one-time programmable read-only memory (OTPROM), an electronically erasable rewritable read-only memory (EEPROM), a compact disc (CD-ROM) or other optical disc storage, magnetic disk storage, magnetic tape storage, or any other computer-readable medium that can be used to carry or store data.

[0114] Finally, it should be noted that the implementation method, device, computer equipment and storage medium of a data service disclosed in the embodiments of the present invention disclose only the preferred embodiments of the present invention, which are only used to illustrate the technical solution of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments can still be modified, or some of the technical features therein can be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for implementing a data service, characterized in that: The method comprises: Based on the preset metadata and the preset data modeling method, a preset data modeling operation is performed on the original data to obtain modeling data; Access the modeling data to the stage data layer according to the preset data access method; Input the data of the stage data layer into a preset data warehouse for a preset data processing operation to obtain data warehouse data; Providing data services to users based on the data warehouse data; The step of accessing the modeling data to the stage data layer according to a preset data access method includes: Access the modeling data to the stage data layer according to the data access method corresponding to the data type of the original data; The step of accessing the modeling data to the stage data layer according to the data access method corresponding to the data type of the original data includes: When the original data is data generated by docking with a preset interface, the request URL information, request header information, request method information, and return result information of the interface are extracted from the modeling data to access the stage data layer; When the original data is data generated by docking with a preset crawler website, the request information of the crawler website, the user name information and password information for verifying and logging in to the crawler website, and the parsing path information of the crawler website are extracted from the modeling data to access the stage data layer; When the original data is data generated by docking with FTP, the IP address information, port information, user name information, password information, file path information, parsed file type information, and parsed class information of the FTP link are extracted from the modeling data to access the stage data layer; The data warehouse includes an ods source layer, a dwd data detail layer, a dws data aggregation layer, and an ads application service layer.

2. The method for implementing data services according to claim 1, characterized in that: The step of accessing the modeling data to the stage data layer according to the data access method corresponding to the data type of the original data also includes: When the original data is data generated by the Internet of Things, the modeling data is connected to the stage data layer through a preset target communication protocol, wherein the target communication protocol includes a 212 communication protocol, an 808 communication protocol, and an 809 communication protocol.

3. The method for implementing data services according to claim 2, characterized in that: The step of accessing the modeling data to the stage data layer according to the data access method corresponding to the data type of the original data also includes: When the original data is data generated by docking with a preset database, the modeling data is accessed to the stage data layer based on the target thread, wherein the number of threads of the target thread is dynamically adjusted according to the data access speed of the modeling data accessed to the stage data layer.

4. The method for implementing data services according to any one of claims 1 to 3, characterized in that: The providing data services to users based on the data warehouse data includes: When a database export request is received, exporting the data warehouse data to a preset database; When receiving an Excel file generation request, generating a corresponding Excel file according to the data warehouse data; When receiving a data sharing interface generation request, generating a corresponding data sharing interface according to the data warehouse data; When an online query request is received, an online query function for the data warehouse data is provided to the user.

5. The method for implementing data services according to any one of claims 1 to 3, characterized in that: The providing data services to users based on the data warehouse data includes: detecting whether a data use request for target data in the data warehouse data is received; When the data use request is received, a preset data approval authority table is queried to determine the target approval terminal identifier corresponding to the data use request, wherein the data approval authority table stores the approval terminal identifier corresponding to the data in the data warehouse data; Sending a data approval request to the target approval terminal corresponding to the target approval terminal identifier, wherein the data approval request is used to prompt a user of the target approval terminal to approve the data use request; detecting whether an approval instruction transmitted back by the target approval terminal is received, wherein the approval instruction is used to indicate that the data use request has been approved by a user of the target approval terminal; When the approval instruction is received, an operation of providing data services to users based on the data warehouse data is triggered.

6. A data service implementation device, characterized in that: The device is used to implement the method for implementing a data service according to any one of claims 1 to 5, and the device includes: A data modeling module, used to perform a preset data modeling operation on the original data based on preset metadata and a preset data modeling method to obtain modeling data; A data access module is used to access the modeling data to the stage data layer according to a preset data access method; A data processing module is used to input the data of the stage data layer into a preset data warehouse for performing preset data processing operations to obtain data warehouse data; The data service module is used to provide data services to users based on the data in the data warehouse.

7. A computer device, characterized in that: The computer device comprises: A memory storing executable program code; a processor connected to the memory; The processor calls the executable program code stored in the memory to execute the method for implementing the data service according to any one of claims 1 to 5.

8. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method for implementing the data service according to any one of claims 1 to 5 is implemented.

Citation Information

Patent Citations

  • ETL architecture management method based on SCA

    CN104834742A

  • Intelligent association analysis method and system for user television and online behaviors

    CN108462888A

  • Data analysis processing method and system

    CN108766542A

  • Electric power big data processing system based on Hadoop

    CN112256782A

  • Message personalization over multiple internet messaging campaigns

    US20160012493A1