Property data processing method, device, equipment, and medium based on real-time data warehouse

Abstract

The present invention proposes a property data processing method, device, equipment, and medium based on a real-time data warehouse. The method includes: constructing a first mapping table that records a preset DW layer, a data type identifier, and a database identifier, and constructing a second mapping table that records the mapping relationship between a preset DM layer, a preset functional unit, and a preset DW layer; storing the real-time business data carrying the first identifier and the second identifier in the associated preset ODS layer, the first identifier is used to characterize the data type, and the second identifier is a database identifier; based on the first mapping table, saving the real-time business data from the preset ODS layer to the preset DW layer; each preset DM layer obtains the real-time business data from the preset DW layer based on the second mapping table, generates functional business data, and then sends it to the corresponding preset functional unit. The first identifier, the second identifier, and the first mapping table are used to realize flexible classification of data storage, and the second mapping table is used to ensure the accuracy of input and output data of each DM layer, thereby improving the reliability of data application.

Description

Technical Field

[0001] The present invention relates to the field of data warehouse technology, and in particular to a property data processing method, device, equipment, and medium based on a real-time data warehouse. Background Art

[0002] Currently, property management companies have begun to use data warehouses to support business needs and management decisions. Common data warehouses are mainly based on DataX systems for data synchronization, Hive systems for data lake storage, Spark systems for data processing, and Clickhouse systems as OLAP engines. However, the large number of existing data warehouse systems has led to complex architectures, and the DataX system is not very timely, the Hive system has a low data processing efficiency, and the Spark system has a large resource consumption, and can usually only implement offline data warehouses. However, property management requires real-time control based on various business data or monitoring data, and existing data warehouses cannot meet the requirements of property management for real-time and high efficiency of data.

[0003] In the related technologies, a real-time data warehouse based on Flink Change Data Capture (FlinkCDC) technology and Doris database is proposed. Flink CDC technology is used to realize real-time data capture, the operational data store (ODS) layer of Doris database is used to store raw data, the data warehouse (DW) layer is used to clean and organize data, and the data market (DM) layer is used to provide customized data services. Although real-time data warehouse can be realized, property management companies usually involve multiple communities or office buildings. The management data of each management entity is independent of each other. It is difficult to have a unified standard for the management data aggregated into the data warehouse. The data warehouse needs to spend a long time to classify the data, and it is difficult to ensure the accuracy of data classification, which affects the reliability of subsequent data applications. Summary of the invention

[0004] The present invention aims to solve at least one of the technical problems existing in the prior art. To this end, the present invention proposes a property data processing method, device, equipment, and medium based on a real-time data warehouse, which can improve the classification efficiency and accuracy of management data and improve the reliability of data application.

[0005] In a first aspect, an embodiment of the present invention provides a property data processing method based on a real-time data warehouse, which is applied to a real-time data warehouse system, wherein the real-time data warehouse system is connected to multiple property databases, and the real-time data warehouse system includes a FlinkCDC module, a Doris database, and multiple preset functional units; the Doris database includes multiple preset ODS layers, multiple preset DW layers, and multiple preset DM layers, and each of the preset ODS layers is associated with one of the property databases and at least one of the preset DW layers; the property data processing method based on the real-time data warehouse includes:

[0006] Constructing a first mapping table and a second mapping table, wherein the first mapping table records a mapping relationship between the preset DW layer, the data type identifier and the database identifier of the property database, and the second mapping table is used to record a mapping relationship between the preset DM layer, the preset functional unit and the preset DW layer;

[0007] Acquire multiple real-time business data from each of the property databases through the Flink CDC module, and store the real-time business data in the preset ODS layer associated with the property database, wherein the real-time business data carries a first identifier and a second identifier, the first identifier is used to characterize the data type of the real-time business data, and the second identifier is the database identifier of the property database corresponding to the real-time business data;

[0008] Based on the first mapping table and the first identifier and the second identifier carried by the real-time service data, the real-time service data is saved from the preset ODS layer to the corresponding preset DW layer;

[0009] Each of the preset DM layers obtains the real-time service data from the associated preset DW layer based on the second mapping table, generates functional service data based on the real-time service data, and sends the generated functional service data to the corresponding preset functional unit.

[0010] According to some embodiments of the present invention, obtaining a plurality of real-time business data from each of the property databases through the Flink CDC module includes:

[0011] In the Flink CDC module, a Flink CDC task is configured for each property database, wherein the Flink CDC task is configured with a third mapping table, and the third mapping table records the mapping relationship between each data table of the property database and the data type identifier;

[0012] Based on the first mapping table, binding the database identifier of the property database to the FlinkCDC task;

[0013] When any of the Flink CDC tasks obtains the real-time business data from the corresponding property database, a source data table of the real-time business data is determined;

[0014] The first identifier of the real-time business data is determined based on the third mapping table and the source data table, and the database identifier bound to the Flink CDC task is determined as the second identifier of the real-time business data.

[0015] According to some embodiments of the present invention, storing the real-time business data in the preset ODS layer associated with the property database includes:

[0016] Performing data cleaning on the real-time business data through the Flink CDC module;

[0017] Converting the real-time business data into real-time structured data based on a preset data structure, wherein the real-time structured data retains the corresponding first identifier and the second identifier;

[0018] The real-time structured data is stored in the associated preset ODS layer.

[0019] According to some embodiments of the present invention, saving the real-time business data from the preset ODS layer to the corresponding preset DW layer includes:

[0020] Based on any of the preset DW layers, determining the data type identifier recorded in the first mapping table as the third identifier, and determining the database identifier recorded in the first mapping table as the fourth identifier;

[0021] The preset ODS layer associated with the preset DW layer is determined as the target ODS layer, and multiple real-time business data are obtained from the target ODS layer based on the third identifier and the fourth identifier, wherein the first identifier of the real-time business data stored in the preset DW layer matches the third identifier, and the second identifier matches the fourth identifier.

[0022] According to some embodiments of the present invention, acquiring the real-time business data from the associated preset DW layer, generating functional business data based on the real-time business data and sending the generated functional business data to the corresponding preset functional unit includes:

[0023] Based on any target DM layer, determining at least one target DW layer and at least one target functional unit according to the second mapping table, acquiring all the real-time service data of the target DW layer,

[0024] Divide the real-time service data into at least one first set based on the second identifier, divide each of the first sets into multiple second sets based on the first identifier, and associate the corresponding first identifier and the second identifier in the second sets;

[0025] Acquire first requirement information of the target functional unit, wherein the first requirement information includes at least one fifth identifier and at least one sixth identifier, the fifth identifier is used to represent the data type, and the sixth identifier is used to ensure the database identifier;

[0026] At least one target set is determined from a plurality of the second sets based on the fifth identifier and the sixth identifier, the functional service data is generated based on the target set, and the functional service data is sent to the target functional unit.

[0027] According to some embodiments of the present invention, after generating functional service data based on the real-time service data and sending it to the corresponding preset functional unit, the method further includes:

[0028] Determine any of the preset DM layers as an updated DM layer, and generate first update information based on the updated DM layer, wherein the first update information records at least one updated functional unit and an updated DW layer;

[0029] Based on the first update information, in the second mapping table, the preset functional unit associated with the updated DM layer is replaced with the updated functional unit, and the preset DW layer associated with the updated DM layer is replaced with the updated DW layer;

[0030] The real-time service data is acquired from the updated DW layer through the updated DM layer, new functional service data is regenerated based on the acquired real-time service data, and the new functional service data is sent to the corresponding updated functional unit.

[0031] According to some embodiments of the present invention, after generating functional service data based on the real-time service data and sending it to the corresponding preset functional unit, the method further includes:

[0032] Constructing a newly added functional unit and corresponding second requirement information, wherein the second requirement information includes at least one seventh identifier and at least one eighth identifier, the seventh identifier is used to characterize a data type, and the eighth identifier is used to ensure the database identifier;

[0033] In the second mapping table, the newly added functional unit is associated with at least one of the preset DM layers, so that the associated preset DM layer outputs the functional service data to the newly added functional unit based on the second requirement information.

[0034] In the second aspect, an embodiment of the present invention provides a property data processing device based on a real-time data warehouse, comprising at least one control processor and a memory for communicating with the at least one control processor; the memory stores instructions that can be executed by the at least one control processor, and the instructions are executed by the at least one control processor so that the at least one control processor can execute the property data processing method based on the real-time data warehouse as described in the first aspect above.

[0035] In a third aspect, an embodiment of the present invention provides an electronic device, comprising a property data processing device based on a real-time data warehouse as described in the second aspect above.

[0036] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium storing computer-executable instructions, wherein the computer-executable instructions are used to execute the property data processing method based on the real-time data warehouse as described in the first aspect above.

[0037] According to the method for processing property data based on a real-time data warehouse according to an embodiment of the present invention, there are at least the following beneficial effects: constructing a first mapping table and a second mapping table, wherein the first mapping table records the mapping relationship between the preset DW layer, the data type identifier and the database identifier of the property database, and the second mapping table is used to record the mapping relationship between the preset DM layer, the preset functional unit and the preset DW layer; obtaining multiple real-time business data from each of the property databases through the Flink CDC module, and storing the real-time business data in the preset ODS layer associated with the property database, wherein the real-time business data carries a first identifier and a second identifier, the first identifier is used to characterize the data type of the real-time business data, and the second identifier is the database identifier of the property database corresponding to the real-time business data; based on the first mapping table and the first identifier and the second identifier carried by the real-time business data, the real-time business data is saved from the preset ODS layer to the corresponding preset DW layer; each of the preset DM layers obtains the real-time business data from the associated preset DW layer based on the second mapping table, generates functional business data based on the real-time business data, and sends it to the corresponding preset functional unit. According to the technical solution of the embodiment of the present invention, real-time business data can be classified by the first identifier and the second identifier, and the DW layer for storing each type of real-time business data can be determined using the first mapping table, thereby realizing flexible classification of data storage and improving data classification efficiency. The second mapping table can be used to ensure that each DM layer can obtain input data from the correct DW layer and output it to the correct functional unit, thereby ensuring the accuracy of functional business data and improving the reliability of data application. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1is a schematic diagram of a real-time data warehouse system provided by an embodiment of the present invention;

[0039] Figure 2 is a flow chart of a method for processing property data based on a real-time data warehouse provided by another embodiment of the present invention;

[0040] Figure 3 is a complete flow chart of a property data processing method based on a real-time data warehouse provided by another embodiment of the present invention;

[0041] Figure 4 It is a structural diagram of a property data processing device based on a real-time data warehouse provided in another embodiment of the present invention. DETAILED DESCRIPTION

[0042] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and cannot be understood as limiting the present invention.

[0043] In the description of the present invention, it should be understood that descriptions involving orientations, such as up, down, front, back, left, right, etc., and orientations or positional relationships indicated are based on the orientations or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be understood as a limitation on the present invention.

[0044] In the description of the present invention, "several" means one or more, "more" means more than two, "greater than", "less than", "exceed" etc. are understood as not including the number itself, and "above", "below", "within" etc. are understood as including the number itself. If there is a description of "first" or "second", it is only used for the purpose of distinguishing the technical features, and cannot be understood as indicating or implying the relative importance or implicitly indicating the number of the indicated technical features or implicitly indicating the order of the indicated technical features.

[0045] In the description of the present invention, unless otherwise clearly defined, terms such as setting, installing, connecting, etc. should be understood in a broad sense, and technicians in the relevant technical field can reasonably determine the specific meanings of the above terms in the present invention based on the specific content of the technical solution.

[0046] The embodiment of the present invention provides a property data processing method, device, equipment, and medium based on a real-time data warehouse, wherein the property data processing method based on a real-time data warehouse includes: constructing a first mapping table and a second mapping table, wherein the first mapping table records the mapping relationship between a preset DW layer, a data type identifier, and a database identifier of a property database, and the second mapping table is used to record the mapping relationship between a preset DM layer, a preset functional unit, and a preset DW layer; obtaining multiple real-time business data from each property database through a Flink CDC module, and storing the real-time business data in a preset ODS layer associated with the property database, wherein the real-time business data carries a first identifier and a second identifier, the first identifier is used to characterize the data type of the real-time business data, and the second identifier is the database identifier of the property database corresponding to the real-time business data; based on the first mapping table and the first identifier and the second identifier carried by the real-time business data, the real-time business data is saved from the preset ODS layer to the corresponding preset DW layer; each preset DM layer obtains the real-time business data from the associated preset DW layer based on the second mapping table, generates functional business data based on the real-time business data, and sends it to the corresponding preset functional unit. According to the technical solution of the embodiment of the present invention, real-time business data can be classified by the first identifier and the second identifier, and the DW layer for storing each type of real-time business data can be determined using the first mapping table, thereby realizing flexible classification of data storage and improving data classification efficiency. The second mapping table can be used to ensure that each DM layer can obtain input data from the correct DW layer and output it to the correct functional unit, thereby ensuring the accuracy of functional business data and improving the reliability of data application.

[0047] First, refer to Figure 1 , Figure 1 Schematic diagram of a real-time data warehouse system provided by an embodiment of the present invention. The real-time data warehouse system is connected to multiple property databases 10. Figure 1 The data interaction diagram of a property database 10 is shown. Each property database 10 is a data source. The property database 10 can be a MySQL database. Each property database 10 corresponds to a property entity. For example, the property system of community A corresponds to one property database 10, and the property system of community B corresponds to another property database 10. After each property database 10 generates real-time business data, it is synchronized to the Doris database 30, and the storage capacity and computing capacity of the Doris database 30 are used to realize data application.

[0048] In addition, the real-time data warehouse system includes a data transmission and computing module 20, a Doris database 30 and a data application module 40. The data transmission and computing module 20 includes a Flink CDC module 21, which uses the real-time detection function of the Flink CDC module 21 to obtain real-time business data from the property database 10 to improve the real-time performance of the data; the Doris database 30 includes multiple preset ODS layers 31, multiple preset DW layers 32 and multiple preset DM layers 33. The data application module 40 includes multiple preset functional units 41, each of which corresponds to different functions, such as Figure 1 The five preset function units 41 shown respectively realize real-time warning function, real-time recommendation function, real-time analysis function, real-time large-screen display function, real-time query function, etc. The specific function realization principle is not within the scope of discussion of this embodiment, and this embodiment is only used to realize data supply.

[0049] The following is based on the Figure 1 The real-time data warehouse system shown is used to further illustrate the technical solution of the embodiment of the present invention.

[0050] Reference Figure 2 , Figure 2 A flowchart of a method for processing property data based on a real-time data warehouse is provided in an embodiment of the present invention. The method for processing property data based on a real-time data warehouse includes but is not limited to the following steps:

[0051] S10, constructing a first mapping table and a second mapping table, wherein the first mapping table records a mapping relationship between a preset DW layer, a data type identifier, and a database identifier of a property database, and the second mapping table is used to record a mapping relationship between a preset DM layer, a preset functional unit, and a preset DW layer;

[0052] S20, obtaining multiple real-time business data from each property database through the Flink CDC module, and storing the real-time business data in a preset ODS layer associated with the property database, wherein the real-time business data carries a first identifier and a second identifier, the first identifier is used to characterize the data type of the real-time business data, and the second identifier is a database identifier of the property database corresponding to the real-time business data;

[0053] S30, based on the first mapping table and the first identifier and the second identifier carried by the real-time business data, saving the real-time business data from the preset ODS layer to the corresponding preset DW layer;

[0054] S40, each preset DM layer obtains real-time service data from the associated preset DW layer based on the second mapping table, generates functional service data based on the real-time service data, and sends the generated functional service data to the corresponding preset functional unit.

[0055] It should be noted that the real-time data warehouse system is connected to multiple property databases. In order to classify the data, this embodiment pre-sets a database identifier for each property database. The specific form of the database identifier is not limited here, as long as it can uniquely distinguish each property database.

[0056] It should be noted that the data type identifier is used to distinguish the data type of real-time business data. The data type of this embodiment can be Figure 1 The user behavior data, business data, point data, system logs, etc. shown are data generated through different functional interfaces. Each data type corresponds to a data acquisition method or data source. Different data types can be set according to actual needs, and a data type identifier is set for each data type. The specific form of the data type identifier is not limited here, as long as it can uniquely distinguish various types of data.

[0057] It should be noted that the first mapping table and the second mapping table can be constructed at any time, for example, they are constructed and saved in the real-time output system when building a real-time data warehouse system; for example, the real-time business data obtained by the real-time data warehouse system from the property database is saved in the preset ODS layer, and the first mapping table and the second mapping table do not involve the preset ODS layer. Therefore, the first mapping table and the second mapping table can also be constructed after the real-time data warehouse system starts to accept real-time business data. After construction, the preset DW layer and the preset DM layer can be triggered to perform data classification and data application.

[0058] It should be noted that the first mapping table and the second mapping table are editable tables, and therefore can be modified at any time when the real-time data warehouse system is running, thereby achieving real-time and flexible adjustment of data classification and business data conversion.

[0059] It should be noted that the first mapping table is as shown in Table 1 below. Each pair of mapping relationships is a preset DW layer, a data type identifier, and a database identifier. The data type identifier and the database identifier can be recorded in the same table unit or separately, and no further limitation is given here. As shown in Table 1 below, the preset DW layers include DW1, DW2, DW3, and DW4. The data type identifier associated with DW1 is business data, and the database identifier is database A, that is, the business data sent by database A is saved to DW1. Similarly, the buried data sent by database A is saved to DW2, and so on. Real-time business data of a data type of a property database (such as DW1, DW2, and DW4) can be saved in a preset DW layer, and multiple data types of a property database (such as DW3) can also be saved in a preset DW layer. It is sufficient to ensure that the same data type of the same database will not be saved to multiple preset DW layers. The classified storage of real-time business data of multiple property databases can be achieved through the first mapping table.

[0060]

[0061] Table 1: First mapping table

[0062] It should be noted that, according to the description of the above embodiment, each preset DW layer stores real-time business data of a data type. It is known to those skilled in the art that the role of the DM layer is to provide the data set required by the functional unit, so the DM layer is capable of classifying the data or performing related calculations. This embodiment records the preset DW layer corresponding to each preset DM layer in the second mapping table to characterize the data source of the preset DM layer, and also records the functional unit corresponding to each preset DM layer. As shown in Table 2 below, the functional units corresponding to DM1 include real-time warning and real-time large screen, so the functional business data processed by DM1 can be output to the real-time warning functional unit and the real-time large screen functional unit respectively. The second mapping table can be set according to actual needs, so as to achieve flexible data allocation, and ensure that the preset DM layer will obtain data from the correct preset DW layer to ensure the accuracy of the functional business data.

[0063]

[0064] Table 2: Second mapping table

[0065] It should be noted that the property database of this embodiment is a MySQL database. Each property database corresponds to a preset ODS layer. After the Flink CDC module obtains multiple real-time business data from a property database, it stores the real-time business data in the corresponding preset ODS layer. For example, the business data, point data, system logs, and user behavior data obtained from database A are all stored in the corresponding ODS layer A, and so on. The preset ODS layer is used to store data and does not require classification operations. It is sufficient to ensure that all real-time business data of the same property database is stored in the same preset ODS layer.

[0066] It should be noted that this embodiment adds a first identifier and a second identifier to the real-time business data. The first identifier is the data type of the real-time business data, such as any data type identifier in the above mapping table, and the second identifier is a database identifier. For a specific description, please refer to the description of the above first mapping table, which will not be repeated here. The real-time property data carries the first identifier and the second identifier, which can be compared with the data type identifier and the database identifier in the first mapping table, so that the preset DW layer can obtain the data of the set type and source from the corresponding preset ODS layer, providing an identification basis for data classification.

[0067] It should be noted that, since the first mapping table records the data type identifier and database identifier corresponding to each preset DW layer, the real-time business data can be saved to the corresponding preset DW layer according to the identifier matching method. Each preset OSD layer is associated with at least one preset DW layer, that is, the ODS layer and the DW layer are in a one-to-many relationship. The data of each property database can be stored in the same preset DW layer, or it can be separated into multiple layers. The first mapping table can be configured according to the actual classification requirements.

[0068] It should be noted that the preset DM layer and the preset DW layer can be in a many-to-many relationship. Each preset DM can obtain real-time business data from multiple preset DW layers for data processing. For example, referring to Table 1 and Table 2 above, DM1 obtains the tracking point data and business data of database A from DW1 and DW2 for related processing. This embodiment does not impose any restrictions on the specific data processing logic of the DM layer.

[0069] It should be noted that the data of the preset DM layer can be output to multiple functional units to achieve flexibility in data output. For example, as shown in Tables 1 and 2 above, DM1 processes the point-of-sale data and business data of database A, and outputs it to the real-time warning functional unit and the real-time large-screen functional unit for related applications.

[0070] In addition, in one embodiment, referring to Figure 3 Step S20 includes but is not limited to the following steps:

[0071] S21, configuring a Flink CDC task for each property database in the Flink CDC module, wherein the Flink CDC task is configured with a third mapping table, and the third mapping table records the mapping relationship between each data table of the property database and the data type identifier;

[0072] S22, based on the first mapping table, binding the database identifier of the property database to the Flink CDC task;

[0073] S23, when any Flink CDC task obtains real-time business data from the corresponding property database, the source data table of the real-time business data is determined;

[0074] S24, determining the first identifier of the real-time business data based on the third mapping table and the source data table, and determining the database identifier bound to the Flink CDC task as the second identifier of the real-time business data.

[0075] It should be noted that, in this embodiment, a Flink CDC task is configured for each property database in the Flink CDC module, and each Flink CDC task is associated with a preset ODS layer, so that the Flink CDC task continuously obtains real-time business data of the property database as a regular service.

[0076] It should be noted that this embodiment pre-configures the source data table required for data synchronization of the property database in each Flink CDC task, and only synchronizes the real-time property data of the source data table to the preset ODS layer to implement data source screening and reduce unnecessary data synchronization.

[0077] It should be noted that each source data table is usually used to transmit the same type of data, and the source data table is configured by the maintenance personnel in the Flink CDC task, so the data type of the source data table is known in advance. This embodiment records the mapping relationship between the source data table and the data type identifier through the third mapping table, so that after the Flink CDC task obtains the real-time business data of a source data table, it can automatically add a first identifier to the real-time business data, and each Flink CDC task corresponds to a business database, so the Flink CDC task can be bound to the corresponding database identifier, and then automatically add a second identifier to each real-time business data. Through the technical solution of this embodiment, after the Flink CDC task obtains the real-time business data, it can automatically add the first identifier and the second identifier to the real-time business data, thereby realizing automatic identification of the data.

[0078] In addition, in one embodiment, referring to Figure 3 Step S20 includes but is not limited to the following steps:

[0079] S25, cleans real-time business data through the Flink CDC module;

[0080] S26, converting the real-time business data into real-time structured data based on a preset data structure, wherein the real-time structured data retains the corresponding first identifier and second identifier;

[0081] S27, storing the real-time structured data into the associated preset ODS layer.

[0082] It should be noted that the Flink CDC module performs data cleaning on real-time business data, which is a technology well known to those skilled in the art. The specific process will not be described in detail here. Data cleaning can remove redundant data, reduce the amount of data in the database, and improve storage efficiency.

[0083] It should be noted that, in order to facilitate the subsequent data application of the Doris database, this embodiment converts the real-time business data into real-time structured data in a preset data structure, for example, extracts the relevant variable names and values ​​from the real-time business data, and arranges them in a preset structure, and the specific data structure is not limited here. Unless otherwise specified, the real-time business data in the subsequent embodiments can be real-time structured data before application.

[0084] In addition, in one embodiment, referring to Figure 3 Step S30 includes but is not limited to the following steps:

[0085] S31, based on any preset DW layer, determining the data type identifier recorded in the first mapping table as the third identifier, and determining the database identifier recorded in the first mapping table as the fourth identifier;

[0086] S32, determining the preset ODS layer associated with the preset DW layer as the target ODS layer, and obtaining multiple real-time business data from the target ODS layer based on the third identifier and the fourth identifier, wherein the first identifier of the real-time business data stored in the preset DW layer matches the third identifier, and the second identifier matches the fourth identifier.

[0087] It should be noted that, according to the above-mentioned first mapping table, the data type identifier associated with each preset DW layer is used as the third identifier, and the database identifier is used as the fourth identifier. After the real-time business data is stored in the preset ODS layer, the preset DW layer performs data search in the corresponding preset ODS layer according to the third identifier and the fourth identifier, and pulls the real-time business data that matches the third identifier with the first identifier and the fourth identifier with the second identifier to the preset DW layer for storage, thereby realizing data classification.

[0088] For example, referring to Table 1 above, DW1 and DW2 correspond to database A. Taking database A corresponding to ODS layer A as an example, DW1 obtains business data from ODS layer A for storage, and DW2 obtains embedded point data from ODS layer A for storage.

[0089] In addition, in one embodiment, referring to Figure 3 Step S40 includes but is not limited to the following steps:

[0090] S41, based on any target DM layer, determining at least one target DW layer and at least one target functional unit according to the second mapping table, and acquiring all real-time business data of the target DW layer,

[0091] S42, dividing the real-time service data into at least one first set based on the second identifier, dividing each first set into multiple second sets based on the first identifier, and associating the corresponding first identifier and second identifier in the second set;

[0092] S43, obtaining first requirement information of the target functional unit, wherein the first requirement information includes at least one fifth identifier and at least one sixth identifier, the fifth identifier is used to represent the data type, and the sixth identifier is used to ensure the database identifier;

[0093] S44, determining at least one target set from the plurality of second sets based on the fifth identifier and the sixth identifier, generating functional service data based on the target set, and sending the functional service data to the target functional unit.

[0094] It should be noted that each target DM layer can obtain real-time business data from different target DW layers, and multiple target DW layers can correspond to different property databases. For example, the functional business data required by the real-time large-screen functional unit is the monitoring image of each community. Therefore, the target DM layer can obtain business data from the target DW layer corresponding to each property database, and filter out the monitoring data for display; multiple target DM layers can also correspond to the same property database. For example, the functional business data required by the real-time analysis functional unit is all real-time business data of community A. The target DM layer can determine multiple preset DW layers corresponding to the same property database as the target DW layers, and the specific mapping relationship can be set in the second mapping table.

[0095] It should be noted that after the target DM layer obtains all the real-time business data of the corresponding target DW layer, it is first divided into at least one first set based on the second identifier. If the first set includes multiple second representations, such as the business data and embedded data of database A, this embodiment is further divided into multiple second sets based on the first identifier. For example, one second set includes business data and another business set includes embedded data. The first identifier and the second identifier are associated in the corresponding second set, for example, "business data" and "database A" are associated in the second set.

[0096] It should be noted that the first demand information of the target functional unit is used to characterize the required real-time business data. The target DM layer matches the target set from multiple second sets according to the fifth identifier and the sixth identifier recorded in the first demand information, and generates the functional business data of the target functional unit based on the second set. The fifth identifier corresponds to the first identifier, and the sixth identifier corresponds to the second identifier. The matching process of the target set will not be elaborated here. The data source of the target functional unit can be flexibly adjusted through the first demand information, so that the target DM layer can output different functional business data to the target functional unit.

[0097] In addition, in one embodiment, referring to Figure 3 After executing step S40, the following steps are also included but not limited to:

[0098] S51, determining any preset DM layer as an updated DM layer, and generating first update information based on the updated DM layer, wherein the first update information records at least one updated functional unit and an updated DW layer;

[0099] S52, based on the first update information, replacing the preset functional unit associated with the updated DM layer with the updated functional unit, and replacing the preset DW layer associated with the updated DM layer with the updated DW layer in the second mapping table;

[0100] S53, acquiring real-time business data from the updated DW layer by updating the DM layer, regenerating new functional business data based on the acquired real-time business data, and sending the new functional business data to the corresponding updated functional unit.

[0101] It should be noted that, according to the description of the above embodiment, different data classifications of the preset DM layer can be realized based on the second mapping table. After obtaining real-time business data carrying different identification combinations, the preset DM layer is automatically divided into multiple sets, so that the preset functional unit can obtain relevant real-time business data according to actual needs. In this embodiment, the second mapping table is set to be configurable, and an updated DM layer is determined from the preset DM layer. After generating the first update information, the content recorded in the second mapping table is updated based on the first update information, thereby realizing the data source change of the preset functional unit. After the preset functional unit is replaced by the updated functional unit in the mapping relationship of the preset DM layer, the data obtained by the preset DM layer is no longer used by the previous preset functional unit, but by the replaced updated functional unit. Similarly, after the preset DW layer corresponding to the updated DM layer is replaced by the updated DW layer, it will also trigger the updated DM layer to obtain different data from the new preset DW layer. The second mapping table is updated through the first update information, thereby adjusting the data output logic.

[0102] Exemplarily, as shown in Table 2 above, the DM layer is updated to DM2, the DW layer is updated to DW3, and the functional unit is updated to real-time query. After the first update information is generated, the DW layer corresponding to DM2 in Table 2 is updated from DW1 and DW2 to DW3, and the functional unit is replaced from real-time recommendation and real-time analysis to real-time query. DM2 obtains the business data and embedded point data of database B from DW3, and outputs it to the real-time query function for user query, thereby realizing flexible switching of functions and data sources.

[0103] In addition, in one embodiment, referring to Figure 3 After executing step S40, the following steps are also included but not limited to:

[0104] S61, constructing a newly added functional unit and corresponding second requirement information, wherein the second requirement information includes at least one seventh identifier and at least one eighth identifier, the seventh identifier is used to represent a data type, and the eighth identifier is used to ensure a database identifier;

[0105] S62: Associating the newly added functional unit with at least one preset DM layer in the second mapping table, so that the associated preset DM layer outputs functional service data to the newly added functional unit based on the second requirement information.

[0106] It should be noted that, unlike the modification function of steps S51 to S53, this embodiment adds a mapping relationship in the second mapping table to achieve flexible configuration of the functions of the real-time data warehouse system.

[0107] It should be noted that the seventh identifier corresponds to the first identifier, and the eighth identifier corresponds to the second identifier, which will not be repeated here.

[0108] It should be noted that the newly added functional unit can be a functional unit different from the current preset functional unit. For example, based on the functional units shown in Table 2, a newly added functional unit is real-time forwarding, the seventh identifier of the second demand information is "system log", and the eighth identifier is "database C", so as to achieve real-time forwarding of the system log of database B to the corresponding staff. At this time, the newly added functional unit can be associated with DM3 in the second mapping table. On the basis of DM3's original association with DW3 and real-time query, DW4 and real-time forwarding functional unit are added, so that DM3 outputs relevant functional business data to the real-time forwarding functional unit, thereby achieving flexible addition of functions.

[0109] like Figure 4 As shown, Figure 4 The present invention also provides a property data processing device based on a real-time data warehouse, including:

[0110] The processor 401 may be implemented by a general-purpose central processing unit (CPU), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application;

[0111] The memory 402 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 402 can store an operating system and other application programs. When the technical solution provided in the embodiment of this specification is implemented by software or firmware, the relevant program code is stored in the memory 402, and the processor 401 calls and executes the property data processing method based on the real-time data warehouse in the embodiment of this application;

[0112] Input / output interface 403, used to implement information input and output;

[0113] Communication interface 404, used to realize communication interaction between the device and other devices, which can be realized through wired mode (such as USB, network cable, etc.) or wireless mode (such as mobile network, WIFI, Bluetooth, etc.);

[0114] A bus 405 that transmits information between various components of the device (e.g., processor 401, memory 402, input / output interface 403, and communication interface 404);

[0115] The processor 401 , the memory 402 , the input / output interface 403 and the communication interface 404 are connected to each other in communication within the device via the bus 405 .

[0116] An embodiment of the present application also provides an electronic device, including the property data processing device based on the real-time data warehouse as described above.

[0117] An embodiment of the present application also provides a storage medium, which is a computer-readable storage medium. The storage medium stores a computer program, and when the computer program is executed by a processor, the above-mentioned property data processing method based on the real-time data warehouse is implemented.

[0118] As a non-transient computer-readable storage medium, the memory can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage devices. In some embodiments, the memory may optionally include a memory remotely arranged relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned networks include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof. The device embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and are implemented to be located in one place, or may also be distributed to multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present embodiment.

[0119] It will be appreciated by those skilled in the art that all or some of the steps and systems in the methods disclosed above may be implemented as software, firmware, hardware, and appropriate combinations thereof. Some or all physical components may be implemented as software executed by a processor, such as a central processing unit, a digital signal processor, or a microprocessor, or as hardware, or as an integrated circuit, such as an application-specific integrated circuit. Such software may be distributed on a computer-readable medium, which may include a computer storage medium (or non-transitory medium) and a communication medium (or transient medium). As known to those skilled in the art, the term computer storage medium includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disk (DVD) or other optical disk storage, magnetic cassettes, magnetic tapes, disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and can be accessed by a computer. Furthermore, it is well known to those skilled in the art that communication media typically include computer readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism, and may include any information delivery media.

[0120] The above is a specific description of the preferred implementation of the present invention, but the present invention is not limited to the above-mentioned implementation mode. Technical personnel familiar with the field can also make various equivalent deformations or substitutions under the shared conditions without violating the spirit of the present invention. These equivalent deformations or substitutions are all included in the scope defined by the claims of the present invention.

Claims

1. A property data processing method based on real-time data warehouse, characterized in that: Applied to a real-time data warehouse system, the real-time data warehouse system is connected to multiple property databases, the real-time data warehouse system includes a Flink CDC module, a Doris database, and multiple preset functional units; the Doris database includes multiple preset ODS layers, multiple preset DW layers, and multiple preset DM layers, each of the preset ODS layers is associated with one of the property databases and at least one of the preset DW layers; The real-time data warehouse-based property data processing method includes: Constructing a first mapping table and a second mapping table, wherein the first mapping table records a mapping relationship between the preset DW layer, the data type identifier and the database identifier of the property database, and the second mapping table is used to record a mapping relationship between the preset DM layer, the preset functional unit and the preset DW layer; Acquire multiple real-time business data from each of the property databases through the Flink CDC module, and store the real-time business data in the preset ODS layer associated with the property database, wherein the real-time business data carries a first identifier and a second identifier, the first identifier is used to characterize the data type of the real-time business data, and the second identifier is the database identifier of the property database corresponding to the real-time business data; Based on the first mapping table and the first identifier and the second identifier carried by the real-time service data, the real-time service data is saved from the preset ODS layer to the corresponding preset DW layer; Each of the preset DM layers obtains the real-time service data from the associated preset DW layer based on the second mapping table, generates functional service data based on the real-time service data, and sends the generated functional service data to the corresponding preset functional unit.

2. The method for processing property data based on real-time data warehouse according to claim 1, characterized in that: The Flink CDC module is used to obtain multiple real-time business data from each of the property databases, including: In the Flink CDC module, a Flink CDC task is configured for each property database, wherein the Flink CDC task is configured with a third mapping table, and the third mapping table records the mapping relationship between each data table of the property database and the data type identifier; Based on the first mapping table, binding the database identifier of the property database to the Flink CDC task; When any of the Flink CDC tasks obtains the real-time business data from the corresponding property database, a source data table of the real-time business data is determined; The first identifier of the real-time business data is determined based on the third mapping table and the source data table, and the database identifier bound to the Flink CDC task is determined as the second identifier of the real-time business data.

3. The method for processing property data based on real-time data warehouse according to claim 2 is characterized in that: Storing the real-time business data in the preset ODS layer associated with the property database includes: Performing data cleaning on the real-time business data through the Flink CDC module; Converting the real-time business data into real-time structured data based on a preset data structure, wherein the real-time structured data retains the corresponding first identifier and the second identifier; The real-time structured data is stored in the associated preset ODS layer.

4. The method for processing property data based on real-time data warehouse according to claim 1, characterized in that: Saving the real-time business data from the preset ODS layer to the corresponding preset DW layer includes: Based on any of the preset DW layers, determining the data type identifier recorded in the first mapping table as the third identifier, and determining the database identifier recorded in the first mapping table as the fourth identifier; The preset ODS layer associated with the preset DW layer is determined as the target ODS layer, and multiple real-time business data are obtained from the target ODS layer based on the third identifier and the fourth identifier, wherein the first identifier of the real-time business data stored in the preset DW layer matches the third identifier, and the second identifier matches the fourth identifier.

5. The method for processing property data based on real-time data warehouse according to claim 4 is characterized in that: Acquiring the real-time business data from the associated preset DW layer, generating functional business data based on the real-time business data and sending the generated functional business data to the corresponding preset functional unit, including: Based on any target DM layer, determining at least one target DW layer and at least one target functional unit according to the second mapping table, acquiring all the real-time service data of the target DW layer, Divide the real-time service data into at least one first set based on the second identifier, divide each of the first sets into multiple second sets based on the first identifier, and associate the corresponding first identifier and the second identifier in the second sets; Acquire first requirement information of the target functional unit, wherein the first requirement information includes at least one fifth identifier and at least one sixth identifier, the fifth identifier is used to represent the data type, and the sixth identifier is used to ensure the database identifier; At least one target set is determined from a plurality of the second sets based on the fifth identifier and the sixth identifier, the functional service data is generated based on the target set, and the functional service data is sent to the target functional unit.

6. The method for processing property data based on real-time data warehouse according to claim 5, characterized in that: After generating functional service data based on the real-time service data and sending it to the corresponding preset functional unit, the method further includes: Determine any of the preset DM layers as an updated DM layer, and generate first update information based on the updated DM layer, wherein the first update information records at least one updated functional unit and an updated DW layer; Based on the first update information, in the second mapping table, the preset functional unit associated with the updated DM layer is replaced with the updated functional unit, and the preset DW layer associated with the updated DM layer is replaced with the updated DW layer; The real-time service data is acquired from the updated DW layer through the updated DM layer, new functional service data is regenerated based on the acquired real-time service data, and the new functional service data is sent to the corresponding updated functional unit.

7. The method for processing property data based on real-time data warehouse according to claim 5, characterized in that: After generating functional service data based on the real-time service data and sending it to the corresponding preset functional unit, the method further includes: Constructing a newly added functional unit and corresponding second requirement information, wherein the second requirement information includes at least one seventh identifier and at least one eighth identifier, the seventh identifier is used to characterize a data type, and the eighth identifier is used to ensure the database identifier; In the second mapping table, the newly added functional unit is associated with at least one of the preset DM layers, so that the associated preset DM layer outputs the functional service data to the newly added functional unit based on the second requirement information.

8. A property data processing device based on real-time data warehouse, characterized in that: It includes at least one control processor and a memory for communicating with the at least one control processor; the memory stores instructions that can be executed by the at least one control processor, and the instructions are executed by the at least one control processor so that the at least one control processor can execute the property data processing method based on real-time data warehouse as described in any one of claims 1 to 7.

9. An electronic device, characterized in that: Including the property data processing device based on real-time data warehouse as described in claim 8.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are used to enable a computer to execute the property data processing method based on the real-time data warehouse as described in any one of claims 1 to 7.

Images