Data flattening processing method, device, equipment and storage medium

Abstract

The present invention relates to the field of big data technology, and discloses a data flattening processing method, device, equipment and storage medium for flattening tree structure data to improve the efficiency of tree structure data query. The data flattening processing method includes: obtaining the initial organization data of the target organization tree from the source database, and synchronizing the initial organization data to the first organization data table according to the preset character encoding format; when a request for flattening organization data is received, querying the first organization data table through the preset calculation engine to obtain the organization data to be processed; flattening the organization data to be processed by the red-black tree algorithm to obtain the target organization data; updating the target organization data to the second organization data table, and generating corresponding response data for the terminal to perform data query operations on the second organization data table according to preset business requirements. The present invention also relates to blockchain technology, and the initial organization data is stored in the blockchain.

Description

Technical Field

[0001] The present invention relates to the field of tree table query of big data technology, and in particular to a data flattening processing method, device, equipment and storage medium. Background Art

[0002] An organization tree is a tree-like data structure that expresses hierarchical relationships between data. Therefore, the organization tree uses branching relationships to define its hierarchy. Each node in the organization tree has zero or more child nodes, and each child node has only one parent node. The node with an empty parent node is the root node, and each organization tree has only one root node. Therefore, recursive queries are required when querying organization data in the organization tree.

[0003] Currently, when recursively querying the organizational data of the organizational tree in a data warehouse, a table-to-table connection query method is often used, and the purpose of recursive querying the organizational data is achieved through multiple connections. This query method is not very versatile, has poor scalability, and lacks flexibility. In addition, custom functions can also be used to recursively query organizational data, but in some business scenarios, user-defined functions, user-defined table generation functions, and user-defined table generation functions need to be combined to realize business functions. Custom functions generated in this way generally lead to slower query speeds and consume more computing resources. Summary of the Invention

[0004] The main purpose of the present invention is to solve the technical problem of low efficiency in querying tree structure data using the existing recursive query method.

[0005] To achieve the above-mentioned purpose, the first aspect of the present invention provides a data flattening processing method, comprising: obtaining the initial organization data of the target organization tree from the source database, and synchronizing the initial organization data of the target organization tree to the first organization data table in the target database according to a preset character encoding format, wherein the initial organization data is multidimensional tree structure data; when a flattening processing organization data request sent by a terminal is received, querying the first organization data table through a preset calculation engine to obtain the organization data to be processed; flattening the organization data to be processed through a red-black tree algorithm to obtain target organization data, wherein the target organization data is an organization data set with a flat data structure; updating the target organization data to the second organization data table in the target database, and generating response data corresponding to the flattening processing organization data request, and sending the response data to the terminal so that the terminal can perform data query operations on the second organization data table according to preset business requirements.

[0006] Optionally, in a first implementation method of the first aspect of the present invention, the initial organization data of the target organization tree is obtained from the source database, and the initial organization data of the target organization tree is synchronized to the first organization data table in the target database according to a preset character encoding format, and the initial organization data is multidimensional tree structure data, including: reading the connection information of the source database, the source data table name and the first organization data table name from the preset configuration information; connecting to the source database using the connection information of the source database to obtain a first connection result, and the source database is a relational database; when the first connection result is a successful connection, reading the initial organization data of the target organization tree according to the source data table name, and encoding the initial organization data of the target organization tree according to the preset character encoding format, and the initial organization data is multidimensional tree structure data; synchronizing the encoded initial organization data to the first organization data table in the target database based on the first organization data table name.

[0007] Optionally, in a second implementation method of the first aspect of the present invention, synchronizing the encoded initial organization data to the first organization data table in the target database based on the first organization data table name includes: obtaining the corresponding target table creation script according to the first organization data table name; calling the corresponding target table creation script to convert the data format of the encoded initial organization data, and writing the converted organization data into the first organization data table in the target database.

[0008] Optionally, in a third implementation of the first aspect of the present invention, when a flattened processing organization data request sent by a terminal is received, the first organization data table is queried through a preset computing engine to obtain organization data to be processed, including: when a flattened processing organization data request sent by a terminal is received, the flattened processing organization data request is parsed to obtain at least one field to be queried, and a query statement is generated based on the at least one field to be queried and the first organization data table name; the connection address, port number and link path corresponding to the target database are obtained, and the target database is connected based on the connection address, the port number and the link path to obtain a second connection result; when the second connection result is a successful connection, the query statement is executed according to the preset parsing engine to obtain query data; the query data is encapsulated according to the preset data format to obtain organization data to be processed.

[0009] Optionally, in a fourth implementation method of the first aspect of the present invention, the to-be-processed organization data is flattened by a red-black tree algorithm to obtain target organization data, and the target organization data is an organization data set with a flat data structure, including: initializing a red-black tree by a red-black tree algorithm, and initializing the root node of the red-black tree; traversing and reading each organization node from the to-be-processed organization data, and inserting each organization node into the red-black tree to obtain multiple leaf nodes; searching and obtaining the organization node path information from each leaf node to the root node respectively, and the organization node path information includes the name of each organization node and the identifier of each organization node; counting and extracting the organization node path information from each leaf node to the root node according to preset business requirements to obtain target organization data, and the target organization data is an organization data set with a flat data structure.

[0010] Optionally, in a fifth implementation method of the first aspect of the present invention, the target organization data is updated to the second organization data table in the target database, and response data corresponding to the flattened organization data request is generated, and the response data is sent to the terminal for the terminal to perform data query operations on the second organization data table according to preset business requirements, including: splitting the target organization data by a preset delimiter to obtain split organization data; mapping the split organization data and preset fields by a mapping operator to obtain mapping data; updating the mapping data to the second organization data table in the target database by the preset calculation engine to obtain an update result, wherein the second organization data table is a matrix data table including row indexes and column indexes; data encapsulating the update result to obtain response data of the flattened organization data request, and sending the response data to the terminal for the terminal to perform data query operations on the second organization data table according to preset business requirements.

[0011] Optionally, in a sixth implementation of the first aspect of the present invention, after updating the target organization data to the second organization data table in the target database and generating response data corresponding to the flattened organization data request, and sending the response data to the terminal so that the terminal can perform data query operations on the second organization data table according to preset business requirements, the data flattening processing method also includes: when a data change is detected in the first organization data table, reading the changed organization data from the first organization data table; and updating the second organization data table based on the changed organization data.

[0012] The second aspect of the present invention provides a data flattening processing device, including: an acquisition module, used to obtain the initial organization data of the target organization tree from the source database, and synchronize the initial organization data of the target organization tree to the first organization data table in the target database according to a preset character encoding format, wherein the initial organization data is multidimensional tree structure data; a query module, when receiving a flattening processing organization data request sent by a terminal, used to query the first organization data table through a preset calculation engine to obtain the organization data to be processed; a flattening module, used to flatten the organization data to be processed through a red-black tree algorithm to obtain target organization data, wherein the target organization data is an organization data set with a flat data structure; a first update module, used to update the target organization data to the second organization data table in the target database, and generate response data corresponding to the flattening processing organization data request, and send the response data to the terminal so that the terminal can perform data query operations on the second organization data table according to preset business requirements.

[0013] Optionally, in a first implementation method of the second aspect of the present invention, the acquisition module further includes: a reading unit, used to read the connection information, source data table name and first organization data table name of the source database from the preset configuration information; a connection unit, used to connect the source database using the connection information of the source database to obtain a first connection result, wherein the source database is a relational database; an encoding unit, when the first connection result is a successful connection, used to read the initial organization data of the target organization tree according to the source data table name, and encode the initial organization data of the target organization tree according to a preset character encoding format, wherein the initial organization data is multidimensional tree structure data; a synchronization unit, used to synchronize the encoded initial organization data to the first organization data table in the target database based on the first organization data table name.

[0014] Optionally, in a second implementation method of the second aspect of the present invention, the synchronization unit is specifically used to: obtain a corresponding target table creation script according to the name of the first organization data table; call the corresponding target table creation script to convert the data format of the encoded initial organization data, and write the converted organization data into the first organization data table in the target database.

[0015] Optionally, in a third implementation method of the second aspect of the present invention, the query module is specifically used to: when receiving a flattened processing organization data request sent by a terminal, parse the flattened processing organization data request to obtain at least one field to be queried, and generate a query statement based on the at least one field to be queried and the first organization data table name; obtain the connection address, port number and link path corresponding to the target database, and connect to the target database based on the connection address, the port number and the link path to obtain a second connection result; when the second connection result is a successful connection, execute the query statement according to a preset parsing engine to obtain query data; encapsulate the query data according to a preset data format to obtain organization data to be processed.

[0016] Optionally, in a fourth implementation method of the second aspect of the present invention, the flattening module is specifically used to: initialize a red-black tree through a red-black tree algorithm, and initialize the root node of the red-black tree; traverse and read each organization node from the organization data to be processed, and insert each organization node into the red-black tree to obtain multiple leaf nodes; search and obtain the organization node path information from each leaf node to the root node respectively, and the organization node path information includes the name of each organization node and the identifier of each organization node; count and extract the organization node path information from each leaf node to the root node according to preset business requirements to obtain target organization data, and the target organization data is an organization data set with a flattened data structure.

[0017] Optionally, in a fifth implementation of the second aspect of the present invention, the first update module is specifically used to: split the target organization data by a preset delimiter to obtain split organization data; map the split organization data and preset fields through a mapping operator to obtain mapping data; update the mapping data to the second organization data table in the target database through the preset calculation engine to obtain an update result, where the second organization data table is a matrix data table including row indexes and column indexes; perform data encapsulation on the update result to obtain response data of the flattened organization data request, and send the response data to the terminal so that the terminal can perform data query operations on the second organization data table according to preset business requirements.

[0018] Optionally, in a sixth implementation of the second aspect of the present invention, the data flattening processing device further includes: a reading module, which is used to read the changed organization data from the first organization data table when a data change is detected in the first organization data table; and a second update module, which is used to update the second organization data table based on the changed organization data.

[0019] The third aspect of the present invention provides a data flattening processing device, comprising: a memory and at least one processor, wherein instructions are stored in the memory, and the memory and the at least one processor are interconnected through lines; the at least one processor calls the instructions in the memory so that the data flattening processing device executes the above-mentioned data flattening processing method.

[0020] A fourth aspect of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores instructions that, when executed on a computer, enable the computer to execute the above-mentioned data flattening processing method.

[0021] In the technical solution provided by the present invention, the initial organization data of the target organization tree is obtained from the source database, and the initial organization data of the target organization tree is synchronized to the first organization data table in the target database according to the preset character encoding format, and the initial organization data is multi-dimensional tree structure data; when the flattening processing organization data request sent by the terminal is received, the first organization data table is queried through the preset calculation engine to obtain the organization data to be processed; the organization data to be processed is flattened by the red-black tree algorithm to obtain the target organization data, and the target organization data is an organization data set with a flat data structure; the target organization data is updated to the second organization data table in the target database, and response data corresponding to the flattening processing organization data request is generated, and the response data is sent to the terminal so that the terminal can perform data query operations on the second organization data table according to preset business requirements. In an embodiment of the present invention, the data processing efficiency is improved by synchronizing the organization data in the relational database to the data warehouse and calling the computing engine to read the organization data in the data warehouse; at the same time, the red-black tree algorithm is used to flatten the organization data in the data warehouse, and the organization data with the flattened data structure is stored in a new organization tree table in the data warehouse. When a request to query organization data is received, the target organization data is obtained from the new organization tree table, thereby improving the query efficiency of the tree-shaped organization data. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 A schematic diagram of an embodiment of a data flattening processing method according to an embodiment of the present invention;

[0023] Figure 2 This is a schematic diagram of another embodiment of the data flattening processing method in an embodiment of the present invention;

[0024] Figure 3 Schematic diagram of an embodiment of a data flattening processing device according to an embodiment of the present invention;

[0025] Figure 4 Schematic diagram of another embodiment of the data flattening processing device in an embodiment of the present invention;

[0026] Figure 5 Schematic diagram of an embodiment of a data flattening processing device in an embodiment of the present invention. DETAILED DESCRIPTION

[0027] Embodiments of the present invention provide a data flattening processing method, apparatus, equipment and storage medium, which are used to improve data processing efficiency by synchronizing organizational data in a relational database to a data warehouse and calling a computing engine to read the organizational data in the data warehouse; at the same time, a red-black tree algorithm is used to flatten the organizational data in the data warehouse to improve the query efficiency of tree-shaped organizational data.

[0028] The terms "first," "second," "third," "fourth," and so on (if any) in the description and claims of the present invention and in the accompanying drawings are used to distinguish similar objects and are not necessarily used to describe a particular order or precedence. It should be understood that the terms used in this manner are interchangeable where appropriate, so that the embodiments described herein can be implemented in an order other than that shown or described herein. In addition, the terms "including" or "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product, or apparatus that includes a series of steps or elements is not necessarily limited to those steps or elements expressly listed, but may include other steps or elements not expressly listed or inherent to such process, method, product, or apparatus.

[0029] For ease of understanding, the specific process of the embodiment of the present invention is described below. Figure 1 , an embodiment of the data flattening processing method in an embodiment of the present invention includes:

[0030] 101. Obtain the initial organization data of the target organization tree from the source database, and synchronize the initial organization data of the target organization tree to the first organization data table in the target database according to a preset character encoding format. The initial organization data is multi-dimensional tree structure data.

[0031] Among them, the source database is a relational database, such as MySQL, and the target database is a data warehouse tool Hive. Furthermore, the server synchronizes the initial organization data from the source database to the first organization data table in the target database through the data synchronization tool Sqoop. Specifically, the server obtains the initial organization data of the target organization tree from the source database, and character encodes the initial organization data according to the preset character encoding format, and then generates a data synchronization script through Sqoop to synchronize the encoded initial organization data to the first organization data table in the target database. The data synchronization script includes a table creation statement, which is used to create the first organization data table. It should be emphasized that in order to further ensure the privacy and security of the above-mentioned initial organization data, the above-mentioned initial organization data can also be stored in a node of a blockchain.

[0032] It should be noted that the synchronization frequency of the data synchronization script can be configured through the preset scheduling platform, that is, the synchronization frequency of the script file can be controlled. The server uses the preset character encoding format for character encoding, which can effectively avoid or control the problem of garbled Chinese data.

[0033] It is understandable that the execution subject of the present invention may be a data flattening processing device, or a terminal or a server, which is not limited here. The embodiment of the present invention is described by taking a server as the execution subject as an example.

[0034] 102. When a request for flattening the organization data is received from the terminal, the first organization data table is queried through a preset calculation engine to obtain the organization data to be processed.

[0035] Specifically, before receiving the flattened processing mechanism data request sent by the terminal, the server receives the pre-packaged preset computing engine spark uploaded by the target terminal, and sends the pre-packaged preset computing engine to the preset computing engine cluster. Therefore, when receiving the flattened processing mechanism data request sent by the terminal, the server connects to the target database through the preset computing engine, and when detecting that the connection is successful, the server queries the first mechanism data table to obtain the mechanism data to be processed. Among them, the preset computing engine spark is a fast and universal cluster computing platform, and is also a general-purpose memory parallel computing framework. It can be used to build low-latency data analysis applications, support more computing modes, including interactive queries and data stream processing, and can also perform data calculations in memory.

[0036] 103. The target organization data is flattened by a red-black tree algorithm to obtain target organization data. The target organization data is an organization data set with a flat data structure.

[0037] The red-black tree algorithm is essentially a binary search tree, but with an additional storage bit at each node representing the node's color, which can be either red or black. Flattening involves converting multidimensional organization data to be processed into one-dimensional target organization data. For example, the name of a parent organization node, which can only be found through a recursive query, is converted to a single query based on the identifier of a child organization node, eliminating the need for recursive processing. This solves the problem of overlapping hierarchies in the target organization tree and inefficient organizational operations.

[0038] Furthermore, the server searches and obtains the data set from each leaf organization node to the root organization node from the organization data to be processed according to the red-black tree algorithm to obtain the target organization data. The target organization data includes the set of leaf organization node identifiers, parent organization node identifiers and root organization nodes. Each organization node identifier uses a unique identifier to identify the corresponding organization node in the tree structure data.

[0039] 104. Update the target organization data to the second organization data table in the target database, generate response data corresponding to the flattened organization data request, and send the response data to the terminal so that the terminal can perform data query operations on the second organization data table according to preset business requirements.

[0040] Specifically, the server updates the target organization data to the second organization data table in the target database according to the field mapping order. When the server detects a successful update, it generates response data corresponding to the request for flattened organization data and sends the response data to the terminal. The second organization data table is used to store the flattened dataset. Therefore, when the terminal sends a query for the data of each organization in the target organization tree, the server directly queries the second organization data table instead of recursively querying the first organization data table, thereby improving the efficiency of querying the target organization data. For example, when calculating premiums for a second-level organization node, it is necessary to query and calculate the premiums of all subordinate organizations below the second-level organization node. The premiums of all subordinate organizations include third-level and fourth-level organizations, etc., which are not specifically limited here. When using join query operations in a data warehouse, join queries need to be performed based on the existing number of sub-organizations. Moreover, when adding new sub-organizations, the original query script also needs to be modified accordingly. Regularly synchronizing the new sub-organization data and triggering conversion instructions to flatten the new sub-organization data not only improves business query efficiency but also improves code development and maintenance efficiency.

[0041] In an embodiment of the present invention, the data processing efficiency is improved by synchronizing the organization data in the relational database to the data warehouse and calling the computing engine to read the organization data in the data warehouse; at the same time, the red-black tree algorithm is used to flatten the organization data in the data warehouse, and the organization data with the flattened data structure is stored in a new organization tree table in the data warehouse. When a request to query organization data is received, the target organization data is obtained from the new organization tree table, thereby improving the query efficiency of the tree-shaped organization data.

[0042] See also Figure 2 Another embodiment of the data flattening processing method in the embodiment of the present invention includes:

[0043] 201. Read the source database connection information, source data table name, and first organization data table name from the preset configuration information.

[0044] The connection information of the source database includes the source database connection address, source database name, source database user name, and source database password. For example, the source database connection address is 1433, the source database name is database_name, the source database user name is root, and the source database password is root123. It can be an empty value and is not limited here. Furthermore, the server can set multiple scheduled tasks based on the number of source data tables in the source database, and read the preset configuration information based on the multiple scheduled tasks to obtain the corresponding target configuration information. The corresponding target configuration information includes the source database connection information, the source data table name, and the first organization data table name, wherein the source data table is used to store the initial organization data of the target organization tree.

[0045] 202. Connect to a source database using the source database connection information to obtain a first connection result, where the source database is a relational database.

[0046] Specifically, the server connects to the database based on the source database connection address, source database name, source database user name, and source database password to obtain a first connection result. The server determines whether the first connection result is a successful connection. Further, the server determines whether the first connection result is a target value. If the first connection result is not the target value, the server determines that the first connection result is a connection failure. If the first connection result is the target value, the server determines that the first connection result is a successful connection. For example, if the first connection result is true or 1, the server determines that the first connection result is a successful connection; if the first connection result is 0, the server determines that the first connection result is a connection failure.

[0047] 203. When the first connection result is successful, the initial organization data of the target organization tree is read according to the source data table name, and the initial organization data of the target organization tree is encoded according to the preset character encoding format. The initial organization data is multi-dimensional tree structure data.

[0048] The preset character encoding format is UTF-8. If the initial organization data includes Chinese characters, the server uses the UTF-8 encoding format to export the initial organization data of the target organization tree from the source data table. This not only unifies the character encoding format of the initial organization data but also prevents the initial organization data from containing garbled Chinese characters. Specifically, the server generates a query statement based on the source data table name; the server executes the query statement to obtain the initial organization data of the target organization tree. The initial organization data is a multidimensional tree structure. Furthermore, the server uses the UTF-8 encoding format to encode the initial organization data of the target organization tree.

[0049] It can be understood that the target organization tree is a tree-shaped data structure. The target organization tree uses branch relationships to define a hierarchical structure. The organization data in the target organization tree includes a root node, a parent node, and a child node.

[0050] 204. Synchronize the encoded initial organization data to the first organization data table in the target database based on the first organization data table name.

[0051] Specifically, the server obtains a corresponding target table creation script according to the name of the first organization data table, where the target table creation script is stored in a preset file directory, for example, / var / www / html / dir / a.shell. The server calls the corresponding target table creation script to convert the data format of the encoded initial organization data and writes the converted organization data into the first organization data table in the target database. The target table creation script includes the target database name and preset table creation statements. The target database is a data warehouse. The target table creation script is used to instruct data synchronization of the encoded initial organization data and write it into the first organization data table.

[0052] Furthermore, after the server flattens the organization data of the source data table in the source database, it cleans the collected initial organization data in the target database, processes the cleaned organization data according to business needs, and puts the processed data into storage.

[0053] It is understandable that the preset table creation statement is also used to check whether the first organization data table already exists; if the first organization data table already exists, the server converts the encoded initial organization data and writes it into the first organization data table; if the first organization data table does not exist, after creating the first organization data table, the server writes the encoded initial organization data into the first organization data table, for example,

[0054] create table if not exists org_bank

[0055] ... / / Omit the included fields

[0056] Among them, org_bank is the name of the first institution data table, which is consistent with the source data table name.

[0057] 205. When a request for flattening the organization data sent by the terminal is received, the first organization data table is queried through a preset calculation engine to obtain the organization data to be processed.

[0058] Specifically, first, when receiving a flattened processing organization data request sent by a terminal, the server parses the flattened processing organization data request, obtains at least one field to be queried, and generates a query statement based on the at least one field to be queried and the first organization data table name. For example, the query statement is as follows:

[0059] select field1, field2 from first institution data table name;

[0060] Secondly, the server obtains the connection address, port number and link path corresponding to the target database, and connects to the target database based on the connection address, port number and link path to obtain a second connection result, for example, hdfs: / / connection address of target database:port number / specific path, where the connection address of the target database is 192.168.40.51 and the port number is 9083. The second connection result includes connection success and connection failure.

[0061] Then, when the second connection result is a successful connection, the server executes the query statement according to the preset parsing engine to obtain the query data, wherein the preset parsing engine includes spark SQL, spark SQL can use structured query language SQL statements on the preset computing engine, and its data source includes elastic distributed data sets and external data sources, such as data warehouses.

[0062] Finally, the server encapsulates the query data according to a preset data format to obtain the organization data to be processed, wherein the preset data format is used to indicate the conversion of the data format of the organization data to be processed according to a preset field mapping relationship, wherein the preset field mapping relationship can be consistent with the field name of the first organization data table, or it can be inconsistent, and the specific details are not limited here. Furthermore, the flattened processing organization data request sent by the target terminal also includes the target user's operation authority. The server obtains the target user's operation authority and determines whether the target user has the operation authority based on the preset authority and the target user's operation authority.

[0063] Before the pre-configured computing engine can connect to the data warehouse, you need to copy the configuration files in the data warehouse, such as the hive-site.xml file, to the pre-configured computing engine's directory, such as the conf directory. This allows you to use the configuration files to find the data warehouse metadata and data storage location.

[0064] 206. Flatten the organization data to be processed using a red-black tree algorithm to obtain target organization data, where the target organization data is an organization data set with a flattened data structure.

[0065] Specifically, the server initializes a red-black tree through a red-black tree algorithm and initializes the root node of the red-black tree; the server traverses and reads each organization node from the organization data to be processed, and inserts each organization node into the red-black tree to obtain multiple leaf nodes. Furthermore, the server marks the root node as black, and in the process of inserting each organization node into the red-black tree, performs color change and rotation processing on each organization in the red-black tree. The color change includes red and black, and the rotation processing includes left and right branch conversion of each node; the server respectively searches for and obtains the organization node path information from each leaf node to the root node. The organization node path information includes the name of each organization node and the identifier of each organization node. Among them, the organization node path information is used to combine organization nodes with hierarchical relationships; the server counts and extracts the organization node path information from each leaf node to the root node according to preset business requirements to obtain target organization data. The target organization data is an organization data set with a flat data structure. Optionally, the target organization data is a mapping MAP set data, and the target organization data includes the parent organization node identifier, the child organization node identifier, the parent organization node name, and the child organization node name.

[0066] It should be noted that the data part of the red-black tree algorithm encapsulates a generic class T <e>When calling the algorithm interface, after passing in the specific class, the class properties, methods, constructors, all fields, and the specific generic type of the previous level are obtained through the function of the JAVA reflection mechanism.

[0067] 207. Update the target organization data to the second organization data table in the target database, generate response data corresponding to the flattened organization data request, and send the response data to the terminal so that the terminal can perform data query operations on the second organization data table according to preset business requirements.

[0068] Specifically, the server splits the target organization data by a preset delimiter to obtain split organization data, wherein the target organization data is serialized data separated by a preset delimiter, and the preset delimiter includes a comma; the server maps the split organization data and the preset fields through a mapping operator to obtain mapping data, wherein the preset fields include the target organization node identifier, the target organization node name and the parent organization node identifier. Further, the server maps the split organization data in the order of the preset fields through a mapping operator to obtain mapping data; the server updates the mapping data to the second organization data table in the target database through a preset calculation engine to obtain an update result, and the second organization data table is a matrix data table including a row index and a column index; the server encapsulates the update result to obtain response data of the flattened organization data request, and sends the response data to the terminal so that the terminal can perform data query operations on the second organization data table according to preset business requirements. Furthermore, when the server receives a request to query institution data, the server directly retrieves the second institution data table to obtain the query result. For example, the request to query institution data includes a target institution node identifier of 101112 and a target institution node name of a first-level branch of Bank A. The preset business requirement is to obtain the root institution node name. However, a query in the first structure data table can only obtain the upper-level institution node identifier corresponding to the target institution node. However, the second institution data table can directly provide the root institution node name corresponding to the target institution node. According to the target institution node identifier 101112, the root institution node name can be queried to obtain Bank A.

[0069] Optionally, the server synchronizes the initial organization data in the initial database to the third database in real time through a preset synchronization tool, wherein the preset synchronization tool includes Oracle's synchronization tool OGG, and the third database includes a Kafka distributed data source; reads data from the Kafka distributed data source through the preset computing engine data processing framework sparkStreaming, and converts the data into an elastic distributed data set; the server preprocesses the data in the elastic distributed data set to obtain a distributed data set, and stores the distributed data set in an in-memory database, and the in-memory database includes redis; the server generates a target interface based on the distributed data set, and updates the distributed data set to the target database, and the target interface is used to indicate the preset scheduling task to schedule the JAR script in real time.

[0070] Optionally, when a data change is detected in the first organization data table, the server reads the changed organization data from the first organization data table; and the server updates the second organization data table based on the changed organization data.

[0071] In an embodiment of the present invention, the data processing efficiency is improved by synchronizing the organization data in the relational database to the data warehouse and calling the computing engine to read the organization data in the data warehouse; at the same time, the red-black tree algorithm is used to flatten the organization data in the data warehouse, and the organization data with the flattened data structure is stored in a new organization tree table in the data warehouse. When a request to query organization data is received, the target organization data is obtained from the new organization tree table, thereby improving the query efficiency of the tree-shaped organization data.

[0072] The above describes the data flattening processing method in the embodiment of the present invention. The following describes the data flattening processing device in the embodiment of the present invention. Figure 3 , an embodiment of the data flattening processing device in the embodiment of the present invention includes:

[0073] An acquisition module 301 is configured to acquire initial organization data of a target organization tree from a source database and synchronize the initial organization data of the target organization tree to a first organization data table in the target database according to a preset character encoding format. The initial organization data is multi-dimensional tree structure data.

[0074] The query module 302 is used to query the first organization data table through the preset calculation engine to obtain the organization data to be processed when receiving the flattened processing organization data request sent by the terminal;

[0075] A flattening module 303 is used to flatten the to-be-processed organization data using a red-black tree algorithm to obtain target organization data, where the target organization data is an organization data set with a flattened data structure;

[0076] The first update module 304 is used to update the target organization data to the second organization data table in the target database, and generate response data corresponding to the flattened organization data request, and send the response data to the terminal so that the terminal can perform data query operations on the second organization data table according to preset business requirements.

[0077] In an embodiment of the present invention, the data processing efficiency is improved by synchronizing the organization data in the relational database to the data warehouse and calling the computing engine to read the organization data in the data warehouse; at the same time, the red-black tree algorithm is used to flatten the organization data in the data warehouse, and the organization data with the flattened data structure is stored in a new organization tree table in the data warehouse. When a request to query organization data is received, the target organization data is obtained from the new organization tree table, thereby improving the query efficiency of the tree-shaped organization data.

[0078] See also Figure 4 Another embodiment of the data flattening processing device in the embodiment of the present invention includes:

[0079] An acquisition module 301 is configured to acquire initial organization data of a target organization tree from a source database and synchronize the initial organization data of the target organization tree to a first organization data table in the target database according to a preset character encoding format. The initial organization data is multi-dimensional tree structure data.

[0080] The query module 302 is used to query the first organization data table through the preset calculation engine to obtain the organization data to be processed when receiving the flattened processing organization data request sent by the terminal;

[0081] A flattening module 303 is used to flatten the to-be-processed organization data using a red-black tree algorithm to obtain target organization data, where the target organization data is an organization data set with a flattened data structure;

[0082] The first update module 304 is used to update the target organization data to the second organization data table in the target database, and generate response data corresponding to the flattened organization data request, and send the response data to the terminal so that the terminal can perform data query operations on the second organization data table according to preset business requirements.

[0083] It should be emphasized that in order to further ensure the privacy and security of the above-mentioned initial institutional data, the above-mentioned initial institutional data can also be stored in a node of a blockchain.

[0084] Optionally, the approval module 301 may also be specifically configured to:

[0085] The reading unit 3011 is used to read the connection information of the source database, the source data table name and the first organization data table name from the preset configuration information;

[0086] A connection unit 3012 is configured to connect to a source database using the connection information of the source database to obtain a first connection result, where the source database is a relational database;

[0087] The encoding unit 3013 is used to read the initial organization data of the target organization tree according to the source data table name and encode the initial organization data of the target organization tree according to the preset character encoding format when the first connection result is successful. The initial organization data is multi-dimensional tree structure data.

[0088] The synchronization unit 3014 is used to synchronize the encoded initial organization data to the first organization data table in the target database based on the first organization data table name.

[0089] Optionally, the synchronization unit 3014 may also be specifically configured to:

[0090] Obtain the corresponding target table creation script according to the first organization's data table name;

[0091] The corresponding target table creation script is called to convert the data format of the encoded initial organization data, and the converted organization data is written into the first organization data table in the target database.

[0092] Optionally, the query module 302 may also be specifically configured to:

[0093] When receiving a flattened processing organization data request sent by a terminal, parsing the flattened processing organization data request to obtain at least one field to be queried, and generating a query statement based on the at least one field to be queried and the first organization data table name;

[0094] Obtaining a connection address, port number, and link path corresponding to a target database, and connecting to the target database based on the connection address, port number, and link path to obtain a second connection result;

[0095] When the second connection result is a successful connection, executing the query statement according to the preset parsing engine to obtain query data;

[0096] The query data is encapsulated according to the preset data format to obtain the organization data to be processed.

[0097] Optionally, the flattening module 303 may also be specifically used for:

[0098] Initialize the red-black tree through the red-black tree algorithm and initialize the root node of the red-black tree;

[0099] Traverse and read each organization node from the organization data to be processed, and insert each organization node into the red-black tree to obtain multiple leaf nodes;

[0100] Search and obtain the organization node path information from each leaf node to the root node respectively, the organization node path information includes the name of each organization node and the identifier of each organization node;

[0101] According to the preset business requirements, the organization node path information from each leaf node to the root node is counted and extracted to obtain the target organization data, which is an organization data set with a flat data structure.

[0102] Optionally, the first updating module 304 may also be specifically configured to:

[0103] Splitting the target organization data using a preset separator to obtain split organization data;

[0104] Mapping the segmented organization data and preset fields using a mapping operator to obtain mapping data;

[0105] Updating the mapping data to a second organization data table in the target database through a preset calculation engine to obtain an update result, where the second organization data table is a matrix data table including row indexes and column indexes;

[0106] The update result is data-encapsulated to obtain response data of the flattened processing organization data request, and the response data is sent to the terminal so that the terminal can perform data query operations on the second organization data table according to preset business requirements.

[0107] Optionally, the data flattening processing device further includes:

[0108] The reading module 305 is used to read the changed organization data from the first organization data table when a data change is detected in the first organization data table;

[0109] The second updating module 306 is configured to update the second organization data table based on the changed organization data.

[0110] In an embodiment of the present invention, the data processing efficiency is improved by synchronizing the organization data in the relational database to the data warehouse and calling the computing engine to read the organization data in the data warehouse; at the same time, the red-black tree algorithm is used to flatten the organization data in the data warehouse, and the organization data with the flattened data structure is stored in a new organization tree table in the data warehouse. When a request to query organization data is received, the target organization data is obtained from the new organization tree table, thereby improving the query efficiency of the tree-shaped organization data.

[0111] above Figure 3 and Figure 4 The data flattening processing apparatus in the embodiment of the present invention is described in detail from the perspective of modular functional entities. The data flattening processing device in the embodiment of the present invention is described in detail from the perspective of hardware processing.

[0112] Figure 5 The diagram is a schematic diagram of the structure of a data flattening device provided in an embodiment of the present invention. The data flattening device 500 may vary significantly depending on configuration or performance. It may include one or more central processing units (CPUs) 510 (e.g., one or more processors), a memory 520, and one or more storage media 530 (e.g., one or more mass storage devices) storing application programs 533 or data 532. The memory 520 and storage media 530 may be either transient or persistent storage. The program stored in the storage medium 530 may include one or more modules (not shown), each of which may include a series of instructions operating on the data flattening device 500. Furthermore, the processor 510 may be configured to communicate with the storage medium 530, executing the series of instructions stored in the storage medium 530 on the data flattening device 500.

[0113] The data flattening processing device 500 may further include one or more power supplies 540, one or more wired or wireless network interfaces 550, one or more input and output interfaces 560, and / or one or more operating systems 531, such as Windows Server, Mac OS X, Unix, Linux, FreeBSD, etc. It will be understood by those skilled in the art that Figure 5 The structure of the data flattening processing device shown does not constitute a limitation on the data flattening processing device, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0114] The present invention also provides a computer-readable storage medium, which may be a non-volatile computer-readable storage medium or a volatile computer-readable storage medium. Instructions are stored in the computer-readable storage medium. When the instructions are executed on a computer, the computer executes the steps of the data flattening processing method.

[0115] Blockchain, as used in this article, refers to a novel application model for computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanisms, and encryption algorithms. Blockchain is essentially a decentralized database, a series of data blocks generated using cryptographic methods. Each block contains information about a batch of online transactions, used to verify the validity of this information (to prevent counterfeiting) and generate the next block. Blockchain can include the underlying blockchain platform, the platform product service layer, and the application service layer.

[0116] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0117] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the portion that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes various media that can store program code, such as a USB flash drive, a mobile hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0118] As described above, the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that the technical solutions described in the above embodiments can still be modified, or some of the technical features thereof can be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.< / e>

Claims

1. A data flattening processing method, characterized in that: The data flattening processing method includes: Acquire initial organization data of a target organization tree from a source database, and synchronize the initial organization data of the target organization tree to a first organization data table in the target database according to a preset character encoding format, wherein the initial organization data is multidimensional tree structure data; When receiving a request for flattening the organization data sent by the terminal, querying the first organization data table through the preset calculation engine to obtain the organization data to be processed; Flattening the to-be-processed organization data using a red-black tree algorithm to obtain target organization data, wherein the target organization data is an organization data set with a flattened data structure; Updating the target organization data to a second organization data table in the target database, generating response data corresponding to the flattened organization data request, and sending the response data to the terminal so that the terminal can perform a data query operation on the second organization data table according to preset business requirements; The target organization data is obtained by flattening the organization data to be processed using a red-black tree algorithm. The target organization data is an organization data set with a flat data structure, including: Initialize the red-black tree through the red-black tree algorithm and initialize the root node of the red-black tree; Traversing and reading each mechanism node from the mechanism data to be processed, and inserting each mechanism node into the red-black tree to obtain a plurality of leaf nodes; Searching and obtaining the organization node path information from each leaf node to the root node respectively, wherein the organization node path information includes the name of each organization node and the identifier of each organization node; According to preset business requirements, the organization node path information from each leaf node to the root node is counted and extracted to obtain target organization data, which is an organization data set with a flat data structure.

2. The data flattening method according to claim 1, wherein: The initial organization data of the target organization tree is obtained from the source database, and the initial organization data of the target organization tree is synchronized to the first organization data table in the target database according to a preset character encoding format. The initial organization data is multi-dimensional tree structure data, including: Read the source database connection information, source data table name and first institution data table name from the preset configuration information; Connecting to a source database using the connection information of the source database to obtain a first connection result, where the source database is a relational database; When the first connection result is successful, the initial organization data of the target organization tree is read according to the source data table name, and the initial organization data of the target organization tree is encoded according to a preset character encoding format, wherein the initial organization data is multi-dimensional tree structure data; The encoded initial organization data is synchronized to the first organization data table in the target database based on the first organization data table name.

3. The data flattening method according to claim 2, wherein: The step of synchronizing the encoded initial organization data to the first organization data table in the target database based on the first organization data table name includes: Obtain the corresponding target table creation script according to the first organization's data table name; The corresponding target table creation script is called to convert the data format of the encoded initial organization data, and the converted organization data is written into the first organization data table in the target database.

4. The data flattening method according to claim 2, wherein: When receiving the flattened processing mechanism data request sent by the terminal, querying the first mechanism data table through the preset calculation engine to obtain the mechanism data to be processed includes: When receiving a flattened processing organization data request sent by a terminal, parsing the flattened processing organization data request to obtain at least one field to be queried, and generating a query statement based on the at least one field to be queried and the first organization data table name; Obtaining a connection address, a port number, and a link path corresponding to the target database, and connecting to the target database based on the connection address, the port number, and the link path to obtain a second connection result; When the second connection result is successful, executing the query statement according to the preset parsing engine to obtain query data; The query data is encapsulated according to a preset data format to obtain the mechanism data to be processed.

5. The data flattening method according to claim 1, wherein: The updating of the target organization data into the second organization data table in the target database, generating response data corresponding to the flattened organization data request, and sending the response data to the terminal so that the terminal can perform a data query operation on the second organization data table according to preset business requirements, includes: Splitting the target organization data using a preset separator to obtain split organization data; Mapping the segmented organization data and the preset fields using a mapping operator to obtain mapping data; Updating the mapping data to a second mechanism data table in the target database by the preset calculation engine to obtain an update result, wherein the second mechanism data table is a matrix data table including row indexes and column indexes; The update result is data-encapsulated to obtain response data of the flattened processing organization data request, and the response data is sent to the terminal so that the terminal can perform a data query operation on the second organization data table according to preset business requirements.

6. The data flattening method according to any one of claims 1 to 5, characterized in that: After updating the target organization data to the second organization data table in the target database, generating response data corresponding to the flattening organization data request, and sending the response data to the terminal so that the terminal can perform a data query operation on the second organization data table according to preset business requirements, the data flattening method further includes: When detecting that data change occurs in the first organization data table, reading the changed organization data from the first organization data table; The second organization data table is updated based on the changed organization data.

7. A data flattening processing device, characterized in that: The data flattening processing device includes: An acquisition module is configured to acquire initial organization data of a target organization tree from a source database, and synchronize the initial organization data of the target organization tree to a first organization data table in the target database according to a preset character encoding format, wherein the initial organization data is multidimensional tree structure data; a query module, which, upon receiving a request for flattened processing of organization data from a terminal, queries the first organization data table through a preset calculation engine to obtain the organization data to be processed; A flattening module is used to flatten the to-be-processed organization data by using a red-black tree algorithm to obtain target organization data, wherein the target organization data is an organization data set with a flattened data structure; A first updating module is configured to update the target organization data to a second organization data table in the target database, generate response data corresponding to the flattened organization data request, and send the response data to the terminal so that the terminal can perform a data query operation on the second organization data table according to preset business requirements; The flattening module is specifically used to: initialize a red-black tree through a red-black tree algorithm, and initialize the root node of the red-black tree; traverse and read each organization node from the to-be-processed organization data, and insert each organization node into the red-black tree to obtain multiple leaf nodes; search and obtain the organization node path information from each leaf node to the root node respectively, and the organization node path information includes the name of each organization node and the identifier of each organization node; count and extract the organization node path information from each leaf node to the root node according to preset business requirements to obtain target organization data, and the target organization data is an organization data set with a flattened data structure.

8. The data flattening processing device according to claim 7, characterized in that: The acquisition module also includes: a reading unit, used to read the connection information, source data table name and first organization data table name of the source database from the preset configuration information; a connection unit, used to connect the source database using the connection information of the source database to obtain a first connection result, wherein the source database is a relational database; an encoding unit, when the first connection result is a successful connection, used to read the initial organization data of the target organization tree according to the source data table name, and encode the initial organization data of the target organization tree according to the preset character encoding format, wherein the initial organization data is multi-dimensional tree structure data; a synchronization unit, used to synchronize the encoded initial organization data to the first organization data table in the target database based on the first organization data table name.

9. A data flattening processing device, characterized in that: The data flattening processing device includes: a memory and at least one processor, wherein the memory stores instructions, and the memory and the at least one processor are interconnected via a line; The at least one processor calls the instruction in the memory to enable the data flattening processing device to execute the data flattening processing method according to any one of claims 1 to 6.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the data flattening processing method according to any one of claims 1 to 6 is implemented.

Images