Unified user identification method and device

Abstract

The present invention provides a unified user identification method and device, which belongs to the technical field of computers. In the method, multiple data tables are cleaned, associated and fused, and the obtained standard data table has strong correlation and more comprehensive and accurate data. In addition, in order to improve the retrieval speed, a fuzzy query dimension table is established in ES, and at the same time, a precise query dimension table is established in Doris, and a unique unified user identification interface is created for ES and Doris. When initiating a query request later, the user only needs to call the unique unified user identification interface, which reduces the interface delay. In addition, while realizing precise query, fuzzy query can also be realized, and the query method is richer. The data obtained by the query will be cached in Redis, which can speed up the next query of the data obtained by the query.

Description

Technical Field

[0001] The present invention relates to the technical field of computers, and in particular to a unified user identification method and device. Background Art

[0002] Currently, the user identification interface is completed by the project calling multiple interfaces on its own. The data warehouse only provides query interfaces for each project. Different projects call different interfaces, and different interface logics lead to different query data results. For example, according to the provided user entity data, in one project, only the instant messaging interface can be called, that is, only the instant messaging account can be determined, and in another project, only the portal website interface can be called, that is, only the portal website account can be determined. It can be seen that the virtual network account determined by one interface is not comprehensive and accurate, resulting in a decrease in the credibility and practicality of the product.

[0003] From the above description, it can be seen that the traditional user identification function has the following problems: First, the obtained virtual network account is not comprehensive and accurate, and the function data update is complicated. The reason is that different projects use different interfaces and conversion logics, resulting in inconsistent virtual network accounts determined by the same entity information in various project systems, which reduces the credibility of the project. In addition, different projects need to learn multiple interface formats (such as instant messaging interfaces, portal website interfaces), and the interface data needs to be re-connected after updating, resulting in cumbersome tasks; Second, the function delay is high. The reason is that when calling the user identification function, it is often necessary to call multiple interfaces (such as instant messaging interfaces, portal website interfaces, etc.). The delay or backlog of one interface will significantly affect the return speed of the entire interface, resulting in the interface speed being limited by the sum of the return times of all interfaces; Third, the data correlation is weak. A single interface only supports queries in a single data table. For example, when querying a mailbox through a user entity data, only the mailbox account associated with the current user entity data can be returned, and other virtual network accounts and entity information associated with the above mailbox account cannot be queried; Fourth, fuzzy query and multi-condition query are not supported. User identification only supports exact matching, and fuzzy query or multi-condition joint query cannot be performed.

[0004] In summary, traditional user identification needs to be iteratively optimized to alleviate the technical problems of traditional user identification, such as incomplete and inaccurate data, high latency, weak data correlation, and single query method. Summary of the invention

[0005] In view of this, the purpose of the present invention is to provide a unified user identification method and device to alleviate the technical problems of traditional user identification such as incomplete and accurate data, high latency, weak data correlation and single query method.

[0006] In a first aspect, an embodiment of the present invention provides a unified user identification method, including:

[0007] Acquire multiple data tables provided by a data provider, and use Hive to clean, associate and merge the multiple data tables to obtain a standard data table, wherein the standard data table includes: user entity data and virtual account data corresponding to the user entity data;

[0008] Determine a fuzzy query field according to the standard data table, and write the fuzzy query field into ES to create an index, thereby obtaining a fuzzy query dimension table, wherein the fuzzy query dimension table is a correspondence between the fuzzy query field and the first primary key value;

[0009] The standard data table is saved to Doris, and a dimension table for precise query is established in Doris, wherein the dimension table for precise query is a correspondence between precise query dimensions and second primary key values;

[0010] Create a unique unified user identification interface for the ES and the Doris, so that the user can call the unified user identification interface to implement a unified user identification function;

[0011] Obtain a query request initiated by a user calling the unified user identification interface, query data according to the query request, and then cache the queried data in Redis to improve the efficiency of re-querying the queried data, wherein the queried data cached in Redis is automatically cleared after reaching the expiration time.

[0012] Furthermore, Hive is used to clean, associate and merge the multiple data tables, including:

[0013] Deleting duplicate data from the plurality of data tables to obtain a plurality of data tables after deleting duplicate data;

[0014] Deleting the erroneous data in the multiple data tables after the duplicate data is deleted to obtain multiple data tables after the erroneous data is deleted, wherein the erroneous data is data that does not meet the corresponding standard format;

[0015] The associated data in the plurality of data tables after the erroneous data is deleted are associated and merged to obtain the standard data table, wherein the associated data are different data associated with the same target data.

[0016] Further, determining the fuzzy query field according to the standard data table includes:

[0017] Determining user entity data in the standard data table;

[0018] The fuzzy query field is determined according to the user entity data.

[0019] Further, the query request includes: a fuzzy query request, and querying data according to the query request includes:

[0020] Determine, in the dimension table of the fuzzy query, a target first primary key value corresponding to the target fuzzy query field carried in the query request;

[0021] According to the target first primary key value, first target standard data corresponding to the target first primary key value is determined in the standard data table, and the first target standard data is used as the data obtained by the query.

[0022] Further, the query request includes: a precise query request, the precise query dimension includes: user entity data, and querying data according to the query request includes:

[0023] Determine, in the dimension table of the precise query, a target second primary key value corresponding to the target user entity data according to the target user entity data carried in the query request;

[0024] According to the target second primary key value, second target standard data corresponding to the target second primary key value is determined in the standard data table, and the second target standard data is used as the data obtained by the query.

[0025] Furthermore, before querying the data according to the query request, the method further includes:

[0026] Determine whether the Redis contains data obtained by the target query corresponding to the query request;

[0027] If included, the data obtained by the target query will be used as the data obtained by the query.

[0028] Furthermore, after obtaining the data obtained by the query, the method further includes:

[0029] The data obtained by the query is returned according to a preset data query strategy, wherein the data query strategy includes: a strategy for querying a preset number of data obtained by the query, and a strategy for returning a preset number of data obtained by the query.

[0030] Furthermore, the method further comprises:

[0031] Obtaining a new data table provided by the data provider, and using Hive to clean the new data table, and then associating and fusing the cleaned new data table with the standard data table to obtain a new standard data table;

[0032] The standard data table in the Doris is updated according to the new standard data table, and the dimension table of the precise query in the Doris is updated, and the update information of the dimension table of the precise query is obtained by statistics;

[0033] The dimension table for fuzzy query in the ES is updated according to the update information of the dimension table for precise query and the cleaned new data table, so that the data in the ES is synchronized with the data in Doris.

[0034] Furthermore, the first primary key value is used to represent the position information of the data corresponding to the corresponding fuzzy query field in the standard data table, and the second primary key value is used to represent the position information of the data corresponding to the corresponding precise query dimension in the standard data table. The data in the standard data table is the data classified according to the fuzzy query field and / or the precise query dimension.

[0035] In a second aspect, an embodiment of the present invention further provides a unified user identification device, including:

[0036] The acquisition and cleaning association fusion processing unit is used to acquire multiple data tables provided by the data provider, and use Hive to perform cleaning association fusion processing on the multiple data tables to obtain a standard data table, wherein the standard data table includes: user entity data and virtual account data corresponding to the user entity data;

[0037] A determination and establishment unit, configured to determine a fuzzy query field according to the standard data table, and write the fuzzy query field into the ES to establish an index, thereby obtaining a dimension table of the fuzzy query, wherein the dimension table of the fuzzy query is a correspondence between the fuzzy query field and the first primary key value;

[0038] A saving and establishing unit, used for saving the standard data table to Doris, and establishing a dimension table for precise query in Doris, wherein the dimension table for precise query is a correspondence between precise query dimensions and second primary key values;

[0039] A creation unit, used to create a unique unified user identification interface for the ES and the Doris, so that the user can call the unified user identification interface to implement a unified user identification function;

[0040] A query unit is used to obtain a query request initiated by a user calling the unified user identification interface, and query data according to the query request, and then cache the queried data in Redis to improve the efficiency of re-querying the queried data, wherein the queried data cached in Redis is automatically cleared after reaching an expiration time.

[0041] In an embodiment of the present invention, a unified user identification method is provided, including: obtaining multiple data tables provided by a data provider, and using Hive to clean, associate and fuse the multiple data tables to obtain a standard data table, wherein the standard data table includes: user entity data and virtual account data corresponding to the user entity data; determining a fuzzy query field according to the standard data table, and writing the fuzzy query field into ES to establish an index, and obtaining a dimension table of fuzzy query, wherein the dimension table of fuzzy query is the correspondence between the fuzzy query field and the first primary key value; saving the standard data table to Doris, and establishing a dimension table of precise query in Doris, wherein the dimension table of precise query is the correspondence between the precise query dimension and the second primary key value; creating a unique unified user identification interface for ES and Doris, so that the user calls the unified user identification interface to implement the unified user identification function; obtaining a query request initiated by the user calling the unified user identification interface, and querying data according to the query request, and then caching the queried data in Redis to improve the efficiency of re-querying the queried data, wherein the queried data cached in Redis is automatically cleared after reaching the expiration time. From the above description, it can be seen that in the unified user identification method of the present invention, multiple data tables are cleaned, associated and fused, and the obtained standard data table has strong correlation and more comprehensive and accurate data. In addition, in order to improve the retrieval speed, a fuzzy query dimension table is established in ES. At the same time, a precise query dimension table is established in Doris, and a unique unified user identification interface is created for ES and Doris. When initiating a query request subsequently, the user only needs to call the unique unified user identification interface, which reduces the interface delay. In addition, while realizing precise query, fuzzy query can also be realized, and the query method is richer. The data obtained by the query will be cached in Redis, which can speed up the next query of the data obtained by the query, alleviating the technical problems of traditional user identification such as incomplete and accurate data, high latency, weak data correlation and single query method. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0043] Figure 1 A flowchart of a unified user identification method provided by an embodiment of the present invention;

[0044] Figure 2A flowchart of another unified user identification method provided by an embodiment of the present invention;

[0045] Figure 3 A schematic diagram of a unified user identification device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0046] The technical solution of the present invention will be clearly and completely described below in conjunction with the embodiments. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0047] Traditional user identification has technical problems such as incomplete and inaccurate data, high latency, weak data correlation and single query method.

[0048] Based on this, the method can be applied to application database management systems. In the unified user identification method of the present invention, multiple data tables are cleaned, associated and fused, and the obtained standard data tables have strong correlation and more comprehensive and accurate data. In addition, in order to improve the speed of retrieval, a fuzzy query dimension table is established in ES, and a precise query dimension table is established in Doris. A unique unified user identification interface is created for ES and Doris. When a query request is initiated later, the user only needs to call the unique unified user identification interface, which reduces the interface delay. In addition, while realizing precise query, fuzzy query can also be realized, and the query method is richer. The data obtained by the query will be cached in Redis, which can speed up the next query of the data obtained by the query.

[0049] To facilitate understanding of this embodiment, a unified user identification method disclosed in an embodiment of the present invention is first introduced in detail.

[0050] Embodiment 1:

[0051] According to an embodiment of the present invention, an embodiment of a unified user identification method is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0052] Figure 1 is a flow chart of a unified user identification method according to an embodiment of the present invention. Figure 1 As shown, the method comprises the following steps:

[0053] Step S102, obtaining multiple data tables provided by the data provider, and using Hive to clean, associate and merge the multiple data tables to obtain a standard data table, wherein the standard data table includes: user entity data and virtual account data corresponding to the user entity data;

[0054] In an embodiment of the present invention, the above-mentioned multiple data tables cover a large amount of data. There are data redundancy and association problems between these tables. In addition, there are many null value data and duplicate data in the data tables, and the processing requires a lot of time. Therefore, it is necessary to clean, associate and merge the above-mentioned multiple data tables to obtain standard data tables.

[0055] The above-mentioned cleaning association fusion processing process can improve resource utilization and data accuracy, reduce data duplication rate and error data rate, reduce the number of underlying tables, and reduce storage space.

[0056] Step S104, determining the fuzzy query field according to the standard data table, and writing the fuzzy query field into the ES to create an index, thereby obtaining a dimension table for the fuzzy query, wherein the dimension table for the fuzzy query is a correspondence between the fuzzy query field and the first primary key value;

[0057] Specifically, the inventors took into account that after data merging (i.e., associative fusion), a single standard data table would occupy too much space, resulting in a slow retrieval speed for the standard data table. To improve the retrieval speed, the inventors designed a method to determine the fuzzy query field based on the standard data table, and write the fuzzy query field into ES to create an index, thereby obtaining a fuzzy query dimension table. When performing a search, a fuzzy query can be performed based on the fuzzy query dimension table first, thereby improving the retrieval speed.

[0058] Step S106, saving the standard data table to Doris, and establishing a dimension table for precise query in Doris, wherein the dimension table for precise query is a correspondence between precise query dimensions and second primary key values;

[0059] Step S108, creating a unique unified user identification interface for ES and Doris, so that the user can call the unified user identification interface to implement the unified user identification function;

[0060] Specifically, the present invention completes a unified user identification interface, converts multiple call interfaces of a project into a single call to a unified user identification interface, thereby improving product credibility (reducing the problem of inaccurate data caused by different interfaces returning different data under the same conditions).

[0061] Step S110, obtain the query request initiated by the user calling the unified user identification interface, and query the data according to the query request, and then cache the queried data in Redis to improve the efficiency of re-querying the queried data, wherein the queried data cached in Redis is automatically cleared after reaching the expiration time.

[0062] Specifically, in order to improve the speed of the next data retrieval, the data obtained by the query can be cached in Redis. When the same query request is initiated next time, the corresponding query data can be directly obtained from Redis, which maximizes the retrieval efficiency. In addition, the query data cached in Redis is automatically cleared after the expiration time is reached, that is, each query data in Redis has an expiration time. When the expiration time is reached, the corresponding query data is automatically cleared.

[0063] It should be noted that when querying data based on the target user entity data carried in the query request, what is obtained is all user entity data and virtual account data corresponding to the target user entity data. Subsequently, each project can obtain the specific data it needs from it according to its own needs (that is, each project adapts according to its own needs).

[0064] In an embodiment of the present invention, a unified user identification method is provided, including: obtaining multiple data tables provided by a data provider, and using Hive to clean, associate and fuse the multiple data tables to obtain a standard data table, wherein the standard data table includes: user entity data and virtual account data corresponding to the user entity data; determining a fuzzy query field according to the standard data table, and writing the fuzzy query field into ES to establish an index, and obtaining a dimension table of fuzzy query, wherein the dimension table of fuzzy query is the correspondence between the fuzzy query field and the first primary key value; saving the standard data table to Doris, and establishing a dimension table of precise query in Doris, wherein the dimension table of precise query is the correspondence between the precise query dimension and the second primary key value; creating a unique unified user identification interface for ES and Doris, so that the user calls the unified user identification interface to implement the unified user identification function; obtaining a query request initiated by the user calling the unified user identification interface, and querying data according to the query request, and then caching the queried data in Redis to improve the efficiency of re-querying the queried data, wherein the queried data cached in Redis is automatically cleared after reaching the expiration time. From the above description, it can be seen that in the unified user identification method of the present invention, multiple data tables are cleaned, associated and fused, and the obtained standard data table has strong correlation and more comprehensive and accurate data. In addition, in order to improve the retrieval speed, a fuzzy query dimension table is established in ES. At the same time, a precise query dimension table is established in Doris, and a unique unified user identification interface is created for ES and Doris. When initiating a query request subsequently, the user only needs to call the unique unified user identification interface, which reduces the interface delay. In addition, while realizing precise query, fuzzy query can also be realized, and the query method is richer. The data obtained by the query will be cached in Redis, which can speed up the next query of the data obtained by the query, alleviating the technical problems of traditional user identification such as incomplete and accurate data, high latency, weak data correlation and single query method.

[0065] The above content briefly introduces the unified user identification method of the present invention. The specific contents involved therein are described in detail below.

[0066] In an optional embodiment of the present invention, Hive is used to perform cleaning, association and fusion processing on multiple data tables, which specifically includes the following steps:

[0067] (1) Deleting duplicate data from multiple data tables to obtain multiple data tables after deleting duplicate data;

[0068] (2) deleting the erroneous data in the multiple data tables after deleting duplicate data, to obtain multiple data tables after deleting the erroneous data, wherein the erroneous data is data that does not meet the corresponding standard format;

[0069] For example, for a portal website ID, the corresponding standard format is a digital format. If the obtained data is a portal website ID containing letters, it is determined that the obtained data is erroneous data.

[0070] (3) After the erroneous data is deleted, the associated data in the multiple data tables are associated and merged to obtain a standard data table, where the associated data are different data associated with the same target data.

[0071] In addition, considering that complex associations between large tables may lead to data skew and memory overflow problems, when performing association fusion, you can consider the strategy of splitting the data table. After splitting the large data table into multiple small data tables, you can then associate and fuse the multiple small data tables.

[0072] When implementing, the specific process can be: ensure that the data table has been stored in HDFS (Hadoop Distributed File System) and the corresponding Hive table has been created to store these data tables; then, perform data cleaning, including: removing duplicate records, filling missing values, correcting erroneous data, etc.; then, use JOIN operations to connect different data tables. Common JOIN types include: inner join (INNER JOIN), left join (LEFT JOIN), right join (RIGHT JOIN) and full join (FULL JOIN); if you need to further merge certain columns in the data table, you can use UNION ALL to merge multiple query results; after completing data processing, you should verify the quality and consistency of the data. You can run some statistical queries to check whether the data meets expectations.

[0073] In an optional embodiment of the present invention, determining the fuzzy query field according to the standard data table specifically includes the following steps:

[0074] (1) Determine user entity data in the standard data table;

[0075] (2) Determine the fuzzy query field based on the user entity data.

[0076] In an optional embodiment of the present invention, the query request includes: a fuzzy query request, and querying data according to the query request specifically includes the following steps:

[0077] (1) determining a target first primary key value corresponding to the target fuzzy query field in the fuzzy query dimension table according to the target fuzzy query field carried in the query request;

[0078] (2) Determine the first target standard data corresponding to the target first primary key value in the standard data table according to the target first primary key value, and use the first target standard data as the data obtained by the query.

[0079] Specifically, the first primary key value is used to represent the position information of the data corresponding to the corresponding fuzzy query field in the standard data table, and the data in the standard data table is the data classified according to the fuzzy query field. Because the data in the standard data table is the data classified according to the fuzzy query field, it is more convenient to search and query. In this way, the position interval information of the data corresponding to the corresponding target fuzzy query field in the standard data table can be determined according to the target first primary key value, and then, according to the position interval information, the first target standard data corresponding to the position interval can be quickly obtained.

[0080] In an optional embodiment of the present invention, the query request includes: a precise query request, the precise query dimension includes: user entity data, and querying the data according to the query request specifically includes the following steps:

[0081] (1) Determine the target second primary key value corresponding to the target user entity data in the dimension table of the precise query according to the target user entity data carried in the query request;

[0082] (2) Determine the second target standard data corresponding to the target second primary key value in the standard data table according to the target second primary key value, and use the second target standard data as the data obtained by the query.

[0083] Specifically, the second primary key value is used to represent the position information of the data corresponding to the corresponding precise query dimension in the standard data table, and the data in the standard data table is the data classified according to the precise query dimension.

[0084] In an optional embodiment of the present invention, before querying the data according to the query request, the method further includes:

[0085] (1) Determine whether Redis contains the data obtained by the target query corresponding to the query request;

[0086] (2) If included, the data obtained by the target query will be used as the data obtained by the query.

[0087] In an optional embodiment of the present invention, after obtaining the queried data, the method further includes:

[0088] The data obtained by the query is returned according to a preset data query strategy, wherein the data query strategy includes: a strategy for querying a preset number of data obtained by the query, and a strategy for returning a preset number of data obtained by the query.

[0089] Specifically, in order to avoid retrieval delays, you can consider limiting the number of query results (that is, the strategy of querying a preset number of query results to obtain data) to reduce the response time of the interface, or you can consider paging queries (the strategy of returning a preset number of query results to obtain data) to reduce the response time of the interface.

[0090] In an optional embodiment of the present invention, the method further comprises the following steps:

[0091] (1) Obtain the new data table provided by the data provider, and use Hive to clean the new data table, and then associate and merge the cleaned new data table with the standard data table to obtain a new standard data table;

[0092] (2) Update the standard data table in Doris according to the new standard data table, and update the dimension table of precise query in Doris, and obtain the updated information of the dimension table of precise query by statistics;

[0093] (3) Update the dimension table for fuzzy query in ES based on the updated information of the dimension table for precise query and the cleaned new data table to synchronize the data in ES with the data in Doris.

[0094] Specifically, it is determined whether the updated information of the dimension table of the precise query is consistent with the new data table after cleaning; specifically: whether the updated information can correctly reflect the new content of the new data table;

[0095] If they match, the dimension table for fuzzy query in ES is updated according to the update information of the dimension table for precise query.

[0096] The process of the unified user identification method of the present invention is as follows Figure 2 As shown in the figure, the data is provided by raw data and third-party data. After data cleaning, it is imported into Hive, where data regularization, deduplication and merging are completed. Finally, the cleaned data is imported into Doris for external display. At the same time, some primary key fields are imported into ES for fuzzy query indexing to improve the query speed. Finally, services are provided through interfaces and the interfaces are exposed externally.

[0097] Embodiment 2:

[0098] The embodiment of the present invention further provides a unified user identification device, which is mainly used to execute the unified user identification method provided in the first embodiment of the present invention. The unified user identification device provided in the embodiment of the present invention is specifically introduced below.

[0099] Figure 3 is a schematic diagram of a unified user identification device according to an embodiment of the present invention. Figure 3As shown, the device mainly includes: an acquisition and cleaning association fusion processing unit 10, a determination and establishment unit 20, a storage and establishment unit 30, a creation unit 40 and a query unit 50, wherein:

[0100] The acquisition and cleaning association fusion processing unit is used to acquire multiple data tables provided by the data provider, and use Hive to perform cleaning association fusion processing on the multiple data tables to obtain a standard data table, wherein the standard data table includes: user entity data and virtual account data corresponding to the user entity data;

[0101] A determination and establishment unit is used to determine the fuzzy query field according to the standard data table, and write the fuzzy query field into the ES to establish an index, so as to obtain a dimension table of the fuzzy query, wherein the dimension table of the fuzzy query is the correspondence between the fuzzy query field and the first primary key value;

[0102] A saving and establishing unit, used to save the standard data table to Doris, and establish a dimension table for precise query in Doris, wherein the dimension table for precise query is the correspondence between the precise query dimension and the second primary key value;

[0103] A creation unit is used to create a unique unified user identification interface for ES and Doris, so that users can call the unified user identification interface to implement a unified user identification function;

[0104] The query unit is used to obtain the query request initiated by the user calling the unified user identification interface, and query the data according to the query request, and then cache the queried data in Redis to improve the efficiency of re-querying the queried data, wherein the queried data cached in Redis is automatically cleared after reaching the expiration time.

[0105] In an embodiment of the present invention, a unified user identification device is provided, including: obtaining multiple data tables provided by a data provider, and using Hive to clean, associate and fuse the multiple data tables to obtain a standard data table, wherein the standard data table includes: user entity data and virtual account data corresponding to the user entity data; determining a fuzzy query field according to the standard data table, and writing the fuzzy query field into ES to establish an index, and obtaining a dimension table of fuzzy query, wherein the dimension table of fuzzy query is the correspondence between the fuzzy query field and the first primary key value; saving the standard data table to Doris, and establishing a dimension table of precise query in Doris, wherein the dimension table of precise query is the correspondence between the precise query dimension and the second primary key value; creating a unique unified user identification interface for ES and Doris, so that the user calls the unified user identification interface to implement the unified user identification function; obtaining a query request initiated by the user calling the unified user identification interface, and querying data according to the query request, and then caching the queried data in Redis to improve the efficiency of re-querying the queried data, wherein the queried data cached in Redis is automatically cleared after reaching the expiration time. From the above description, it can be seen that in the unified user identification device of the present invention, multiple data tables are cleaned, associated and fused, and the obtained standard data table has strong correlation and more comprehensive and accurate data. In addition, in order to improve the retrieval speed, a fuzzy query dimension table is established in ES. At the same time, a precise query dimension table is established in Doris, and a unique unified user identification interface is created for ES and Doris. When initiating a query request subsequently, the user only needs to call the unique unified user identification interface, which reduces the interface delay. In addition, while realizing precise query, fuzzy query can also be realized, and the query method is richer. The data obtained by the query will be cached in Redis, which can speed up the next query of the data obtained by the query, alleviating the technical problems of traditional user identification such as incomplete and accurate data, high latency, weak data correlation and single query method.

[0106] Optionally, the acquisition and cleaning association fusion processing unit is also used to: delete duplicate data in multiple data tables to obtain multiple data tables after duplicate data is deleted; delete erroneous data in the multiple data tables after duplicate data is deleted to obtain multiple data tables after erroneous data is deleted, wherein the erroneous data is data that does not meet the corresponding standard format; associate and fuse the associated data in the multiple data tables after the erroneous data is deleted to obtain a standard data table, wherein the associated data is different data associated with the same target data.

[0107] Optionally, the determining and establishing unit is further used to: determine user entity data in a standard data table; and determine a fuzzy query field according to the user entity data.

[0108] Optionally, the query request includes: a fuzzy query request, and the query unit is also used to: determine the target first primary key value corresponding to the target fuzzy query field in the dimension table of the fuzzy query according to the target fuzzy query field carried in the query request; determine the first target standard data corresponding to the target first primary key value in the standard data table according to the target first primary key value, and use the first target standard data as the data obtained by the query.

[0109] Optionally, the query request includes: a precise query request, the precise query dimension includes: user entity data, and the query unit is also used to: determine the target second primary key value corresponding to the target user entity data in the precise query dimension table according to the target user entity data carried in the query request; determine the second target standard data corresponding to the target second primary key value in the standard data table according to the target second primary key value, and use the second target standard data as the data obtained by the query.

[0110] Optionally, the device is also used to: determine whether Redis contains data obtained by the target query corresponding to the query request; if contained, use the data obtained by the target query as the data obtained by the query.

[0111] Optionally, the device is further used to: return the queried data according to a preset data query strategy, wherein the data query strategy includes: a strategy for querying a preset number of queried data, and a strategy for returning a preset number of queried data.

[0112] Optionally, the device is also used to: obtain a new data table provided by a data provider, and use Hive to clean the new data table, and then associate and fuse the cleaned new data table with the standard data table to obtain a new standard data table; update the standard data table in Doris according to the new standard data table, and update the dimension table of the precise query in Doris, and obtain the update information of the dimension table of the precise query by statistics; update the dimension table of the fuzzy query in ES according to the update information of the dimension table of the precise query and the cleaned new data table, so as to synchronize the data in ES with the data in Doris.

[0113] Optionally, the first primary key value is used to represent the location information of the data corresponding to the corresponding fuzzy query field in the standard data table, and the second primary key value is used to represent the location information of the data corresponding to the corresponding precise query dimension in the standard data table. The data in the standard data table is data classified according to the fuzzy query field and / or precise query dimension.

[0114] The computer program product of the unified user identification method provided in the embodiment of the present invention includes a computer-readable storage medium storing program code. The instructions included in the program code can be used to execute the method in the previous method embodiment. The specific implementation can be found in the method embodiment and will not be repeated here.

[0115] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the system and device described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0116] In addition, in the description of the embodiments of the present invention, unless otherwise clearly specified and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0117] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc., which can store program codes.

[0118] In the description of the present invention, it should be noted that the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inner", "outer", etc., indicating the orientation or positional relationship, are based on the orientation or positional relationship shown in the drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as limiting the present invention. In addition, the terms "first", "second", and "third" are used for descriptive purposes only, and cannot be understood as indicating or implying relative importance.

[0119] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein by equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A unified user identification method, It is characterized in that include: Acquire multiple data tables provided by a data provider, and use Hive to clean, associate and merge the multiple data tables to obtain a standard data table, wherein the standard data table includes: user entity data and virtual account data corresponding to the user entity data; Determine a fuzzy query field according to the standard data table, and write the fuzzy query field into ES to create an index, thereby obtaining a fuzzy query dimension table, wherein the fuzzy query dimension table is a correspondence between the fuzzy query field and the first primary key value; The standard data table is saved to Doris, and a dimension table for precise query is established in Doris, wherein the dimension table for precise query is a correspondence between precise query dimensions and second primary key values; Create a unique unified user identification interface for the ES and the Doris, so that the user can call the unified user identification interface to implement a unified user identification function; Obtaining a query request initiated by a user calling the unified user identification interface, querying data according to the query request, and then caching the queried data in Redis to improve the efficiency of re-querying the queried data, wherein the queried data cached in Redis is automatically cleared after reaching an expiration time; Wherein, determining the fuzzy query field according to the standard data table includes: Determining user entity data in the standard data table; Determine the fuzzy query field according to the user entity data; The query request includes: a fuzzy query request, and querying data according to the query request includes: Determine, in the dimension table of the fuzzy query, a target first primary key value corresponding to the target fuzzy query field carried in the query request; Determine, in the standard data table, first target standard data corresponding to the target first primary key value according to the target first primary key value, and use the first target standard data as the data obtained by the query; The query request includes: a precise query request, the precise query dimension includes: user entity data, and querying data according to the query request includes: Determine, in the dimension table of the precise query, a target second primary key value corresponding to the target user entity data according to the target user entity data carried in the query request; Determine, in the standard data table, second target standard data corresponding to the target second primary key value according to the target second primary key value, and use the second target standard data as the data obtained by the query; The first primary key value is used to represent the position information of the data corresponding to the corresponding fuzzy query field in the standard data table, and the second primary key value is used to represent the position information of the data corresponding to the corresponding precise query dimension in the standard data table. The data in the standard data table is the data classified according to the fuzzy query field and / or the precise query dimension; The method further comprises: Obtaining a new data table provided by the data provider, and using Hive to clean the new data table, and then associating and fusing the cleaned new data table with the standard data table to obtain a new standard data table; The standard data table in the Doris is updated according to the new standard data table, and the dimension table of the precise query in the Doris is updated, and the update information of the dimension table of the precise query is obtained by statistics; The dimension table of the fuzzy query in the ES is updated according to the updated information of the dimension table of the precise query and the cleaned new data table, so that the data in the ES is synchronized with the data in the Doris, wherein the dimension table of the fuzzy query in the ES is updated according to the updated information of the dimension table of the precise query and the cleaned new data table, including: judging whether the updated information of the dimension table of the precise query is consistent with the cleaned new data table; if consistent, updating the dimension table of the fuzzy query in the ES according to the updated information of the dimension table of the precise query.

2. The method according to claim 1, It is characterized in that Hive is used to clean, associate and merge the multiple data tables, including: Deleting duplicate data from the plurality of data tables to obtain a plurality of data tables after deleting duplicate data; Deleting the erroneous data in the multiple data tables after the duplicate data is deleted to obtain multiple data tables after the erroneous data is deleted, wherein the erroneous data is data that does not meet the corresponding standard format; The associated data in the plurality of data tables after the erroneous data is deleted are associated and merged to obtain the standard data table, wherein the associated data are different data associated with the same target data.

3. The method according to claim 1, It is characterized in that Before querying data according to the query request, the method further includes: Determine whether the Redis contains data obtained by the target query corresponding to the query request; If included, the data obtained by the target query will be used as the data obtained by the query.

4. The method according to claim 1, It is characterized in that After obtaining the data obtained by the query, the method further includes: The data obtained by the query is returned according to a preset data query strategy, wherein the data query strategy includes: a strategy for querying a preset number of data obtained by the query, and a strategy for returning a preset number of data obtained by the query.

5. A unified user identification device, It is characterized in that include: The acquisition and cleaning association fusion processing unit is used to acquire multiple data tables provided by the data provider, and use Hive to perform cleaning association fusion processing on the multiple data tables to obtain a standard data table, wherein the standard data table includes: user entity data and virtual account data corresponding to the user entity data; A determination and establishment unit, configured to determine a fuzzy query field according to the standard data table, and write the fuzzy query field into the ES to establish an index, thereby obtaining a dimension table of the fuzzy query, wherein the dimension table of the fuzzy query is a correspondence between the fuzzy query field and the first primary key value; A saving and establishing unit, used for saving the standard data table to Doris, and establishing a dimension table for precise query in Doris, wherein the dimension table for precise query is a correspondence between precise query dimensions and second primary key values; A creation unit, used to create a unique unified user identification interface for the ES and the Doris, so that the user can call the unified user identification interface to implement a unified user identification function; A query unit, used to obtain a query request initiated by a user calling the unified user identification interface, query data according to the query request, and then cache the queried data in Redis to improve the efficiency of re-querying the queried data, wherein the queried data cached in Redis is automatically cleared after reaching an expiration time; Wherein, the determining and establishing unit is further used to: determine user entity data in the standard data table; determine the fuzzy query field according to the user entity data; Wherein, the query request includes: a fuzzy query request, and the query unit is further used to: determine a target first primary key value corresponding to the target fuzzy query field in the dimension table of the fuzzy query according to the target fuzzy query field carried in the query request; determine first target standard data corresponding to the target first primary key value in the standard data table according to the target first primary key value, and use the first target standard data as the data obtained by the query; Wherein, the query request includes: a precise query request, the precise query dimension includes: user entity data, and the query unit is further used to: determine a target second primary key value corresponding to the target user entity data in the dimension table of the precise query according to the target user entity data carried in the query request; determine second target standard data corresponding to the target second primary key value in the standard data table according to the target second primary key value, and use the second target standard data as the data obtained by the query; The first primary key value is used to represent the position information of the data corresponding to the corresponding fuzzy query field in the standard data table, and the second primary key value is used to represent the position information of the data corresponding to the corresponding precise query dimension in the standard data table. The data in the standard data table is the data classified according to the fuzzy query field and / or the precise query dimension; The device is also used for: obtaining a new data table provided by the data provider, and using Hive to clean the new data table, and then associating and fusing the cleaned new data table with the standard data table to obtain a new standard data table; updating the standard data table in the Doris according to the new standard data table, and updating the dimension table of the precise query in the Doris, and obtaining the update information of the dimension table of the precise query by statistics; updating the dimension table of the fuzzy query in the ES according to the update information of the dimension table of the precise query and the cleaned new data table, so as to synchronize the data in the ES with the data in the Doris, wherein updating the dimension table of the fuzzy query in the ES according to the update information of the dimension table of the precise query and the cleaned new data table includes: judging whether the update information of the dimension table of the precise query is consistent with the cleaned new data table; if it is consistent, updating the dimension table of the fuzzy query in the ES according to the update information of the dimension table of the precise query.

Images