Data processing method and device, electronic equipment and storage medium

By using the prefix tree structure to organize data in the database, the problem of inefficient data update caused by hash conflicts is solved, and more efficient data update processing is achieved.

CN118964366APending Publication Date: 2024-11-15BEIJING WODONG TIANJUN INFORMATION TECH CO LTD +1
View PDF 5 Cites 0 Cited by

Patent Information

Application Number
CN202310545372.1
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2023-05-15
Publication Date
2024-11-15

AI Technical Summary

Technical Problem

In the big data scenario, the data update processing efficiency caused by hash conflicts in the database is low, and the hash time and conflicts of the hash function are uncertain, which affects the processing efficiency of data updates.

Method used

By storing database data as a prefix tree structure in the cache and obtaining the prefix tree according to the target field, iterating over the matching data set to determine the newly added and invalid data sets, thus performing database updates, replacing the data filtering method of hash tables.

Benefits of technology

Eliminates the hashing time and hash conflict caused by hash function, improves the processing efficiency of data updates, and solves the inefficiency problem caused by hash conflicts.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118964366A_ABST
    Figure CN118964366A_ABST
Patent Text Reader

Abstract

The invention provides a data processing method and device, electronic equipment and a storage medium, and can be applied to the technical field of big data. The method comprises the steps that a data processing request is received, and the data processing request comprises a target field and a data set related to the target field; acquiring a first prefix tree from the cache based on the primary key of the target field under the condition of determining that the primary key of the target field exists in the cache; based on the first prefix tree and the data set, determining a newly added data set and an invalid data set; and performing data updating on the database associated with the cache based on the newly added data set and the invalid data set.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present disclosure relates to the field of big data technology, and more specifically, to a data processing method, device, electronic device and storage medium. Background Art

[0002] With the rapid development of information technology, changes in business coverage data are becoming more frequent in various business scenarios. When updating data in the database, the server can find the corresponding data from the database and compare it with the data entered by the user to filter out new data and invalid data, and finally perform data insertion and update operations respectively.

[0003] In related technologies, when updating data in a database, a hash table is often used to compare and filter data. However, in a big data scenario, there is a high probability that hash conflicts will occur in the data in the database. The uncertainty of the time consumed by hash conflicts will make the processing efficiency of data updates low. Summary of the invention

[0004] In view of this, the present disclosure provides a data processing method, apparatus, electronic device, readable storage medium, and computer program product.

[0005] One aspect of the present disclosure provides a data processing method, including: receiving a data processing request, wherein the data processing request includes a target field and a data set related to the target field; obtaining a first prefix tree from the cache based on the primary key of the target field when it is determined that the primary key of the target field exists in the cache; determining a newly added data set and an invalidated data set based on the first prefix tree and the data set; and updating data in a database associated with the cache based on the newly added data set and the invalidated data set, respectively.

[0006] According to an embodiment of the present disclosure, the above-mentioned data set includes multiple input parameter data; wherein, the above-mentioned determination of a newly added data set and an invalid data set based on the above-mentioned first prefix tree and the above-mentioned data set includes: updating the version number configured in the above-mentioned first prefix tree to obtain a target version number; for each of the above-mentioned input parameter data, using the above-mentioned first prefix tree to traverse and match multiple characters included in the above-mentioned input parameter data to obtain a first matching result; when the above-mentioned first matching result indicates that the above-mentioned input parameter data exists in the above-mentioned first prefix tree, determining a first path in the above-mentioned first prefix tree that matches the above-mentioned input parameter data; using the above-mentioned target version number to update the value of the first leaf node included in the above-mentioned first path; when the above-mentioned first matching result indicates that the above-mentioned input parameter data does not exist in the above-mentioned first prefix tree, adding the above-mentioned input parameter data to the above-mentioned newly added data set; and determining the above-mentioned invalid data set from the above-mentioned first prefix tree based on the respective values ​​of multiple leaf nodes in the above-mentioned first prefix tree.

[0007] According to an embodiment of the present disclosure, the first prefix tree includes multiple node layers, the root node of the first prefix tree belongs to the first node layer of the first prefix tree, and each of the node layers includes at least one node; wherein, for each of the input parameter data, the first prefix tree is used to traverse and match the multiple characters included in the input parameter data to obtain a first matching result, including: for each of the input parameter data, when it is determined that the value of the first node in the i-th node layer of the first prefix tree matches the i-1-th character of the input parameter data, determine from the i+1-th node layer of the first prefix tree the nodes connected to the first node. at least one second node, wherein the above i is an integer greater than 1, and the above i is less than or equal to the number of characters included in the above input data; the i-th character of the above input data is matched with the respective values ​​of the above at least one second node to obtain at least one second matching result; when the above at least one second matching result indicates an incomplete match, the above first matching result indicating that the above input data does not exist in the above first prefix tree is obtained; and when the last character of the above input data matches the third node of the above first prefix tree, the above first matching result is obtained based on the child node connected to the above third node.

[0008] According to an embodiment of the present disclosure, in a case where the last character of the above-mentioned input parameter data matches the third node of the above-mentioned first prefix tree, the above-mentioned first matching result is obtained based on the child nodes connected to the above-mentioned third node, including: in a case where it is determined that there is a leaf node in the child nodes connected to the above-mentioned third node, the above-mentioned first matching result indicating that the above-mentioned input parameter data exists in the above-mentioned first prefix tree is obtained; and in a case where it is determined that there is no leaf node in the child nodes connected to the above-mentioned third node, the above-mentioned first matching result indicating that the above-mentioned input parameter data does not exist in the above-mentioned first prefix tree is obtained.

[0009] According to an embodiment of the present disclosure, the above-mentioned invalid data set is determined from the above-mentioned first prefix tree based on the respective values ​​of the multiple leaf nodes in the above-mentioned first prefix tree, including: using the above-mentioned target version number to traverse and match the respective values ​​of the above-mentioned multiple leaf nodes to determine at least one second leaf node from the above-mentioned multiple leaf nodes, wherein the value of the above-mentioned second leaf node does not match the above-mentioned target version number; and determining the above-mentioned invalid data set based on the second path to which each of the above-mentioned at least one second leaf node belongs.

[0010] According to an embodiment of the present disclosure, the method further includes: when the first matching result indicates that the input parameter data does not exist in the first prefix tree, writing the input parameter data into the first prefix tree.

[0011] According to an embodiment of the present disclosure, the above-mentioned input parameter data is written into the above-mentioned first prefix tree, including: when it is determined that the value of the fourth node in the j-th node layer of the above-mentioned first prefix tree matches the j-1-th character of the above-mentioned input parameter data, at least one fifth node associated with the above-mentioned fourth node is determined from the j+1-th node layer of the above-mentioned first prefix tree, wherein the above-mentioned j is an integer greater than 1, and the above-mentioned j is less than or equal to the number of characters included in the above-mentioned input parameter data; the j-th character of the above-mentioned input parameter data is matched with the respective values ​​of the above-mentioned at least one fifth node, respectively, to obtain at least one third matching result; when the above-mentioned at least one third matching result indicates an incomplete match, based on the j-th character of the above-mentioned input parameter data, a sixth node connected to the above-mentioned fourth node is generated at the j+1-th node layer of the above-mentioned first prefix tree; and when the j-th character of the above-mentioned input parameter data is the last character of the above-mentioned input parameter data, based on the above-mentioned target version number, a third leaf node connected to the above-mentioned sixth node is generated at the j+2-th node layer of the above-mentioned first prefix tree.

[0012] According to an embodiment of the present disclosure, the method further includes: pruning the first prefix tree based on respective values ​​of a plurality of leaf nodes in the first prefix tree.

[0013] According to an embodiment of the present disclosure, the above-mentioned first prefix tree is pruned based on the respective values ​​of multiple leaf nodes in the above-mentioned first prefix tree, including: for each of the above-mentioned leaf nodes, when the value of the above-mentioned leaf node does not match the above-mentioned target version number, determining the seventh node connected to the above-mentioned leaf node, and deleting the above-mentioned leaf node; querying to obtain the sibling nodes of the above-mentioned seventh node, wherein the sibling nodes of the above-mentioned seventh node have the same parent node as the above-mentioned seventh node; when it is determined that there is a leaf node among the sibling nodes of the above-mentioned seventh node, deleting the above-mentioned seventh node to complete the pruning of the above-mentioned leaf node; and when it is determined that there is no leaf node among the sibling nodes of the above-mentioned seventh node, determining the eighth node connected to the above-mentioned seventh node, deleting the above-mentioned seventh node, and continuing to query the sibling nodes of the above-mentioned eighth node.

[0014] According to an embodiment of the present disclosure, the above method also includes: when it is determined that the primary key of the above target field does not exist in the above cache, using the above data set to update data in the above database; using the primary key of the above target field as the root node and generating a second prefix tree based on the above data set; and writing the above second prefix tree into the above cache.

[0015] Another aspect of the present disclosure provides a data processing device, including: a receiving module, used to receive a data processing request, wherein the data processing request includes a target field and a data set related to the target field; an acquisition module, used to acquire a first prefix tree from the cache based on the primary key of the target field when it is determined that the primary key of the target field exists in the cache; a determination module, used to determine a newly added data set and an invalidated data set based on the first prefix tree and the data set; and a first processing module, used to update data in a database associated with the cache based on the newly added data set and the invalidated data set, respectively.

[0016] Another aspect of the present disclosure provides an electronic device, comprising: one or more processors; and a memory for storing one or more instructions, wherein when the one or more instructions are executed by the one or more processors, the one or more processors implement the method described above.

[0017] Another aspect of the present disclosure provides a computer-readable storage medium storing computer-executable instructions, which are used to implement the above method when executed.

[0018] Another aspect of the present disclosure provides a computer program product, the computer program product comprising computer executable instructions, and the instructions are used to implement the above method when executed.

[0019] According to an embodiment of the present disclosure, data belonging to the same field in a database can be heterogeneously formed into prefix trees and written into a cache for storage. When a data processing request is received for data update, a first prefix tree can be obtained from the cache according to the target field, and based on the first prefix tree and the data set related to the target field, an invalid data set can be determined from the first prefix tree, and a newly added data set can be determined from the data set, and then the newly added data set and the invalid data set can be used to update the data related to the target field in the database. By organizing the data in the database in the form of a prefix tree, a prefix tree can be used to implement the screening of newly added data and invalid data, so as to replace the method of using a hash table for data screening in the related art, so that the technical problem of low processing efficiency of data update in the related art can be at least partially overcome, the hashing time of the hash function and the time consumption caused by hash conflicts can be eliminated, and the processing efficiency of data update can be effectively improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The above and other objects, features and advantages of the present disclosure will become more apparent through the following description of the embodiments of the present disclosure with reference to the accompanying drawings, in which:

[0021] Figure 1The figure schematically shows a data updating method in the related art.

[0022] Figure 2 An exemplary system architecture to which the data processing method and apparatus according to an embodiment of the present disclosure can be applied is schematically shown.

[0023] Figure 3 The flowchart of the data processing method according to the embodiment of the present disclosure is schematically shown.

[0024] Figure 4 A schematic diagram of a first prefix tree according to an embodiment of the present disclosure is schematically shown.

[0025] Figure 5 A schematic diagram of a first intermediate prefix tree according to an embodiment of the present disclosure is schematically shown.

[0026] Figure 6 A schematic diagram of a second intermediate prefix tree according to an embodiment of the present disclosure is schematically shown.

[0027] Figure 7 The flowchart schematically shows a data processing method according to another embodiment of the present disclosure.

[0028] Figure 8 The block diagram schematically shows a data processing device according to an embodiment of the present disclosure.

[0029] Fig. 9 A block diagram of an electronic device suitable for implementing a data processing method according to an embodiment of the present disclosure is schematically shown. DETAILED DESCRIPTION

[0030] Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the present disclosure. In the following detailed description, for ease of explanation, many specific details are set forth to provide a comprehensive understanding of the embodiments of the present disclosure. However, it is apparent that one or more embodiments may also be implemented without these specific details. In addition, in the following description, descriptions of known structures and technologies are omitted to avoid unnecessary confusion of the concepts of the present disclosure.

[0031] The terms used herein are only for describing specific embodiments and are not intended to limit the present disclosure. The terms "comprise", "include", etc. used herein indicate the existence of the features, steps, operations and / or components, but do not exclude the existence or addition of one or more other features, steps, operations or components.

[0032] All terms (including technical and scientific terms) used herein have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of this specification, and should not be interpreted in an idealized or overly rigid manner.

[0033] When using expressions such as "at least one of A, B, and C, etc.", they should generally be interpreted according to the meaning of the expression commonly understood by those skilled in the art (for example, "a system having at least one of A, B, and C" should include but is not limited to a system having A alone, B alone, C alone, A and B, A and C, B and C, and / or A, B, C, etc.).

[0034] With the rapid development of information technology, changes in business coverage data are becoming more frequent in various business scenarios. In production practice, in order to maintain the stability of the database, logical deletion is generally used to delete the original invalid data. Therefore, if the original data is deleted and new data is written when the data is updated, the amount of data in the database will continue to increase, which will eventually affect the normal use of the database. Therefore, in related technologies, collection queues or hash tables are often used to filter out new data and invalid data through data comparison and screening, and then the new data and invalid data are used to update the database data.

[0035] Figure 1 The figure schematically shows a data updating method in the related art.

[0036] like Figure 1 As shown, when receiving a request to update the data in the database, the server can use the index field in the data list as the screening basis to filter out the corresponding data in the database as listDB. Afterwards, the server can compare the data list with listDB to find the newly added data listI and the invalid data listU, and then perform database insertion according to listI and perform database update according to listU.

[0037] The most critical step of the data update method in the related art is to determine the difference between the data set in the data list and the original data set in the database, that is, to filter out the newly added data and invalid data. Its essence is to traverse each element in a set to see if the element exists in another set. Using the acquisition queue to complete the data screening requires two loop traversals, and the time complexity is 2*O(N*M*L), where N represents the amount of data in the data set in the data list, M represents the amount of data in the original data set, and L represents the average character length of each data. Using a hash table to complete the data screening only requires one loop traversal. Taking the query of a value as an example, its execution process includes: 1. Use a hash function to find the hash value of the value. 2. Go to the hash address to get the stored value, and determine whether the value is the same as the stored value. 3. When there is a lot of data, there may be a hash conflict in the hash table. If the value of the first node on the hash address is not the query target value, then the value needs to be found in the subsequent nodes of the first node on the hash address. Therefore, the time complexity of using a hash table to complete the data processing is O(N*t*L)+O(M*L)+T, where t represents the average hashing time of the hash function and T represents the time consumed by hash conflicts. In the big data scenario, the possibility of hash conflicts in the data in the database will continue to increase. When using a hash table to complete data screening, the hashing time of the hash function is long and the time consumed to resolve hash conflicts is uncertain, resulting in low processing efficiency for data updates.

[0038] In view of this, the embodiments of the present disclosure provide a data processing method, device, electronic device and storage medium, which can eliminate the query time caused by the hash time and hash conflict of the hash function and improve the request response rate. Specifically, the data processing method includes: receiving a data processing request, wherein the data processing request includes a target field and a data set related to the target field; in the case of determining that the primary key of the target field exists in the cache, obtaining a first prefix tree from the cache based on the primary key of the target field; based on the first prefix tree and the data set, determining a newly added data set and an invalid data set; and updating the data in the database associated with the cache based on the newly added data set and the invalid data set respectively.

[0039] In the technical solution of the present disclosure, the collection, storage, use, processing, transmission, provision, disclosure and application of the data involved (including but not limited to user personal information) comply with the provisions of relevant laws and regulations, take necessary confidentiality measures, and do not violate public order and good morals.

[0040] It should be noted that, unless it is explicitly stated that there is a sequence of execution between different operations shown in the flowchart in the embodiments of the present disclosure, or there is a sequence of execution between different operations in technical implementation, otherwise, the execution order of multiple operations may not be prioritized, and multiple operations may also be executed simultaneously.

[0041] Figure 2 The exemplary system architecture to which the data processing method and apparatus according to the embodiments of the present disclosure can be applied is schematically shown. It should be noted that: Figure 2 What is shown is merely an example of a system architecture to which the embodiments of the present disclosure can be applied, in order to help those skilled in the art understand the technical content of the present disclosure, but it does not mean that the embodiments of the present disclosure cannot be used in other devices, systems, environments or scenarios.

[0042] like Figure 2 As shown, the system architecture 200 according to this embodiment may include terminal devices 201 , 202 , 203 , a network 204 and a server 205 .

[0043] The terminal devices 201 , 202 , and 203 may be various electronic devices, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, and the like.

[0044] The network 204 is used to provide a medium for communication links between the terminal devices 201, 202, 203 and the server 205. The network 204 may include various connection types, such as wired and / or wireless communication links, and the like.

[0045] The server 205 may be a server that provides various services, such as a server that processes data generated when a user uses the terminal devices 201, 202, and 203. The server may update the stock data in the database based on the generated data.

[0046] It should be noted that the data processing method provided in the embodiment of the present disclosure can generally be executed by the server 205. Accordingly, the data processing device provided in the embodiment of the present disclosure can generally be set in the server 205. The data processing method provided in the embodiment of the present disclosure can also be executed by a server or server cluster that is different from the server 205 and can communicate with the terminal device 201, 202, 203 and / or the server 205. Accordingly, the data processing device provided in the embodiment of the present disclosure can also be set in a server or server cluster that is different from the server 205 and can communicate with the terminal device 201, 202, 203 and / or the server 205. Alternatively, the data processing method provided in the embodiment of the present disclosure can also be executed by the terminal device 201, 202, or 203, or it can also be executed by other terminal devices different from the terminal device 201, 202, or 203. Accordingly, the data processing device provided in the embodiment of the present disclosure can also be set in the terminal device 201, 202, or 203, or it can be set in other terminal devices different from the terminal device 201, 202, or 203.

[0047] It should be understood that Figure 2 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0048] Figure 3 The flowchart of the data processing method according to the embodiment of the present disclosure is schematically shown.

[0049] like Figure 3 As shown, the method includes operations S301 to S304.

[0050] In operation S301 , a data processing request is received, wherein the data processing request includes a target field and a data set related to the target field.

[0051] In operation S302, when it is determined that the primary key of the target field exists in the cache, a first prefix tree is obtained from the cache based on the primary key of the target field.

[0052] In operation S303 , a newly added data set and an invalid data set are determined based on the first prefix tree and the data set.

[0053] In operation S304, data in a database associated with the cache is updated based on the newly added data set and the invalid data set.

[0054] According to an embodiment of the present disclosure, a data processing request may include multiple target fields, each target field may be associated with a data object, and a data set associated with the target field may be composed of multiple pieces of data belonging to the data object.

[0055] According to an embodiment of the present disclosure, a cache may refer to any cache database, such as Redis, SQLite, etc., which is not limited here. A cache may have a corresponding database, which may be any type of database, which is not limited here. The data of each data object may be originally stored in a database corresponding to the cache. At the same time, the data of each data object stored in the database may be heterogeneous into a prefix tree and written into the cache. The prefix tree may be a multi-branch tree structure, that is, a node in the prefix tree may have multiple child nodes.

[0056] According to an embodiment of the present disclosure, determining whether the primary key of the target field exists in the cache may be to first determine the primary key of the target field based on the target field, and then match the primary key of the target field with the value of the root node of each prefix tree stored in the cache. Determining the primary key of the target field based on the target field may be to directly use the name of the target field or a part of the name as the primary key of the target field, or may be to use a feature data in a data set related to the target field as the primary key of the target field, which is not limited here. Accordingly, in the case where the primary key of the target field exists in the cache, the value of the root node of the first prefix tree obtained from the cache is the primary key of the target field.

[0057] According to the embodiments of the present disclosure, the prefix tree can be implemented by means of an array, a hash table, etc., which is not limited here. For example, in the method of implementing the prefix tree using an array, each node of the prefix tree can be represented as an array, the value of the node is the array name of the array, the various child nodes of the node can be various elements of the array, and each element of the array can also be an array. For another example, in the method of implementing the prefix tree using a hash table, each node in the prefix tree can be a hash table, the primary key corresponding to the hash table can be the value of the node, the various child nodes of the node can be various primary keys included in the hash table, and each primary key can have its own corresponding hash table.

[0058] According to an embodiment of the present disclosure, based on the first prefix tree and the data set, determining the newly added data set and the invalid data set can be performed by traversing the data set and the first prefix tree respectively, judging whether each piece of data in the data set exists in the first prefix tree, and determining that the data not existing in the first prefix tree is the newly added data, and then judging whether each piece of data in the first prefix tree exists in the data set, and determining that the data not existing in the data set is the invalid data. The newly added data set can be a set consisting of the newly added data, and the invalid data set can be a set consisting of the invalid data.

[0059] According to an embodiment of the present disclosure, data updating of a database associated with a cache based on a newly added data set and an invalidated data set may include writing each newly added data in the newly added data set into the database, and logically deleting data in the database that matches each invalidated data in the invalidated data set. Logical deletion may be achieved, for example, by adding specific characters to the front end of the data to convert the data into a comment, which is not limited here.

[0060] According to an embodiment of the present disclosure, data belonging to the same field in a database can be heterogeneously formed into prefix trees and written into a cache for storage. When a data processing request is received for data update, a first prefix tree can be obtained from the cache according to the target field, and based on the first prefix tree and the data set related to the target field, an invalid data set can be determined from the first prefix tree, and a newly added data set can be determined from the data set, and then the newly added data set and the invalid data set can be used to update the data related to the target field in the database. By organizing the data in the database in the form of a prefix tree, a prefix tree can be used to implement the screening of newly added data and invalid data, so as to replace the method of using a hash table for data screening in the related art, so that the technical problem of low processing efficiency of data update in the related art can be at least partially overcome, the hashing time of the hash function and the time consumption caused by hash conflicts can be eliminated, and the processing efficiency of data update can be effectively improved.

[0061] Reference below Figures 4 to 7 , combined with specific embodiments Figure 3 The method shown is further explained.

[0062] According to an embodiment of the present disclosure, the leaf node of each path included in the first prefix tree may record the version number of the path.

[0063] According to an embodiment of the present disclosure, a data set may include multiple input parameter data.

[0064] According to an embodiment of the present disclosure, determining a newly added data set and an invalid data set based on the first prefix tree and the data set may include the following operations:

[0065] The version number configured in the first prefix tree is updated to obtain a target version number; for each input parameter data, the first prefix tree is used to traverse and match multiple characters included in the input parameter data to obtain a first matching result; when the first matching result indicates that the input parameter data exists in the first prefix tree, a first path in the first prefix tree that matches the input parameter data is determined; the target version number is used to update the value of the first leaf node included in the first path; when the first matching result indicates that the input parameter data does not exist in the first prefix tree, the input parameter data is added to a newly added data set; and based on the respective values ​​of multiple leaf nodes in the first prefix tree, an invalid data set is determined from the first prefix tree.

[0066] According to an embodiment of the present disclosure, the version number of the first prefix tree configuration may be recorded in the root node of the first prefix tree, or the version number of the first prefix tree configuration may also be recorded in a leaf node directly connected to the root node, which is not limited here.

[0067] According to an embodiment of the present disclosure, the version number configured in the first prefix tree is updated to obtain a target version number, which can be generated according to the update rule of the version number based on the version number originally configured in the first prefix tree. For example, the original version number can be composed of decimal numbers, such as 1, 5, 20, etc., and the update operation of the version number can be to add 1 to the original version number. For another example, the original version number can be composed of timestamp information, version identification information, etc., for example, it can be expressed as 1.01.02.200001010000, where 200001010000 can be expressed as timestamp information, and 1.01.02. can be expressed as version identification information, and the update operation of the version number can be to generate new version identification information based on the version identification information of the original version number, and then splice the new version identification information with the timestamp information of the current moment to obtain the target version number, and the target version number can be expressed as 1.01.03.200002022359, for example.

[0068] According to an embodiment of the present disclosure, after the target version number is obtained, the target version number may be used to replace the version number originally configured in the first prefix tree.

[0069] According to an embodiment of the present disclosure, the first prefix tree may include multiple node layers, each node layer includes at least one node, and the first node layer of the first prefix tree is the node layer where the root node of the first prefix tree is located. For the data stored in the first prefix tree, the first character of the data can be located as a node in the second node layer of the first prefix tree, the second character of the data can be located as a node in the third node layer of the first prefix tree, and the node of the second character of the data can be a child node of the node of the first character of the data, and so on.

[0070] According to an embodiment of the present disclosure, based on the data storage structure in the first prefix tree, for each input parameter data, using the first prefix tree to traverse and match multiple characters included in the input parameter data, obtaining a first matching result may include the following operations:

[0071] For each input parameter data, when it is determined that the value of the first node in the i-th node layer of the first prefix tree matches the i-1th character of the input parameter data, at least one second node connected to the first node is determined from the i+1th node layer of the first prefix tree, wherein i is an integer greater than 1, and i is less than or equal to the number of characters included in the input parameter data; the i-th character of the input parameter data is matched with the respective values ​​of at least one second node to obtain at least one second matching result; when at least one second matching result indicates an incomplete match, a first matching result indicating that the input parameter data does not exist in the first prefix tree is obtained; and when the last character of the input parameter data matches the third node of the first prefix tree, a first matching result is obtained based on the child node connected to the third node.

[0072] According to an embodiment of the present disclosure, when the prefix tree is implemented by data or a hash table, the time complexity of matching the i-th character of the input data with the respective values ​​of at least one second node can be O(1). For example, the first node can be represented as an array, the respective values ​​of at least one second node can be the respective values ​​of at least one element in the data, and matching the i-th character of the input data with the respective values ​​of at least one second node can be matching the i-th character of the input data with the respective values ​​of at least one element in the array, and the number of comparisons can be a constant level, that is, the time complexity can be O(1). For another example, the first node can be represented as a hash table, the respective values ​​of at least one second node can be represented as at least one primary key in the hash table, and matching the i-th character of the input data with the respective values ​​of at least one second node can be matching the i-th character of the input data with at least one primary key in the hash table.

[0073] According to an embodiment of the present disclosure, when there is a second matching result indicating that the match is completed in at least one second matching result, it can be determined that there is a target second node in at least one second node that matches the i-th character of the input parameter data. At this time, the above operation can be repeated to match the child node of the target second node with the i+1-th character of the input parameter data until all characters of the input parameter data have been matched.

[0074] According to an embodiment of the present disclosure, the input parameter data may be sub-data of the data already contained in the first prefix tree, that is, the input parameter data may be a part of the data in the first prefix tree. In the case where the last character of the input parameter data matches the third node of the first prefix tree, the third node may also be queried to determine whether the input parameter data is newly added data. Specifically, in the case where it is determined that there is a leaf node in the child node connected to the third node, a first matching result indicating that the input parameter data exists in the first prefix tree is obtained; and in the case where it is determined that there is no leaf node in the child node connected to the third node, a first matching result indicating that the input parameter data does not exist in the first prefix tree is obtained.

[0075] According to an embodiment of the present disclosure, for example, the input data is AB, and there is data ABC in the first prefix tree. Since data ABC includes the input data, the input data AB can be matched with the data ABC when matching with the first prefix tree. At this time, the child nodes of node B can be queried to see if there are leaf nodes. If there are leaf nodes, it means that data AB also exists in the first prefix tree. If there are no leaf nodes, it means that the input data AB can be new data relative to the first prefix tree.

[0076] The following describes a matching process between input parameter data and the first prefix tree in conjunction with a specific embodiment.

[0077] Figure 4 A schematic diagram of a first prefix tree according to an embodiment of the present disclosure is schematically shown.

[0078] like Figure 4As shown, the first prefix tree may be composed of the following data: BOB, BOY, BUY, BO, JACK. The root node of the first prefix tree may be the primary key of the target field, for example, field, and the root node may be located at the first node layer of the first prefix tree. The second node layer may include node B and node J. The third node layer may include node O, node U, and node A, node O may be a child node of node B of the second node layer, node U may be a child node of node B of the second node layer, and node A may be a child node of node J of the second node layer. The fourth node layer may include node B, two nodes Y, node C, and leaf node 1, node B may be a child node of node O of the third node layer, the first node Y may be a child node of node O of the third node layer, the second node Y may be a child node of node U of the third node layer, node C may be a child node of node A of the third node layer, and leaf node 1 may be a child node of node O of the third node layer. The fifth node layer may include leaf node 2, leaf node 3, leaf node 4 and node K, leaf node 2 may be a child node of node B of the fourth node layer, leaf node 3 may be a child node of the first node Y of the fourth node layer, leaf node 4 may be a child node of the second node Y of the fourth node layer, and node K may be a child node of node C of the fourth node layer. The sixth node layer may include leaf node 5, and leaf node 5 may be a child node of node K of the fifth node layer.

[0079] According to an embodiment of the present disclosure, the input parameter data may be, for example, BAD, and the input parameter data includes three characters, namely B, A and D. When matching the input parameter data, the character B may be matched with each node of the second node layer first, and the matching result obtained is that the character B matches the node B of the second node layer. Afterwards, the character A may be matched with each child node of the node B of the second node layer, that is, the node O of the third node layer and the node U of the third node layer, and the matching result obtained is that the character A does not match, thereby obtaining the first matching result that the input parameter data does not match the first prefix tree.

[0080] According to an embodiment of the present disclosure, for another example, the input parameter data may be BO. According to the above example, the character B matches the node B of the second node layer. When the character O is matched with each child node of the node B of the second node layer, the matching result obtained is that the character O matches the node O of the third node layer. By querying the child nodes of the node O of the third node layer, it can be determined that the child nodes of the node O of the third node layer include the leaf node 1, thereby obtaining the first matching result of the input parameter data matching the first prefix tree.

[0081] According to an embodiment of the present disclosure, the first path in the first prefix tree that matches the input parameter data can be composed of leaf nodes and nodes that match various parameters of the input parameter data. The first path can be a unidirectional link starting from the root node and ending at the leaf node.

[0082] According to an embodiment of the present disclosure, updating the value of the first leaf node included in the first path using the target version number may be replacing the value of the first leaf node with the target version number.

[0083] According to the embodiments disclosed this year, for each input parameter data, when it is determined that the first matching result of the input parameter data indicates that the input parameter data does not exist in the first prefix tree, the input parameter data can be added as new data to the newly added data set. Before performing traversal matching of the data set, the newly added data set can be an empty data set. During the traversal matching process, all input parameter data whose first matching results indicate that they do not match can be added to the empty data set to finally obtain the newly added data set.

[0084] According to an embodiment of the present disclosure, for each input parameter data, when the first matching result indicates that the input parameter data does not exist in the first prefix tree, the input parameter data can be written into the first prefix tree.

[0085] According to an embodiment of the present disclosure, the operation of writing the input parameter data into the first prefix tree can be performed synchronously with the operation of adding the input parameter data to the newly added data set, so as to reduce the number of traversals of the data set and the first prefix tree and improve data update efficiency.

[0086] According to an embodiment of the present disclosure, writing the input parameter data into the first prefix tree may specifically include the following operations:

[0087] When it is determined that the value of the fourth node in the j-th node layer of the first prefix tree matches the j-1-th character of the input parameter data, determine at least one fifth node associated with the fourth node from the j+1-th node layer of the first prefix tree, wherein j is an integer greater than 1, and j is less than or equal to the number of characters included in the input parameter data; match the j-th character of the input parameter data with the respective values ​​of at least one fifth node, respectively, to obtain at least one third matching result; when at least one third matching result indicates an incomplete match, generate a sixth node connected to the fourth node at the j+1-th node layer of the first prefix tree based on the j-th character of the input parameter data; and when the j-th character of the input parameter data is the last character of the input parameter data, generate a third leaf node connected to the sixth node at the j+2-th node layer of the first prefix tree based on the target version number.

[0088] According to an embodiment of the present disclosure, after the generation of the sixth node is completed, the sixth node can be used as a starting point, and nodes can be generated based on the j+1th character of the input parameter data, the j+2th character of the input parameter data, ..., and the last character of the input parameter data. Specifically, the generation method of the sixth node can be referred to, and based on the j+1th character of the input parameter data, a node connected to the sixth node can be generated at the j+2th node layer of the first prefix tree, and so on, until a node corresponding to the last character of the input parameter data is generated, after which a leaf node generated based on the target version number can be added to the child nodes of the node.

[0089] The following is a specific embodiment and Figure 4 The first prefix tree shown illustrates a process of writing the input parameter data into the first prefix tree. The prefix tree obtained after the input parameter data is written into the first prefix tree may be named, for example, a first intermediate prefix tree.

[0090] Figure 5 A schematic diagram of a first intermediate prefix tree according to an embodiment of the present disclosure is schematically shown.

[0091] like Figure 5 As shown, write the input parameter data BAD into Figure 4 After the first prefix tree shown, the first intermediate prefix tree can be obtained.

[0092] According to an embodiment of the present disclosure, the input data BAD may include character B, character A, and character D. When writing the input data BAD, since character A does not match any of the child nodes of node B of the second node layer, it can be determined that the starting position of the path growth is node B of the second node layer. When performing path growth, first, node A connected to node B of the second node layer can be generated at the third node layer, and then node D connected to node A of the third node layer can be generated at the fourth node layer. Then, leaf node 6 connected to node D of the fourth node layer can be generated at the fifth node layer, thereby completing the write operation of the input data BAD.

[0093] According to the embodiments of the present disclosure, after completing the traversal of the data set, the values ​​of each leaf node in the first prefix tree can be divided into two categories, one is the updated target version number, and the other is the original version number that has not been updated, so that it can be determined that the data to which the leaf node with the value of the original version number belongs may be data that does not exist in the data set of this data processing request, that is, invalid data. Specifically, based on the values ​​of each of the multiple leaf nodes in the first prefix tree, determining the invalid data set from the first prefix tree may include the following operations:

[0094] The target version number is used to traverse and match the respective values ​​of multiple leaf nodes to determine at least one second leaf node from the multiple leaf nodes, wherein the value of the second leaf node does not match the target version number; and based on the second path to which the at least one second leaf node belongs, an invalid data set is determined.

[0095] According to an embodiment of the present disclosure, based on the second path to which the second leaf node belongs, the data represented by the second path, ie, the failure data, can be determined, and the failure data set can be obtained by combining at least one piece of failure data.

[0096] According to an embodiment of the present disclosure, traversing the leaf nodes in the first prefix tree may be implemented by using any multi-branch tree traversal method, which is not limited here.

[0097] According to an embodiment of the present disclosure, the first prefix tree can be pruned based on the respective values ​​of multiple leaf nodes in the first prefix tree. That is, in the process of traversing the leaf nodes in the first prefix tree, when it is determined that the leaf node belongs to invalid data, the nodes in the first prefix tree that are only related to the invalid data can be deleted to implement pruning of the implementation data.

[0098] According to an embodiment of the present disclosure, the pruning process of the first prefix tree can be completed in an order from the leaf node upward. Specifically, based on the values ​​of each of the plurality of leaf nodes in the first prefix tree, the pruning process of the first prefix tree may include the following operations:

[0099] For each leaf node, when the value of the leaf node does not match the target version number, determine the seventh node connected to the leaf node and delete the leaf node; query to obtain the sibling nodes of the seventh node, wherein the sibling nodes of the seventh node have the same parent node as the seventh node; when it is determined that there is a leaf node among the sibling nodes of the seventh node, delete the seventh node to complete the pruning of the leaf nodes; and when it is determined that there is no leaf node among the sibling nodes of the seventh node, determine the eighth node connected to the seventh node, delete the seventh node, and continue to query the sibling nodes of the eighth node.

[0100] According to an embodiment of the present disclosure, the sibling node of the seventh node may refer to other nodes having the same parent node as the seventh node. For example, the seventh node may be located in the kth node layer of the first prefix tree, then the sibling node of the seventh node may also be located in the kth node layer of the first prefix tree, and the seventh node and the sibling node of the seventh node are both connected to a node in the k-1th node layer of the first prefix tree.

[0101] According to an embodiment of the present disclosure, when it is determined that there is a leaf node among the sibling nodes of the seventh node, it can be determined that the seventh node may be a common node between the current invalid data and other data. At this time, it can be determined that all nodes in the first prefix tree that are only related to the invalid data have been deleted, that is, the pruning process for the invalid data is completed.

[0102] According to an embodiment of the present disclosure, continuing to query the sibling nodes of the eighth node may be to query to obtain the sibling nodes of the eighth node, and continue to judge and process whether the sibling nodes of the eighth node include leaf nodes according to the aforementioned method.

[0103] According to an embodiment of the present disclosure, during the pruning process, if it is determined that the node currently being processed is a root node, it can be determined that the pruning process operation has been completed.

[0104] The following is a specific embodiment and Figure 5 The first intermediate prefix tree shown illustrates the process of removing invalid data from the first intermediate prefix tree. The prefix tree obtained after removing the invalid data from the first intermediate prefix tree may be named, for example, a second intermediate prefix tree.

[0105] Figure 6 A schematic diagram of a second intermediate prefix tree according to an embodiment of the present disclosure is schematically shown.

[0106] like Figure 6 As shown, in the traversal process of the first intermediate prefix tree, the leaf node whose determined value does not match the target version number may be leaf node 3, that is, the data BOY represented by the path to which leaf node 3 belongs is invalid data. In the pruning process for leaf node 3, since leaf node 3 is located in the fifth node layer, the node connected to leaf node 3 can be queried from the fourth node layer, and the node can be queried to be the first node Y of the fourth node layer. Afterwards, the brother node of the first node Y of the fourth node layer can be queried, and the query result is represented by the brother node of the first node Y of the fourth node layer including leaf node 1 and node B of the fourth node layer. Since leaf node 1 exists among the brother nodes of the first node Y of the fourth node layer, it means that the parent node of the node is not a node owned by the invalid data BOY alone. At this time, the first node Y of the fourth node layer can be deleted, and it is determined that the pruning process for leaf node 3 has been completed.

[0107] According to an embodiment of the present disclosure, by performing pruning processing on the first prefix tree, the amount of useless data contained in the first prefix tree can be reduced, and the storage pressure of the first prefix tree on the cache can be reduced.

[0108] Figure 7 The flowchart schematically shows a data processing method according to another embodiment of the present disclosure.

[0109] like Figure 7 As shown, the data processing method may include operations S701 to S712.

[0110] In operation S701, a data processing request is received.

[0111] In operation S702, it is determined whether the primary key of the target field exists in the cache. If it is determined that the primary key of the target field exists, operation S703 is performed. If it is determined that the primary key of the target field does not exist, operation S711 is performed.

[0112] In operation S703, a first prefix tree is obtained, and a target version number is determined.

[0113] In operation S704, each input parameter data in the data set is traversed in sequence.

[0114] In operation S705, it is determined whether the input parameter data exists in the first prefix tree. If it is determined that the input parameter data exists in the first prefix tree, operation S706 is performed. If it is determined that the input parameter data does not exist in the first prefix tree, operation S707 is performed.

[0115] In operation S706, the value of the leaf node related to the input parameter data in the first prefix tree is updated to the target version number. After completing operation S706, operation S708 is performed.

[0116] In operation S707, the input parameter data is written into the first prefix tree and included in the newly added data set.

[0117] In operation S708, it is determined whether the traversal of the data set is completed. If it is determined that the traversal of the data set is completed, operation S709 is performed. If it is determined that the traversal of the data set is not completed, the operation returns to operation S704.

[0118] In operation S709, the first prefix tree is traversed to find leaf nodes whose values ​​are not the target version number, perform pruning processing, and obtain an invalid data set.

[0119] In operation S710 , the database is updated using the newly added data set and the invalid data set.

[0120] In operation S711, the database is updated using the data set.

[0121] In operation S712, a second prefix tree is generated based on the data set with the target field as a root node, and the second prefix tree is written into a cache.

[0122] According to an embodiment of the present disclosure, generating the second prefix tree based on the data set can be implemented using the same or similar method as when writing the input parameter data into the first prefix tree, which will not be described in detail here.

[0123] According to an embodiment of the present disclosure, the determination of a newly added data set and the writing of the input data to the first prefix tree can be completed when the data set is traversed, and the time complexity thereof can be O(N*L), where N can represent the number of input data in the data set, and L can represent the average character length of the input data. The determination of an invalid data set and the pruning of invalid data in the first prefix tree can be completed when the first prefix tree is traversed, and the time complexity thereof can be O(M*L), where M can represent the amount of data in the first prefix tree. That is, the first prefix tree is used to complete the screening process of the newly added data and the invalid data, and the time complexity of the screening process is reduced to O(N*L)+O(M*L). Compared with the hash table method used in the related art, the delay caused by the hashing of the hash function and possible hash conflicts when searching for data is eliminated.

[0124] Figure 8 The block diagram schematically shows a data processing device according to an embodiment of the present disclosure.

[0125] like Figure 8 As shown, the data processing device 800 includes a receiving module 810 , an acquiring module 820 , a determining module 830 and a first processing module 840 .

[0126] The receiving module 810 is used to receive a data processing request, wherein the data processing request includes a target field and a data set related to the target field.

[0127] The acquisition module 820 is configured to acquire a first prefix tree from the cache based on the primary key of the target field when it is determined that the primary key of the target field exists in the cache.

[0128] The determination module 830 is used to determine a newly added data set and an invalid data set based on the first prefix tree and the data set.

[0129] The first processing module 840 is used to update the data in the database associated with the cache based on the newly added data set and the invalid data set respectively.

[0130] According to an embodiment of the present disclosure, a data set includes multiple input parameter data.

[0131] According to an embodiment of the present disclosure, the determination module 830 includes a first determination submodule, a second determination submodule, a third determination submodule, a fourth determination submodule, a fifth determination submodule, and a sixth determination submodule.

[0132] The first determination submodule is used to update the version number of the first prefix tree configuration to obtain a target version number.

[0133] The second determination submodule is used to traverse and match multiple characters included in each input parameter data using the first prefix tree to obtain a first matching result.

[0134] The third determination submodule is used to determine a first path matching the input parameter data in the first prefix tree when the first matching result indicates that the input parameter data exists in the first prefix tree.

[0135] The fourth determination submodule is used to update the value of the first leaf node included in the first path by using the target version number.

[0136] The fifth determining submodule is configured to add the input parameter data to the newly added data set when the first matching result indicates that the input parameter data does not exist in the first prefix tree.

[0137] The sixth determining submodule is used to determine the invalid data set from the first prefix tree based on the respective values ​​of the plurality of leaf nodes in the first prefix tree.

[0138] According to an embodiment of the present disclosure, the first prefix tree includes multiple node layers, the root node of the first prefix tree belongs to the first node layer of the first prefix tree, and each node layer includes at least one node.

[0139] According to an embodiment of the present disclosure, the second determining submodule includes a first determining unit, a second determining unit, a third determining unit, and a fourth determining unit.

[0140] The first determination unit is used to determine, for each input parameter data, at least one second node connected to the first node from the i+1th node layer of the first prefix tree when it is determined that the value of the first node in the i-th node layer of the first prefix tree matches the i-1th character of the input parameter data, wherein i is an integer greater than 1 and i is less than or equal to the number of characters included in the input parameter data.

[0141] The second determination unit is used to match the i-th character of the input parameter data with the respective values ​​of at least one second node to obtain at least one second matching result.

[0142] The third determining unit is configured to obtain a first matching result indicating that the input parameter data does not exist in the first prefix tree when at least one of the second matching results indicates that the matching is not completed.

[0143] The fourth determination unit is used to obtain a first matching result based on a child node connected to the third node when the last character of the input parameter data matches the third node of the first prefix tree.

[0144] According to an embodiment of the present disclosure, the fourth determining unit includes a first determining subunit and a second determining subunit.

[0145] The first determining subunit is used to obtain a first matching result indicating that the input parameter data exists in the first prefix tree when it is determined that there is a leaf node in the child node connected to the third node.

[0146] The second determining subunit is used to obtain a first matching result indicating that the input parameter data does not exist in the first prefix tree when it is determined that there is no leaf node in the child nodes connected to the third node.

[0147] According to an embodiment of the present disclosure, the sixth determining submodule includes a fifth determining unit and a sixth determining unit.

[0148] The fifth determination unit is used to use the target version number to traverse and match the values ​​of each of the multiple leaf nodes to determine at least one second leaf node from the multiple leaf nodes, wherein the value of the second leaf node does not match the target version number.

[0149] The sixth determining unit is used to determine the invalid data set based on the second path to which each of the at least one second leaf nodes belongs.

[0150] According to an embodiment of the present disclosure, the data processing device 800 further includes a first writing module.

[0151] The first writing module is used to write the input parameter data into the first prefix tree when the first matching result indicates that the input parameter data does not exist in the first prefix tree.

[0152] According to an embodiment of the present disclosure, the first writing module includes a first writing unit, a second writing unit, a third writing unit, and a fourth writing unit.

[0153] A first writing unit is used to determine at least one fifth node associated with the fourth node from the j+1th node layer of the first prefix tree when it is determined that the value of the fourth node in the jth node layer of the first prefix tree matches the j-1th character of the input parameter data, wherein j is an integer greater than 1 and j is less than or equal to the number of characters included in the input parameter data.

[0154] The second writing unit is used to match the j-th character of the input parameter data with the value of at least one fifth node respectively, to obtain at least one third matching result.

[0155] The third writing unit is used to generate a sixth node connected to the fourth node at the j+1th node layer of the first prefix tree based on the jth character of the input parameter data when at least one third matching result indicates an incomplete match.

[0156] The fourth writing unit is used to generate a third leaf node connected to the sixth node at the j+2th node layer of the first prefix tree based on the target version number when the jth character of the input parameter data is the last character of the input parameter data.

[0157] According to an embodiment of the present disclosure, the data processing device 800 further includes a second processing module.

[0158] The second processing module is used to perform pruning processing on the first prefix tree based on the respective values ​​of multiple leaf nodes in the first prefix tree.

[0159] According to an embodiment of the present disclosure, the second processing module includes a first processing unit, a second processing unit, a third processing unit, and a fourth processing unit.

[0160] The first processing unit is used to determine, for each leaf node, a seventh node connected to the leaf node when the value of the leaf node does not match the target version number, and delete the leaf node.

[0161] The second processing unit is configured to query and obtain a sibling node of the seventh node, wherein the sibling node of the seventh node has the same parent node as the seventh node.

[0162] The third processing unit is used to delete the seventh node when it is determined that there is a leaf node among the sibling nodes of the seventh node, so as to complete the pruning process of the leaf nodes.

[0163] The fourth processing unit is used to determine the eighth node connected to the seventh node when it is determined that there is no leaf node among the sibling nodes of the seventh node, delete the seventh node, and continue to query the sibling nodes of the eighth node.

[0164] According to an embodiment of the present disclosure, the data processing device 800 further includes a third processing module, a generating module and a second writing module.

[0165] The third processing module is used to update the database using the data set when it is determined that the primary key of the target field does not exist in the cache.

[0166] The generation module is used to generate a second prefix tree based on the data set with the primary key of the target field as the root node.

[0167] The second writing module is used to write the second prefix tree into the cache.

[0168] According to the embodiments of the present invention, any one or more of the modules, submodules, units, and subunits, or at least part of the functions of any one of them can be implemented in one module. According to the embodiments of the present invention, any one or more of the modules, submodules, units, and subunits can be split into multiple modules for implementation. According to the embodiments of the present invention, any one or more of the modules, submodules, units, and subunits can be at least partially implemented as hardware circuits, such as field programmable gate arrays (FPGAs), programmable logic arrays (PLAs), systems on chips, systems on substrates, systems on packages, application specific integrated circuits (ASICs), or can be implemented by hardware or firmware in any other reasonable way of integrating or packaging the circuit, or implemented in any one of the three implementation methods of software, hardware, and firmware, or in any appropriate combination of any of them. Alternatively, according to the embodiments of the present invention, one or more of the modules, submodules, units, and subunits can be at least partially implemented as computer program modules, and when the computer program modules are run, the corresponding functions can be performed.

[0169] For example, any multiple of the receiving module 810, the acquisition module 820, the determination module 830, and the first processing module 840 can be combined in one module / unit / subunit for implementation, or any one of the modules / units / subunits can be split into multiple modules / units / subunits. Alternatively, at least part of the functions of one or more of these modules / units / subunits can be combined with at least part of the functions of other modules / units / subunits and implemented in one module / unit / subunit. According to an embodiment of the present disclosure, at least one of the receiving module 810, the acquisition module 820, the determination module 830, and the first processing module 840 can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on a chip, a system on a substrate, a system on a package, an application specific integrated circuit (ASIC), or can be implemented by hardware or firmware such as any other reasonable way of integrating or packaging the circuit, or by any one of the three implementation methods of software, hardware, and firmware, or by a suitable combination of any of them. Alternatively, at least one of the receiving module 810 , the acquiring module 820 , the determining module 830 and the first processing module 840 may be at least partially implemented as a computer program module, and when the computer program module is executed, a corresponding function may be performed.

[0170] It should be noted that the data processing device part in the embodiments of the present disclosure corresponds to the data processing method part in the embodiments of the present disclosure. The description of the data processing device part specifically refers to the data processing method part, which will not be repeated here.

[0171] Fig. 9A block diagram of an electronic device suitable for implementing a data processing method according to an embodiment of the present disclosure is schematically shown. Fig. 9 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0172] like Fig. 9 As shown, the computer electronic device 900 according to the embodiment of the present disclosure includes a processor 901, which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM) 902 or the program loaded from the storage part 908 to the random access memory (RAM) 903. The processor 901 may include, for example, a general-purpose microprocessor (such as a CPU), an instruction set processor and / or a related chipset and / or a special-purpose microprocessor (for example, an application-specific integrated circuit (ASIC)), etc. The processor 901 may also include an onboard memory for caching purposes. The processor 901 may include a single processing unit or multiple processing units for performing different actions of the method flow according to the embodiment of the present disclosure.

[0173] In RAM 903, various programs and data required for the operation of electronic device 900 are stored. Processor 901, ROM 902 and RAM 903 are connected to each other via bus 904. Processor 901 performs various operations of the method flow according to the embodiment of the present disclosure by executing the program in ROM 902 and / or RAM 903. It should be noted that the program can also be stored in one or more memories other than ROM 902 and RAM 903. Processor 901 can also perform various operations of the method flow according to the embodiment of the present disclosure by executing the program stored in the one or more memories.

[0174] According to an embodiment of the present disclosure, the electronic device 900 may further include an input / output (I / O) interface 905, which is also connected to the bus 904. The electronic device 900 may further include one or more of the following components connected to the I / O interface 905: an input portion 906 including a keyboard, a mouse, etc.; an output portion 907 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage portion 908 including a hard disk, etc.; and a communication portion 909 including a network interface card such as a LAN card, a modem, etc. The communication portion 909 performs communication processing via a network such as the Internet. A drive 910 is also connected to the I / O interface 905 as needed. A removable medium 911, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 910 as needed, so that a computer program read therefrom is installed into the storage portion 908 as needed.

[0175] According to an embodiment of the present disclosure, the method flow according to an embodiment of the present disclosure can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable storage medium, and the computer program contains a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 909, and / or installed from the removable medium 911. When the computer program is executed by the processor 901, the above-mentioned functions defined in the system of the embodiment of the present disclosure are executed. According to an embodiment of the present disclosure, the system, equipment, device, module, unit, etc. described above can be implemented by a computer program module.

[0176] The present disclosure also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or may exist independently without being assembled into the device / apparatus / system. The above computer-readable storage medium carries one or more programs, and when the above one or more programs are executed, the method according to the embodiment of the present disclosure is implemented.

[0177] According to an embodiment of the present disclosure, the computer-readable storage medium may be a non-volatile computer-readable storage medium. For example, it may include, but is not limited to: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, apparatus, or device.

[0178] For example, according to an embodiment of the present disclosure, the computer-readable storage medium may include the ROM 902 and / or the RAM 903 described above and / or one or more memories other than the ROM 902 and the RAM 903 .

[0179] The embodiments of the present disclosure also include a computer program product, which includes a computer program, which contains program code for executing the method provided by the embodiments of the present disclosure. When the computer program product runs on an electronic device, the program code is used to enable the electronic device to implement the data processing method provided by the embodiments of the present disclosure.

[0180] When the computer program is executed by the processor 901, the above functions defined in the system / device of the embodiment of the present disclosure are executed. According to the embodiment of the present disclosure, the system, device, module, unit, etc. described above can be implemented by a computer program module.

[0181] In one embodiment, the computer program may be based on a tangible storage medium such as an optical storage device, a magnetic storage device, etc. In another embodiment, the computer program may also be transmitted and distributed in the form of a signal on a network medium, and downloaded and installed through the communication part 909, and / or installed from a removable medium 911. The program code contained in the computer program may be transmitted using any appropriate network medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.

[0182] According to an embodiment of the present disclosure, the program code for executing the computer program provided by the embodiment of the present disclosure can be written in any combination of one or more programming languages. Specifically, these computing programs can be implemented using high-level process and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, Java, C++, python, "C" language or similar programming languages. The program code can be executed entirely on the user computing device, partially on the user device, partially on the remote computing device, or entirely on the remote computing device or server. In the case of a remote computing device, the remote computing device can be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device (for example, using an Internet service provider to connect through the Internet).

[0183] The flowcharts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flowchart or block diagram may represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of the code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box may also occur in an order different from that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of boxes in the block diagram or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions. It can be understood by those skilled in the art that the features recorded in the various embodiments and / or claims of the present disclosure can be combined and / or combined in a variety of ways, even if such a combination or combination is not explicitly recorded in the present disclosure. In particular, without departing from the spirit and teaching of the present disclosure, the features described in the various embodiments and / or claims of the present disclosure may be combined and / or combined in a variety of ways. All of these combinations and / or combinations fall within the scope of the present disclosure.

[0184] The embodiments of the present disclosure are described above. However, these embodiments are only for illustrative purposes and are not intended to limit the scope of the present disclosure. Although the embodiments are described above separately, this does not mean that the measures in the various embodiments cannot be used in combination to advantage. The scope of the present disclosure is defined by the attached claims and their equivalents. Without departing from the scope of the present disclosure, those skilled in the art may make a variety of substitutions and modifications, which should all fall within the scope of the present disclosure.

Claims

1. A data processing method, comprising: receiving a data processing request, wherein the data processing request includes a target field and a data set related to the target field; When it is determined that the primary key of the target field exists in the cache, obtaining a first prefix tree from the cache based on the primary key of the target field; Determine a newly added data set and an invalid data set based on the first prefix tree and the data set; and Based on the newly added data set and the invalid data set, data in the database associated with the cache is updated respectively.

2. The method according to claim 1, wherein: The data set includes multiple input parameter data; The step of determining a newly added data set and an invalid data set based on the first prefix tree and the data set includes: Updating the version number of the first prefix tree configuration to obtain a target version number; For each piece of the input parameter data, use the first prefix tree to traverse and match multiple characters included in the input parameter data to obtain a first matching result; When the first matching result indicates that the input parameter data exists in the first prefix tree, determining a first path in the first prefix tree that matches the input parameter data; Using the target version number to update the value of the first leaf node included in the first path; If the first matching result indicates that the input parameter data does not exist in the first prefix tree, adding the input parameter data to the newly added data set; and The invalid data set is determined from the first prefix tree based on respective values ​​of a plurality of leaf nodes in the first prefix tree.

3. The method according to claim 2, wherein: The first prefix tree includes a plurality of node layers, the root node of the first prefix tree belongs to the first node layer of the first prefix tree, and each of the node layers includes at least one node; Wherein, for each of the input parameter data, using the first prefix tree to traverse and match multiple characters included in the input parameter data to obtain a first matching result includes: For each of the input parameter data, upon determining that the value of the first node in the i-th node layer of the first prefix tree matches the i-1-th character of the input parameter data, determine at least one second node connected to the first node from the i+1-th node layer of the first prefix tree, wherein i is an integer greater than 1, and i is less than or equal to the number of characters included in the input parameter data; Match the i-th character of the input parameter data with the respective values ​​of the at least one second node to obtain at least one second matching result; When the at least one second matching result indicates that the match is not completed, obtaining the first matching result indicating that the input parameter data does not exist in the first prefix tree; and When the last character of the input parameter data matches the third node of the first prefix tree, the first matching result is obtained based on the child nodes connected to the third node.

4. The method according to claim 3, wherein: In the case where the last character of the input parameter data matches the third node of the first prefix tree, obtaining the first matching result based on the child node connected to the third node includes: When it is determined that a leaf node exists in the child node connected to the third node, obtaining the first matching result indicating that the input parameter data exists in the first prefix tree; and When it is determined that there is no leaf node in the child nodes connected to the third node, the first matching result is obtained, indicating that the input parameter data does not exist in the first prefix tree.

5. The method according to claim 2, wherein: The determining the invalid data set from the first prefix tree based on respective values ​​of a plurality of leaf nodes in the first prefix tree comprises: traversing and matching the values ​​of each of the plurality of leaf nodes using the target version number to determine at least one second leaf node from the plurality of leaf nodes, wherein the value of the second leaf node does not match the target version number; and The invalid data set is determined based on the second path to which each of the at least one second leaf nodes belongs.

6. The method according to claim 2, further comprising: When the first matching result indicates that the input parameter data does not exist in the first prefix tree, the input parameter data is written into the first prefix tree.

7. The method according to claim 6, wherein: The step of writing the input parameter data into the first prefix tree comprises: In the case of determining that the value of the fourth node in the j-th node layer of the first prefix tree matches the j-1-th character of the input parameter data, determining at least one fifth node associated with the fourth node from the j+1-th node layer of the first prefix tree, wherein j is an integer greater than 1, and j is less than or equal to the number of characters included in the input parameter data; Match the j-th character of the input parameter data with the respective values ​​of the at least one fifth node respectively, to obtain at least one third matching result; When the at least one third matching result indicates an incomplete match, generating a sixth node connected to the fourth node at the j+1th node level of the first prefix tree based on the jth character of the input parameter data; and When the j-th character of the input parameter data is the last character of the input parameter data, based on the target version number, a third leaf node connected to the sixth node is generated at the j+2-th node layer of the first prefix tree.

8. The method according to claim 2, further comprising: The first prefix tree is pruned based on respective values ​​of a plurality of leaf nodes in the first prefix tree.

9. The method according to claim 8, wherein: The pruning of the first prefix tree based on respective values ​​of a plurality of leaf nodes in the first prefix tree comprises: For each of the leaf nodes, if the value of the leaf node does not match the target version number, determine the seventh node connected to the leaf node, and delete the leaf node; Query and obtain a sibling node of the seventh node, wherein the sibling node of the seventh node has the same parent node as the seventh node; When it is determined that there is a leaf node among the sibling nodes of the seventh node, the seventh node is deleted to complete the pruning process of the leaf node; and When it is determined that there is no leaf node among the sibling nodes of the seventh node, an eighth node connected to the seventh node is determined, the seventh node is deleted, and the sibling nodes of the eighth node are continued to be queried.

10. The method according to claim 1, further comprising: When it is determined that the primary key of the target field does not exist in the cache, updating the database using the data set; Taking the primary key of the target field as a root node, generating a second prefix tree based on the data set; as well as The second prefix tree is written into the cache.

11. A data processing device, comprising: A receiving module, configured to receive a data processing request, wherein the data processing request includes a target field and a data set related to the target field; an acquisition module, configured to acquire a first prefix tree from the cache based on the primary key of the target field when it is determined that the primary key of the target field exists in the cache; a determination module, configured to determine a newly added data set and an invalid data set based on the first prefix tree and the data set; and The first processing module is used to update data in a database associated with the cache based on the newly added data set and the invalid data set respectively.

12. An electronic device comprising: one or more processors; a memory for storing one or more instructions, Wherein, when the one or more instructions are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 10.

13. A computer-readable storage medium having executable instructions stored thereon, wherein when the executable instructions are executed by a processor, the processor is enabled to implement the method according to any one of claims 1 to 10.

14. A computer program product, comprising computer executable instructions, which are used to implement the method of any one of claims 1 to 10 when executed.

Citation Information

Patent Citations

  • Data updating method and device

    CN110209677A

  • Interest point data processing method and device, electronic equipment and storage medium

    CN113360789A

  • Prefix tree processing method and device, terminal and storage medium

    CN114676289A

  • Efficient storage and retrieval of time series data

    US10769132B1

  • Decentralized database optimizations

    US20190288850A1