Method and device for correctly constructing ID atlas, equipment and storage medium

Abstract

The invention discloses a method for correctly constructing an ID map, which comprises the following steps of: collecting ID data in real time, sorting credibility of data collection channels based on a preset rule, and granting map modification authority in sequence according to the sorting; setting a version number for each piece of ID data through an optimistic lock mechanism, when updating occurs, adding 1 to the version number and then performing writing, and if conflicts occur, performing retry; in the process of obtaining the ID data, the father-son relationship of the ID data at the channel end is reserved, and when the father ID is robbed and migrated to a new map, the corresponding son ID follows the father ID together; and setting an atlas correction function, and correcting the atlas based on the correction function. According to the method, when ID data is collected for a plurality of source systems, the source systems carry out data retransmission and data sending sequence disruption, the correctness of ID map construction can be ensured, the system can carry out self-repair, the final consistency of data and the correctness of the map are ensured, and the ID can distinguish different channels based on the channel with high credibility.

Description

Technical Field

[0001] The present invention relates to the field of Internet algorithm applications, and in particular to a method, device and storage medium for correctly constructing an ID graph. Background Art

[0002] The current Internet system has realized an advanced function, which integrates multiple identity information such as mobile phone number, ID number, system user ID and device ID, and combines precise business rules and intelligent algorithms to perform ID-Mapping, thereby building a comprehensive ID association map. This map can connect the originally isolated data points to form a data panorama, providing strong support for application scenarios such as data analysis and marketing.

[0003] However, in the process of real-time collection, there are still some problems that have not been overcome so far. First, in certain scenarios, data loss often occurs, which may affect the integrity of the graph. Secondly, when data is lost or errors occur in the graph, it becomes relatively difficult to fix these problems due to the characteristics of real-time collection. In addition, if there is a disorder problem during data collection, it will also cause errors in graph construction, further affecting the accuracy of the data. Finally, once the ID data is cleaned up by the flow log during the graph construction process, it will be difficult to trace its original source.

[0004] Therefore, in order to solve these problems, it is a technical problem that technical personnel in this field urgently need to solve to propose an algorithm that can continuously optimize to improve the stability and accuracy of data acquisition, while enhancing the self-healing ability of the map to ensure that it can be repaired quickly and accurately even when data is lost or errors occur. Summary of the invention

[0005] To this end, the present invention proposes a method, device and storage medium for correctly constructing an ID graph, so as to support a wider range of business needs and data analysis work by constructing a more robust and reliable ID association graph.

[0006] To achieve the above objectives, in a first aspect, the present invention provides a method for correctly constructing an ID map, comprising:

[0007] Collect ID data in real time, sort the credibility of data collection channels based on preset rules, and grant map modification permissions according to the sorting sequence;

[0008] Through the optimistic locking mechanism, a version number is set for each ID data. When an update occurs, the version number is incremented by 1 and then written. If a conflict occurs, a retry is performed;

[0009] In the process of obtaining the ID data, the parent-child relationship of the ID data on the channel side is retained. When the parent ID is snatched and migrated to a new graph, the corresponding child ID follows it;

[0010] A spectrum correction function is set, and the spectrum is corrected based on the correction function.

[0011] Furthermore, the correction function includes:

[0012] Query the updated ID information within the set range, and check whether each ID modification is completed. If completed, end the modification;

[0013] If not completed, determine whether the ID is in the previously repaired ID map.

[0014] If not, query the graph through the GID of the ID and add it to the list of IDs to be processed.

[0015] Traverse the data graph to determine whether the GID is the same as the parent ID;

[0016] If they are different, query the parent node graph through the parent node GID and add it to the ID list to be processed;

[0017] The list of IDs to be processed is deduplicated, the core construction logic is called, and all IDs are traversed;

[0018] If it is not completed, refresh the GID to the GID of the effective channel parent node and traverse again;

[0019] If completed, all nodes are grouped by GID first, and then grouped by the same certificate type for traversal processing;

[0020] After the traversal is completed, the copyright numbers of all updated nodes are increased by 1 and written into the database;

[0021] The traversal is not finished. Only one ID type of the same type is allowed. If there are multiple IDs with low channel credibility, remove them, generate a new GID and write it in, and traverse again.

[0022] Furthermore, the correction function includes:

[0023] Query the updated ID information within the set range, and check whether each ID modification is completed. If completed, end the modification;

[0024] If the traversal is not completed, determine whether the ID is in the previously repaired ID map. If the ID is in the repaired ID map, traverse again.

[0025] If the ID is not in the repair ID map, query the map through the GID of the ID and add it to the list of IDs to be processed.

[0026] Traverse the data graph to determine whether the ID is the same as the GID of the parent ID. If they are different, query the parent node graph through the parent node GID and add it to the list of IDs to be processed;

[0027] The list of IDs to be processed is deduplicated, the core construction logic is called, and all IDs are traversed;

[0028] If the traversal is not completed, refresh the GID to the GID of the effective channel parent node and traverse again;

[0029] If completed, all nodes are grouped by GID first, and then grouped by the same certificate type for traversal processing;

[0030] After the traversal is completed, the copyright numbers of all updated nodes are increased by 1 and written into the database;

[0031] The traversal is not finished. Only one ID type of the same type is allowed. If there are multiple IDs with low channel credibility, remove them, generate a new GID and write it in, and traverse again.

[0032] Furthermore, synchronized data is obtained from the source system at a set time for offline batch verification. The offline synchronized data is converted into KV format, and the ID data in the database is converted into KV format, and data comparison is performed through a unified data format.

[0033] In a second aspect, the present invention also provides a device for constructing a correct ID map, comprising: a sorting unit, a first control unit, a second control unit, and a correction unit;

[0034] The sorting unit is used to collect ID data in real time, sort the credibility of data collection channels based on preset rules, and grant map modification permissions according to the sorting sequence;

[0035] The first control unit is used to set a version number for each ID data through an optimistic locking mechanism. When an update occurs, the version number is incremented by 1 and then written. If a conflict occurs, a retry is performed;

[0036] The second control unit is used to retain the parent-child relationship of the ID data on the channel side during the process of obtaining the ID data. When the parent ID is snatched and migrated to a new map, the corresponding child ID follows it;

[0037] The correction unit is used to set a graph correction function and correct the graph based on the correction function.

[0038] Furthermore, the correction function includes:

[0039] Query the updated ID information within the set range, and check whether each ID modification is completed. If completed, end the modification;

[0040] If not completed, determine whether the ID is in the previously repaired ID map.

[0041] If not, query the graph through the GID of the ID and add it to the list of IDs to be processed.

[0042] Traverse the data graph to determine whether the GID is the same as the parent ID;

[0043] If they are different, query the parent node graph through the parent node GID and add it to the ID list to be processed;

[0044] The list of IDs to be processed is deduplicated, the core construction logic is called, and all IDs are traversed;

[0045] If it is not completed, refresh the GID to the GID of the effective channel parent node and traverse again;

[0046] If completed, all nodes are grouped by GID first, and then grouped by the same certificate type for traversal processing;

[0047] After the traversal is completed, the copyright numbers of all updated nodes are increased by 1 and written into the database;

[0048] The traversal is not finished. Only one ID type of the same type is allowed. If there are multiple IDs with low channel credibility, remove them, generate a new GID and write it in, and traverse again.

[0049] Furthermore, the correction function includes:

[0050] Query the updated ID information within the set range, and check whether each ID modification is completed. If completed, end the modification;

[0051] If the traversal is not completed, determine whether the ID is in the previously repaired ID map. If the ID is in the repaired ID map, traverse again.

[0052] If the ID is not in the repair ID map, query the map through the GID of the ID and add it to the list of IDs to be processed.

[0053] Traverse the data graph to determine whether the ID is the same as the GID of the parent ID. If they are different, query the parent node graph through the parent node GID and add it to the list of IDs to be processed;

[0054] The list of IDs to be processed is deduplicated, the core construction logic is called, and all IDs are traversed;

[0055] If the traversal is not completed, refresh the GID to the GID of the effective channel parent node and traverse again;

[0056] If completed, all nodes are grouped by GID first, and then grouped by the same certificate type for traversal processing;

[0057] After the traversal is completed, the copyright numbers of all updated nodes are increased by 1 and written into the database;

[0058] The traversal is not finished. Only one ID type of the same type is allowed. If there are multiple IDs with low channel credibility, remove them, generate a new GID and write it in, and traverse again.

[0059] Furthermore, synchronized data is obtained from the source system at a set time for offline batch verification. The offline synchronized data is converted into KV format, and the ID data in the database is converted into KV format, and data comparison is performed through a unified data format.

[0060] In a third aspect, the present invention provides a computing device, comprising:

[0061] A memory for storing program instructions;

[0062] The processor is used to call the program instructions stored in the memory and execute the correct method for constructing an ID map described above according to the obtained program instructions.

[0063] In a fourth aspect, the present invention provides a computer-readable storage medium, including computer-readable instructions. When a computer reads and executes the computer-readable instructions, the above-mentioned method for correctly constructing an ID map is implemented. The readable medium may be multiple, and the multiple readable media can operate independently of each other.

[0064] Compared with the existing technology, this solution collects ID data in real time, sorts the credibility of data collection channels based on preset rules, and grants map modification permissions in order of sorting; through the optimistic locking mechanism, a version number is set for each ID data. When an update occurs, the version number is increased by 1 and written. If a conflict occurs, a retry is performed; in the process of obtaining ID data, the parent-child relationship of the ID data on the channel side is retained. When the parent ID is snatched and migrated to a new map, the corresponding child ID follows; a map correction function is set, and the map is corrected based on the correction function. When collecting ID data for multiple source systems, the source system resends the data and the order of data transmission is disrupted, which can ensure the correctness of the ID map construction. The system can perform self-repair to ensure the final consistency of the data and the correctness of the map, and the ID can distinguish different channels, with the channel with higher credibility as the standard. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] In order to more clearly illustrate the embodiments of the present disclosure or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0066] Figure 1 A schematic diagram of a process for constructing a correct ID map provided by the present invention;

[0067] Figure 2 A structural block diagram of a device for constructing a correct ID map provided by the present invention;

[0068] Figure 3 The present invention provides a schematic structural diagram of an electronic device. DETAILED DESCRIPTION

[0069] The embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present application, and cannot be interpreted as limiting the present application.

[0070] It will be understood by those skilled in the art that, unless otherwise stated, the singular forms "a", "an" and "the" used herein may also include plural forms. It should be further understood that the term "comprising" used in the specification of the present application refers to the presence of features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof. The term "and / or" used herein includes all or any unit and all combinations of one or more associated listed items.

[0071] For the convenience of description, some nouns or terms involved in the embodiments of the present application are explained below:

[0072] Optimistic locking: A concurrency control mechanism used to manage access to shared resources in a multi-user environment. It is based on the assumption that in most cases, multiple users rarely modify the same resource at the same time, so there is no need to strictly lock it every time it is accessed. The operation steps are as follows:

[0073] Step 1: Add a version number field to the data table and obtain the version number each time you read data.

[0074] Step 2: After reading the data, the user performs business operations and modifies the data.

[0075] Step 3: When submitting an update, check whether the version number has changed. If the version number has not changed, it means that no other user has modified the data during this period, and you can safely update the data and increase the version number. If the version number has changed, it means that the data has been modified by other users. At this time, you can refuse to update or re-read the latest data and try to update.

[0076] Step 4: If a version number conflict is detected, different strategies can be adopted, such as retrying the operation, notifying the user of the conflict, or using other mechanisms to resolve the conflict.

[0077] In order to make the objectives, technical solutions and advantages of the present application more clear, the specific implementation methods of the present application will be further described in detail below with reference to the accompanying drawings.

[0078] The core of the present invention is to provide a method for constructing a correct ID map. Figure 1 The figure shows a flow chart of the correct method for constructing the ID map provided in this embodiment. In order to better understand the method, Figure 1 The technical solution of this embodiment is described in detail:

[0079] Step S1: Collect ID data in real time, sort the credibility of data collection channels based on preset rules, and grant map modification permissions according to the sorting sequence;

[0080] Specifically, as shown in Table 1, there are many channels for collecting IDs. For example, when the ID mapping data from channels A and B modify the same graph at the same time, the system makes a judgment based on the credibility of the acquisition channel. Only channel A with high credibility can modify the graph, and the data from channel B is invalid.

[0081] channel Credibility Channel A 100 Channel B 99 ... ...

[0082] Table 1

[0083] Step S2: By using an optimistic locking mechanism, a version number is set for each ID data. When an update occurs, the version number is incremented by 1 and then written. If a conflict occurs, a retry is performed;

[0084] Specifically, the system collects various ID data in real time, and adopts an optimistic locking mechanism to ensure the correctness of graph construction in the case of high concurrency. Each ID has a version number, and each time it is updated, the version number is added by 1 before writing. If a conflict occurs, a retry is performed.

[0085] Step S3: In the process of obtaining the ID data, the parent-child relationship of the ID data on the channel side is retained. When the parent ID is snatched and migrated to a new graph, the corresponding child ID follows it;

[0086] Specifically, during the ID collection process, the parent-child relationship of the ID is retained;

[0087] For example, when collecting data through channel A, the primary key of mobile phone number 1 is ID number 1, then the parent ID of mobile phone number 1 is ID number 1, and the parent ID of ID number 1 is itself. Through redundant design, the blood relationship of ID on the channel side is retained, which can meet the needs of subsequent data comparison and traceability.

[0088] When ID snatching occurs, if the parent ID is snatched and migrated to the new graph, the corresponding child ID also needs to follow.

[0089] Step S4: Setting a spectrum correction function, and correcting the spectrum based on the correction function.

[0090] Specifically, after data is written, if it is a non-primary key query, it cannot be queried immediately and can only be found after refresh. The existing technology generally uses redis to lock multiple IDs. In the case of repeated data and high concurrency, this approach will result in a large number of lock acquisition failures, causing a sharp drop in program TPS and inability to provide normal services.

[0091] This embodiment adopts the above-mentioned version number design. For data that has not been successfully updated due to being invisible, the map is corrected after the data is refreshed through the correction function to achieve the final map correctness.

[0092] In a preferred embodiment, the correction function includes:

[0093] Query the updated ID information within the set range, and check whether each ID modification is completed. If completed, end the modification;

[0094] If not completed, determine whether the ID is in the previously repaired ID map.

[0095] If not, query the graph through the GID of the ID and add it to the list of IDs to be processed.

[0096] Traverse the data graph to determine whether the GID is the same as the parent ID;

[0097] If they are different, query the parent node graph through the parent node GID and add it to the ID list to be processed;

[0098] The list of IDs to be processed is deduplicated, the core construction logic is called, and all IDs are traversed;

[0099] If it is not completed, refresh the GID to the GID of the effective channel parent node and traverse again;

[0100] If completed, all nodes are grouped by GID first, and then grouped by the same certificate type for traversal processing;

[0101] After the traversal is completed, the copyright numbers of all updated nodes are increased by 1 and written into the database;

[0102] The traversal is not finished. Only one ID type of the same type is allowed. If there are multiple IDs with low channel credibility, remove them, generate a new GID and write it in, and traverse again.

[0103] In a preferred embodiment, the correction function includes:

[0104] Query the updated ID information within the set range, and check whether each ID modification is completed. If completed, end the modification;

[0105] If the traversal is not completed, determine whether the ID is in the previously repaired ID map. If the ID is in the repaired ID map, traverse again.

[0106] If the ID is not in the repair ID map, query the map through the GID of the ID and add it to the list of IDs to be processed.

[0107] Traverse the data graph to determine whether the ID is the same as the GID of the parent ID. If they are different, query the parent node graph through the parent node GID and add it to the list of IDs to be processed;

[0108] The list of IDs to be processed is deduplicated, the core construction logic is called, and all IDs are traversed;

[0109] If the traversal is not completed, refresh the GID to the GID of the effective channel parent node and traverse again;

[0110] If completed, all nodes are grouped by GID first, and then grouped by the same certificate type for traversal processing;

[0111] After the traversal is completed, the copyright numbers of all updated nodes are increased by 1 and written into the database;

[0112] The traversal is not finished. Only one ID type of the same type is allowed. If there are multiple IDs with low channel credibility, remove them, generate a new GID and write it in, and traverse again.

[0113] In a preferred embodiment, synchronized data is obtained from the source system at a set time and offline batch verification is performed. The offline synchronized data is converted into KV format, and the ID data in the database is converted into KV format, and data comparison is performed using a unified data format.

[0114] Specifically, since data collection is real-time, there is a possibility of data loss. To ensure data accuracy, it is necessary to synchronize T-1 data from the source system every day and perform offline batch verification to ensure that data is not lost. All offline synchronized data is converted into KV format and placed in the above intermediate table. The ID data in the database is also converted into KV format. Through the unified data format, data comparison is performed, which is also convenient for subsequent expansion.

[0115] In the scheme of this embodiment, by collecting ID data in real time, the credibility of data collection channels is sorted based on preset rules, and the map modification permission is granted in order of sorting; through the optimistic locking mechanism, a version number is set for each ID data, and when an update occurs, the version number is increased by 1 and written, and if a conflict occurs, a retry is performed; in the process of obtaining the ID data, the parent-child relationship of the ID data on the channel side is retained, and when the parent ID is snatched and migrated to a new map, the corresponding child ID follows; a map correction function is set, and the map is corrected based on the correction function. When collecting ID data from multiple source systems, the source system resends the data and the order of data transmission is disrupted, which can ensure the correctness of the ID map construction, and the system can perform self-repair to ensure the final consistency of the data and the correctness of the map, and the ID can distinguish different channels, with the channel with higher credibility as the standard.

[0116] In one embodiment, the present invention provides a device for constructing a correct ID map, see Figure 2 The figure shows a structural block diagram of a device for constructing a correct ID map provided by this embodiment, wherein the device comprises: a sorting unit 10, a first control unit 20, a second control unit 30, and a correction unit 40;

[0117] The sorting unit 10 is used to collect ID data in real time, sort the credibility of data collection channels based on preset rules, and grant map modification permissions according to the sorting sequence;

[0118] The first control unit 20 is used to set a version number for each ID data through an optimistic locking mechanism. When an update occurs, the version number is incremented by 1 and then written. If a conflict occurs, a retry is performed;

[0119] The second control unit 30 is used to retain the parent-child relationship of the ID data at the channel end during the process of acquiring the ID data. When the parent ID is snatched and migrated to a new map, the corresponding child ID follows it;

[0120] The correction unit 40 is used to set a graph correction function and correct the graph based on the correction function.

[0121] In the scheme of this embodiment, by collecting ID data in real time, the credibility of data collection channels is sorted based on preset rules, and the map modification permission is granted in order of sorting; through the optimistic locking mechanism, a version number is set for each ID data, and when an update occurs, the version number is increased by 1 and written, and if a conflict occurs, a retry is performed; in the process of obtaining the ID data, the parent-child relationship of the ID data on the channel side is retained, and when the parent ID is snatched and migrated to a new map, the corresponding child ID follows; a map correction function is set, and the map is corrected based on the correction function. When collecting ID data from multiple source systems, the source system resends the data and the order of data transmission is disrupted, which can ensure the correctness of the ID map construction, and the system can perform self-repair to ensure the final consistency of the data and the correctness of the map, and the ID can distinguish different channels, with the channel with higher credibility as the standard.

[0122] The device for correctly constructing an ID map in this embodiment is used to implement the aforementioned method for correctly constructing an ID map. Therefore, the specific implementation method of the device for correctly constructing an ID map can be seen in the specific implementation method of the method for correctly constructing an ID map in the previous text. For example, the sorting unit 10 is used to implement step S1 in the above-mentioned method for correctly constructing an ID map, and the first control unit 20 is used to implement step S2 in the above-mentioned method for correctly constructing an ID map, etc. Therefore, its specific implementation method can refer to the description of the corresponding embodiments of each part. The device provided by the present invention is attached to the method provided by the present invention, and its specific effects can refer to the above-mentioned method, which will not be repeated here.

[0123] In one embodiment, a computing device is provided, comprising:

[0124] A memory for storing program instructions;

[0125] A processor is used to implement the steps of any of the above-mentioned methods for correctly constructing an ID map when executing the computer program.

[0126] As an example, Figure 3 A schematic diagram of the structure of an electronic device applicable to an embodiment of the present application is shown. Figure 3 As shown, the electronic device 2000 includes: a processor 2001 and a memory 2003. The processor 2001 and the memory 2003 are connected, such as through a bus 2002. Optionally, the electronic device 2000 may also include a transceiver 2004. It should be noted that in actual applications, the transceiver 2004 is not limited to one, and the structure of the electronic device 2000 does not constitute a limitation on the embodiments of the present application.

[0127] The processor 2001 is applied in the embodiment of the present application to implement the method shown in the above method embodiment. The transceiver 2004 may include a receiver and a transmitter. The transceiver 2004 is applied in the embodiment of the present application to implement the function of the electronic device of the embodiment of the present application communicating with other devices when executed.

[0128] Processor 2001 may be a CPU (Central Processing Unit), a general purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array) or other programmable logic devices, transistor logic devices, hardware components or any combination thereof. It may implement or execute various exemplary logic blocks, modules and circuits described in conjunction with the disclosure of this application. Processor 2001 may also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, etc.

[0129] The bus 2002 may include a path to transmit information between the above components. The bus 2002 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus. The bus 2002 may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 3 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0130] The memory 2003 can be a ROM (Read Only Memory) or other types of static storage devices that can store static information and instructions, a RAM (Random Access Memory) or other types of dynamic storage devices that can store information and instructions, or an EEPROM (Electrically Erasable Programmable Read Only Memory), a CD-ROM (Compact Disc Read Only Memory) or other optical disk storage, optical disk storage (including compressed optical disk, laser disk, optical disk, digital versatile disk, Blu-ray disk, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited to these.

[0131] Optionally, the memory 2003 is used to store application code for executing the solution of the present application, and the execution is controlled by the processor 2001. The processor 2001 is used to execute the application code stored in the memory 2003 to implement the investment and financing project recommendation method provided in any embodiment of the present application.

[0132] The electronic device provided in the embodiment of the present application is applicable to any embodiment of the above method and will not be described in detail here.

[0133] The method for constructing a correct ID map provided by the present invention has obvious advantages and beneficial effects compared with the prior art. It collects ID data in real time, sorts the credibility of data collection channels based on preset rules, and grants map modification permissions in sequence according to the sorting; sets a version number for each ID data through an optimistic locking mechanism, and when an update occurs, the version number is added by 1 and written, and if a conflict occurs, a retry is performed; in the process of obtaining the ID data, the parent-child relationship of the ID data on the channel side is retained, and when the parent ID is snatched and migrated to a new map, the corresponding child ID follows; a map correction function is set, and the map is corrected based on the correction function. It is realized that when collecting ID data from multiple source systems, the source system resends data and the data sending order is disrupted, which can ensure the correctness of the ID map construction, and the system can perform self-repair to ensure the final consistency of the data and the correctness of the map, and the ID can distinguish different channels, with the channel with high credibility as the standard.

[0134] In one embodiment, a computer-readable storage medium is provided, including computer-readable instructions. When a computer reads and executes the computer-readable instructions, the steps of the method for correctly constructing an ID map described above are implemented. The method for correctly constructing an ID map provided by the present invention has obvious advantages and beneficial effects compared with the prior art. The ID data is collected in real time, the credibility of the data collection channels is sorted based on preset rules, and the map modification permission is granted in sequence according to the sorting; a version number is set for each ID data through an optimistic locking mechanism, and when an update occurs, the version number is added by 1 and written, and a retry is performed if a conflict occurs; in the process of obtaining the ID data, the parent-child relationship of the ID data on the channel side is retained, and when the parent ID is snatched and migrated to a new map, the corresponding child ID follows together; a map correction function is set, and the map is corrected based on the correction function. When collecting ID data from multiple source systems, the source systems resend data and disrupt the order of data transmission, which can ensure the correctness of ID map construction. The system can perform self-repair to ensure the final consistency of data and the correctness of the map. The ID can distinguish different channels, with the most credible channel being the standard.

[0135] Through the description of the above implementation methods, technical personnel in the relevant field can understand that for the convenience and simplicity of description, only the division of the above-mentioned functional modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the specific device can be divided into different functional modules to complete all or part of the functions described above.

[0136] Those of ordinary skill in the art will appreciate that the modules and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in terms of function in the above description. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0137] In several embodiments covered by the present application, it should be understood that the disclosed devices, equipment and methods can be implemented in other ways. For example, the division of modules is only a logical function division, and there may be other division methods in actual implementation, such as multiple modules or components can be combined or integrated into another device, or some features can be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or modules, or it can be an electrical, mechanical or other form of connection.

[0138] The modules described as separate components may or may not be physically separated, and the components displayed as modules may or may not be physical modules, that is, they may be located in one place or distributed on multiple network modules. Some or all of the modules may be selected according to actual needs to achieve the purpose of the embodiments of the present application.

[0139] In addition, each functional module in each embodiment of the present application can be implemented in the form of hardware or in the form of software functional modules. If these functional modules are implemented in the form of software functional modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the present application is essentially or part of the prior art that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a computer program product, which includes one or more computer instructions. When loading and executing computer program instructions on a computer, the process or function according to the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer program product is stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, a computer instruction can be transmitted from a website site, a computer, a server or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or a data center that includes one or more available media integrations. The available media may be magnetic media (eg, floppy disks, hard disks, magnetic tapes), optical media (eg, DVDs), or semiconductor media (eg, solid state disks (SSDs)).

[0140] The above contents are merely examples and explanations of the structure of the present invention. The technicians in this technical field may make various modifications or additions to the specific embodiments described or replace them in a similar manner. As long as they do not deviate from the structure of the invention or exceed the scope defined by the claims, they should all fall within the protection scope of the present invention.

Claims

1. A method for constructing a correct ID map, characterized in that: include: Collect ID data in real time, sort the credibility of data collection channels based on preset rules, and grant map modification permissions according to the sorting sequence; Through the optimistic locking mechanism, a version number is set for each ID data. When an update occurs, the version number is incremented by 1 and then written. If a conflict occurs, a retry is performed; In the process of obtaining the ID data, the parent-child relationship of the ID data on the channel side is retained. When the parent ID is snatched and migrated to a new graph, the corresponding child ID follows it; A spectrum correction function is set, and the spectrum is corrected based on the correction function.

2. The method for constructing a correct ID map according to claim 1, characterized in that: The correction function includes: Query the updated ID information within the set range, and check whether each ID modification is completed. If completed, end the modification; If not completed, determine whether the ID is in the previously repaired ID map. If not, query the graph through the GID of the ID and add it to the list of IDs to be processed. Traverse the data graph to determine whether the GID is the same as the parent ID; If they are different, query the parent node graph through the parent node GID and add it to the ID list to be processed; The list of IDs to be processed is deduplicated, the core construction logic is called, and all IDs are traversed; If it is not completed, refresh the GID to the GID of the effective channel parent node and traverse again; If completed, all nodes are grouped by GID first, and then grouped by the same certificate type for traversal processing; After the traversal is completed, the copyright numbers of all updated nodes are increased by 1 and written into the database; The traversal is not finished. Only one ID type of the same type is allowed. If there are multiple IDs with low channel credibility, remove them, generate a new GID and write it in, and traverse again.

3. The method for constructing a correct ID map according to claim 2, characterized in that: The correction function includes: Query the updated ID information within the set range, and check whether each ID modification is completed. If completed, end the modification; If the traversal is not completed, determine whether the ID is in the previously repaired ID map. If the ID is in the repaired ID map, traverse again. If the ID is not in the repair ID map, query the map through the GID of the ID and add it to the list of IDs to be processed. Traverse the data graph to determine whether the ID is the same as the GID of the parent ID. If they are different, query the parent node graph through the parent node GID and add it to the list of IDs to be processed; The list of IDs to be processed is deduplicated, the core construction logic is called, and all IDs are traversed; If the traversal is not completed, refresh the GID to the GID of the effective channel parent node and traverse again; If completed, all nodes are grouped by GID first, and then grouped by the same certificate type for traversal processing; After the traversal is completed, the copyright numbers of all updated nodes are increased by 1 and written into the database; The traversal is not finished. Only one ID type of the same type is allowed. If there are multiple IDs with low channel credibility, remove them, generate a new GID and write it in, and traverse again.

4. The method for constructing a correct ID map according to claim 1, characterized in that: At the set time, synchronized data is obtained from the source system and offline batch verification is performed. The offline synchronized data is converted into KV format, and the ID data in the database is converted into KV format. Data comparison is performed through a unified data format.

5. A device for constructing a correct ID map, characterized in that: The device comprises: a sorting unit, a first control unit, a second control unit, and a correction unit; The sorting unit is used to collect ID data in real time, sort the credibility of data collection channels based on preset rules, and grant map modification permissions according to the sorting sequence; The first control unit is used to set a version number for each ID data through an optimistic locking mechanism. When an update occurs, the version number is incremented by 1 and then written. If a conflict occurs, a retry is performed; The second control unit is used to retain the parent-child relationship of the ID data on the channel side during the process of obtaining the ID data. When the parent ID is snatched and migrated to a new map, the corresponding child ID follows it; The correction unit is used to set a graph correction function and correct the graph based on the correction function.

6. The device for constructing a correct ID map according to claim 5, characterized in that: The correction function includes: Query the updated ID information within the set range, and check whether each ID modification is completed. If completed, end the modification; If not completed, determine whether the ID is in the previously repaired ID map. If not, query the graph through the GID of the ID and add it to the list of IDs to be processed. Traverse the data graph to determine whether the GID is the same as the parent ID; If they are different, query the parent node graph through the parent node GID and add it to the ID list to be processed; The list of IDs to be processed is deduplicated, the core construction logic is called, and all IDs are traversed; If it is not completed, refresh the GID to the GID of the effective channel parent node and traverse again; If completed, all nodes are grouped by GID first, and then grouped by the same certificate type for traversal processing; After the traversal is completed, the copyright numbers of all updated nodes are increased by 1 and written into the database; The traversal is not finished. Only one ID type of the same type is allowed. If there are multiple IDs with low channel credibility, remove them, generate a new GID and write it in, and traverse again.

7. The device for constructing a correct ID map according to claim 6, characterized in that: The correction function includes: Query the updated ID information within the set range, and check whether each ID modification is completed. If completed, end the modification; If the traversal is not completed, determine whether the ID is in the previously repaired ID map. If the ID is in the repaired ID map, traverse again. If the ID is not in the repair ID map, query the map through the GID of the ID and add it to the list of IDs to be processed. Traverse the data graph to determine whether the ID is the same as the GID of the parent ID. If they are different, query the parent node graph through the parent node GID and add it to the list of IDs to be processed; The list of IDs to be processed is deduplicated, the core construction logic is called, and all IDs are traversed; If the traversal is not completed, refresh the GID to the GID of the effective channel parent node and traverse again; If completed, all nodes are grouped by GID first, and then grouped by the same certificate type for traversal processing; After the traversal is completed, the copyright numbers of all updated nodes are increased by 1 and written into the database; The traversal is not finished. Only one ID type of the same type is allowed. If there are multiple IDs with low channel credibility, remove them, generate a new GID and write it in, and traverse again.

8. The device for constructing a correct ID map according to claim 5, characterized in that: At the set time, synchronized data is obtained from the source system and offline batch verification is performed. The offline synchronized data is converted into KV format, and the ID data in the database is converted into KV format. Data comparison is performed through a unified data format.

9. A computing device, characterized in that include: A memory for storing program instructions; A processor is used to call the program instructions stored in the memory, and execute the method according to any one of claims 1 to 4 according to the obtained program instructions.

10. A computer-readable storage medium, characterized in that: It includes computer-readable instructions. When a computer reads and executes the computer-readable instructions, it implements the method as claimed in any one of claims 1 to 4. There can be multiple readable media, and the multiple readable media can operate independently of each other.

Images