Multi-source data integration method and device based on graph algorithm, platform and medium

Abstract

The embodiment of the invention discloses a multi-source data integration method and device based on a graph algorithm, a platform and a medium. The method comprises the steps that business data from multiple different channels are acquired, the business data comprise multiple pieces of consumer data, and the consumer data comprise multiple pieces of consumer attribute data; generating consumer identifiers respectively corresponding to the consumer data; grouping the plurality of consumer data based on a preset graph algorithm and a preset identity recognition rule to obtain a plurality of consumer data groups; for each consumer data group, generating a unified identity identifier corresponding to the consumer data group, and associating the unified identity identifier with each consumer identifier in the consumer data group to obtain an identifier integration graph; and for each identification integration graph, performing association integration processing on the consumer data associated with the identification integration graph to obtain omni-channel consumer integration data. According to the embodiment of the invention, the consumer data of the same consumer in different channels can be associated and integrated.

Description

Technical Field

[0001] This application relates to the technical field of data processing, and particularly to a multi-source data integration method, apparatus, platform and medium based on graph algorithms. Background Art

[0002] With the development of Internet technology, merchants have more and more sales channels. Understanding consumer data can help merchants further understand consumers' consumption habits, thereby optimizing marketing strategies and improving business performance.

[0003] However, the same consumer has different accounts on different channels, and the consumer data between different channels is not connected, resulting in difficulties in identifying the identity of the same consumer in multiple channels, making it difficult to integrate consumer data from different channels, and there is a problem of data islands in consumer data in different channels. Summary of the Invention

[0004] The embodiments of this application provide a multi-source data integration method, apparatus, platform and medium based on graph algorithms, which can integrate and process consumer data from different channels, and solve the problem of data islands in consumer data in different channels.

[0005] In a first aspect, the embodiments of this application provide a multi-source data integration method based on graph algorithms, which includes: Obtain business data from multiple different channels, the business data includes multiple pieces of consumer data, and the consumer data includes multiple pieces of consumer attribute data; Generate a consumer identifier corresponding to each piece of consumer data according to the channel identifier of the channel corresponding to each piece of consumer data; Perform grouping processing on multiple pieces of consumer data based on a preset graph algorithm and a preset identity recognition rule to obtain multiple consumer data groups; For each consumer data group, generate a unified identity identifier corresponding to the consumer data group, and associate the unified identity identifier with each consumer identifier in the consumer data group to obtain an identity integration graph; For each identity integration graph, perform associated integration processing on the consumer data associated with the identity integration graph to obtain full-channel consumer integration data.

[0006] In a second aspect, the embodiments of this application further provide a multi-source data integration apparatus based on graph algorithms, which includes: A transceiver unit, configured to obtain business data from multiple different channels, the business data includes multiple pieces of consumer data, and the consumer data includes multiple pieces of consumer attribute data; A processing unit, configured to generate respective consumer identifiers corresponding to the respective consumer data according to the channel identifiers of the channels corresponding to the respective consumer data; perform grouping processing on the multiple pieces of consumer data based on a preset graph algorithm and a preset identity recognition rule to obtain multiple consumer data groups; for each of the consumer data groups, generate a unified identity identifier corresponding to the consumer data group, and associate the unified identity identifier with each consumer identifier in the consumer data group to obtain an identifier integration graph; for each of the identifier integration graphs, perform associated integration processing on the consumer data associated with the identifier integration graph to obtain omnichannel consumer integration data.

[0007] In a third aspect, an embodiment of the present application further provides a customer data platform, which includes a memory and a processor, and a computer program is stored on the memory. When the processor executes the computer program, the above method is implemented.

[0008] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium, and a computer program is stored on the storage medium. The computer program includes program instructions, and when the program instructions are executed by a processor, the above method can be implemented.

[0009] An embodiment of the present application provides a multi-source data integration method, device, platform and medium based on a graph algorithm. Among them, the method includes: obtaining service data from multiple different channels, the service data includes multiple pieces of consumer data, and the consumer data includes multiple pieces of consumer attribute data; generating respective consumer identifiers corresponding to the respective consumer data according to the channel identifiers of the channels corresponding to the respective consumer data; performing grouping processing on the multiple pieces of consumer data based on a preset graph algorithm and a preset identity recognition rule to obtain multiple consumer data groups; for each of the consumer data groups, generate a unified identity identifier corresponding to the consumer data group, and associate the unified identity identifier with each consumer identifier in the consumer data group to obtain an identifier integration graph; for each of the identifier integration graphs, perform associated integration processing on the consumer data associated with the identifier integration graph to obtain omnichannel consumer integration data. An embodiment of the present application can group consumer data through a graph algorithm and a preset identity recognition rule to determine the consumer data (consumer data group) of the same consumer in different channels, and then perform associated integration on the consumer data of the same consumer in different channels, thereby solving the problem of data islands of consumer data in different channels. Description of the Drawings

[0010] To more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the drawings required for the description of the embodiments. Obviously, the drawings in the following description are some embodiments of the present application. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0011] Figure 1 It is a schematic flowchart of the multi-source data integration method based on graph algorithms provided by the embodiments of the present application; Figure 2 It is a schematic sub-flowchart of the multi-source data integration method based on graph algorithms provided by the embodiments of the present application; Figure 3 It is a schematic diagram of the data acquisition structure in the multi-source data integration method based on graph algorithms provided by the embodiments of the present application; Figure 4 It is another schematic sub-flowchart of the multi-source data integration method based on graph algorithms provided by the embodiments of the present application; Figure 5 It is a schematic diagram of the identity integration graph of the multi-source data integration method based on graph algorithms provided by the embodiments of the present application; Figure 6 It is a schematic diagram of the consumer omni-channel ID integration data of the multi-source data integration method based on graph algorithms provided by the embodiments of the present application; Figure 7 It is yet another schematic sub-flowchart of the multi-source data integration method based on graph algorithms provided by the embodiments of the present application; Figure 8 It is a schematic block diagram of the multi-source data integration device based on graph algorithms provided by the embodiments of the present application; Figure 9 It is a schematic block diagram of the customer data platform provided by the embodiments of the present application. Detailed implementation manners

[0012] The following will clearly and completely describe the technical solutions in the embodiments of the present application in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are some, but not all, of the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts fall within the scope of protection of the present application.

[0013] It should be understood that when used in this specification and the appended claims, the terms "comprising" and "including" indicate the presence of the described features, wholes, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components, and / or their combinations.

[0014] It should also be understood that the terms used in the specification of this application are only for the purpose of describing specific embodiments and are not intended to limit this application. As used in the specification of this application and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an", and "the" are intended to include the plural forms.

[0015] It should be further understood that the term "and / or" used in the specification of this application and the appended claims refers to any combination and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0016] The embodiments of this application provide a multi-source data integration method, device, platform, and medium based on graph algorithms.

[0017] The execution subject of the multi-source data integration method based on graph algorithms can be the multi-source data integration device provided by the embodiments of this application, or a customer data platform (Customer Data Platform, CDP) integrated with the multi-source data integration device based on graph algorithms. Among them, the multi-source data integration device based on graph algorithms can be implemented in a hardware or software manner, and the customer data platform can be a terminal or a server.

[0018] Among them, a customer data platform is deployed in the above customer data platform. Through the customer data platform provided by this application, consumer data from different channels can be integrated and processed, solving the problem of data islands of consumer data in different channels.

[0019] Figure 1 It is a schematic flowchart of the multi-source data integration method based on graph algorithms provided by the embodiments of this application. As Figure 1 shown, the method includes the following steps S110 - S150.

[0020] S110. Obtain business data from multiple different channels, where the business data includes multiple pieces of consumer data, and the consumer data includes multiple pieces of consumer attribute data.

[0021] In this embodiment, the multiple different channels include various online sales platforms, customer relationship management (Customer Relationship Management, CRM), and offline store systems.

[0022] In some embodiments, the method is applied to a customer data platform, and the obtained business data is stored in the data warehouse of the customer data platform. The business data includes first business data, second business data, and third business data; as Figure 2 shown, the obtaining of business data from multiple different channels includes: S1101. According to the preset data synchronization service, batch data synchronization processing is performed on the channel databases for storing the service data in each of the channels and the data warehouse, so as to obtain the first service data from multiple different channels.

[0023] This embodiment is used to batch synchronize structured data. The channel database is generally a relational database (such as MySQL, Oracle). The preset data synchronization service in this embodiment is a service with the function of batch data synchronization between databases. For example, Apache Spark (a distributed computing framework that supports large-scale data batch processing), or it can also be a self-developed data synchronization service according to specific data synchronization scenarios. The specific technology of the data synchronization service in this embodiment is not limited.

[0024] Among them, after obtaining the first service data from multiple different channels, it is also necessary to perform data cleaning on the first service data, including cleaning steps such as handling missing values, removing duplicates, and standardizing formats.

[0025] S1102. According to the preset Kafka message queue, collect the buried point data in each of the channels, so as to obtain the second service data from multiple different channels.

[0026] This embodiment is used to synchronize the behavior / event data of consumers. The buried point data in each channel (such as the buried point data of the corresponding APP) includes the buried point data of user behavior logs and / or device data, and supports distributed deployment and flexible routing strategies.

[0027] Among them, after obtaining the second service data from multiple different channels, it is also necessary to perform data cleaning on the second service data, including cleaning steps such as handling missing values, removing duplicates, and standardizing formats.

[0028] Among them, through this step, the service data in the data warehouse can be updated in real time, and the integrated data of all-channel consumers can be updated in real time based on the updated data, so as to ensure that the obtained consumer information is the latest consumer information.

[0029] S1103. Pull the interface data of each of the channels respectively through the preset ETL script and the preset time interval, so as to obtain the third service data from multiple different channels.

[0030] In this embodiment, the data of the third-party APIs of each channel can be pulled. For example, the platform regularly pulls the API data of each channel through a custom ETL script, and supports dynamic task scheduling.

[0031] After obtaining the third business data from multiple different channels, it is also necessary to clean the third business data, including cleaning steps such as handling missing values, removing duplicates, and standardizing formats.

[0032] Among them, the data acquisition structure in this embodiment is as Figure 3 shown.

[0033] S120. Generate corresponding consumer identifiers for each of the consumer data according to the channel identifiers of the channels corresponding to the respective consumer data.

[0034] In this embodiment, in order to distinguish the source channels of different consumer data and set a unique consumer identifier for consumer data from different channels on the platform, this embodiment generates corresponding consumer identifiers for each of the consumer data according to the channel identifiers of the channels corresponding to the consumer data.

[0035] For example, consumer data is obtained from three channels, namely channel A, channel B, and channel C. The channel identifier of channel A is: userId, the channel identifier of channel B is: memberId, and the channel identifier of channel C is: openId. Since multiple different consumer data are obtained from the same channel, at this time, it is necessary to generate consumer identifiers for each piece of consumer data. Further, this embodiment also combines the order of each piece of consumer data to generate consumer identifiers, so that the generated consumer identifiers can not only uniquely reflect the current consumer data, the source channel, but also the number of data entries.

[0036] For example, the consumer data after generating the consumer identifiers is as follows: Channel A (userId: 1, mobile: 1891111111x, name: Zhang San, email: zhansan@email.com); Channel B (memberId: 2, phone: 1891111111x, name: Wang Wu); Channel C (openId: 3, mobile: 1891111111x, name: Li Si, email: zhansan10@email.com); Channel A (userId: 4, mobile: 1891111222x, name: Liu Yi, email: liuyi@email.com); Channel A (userId: 5, mobile: 1891111333x, name: Chen Er, email: chener@email.com); The consumer identifiers corresponding to the above 5 pieces of consumer data are respectively: "userId: 1", "memberId: 2", "openId: 3", "userId", and "userId: 5".

[0037] S130. Group the multiple pieces of consumer data based on a preset graph algorithm and a preset identity recognition rule to obtain multiple groups of consumer data.

[0038] Specifically, as Figure 4 shown, in some embodiments, step S130 includes: S1301. Traverse the consumer attribute data in the multiple pieces of consumer data through the target consumer attribute data type specified by the identity recognition rule; Among them, each piece of consumer attribute data in the multiple pieces of consumer attribute data included in the consumer data in this embodiment corresponds to a different consumer attribute data type. For example, the consumer attribute data types are telephone numbers, names, or email addresses, etc.

[0039] The identity recognition rule in this embodiment can be obtained according to the configuration of the user. For example, according to the configuration of the user, the target consumer attribute data type specified in the identity recognition rule is a telephone number. At this time, the telephone number is used as the recognition field, and the consumer attribute data in each piece of consumer data is traversed to analyze whether the telephone numbers in each piece of consumer data are the same.

[0040] In addition, through configuration, other consumer attribute data types can also be set as the target consumer attribute data type. For example, an email address can also be used as the target consumer attribute data type. At this time, during traversal, the email address is used as the recognition field for traversal. The specific type of the target consumer attribute data type in this embodiment is not limited.

[0041] S1302. Group the consumer data obtained by traversal that contains the same target consumer attribute data into one group to obtain multiple groups of consumer data, where the target consumer attribute data is the consumer attribute data of the target consumer attribute data type.

[0042] In this embodiment, the consumer data obtained by traversal that contains the same target consumer attribute data is grouped into one group. For example, when the target consumer attribute data is a telephone number, the consumer data that contains the same telephone number is grouped into one group.

[0043] In the above example, the telephone numbers in the consumer data corresponding to userId: 1, memberId: 2, and openId: 3 are all "1891111111x". Since the telephone numbers are the same, these pieces of data are grouped into one group and regarded as the data of the same consumer.

[0044] S140. For each of the consumer data groups, generate a unified identity identifier corresponding to the consumer data group, and associate the unified identity identifier with each consumer identifier in the consumer data group to obtain an identifier integration graph.

[0045] In this embodiment, a unified identity identifier OneID is generated for each consumer data group. For example, Figure 5 as shown, the OneID generated for a certain consumer data group is OneID: 100 (unified identifier 100), and the associated consumer identifiers are respectively userId: 1 (consumer identifier 1), memberId: 2 (consumer identifier 2), and openId: 3 (consumer identifier 3).

[0046] S150. For each of the identifier integration graphs, perform an association integration process on the consumer data associated with the identifier integration graph to obtain omnichannel consumer integration data.

[0047] In this embodiment, after generating an identifier integration graph corresponding to each consumer (consumer data group), for each identifier integration graph, further perform an association integration process on the consumer data associated with the identifier integration graph.

[0048] Furthermore, the obtained omnichannel consumer integration data includes omnichannel consumer ID integration data and omnichannel consumer information integration data. For example, Figure 6 as shown, the omnichannel consumer ID integration data integrates all IDs of the same consumer across all channels. Specifically, by analyzing the upstream IDs of each consumer identifier in the identifier integration graph, the upstream IDs of each consumer identifier are added to the omnichannel consumer ID integration data. For example, Figure 6 as shown, the upstream ID of the openId of the online marketing platform official account is unionId (associated identifier), and the upstream ID of unionId is the brand member ID (brand member identifier). Then, the unionId and the brand member ID under the same phone number are added to the omnichannel consumer ID integration data.

[0049] Furthermore, when calculating the omnichannel consumer information integration data, for example, Figure 7 as shown, step S150 includes: S1501. Obtain the consumer attribute data of the consumer data associated with the identifier integration graph as candidate consumer attribute data; For example, for the following example: Channel A (userId: 1, mobile: 1891111111x, name: Zhang San, email: zhansan@email.com); Channel B (memberId: 2, phone: 1891111111x, name: Wang Wu); Channel C (openId: 3, mobile: 1891111111x, name: Li Si, email: zhansan10@email.com); The candidate consumer attribute data includes the name and email data in each piece of data.

[0050] S1502. Use the candidate consumer attribute data corresponding to the specified consumer attribute data type among the multiple pieces of candidate consumer attribute data as the consumer attribute data to be merged; For example, if the specified consumer attribute data types are name and email, at this time, the candidate consumer attribute data of the types name and email are the consumer attribute data to be merged. If the consumer data also includes other types of data, such as age, it is deleted.

[0051] S1503. Determine whether there are multiple pieces of consumer attribute data to be processed in the consumer attribute data to be merged, where the types of the consumer attribute data are the same but the contents are different; For example, in the above example, the name data types of userId: 1, memberId: 2, and openId: 3 are the same, but the contents are different. The email types of userId: 1 and openId: 3 are the same but the contents are different. At this time, the name in userId: 1, memberId: 2, and openId: 3 is the consumer attribute data to be processed, and the email in userId: 1 and openId: 3 is also the consumer attribute data to be processed.

[0052] S1504. If there is the consumer attribute data to be processed, determine the target consumer attribute data from the multiple pieces of consumer attribute data to be processed according to the preset consumer attribute data screening rules; Among them, if there is no consumer attribute data to be processed, the consumer attribute data to be merged can be directly processed by associative merging. If there is consumer attribute data to be processed, at this time, for each type of consumer attribute data to be processed, it is also necessary to further determine the target consumer attribute data from the multiple pieces of consumer attribute data to be processed.

[0053] In some embodiments, the target consumer attribute data can be determined through the following steps: Obtain the channel identifier corresponding to each piece of the to-be-processed consumer attribute data; determine the channel priority corresponding to each piece of the to-be-processed consumer attribute data according to the preset corresponding relationship between the channel and the priority and the channel identifier corresponding to each piece of the to-be-processed consumer attribute data; determine the to-be-processed consumer attribute data with the highest channel priority as the target consumer attribute data.

[0054] In this embodiment, a corresponding relationship between the channel and the priority is set, that is, different priorities are respectively set for multiple different channels in advance. After obtaining the channel identifier corresponding to the to-be-processed consumer attribute data in this embodiment, the corresponding channel priority can be determined according to this corresponding relationship, and finally, the to-be-processed consumer attribute data with the highest channel priority among the multiple pieces of to-be-processed consumer attribute data is determined as the target consumer attribute data.

[0055] In some other embodiments, the target consumer attribute data can also be determined through the following steps: Obtain the channel identifier corresponding to each piece of the to-be-processed consumer attribute data; determine the target channel priority sorting according to the preset corresponding relationship between each type of consumer attribute data and the channel priority sorting, and the type of consumer attribute data corresponding to each piece of the to-be-processed consumer attribute data; determine the target consumer attribute data according to the target channel priority sorting and the channel identifier corresponding to each piece of the to-be-processed consumer attribute data.

[0056] In this embodiment, different channel priority sortings are respectively set for different types of consumer attribute data. The target channel priority sorting can be determined according to the type of consumer attribute data of the currently to-be-processed consumer attribute data, and then the target consumer attribute data is determined according to the target channel priority sorting and the channel identifier corresponding to each piece of the to-be-processed consumer attribute data.

[0057] S1505. Perform an association integration process on the target consumer attribute data to obtain the omnichannel consumer integration data.

[0058] In this embodiment, after determining the target consumer attribute data, perform an association integration process on the target consumer attribute data according to the identification integration diagram, and add each piece of target consumer attribute data to the omnichannel consumer integration data of the corresponding identification integration diagram.

[0059] Further, in some embodiments, after performing the association integration process on the consumer data associated with the identification integration diagram to obtain the omnichannel consumer integration data, the method further includes: Generate a cross-platform user profile based on the omnichannel consumer integration data; obtain a user profile analysis request sent by a user terminal, and generate a user profile analysis report according to the cross-platform user profile; send the user profile analysis report to the user terminal.

[0060] Among them, the user profile analysis report can indicate the channels associated with different consumers, as well as information such as the proportion of consumers in each channel, so that the user can optimize the marketing strategy based on the obtained user profile analysis report.

[0061] In summary, in this embodiment, business data from multiple different channels is obtained. The business data includes multiple pieces of consumer data, and the consumer data includes multiple pieces of consumer attribute data; consumer identifiers corresponding to the respective pieces of consumer data are generated according to the channel identifiers of the channels corresponding to the respective pieces of consumer data; multiple pieces of the consumer data are grouped based on a preset graph algorithm and a preset identity recognition rule to obtain multiple consumer data groups; for each of the consumer data groups, a unified identity identifier corresponding to the consumer data group is generated, and the unified identity identifier is associated with each consumer identifier in the consumer data group to obtain an identity integration graph; for each of the identity integration graphs, the consumer data associated with the identity integration graph is subjected to associated integration processing to obtain omnichannel consumer integration data. The embodiment of the present application can group consumer data through a graph algorithm and a preset identity recognition rule to determine the consumer data (consumer data group) of the same consumer in different channels, and then perform associated integration on the consumer data of the same consumer in different channels, thereby solving the data island problem of consumer data in different channels. In addition, by integrating consumer data from different channels, the accuracy and consistency of consumer data are also improved.

[0062] Figure 8 It is a schematic block diagram of a multi-source data integration device 800 based on a graph algorithm provided by an embodiment of the present application. As Figure 8 shown, corresponding to the above multi-source data integration method based on a graph algorithm, the present application also provides a multi-source data integration device 800 based on a graph algorithm. The multi-source data integration device 800 based on a graph algorithm includes units for executing the above multi-source data integration method based on a graph algorithm, and the multi-source data integration device 800 based on a graph algorithm can be configured in a customer data platform. Specifically, please refer to Figure 8 . The multi-source data integration device 800 based on a graph algorithm includes a transceiver unit 801 and a processing unit 802.

[0063] The transceiver unit 801 is configured to obtain business data from multiple different channels, the business data including multiple pieces of consumer data, and the consumer data including multiple pieces of consumer attribute data; Multiple different channels include various online sales platforms, Customer Relationship Management (CRM), and offline store systems.

[0064] A processing unit 802 is configured to generate respective consumer identifiers corresponding to the respective consumer data based on the channel identifiers of the channels corresponding to the respective consumer data; group the multiple pieces of consumer data based on a preset graph algorithm and a preset identity recognition rule to obtain multiple groups of consumer data; for each group of consumer data, generate a unified identity identifier corresponding to the group of consumer data, and associate the unified identity identifier with each consumer identifier in the group of consumer data to obtain an identity integration graph; for each identity integration graph, perform an associated integration process on the consumer data associated with the identity integration graph to obtain omnichannel consumer integration data.

[0065] In order to distinguish the source channels of different consumer data and set a unique consumer identifier for consumer data from different channels on the platform, in this embodiment, respective consumer identifiers corresponding to the respective consumer data are generated based on the channel identifiers of the channels corresponding to the consumer data.

[0066] For example, consumer data is obtained from three channels, namely Channel A, Channel B, and Channel C. The channel identifier of Channel A is: userId, the channel identifier of Channel B is: memberId, and the channel identifier of Channel C is: openId. Since multiple different pieces of consumer data are obtained from the same channel, at this time, it is necessary to generate consumer identifiers for each piece of consumer data. Further, in this embodiment, the generation of consumer identifiers is also combined with the order of each piece of consumer data, so that the generated consumer identifiers can not only uniquely reflect the current consumer data, the source channel, but also the number of data items.

[0067] For example, the consumer data after generating the consumer identifiers is as follows: Channel A (userId: 1, mobile: 1891111111x, name: Zhang San, email: zhansan@email.com); Channel B (memberId: 2, phone: 1891111111x, name: Wang Wu); Channel C (openId: 3, mobile: 1891111111x, name: Li Si, email: zhansan10@email.com); Channel A (userId: 4, mobile: 1891111222x, name: Liu Yi, email: liuyi@email.com); Channel A (userId: 5, mobile: 1891111333x, name: Chen Er, email: chener@email.com); The consumer identifiers corresponding to the above 5 pieces of consumer data are respectively: "userId: 1", "memberId: 2", "openId: 3", "userId", and "userId: 5".

[0068] In some embodiments, when the processing unit 802 executes the step of grouping multiple pieces of the consumer data based on a preset graph algorithm and a preset identity recognition rule to obtain multiple groups of consumer data, it specifically is used for: Traverse the consumer attribute data in multiple pieces of the consumer data through the target consumer attribute data type specified by the identity recognition rule; group the consumer data obtained by traversal that contains the same target consumer attribute data into one group to obtain multiple groups of consumer data, where the target consumer attribute data is the consumer attribute data of the target consumer attribute data type.

[0069] Among them, each piece of consumer attribute data in multiple pieces of consumer attribute data included in the consumer data of this embodiment corresponds to a different consumer attribute data type, for example, the consumer attribute data type is a telephone number, a name, an email, etc.

[0070] The identity recognition rule in this embodiment can be obtained according to the configuration of the user. For example, according to the configuration of the user, the target consumer attribute data type specified in the identity recognition rule is a telephone number. At this time, the telephone number is used as the recognition field, and the consumer attribute data in each piece of consumer data is traversed to analyze whether the telephone numbers in each piece of consumer data are the same.

[0071] In addition, through configuration, other consumer attribute data types can also be set as the target consumer attribute data type. For example, an email can also be used as the target consumer attribute data type. At this time, during traversal, the email is used as the recognition field for traversal. The specific type of the target consumer attribute data type in this embodiment is not limited.

[0072] Group the consumer data obtained by traversal that contains the same target consumer attribute data into one group. For example, when the target consumer attribute data is a telephone number, group the consumer data that contains the same telephone number into one group.

[0073] In the above example, the phone numbers in the consumer data corresponding to userId: 1, memberId: 2, and openId: 3 are all "1891111111x". Since the phone numbers are the same, these pieces of data are grouped together and regarded as the data of the same consumer.

[0074] For each group of consumer data, a unified identity identifier OneID is generated. The OneID generated for a certain group of consumer data is OneID: 100, and the associated consumer identifiers are userId: 1, memberId: 2, and openId: 3 respectively.

[0075] After generating the identification integration diagram corresponding to each consumer (consumer data group), for each identification integration diagram, further perform an association integration process on the consumer data associated with the identification integration diagram.

[0076] In some embodiments, when the processing unit 802 executes the step of performing an association integration process on the consumer data associated with the identification integration diagram to obtain the omnichannel consumer integration data, it specifically is used for: Obtain the consumer attribute data of the consumer data associated with the identification integration diagram as candidate consumer attribute data; use the candidate consumer attribute data corresponding to the specified consumer attribute data type among the multiple candidate consumer attribute data as the consumer attribute data to be merged; determine whether there are multiple pieces of consumer attribute data to be processed with the same consumer attribute data type but different contents of the consumer attribute data among the consumer attribute data to be merged; if there are the consumer attribute data to be processed, determine the target consumer attribute data from the multiple pieces of consumer attribute data to be processed according to the preset consumer attribute data screening rules; perform an association integration process on the target consumer attribute data to obtain the omnichannel consumer integration data.

[0077] For example, for the following example: Channel A (userId: 1, mobile: 1891111111x, name: Zhang San, email: zhansan@email.com); Channel B (memberId: 2, phone: 1891111111x, name: Wang Wu); Channel C (openId: 3, mobile: 1891111111x, name: Li Si, email: zhansan10@email.com); The candidate consumer attribute data includes the name and email data in each piece of data.

[0078] For example, if the specified consumer attribute data types are name and email, at this time, the candidate consumer attribute data of types name and email are the consumer attribute data to be merged. If the consumer data also includes other types of data, such as age, it is deleted.

[0079] For example, in the above example, the name data types in userId: 1, memberId: 2, and openId: 3 are the same, but the contents are different. The email types in userId: 1 and openId: 3 are the same but the contents are different. At this time, the name in userId: 1, memberId: 2, and openId: 3 is the consumer attribute data to be processed, and the email in userId: 1 and openId: 3 is also the consumer attribute data to be processed.

[0080] Among them, if there is no consumer attribute data to be processed, the consumer attribute data to be merged can be directly associated and merged. If there is consumer attribute data to be processed, at this time, for each type of consumer attribute data to be processed, it is also necessary to further determine the target consumer attribute data from multiple pieces of consumer attribute data to be processed.

[0081] In some embodiments, when the processing unit 802 executes the step of determining the target consumer attribute data from multiple pieces of the consumer attribute data to be processed according to the preset consumer attribute data screening rule, it specifically is used for: Obtain the channel identifier corresponding to each piece of the consumer attribute data to be processed; determine the channel priority corresponding to each piece of the consumer attribute data to be processed according to the corresponding relationship between the preset channel and the priority and the channel identifier corresponding to each piece of the consumer attribute data to be processed; determine the consumer attribute data with the highest channel priority as the target consumer attribute data.

[0082] In this embodiment, a corresponding relationship between the channel and the priority is set, that is, different priorities are respectively set for multiple different channels in advance. After obtaining the channel identifier corresponding to the consumer attribute data to be processed in this embodiment, the corresponding channel priority can be determined according to this corresponding relationship, and finally, the consumer attribute data with the highest channel priority among multiple pieces of the consumer attribute data to be processed is determined as the target consumer attribute data.

[0083] In some embodiments, when the processing unit 802 executes the step of determining the target consumer attribute data from multiple pieces of the consumer attribute data to be processed according to the preset consumer attribute data screening rule, it specifically is used for: Obtain the channel identifier corresponding to each piece of to-be-processed consumer attribute data; determine the target channel priority sorting according to the corresponding relationship between each preset consumer attribute data type and the channel priority sorting, and the consumer attribute data type corresponding to each piece of to-be-processed consumer attribute data; determine the target consumer attribute data according to the target channel priority sorting and the channel identifier corresponding to each piece of to-be-processed consumer attribute data.

[0084] In this embodiment, different channel priority sortings are set for different consumer attribute data types. The target channel priority sorting can be determined according to the consumer attribute data type of the currently to-be-processed consumer attribute data, and then the target consumer attribute data can be determined based on the target channel priority sorting and the channel identifier corresponding to each piece of to-be-processed consumer attribute data.

[0085] In some embodiments, the method is applied to a customer data platform. The obtained service data is stored in the data warehouse of the customer data platform. The service data includes first service data, second service data, and third service data. When the transceiver unit 801 executes the step of obtaining service data from multiple different channels, it specifically is used for: Perform data batch synchronization processing on the channel database for storing the service data in each channel and the data warehouse according to a preset data synchronization service to obtain the first service data from multiple different channels; This embodiment is used to batch synchronize structured data. The channel database is generally a relational database (such as MySQL, Oracle). The preset data synchronization service in this embodiment is a service with the function of batch data synchronization between databases. For example, Apache Spark (a distributed computing framework that supports large-scale data batch processing), or it can also be a self-developed data synchronization service according to specific data synchronization scenarios. The specific technology of the data synchronization service in this embodiment is not limited.

[0086] Among them, after obtaining the first service data from multiple different channels, it is also necessary to perform data cleaning on the first service data, including cleaning steps such as processing missing values, removing duplicates, and standardizing formats.

[0087] Collect the buried point data in each channel according to a preset Kafka message queue to obtain the second service data from multiple different channels; This embodiment is used to synchronize the behavior / event data of consumers. The buried point data in each channel (such as the buried point data corresponding to the APP) includes the buried point data of user behavior logs and / or device data, and supports distributed deployment and flexible routing strategies.

[0088] After obtaining the second business data from multiple different channels, it is also necessary to clean the second business data, including cleaning steps such as handling missing values, removing duplicates, and standardizing formats.

[0089] Among them, through this step, the business data in the data warehouse can be updated in real time, and the omnichannel consumer integration data can be updated in real time based on the updated data to ensure that the obtained consumer information is the latest consumer information.

[0090] Pull the interface data of each of the channels through a preset ETL script and a preset time interval to obtain the third business data from multiple different channels.

[0091] In this embodiment, the data of the third-party APIs of each channel can be pulled. For example, the platform regularly pulls the API data of each channel through a custom ETL script to support dynamic task scheduling.

[0092] After obtaining the third business data from multiple different channels, it is also necessary to clean the third business data, including cleaning steps such as handling missing values, removing duplicates, and standardizing formats.

[0093] In some embodiments, after the processing unit 802 executes the step of associating and integrating the consumer data associated with the identity integration graph to obtain the omnichannel consumer integration data, it is further configured to: Generate a cross-platform user portrait according to the omnichannel consumer integration data; obtain a user portrait analysis request sent by the user terminal through the transceiver unit 801, generate a user portrait analysis report according to the cross-platform user portrait; and send the user portrait analysis report to the user terminal.

[0094] Among them, the user portrait analysis report can indicate information such as the channels associated with different consumers and the proportion of consumers in each channel, so that the user can optimize the marketing strategy based on the obtained user portrait analysis report.

[0095] In summary, the multi-source data integration device 800 based on the graph algorithm in this embodiment can group consumer data through the graph algorithm and a preset identity recognition rule to determine the consumer data (consumer data group) of the same consumer in different channels, and then perform associative integration on the consumer data of the same consumer in different channels, thereby solving the data island problem of consumer data in different channels.

[0096] It should be noted that those skilled in the art can clearly understand that the specific implementation processes of the above-mentioned multi-source data integration device 800 based on the graph algorithm and each unit can refer to the corresponding descriptions in the foregoing method embodiments. For the sake of convenience and conciseness of description, they will not be elaborated here.

[0097] The above multi-source data integration device based on the graph algorithm can be implemented in the form of a computer program, and this computer program can run on a customer data platform as shown in Figure 9 the following.

[0098] Please refer to Figure 9 , Figure 9 which is a schematic block diagram of a customer data platform provided by an embodiment of the present application. The customer data platform 900 can be a terminal or a server.

[0099] Referring to Figure 9 , the customer data platform 900 includes a processor 902, a memory, and a network interface 905 connected through a system bus 901. Among them, the memory can include a non-volatile storage medium 903 and an internal memory 904.

[0100] The non-volatile storage medium 903 can store an operating system 9031 and a computer program 9032. The computer program 9032 includes program instructions, and when these program instructions are executed, the processor 902 can be made to execute a multi-source data integration method based on a graph algorithm.

[0101] The processor 902 is used to provide computing and control capabilities to support the operation of the entire customer data platform 900.

[0102] The internal memory 904 provides an environment for the operation of the computer program 9032 in the non-volatile storage medium 903. When the computer program 9032 is executed by the processor 902, the processor 902 can be made to execute a multi-source data integration method based on a graph algorithm.

[0103] The network interface 905 is used for network communication with other devices. Those skilled in the art can understand that Figure 9 the structure shown in

[0104] is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the customer data platform 900 to which the solution of the present application is applied. The specific customer data platform 900 may include more or fewer components than those shown in the figure, or combine certain components, or have a different component layout. Obtain service data from multiple different channels, where the service data includes multiple pieces of consumer data, and the consumer data includes multiple pieces of consumer attribute data; Generate a consumer identifier corresponding to each piece of consumer data according to the channel identifier of the channel corresponding to each piece of consumer data; Group the multiple pieces of consumer data based on a preset graph algorithm and a preset identity recognition rule to obtain multiple groups of consumer data; For each group of consumer data, generate a unified identity identifier corresponding to the group of consumer data, and associate the unified identity identifier with each consumer identifier in the group of consumer data to obtain an identity integration graph; For each identity integration graph, perform an association integration process on the consumer data associated with the identity integration graph to obtain omnichannel consumer integration data.

[0105] It should be understood that in the embodiment of the present application, the processor 902 may be a central processing unit (CPU), and the processor 902 may also be other general-purpose processors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0106] Those of ordinary skill in the art can understand that all or part of the processes of the methods in the above embodiments can be completed by instructing relevant hardware through a computer program. The computer program includes program instructions, and the computer program can be stored in a storage medium, and the storage medium is a computer-readable storage medium. The program instructions are executed by at least one processor in the computer system to implement the process steps of the embodiments of the above methods.

[0107] Therefore, the present application also provides a storage medium. The storage medium may be a computer-readable storage medium. The storage medium stores a computer program, where the computer program includes program instructions. When the program instructions are executed by a processor, the processor is caused to execute the following steps: Obtain service data from multiple different channels, where the service data includes multiple pieces of consumer data, and the consumer data includes multiple pieces of consumer attribute data; Generate a consumer identifier corresponding to each piece of consumer data according to the channel identifier of the channel corresponding to each piece of consumer data; Group the multiple pieces of consumer data based on a preset graph algorithm and a preset identity recognition rule to obtain multiple groups of consumer data; For each of the consumer data groups, generate a unified identity identifier corresponding to the consumer data group, and associate the unified identity identifier with each consumer identifier in the consumer data group to obtain an identifier integration graph; For each of the identifier integration graphs, perform an association integration process on the consumer data associated with the identifier integration graph to obtain omnichannel consumer integration data.

[0108] The storage medium may be various computer-readable storage media such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a magnetic disk, or an optical disc that can store program code.

[0109] Those of ordinary skill in the art can realize that the units and algorithm steps of the examples described in conjunction with the embodiments disclosed herein can be implemented by electronic hardware, computer software, or a combination of the two. To clearly illustrate the interchangeability of hardware and software, the components and steps of the examples have been generally described according to their functions in the above description. Whether these functions are executed in a hardware or software manner depends on the specific application and design constraints of the technical solution. Professional technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of this application.

[0110] In several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of each unit is only a logical function division, and there may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed.

[0111] The steps in the method embodiments of this application can be adjusted, combined, and deleted according to actual needs. The units in the device embodiments of this application can be combined, divided, and deleted according to actual needs. In addition, the functional units in each embodiment of this application can be integrated into one processing unit, or each unit can exist physically alone, or two or more units can be integrated into one unit.

[0112] When the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to enable a customer data platform (which can be a personal computer, a terminal, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application.

[0113] As described above, the above are only specific implementation manners of this application, but the protection scope of this application is not limited thereto. Any person skilled in the art in the technical field disclosed by this application can easily think of various equivalent modifications or replacements within the technical scope disclosed by this application, and these modifications or replacements should all be covered within the protection scope of this application. Therefore, the protection scope of this application shall be subject to the protection scope of the claims.

Claims

1. A multi-source data integration method based on graph algorithms, characterized in that, Including: Obtain business data from multiple different channels, where the business data includes multiple pieces of consumer data, and the consumer data includes multiple pieces of consumer attribute data; Generate respective consumer identifiers corresponding to each piece of the consumer data according to the channel identifiers of the channels corresponding to each piece of the consumer data; Perform grouping processing on multiple pieces of the consumer data based on a preset graph algorithm and a preset identity recognition rule to obtain multiple consumer data groups; For each of the consumer data groups, generate a unified identity identifier corresponding to the consumer data group, and associate the unified identity identifier with each consumer identifier in the consumer data group to obtain an identity integration graph; For each of the identity integration graphs, perform associated integration processing on the consumer data associated with the identity integration graph to obtain omnichannel consumer integration data.

2. The method according to claim 1, wherein The performing grouping processing on multiple pieces of the consumer data based on a preset graph algorithm and a preset identity recognition rule to obtain multiple consumer data groups includes: Traverse the consumer attribute data in multiple pieces of the consumer data through the target consumer attribute data type specified by the identity recognition rule; Group the consumer data obtained by traversal that contains the same target consumer attribute data into one group to obtain multiple consumer data groups, where the target consumer attribute data is the consumer attribute data of the target consumer attribute data type.

3. The method according to claim 1, characterized in that, The performing associated integration processing on the consumer data associated with the identity integration graph to obtain omnichannel consumer integration data includes: Obtain the consumer attribute data of the consumer data associated with the identity integration graph as candidate consumer attribute data; Use the candidate consumer attribute data corresponding to the specified consumer attribute data type among multiple pieces of the candidate consumer attribute data as the consumer attribute data to be merged; Determine whether there are multiple pieces of consumer attribute data to be processed in the consumer attribute data to be merged that have the same consumer attribute data type but different contents of the consumer attribute data; If there are the consumer attribute data to be processed, determine target consumer attribute data from multiple pieces of the consumer attribute data to be processed according to a preset consumer attribute data screening rule; Perform associated integration processing on the target consumer attribute data to obtain the omnichannel consumer integration data.

4. The method according to claim 3, wherein The determining target consumer attribute data from multiple pieces of the consumer attribute data to be processed according to a preset consumer attribute data screening rule includes: Obtain the channel identifiers corresponding to each piece of the consumer attribute data to be processed; Determine the channel priorities corresponding to each piece of the consumer attribute data to be processed according to the corresponding relationship between the channels and priorities preset and the channel identifiers corresponding to each piece of the consumer attribute data to be processed; Determine the consumer attribute data to be processed with the highest channel priority as the target consumer attribute data.

5. The method according to claim 3, characterized in that, The determining target consumer attribute data from multiple pieces of the consumer attribute data to be processed according to a preset consumer attribute data screening rule includes: Obtain the channel identifiers corresponding to each piece of the consumer attribute data to be processed; Determine the target channel priority ranking according to the corresponding relationship between the preset data types of each consumer attribute and the channel priority ranking, and the data types of consumer attributes corresponding to each of the to-be-processed consumer attribute data. Determine the target consumer attribute data according to the target channel priority ranking and the channel identifiers corresponding to each of the to-be-processed consumer attribute data.

6. The method according to claim 1, wherein The method is applied to a customer data platform, and the obtained business data is stored in the data warehouse of the customer data platform. The business data includes first business data, second business data, and third business data; the obtaining of business data from multiple different channels includes: Perform batch data synchronization processing on the channel databases for storing the business data in each of the channels and the data warehouse according to a preset data synchronization service, so as to obtain the first business data from multiple different channels. Collect the buried point data in each of the channels according to a preset Kafka message queue, so as to obtain the second business data from multiple different channels. Pull the interface data of each of the channels through a preset ETL script and a preset time interval, so as to obtain the third business data from multiple different channels.

7. The method according to any one of claims 1 to 6, characterized in that, After performing the association and integration processing on the consumer data associated with the identification integration graph to obtain the omnichannel consumer integration data, the method further includes: Generate a cross-platform user portrait according to the omnichannel consumer integration data. Obtain a user portrait analysis request sent by a user terminal, and generate a user portrait analysis report according to the cross-platform user portrait. Send the user portrait analysis report to the user terminal.

8. A multi-source data integration device based on graph algorithms, characterized in that Including: A transceiver unit, configured to obtain business data from multiple different channels, where the business data includes multiple pieces of consumer data, and the consumer data includes multiple pieces of consumer attribute data. A processing unit, configured to generate a consumer identifier corresponding to each of the consumer data according to the channel identifier of the channel corresponding to each of the consumer data; perform grouping processing on the multiple pieces of consumer data based on a preset graph algorithm and a preset identity recognition rule to obtain multiple consumer data groups. For each of the consumer data groups, generate a unified identity identifier corresponding to the consumer data group, and associate the unified identity identifier with each consumer identifier in the consumer data group to obtain an identification integration graph; for each of the identification integration graphs, perform association and integration processing on the consumer data associated with the identification integration graph to obtain the omnichannel consumer integration data.

9. A customer data platform, comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the multi-source data integration method based on a graph algorithm according to any one of claims 1-7.

10. A storage medium, characterized in that, The storage medium stores a computer program, and the computer program includes program instructions, and when the program instructions are executed by a processor, the processor is caused to execute the multi-source data integration method based on a graph algorithm according to any one of claims 1-7.

Images