Data asset sharing method and system for multi-source heterogeneous data source

Abstract

The invention relates to the technical field of data processing, in particular to a data asset sharing method and system for multi-source heterogeneous data sources. The method comprises the following steps: firstly, clustering all displayed advertisements according to a difference condition of user characteristic indexes of all displayed advertisements and a difference condition of advertisement query durations to obtain each cluster; according to the numerical value distribution condition of each to-be-dimensionally-reduced index in all the clustering clusters, obtaining the retention metric of each to-be-dimensionally-reduced index; and according to the retention metric of each index to be subjected to dimensionality reduction, performing dimensionality reduction on the data set to be subjected to dimensionality reduction, obtaining a dimensionality-reduced data set of the advertisement category, and performing data resource sharing. According to the method, reasonable dimension reduction is carried out on the data set to be subjected to dimension reduction by fully considering the user preference condition influencing the advertisement effect, so that the data set after dimension reduction can effectively reflect the advertisement effect, and the value of data resource sharing is guaranteed.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a data asset sharing method and system for multi-source heterogeneous data sources. Background Art

[0002] Data resource sharing refers to the behavior of directly exchanging and sharing data between different users or systems. The cloud platform provides unified data storage and management services. Users can upload data to the cloud platform and share data through the cloud platform. Data from multi-source heterogeneous data sources refers to data from different sources with different structures and types. Advertising data is a typical multi-source heterogeneous data source.

[0003] When sharing advertising data resources, dimensionality reduction is an important step. Dimensionality reduction can significantly reduce the dimension of data, thereby reducing data processing costs. PCA (Principal Component Analysis) is a commonly used dimensionality reduction technology. The existing technology uses PCA to reduce the dimensionality of advertising data. When using the existing technology for dimensionality reduction, it does not fully consider that user preferences are an important basis for affecting advertising effects, making it difficult for the data after dimensionality reduction to effectively reflect the advertising effect, reducing the value of data resource sharing. Summary of the invention

[0004] In order to solve the technical problem that the existing technology does not reasonably reduce the dimension of advertising data and reduces the value of data resource sharing, the purpose of the present invention is to provide a data asset sharing method and system for multi-source heterogeneous data sources. The technical solution adopted is as follows:

[0005] A data asset sharing method for multi-source heterogeneous data sources, the method comprising:

[0006] Acquire a data set of advertisement categories; the data set includes a data set to be reduced in dimension and a user characteristic index and advertisement query duration of each displayed advertisement; the data set to be reduced in dimension includes an index to be reduced in dimension of each displayed advertisement;

[0007] In the data set of the advertisement category, all the display advertisements are clustered according to the differences of the user characteristic indicators and the differences of the advertisement query durations of all the display advertisements to obtain various clusters; according to the numerical distribution of each indicator to be reduced in all the clusters, the retention metric of each indicator to be reduced is obtained;

[0008] According to the retention metric of each indicator to be reduced in dimension, the data set to be reduced in dimension is reduced in dimension, and the reduced-dimensional data sets of the advertising categories are obtained and merged for data resource sharing.

[0009] Furthermore, the method for obtaining the clustering clusters includes:

[0010] Constructing a distance metric value between each two of the display advertisements according to differences in all the user characteristic indicators and differences in the advertisement query durations between each two of the display advertisements;

[0011] The k-means clustering algorithm is used to cluster all the display advertisements according to the distance metric value between every two display advertisements to obtain various clusters.

[0012] Furthermore, the method for obtaining the distance metric value includes:

[0013] Calculate the absolute value of the difference between the corresponding values ​​of each user characteristic indicator of each two display advertisements to obtain the difference parameter of each user characteristic indicator of each two display advertisements; calculate the mean value of the difference parameter of all the user characteristic indicators of each two display advertisements to obtain the user characteristic difference degree of each two display advertisements;

[0014] Calculate the absolute value of the difference between the corresponding values ​​of the advertisement query duration of each two of the display advertisements to obtain the advertisement query difference degree of each two of the display advertisements;

[0015] The product of the user feature difference and the advertisement query difference is calculated to obtain a distance measurement value between every two displayed advertisements.

[0016] Furthermore, the method for obtaining the retention metric includes:

[0017] For any indicator to be reduced in dimension, in each cluster, according to the numerical distribution and aggregation of the indicator to be reduced in dimension of all the display advertisements, a representative indicator of each cluster is obtained;

[0018] Forward fusion of the representative indicators of all clusters to obtain the first retained indicator of the indicator to be reduced in dimension;

[0019] Obtaining a second retained index of the index to be reduced in dimension according to differences in overall distribution values ​​of the index to be reduced in dimension of all the display advertisements of all clusters;

[0020] The first retained indicator and the second retained indicator are forwardly fused to obtain a retained metric of the indicator to be reduced in dimension.

[0021] Furthermore, the method for obtaining the representative indicator includes:

[0022] For any indicator to be reduced in dimension, negatively correlate the range of the display ads in the cluster corresponding to the indicator to be reduced in dimension, and obtain the first clustering factor of the cluster;

[0023] In the cluster cluster, the number of values ​​of each dimensionality reduction index is counted, and the maximum number is used as the first parameter; the total number of values ​​of all dimensionality reduction indicators in the cluster cluster is used as the second parameter; the ratio of the first parameter to the second parameter is calculated to obtain the second clustering factor of the cluster cluster;

[0024] The mean of the first clustering factor and the second clustering factor is calculated to obtain a representative index of the cluster.

[0025] Furthermore, the method for obtaining the first retention index includes:

[0026] The mean of the representative index of all clusters is calculated to obtain the first retained index of the index to be reduced in dimension.

[0027] Furthermore, the method for obtaining the second retention index includes:

[0028] Calculating the mean of the corresponding values ​​of the to-be-reduced dimension index of all the display advertisements of each cluster to obtain the overall distribution value of the cluster;

[0029] The absolute value of the difference between the overall distribution values ​​of every two clusters is calculated, the mean of all the absolute values ​​of the differences is calculated and normalized, and a second retained index of the index to be reduced in dimension is obtained.

[0030] Furthermore, the method of forward fusing the first retained index and the second retained index to obtain the retained metric of the index to be reduced in dimension includes:

[0031] The mean of the first retained index and the second retained index is calculated to obtain the retained metric of the index to be reduced in dimension.

[0032] Furthermore, the method for obtaining the reduced-dimensionality data set includes:

[0033] The indicator to be reduced in dimension whose retention metric is greater than the preset retention value is used as the retention indicator; in the data set to be reduced in dimension of the advertisement category, the retention indicators of all displayed advertisements are counted to obtain the data set after dimension reduction of the advertisement category.

[0034] A data asset sharing system for multi-source heterogeneous data sources, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of a data asset sharing method for multi-source heterogeneous data sources as described in any one of the above items when executing the computer program.

[0035] The present invention proposes a data asset sharing system for multi-source heterogeneous data sources, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of a data asset sharing method for multi-source heterogeneous data sources are implemented.

[0036] The present invention has the following beneficial effects:

[0037] First, according to the differences in the user characteristic indicators of all the display advertisements and the differences in the advertisement query durations, all the display advertisements are clustered to obtain various clustering clusters; clustering clusters are used to reflect display advertisements with a high degree of similarity in user preferences, and according to the numerical distribution of each indicator to be reduced in each clustering cluster, the retention metric of the indicator to be reduced is obtained; the retention metric can more accurately reflect the importance of the indicator to be reduced in understanding user preferences, thereby reasonably reducing the dimensionality of the data set to be reduced, so that the data set after dimensionality reduction can effectively reflect the advertising effect and ensure the value of data resource sharing. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the technical solutions and advantages in the embodiments of the present invention or the prior art, the drawings required for use in the embodiments or the prior art descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0039] Figure 1 A flow chart of a data asset sharing method for multi-source heterogeneous data sources provided by one embodiment of the present invention;

[0040] Figure 2 A flow chart of a method for obtaining clusters provided by an embodiment of the present invention;

[0041] Figure 3 A flow chart of a method for obtaining a retention metric provided by an embodiment of the present invention;

[0042] Figure 4 A structural diagram of a data asset sharing system for multi-source heterogeneous data sources provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0043] In order to further explain the technical means and effects adopted by the present invention to achieve the predetermined invention purpose, the following is a detailed description of the data asset sharing method and system for multi-source heterogeneous data sources proposed by the present invention, in combination with the accompanying drawings and preferred embodiments, as well as its specific implementation, structure, features and effects. In the following description, different "one embodiment" or "another embodiment" does not necessarily refer to the same embodiment. In addition, specific features, structures or characteristics in one or more embodiments may be combined in any suitable form.

[0044] Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs.

[0045] The following is a detailed description of a method and system for sharing data assets from multiple heterogeneous data sources provided by the present invention in conjunction with the accompanying drawings.

[0046] The present invention provides a data asset sharing method and system for multi-source heterogeneous data sources. Figure 1 , which shows a flow chart of a data asset sharing method for multi-source heterogeneous data sources provided by an embodiment of the present invention, the method comprising the following steps:

[0047] Step S1: Acquire a data set of advertisement categories; the data set includes a data set to be reduced in dimension and user characteristic indicators and advertisement query duration of each displayed advertisement; the data set to be reduced in dimension includes indicators to be reduced in dimension of each displayed advertisement.

[0048] In order to share data resources, you first need to obtain a data set of the advertising category.

[0049] Considering that advertisements are a multi-source heterogeneous data source, the present invention uses advertisements as an example to illustrate data dimension reduction and data resource sharing for multi-source heterogeneous data sources. The present invention regards advertisements with the same display page content as one advertisement category, and regards advertisements displayed to a specific user at a specific time through a specific channel as an independent display advertisement.

[0050] From the advertisement database, obtain a data set of advertisement categories. The data set to be reduced in dimension includes the indicators to be reduced in dimension for each display advertisement; the indicators to be reduced in dimension include: advertisement channel, advertisement delivery time, advertisement query duration, advertisement click-through rate, advertisement conversion rate, advertisement title format, advertisement subtitle format, advertisement content, user income, user education level, etc.

[0051] Considering that users with different characteristics query display ads for different durations is a core factor affecting the effect of advertising, in order to ensure the effect of data dimensionality reduction, it is necessary to obtain user characteristics and data on query advertising time, and obtain user characteristic indicators and advertising query duration for each display ad from the advertising database; in one embodiment of the present invention, user characteristic indicators include: user age, user gender, user region, user interests and hobbies, etc.; advertising query duration, that is, the total duration from the user clicking on the ad to closing the ad. Among them, the user characteristic indicator directly reflects the characteristics of the display user facing the display ad, and indirectly reflects the user's preference; the advertising query duration directly reflects the user's interest in the display ad, and indirectly reflects the user's preference.

[0052] In order to facilitate computer processing, the text information of the indicators to be reduced in dimension of display ads and the user characteristic indicators are first converted into digital information. For example, the indicator to be reduced in dimension of user education level is used as an example. For example, the code corresponding to the user education level is 001. The user education levels are: primary school and below, junior high school, high school, college and above, and the corresponding codes are 1000, 0100, 0010, 0001. For example, the indicator to be reduced in dimension of the user education level of a display ad is expressed as high school, and the corresponding code is 0010010, and the corresponding decimal number is 36. The values ​​of the indicators to be reduced in dimension are expressed in decimal.

[0053] It should be noted that, in order to facilitate calculations, all indicator data involved in the calculations in the embodiments of the present invention are subjected to data preprocessing to eliminate the dimension effect. The specific means of removing the dimension effect are technical means well known to those skilled in the art and are not limited here. It should be noted that the present invention describes the dimensionality reduction for a data set to be reduced in any advertising category. Similarly, the dimensionality reduction can be performed for data sets to be reduced in other advertising categories in the advertising database, which will not be elaborated here.

[0054] In the process of reducing the dimensionality of the data set, it is considered that the user characteristic indicators reflect the personalized characteristics of the users, the advertising query duration reflects the users' attention to the advertising content, the display advertisements are aimed at the users' attention to the advertising content and the personalized needs of the users, which are the core factors affecting the effectiveness of advertising and reflect the user's preferences. If they are ignored in the process of reducing the dimensionality of the advertising data, the reduced data may not accurately reflect all the information of the original data, thereby reducing the value of data resource sharing.

[0055] Step S2: In the data set of the advertising category, all displayed advertisements are clustered according to the differences in the user characteristic indicators of all displayed advertisements and the differences in the advertisement query durations to obtain various clustering clusters; according to the numerical distribution of each indicator to be reduced in all clustering clusters, the retention measure of each indicator to be reduced is obtained.

[0056] First, all display ads are clustered according to the differences in user characteristic indicators and ad query duration to obtain clusters. Clusters are used to reflect the display ads browsed by user groups with similar preferences. According to the numerical distribution of each indicator to be reduced in each cluster, the retention metric of the indicator to be reduced is obtained. The retention metric reflects the degree to which the indicator to be reduced needs to be retained, providing a reference for subsequent dimensionality reduction.

[0057] Considering that ads for a certain ad category will be displayed to different users, users with different characteristics have different preferences for ads. For example, young women are often more interested in cosmetics ads and tend to spend more time searching for them; older men are often not interested and rarely click on such ads. In order to identify user groups with similar characteristics and behaviors, all display ads are clustered according to the differences in user characteristic indicators and ad search durations to obtain clusters. See Figure 2 , which shows a flow chart of a method for obtaining clusters in one embodiment of the present invention. Preferably, in one embodiment of the present invention, the method for obtaining clusters includes:

[0058] Step S201: constructing a distance metric value between every two display advertisements according to the differences in all user characteristic indicators and the differences in advertisement query durations between every two display advertisements.

[0059] Considering the ideal advertising effect, personalized push is carried out according to user preferences to increase the exposure rate of advertisements. User characteristic indicators reflect user portraits and are an important basis for evaluating user preferences for advertisements; advertisement query duration indicates the length of time users browse or query advertisements, reflecting the degree of user preference for advertisement content; through the differences in all user characteristic indicators of every two displayed advertisements and the differences in advertisement query duration, the distance measurement value of every two displayed advertisements is constructed. The distance measurement value combines the differences in user characteristics and query duration, and is used to measure the differences in user preferences for advertisements.

[0060] Preferably, in one embodiment of the present invention, the absolute value of the difference between the numerical values ​​corresponding to each user characteristic indicator of every two display advertisements is calculated to obtain the difference parameter of each user characteristic indicator of every two display advertisements; the mean of the difference parameters of all user characteristic indicators of every two display advertisements is calculated to obtain the user characteristic difference degree of every two display advertisements; the absolute value of the difference between the numerical values ​​corresponding to the advertising query duration of every two display advertisements is calculated to obtain the advertising query difference degree of every two display advertisements; the product of the user characteristic difference degree and the advertising query difference degree is calculated to obtain the distance measurement value of every two display advertisements.

[0061] For the above steps, considering that the user feature index is a dimension describing the user portrait, the difference parameter reflects the degree of difference in a certain user feature index of the users targeted by the two advertisements. The user feature difference reflects the overall degree of difference in user features between the two display ads. The greater the user feature difference, the greater the overall degree of difference in user features targeted by the two ads; the ad query difference is constructed. The greater the ad query difference, the greater the degree of difference in the ad viewing time of the users targeted by the two display ads, reflecting the degree of difference in attracting user attention between the two display ads; the user feature difference and the ad query difference are combined to obtain the distance metric value of each two display ads. The distance metric value combines the differences in user features and query time, and is used to measure the difference in user preferences for ads. The smaller the distance metric value, the more similar the two display ads are in terms of user features and ad query time, that is, the closer their preferences for users are. Conversely, the larger the distance metric value, the greater the difference between the two display ads in these aspects, and the more different their preferences for users are.

[0062] Step S202: using a clustering algorithm, cluster all display advertisements according to the distance measurement value between every two display advertisements to obtain various clusters.

[0063] The display ads in the clusters all have similar user characteristics and ad query durations, that is, the users targeted by all the display ads in the clusters have highly similar user preferences.

[0064] It should be noted that the present invention uses the k-means clustering algorithm to cluster all display ads according to the distance measurement value between each two display ads to obtain each cluster cluster. The k-means clustering algorithm is a technical means well known to those skilled in the art, and will not be described in detail here. Only a brief description of the process of using the k-means clustering algorithm in one embodiment of the present invention to cluster all display ads in an advertising category according to the distance measurement value between each two display ads to obtain each cluster cluster is given:

[0065] Considering that the larger the distance metric value between two display ads, the greater the difference in user preferences, the optimal k value is obtained through the elbow method, and the distance metric value between each two display ads is substituted into the clustering process, and all display ads in the advertising category are clustered to obtain clustering results and various clustering clusters. The differences in display ads in different clustering clusters are made as large as possible, and the differences in display ads in the same clustering cluster are made as small as possible. The similarity of user features of display ads and query ad duration in each clustering cluster is made as large as possible. In other embodiments of the present invention, the optimal k value of the k-means clustering algorithm can be determined by other methods such as the silhouette coefficient method, which is not limited here.

[0066] Through the above steps, advertisements for similar user characteristics are clustered into a cluster. Considering that the indicators to be reduced in dimension show significant differences between different clusters, they have high consistency in the same cluster, which reflects that the indicators to be reduced in dimension are more important for distinguishing user preferences. It is important information for understanding user preferences and optimizing advertising strategies, and needs to be retained. The retention metric is constructed to reflect the degree to which the indicators to be reduced in dimension need to be retained. Please refer to Figure 3 , which shows a flow chart of a method for obtaining a retention metric in one embodiment of the present invention. Preferably, in one embodiment of the present invention, the method for obtaining a retention metric includes:

[0067] Step S211: for any indicator to be reduced in dimension, in each cluster, according to the numerical distribution and aggregation of the indicator to be reduced in dimension of all displayed advertisements, a representative indicator of each cluster is obtained.

[0068] In order to retain those important indicators that have significant differences between different clusters and high consistency within the same cluster in the subsequent dimensionality reduction analysis, first, we select an indicator to be reduced for analysis. First, the representative indicator of the cluster is used to reflect the consistency of the cluster in the indicator to be reduced.

[0069] Preferably, in one embodiment of the present invention, the method for obtaining the representative indicator includes:

[0070] For any indicator to be reduced in dimension, negatively correlate the range of the display ads in the cluster corresponding to the indicator to be reduced in dimension, and obtain the first clustering factor of the cluster;

[0071] In the clustering cluster, the number of values ​​of each dimensionality reduction index is counted, and the maximum number is used as the first parameter; the total number of values ​​of all dimensionality reduction indicators in the clustering cluster is used as the second parameter; the ratio of the first parameter to the second parameter is calculated to obtain the second clustering factor of the clustering cluster;

[0072] The means of the first clustering factor and the second clustering factor are calculated to obtain the representative index of the cluster. It should be noted that negative correlation mapping is a technical means well known to those skilled in the art, and negative correlation mapping can be in the form of inverse proportion or negative exponential power without limitation.

[0073] According to the above steps, the range of the indicator to be reduced in dimension corresponding to the display advertisement in the cluster is calculated. Since the larger the range, the lower the degree of aggregation, the negative correlation mapping is performed on the range to obtain the first clustering factor of the cluster; the larger the first clustering factor, the higher the degree of aggregation, which means that the numerical consistency of the indicator to be reduced in the cluster is higher; the number of values ​​of each indicator to be reduced is counted, and the value with the largest number is found as the first parameter, and the total number of values ​​of all indicators to be reduced in the cluster is calculated as the second parameter. Then, the ratio of the first parameter to the second parameter is calculated to obtain the second clustering factor. The second clustering factor reflects the proportion of the corresponding values ​​of the dominant indicator to be reduced in the cluster. The higher the proportion, the higher the degree of aggregation, which means that the numerical consistency of the indicator to be reduced in the cluster is higher. Finally, the mean of the first clustering factor and the second clustering factor is calculated to obtain the representative indicator of the cluster. The representative indicator can better reflect the numerical consistency characteristics of the cluster on the indicator to be reduced.

[0074] Step S212: forwardly fuse the representative indices of all clusters to obtain the first retained index of the index to be reduced in dimension.

[0075] The first retention index comprehensively reflects the consistency of all clusters in the index to be reduced in the cluster; the larger the value of the first retention index is, the more the index to be reduced needs to be retained.

[0076] It should be noted that forward fusion is an existing technology well known to those skilled in the art, and forward fusion can adopt simple product, arithmetic mean or other suitable fusion methods. Preferably, in one embodiment of the present invention, the method for obtaining the first retained index includes: calculating the mean of the representative indexes of all clusters to obtain the first retained index of the index to be reduced in dimension.

[0077] According to the above steps, the mean of the representative index of all clusters is calculated to obtain the first retained index of the index to be reduced in dimension. This index reflects the consistency of all clusters in the index to be reduced in dimension, and is the basis for evaluating the overall importance of the index to be reduced in dimension.

[0078] Step S213: obtaining a second retained indicator of the indicator to be reduced in dimension according to the difference in the overall distribution values ​​of the indicator to be reduced in dimension of all display advertisements in all clusters.

[0079] The second retention index reflects the numerical differences of the indicators to be reduced in different clusters. The larger the value of the second retention index is, the more the indicator to be reduced needs to be retained.

[0080] Preferably, in one embodiment of the present invention, the method for obtaining the second retention index includes:

[0081] Calculate the mean of the corresponding values ​​of the to-be-reduced dimension index of all display ads in each cluster to obtain the overall distribution value of the cluster;

[0082] The absolute value of the difference between the overall distribution values ​​of each two clusters is calculated, the mean of all the absolute values ​​of the differences is calculated and normalized, and the second retained index of the index to be reduced in dimension is obtained. It should be noted that the normalization processing is a technical means well known to those skilled in the art, and the normalization processing can adopt linear normalization, etc., which is not limited here.

[0083] For the above steps, calculate the mean of the corresponding values ​​of all display ads in each cluster on the indicator to be reduced in dimension to obtain the overall distribution value of the cluster. Then, calculate the absolute value of the difference between the overall distribution values ​​of each two clusters to measure the difference between them. The mean of the absolute values ​​of all differences should be calculated and normalized to obtain the second retained indicator of the indicator to be reduced in dimension. The second retained indicator reflects the degree of difference between different clusters on the indicator to be reduced in dimension. The greater the difference, the stronger the ability of this indicator to distinguish different user preferences.

[0084] Step S214: forwardly fuse the first retained indicator and the second retained indicator to obtain the retained metric of the indicator to be reduced in dimension.

[0085] The retention metric reflects the degree to which the indicator to be reduced in dimension needs to be retained. The larger the retention metric, the higher the degree to which the indicator to be reduced in dimension needs to be retained.

[0086] It should be noted that forward fusion is an existing technology well known to those skilled in the art, and forward fusion can adopt simple product, arithmetic mean or other suitable fusion methods. Preferably, in one embodiment of the present invention, the mean of the first retention index and the second retention index is calculated to obtain the retention metric of the index to be reduced in dimension. Here, by selecting an index to be reduced in dimension for analysis, the retention metric of the index to be reduced in dimension is explained. Similarly, the retention metric of other indexes to be reduced in dimension can be obtained without further explanation.

[0087] According to the above steps, the mean of the first retention index and the second retention index is calculated to obtain the retention metric of the index to be reduced in dimension. The retention metric can more accurately reflect the importance of the index to be reduced in dimension to understanding user preferences.

[0088] Step S3: According to the retention metric of each indicator to be reduced in dimension, the data set to be reduced in dimension is reduced in dimension, and the reduced-dimension data sets of the advertising category are obtained and merged for data resource sharing.

[0089] Through the above steps, we can obtain a retention measure of the importance of understanding user preferences, and perform reasonable dimensionality reduction on the dimensionality reduction data set, so that the dimensionality reduction data set can effectively reflect the advertising effect and ensure the value of data resource sharing.

[0090] In order to reasonably reduce the dimension of the data set to be reduced, preferably, in one embodiment of the present invention, the method for obtaining the data set after dimension reduction includes:

[0091] The indicator to be reduced in dimension whose retention metric is greater than the preset retention value is used as the retention indicator; in the data set to be reduced in dimension of the advertising category, the retention indicators of all displayed advertisements are counted to obtain the reduced dimension data set of the advertising category. In one embodiment of the present invention, the preset retention value is determined based on an empirical value, and the preset retention value is set to 0.54, which can be set by the implementer according to the implementation scenario.

[0092] For the above steps, considering that the retained metric reflects the importance of the indicator to be reduced in understanding user preferences, the indicator to be reduced whose retained metric is greater than the preset retained value is used as the retained indicator; in the data set to be reduced in the advertising category, these retained indicators are counted for all display ads to form the data set after dimensionality reduction. Since the most important feature information is retained, the data set after dimensionality reduction is easier to process than the original data set, while ensuring that the data can effectively reflect the advertising effect.

[0093] In order to share data resources, in one embodiment of the present invention, a method for sharing data resources includes:

[0094] Upload the reduced-dimensional data set to the cloud platform for resource sharing.

[0095] Specifically, upload the reduced-dimensionality data set to the designated storage location on the cloud platform, and use the storage service provided by the cloud platform to manage the reduced-dimensionality data set. Furthermore, in order to ensure that the interface supports common query operations and provides necessary data format conversion and output functions, a data access interface is developed so that other systems or users can easily access and query the reduced-dimensionality data set. Consider the copyright and privacy protection issues of the data, ensure that the data will not leak sensitive information or infringe on the rights of others during the sharing process, formulate a data sharing strategy, clarify the scope, method and conditions of data sharing, and finally realize the resource sharing of the reduced-dimensionality data set.

[0096] The present invention also proposes a data asset sharing system for multi-source heterogeneous data sources, see Figure 4 , which shows a structure diagram of a data asset sharing system for multi-source heterogeneous data sources provided by an embodiment of the present invention. The system includes: a data acquisition module 101, a retention metric analysis module 102 and a resource sharing module 103.

[0097] The data acquisition module 101 is used to acquire a data set of an advertisement category; the data set includes a data set to be reduced in dimension and user characteristic indicators and advertisement query duration of each displayed advertisement; the data set to be reduced in dimension includes indicators to be reduced in dimension of each displayed advertisement.

[0098] The retention metric analysis module 102 is used to cluster all displayed advertisements in the data set of the advertisement category according to the differences in the user characteristic indicators of all displayed advertisements and the differences in the advertisement query durations, and obtain various clustering clusters; and obtain the retention metric of each indicator to be reduced in dimension according to the numerical distribution of each indicator to be reduced in dimension among all clustering clusters.

[0099] The resource sharing module 103 is used to reduce the dimension of the data set to be reduced according to the retention metric of each indicator to be reduced, obtain the reduced dimension data set of the advertising category and merge them for data resource sharing.

[0100] It should be noted that the system provided in the above embodiment is only illustrated by the division of the above functional modules. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the computer device is divided into different functional modules to complete all or part of the functions described above. In addition, the data asset sharing system for multi-source heterogeneous data sources and the data asset sharing method embodiment for multi-source heterogeneous data sources provided in the above embodiment belong to the same concept. The specific implementation process is detailed in the method embodiment and will not be repeated here.

[0101] In summary, the embodiment of the present invention provides a data asset sharing method and system for multi-source heterogeneous data sources. First, all display ads are clustered according to the differences in user characteristic indicators of all display ads and the differences in ad query duration to obtain various clustering clusters; according to the numerical distribution of each indicator to be reduced in all clustering clusters, the retention metric of each indicator to be reduced is obtained; according to the retention metric of each indicator to be reduced, the dimension of the data set to be reduced is reduced, and the reduced dimension data set of the advertising category is obtained and merged for data resource sharing. In the embodiment of the present invention, by fully considering the user preferences that affect the advertising effect, the dimension of the data set to be reduced is reasonably reduced to ensure the value of data resource sharing.

[0102] It should be noted that the sequence of the above embodiments of the present invention is only for description and does not represent the advantages and disadvantages of the embodiments. The processes depicted in the accompanying drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0103] The various embodiments in this specification are described in a progressive manner, and the same or similar parts between the various embodiments can be referenced to each other, and each embodiment focuses on the differences from other embodiments.

Claims

1. A data asset sharing method for multi-source heterogeneous data sources, characterized in that: The method comprises: Acquire a data set of advertisement categories; the data set includes a data set to be reduced in dimension and a user characteristic index and advertisement query duration of each displayed advertisement; the data set to be reduced in dimension includes an index to be reduced in dimension of each displayed advertisement; In the data set of the advertisement category, all the display advertisements are clustered according to the differences of the user characteristic indicators and the differences of the advertisement query durations of all the display advertisements to obtain various clusters; according to the numerical distribution of each indicator to be reduced in all the clusters, the retention metric of each indicator to be reduced is obtained; According to the retention metric of each indicator to be reduced in dimension, the data set to be reduced in dimension is reduced in dimension, and the reduced-dimensional data sets of the advertising categories are obtained and merged for data resource sharing.

2. A data asset sharing method for multi-source heterogeneous data sources according to claim 1, characterized in that: The method for obtaining the clusters includes: Constructing a distance metric value between each two of the display advertisements according to differences in all the user characteristic indicators and differences in the advertisement query durations between each two of the display advertisements; The k-means clustering algorithm is used to cluster all the display advertisements according to the distance metric value between every two display advertisements to obtain various clusters.

3. A data asset sharing method for multi-source heterogeneous data sources according to claim 2, characterized in that: The method for obtaining the distance metric value includes: Calculate the absolute value of the difference between the corresponding values ​​of each user characteristic indicator of each two display advertisements to obtain the difference parameter of each user characteristic indicator of each two display advertisements; calculate the mean value of the difference parameter of all the user characteristic indicators of each two display advertisements to obtain the user characteristic difference degree of each two display advertisements; Calculate the absolute value of the difference between the corresponding values ​​of the advertisement query duration of each two of the display advertisements to obtain the advertisement query difference degree of each two of the display advertisements; The product of the user feature difference and the advertisement query difference is calculated to obtain a distance measurement value between every two displayed advertisements.

4. A data asset sharing method for multi-source heterogeneous data sources according to claim 1, characterized in that: The method for obtaining the retention metric includes: For any indicator to be reduced in dimension, in each cluster, according to the numerical distribution and aggregation of the indicator to be reduced in dimension of all the display advertisements, a representative indicator of each cluster is obtained; Forward fusion of the representative indicators of all clusters to obtain the first retained indicator of the indicator to be reduced in dimension; Obtaining a second retained index of the index to be reduced in dimension according to differences in overall distribution values ​​of the index to be reduced in dimension of all the display advertisements of all clusters; The first retained indicator and the second retained indicator are forwardly fused to obtain a retained metric of the indicator to be reduced in dimension.

5. A data asset sharing method for multi-source heterogeneous data sources according to claim 4, characterized in that: The method for obtaining the representative indicator includes: For any indicator to be reduced in dimension, negatively correlate the range of the display ads in the cluster corresponding to the indicator to be reduced in dimension, and obtain the first clustering factor of the cluster; In the cluster cluster, the number of values ​​of each dimensionality reduction index is counted, and the maximum number is used as the first parameter; the total number of values ​​of all dimensionality reduction indicators in the cluster cluster is used as the second parameter; the ratio of the first parameter to the second parameter is calculated to obtain the second clustering factor of the cluster cluster; The mean of the first clustering factor and the second clustering factor is calculated to obtain a representative index of the cluster.

6. A data asset sharing method for multi-source heterogeneous data sources according to claim 4, characterized in that: The method for obtaining the first retention index includes: The mean of the representative index of all clusters is calculated to obtain the first retained index of the index to be reduced in dimension.

7. A data asset sharing method for multi-source heterogeneous data sources according to claim 4, characterized in that: The method for obtaining the second retention index includes: Calculating the mean of the corresponding values ​​of the to-be-reduced dimension index of all the display advertisements of each cluster to obtain the overall distribution value of the cluster; The absolute value of the difference between the overall distribution values ​​of every two clusters is calculated, the mean of all the absolute values ​​of the differences is calculated and normalized, and a second retained index of the index to be reduced in dimension is obtained.

8. A data asset sharing method for multi-source heterogeneous data sources according to claim 4, characterized in that: The method of forward fusing the first retention index and the second retention index to obtain the retention metric of the index to be reduced in dimension includes: The mean of the first retained index and the second retained index is calculated to obtain the retained metric of the index to be reduced in dimension.

9. A data asset sharing method for multi-source heterogeneous data sources according to claim 1, characterized in that: The method for obtaining the reduced-dimensionality data set includes: The indicator to be reduced in dimension whose retention metric is greater than the preset retention value is used as the retention indicator; in the data set to be reduced in dimension of the advertisement category, the retention indicators of all displayed advertisements are counted to obtain the data set after dimension reduction of the advertisement category.

10. A data asset sharing system for multi-source heterogeneous data sources, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of a data asset sharing method for multi-source heterogeneous data sources as described in any one of claims 1 to 9 are implemented.

Images