Data processing method, device, electronic device and storage medium

Abstract

The present application discloses a data processing method, device, electronic device and storage medium, and relates to the field of data processing, and further to the field of intelligent recommendation. The specific implementation scheme is: obtain a video data set with features to be filled; wherein each video resource in the video data set contains N features, and the video resources in the video data set come from multiple different sites; N is a positive integer; perform similarity clustering on the video resources in the video data set to obtain at least one clustering cluster; determine the distribution histogram of each feature of the video resources from the same site in the video data set; fill the feature values ​​of the video resources with missing feature values ​​in each clustering cluster according to the distribution histogram of each feature of the video resources from the same site in the video data set. The present application makes full use of existing video resources for feature filling, which is low-cost and does not affect the filling effect, and can improve the feature coverage of video resources.

Description

Technical Field

[0001] The present application relates to the field of data processing, and further to the field of intelligent recommendation, and in particular to a data processing method, device, electronic device and storage medium. Background Art

[0002] In recent years, more and more short video apps have emerged, providing convenience and entertainment for people's lives. In order to recommend better and more personalized video content to users, a large number of video resources will be captured from the Internet, and models will be trained based on the features of multiple dimensions of video resources to optimize the user experience. Summary of the invention

[0003] The present application provides a data processing method, device, electronic device and storage medium.

[0004] According to one aspect of the present application, a data processing method is provided, comprising:

[0005] Obtain a video data set with features to be filled; wherein each video resource in the video data set contains N features, and the video resources in the video data set come from multiple different sites; N is a positive integer;

[0006] Performing similarity clustering processing on the video resources in the video data set to obtain at least one cluster;

[0007] Determine a distribution histogram of video resources from the same site in the video data set on each feature;

[0008] According to the distribution histogram of each feature of the video resources from the same site in the video data set, feature values ​​are filled for the video resources with missing feature values ​​in each of the clusters.

[0009] According to a second aspect of the present application, there is provided a data processing device, comprising:

[0010] An acquisition module, used to acquire a video data set with features to be filled; wherein each video resource in the video data set contains N features, and the video resources in the video data set come from multiple different sites; and N is a positive integer;

[0011] A clustering module, used to perform similarity clustering processing on the video resources in the video data set to obtain at least one clustering cluster;

[0012] A determination module, used to determine a distribution histogram of video resources from the same site in the video data set on each feature;

[0013] The first filling module is used to fill in feature values ​​for video resources with missing feature values ​​in each of the clusters according to the distribution histogram of each feature of video resources from the same site in the video data set.

[0014] According to a third aspect of the present application, an electronic device is provided, including:

[0015] at least one processor; and

[0016] a memory communicatively connected to at least one processor; wherein,

[0017] The memory stores instructions that can be executed by at least one processor, and the instructions are executed by at least one processor so that the at least one processor can execute the data processing method of the first aspect mentioned above.

[0018] According to a fourth aspect of the present application, a non-transitory computer-readable storage medium storing computer instructions is provided, where the computer instructions are used to enable a computer to execute the data processing method of the aforementioned first aspect.

[0019] According to a fifth aspect of the present application, a computer program product is provided, comprising a computer program, which implements the steps of the data processing method according to the aforementioned first aspect when executed by a processor.

[0020] According to the technical solution of the present application, existing video resources are fully utilized for feature filling, which is low-cost and does not affect the filling effect, thereby improving the feature coverage of video resources.

[0021] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present application, nor is it intended to limit the scope of the present application. Other features of the present application will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present application.

[0023] Figure 1 It is a flowchart of a data processing method provided according to an embodiment of the present application;

[0024] Figure 2 is a schematic diagram of a distribution histogram of video resources from the same site on one feature in a video data set provided according to an embodiment of the present application;

[0025] Figure 3 is a flowchart of another data processing method provided according to an embodiment of the present application;

[0026] Figure 4is a flowchart of another data processing method provided according to an embodiment of the present application;

[0027] Figure 5 is a schematic diagram of a distribution histogram of the “number of likes” feature of video resources from site C in a video data set provided according to an embodiment of the present application;

[0028] Figure 6 is a schematic diagram of a distribution histogram of the “number of likes” feature of video resources from site A in a video data set provided according to an embodiment of the present application;

[0029] Figure 7 is a schematic diagram of a distribution histogram of the “number of comments” feature of video resources from site A in a video data set provided according to an embodiment of the present application;

[0030] Figure 8 is a structural block diagram of a data processing device provided according to an embodiment of the present application;

[0031] Fig. 9 is a structural block diagram of another data processing device provided according to an embodiment of the present application;

[0032] Fig.10 It is a block diagram of an electronic device used to implement the data processing method of an embodiment of the present application. DETAILED DESCRIPTION

[0033] The following is a description of exemplary embodiments of the present application in conjunction with the accompanying drawings, including various details of the embodiments of the present application to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present application. Similarly, for the sake of clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0034] Since some video websites are closed-loop, video sites do not share unique resources with each other. The features of a large number of video resources from different sites captured on the Internet are often missing, for example, features of multiple dimensions such as the number of views, likes, dislikes, and comments of video resources. The lack of video resource features often greatly affects the training effect of downstream models. Therefore, how to solve the missing of video resource features has become a difficulty in related technologies.

[0035] To this end, the present application provides a data processing method, device, electronic device and storage medium. Specifically, the data processing method, device, electronic device and storage medium of the embodiments of the present application are described below with reference to the accompanying drawings.

[0036] Figure 1It is a flowchart of a data processing method provided according to an embodiment of the present application. It should be noted that the data processing method of the embodiment of the present application can be applied to the data processing device of the embodiment of the present application, and the data processing device can be configured on an electronic device.

[0037] like Figure 1 As shown, the data processing method may include the following steps:

[0038] Step 101, obtaining a video data set to be filled with features. Each video resource in the video data set contains N features, and the video resources in the video data set come from multiple different sites. N is a positive integer.

[0039] As an example, the features included in each video resource in the video data set may be features such as the number of likes, the number of plays, and the number of comments on the video resource.

[0040] Step 102: Perform similarity clustering processing on the video resources in the video data set to obtain at least one cluster.

[0041] Optionally, in some embodiments of the present application, the video resources in the video data set can be clustered by similarity through fingerprint clustering. As an example, each video resource in the video data set can be processed to obtain the video fingerprint information of each video resource. Based on the video fingerprint information of each video resource, the video resources in the video data set are clustered by similarity to obtain at least one clustering cluster. Among them, the video resources in the same clustering cluster are video resources with similar video features. In addition, the video resources in the video data set can also be clustered by similarity through other clustering methods, such as K-MEANS clustering algorithm, DBSCAN clustering algorithm, etc.

[0042] Step 103: determine a distribution histogram of each feature of video resources from the same site in the video data set.

[0043] The above-mentioned determination of the distribution histogram of each feature of the video resources originating from the same site in the video data set is to count the distribution of the characteristic values ​​of the video resources originating from the same site in each feature in the video data set. As a possible implementation method, multiple interval ranges corresponding to at least one feature can be determined. According to each characteristic value of the video resources originating from the same site in the video data set, the number of video resources originating from the same site in the video data set distributed in each interval range is counted. According to the number distributed in each interval range, the distribution histogram of each feature of the video resources originating from the same site in the video data set is determined.

[0044] As an example, the interval range corresponding to the "number of plays" feature is determined to be 0, (0,10], (10,100], (100,1000], (1000,10000], (10000,10000], (10000,100000], (100000,+∞), and according to the "number of plays" feature value of the video resources from site A in the video data set, the number of "number of plays" feature values ​​of the video resources from site A in the video data set distributed in each interval range is counted. According to the number of "number of plays" feature values ​​of the video resources from site A distributed in each interval range, the proportion of the "number of plays" feature values ​​of the video resources from site A distributed in each interval range is calculated, so as to determine Figure 2 The distribution histogram of the video resources from site A in the video dataset on the “number of plays” feature is shown. Similarly, the distribution histogram of the video resources from site A in the video dataset on other features, as well as the distribution histogram of the video resources from other sites on each feature can be determined.

[0045] Step 104 , according to the distribution histogram of each feature of the video resources from the same site in the video data set, fill in the feature values ​​of the video resources with missing feature values ​​in each cluster.

[0046] It should be noted that this step is mainly aimed at the situation where all video resources from the same site in the same cluster do not have the characteristic value of the i-th feature. According to the distribution histogram of each feature of the video resources from the same site in the video data set, and the video resources from other sites in the same cluster that have the characteristic value of the i-th feature, the feature filling value can be obtained through a certain mapping, and the characteristic value of the video resources missing in each cluster can be filled.

[0047] In addition, for the case where some video resources originating from the same site in the same cluster have a feature value of the i-th feature, the feature value of the i-th feature of other video resources in the same cluster that at least some of the video resources originate from the same site can be filled in based on the feature value of the i-th feature of at least some of the video resources. As an example, assume that in cluster C1, some video resources originating from site B have a feature value of the i-th feature, while the rest of the video resources of site B do not have a feature value of the i-th feature. The feature average value of the i-th feature of the video resource of site B that has the feature value of the i-th feature can be calculated, and the feature average value can be used as the feature filling value to fill in the feature value of the i-th feature of the video resource in cluster C1 that originates from site B and lacks the feature value.

[0048] According to the data processing method of the embodiment of the present application, by clustering the video resources in the data set by similarity, using the distribution histogram of each feature of the video resources from the same site in the video data set, and the video resources with similar video features in the same cluster, the feature values ​​of the video resources missing in each cluster are filled, making full use of the existing video resources for feature filling, with low cost and without affecting the filling effect. The present application can also improve the feature coverage of video resources, and train the downstream modules based on the data set after feature filling, so as to optimize the model effect.

[0049] Figure 3 FIG. 1 is a flow chart of another data processing method provided according to an embodiment of the present application. Figure 3 As shown, the data processing method may include the following steps:

[0050] Step 301, obtaining a video data set to be filled with features. Each video resource in the video data set contains N features, and the video resources in the video data set come from multiple different sites. N is a positive integer.

[0051] Step 302: Perform similarity clustering processing on the video resources in the video data set to obtain at least one cluster.

[0052] Step 303: determine a distribution histogram of each feature of video resources from the same site in the video data set.

[0053] Step 304: for the i-th feature, in response to all video resources from the same site in the same cluster not having a feature value of the i-th feature, determine a target site in the same cluster that has a feature value of the i-th feature, where i is a positive integer less than or equal to N.

[0054] As an example, assume that cluster C2 includes video resources from site A, video resources from site B, and video resources from site C. For the i-th feature, all video resources from site A in cluster C2 do not have the feature value of the i-th feature, while at least one of the multiple video resources from site B has the feature value of the i-th feature. Therefore, in cluster C2, site B can be determined as the target site. In addition, if at least one of the multiple video resources from site C has the feature value of the i-th feature, in cluster C2, site C can also be determined as the target site.

[0055] Step 305 : Determine the distribution histogram of the video resources of the target site on the i-th feature according to the distribution histogram of the video resources from the same site in the video data set on each feature.

[0056] Step 306: fill the feature value of the i-th feature of all video resources from the same site according to the feature value of the i-th feature of the video resources from the target site and the distribution histogram of the video resources of the target site on the i-th feature.

[0057] As a possible implementation method, the feature average value of the i-th feature of the video resources from the target site in the same cluster can be calculated, and the range of the feature average value in the distribution histogram of the i-th feature of the video resources of the target site is determined. A feature value of the i-th feature is randomly selected within the range as the feature fill value of the i-th feature of all video resources from the same site.

[0058] Step 307, in response to the existence of a feature value of the i-th feature in some video resources in the same cluster that originate from the same site, fill in the feature value of the i-th feature of other video resources in the same cluster that originate from the same site as at least some of the video resources according to the feature value of the i-th feature of at least some of the video resources.

[0059] In the embodiments of the present application, steps 301 to 303 and step 307 can be implemented in any of the embodiments of the present application, and the present application does not make any specific limitation on this and will not elaborate on it.

[0060] According to the data processing method of the embodiment of the present application, by performing similarity clustering on the video resources in the data set, for video resources in the same clustering cluster where all video resources originating from the same site do not have the feature value of the i-th feature, the target site video resources in the same clustering cluster that have the feature value of the i-th feature, and the distribution histogram of the target site on each feature, are used to fill the feature value of the video resources in the clustering cluster that are missing the feature value from the same site. For the case where some video resources in the same clustering cluster that are originating from the same site have the feature value of the i-th feature, the video resources in the same site that have the feature value of the i-th feature are used to fill the feature of the video resources that are missing the feature value of the i-th feature from the same site. The present application makes full use of the existing video resources for feature filling, which is low in cost and does not affect the filling effect, thereby improving the accuracy of feature filling. The present application can also improve the feature coverage of video resources, and train downstream modules based on the data set after feature filling, thereby optimizing the model effect.

[0061] In order to further improve the accuracy of video feature filling, taking into account the differences in the distribution of video features of different sites (for example, the playback volume of site A is generally higher than that of site B), when using the video resources of the target site to fill the features of the site video resources with missing features in the same cluster, the feature filling value can be determined based on a certain mapping relationship. Figure 4 FIG. 1 is a flow chart of another data processing method provided according to an embodiment of the present application. Figure 4 As shown, the data processing method may include the following steps:

[0062] Step 401, obtain a video data set with features to be filled, wherein each video resource in the video data set contains N features, and the video resources in the video data set come from multiple different sites, and N is a positive integer.

[0063] Step 402: Perform similarity clustering processing on the video resources in the video data set to obtain at least one cluster.

[0064] Step 403: determine a distribution histogram of each feature of video resources from the same site in the video data set.

[0065] Step 404: for the i-th feature, in response to all video resources from the same site in the same cluster not having a feature value of the i-th feature, determine a target site in the same cluster that has a feature value of the i-th feature, where i is a positive integer less than or equal to N.

[0066] Step 405 : determining a distribution histogram of the video resources of the target site on the i-th feature according to the distribution histogram of the video resources from the same site in the video data set on each feature.

[0067] Step 406: Calculate the feature average value of the i-th feature of the target site according to the feature value of the i-th feature of the video resource from the target site.

[0068] It should be noted that the video resource from the target site and the video resource from the same site and neither of which has the feature value of the i-th feature are in the same cluster. According to the feature value of the i-th feature of the video resource from the target site in the same cluster, the feature average value of the i-th feature of the target site is calculated.

[0069] Step 407 : Determine a first target interval range within which the feature average falls according to the feature average of the i-th feature of the target site and the distribution histogram of the video resources of the target site on the i-th feature.

[0070] The distribution histogram of the target site's video resources on the i-th feature reflects the distribution of the feature values ​​of the target site's video resources on the i-th feature. As an example, assume that in cluster C3, all video resources from site A do not have a feature value of the "number of likes" feature, and video resources from site C have a feature value of the "number of likes" feature. Site C is taken as the target site. Figure 5 is the distribution histogram of the “number of likes” feature of the video resources from site C in the video dataset, such as Figure 5 As shown, it can be seen that among the video resources of site C, 3% of the video resources have 0 likes, 9% of the video resources have likes in the interval (0,10], 20% of the video resources have likes in the interval (10,100], 25% of the video resources have likes in the interval (100,1000], 25% of the video resources have likes in the interval (1000,10000], 13% of the video resources have likes in the interval (10000,100000], and 5% of the video resources have likes greater than 100000. Assuming that the feature average value of the "likes" feature of the video resources of site C in cluster C3 is 9, according to Figure 5 The distribution histogram of the video resources of site C on the “number of likes” feature shows that the first target interval range in which the average value of the feature falls is (0, 10].

[0071] Step 408: Determine the first resource proportion of the target site whose feature value of the i-th feature is less than or equal to the target value based on the distribution histogram of the target site's video resources on the i-th feature, wherein the target value is the upper limit value within the first target interval.

[0072] It should be noted that the first resource ratio can be understood as the cumulative ratio of video resources whose feature value of the i-th feature in the target site is less than or equal to the target value in all video resources of the target site. Figure 5 Taking the embodiment shown as an example, the first target interval range is (0, 10], and the upper limit value is 10, so the upper limit value 10 is determined as the target value. Figure 5 In the illustrated embodiment, according to the distribution histogram of the “number of likes” feature of the video resources in site C, the video resources in site C whose characteristic value of the “number of likes” feature is less than or equal to the target value 10 include video resources whose number of likes is 0 and video resources in the interval (0,10]. Therefore, it can be obtained that the proportion of the first resources in site C whose characteristic value of the “number of likes” feature is less than or equal to the target value 10 is 0.12.

[0073] Step 409: Determine a second target interval range having a resource ratio similar to the first resource ratio from the distribution histogram of the video resources of the same site on the i-th feature.

[0074] Among them, the similar resource ratio can be understood as the resource ratio that is closest to the first resource ratio among the resource ratios corresponding to each interval range. The calculation method of the resource ratio corresponding to each interval range is the same as the calculation method of the first resource ratio, that is, the cumulative ratio of video resources whose feature value of the i-th feature in the same site is less than or equal to the upper limit value in each interval range in all video resources in the same site.

[0075] In one implementation, the resource proportions corresponding to the respective intervals that are just greater than the first resource proportion may be determined as similar resource proportions. In another implementation, the resource proportions corresponding to the respective intervals that are just less than the first resource proportion may also be determined as similar resource proportions.

[0076] As an example, Figure 5 Taking the embodiment shown as an example, in cluster C3, all video resources from site A do not have a feature value of the "number of likes" feature. Figure 6 is the distribution histogram of the “number of likes” feature of the video resources from site A in the video dataset. Figure 5 It can be seen that the proportion of first resources whose feature value of the "number of likes" feature is less than or equal to the target value 10 in the target site C is 0.12. Figure 6 From the distribution histogram of the video resources of site A on the "likes" feature, it can be seen that the proportion of resources with 0 likes of site A's video resources is 0.03, the proportion of resources corresponding to the (0,10] interval is 0.07, the proportion of resources corresponding to the (10,100] interval is 0.3, the proportion of resources corresponding to the (100,1000] interval is 0.64, the proportion of resources corresponding to the (1000,10000] interval is 0.86, and the proportion of resources corresponding to the (10000,100000] interval is 0.98. Among the resource proportions corresponding to multiple intervals, the resource proportion of 0.3 corresponding to the (10,100] interval is just greater than the first resource proportion of 0.12, so the resource proportion of 0.3 corresponding to the (10,100] interval can be determined as the similar resource proportion, and the (10,100] interval can be determined as the second target interval.

[0077] As another example, in cluster C4, all video resources from site A do not have a feature value of the "number of comments" feature, while video resources from site C do have a feature value of the "number of comments" feature, so site C can be used as the target site.

[0078] Figure 7is the distribution histogram of the video resources from site A in the video dataset on the “number of comments” feature. Assume that in cluster C4, the first resource in the target site C accounts for 0.29. Figure 7 As shown, the proportion of resources with 0 comments on the video resources of site A is 0.16, the proportion of resources corresponding to the (0,10] interval is 0.28, the proportion of resources corresponding to the (10,100] interval is 0.48, the proportion of resources corresponding to the (100,1000] interval is 0.72, the proportion of resources corresponding to the (1000,10000] interval is 0.92, and the proportion of resources corresponding to the (10000,100000] interval is 0.94. Among the resource proportions corresponding to multiple intervals, the resource proportion of 0.28 corresponding to the (0,10] interval is just less than the first resource proportion of 0.29, so the resource proportion of 0.28 corresponding to the (0,10] interval can be determined as the similar resource proportion, and the (0,10] interval can be determined as the second target interval.

[0079] Step 410: Obtain a target filling value according to the second target interval range.

[0080] Optionally, a characteristic value may be randomly selected within the second target interval as a target filling value.

[0081] Step 411, filling feature values ​​of the i-th feature of all video resources from the same site according to the target filling value.

[0082] That is to say, Figure 5 , Figure 6 In the illustrated embodiment, according to the target filling value, the “number of likes” feature of all video resources from site A in cluster C3 is filled with feature values.

[0083] Step 412, in response to the existence of a feature value of the i-th feature in some video resources in the same cluster that originate from the same site, fill in the feature value of the i-th feature of other video resources in the same cluster that originate from the same site as at least some of the video resources based on the feature value of the i-th feature of at least some of the video resources.

[0084] In the embodiments of the present application, steps 401 to 405 and step 412 may be implemented in any of the embodiments of the present application, and the present application does not make any specific limitation on this and will not elaborate on it.

[0085] According to the data processing method of the embodiment of the present application, by performing similarity clustering on the video resources in the data set, for video resources where the feature value of the i-th feature does not exist in all video resources from the same site in the same clustering cluster, considering the differences in the distribution of video features of different sites, the feature filling value is determined based on the mapping relationship between the distribution histogram of the site to be filled in the feature to be filled in the same clustering cluster and the distribution histogram of the target site on the feature to be filled in the video resource, and the feature value is filled for the video resources with missing feature values ​​from the same site in the clustering cluster. The present application makes full use of the existing video resources for feature filling, which is low-cost and does not affect the filling effect, is more in line with the actual situation, and further improves the accuracy of feature filling. The present application can also improve the feature coverage of video resources, and train downstream modules based on the data set after feature filling, thereby optimizing the model effect.

[0086] Figure 8 is a structural block diagram of a data processing device provided according to an embodiment of the present application. Figure 8 As shown, the data processing device includes an acquisition module 801 , a clustering module 802 , a determination module 803 and a first filling module 804 .

[0087] Specifically, the acquisition module 801 is used to acquire a video data set with features to be filled, wherein each video resource in the video data set contains N features, and the video resources in the video data set come from multiple different sites, and N is a positive integer.

[0088] The clustering module 802 is used to perform similarity clustering processing on the video resources in the video data set to obtain at least one cluster.

[0089] The determination module 803 is used to determine the distribution histogram of the video resources from the same site in the video data set on each feature.

[0090] The first filling module 804 is used to fill the feature values ​​of the video resources with missing feature values ​​in each cluster according to the distribution histogram of each feature of the video resources from the same site in the video data set.

[0091] In some embodiments of the present application, the determination module 803 is specifically used to: determine multiple interval ranges corresponding to the at least one feature; based on each feature value of the video resources originating from the same site in the video data set, count the number of video resources originating from the same site in the video data set distributed in each of the interval ranges; based on the number distributed in each of the interval ranges, determine the distribution histogram of the video resources originating from the same site in the video data set on each feature.

[0092] In some embodiments of the present application, the first filling module 804 is specifically used to: for the i-th feature, in response to the fact that all video resources in the same cluster originating from the same site do not have a feature value of the i-th feature, determine a target site from the same cluster that has a feature value of the i-th feature; wherein i is a positive integer less than or equal to N; determine a distribution histogram of the video resources of the target site on the i-th feature based on a distribution histogram of each feature of the video resources originating from the same site in the video data set; and fill the feature value of the i-th feature of all video resources originating from the same site based on the feature value of the i-th feature of the video resources originating from the target site and the distribution histogram of the video resources of the target site on the i-th feature.

[0093] In some embodiments of the present application, the first filling module 804 is also used to: calculate the feature average value of the i-th feature of the target site based on the feature value of the i-th feature of the video resources originating from the target site; map the distribution histogram of the video resources at the same site on the i-th feature based on the feature average value of the i-th feature of the target site and the distribution histogram of the video resources of the target site on the i-th feature to obtain a target filling value; and fill the feature value of the i-th feature of all video resources originating from the same site according to the target filling value.

[0094] In some embodiments of the present application, the first filling module 804 is also used to: determine the first target interval range in which the feature average value falls according to the feature average value of the i-th feature of the target site and the distribution histogram of the video resources of the target site on the i-th feature; determine the first resource proportion in which the feature value of the i-th feature in the target site is less than or equal to the target value according to the distribution histogram of the video resources of the target site on the i-th feature; wherein the target value is the upper limit value within the first target interval range; determine a second target interval range having a similar resource proportion to the first resource proportion from the distribution histogram of the video resources of the same site on the i-th feature; and obtain a target filling value according to the second target interval range.

[0095] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0096] According to the data processing device of the embodiment of the present application, by clustering the video resources in the data set by similarity, using the distribution histogram of each feature of the video resources from the same site in the video data set, and the video resources with similar video features in the same cluster, the feature value of the video resources missing in each cluster is filled, and the existing video resources are fully utilized for feature filling, which is low-cost and does not affect the filling effect. The present application can also improve the feature coverage of video resources, and train downstream modules based on the data set after feature filling, thereby optimizing the model effect.

[0097] Fig. 9 A structural block diagram of another data processing device provided according to an embodiment of the present application. In some embodiments of the present application, such as Fig. 9 As shown, based on the above embodiment, the data processing device also includes a second filling module 905.

[0098] Specifically, the second filling module 905 is used to respond to the existence of the feature value of the i-th feature in some video resources originating from the same site in the same cluster, and fill the feature value of the i-th feature of other video resources in the same cluster that originate from the same site as at least part of the video resources according to the feature value of the i-th feature of at least part of the video resources.

[0099] in, Fig. 9 901-904 and Figure 8 801-804 have the same function and structure.

[0100] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0101] According to the data processing device of the embodiment of the present application, by performing similarity clustering on the video resources in the data set, for video resources in the same clustering cluster where all video resources originating from the same site do not have the feature value of the i-th feature, the target site video resources in the same clustering cluster that have the feature value of the i-th feature, and the distribution histogram of the target site on each feature, are used to fill the feature value of the video resources in the clustering cluster that are missing the feature value of the i-th feature. For the case where some video resources originating from the same site in the same clustering cluster have the feature value of the i-th feature, the video resources in the same site that have the feature value of the i-th feature are used to fill the feature of the video resources that are missing the feature value of the i-th feature of the same site. The present application makes full use of the existing video resources for feature filling, which is low in cost and does not affect the filling effect, thereby improving the accuracy of feature filling. The present application can also improve the feature coverage of video resources, and train downstream modules based on the data set after feature filling, thereby optimizing the model effect.

[0102] According to an embodiment of the present application, the present application also provides an electronic device, a readable storage medium and a computer program product.

[0103] like Fig.10 , is a block diagram of an electronic device for implementing a data processing method according to an embodiment of the present application. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present application described and / or required herein.

[0104] like Fig.10 As shown, the electronic device includes: one or more processors 1001, a memory 1002, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. The various components are interconnected using different buses and can be installed on a common mainboard or installed in other ways as needed. The processor can process instructions executed in the electronic device, including instructions stored in or on the memory to display the graphical information of the GUI on an external input / output device (such as a display device coupled to the interface). In other embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Similarly, multiple electronic devices can be connected, and each device provides some necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). Fig.10 A processor 1001 is taken as an example.

[0105] The memory 1002 is a non-transient computer-readable storage medium provided in the present application. The memory stores instructions executable by at least one processor to enable the at least one processor to perform the data processing method provided in the present application. The non-transient computer-readable storage medium of the present application stores computer instructions, which are used to enable a computer to perform the data processing method provided in the present application.

[0106] The memory 1002 is a non-transient computer-readable storage medium that can be used to store non-transient software programs, non-transient computer executable programs and modules, such as program instructions / modules corresponding to the data processing method in the embodiment of the present application (for example, the attached Figure 8 The acquisition module 801, clustering module 802, determination module 803 and first filling module 804 shown in the figure, and the attached Fig. 9 The processor 1001 executes various functional applications and data processing of the server by running the non-transient software programs, instructions and modules stored in the memory 1002, that is, the data processing method in the above method embodiment is implemented.

[0107] The memory 1002 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required by at least one function; the data storage area may store data created by the use of an electronic device for implementing the data processing method, etc. In addition, the memory 1002 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory 1002 may optionally include a memory remotely arranged relative to the processor 1001, and these remote memories may be connected to the electronic device for implementing the data processing method via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0108] The electronic device for implementing the data processing method may further include: an input device 1003 and an output device 1004. The processor 1001, the memory 1002, the input device 1003 and the output device 1004 may be connected via a bus or other means. Fig.10 The example of connecting through bus is taken in the following.

[0109] The input device 1003 can receive input digital or character information, and generate key signal input related to user settings and function control of the electronic device used to implement the data processing method, such as a touch screen, a keypad, a mouse, a track pad, a touch pad, an indicator rod, one or more mouse buttons, a trackball, a joystick and other input devices. The output device 1004 may include a display device, an auxiliary lighting device (e.g., an LED) and a tactile feedback device (e.g., a vibration motor), etc. The display device may include, but is not limited to, a liquid crystal display (LCD), a light emitting diode (LED) display and a plasma display. In some embodiments, the display device may be a touch screen.

[0110] Various embodiments of the systems and techniques described herein can be implemented in digital electronic circuit systems, integrated circuit systems, dedicated ASICs (application-specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include: being implemented in one or more computer programs, that is, the present application also proposes a computer program, which, when executed by a processor, implements the data processing method described in the above embodiments, and the one or more computer programs can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a dedicated or general programmable processor, which can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0111] These computer programs (also referred to as programs, software, software applications, or code) include machine instructions for programmable processors and can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, device, and / or means (e.g., disk, optical disk, memory, programmable logic device (PLD)) for providing machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal for providing machine instructions and / or data to a programmable processor.

[0112] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0113] The systems and techniques described herein can be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), the Internet, and a blockchain network.

[0114] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship between the client and the server is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services ("Virtual Private Server", or "VPS" for short). The server may also be a server of a distributed system, or a server combined with a blockchain.

[0115] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution disclosed in this application can be achieved, and this document is not limited here.

[0116] The above specific implementations do not constitute a limitation on the protection scope of this application. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions and improvements made within the spirit and principles of this application should be included in the protection scope of this application.

Claims

1. A data processing method, comprising: Obtain a video data set with features to be filled; wherein each video resource in the video data set contains N features, and the video resources in the video data set come from multiple different sites; N is a positive integer; Performing similarity clustering processing on the video resources in the video data set to obtain at least one cluster; Determine a distribution histogram of video resources from the same site in the video data set on each feature; wherein determining a distribution histogram of video resources from the same site in the video data set on each feature includes: Determine a plurality of interval ranges corresponding to the N features; According to each feature value of the video resources from the same site in the video data set, counting the number of video resources from the same site in the video data set distributed in each interval range; Determine, according to the number of distributions in each of the intervals, a distribution histogram of video resources from the same site in the video data set on each feature; According to the distribution histogram of each feature of the video resources from the same site in the video data set, feature values ​​are filled for the video resources with missing feature values ​​in each cluster; wherein, according to the distribution histogram of each feature of the video resources from the same site in the video data set, feature values ​​are filled for the video resources with missing feature values ​​in each cluster, including: For the i-th feature, in response to the fact that all video resources from the same site in the same cluster do not have a feature value of the i-th feature, a target site that has a feature value of the i-th feature is determined from the same cluster; wherein i is a positive integer less than or equal to N; Determine a distribution histogram of the video resources of the target site on the i-th feature according to a distribution histogram of the video resources from the same site in the video data set on each feature; According to the feature value of the i-th feature of the video resources from the target site and the distribution histogram of the video resources of the target site on the i-th feature, feature value filling is performed on the i-th feature of all the video resources from the same site.

2. The method of claim 1, wherein: The step of filling the feature value of the i-th feature of all the video resources from the same site according to the feature value of the i-th feature of the video resources from the target site and the distribution histogram of the video resources of the target site on the i-th feature comprises: Calculating a feature average value of the i-th feature of the target site according to the feature value of the i-th feature of the video resource from the target site; According to the feature average value of the i-th feature of the target site and the distribution histogram of the video resources of the target site on the i-th feature, the distribution histogram of the video resources of the same site on the i-th feature is mapped to obtain a target filling value; According to the target filling value, the i-th feature of all the video resources from the same site is filled with feature values.

3. The method of claim 2, wherein: The step of mapping the distribution histogram of the video resources of the target site on the i-th feature according to the feature average value of the i-th feature of the target site and the distribution histogram of the video resources of the target site on the i-th feature to obtain the target filling value includes: Determine, according to a feature average value of the i-th feature of the target site and a distribution histogram of the video resources of the target site on the i-th feature, a first target interval range within which the feature average value falls; Determine, according to a distribution histogram of the video resources of the target site on the i-th feature, a first resource proportion of the target site whose feature value of the i-th feature is less than or equal to a target value; wherein the target value is an upper limit value within the first target interval; Determine, from the distribution histogram of the video resources of the same site on the i-th feature, a second target interval range having a resource proportion similar to the first resource proportion; A target filling value is obtained according to the second target interval range.

4. The method of claim 1, further comprising: In response to the existence of a feature value of the i-th feature in some video resources originating from the same site in the same cluster, the feature value of the i-th feature of other video resources in the same cluster that originate from the same site as the part of the video resources is filled according to the feature value of the i-th feature of the part of the video resources.

5. The method according to any one of claims 1 to 4, wherein: The performing similarity clustering processing on the video resources in the video data set to obtain at least one clustering cluster includes: Processing each video resource in the video data set to obtain video fingerprint information of each video resource; Based on the video fingerprint information of each video resource, similarity clustering processing is performed on the video resources in the video data set to obtain at least one cluster.

6. A data processing device, comprising: An acquisition module, used to acquire a video data set with features to be filled; wherein each video resource in the video data set contains N features, and the video resources in the video data set come from multiple different sites; and N is a positive integer; A clustering module, used to perform similarity clustering processing on the video resources in the video data set to obtain at least one clustering cluster; A determination module is used to determine a distribution histogram of video resources from the same site in the video data set on each feature; wherein the determination module is specifically used to: Determine a plurality of interval ranges corresponding to the N features; According to each feature value of the video resources from the same site in the video data set, counting the number of video resources from the same site in the video data set distributed in each interval range; Determine, according to the number of distributions in each of the intervals, a distribution histogram of video resources from the same site in the video data set on each feature; A first filling module is used to fill feature values ​​of video resources with missing feature values ​​in each cluster according to the distribution histogram of each feature of video resources from the same site in the video data set; wherein the first filling module is specifically used to: For the i-th feature, in response to the fact that all video resources from the same site in the same cluster do not have a feature value of the i-th feature, a target site that has a feature value of the i-th feature is determined from the same cluster; wherein i is a positive integer less than or equal to N; Determine a distribution histogram of the video resources of the target site on the i-th feature according to a distribution histogram of the video resources from the same site in the video data set on each feature; According to the feature value of the i-th feature of the video resources from the target site and the distribution histogram of the video resources of the target site on the i-th feature, feature value filling is performed on the i-th feature of all the video resources from the same site.

7. The device according to claim 6, wherein: The first filling module is also used for: Calculating a feature average value of the i-th feature of the target site according to the feature value of the i-th feature of the video resource from the target site; According to the feature average value of the i-th feature of the target site and the distribution histogram of the video resources of the target site on the i-th feature, the distribution histogram of the video resources of the same site on the i-th feature is mapped to obtain a target filling value; According to the target filling value, the i-th feature of all the video resources from the same site is filled with feature values.

8. The device according to claim 7, wherein: The first filling module is also used for: Determine, according to a feature average value of the i-th feature of the target site and a distribution histogram of the video resources of the target site on the i-th feature, a first target interval range within which the feature average value falls; Determine, according to a distribution histogram of the video resources of the target site on the i-th feature, a first resource proportion of the target site whose feature value of the i-th feature is less than or equal to a target value; wherein the target value is an upper limit value within the first target interval; Determine, from the distribution histogram of the video resources of the same site on the i-th feature, a second target interval range having a resource proportion similar to the first resource proportion; A target filling value is obtained according to the second target interval range.

9. The apparatus of claim 6, further comprising: The second filling module is used to respond to the existence of the feature value of the i-th feature in some video resources originating from the same site in the same cluster, and fill the feature value of the i-th feature of other video resources in the same cluster that originate from the same site as the part of the video resources according to the feature value of the i-th feature of the part of the video resources.

10. The device according to any one of claims 6 to 9, wherein: The clustering module is specifically used for: Processing each video resource in the video data set to obtain video fingerprint information of each video resource; Based on the video fingerprint information of each video resource, similarity clustering processing is performed on the video resources in the video data set to obtain at least one cluster.

11. An electronic device, comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 5.

12. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1 to 5.

13. A computer program product, comprising a computer program, which, when executed by a processor, implements the steps of the method according to any one of claims 1 to 5.

Images