Verification method and device, electronic equipment and storage medium

Abstract

The present disclosure relates to a verification method and device, electronic equipment and a storage medium. The verification method comprises: acquiring multimedia resource usage information of a plurality of sample accounts within a preset time period; for each sample account in the plurality of sample accounts, determining repeated use event information for each sample account for representing repeated use of the same multimedia resource based on the multimedia resource usage information of each sample account acquired this time and the multimedia resource usage information of each sample account acquired in previous times; and verifying the deduplication performance of a multimedia resource recommendation model based on the repeated use event information for the plurality of sample accounts. The multimedia resource usage information is acquired in an offline manner, and then the deduplication performance of the multimedia resource recommendation model is verified, without accessing offline historical service data, which can avoid the situation that the offline historical service data is misoperated, thereby ensuring the stability of multimedia resource recommendation.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of computer, and more particularly, to a verification method and device, an electronic device and a storage medium. BACKGROUND

[0002] With the continuous increase of multimedia resource users, the number of various problems fed back by users also increases sharply. Through the analysis of the collected multimedia resource recommendation problems, it is found that the problem of repeated recommendation of multimedia resources is the main problem affecting user experience.

[0003] In the related art, when testing whether the deduplication performance of the multimedia resource recommendation model meets the requirements, the online historical service data of the multimedia resource recommendation model needs to be accessed, and based on the access result of the online historical service data, it is determined whether there is a case of repeatedly recommending the same multimedia resource, and then the deduplication performance of the multimedia resource recommendation model is evaluated according to the statistical result. However, using this method to test the deduplication performance of the multimedia resource recommendation model may cause the online historical service data to be misoperated, thereby degrading the recommendation performance of the multimedia resource. SUMMARY

[0004] The present disclosure provides a verification method and device, an electronic device and a storage medium to at least solve the problem that in the related art, when testing whether the deduplication performance of the multimedia resource recommendation model meets the requirements, the online historical service data may be misoperated, thereby degrading the recommendation performance of the multimedia resource.

[0005] According to a first aspect of an embodiment of the present disclosure, a verification method is provided, the verification method comprising: obtaining multimedia resource usage information of a plurality of sample accounts within a preset time period, wherein the multimedia resource usage information of a sample account is used to describe the usage of a multimedia resource recommended by a multimedia resource recommendation model by the sample account; for each sample account in the plurality of sample accounts, based on the multimedia resource usage information of the each sample account obtained this time and the multimedia resource usage information of the each sample account obtained in previous times, determining repeated usage event information of the each sample account, which is used to represent that the same multimedia resource is repeatedly used; and based on the repeated usage event information of the plurality of sample accounts, verifying the deduplication performance of the multimedia resource recommendation model, wherein the deduplication performance indicates the performance of avoiding repeatedly recommending the same multimedia resource.

[0006] Optionally, the step of verifying the deduplication performance of the multimedia resource recommendation model based on the repeated use event information of the plurality of sample accounts comprises: obtaining an index value of each deduplication evaluation index based on the repeated use event information of the plurality of sample accounts; and determining that the deduplication performance of the multimedia resource recommendation model is abnormal when the index value of the determined deduplication evaluation index exceeds a corresponding alarm threshold.

[0007] Optionally, the step of obtaining an index value of each deduplication evaluation index based on the repeated use event information of the plurality of sample accounts comprises: determining, for each time interval in the preset time period, a first ratio between the number of repeated use events in the each time interval and the number of multimedia resources used in the each time interval; and taking a statistical value of the first ratio corresponding to each time interval as the index value of the deduplication evaluation index.

[0008] Optionally, the step of obtaining an index value of each deduplication evaluation index based on the repeated use event information of the plurality of sample accounts comprises: determining, for each time interval in the preset time period, a second ratio between the sum of repeated use times of the multimedia resources repeatedly used as indicated by the repeated use events in the each time interval and the number of multimedia resources used in the each time interval; and taking a statistical value of the second ratio corresponding to each time interval as the index value of the deduplication evaluation index.

[0009] Optionally, the step of determining, for each sample account in the plurality of sample accounts, repeated use event information for representing that a same multimedia resource is repeatedly used based on the multimedia resource use information of the each sample account obtained this time and the multimedia resource use information of the each sample account obtained in previous times comprises: performing repeated determination on the multimedia resource use information of the each sample account obtained this time with respect to each sample account to obtain first repeated use statistical information; performing repeated determination on the multimedia resource use information of the each sample account obtained this time and the multimedia resource use information of the each sample account obtained in previous times with respect to each sample account to obtain second repeated use statistical information; and determining, based on the first repeated use statistical information and the second repeated use statistical information of the each sample account, the repeated use event information for representing that a same multimedia resource is repeatedly used with respect to the each sample account.

[0010] Optionally, the verification method further comprises: performing a modulo operation between each account in the plurality of accounts that report multimedia resource usage information within the preset time length and a preset sampling value to obtain at least one operation result; and determining, as the plurality of sampling accounts, the accounts in the plurality of accounts for which the corresponding operation results are preset operation results.

[0011] According to a second aspect of the embodiments of the present disclosure, a verification device is provided, comprising: an acquisition module configured to acquire multimedia resource usage information of a plurality of sampling accounts within a preset time length, wherein the multimedia resource usage information of a sampling account is used to describe usage of multimedia resources recommended by a multimedia resource recommendation model by the sampling account; a first determination module configured to, for each sampling account in the plurality of sampling accounts, determine, based on the multimedia resource usage information of the each sampling account acquired this time and the multimedia resource usage information of the each sampling account acquired in previous times, repeated usage event information of the each sampling account for indicating repeated usage of a same multimedia resource; and a verification module configured to verify, based on the repeated usage event information of the plurality of sampling accounts, a deduplication performance of the multimedia resource recommendation model, wherein the deduplication performance indicates a performance of avoiding repeated recommendation of a same multimedia resource.

[0012] Optionally, the verification module is configured to: obtain an index value of each deduplication evaluation index based on the repeated usage event information of the plurality of sampling accounts; and determine that the deduplication performance of the multimedia resource recommendation model is abnormal when the index value of the determined deduplication evaluation index exceeds a corresponding alarm threshold.

[0013] Optionally, the verification module is configured to: for each time interval in each time interval included in the preset time length, determine a first ratio between a number of repeated usage events in the each time interval and a number of multimedia resources used in the each time interval; and take a statistical value of the corresponding first ratios of the each time interval as the index value of the deduplication evaluation index.

[0014] Optionally, the verification module is configured to: for each time interval in each time interval included in the preset time length, determine a second ratio between a sum of repeated usage times of multimedia resources repeatedly used as indicated by repeated usage events in the each time interval and a number of multimedia resources used in the each time interval; and take a statistical value of the corresponding second ratios of the each time interval as the index value of the deduplication evaluation index.

[0015] Optionally, the first determining module is configured to: for each sample account, repeatedly determine the multimedia resource usage information of the sample account obtained this time from each other to obtain first repeated usage statistical information; for each sample account, repeatedly determine the multimedia resource usage information of the sample account obtained this time and the multimedia resource usage information of the sample account obtained in previous times to obtain second repeated usage statistical information; and determine, based on the first repeated usage statistical information and the second repeated usage statistical information of the sample account, repeated usage event information of the sample account for characterizing repeated usage of the same multimedia resource.

[0016] Optionally, the verification apparatus further includes: a modulo operation module configured to perform modulo operation between each account in the plurality of accounts that report multimedia resource usage information within the preset time length and a preset sampling value to obtain at least one operation result; and a second determining module configured to determine, as the plurality of sample accounts, the accounts in the plurality of accounts whose corresponding operation results are preset operation results.

[0017] According to a third aspect of embodiments of the present disclosure, an electronic device is provided, including: a processor; a memory for storing instructions executable by the processor; and wherein the processor is configured to execute the instructions to implement the verification method according to the present disclosure.

[0018] According to a fourth aspect of embodiments of the present disclosure, a computer readable storage medium is provided, when instructions in the computer readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to perform the verification method according to the present disclosure.

[0019] According to a fifth aspect of embodiments of the present disclosure, a computer program product is provided, including a computer program, when the computer program is executed by a processor, the verification method according to the present disclosure is implemented.

[0020] The technical solutions provided by the embodiments of the present disclosure at least bring the following beneficial effects:

[0021] The present disclosure obtains multimedia resource usage information in an offline manner, and verifies the deduplication performance of a multimedia resource recommendation model based on the obtained multimedia resource usage information, without accessing online historical service data, so that the situation that the online historical service data is misoperated can be avoided, the online historical service data can be ensured not to be damaged, and the stability of multimedia resource recommendation is ensured.

[0022] It should be understood that the foregoing general description and the following detailed description are only exemplary and explanatory, and cannot limit the present disclosure. BRIEF DESCRIPTION OF DRAWINGS

[0023] The accompanying drawings, which are incorporated herein and constitute part of the specification, illustrate implementations of the embodiments consistent with the present disclosure and, together with the description, further serve to explain the principles of the present disclosure and, do not limit the present disclosure in any inappropriable way.

[0024] Figure 1 is a flowchart illustrating a verification method according to an example embodiment of the present disclosure;

[0025] Figure 2 is a detailed implementation flowchart illustrating a verification method according to an example embodiment of the present disclosure;

[0026] Figure 3 is a block diagram illustrating a verification device according to an example embodiment of the present disclosure;

[0027] Figure 4 is a block diagram illustrating an electronic device according to an example embodiment of the present disclosure. DETAILED DESCRIPTION

[0028] In order for those skilled in the art to better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be described clearly and completely below with reference to the drawings.

[0029] It should be noted that the terms "first", "second", and the like in the specification and claims of the present disclosure and the above-described drawings are used to distinguish similar objects, and do not necessarily have to describe a specific order or a chronological sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the present disclosure described herein can be implemented in an order other than that illustrated or described herein. The embodiments described in the following embodiments do not represent all the embodiments consistent with the present disclosure. Rather, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0030] It should be noted that "at least one of a plurality of items" appearing in the present disclosure means that the three types of alternatives of "any one of the plurality of items", "a combination of any two or more of the plurality of items", and "all of the plurality of items" are included. For example, "including at least one of A and B" includes the following three alternatives: (1) including A; (2) including B; and (3) including A and B. For another example, "performing at least one of step one and step two" means the following three alternatives: (1) performing step one; (2) performing step two; and (3) performing step one and step two.

[0031] Figure 1 is a flowchart illustrating a verification method according to an example embodiment of the present disclosure.

[0032] Referring toFigure 1 At step 101, multimedia resource usage information of a plurality of sample accounts within a preset time period can be acquired. The multimedia resource usage information of the sample accounts is used to describe the usage of the multimedia resources recommended by the multimedia resource recommendation model by the sample accounts.

[0033] It should be noted that the aforementioned "preset time period" can be a recent time period, for example, the recent one day, the recent one week, or the recent one month, etc., which can be set according to actual conditions. In each time interval within the recent time period, a large number of accounts can send multimedia resource acquisition requests to the multimedia resource recommendation model, for example, in each time interval, there can be tens of thousands of accounts simultaneously requesting multimedia resources from the multimedia resource recommendation model. The multimedia resource recommendation model can return at least one multimedia resource to each account in response to the multimedia resource acquisition request from each account. After each account uses the multimedia resources pushed by the multimedia resource recommendation model, the terminal of the account transmits the information of the multimedia resources used by the account, i.e., the multimedia resource usage information, into the open source stream processing platform (Kafka). The disclosure can pull the multimedia resource usage information of a plurality of accounts from the Kafka, and check the deduplication performance of the multimedia resource recommendation model based on the pulled multimedia resource usage information of the plurality of accounts. The aforementioned "sample account" is an account extracted from a large number of accounts requesting multimedia resources from the multimedia resource recommendation model.

[0034] According to an example embodiment of the disclosure, the multimedia resource usage information of the sample account can include the identification (photoId) of the used multimedia resource, the usage timestamp of the multimedia resource, the usage type, and the sample account (userID). The usage type can include: display and / or play. The multimedia resource can be a video, a picture, an audio, or an article, etc.

[0035] At step 102, for each sample account in the plurality of sample accounts, the repeated usage event information for each sample account for characterizing the repeated usage of the same multimedia resource can be determined based on the multimedia resource usage information of each sample account acquired this time and the multimedia resource usage information of each sample account acquired in previous times.

[0036] According to an example embodiment of the disclosure, the repeated usage event information of the sample account can include the identification (photoId) of the used multimedia resource, the usage timestamp of each usage, the total number of repetitions of each usage type, and the sample account (userID). The usage type can include: display and / or play.

[0037] For example, assuming that the multimedia resource recommendation model pushes 8 videos to the terminal of the sampling account A this time, the sampling account A only browses 6 videos of the 8 videos, and does not play any video, that is, the terminal of the sampling account A only displays 6 videos of the 8 videos, and the video usage information corresponding to each of the 6 videos can be: "the userID of the sampling account A displaying the video - the photoId of the video - the timestamp of this time display of the video". Through repeated determination of the video usage information of the sampling account A in the history, it is known that the video has been displayed 3 times and played twice in the history by the terminal of the sampling account A, and the repeated use event information corresponding to the video can be: "the userID of the sampling account A displaying the video - the photoId of the video - the timestamp of this time display of the video - the video is repeatedly displayed 3 times - the video is repeatedly played 1 time".

[0038] It should be noted that the information format of the multimedia resource usage information and the repeated use event information is not limited to the above form, and can also be other forms. The above form is only an exemplary description.

[0039] According to the exemplary embodiments of the present disclosure, for each sampling account, the multimedia resource usage information of the sampling account obtained this time can be repeatedly determined with each other to obtain first repeated use statistical information. For each sampling account, the multimedia resource usage information of the sampling account obtained this time and the multimedia resource usage information of the sampling account obtained in the history can be repeatedly determined to obtain second repeated use statistical information. Next, based on the first repeated use statistical information and the second repeated use statistical information of the sampling account, the repeated use event information for representing that the same multimedia resource is repeatedly used can be determined for the sampling account.

[0040] For example, assuming that the terminal of the sampling account A only browses 6 videos of the 8 videos recommended by the multimedia resource recommendation model this time, and does not play any video, that is, the terminal of the sampling account A only displays 6 videos of the 8 videos, and assuming that the photoId of 2 videos of the 6 videos displayed by the terminal of the sampling account A is the same, that is, there are two same videos in the 6 videos displayed by the terminal of the sampling account A this time, the first repeated use statistical information obtained by repeatedly determining the video usage information of the sampling account A obtained this time with each other can be: "the userID of the sampling account A displaying the video - the photoId of the video - the two display timestamps of the video this time - the video is repeatedly displayed 1 time this time".

[0041] If the video usage information of the sample account A in the past is checked, it is known that the video is displayed twice this time, that is, the repeatedly displayed video is displayed 4 times and played 3 times in the history by the terminal of the sample account A, the second repeated use statistical information obtained by the repeated determination between the video usage information of the sample account A obtained this time and the video usage information of the sample account A obtained in the past can be: "the userID of the sample account A displaying the video-the photoId of the video-the video is repeatedly displayed 4 times in the history-the video is repeatedly played 2 times in the history". Next, based on the first repeated use statistical information of the sample account A: "the userID of the sample account A displaying the video-the photoId of the video-two display time stamps of the video displayed twice this time-the video is repeatedly displayed 1 time this time", and the second repeated use statistical information: "the userID of the sample account A displaying the video-the photoId of the video-the video is repeatedly displayed 4 times in the history-the video is repeatedly played 2 times in the history", it can be determined that the repeated use event information for the sample account A for representing the same video is: "the userID of the sample account A displaying the video-the photoId of the video-two display time stamps of the video displayed twice this time-the video is repeatedly displayed 5 times-the video is repeatedly played 2 times." In this way, for each sample account, when determining the repeated use event information of the sample account, the multimedia resource usage information of the sample account obtained this time and the multimedia resource usage information of the sample account obtained in the past can be repeatedly determined, and the multimedia resource usage information of the sample account obtained this time can also be repeatedly determined. Since the results of the two repeated determinations are considered, more factors are considered, and the final repeated use event information of the sample account can be more perfect.

[0042] In step 103, based on the repeated use event information of the plurality of sample accounts, the deduplication performance of the multimedia resource recommendation model can be verified, wherein the deduplication performance is used to indicate the performance of avoiding repeatedly recommending the same multimedia resource.

[0043] According to the exemplary embodiments of the present disclosure, the index value of each deduplication evaluation index can be obtained based on the repeated use event information of the plurality of sample accounts. When the index value of the determined deduplication evaluation index exceeds the corresponding alarm threshold, it can be determined that the deduplication performance of the multimedia resource recommendation model is abnormal.

[0044] According to the exemplary embodiments of the present disclosure, in each of the time intervals contained in the recent time period, a large number of accounts can send multimedia resource obtaining requests to the multimedia resource recommendation model, for example, in each time interval, ten thousand accounts can simultaneously request multimedia resources from the multimedia resource recommendation model. Therefore, the number of final reuse event information is also relatively large, and the format of each "reuse event information" can be "userID of the sample account that displays the video-photoId of the video-current display timestamp of the video-current play timestamp of the video (if exists)-number of repeated displays of the video (if exists)-number of repeated plays of the video (if exists)".

[0045] According to the exemplary embodiments of the present disclosure, for each of the time intervals contained in the preset time period, a first ratio between the number of reuse events in each time interval and the number of multimedia resources used in each time interval can be determined. For example, based on the "current display timestamp" contained in the reuse event information of the plurality of sample accounts, the number of reuse events of the "current display timestamp" in the time interval can be counted, and the first ratio between the number of reuse events of the "current display timestamp" and the number of multimedia resources used in the time interval can be counted. Next, the statistical value of the first ratio corresponding to each time interval can be used as the index value of the deduplication evaluation index. For example, the average value of the first ratio corresponding to each time interval can be used as the index value of the deduplication evaluation index. The larger the average value of the first ratio corresponding to each time interval, the more multimedia resources are reused, and the worse the deduplication performance of the multimedia resource recommendation model.

[0046] In addition, each of the aforementioned time intervals can be 3 minutes or 5 minutes, etc., which can be flexibly configured according to actual conditions.

[0047] According to an example embodiment of the present disclosure, for each of the time intervals contained in the preset time length, a second ratio between the sum of the reuse times of the reused multimedia resources indicated by each of the reuse events in each of the time intervals and the number of the multimedia resources used in each of the time intervals can be determined. For example, the sum of the reuse times of the reused multimedia resources indicated by each of the reuse events in which the current display timestamp is located in the time interval can be counted based on the current display timestamp contained in the reuse event information of the plurality of sampling accounts. For example, assuming that there are 3 reuse events in which the current display timestamp is located in the time interval, and the reuse times of the reused multimedia resources indicated by the 3 reuse events are 2 times, 3 times and 5 times respectively, the sum of the reuse times of the reused multimedia resources indicated by each of the reuse events in which the current display timestamp is located in the time interval is 2+3+5=10 times.

[0048] Next, the statistical value of the second ratio corresponding to each of the time intervals can be taken as the index value of the deduplication evaluation index. For example, the average value of the second ratio corresponding to each of the time intervals can be taken as the index value of the deduplication evaluation index. The larger the average value of the second ratio corresponding to each of the time intervals, the more the reused multimedia resources, and the more likely the reused multimedia resources are reused multiple times, and in this case, it can be determined that the deduplication performance of the multimedia resource recommendation model is poor.

[0049] According to an example embodiment of the present disclosure, as described above, in each of the plurality of time intervals contained in the recent time period, a large number of accounts can send multimedia resource acquisition requests to the multimedia resource recommendation model, for example, tens of thousands of accounts can simultaneously request multimedia resources from the multimedia resource recommendation model in each of the time intervals. However, as a test task for verifying the deduplication performance of the multimedia resource recommendation model, it is not necessary to test with such a large amount of data, and therefore, a large number of accounts sending multimedia resource acquisition requests to the multimedia resource recommendation model can be sampled, and only the sampling accounts obtained by sampling can be tested. For example, a modulo operation can be performed between each of the plurality of accounts reporting multimedia resource usage information in the preset time length and a preset sampling value, at least one operation result can be obtained, and then the accounts corresponding to the operation results of the plurality of accounts being the preset operation result can be determined as the plurality of sampling accounts.

[0050] For example, the preset sampling value can be 600, each account is a string of numbers, a modulo operation can be performed between each account and the preset sampling value 600, the remainder corresponding to the account is obtained, and then the accounts corresponding to the remainder being 1 can be selected from the plurality of accounts as the sampling accounts.

[0051] It should be noted that in the related art, when testing whether the deduplication performance of the multimedia resource recommendation model meets the requirements, the online distribution history service data of the multimedia resource recommendation model needs to be accessed. However, as a test task, it is not safe to access the online distribution history service data, and it is likely that the online distribution history service data will be misoperated.

[0052] For example, some data in the online distribution history service data may be mistakenly deleted. Assuming that the data related to video A in the online distribution history service data is mistakenly deleted, the multimedia resource recommendation model will consider that video A has never been pushed to the terminal of the user in the history process, and will push video A, which has actually been pushed in the history process, to the terminal of the user, causing the user to see video A again, which affects the user experience.

[0053] Or, some data related to a video may be mistakenly written into the online distribution history service data. Assuming that the data related to video B is mistakenly written into the online distribution history service data, the multimedia resource recommendation model will consider that video B has been pushed to the terminal of the user in the history process, and will not push video B, which has not actually been pushed, to the terminal of the user in order to avoid the user repeatedly seeing video B, causing the user to never see video B in the future, which affects the recommendation performance of the multimedia resource.

[0054] The verification method of the present disclosure is to obtain multimedia resource usage information in an offline manner. As mentioned earlier, each account in the plurality of accounts can upload the multimedia resource usage information to Kafka, and the present disclosure can pull the multimedia resource usage information of the plurality of accounts from Kafka and verify the deduplication performance of the multimedia resource recommendation model based on the obtained multimedia resource usage information. That is, the present disclosure is equivalent to maintaining a set of distribution history services itself, without accessing the online distribution history service data, which can avoid the misoperation of the online distribution history service data and ensure that the online distribution history service data is not damaged, thereby ensuring the stability of the multimedia resource recommendation.

[0055] Referring to Figure 2 , Figure 2 is a specific implementation flowchart of a verification method according to an example embodiment of the present disclosure. In Figure 2 , the specific implementation flowchart of the verification method is divided into an application layer, a processing layer and a storage layer at a logical level, and is described taking a video as an example.

[0056] Figure 2The middle part involves open source stream processing platform (Kafka), cross-platform open source metric analysis and visualization tool (Grafana) and open source stream processing framework (Apache Flink). Kafka is a high-throughput distributed publish-subscribe message system that can handle all action stream data of users in the website. Grafana can display the collected data in various visualized ways, such as heat map, line chart and the like. The core of Apache Flink is a distributed stream data stream engine written in Java and Scala. Flink executes any stream data program in a data parallel and pipeline manner. In addition, the running of Flink itself supports the execution of iterative algorithms.

[0057] In step 201, in the recent time period, each account in the plurality of accounts uploads the related information of the played video to Kafka in real time. The related information of the played video can be "userID of the account playing the video-photoId of the video-play timestamp". It should be noted that the related information of the played video can also include the type of the page playing the video, such as "attention page", "discovery page" or "selected page" of the video push software on the terminal of the sampled account and the like.

[0058] In step 202, in the recent time period, each account in the plurality of accounts uploads the related information of the played video to Kafka in real time. The related information of the played video can be "userID of the account playing the video-photoId of the video-play timestamp". It should be noted that the related information of the played video can also include the type of the page playing the video, such as "attention page", "discovery page" or "selected page" of the video push software on the terminal of the sampled account and the like.

[0059] In step 203, a plurality of sampled accounts are selected from the plurality of accounts by using the modulo operation, and the related information of the video played by each sampled account in the plurality of sampled accounts is pulled from Kafka by using flink sql. The sampling process has been described in the previous embodiment, and will not be repeated here.

[0060] In step 204, a plurality of sampled accounts are selected from the plurality of accounts by using the modulo operation, and the related information of the video played by each sampled account in the plurality of sampled accounts is pulled from Kafka by using flink sql.

[0061] In step 205, photoId-list is spliced and rewritten.

[0062] For example, if the terminal of the sampling account A displays and plays the P video, two pieces of information related to the P video will be pulled from Kafka, i.e., "the userID of the sampling account A displays the photoId of the P video and the display timestamp" and "the userID of the sampling account A plays the photoId of the P video and the play timestamp". At this time, the two pieces of information related to the P video can be combined into one piece of information, i.e., "the userID of the sampling account A displays the photoId of the P video and the display timestamp and the play timestamp". In this way, the storage resource can be saved by combining multiple pieces of information related to the video.

[0063] In step 206, the obtained combined information is stored in the storage system (Rocks DB) by splicing and rewriting multiple pieces of related information of the same video.

[0064] In step 207, the repeated use of the video is determined.

[0065] In this step, the repeated use of the video can include two steps: first, the multiple used videos returned by a certain sampling account this time can be compared with each other; then, the information of each video used by the sampling account in the historical process can be read from the Rocks DB, and each used video returned by the sampling account this time can be compared with each video used by the sampling account in the historical process. Further, the user-defined aggregation function (UDAF) can be used for comparison. The specific comparison process has been described in the previous embodiment, and will not be repeated here.

[0066] It should be noted that when determining the repeated use of the video, the types of pages mentioned in steps 201 and 202 can also be used for comparison. The repeated use of the video between each page type and itself can be determined, for example, the repeated use of the video between the "attention page" and itself can be determined; the repeated use of the video between each page type and each page type other than itself can be determined, for example, the repeated use of the video between the "attention page" and each page type other than itself, such as the "discovery page" and the "selected page"; the repeated use of the video between each page type and all page types including itself can be determined, for example, the repeated use of the video between the "attention page" and the "attention page", the "discovery page" and the "selected page" can be determined, and so on.

[0067] In step 208, the format of the repeated use of the video is adjusted to a unified format.

[0068] In step 209, the repeated use judgment result of the used video unified in the format is transmitted into Kafka.

[0069] In step 2010, the repeated use judgment result of the used video unified in the format is transmitted into the database (click house) through the klog data delivery mode.

[0070] In step 2011, the click house receives the repeated use judgment result of the used video unified in the format delivered by Kafka through the klog data delivery mode, which is the “repeated use event information” described in the previous embodiment.

[0071] It should be noted that the click house stores a lot of repeated use event information of the sampling accounts for representing that the video is repeatedly used. As described in the previous embodiment, the format of the “repeated use event information” can be “userID of the sampling account showing the video-photoId of the video-time stamp of this time showing the video-time stamp of this time playing the video (if exists)-number of times of repeatedly showing the video (if exists)-number of times of repeatedly playing the video (if exists)”.

[0072] In step 2012, the Grafana calculates the index value of each deduplication evaluation index.

[0073] It should be noted that the Grafana can read the repeated use event information of a lot of sampling accounts from the click house, and generate a curve graph of the repeatedly used videos in the recent time period based on the read repeated use event information of a lot of sampling accounts.

[0074] The horizontal axis of the curve graph can be time, and the vertical axis can be the number of the repeatedly used videos. At this time, the meaning of a data point in the curve graph is the number of the repeatedly used videos at a certain moment in the recent time period, and further, the “statistical value of the first ratio corresponding to each time interval” described in the previous embodiment can also be calculated by using the curve graph.

[0075] Alternatively, the horizontal axis of the curve graph can be time, and the vertical axis can be the sum of the number of times of repeatedly using the repeatedly used videos. At this time, the meaning of a data point in the curve graph is the sum of the number of times of repeatedly using all the repeatedly used videos at a certain moment in the recent time period, and further, the “statistical value of the second ratio corresponding to each time interval” described in the previous embodiment can also be calculated by using the curve graph.

[0076] At step 2013, when the index value of a certain de-duplication evaluation index exceeds the corresponding alarm threshold, Grafana outputs an alarm prompt information.

[0077] Figure 3 is a block diagram illustrating a verification device according to an example embodiment of the present disclosure.

[0078] With reference to Figure 3 The device 300 can include an acquisition module 301, a first determination module 302, and a verification module 303.

[0079] The acquisition module 301 is configured to acquire multimedia resource usage information of a plurality of sample accounts within a preset time period, wherein the multimedia resource usage information of a sample account is used to describe the usage of a multimedia resource recommended by a multimedia resource recommendation model by the sample account.

[0080] The first determination module 302 is configured to, for each sample account in the plurality of sample accounts, determine, based on the multimedia resource usage information of the each sample account acquired this time and the multimedia resource usage information of the each sample account acquired in previous times, repeated usage event information for the each sample account, which is used to represent repeated usage of a same multimedia resource.

[0081] The verification module 303 is configured to verify a de-duplication performance of the multimedia resource recommendation model based on the repeated usage event information for the plurality of sample accounts, wherein the de-duplication performance indicates a performance of avoiding repeated recommendation of a same multimedia resource.

[0082] According to an example embodiment of the present disclosure, the verification module 303 is configured to:

[0083] obtain an index value of each de-duplication evaluation index based on the repeated usage event information for the plurality of sample accounts;

[0084] determine that the de-duplication performance of the multimedia resource recommendation model is abnormal when the index value of the determined de-duplication evaluation index exceeds a corresponding alarm threshold.

[0085] According to an example embodiment of the present disclosure, the verification module 303 is configured to:

[0086] determine, for each time interval in each time interval included in the preset time period, a first ratio value between a number of repeated usage events in the each time interval and a number of multimedia resources used in the each time interval;

[0087] take a statistical value of the first ratio value corresponding to each time interval as the index value of the de-duplication evaluation index.

[0088] According to an example embodiment of the present disclosure, the checking module 303 is configured to:

[0089] determining, for each of the time intervals included in the preset time length, a second ratio between a sum of the number of times of reuse of the multimedia resource indicated by the reuse event in each of the time intervals and the number of the multimedia resources used in each of the time intervals;

[0090] taking the statistical value of the second ratio corresponding to each of the time intervals as the index value of the deduplication evaluation index.

[0091] According to an example embodiment of the present disclosure, the first determining module 302 is configured to:

[0092] for each of the sample accounts, repeatedly determining the multimedia resource usage information of the sample account obtained this time and the multimedia resource usage information of the sample account obtained in previous times to obtain first reuse statistical information;

[0093] for each of the sample accounts, repeatedly determining the multimedia resource usage information of the sample account obtained this time and the multimedia resource usage information of the sample account obtained in previous times to obtain second reuse statistical information;

[0094] based on the first reuse statistical information and the second reuse statistical information of each of the sample accounts, determining reuse event information of the same multimedia resource for each of the sample accounts.

[0095] According to an example embodiment of the present disclosure, the checking device further includes:

[0096] a modulo operation module configured to perform a modulo operation between each of a plurality of accounts reporting multimedia resource usage information in the preset time length and a preset sampling value to obtain at least one operation result;

[0097] a second determining module configured to determine, as the plurality of sample accounts, the accounts corresponding to the operation result being a preset operation result in the plurality of accounts.

[0098] Figure 4 is a block diagram illustrating an electronic device 400 according to an example embodiment of the present disclosure.

[0099] Referring to Figure 4 , the electronic device 400 includes at least one memory 401 and at least one processor 402, and the at least one memory 401 stores instructions which, when executed by the at least one processor 402, perform the checking method according to an example embodiment of the present disclosure.

[0100] As an example, the electronic device 400 can be a PC computer, a tablet device, a personal digital assistant, a smart phone, or other device capable of executing the instructions described above. Here, the electronic device 400 need not be a single electronic device, but can be a collection of any devices or circuits capable of executing the instructions (or set of instructions) individually or jointly. The electronic device 400 can also be part of an integrated control system or system manager, or can be configured as a portable electronic device that interfaces with local or remote (e.g., via wireless transmission) devices.

[0101] In the electronic device 400, the processor 402 can include a central processor (CPU), a graphics processor (GPU), a programmable logic device, a dedicated processor system, a microcontroller, or a microprocessor. As an example and not by way of limitation, the processor can also include an analog processor, a digital processor, a microprocessor, a multi-core processor, a processor array, a network processor, etc.

[0102] The processor 402 can execute instructions or code stored in the memory 401, which can also store data. The instructions and data can also be sent and received over a network via a network interface device, which can employ any known transmission protocol.

[0103] The memory 401 can be integrated with the processor 402, such as by being disposed within an integrated circuit microprocessor, etc. Further, the memory 401 can include a separate device, such as an external disk drive, a storage array, or other storage device usable by any database system. The memory 401 and the processor 402 can be operatively coupled, or can communicate with each other, such as through I / O ports, network connections, etc., so that the processor 402 can read files stored in the memory.

[0104] Further, the electronic device 400 can also include a video display (such as a liquid crystal display) and a user interface (such as a keyboard, a mouse, a touch input device, etc.). All components of the electronic device 400 can be connected via a bus and / or network.

[0105] According to an exemplary embodiment of the present disclosure, there can also be provided a computer-readable storage medium that, when instructions stored therein are executed by a processor of an electronic device, enables the electronic device to perform the above-described verification method. Examples of the computer-readable storage medium herein include read-only memory (ROM), random-access programmable read-only memory (PROM), electrically erasable programmable read-only memory (EEPROM), random-access memory (RAM), dynamic random-access memory (DRAM), static random-access memory (SRAM), flash memory, non-volatile memory, CD-ROM, CD-R, CD+R, CD-RW, CD+RW, DVD-ROM, DVD-R, DVD+R, DVD-RW, DVD+RW, DVD-RAM, BD-ROM, BD-R, BD-R LTH, BD-RE, Blu-ray or optical disk memory, a hard disk drive (HDD), a solid state drive (SSD), a card-type memory such as a multimedia card, a secure digital (SD) card, or an extreme digital (XD) card, a magnetic tape, a floppy disk, a magneto-optical data storage device, an optical data storage device, a hard disk, a solid state disk, and any other device configured to store a computer program and any associated data, data files, and data structures in a non-transitory manner and provide the computer program and any associated data, data files, and data structures to a processor or computer so that the processor or computer can execute the computer program. The computer program in the above-described computer-readable storage medium can be executed in an environment deployed in a computer device such as a client, a host, a proxy device, a server, etc., and, in addition, in one example, the computer program and any associated data, data files, and data structures are distributed over a networked computer system so that the computer program and any associated data, data files, and data structures are stored, accessed, and executed in a distributed manner by one or more processors or computers.

[0106] According to an exemplary embodiment of the present disclosure, there can also be provided a computer program product including a computer program that, when executed by a processor, implements the verification method according to the present disclosure.

[0107] According to the verification method and device, the electronic device and the storage medium provided in the present disclosure, the multimedia resource usage information is acquired in an offline manner, and the deduplication performance of the multimedia resource recommendation model is verified based on the acquired multimedia resource usage information, without accessing the online distribution history service data, so that the situation that the online distribution history service data is misoperated can be avoided, the online distribution history service data can be ensured not to be damaged, and the stability of the multimedia resource recommendation is ensured. Further, for each sample account, when the repeated usage event information of the sample account is determined, the multimedia resource usage information of the sample account acquired this time and the multimedia resource usage information of the sample account acquired in the past time can be repeatedly determined, and the multimedia resource usage information of the sample account acquired this time can also be repeatedly determined, since the results of the two repeated determinations are combined, more factors are considered, and the final repeated usage event information of the sample account can be more perfect.

[0108] Other embodiments of the present disclosure will be apparent to those skilled in the art from consideration of the specification and practice of the present disclosure disclosed herein. The present disclosure is intended to cover any variations, uses or adaptations of the present disclosure following, in general, the principles of the present disclosure and including such

[0109] It should be understood that the present disclosure is not limited to the precise construction that has been described and shown in the accompanying drawings, and that various modifications and changes can be made by those skilled in the art without departing from the scope of the present disclosure. The scope of the present disclosure is limited only by the claims that follow.

Claims

1. A method of checking, characterized by, The verification method comprises: obtaining multimedia resource usage information of a plurality of sample accounts within a preset time period, wherein the multimedia resource usage information of the sample accounts is used to describe usage of multimedia resources recommended by a multimedia resource recommendation model by the sample accounts; for each sample account in the plurality of sample accounts, based on the multimedia resource usage information of the sample account obtained this time and the multimedia resource usage information of the sample account obtained in previous times, determining repeated usage event information for the sample account, the repeated usage event information being used to represent repeated usage of the same multimedia resource; based on the repeated usage event information for the plurality of sample accounts, verifying a deduplication performance of the multimedia resource recommendation model, wherein the deduplication performance indicates a performance of avoiding repeated recommendation of the same multimedia resource.

2. The checking method of claim 1, wherein, The step of verifying the deduplication performance of the multimedia resource recommendation model based on the repeated usage event information for the plurality of sample accounts comprises: based on the repeated usage event information for the plurality of sample accounts, obtaining an index value of each deduplication evaluation index; when the index value of the determined deduplication evaluation index exceeds a corresponding alarm threshold, determining that the deduplication performance of the multimedia resource recommendation model is abnormal.

3. The method of claim 2, wherein, The step of obtaining the index value of each deduplication evaluation index based on the repeated usage event information for the plurality of sample accounts comprises: for each time interval in each time interval included in the preset time period, determining a first ratio between a number of repeated usage events in the time interval and a number of multimedia resources used in the time interval; taking a statistical value of the first ratio corresponding to each time interval as the index value of the deduplication evaluation index.

4. The method of claim 2, wherein, The step of obtaining the index value of each deduplication evaluation index based on the repeated usage event information for the plurality of sample accounts comprises: for each time interval in each time interval included in the preset time period, determining a second ratio between a sum of repeated usage times of multimedia resources repeatedly used as indicated by repeated usage events in the time interval and a number of multimedia resources used in the time interval; taking a statistical value of the second ratio corresponding to each time interval as the index value of the deduplication evaluation index.

5. The method of claim 1, wherein, The step of determining, for each sample account in the plurality of sample accounts, repeated usage event information for the sample account based on the multimedia resource usage information of the sample account obtained this time and the multimedia resource usage information of the sample account obtained in previous times, the repeated usage event information being used to represent repeated usage of the same multimedia resource, comprises: for each sample account, performing repeated determination on the multimedia resource usage information of the sample account obtained this time to obtain first repeated usage statistical information; for each sample account, performing repeated determination between the multimedia resource usage information of the sample account obtained this time and the multimedia resource usage information of the sample account obtained in previous times to obtain second repeated usage statistical information; and for each sample account, performing repeated determination between the multimedia resource usage information of the sample account obtained this time and the multimedia resource usage information of the sample account obtained in previous times to obtain second repeated usage statistical information. determine, based on the first and second reuse statistics of each of the sampled accounts, reuse event information for each of the sampled accounts, the reuse event information being indicative of reuse of a same multimedia resource.

6. The method of claim 1, wherein, The verification method further includes: performing a modulo operation between each of a plurality of accounts reporting multimedia resource usage information within the preset time length and a preset sampling value to obtain at least one operation result; determining, as the plurality of sampled accounts, accounts corresponding to the operation result being a preset operation result among the plurality of accounts.

7. A checking device, characterized in that comprise: an acquisition module configured to acquire multimedia resource usage information of a plurality of sampled accounts within a preset time length, wherein the multimedia resource usage information of a sampled account is used to describe usage of a multimedia resource recommended by a multimedia resource recommendation model by the sampled account; a first determination module configured to, for each of the plurality of sampled accounts, determine, based on the multimedia resource usage information of the account acquired this time and the multimedia resource usage information of the account acquired in previous times, reuse event information for the account, the reuse event information being indicative of reuse of a same multimedia resource; a verification module configured to verify a deduplication performance of the multimedia resource recommendation model based on the reuse event information of the plurality of sampled accounts, wherein the deduplication performance is indicative of a performance of avoiding repeated recommendation of a same multimedia resource.

8. The verification device of claim 7, wherein, The verification module is configured to: obtain an index value of each deduplication evaluation index based on the reuse event information of the plurality of sampled accounts; determine that the deduplication performance of the multimedia resource recommendation model is abnormal when the index value of the deduplication evaluation index determined exceeds a corresponding alarm threshold.

9. The verification device of claim 8, wherein, The verification module is configured to: determine, for each time interval in each time interval included in the preset time length, a first ratio between a number of reuse events in the time interval and a number of multimedia resources used in the time interval; take a statistical value of the first ratio corresponding to each time interval as the index value of the deduplication evaluation index.

10. The verification device of claim 8, wherein, The verification module is configured to: determine, for each time interval in each time interval included in the preset time length, a second ratio between a sum of reuse times of a multimedia resource reused as indicated by reuse events in the time interval and a number of multimedia resources used in the time interval; take a statistical value of the second ratio corresponding to each time interval as the index value of the deduplication evaluation index.

11. The verification device of claim 7, wherein, The first determination module is configured to: perform reuse determination between the multimedia resource usage information of each sampled account acquired this time and each other to obtain first reuse statistics for each sampled account; perform reuse determination between the multimedia resource usage information of each sampled account acquired this time and the multimedia resource usage information of the account acquired in previous times to obtain second reuse statistics for each sampled account; Based on the first and second reuse statistics of each of the sampling accounts, determine reuse event information for each of the sampling accounts to represent reuse of the same multimedia resource.

12. The verification device of claim 7, wherein, The verification device further comprises: a modulo operation module configured to perform a modulo operation between each of a plurality of accounts that reported multimedia resource usage information within the preset time length and a preset sampling value, to obtain at least one operation result; a second determination module configured to determine, as the plurality of sampling accounts, accounts of the plurality of accounts whose corresponding operation results are preset operation results.

13. An electronic device, comprising: comprise: a processor; a memory for storing instructions executable by the processor; wherein the processor is configured to execute the instructions to implement the verification method of any one of claims 1 to 6.

14. A computer readable storage medium characterized by: When the instructions in the computer-readable storage medium are executed by the processor of the electronic device, the electronic device is enabled to perform the verification method of any one of claims 1 to 6.

Images