Advertisement information recommendation method and device, equipment and storage medium

Abstract

The invention discloses an advertisement information recommendation method and device, equipment and a storage medium, an advertisement information queue comprising multiple pieces of candidate advertisement information is acquired, the advertisement information queue is divided, at least two advertisement information sets are obtained, and the same advertisement information set comprises at least two pieces of similar candidate advertisement information. Then, the interest tendency of the target object to each piece of candidate advertisement information is predicted through the recommendation system, a first estimation result corresponding to each piece of candidate advertisement information is obtained, a first estimation deviation can be obtained by comparing the first estimation results, and a stability detection result of the recommendation system can be determined according to the first estimation deviation. And when the stability detection result of the recommendation system is passed, recommending the advertisement information by using the recommendation system. The recommendation method can improve the application effect of the recommendation system and improve the recommendation performance of the advertisement information. The technical scheme can be widely applied to the technical field of information.

Description

Technical Field

[0001] The present application relates to the field of information technology, and in particular to a method, device, equipment and storage medium for recommending advertising information. Background Art

[0002] At present, with the development of information technology, more and more related applications have been developed, bringing many conveniences to people's work, life and study. For example, the recommendation system developed based on information technology and artificial intelligence technology can recommend advertising information that people may be interested in, and has been widely used in product promotion, online shopping, video entertainment and other scenarios.

[0003] In the related art, in order to improve the application effect of the recommendation system as much as possible, it is generally necessary to test the recommendation system to detect the stability of the recommendation system. Commonly used detection methods are generally based on the AUC (area under the curve) indicator or the Bias (deviation degree) indicator to determine the performance of the recommendation system. In actual applications, it is found that these detection methods need to be implemented based on the real results of object feedback, and there is a lag problem, which makes it difficult to conduct timely and effective detection of newly developed recommendation systems; moreover, these detection methods can only measure the performance of the recommendation system in large quantities of sample dimensions, and cannot effectively detect smaller sample granularities, which may lead to low credibility of the stability detection results of the recommendation system, poor application effect of the actual deployment of the recommendation system, and affect the recommendation performance of advertising information.

[0004] In summary, the technical problems existing in the relevant technologies need to be improved. Summary of the invention

[0005] The embodiments of the present application provide a method, apparatus, device and storage medium for recommending advertising information, which can improve the application effect of the recommendation system and enhance the recommendation performance of advertising information.

[0006] An aspect of an embodiment of the present application provides a method for recommending advertising information, the method comprising:

[0007] Acquire an advertisement information queue; the advertisement information queue includes a plurality of candidate advertisement information;

[0008] Dividing the advertisement information queue to obtain at least two advertisement information sets; wherein the advertisement information set includes at least two of the candidate advertisement information;

[0009] Predicting the interest tendency of the target object to each of the candidate advertising information through the recommendation system, and obtaining a first estimation result corresponding to each of the candidate advertising information;

[0010] Determine a first estimation deviation between first estimation results corresponding to the candidate advertisement information in each of the advertisement information sets;

[0011] Obtaining a stability detection result of the recommendation system according to the first estimated deviation corresponding to each of the advertising information sets;

[0012] When the stability detection result of the recommendation system is passed, advertising information is recommended based on the recommendation system.

[0013] On the other hand, an embodiment of the present application provides a device for recommending advertising information, the device comprising:

[0014] An acquisition unit, used for acquiring an advertisement information queue; the advertisement information queue includes a plurality of candidate advertisement information;

[0015] A division unit, configured to divide the advertisement information queue to obtain at least two advertisement information sets; wherein the advertisement information set includes at least two of the candidate advertisement information;

[0016] An estimation unit, used for predicting the interest tendency of the target object to each of the candidate advertisement information through a recommendation system, and obtaining a first estimation result corresponding to each of the candidate advertisement information;

[0017] a comparing unit, configured to determine a first estimation deviation between first estimation results corresponding to the candidate advertisement information in each of the advertisement information sets;

[0018] A processing unit, configured to obtain a stability detection result of the recommendation system according to the first estimated deviation corresponding to each of the advertising information sets;

[0019] The recommendation unit is used to recommend advertising information based on the recommendation system when the stability detection result of the recommendation system is passed.

[0020] Optionally, the division unit is specifically used for:

[0021] Extracting fingerprint information corresponding to each of the candidate advertisement information, and dividing the advertisement information queue according to the fingerprint information;

[0022] Alternatively, extracting first characteristic information corresponding to each of the candidate advertisement information, and dividing the advertisement information queue according to the first characteristic information;

[0023] Alternatively, obtaining industry category information corresponding to each of the candidate advertisement information, and dividing the advertisement information queue according to the industry category information;

[0024] Alternatively, standardized product unit information corresponding to each of the candidate advertisement information is obtained, and the advertisement information queue is divided according to the standardized product unit information.

[0025] Optionally, the fingerprint information includes an information summary value; and the dividing unit is specifically used to:

[0026] Extracting information summary values ​​corresponding to each of the candidate advertisement information;

[0027] The information summary values ​​are compared, and the candidate advertisement information with the same corresponding information summary values ​​are grouped into the same advertisement information set.

[0028] Optionally, the division unit is specifically used for:

[0029] Extracting first feature information corresponding to each of the candidate advertisement information;

[0030] Taking the first characteristic information corresponding to any of the candidate advertisement information as the initial cluster center, establishing a cluster;

[0031] Calculating the third similarity between each feature information to be clustered and the initial cluster center; the feature information to be clustered is the first feature information that has not been added to the clustering cluster;

[0032] When the third similarity between the feature information to be clustered and the initial cluster center is greater than or equal to the second threshold, the feature information to be clustered is added to the cluster corresponding to the initial cluster center, and the position of the initial cluster center is updated; or, when the third similarity between the feature information to be clustered and any of the initial cluster centers is less than the second threshold, the feature information to be clustered is used as the initial cluster center, and a new cluster is additionally established;

[0033] The candidate advertisement information corresponding to the first characteristic information belonging to the same cluster is divided into the same advertisement information set.

[0034] Optionally, the first estimation result includes an estimation result of a click rate or a conversion rate of the target object for the candidate advertisement information; and the estimation unit is specifically configured to:

[0035] Acquire first associated information of the target object;

[0036] The recommendation system predicts, based on the candidate advertisement information and the first associated information, a click rate estimate result or a conversion rate estimate result of the target object for the candidate advertisement information.

[0037] Optionally, the comparison unit is specifically used for:

[0038] Determine an estimated mean and an estimated standard deviation of the first estimated results corresponding to each of the candidate advertisement information in the advertisement information set;

[0039] A first estimated deviation corresponding to the advertisement information set is obtained according to a quotient of the estimated standard deviation and the estimated mean.

[0040] Optionally, the processing unit is specifically configured to:

[0041] Counting the number of the candidate advertisement information in each of the advertisement information sets;

[0042] Determining, according to the number, a weighted weight corresponding to each of the advertising information sets;

[0043] According to the weighted weights, weighted summation is performed on the first estimated deviations corresponding to the respective advertising information sets to obtain a third estimated deviation;

[0044] determining a stability detection result of the recommendation system according to the third estimated deviation;

[0045] The weighted weight corresponding to the advertisement information set is positively correlated with the number of candidate advertisement information in the advertisement information set.

[0046] Optionally, the processing unit is specifically configured to:

[0047] Obtaining third estimated deviations corresponding to the plurality of target objects;

[0048] Performing weighted summation on the third estimated deviations corresponding to the target objects to obtain a fourth estimated deviation;

[0049] A stability detection result of the recommendation system is determined according to the fourth estimated deviation.

[0050] Optionally, the processing unit is specifically configured to:

[0051] comparing the fourth estimated deviation with a preset deviation threshold;

[0052] If the fourth estimated deviation is greater than or equal to the deviation threshold, it is determined that the stability test result of the recommendation system is failed; or, if the fourth estimated deviation is less than the deviation threshold, it is determined that the stability test result of the recommendation system is passed.

[0053] Optionally, the recommendation system is configured with a recommendation model; and the estimation unit is specifically used for:

[0054] Acquire first associated information of the target object;

[0055] extracting, by the recommendation model, first feature information corresponding to the candidate advertisement information and second feature information corresponding to the first associated information;

[0056] A first estimation result is obtained by predicting the recommendation model according to the first feature information and the second feature information.

[0057] Optionally, the recommendation device further includes a training unit, and the training unit is specifically used for:

[0058] Acquire sample advertising information, second associated information of the sample object, and label information; wherein the label information is used to represent the actual result of the sample object's interest tendency in the sample advertising information;

[0059] extracting the third characteristic information of the sample advertisement information and the fourth characteristic information of the second associated information through a recommendation model;

[0060] According to the third feature information and the fourth feature information, predicting the interest tendency of the sample object to the sample advertising information by the recommendation model to obtain a fourth estimation result;

[0061] Determining a loss value of training according to the fourth estimation result and the label information;

[0062] The parameters in the recommendation model are updated according to the loss value to obtain a trained recommendation model.

[0063] Optionally, the recommendation device further includes an alarm unit, and the alarm unit is specifically used to:

[0064] If the stability test result of the recommendation system fails, an alarm message is generated;

[0065] The warning information is displayed, or the warning information is sent to an associated object of the recommendation system.

[0066] On the other hand, an embodiment of the present application provides an electronic device, including a processor and a memory;

[0067] The memory is used to store computer programs;

[0068] The processor executes the computer program to implement the aforementioned method for recommending advertising information.

[0069] On the other hand, an embodiment of the present application provides a computer-readable storage medium, wherein the storage medium stores a computer program, and the computer program is executed by a processor to implement the aforementioned method for recommending advertising information.

[0070] On the other hand, an embodiment of the present application further discloses a computer program product, including a computer program, which implements the aforementioned method for recommending advertising information when executed by a processor.

[0071] The embodiments of the present application include at least the following beneficial effects: The present application provides a method, apparatus, device and storage medium for recommending advertising information, wherein the recommendation method obtains an advertising information queue including multiple candidate advertising information, divides the advertising information queue, and obtains at least two advertising information sets, wherein the same advertising information set includes at least two similar candidate advertising information. Then, the recommendation system predicts the interest tendency of the target object for each candidate advertising information, obtains the first estimated result corresponding to each candidate advertising information, and compares the first estimated result to obtain a first estimated deviation. Since similar candidate advertising information is presented to the same target object, the first estimated result obtained by the recommendation system should be close, so the size of the first estimated deviation can reflect the stability of the recommendation system to a certain extent, and the stability detection result of the recommendation system can be determined based on the first estimated deviation. In this way, the stability test of the recommendation system can be realized without collecting the real results of the target object feedback, which can reduce the time cost required for the test, improve the lag problem existing in the current detection method, and is suitable for the newly developed recommendation system. It can also improve the granularity of the sample tested, and facilitate obtaining more accurate and reliable stability test results. When the stability test result of the recommendation system is passed, the recommendation method uses the recommendation system to recommend advertising information, which can improve the application effect of the recommendation system and improve the recommendation performance of advertising information. BRIEF DESCRIPTION OF THE DRAWINGS

[0072] The accompanying drawings are used to provide a further understanding of the technical solution of the present application, and constitute a part of the specification, and together with the embodiments of the present application, are used to explain the technical solution of the present application. Obviously, the drawings described below are only some embodiments of the present application and do not constitute a limitation on the technical solution of the present application. For those skilled in the art, other drawings can also be obtained based on these drawings without paying creative work.

[0073] Figure 1 A schematic diagram of a ROC curve provided in an embodiment of the present application;

[0074] Figure 2 A schematic diagram of an implementation environment of a method for recommending advertising information provided in an embodiment of the present application;

[0075] Figure 3 A schematic diagram of an implementation environment of another method for recommending advertising information provided in an embodiment of the present application;

[0076] Figure 4A schematic diagram of a flow chart of a stability detection method of a recommendation system provided in an embodiment of the present application;

[0077] Figure 5 A schematic diagram of a flow chart of another method for detecting stability of a recommendation system provided in an embodiment of the present application;

[0078] Figure 6 A flowchart of a method for recommending advertising information provided in an embodiment of the present application;

[0079] Figure 7 A schematic diagram of displaying advertising information on an interface of an application provided in an embodiment of the present application;

[0080] Figure 8 A schematic diagram of determining an estimated deviation of a recommendation system for advertising information provided in an embodiment of the present application;

[0081] Fig. 9 A schematic diagram of dividing an advertisement information set provided in an embodiment of the present application;

[0082] Fig.10 A schematic diagram of determining a stability test result according to an estimated deviation provided in an embodiment of the present application;

[0083] Fig.11 A schematic diagram of the structure of a device for recommending advertising information provided in an embodiment of the present application;

[0084] Fig.12 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application;

[0085] Fig.13 A schematic diagram of the structure of another electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0086] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application is further described in detail below in conjunction with the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the embodiments of the present application. They are only examples of devices and methods consistent with some aspects of the embodiments of the present application as detailed in the attached claims.

[0087] It is understood that the terms "first", "second", etc. used in this application can be used to describe various concepts in this article, but unless otherwise specified, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another concept. For example, without departing from the scope of the embodiment of the present application, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the words "if" and "if" as used herein can be interpreted as "at the time of" or "when" or "in response to determination".

[0088] The terms "at least one", "multiple", "each", "any", etc. used in this application, at least one includes one, two or more, multiple includes two or more, each refers to each of the corresponding multiple, and any refers to any one of the multiple.

[0089] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0090] Before describing the embodiments of the present application in detail, some nouns and terms involved in the embodiments of the present application are first described. The nouns and terms involved in the embodiments of the present application are subject to the following explanations.

[0091] 1) Recommendation system: A system that uses information technology and artificial intelligence technology to recommend advertising information that may be of interest to the target based on the target's relevant information.

[0092] 2) AUC (Area Under Curve), the area enclosed by the ROC curve and the coordinate axis below it, is often used as an evaluation indicator for classification. Since the ROC curve is generally above the 45-degree diagonal in the coordinate system, the value range of the AUC indicator is usually between 0.5 and 1. The larger the value of the AUC indicator, the better the classification effect and the closer it is to the actual result.

[0093] 3) ROC (Receiver Operating Characteristic) curve, also known as sensitivity curve. When generating the ROC curve, the true positive rate of classification (the ratio of the number of samples that are actually positive samples and predicted to be positive samples to the total number of samples that are actually positive samples) is used as the vertical axis, and the false positive rate (the ratio of the number of samples that are actually negative samples and predicted to be positive samples to the total number of samples that are actually negative samples) is used as the horizontal axis. Coordinate points are obtained under different classification thresholds, and each coordinate point is connected to obtain the ROC curve.

[0094] 4) Bias (deviation degree), an evaluation indicator used to measure the accuracy of prediction, indicating the difference between the estimated result and the actual result.

[0095] 5) Artificial Intelligence (AI) refers to theories, methods, technologies and application systems that use digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new type of intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines so that machines have the functions of perception, reasoning and decision-making. Artificial intelligence technology is an interdisciplinary subject that covers a wide range of fields, including both hardware-level and software-level technologies. Basic artificial intelligence technologies generally include sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics and other technologies. Artificial intelligence software technology mainly includes computer vision technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0096] 6) Machine Learning (ML), a multi-disciplinary interdisciplinary subject, involves probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory and other disciplines. It specializes in studying how computers simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications are spread across all areas of artificial intelligence. Machine learning (deep learning) usually includes artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and self-learning.

[0097] 7) Blockchain, a new application model of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanism, encryption algorithm, etc. Blockchain is essentially a decentralized database, a string of data blocks generated by cryptographic methods. Each data block contains a batch of network transaction information, which is used to verify the validity of its information (anti-counterfeiting) and generate the next block. Blockchain can include the underlying blockchain platform, platform product service layer and application service layer. Blockchain can include public chain, alliance chain and private chain. Among them, public chain refers to a blockchain that anyone can enter the blockchain network at any time to read data, send data or compete for accounting; alliance chain refers to a blockchain jointly managed by several organizations or institutions; private chain refers to a blockchain with certain centralized control. The write right of the private chain account book is controlled by a certain organization or institution, and the access and use of data are strictly managed by permissions.

[0098] At present, with the development of information technology, more and more related applications have been developed, bringing many conveniences to people's work, life and study. For example, the recommendation system developed based on information technology and artificial intelligence technology can recommend advertising information that people may be interested in, and has been widely used in product promotion, online shopping, video entertainment and other scenarios.

[0099] The popularity of recommendation systems provides a channel for personalized and accurate advertising information recommendations. Both upstream and downstream of advertising information can benefit from the application of recommendation systems. On the one hand, for the objects of recommended advertising information, the recommendation system can make personalized recommendations based on personal interests and preferences, provide advertising information that meets personal needs as much as possible, and improve the satisfaction and browsing experience of advertising information recommendations; in the recommendation process, the recommendation system can help filter out a large amount of redundant information and uninteresting advertising information, making it easier for the object to find the desired content from the massive amount of information, saving the time and energy consumed by information search; and the recommendation system can also recommend advertising information related to the object's interests but may not have been exposed to, helping to discover new content of interest, helping to broaden the information horizon and enrich interests and hobbies. On the other hand, for objects with advertising information recommendation needs, the recommendation system can help achieve accurate delivery of advertising information, improve indicators such as the click-through rate and conversion rate of advertising information, and facilitate the promotion of related products or services.

[0100] In the related art, when developing or deploying a recommendation system, it is generally necessary to test it to detect the stability of the recommendation system, so as to facilitate targeted iterative optimization and improve the application effect of the recommendation system as much as possible. Here, the stability of the recommendation system mainly includes two meanings: reproducibility and accuracy. That is, when the recommendation system can continuously output accurate prediction results, it is considered to have good stability. Commonly used detection methods generally determine the performance of the recommendation system based on the AUC (area under the curve) indicator or the Bias (deviation degree) indicator, where the "curve" in the AUC indicator refers to the ROC (receiver operating characteristic) curve. For binary classification applications, the ROC curve can be made according to the set classification threshold, true positive rate, and false positive rate. Here, the true positive rate refers to the ratio of the number of samples that are actually positive samples and predicted to be positive samples to the total number of samples that are actually positive samples; the false positive rate refers to the ratio of the number of samples that are actually negative samples and predicted to be positive samples to the total number of samples that are actually negative samples.

[0101] In the scenarios involved in this application, the concepts of positive samples and negative samples can be set according to specific needs. For example, for example, a recommendation system can be used to predict whether multiple objects are interested in a certain advertisement information. If the object is interested in the advertisement information, the object can be identified as a positive sample; conversely, if the object is not interested in the advertisement information, the object can be identified as a negative sample. Of course, the above is only used to illustrate the concepts of positive samples and negative samples involved in the production of ROC curves. In actual applications, it can be flexibly set according to specific needs, and this application does not limit this.

[0102] Reference Figure 1 , Figure 1 A schematic diagram of a ROC curve provided in an embodiment of the present application is shown. Figure 1For the coordinate system shown, the horizontal axis is the true positive rate and the vertical axis is the false positive rate. According to the different classification thresholds set, the corresponding points can be determined in the coordinate space, and the ROC curve 110 can be obtained by connecting these points. On the ROC curve 110, the larger the vertical coordinate corresponding to each point, the more actual positive samples there are in the predicted positive samples, and the better the classification effect; the larger the horizontal coordinate corresponding to each point, the more actual negative samples there are in the predicted positive samples, and the worse the classification effect. Therefore, the ideal goal is to make the ROC curve 110 as close to the coordinate point (0, 1) as possible, that is, the more it deviates from the 45-degree diagonal and the closer it is to the upper left corner, the better. Here, the area enclosed by the ROC curve and the horizontal axis of the coordinate system, that is, the area under the curve (AUC), is generally used to characterize the corresponding index. The essence of the AUC index is a probability value, and its size corresponds to the meaning of: when a positive sample and a negative sample are randomly selected, the probability that the current classification algorithm will rank this positive sample in front of the negative sample according to the calculated estimated result. It is understandable that the larger the value of the AUC indicator, the more likely the current classification algorithm is to rank positive samples ahead of negative samples, thereby obtaining more accurate and reliable classification results.

[0103] The Bias (degree of deviation) indicator can be used to measure the distribution deviation between the estimated results and the actual results of the recommendation system, and is generally obtained by comparing the differences between multiple actual results and the estimated results. For example, in the scenario involved in this application, for example, a recommendation system is used to predict the click-through rate data of an object for advertising information. When detecting the Bias indicator of the recommendation system, the click-through rate estimation results of the recommendation system for an object for multiple advertising information and the number of advertising information for which the object actually has click behavior can be counted, denoted as Y; then, the click-through rate estimation results corresponding to each advertising information are summed to obtain the total estimation result, denoted as Y', and the result of calculating Y' / Y-1 can be used to obtain the Bias indicator. Of course, it is also possible to count the clicks of multiple objects on one advertising information, or to count the clicks of multiple objects on multiple advertising information to determine the Bias indicator, and this application does not limit this. It can be understood that the smaller the absolute value of the Bias indicator, the smaller the overall deviation between the estimated results and the actual results of the recommendation system, and the better the application effect; conversely, the larger the absolute value of the Bias indicator, the greater the overall deviation between the estimated results and the actual results of the recommendation system, and the worse the application effect.

[0104] In actual applications, it is found that the detection methods using the above indicators need to be implemented based on the real results of the object feedback, that is, it is necessary to compare the estimated results of the recommendation system with the real results of the object feedback in order to realize the performance detection of the recommendation system. However, the collection of real results is often time-consuming, costly, and has the problem of lag, making it difficult to conduct timely and effective detection of newly developed recommendation systems; moreover, these detection methods can only measure the performance of the recommendation system in the dimension of large-scale samples, and cannot effectively detect smaller sample granularity. Here, a single sample refers to the estimated result of the recommendation system predicting an object's interest in an advertising message.

[0105] Specifically, for the AUC indicator, it is difficult to obtain a reliable and valuable AUC indicator value based on a small number of samples, and it may not even be possible to draw the ROC curve; for the Bias indicator, the Bias indicator value obtained on a small number of samples has no actual reference value. For example, for example, a recommendation system is used to predict the estimated click-through rate of an object for advertising information. Assuming that the estimated click-through rate result is 2%, if the object does not click on the corresponding advertising information, the actual true result is 0, and the Bias indicator is infinite; if the object clicks on the corresponding advertising information, the actual true result is 1, and the Bias indicator is -98%. For each object, the click-through rate prediction result given by the recommendation system is generally a value between 0-1, while the actual true result can only be 0 or 1. Therefore, the Bias indicator value obtained on a small number of samples is easily affected by individual samples, and there is a serious floating situation. Only on a large number of samples can the application effect of the recommendation system be reflected.

[0106] In summary, the above-mentioned detection methods are difficult to reflect the performance of the recommendation system at a smaller sample granularity. There may be a situation where the recommendation system shows good stability in the dimension of a large number of samples, but the actual estimated results of a single sample perform poorly, resulting in low credibility of the stability detection results of the recommendation system, poor application effect of the actual deployment of the recommendation system, and affecting the recommendation performance of advertising information.

[0107] In view of this, the present application provides a stability detection method for a recommendation system, a recommendation method for advertising information, an apparatus, a device and a storage medium, wherein the stability detection method for the recommendation system obtains similar first advertising information and second advertising information, and predicts the interest tendency of the same target object to the first advertising information and the second advertising information through the recommendation system, and obtains the corresponding second estimated result and third estimated result. By comparing the second estimated result and the third estimated result, a second estimated deviation is obtained. Since the similar first advertising information and the second advertising information are presented to the same target object, the second estimated result and the third estimated result obtained by the recommendation system should be close to each other, so the size of the second estimated deviation can reflect the stability of the recommendation system to a certain extent, and the stability detection result of the recommendation system can be determined according to the second estimated deviation. The detection method can realize the stability detection of the recommendation system without collecting the real results of the target object feedback, can reduce the time cost required for detection, can improve the hysteresis problem existing in the current detection method, and is suitable for newly developed recommendation systems; moreover, the detection method can effectively detect the stability of the recommendation system in the dimension of a small number of samples, can improve the fine granularity of the detected samples, and facilitate the acquisition of more accurate and reliable stability detection results. The recommendation method can use the recommendation system to recommend advertising information when the stability detection result of the recommendation system is passed, which can improve the application effect of the recommendation system and enhance the recommendation performance of the advertising information.

[0108] The stability detection method of the recommendation system and the recommendation method of advertising information provided in the embodiments of the present application mainly involve technologies in the field of information technology and related to recommendation applications. Specifically, the stability detection method of the recommendation system and the recommendation method of advertising information provided in the embodiments of the present application can be executed in application scenarios of various recommendation systems, without limiting the specific form and content of the recommended advertising information. The following are some examples of application scenarios of the recommendation system provided in the embodiments of the present application:

[0109] For example, in an e-commerce application scenario, the recommendation system can recommend relevant goods or services to a subject based on the subject's purchase history, browsing behavior, and personal preferences, thereby improving the subject's shopping satisfaction and conversion rate.

[0110] For example, in the application scenario of video streaming, the recommendation system can recommend movies, TV series, music and other content that meet the individual needs of the subject based on the subject's viewing history, film and television ratings and viewing preferences, thereby improving the subject's viewing experience and subscription rate.

[0111] For example, in the application scenario of news information, the recommendation system can recommend relevant news, articles, blogs and other content to the subject based on the subject's interests, browsing history and social relationships, thereby improving the subject's browsing experience.

[0112] It is understandable that in the above application scenarios, before or after the actual deployment of the recommendation system, the stability detection method of the recommendation system provided in the embodiments of the present application can be used to detect the stability of the recommendation system. When the stability detection result of the recommendation system is passed, advertising information can be recommended based on the recommendation system; when the stability detection result of the recommendation system is not passed, the recommendation system can be updated and optimized to improve its application effect. Therefore, the stability detection method of the recommendation system and the recommendation method of advertising information provided in the embodiments of the present application are also applicable to the above application scenarios.

[0113] Of course, it is understandable that the above application scenarios are only exemplary and do not limit the actual application of the stability detection method of the recommendation system and the recommendation method of advertising information provided in the embodiments of the present application. Those skilled in the art can understand that in different application scenarios, the stability detection method of the recommendation system and the recommendation method of advertising information provided in the embodiments of the present application can be used to perform specified tasks.

[0114] It should be noted that in each specific implementation of the present application, when it comes to the need to perform relevant processing based on the object's information, the object's behavioral data, the object's historical data, the object's location information, and other data related to the object's identity or characteristics, the object's permission or consent will be obtained first, and the collection, use, and processing of these data will comply with the relevant laws, regulations, and standards of the relevant countries and regions. In addition, when the embodiment of the present application needs to obtain sensitive information of the object, the object's separate permission or separate consent will be obtained through a pop-up window or by jumping to a confirmation page. After clearly obtaining the object's separate permission or separate consent, the necessary object-related data for the normal operation of the embodiment of the present application will be obtained.

[0115] The specific implementation of the embodiment of the present application is described in detail below in conjunction with the accompanying drawings.

[0116] Please refer to Figure 2 , Figure 2 A schematic diagram of the implementation environment of a method for recommending advertising information provided in an embodiment of the present application is shown. Of course, the stability detection method of the recommendation system provided in an embodiment of the present application can also be applied in this implementation environment. In this implementation environment, the main hardware and software entities involved include a recommendation system 210 and a client 220, and a communication connection can be achieved between the recommendation system 210 and the client 220. Among them, the recommendation system 210 can send corresponding advertising information to the object through the client 220, and the client 220 can receive the advertising information sent by the recommendation system 210 and display it to achieve the recommendation of advertising information.

[0117] The stability detection method of the recommendation system provided in the embodiment of the present application can be executed on the recommendation system 210 side. For example, the recommendation system 210 can be integrated with a computer program 230 for executing the stability detection method of the recommendation system, and the recommendation system 210 can realize its own stability detection by running the computer program 230. Of course, relatively speaking, the stability detection method of the recommendation system provided in the embodiment of the present application can also be executed on the client 220 side, and its principle is similar to that executed on the recommendation system 210 side, which will not be described in detail here.

[0118] It is understandable that in some embodiments, the recommendation system 210 and the client 220 can be software programs themselves, which can run on physical machines or virtual machines. In other embodiments, the recommendation system 210 and the client 220 can also be hardware devices. For example, the recommendation system 210 can be a server device for implementing advertising information push, and the client 220 can be a terminal device. For specific device types, the server device can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms, or a node server in a blockchain network; terminal devices can include smart phones, tablet computers, laptops, desktop computers, smart speakers, smart watches, and vehicle-mounted terminals, but are not limited to these.

[0119] The recommendation system 210 and the client 220 may establish a communication connection through a wireless network or a wired network. The wireless network or wired network uses standard communication technology and / or protocols, and the network may be set to the Internet or any other network, such as but not limited to a local area network (LAN), a metropolitan area network (MAN), a wide area network (WAN), a mobile, wired or wireless network, a private network or any combination of a virtual private network.

[0120] The above description and introduction of the recommendation system 210 and the client 220 are also applicable to the subsequent embodiments. In the subsequent embodiments, the same or repeated parts will not be described in detail.

[0121] Please refer to Figure 3 , Figure 3A schematic diagram of the implementation environment of another method for recommending advertising information provided in an embodiment of the present application is shown. Similarly, the stability detection method of the recommendation system provided in an embodiment of the present application can also be applied in this implementation environment. In this implementation environment, the main hardware and software entities involved include a recommendation system 210, a client 220, and a performance detection system 240, and communication connections can be achieved between the recommendation system 210 and the client 220, and between the recommendation system 210 and the performance detection system 240. The performance detection system 240 can be an independent software program or hardware device, which can be used to implement performance detection of the recommendation system.

[0122] The stability detection method of the recommendation system provided in the embodiment of the present application can be executed on the performance detection system 240 side. For example, the performance detection system 240 can be integrated with a computer program 230 for executing the stability detection method of the recommendation system, and the stability detection of the recommendation system 210 can be implemented by running the computer program 230, and the detection result is fed back to the recommendation system 210. When the stability detection result is passed, the recommendation system 210 can run normally and recommend advertising information; when the stability detection result is not passed, the recommendation operation of the recommendation system 210 can be temporarily stopped.

[0123] It should be noted that in the above embodiments, the various software and hardware entities may use the same communication connection method or different communication connection methods, and this application does not impose any specific restrictions on this.

[0124] Of course, it is understandable that Figure 2 , Figure 3 The implementation environment in the present application is just some optional application scenarios of the stability detection method of the recommendation system and the recommendation method of advertising information provided in the embodiments of the present application. The actual application is not fixed. Figure 2 , Figure 3 The hardware and software environment shown in .

[0125] The method provided in the embodiments of the present application can be applied to various technical fields, such as shopping, games, social networking, cloud technology, artificial intelligence, smart transportation, assisted driving, etc., and the present application does not impose any specific restrictions on this.

[0126] Before explaining the stability detection method of the recommendation system and the recommendation method of advertising information provided in the embodiments of the present application, first, the recommendation system mentioned in the embodiments of the present application is introduced and explained.

[0127] In the embodiments of the present application, the recommendation system refers to a system that uses information technology and artificial intelligence technology to recommend advertising information that may be of interest to an object based on the object's relevant information. The present application does not limit the object information and the type of artificial intelligence technology used by the recommendation system. For example, in some embodiments, the recommendation system can use machine learning, data mining, portrait analysis and other technologies based on the object's attribute information, historical behavior information, personal preference information, etc. to provide personalized and accurate advertising information recommendations for the object.

[0128] In the embodiments of the present application, the recommendation system can recommend advertising information based on various recommendation algorithms. Exemplarily, in some embodiments, the recommendation system can use a recommendation algorithm based on collaborative filtering, for example, based on the historical behavior information of the target object, find other objects with similar interests to the target object, and then recommend advertising information to the target object based on the interest preferences of other objects; or first determine the target advertising information preferred by the target object, find other advertising information similar to the target advertising information, and recommend these other advertising information to the target user. In some embodiments, the recommendation system can use a recommendation algorithm based on content filtering, and recommend advertising information by matching the interest preferences of the object with the content characteristics of the advertising information according to the content characteristics of the advertising information. In some embodiments, the recommendation system can recommend advertising information based on an artificial intelligence model. For example, the recommendation system can use machine learning or deep learning technology to build a corresponding recommendation model, and recommend advertising information based on the built recommendation model. In some embodiments, the recommendation system can also combine multiple recommendation algorithms or strategies, comprehensively consider different factors, and provide more accurate and personalized advertising information recommendation results.

[0129] Of course, the above is only used to exemplify some types of recommendation algorithms that can be used in the recommendation system involved in the embodiments of the present application, and does not mean to limit the implementation of the recommendation system in the embodiments of the present application. Those skilled in the art can understand that in actual applications, the recommendation system may vary according to different application scenarios and needs. Specifically, it can comprehensively consider factors such as the characteristics of the object, the characteristics of the advertising information, and the recommendation algorithm, and make flexible selections and configurations. The present application does not impose any restrictions on this.

[0130] Below, in combination with the above description of the implementation environment and the recommendation system, a stability detection method for a recommendation system and a method for recommending advertising information provided in an embodiment of the present application are introduced and explained in detail.

[0131] like Figure 4As shown, in an embodiment of the present application, a stability detection method for a recommendation system is provided, which can be applied to the above-mentioned recommendation system 210, client 220 or performance detection system 240, and can also be executed on any other software program or hardware device with similar data processing capabilities.

[0132] Reference Figure 4 The stability detection method of the recommendation system provided in the embodiment of the present application specifically includes but is not limited to steps 410 to 440:

[0133] Step 410: Obtain first advertisement information and second advertisement information;

[0134] In this step, when the stability test of the recommendation system is performed, two similar advertising information can be obtained, and these two advertising information are recorded as the first advertising information and the second advertising information. Here, the advertising information refers to the relevant information of the resources recommended by the recommendation system. It can be understood that in the embodiment of the present application, the types of resources recommended by the recommendation system can be various, for example, they can include tangible physical items, intangible virtual products, commercial services or a combination thereof. For example, clothes on a shopping platform, computers, props or paid services in online games, stocks and funds on a securities platform, etc. can all fall within the scope of resources in the embodiment of the present application. Of course, the actual resource types are not limited to this, and the present application does not limit this.

[0135] In the embodiments of the present application, the specific data format and content of the advertising information can be set according to the corresponding resource type. For example, in some embodiments, the resources recommended by the recommendation system may be physical goods. In this case, the advertising information may be information related to the goods, such as the introduction information and image information of the goods. In some embodiments, the resources recommended by the recommendation system may be digital products, such as videos or music. In this case, the advertising information may be information related to the digital products, or the digital products themselves (for example, the audio data corresponding to the video data or the music may be used as advertising information), or a combination of the two. In the embodiments of the present application, there is no limitation on this.

[0136] In this step, when obtaining the first advertisement information and the second advertisement information, there is no restriction on the actual source of the first advertisement information and the second advertisement information. For example, in some embodiments, the first advertisement information and the second advertisement information may be transmitted by other systems or electronic devices; in other embodiments, the first advertisement information and the second advertisement information may also be manually input and uploaded. Figure 2Taking the implementation scenario shown as an example, the developer and tester of the recommendation system can collect the first advertisement information and the second advertisement information as required, and upload them to the recommendation system. The recommendation system can obtain the corresponding stability detection result by executing the stability detection method provided in the embodiment of the present application, and the stability detection result can characterize the performance of the recommendation system. The developer and tester of the recommendation system can optimize and update the relevant program code of the recommendation system according to the stability detection result, so as to improve the performance of the recommendation system in a targeted manner and facilitate the improvement of the application effect of the recommendation system after deployment.

[0137] In an embodiment of the present application, the first advertising information and the second advertising information are similar, and the relationship between the two can be defined by similarity. Here, similarity is used to characterize the degree of similarity between two given items. In an embodiment of the present application, the similarity between the first advertising information and the second advertising information is recorded as the first similarity. The higher the value of the first similarity, the more similar the first advertising information and the second advertising information are. Therefore, it can be understood that in order to limit the degree of similarity between the first advertising information and the second advertising information, in an embodiment of the present application, a similarity threshold can be set accordingly, recorded as the first threshold. Constraining the first similarity between the first advertising information and the second advertising information to be greater than or equal to the first threshold can make the two be within a similar range.

[0138] Here, it should be noted that in the embodiments of the present application, there is no restriction on the numerical representation of similarity. For example, in some embodiments, a percentage can be used to characterize the magnitude of the similarity, and 100% can be set as the maximum value of the similarity. When the first similarity between the first advertising information and the second advertising information reaches 100%, it means that the two are completely consistent. The size of the first threshold can be set to 80%. When the first similarity between the first advertising information and the second advertising information is less than 80%, it can be determined that the two are not similar. Of course, the embodiment of the present application also does not limit the size of the first threshold, and it can be flexibly set according to actual needs.

[0139] Generally speaking, commonly used similarity algorithms include cosine similarity algorithm, Jaccard similarity algorithm, Hamming distance algorithm, etc. In the embodiments of the present application, there is no restriction on the specific algorithm type used. Exemplarily, in some embodiments, the first similarity between the first advertising information and the second advertising information can be determined by the cosine similarity algorithm. First, the feature information of the first advertising information and the feature information of the second advertising information can be extracted by a feature extraction algorithm. Here, the feature information of both of them can be vector data, the first advertising information corresponds to the first feature vector, and the second advertising information corresponds to the second feature vector. Then, the vector length of the first feature vector can be determined, recorded as the first length, and the length of the second feature vector can be recorded as the second length. Next, the product of the first length and the second length is calculated as the first value, and the inner product of the first feature vector and the second feature vector is calculated as the second value, and then the quotient of the first value and the second value is calculated as the similarity between the first feature vector and the second feature vector, that is, the first similarity between the first advertising information and the second advertising information.

[0140] It should be noted that, in the embodiment of the present application, the first similarity between the first advertisement information and the second advertisement information is limited to be greater than or equal to the first threshold, and the goal is to make the first advertisement information and the second advertisement information as similar as possible. When actually executing the stability detection method provided in the embodiment of the present application, a variety of methods can be used to constrain the first advertisement information and the second advertisement information to be similar, and there is no need to actually calculate the specific value of the first similarity.

[0141] Step 420: predicting the interest tendency of the target object to the first advertisement information through the recommendation system to obtain a second estimation result;

[0142] In this step, for the first advertisement information obtained, the recommendation system to be detected can be used to predict the interest tendency of the target object in the first advertisement information, and an estimated result can be obtained, which is recorded as the second estimated result. Here, the target object refers to the object to which the recommendation system recommends the advertisement information. Generally speaking, the target object can be any individual. Of course, in some embodiments, the target object can also be a collective (such as a company or organization) or an electronic device with intelligent decision-making capabilities, and this application does not limit this.

[0143] In this step, the target object's interest tendency for the first advertising information is predicted by the recommendation system. The recommendation system can be implemented with reference to the introduction of the above-mentioned embodiment, which will not be described in detail here. The recommendation system predicts the target object's interest tendency for the first advertising information, and there may be multiple implementation methods. Exemplarily, in some embodiments, the recommendation system can predict the target object's interest tendency for the first advertising information by means of numerical prediction. For example, the recommendation system can analyze the target object's past click behavior or conversion behavior on the advertising information, and predict the target object's click rate prediction result or conversion rate prediction result for the first advertising information as a characterization result of the interest tendency. In some embodiments, the recommendation system can predict the target object's interest tendency for the first advertising information by means of classification prediction. For example, the recommendation system can predict whether the first advertising information should be recommended to the target object, or predict whether the target object is interested in the first advertising information. In this case, the prediction task can be regarded as a classification task, such as the recommendation system can obtain the classification result of whether the target object is interested or not interested in the first advertising information through the classification model, or can obtain the classification result of recommending the first advertising information to the target object or not recommending the first advertising information to the target object through the classification model as a characterization result of the interest tendency. In the embodiment of the present application, there is no limitation on the manner in which the recommendation system predicts the target object's interest tendency toward the first advertisement information.

[0144] It is understandable that, since there are various ways for the recommendation system to predict the target object's interest in the first advertising information, there are correspondingly various ways to implement the data form and meaning of the obtained second estimated result. Exemplarily, in some embodiments, the second estimated result may be data with practical significance, for example, the second estimated result may be data reflecting the possibility of the target object performing a corresponding operation on the first advertising information, for example, the second estimated result may be click-through rate data, conversion rate data, collection rate data, or purchase rate data, etc.

[0145] In some embodiments, the second estimated result may be data used to characterize the classification result. For example, when the recommendation system obtains the classification result that the target object is interested in or not interested in the first advertising information through the classification model, the second estimated result corresponding to the prediction that the target object is interested in the first advertising information may be set to 1, and the second estimated result corresponding to the prediction that the target object is not interested in the first advertising information may be set to 0. Of course, in the embodiments of the present application, there is no restriction on the specific data form and corresponding meaning of the second estimated result in this case, and it can be flexibly adjusted as needed. For example, in some cases, the data form of the second estimated result may be a vector, and the second estimated result corresponding to the prediction that the target object is interested in the first advertising information may be set to (1, 0), and the second estimated result corresponding to the prediction that the target object is not interested in the first advertising information may be set to (0, 1). The present application does not impose any restriction on this.

[0146] In some embodiments, the second estimated result can also be some indicator data obtained based on comprehensive analysis. For example, the recommendation system can predict some rating data of the target object for the first advertising information, and personalized recommendations can be implemented based on the rating data. In this case, the rating data can be used as the second estimated result. In the embodiments of the present application, there is no restriction on the specific reference factors and implementation logic of the rating data.

[0147] Step 430: predicting the interest tendency of the target object to the second advertisement information through the recommendation system to obtain a third estimation result;

[0148] In this step, similar to the first advertisement information, for the obtained second advertisement information, the recommendation system to be detected can also be used to predict the interest tendency of the target object in the second advertisement information, and an estimated result is obtained, which is recorded as the third estimated result. This process is similar to the processing process of the first advertisement information, and will not be repeated here.

[0149] It can be understood that in the embodiment of the present application, the same recommendation system is used to predict the first advertising information and the second advertising information. Therefore, the data structure and meaning of the second estimated result and the third estimated result obtained are corresponding, which can facilitate the subsequent comparison or calculation processing of the second estimated result and the third estimated result.

[0150] Step 440: determine a second estimated deviation between the second estimated result and the third estimated result, and obtain a stability detection result of the recommendation system according to the second estimated deviation.

[0151] In this step, after obtaining the second estimated result and the third estimated result, the estimated deviation between the second estimated result and the third estimated result can be determined, and in the embodiment of the present application, it is recorded as the second estimated deviation. It can be understood that in the embodiment of the present application, the estimated result of the target object for two similar advertising information is predicted by the recommendation system. Since the recommendation system used is the same recommendation system, the predicted target object is the same target object, and the predicted advertising information is similar advertising information, the second estimated result and the third estimated result obtained should be close, that is, the second estimated deviation should be smaller. Therefore, the actual size of the second estimated deviation can reflect the stability of the recommendation system to a certain extent, and the stability test result of the recommendation system can be determined according to the second estimated deviation. The larger the size of the second estimated deviation, the worse the stability of the recommendation system and the less ideal the application effect; conversely, the smaller the size of the second estimated deviation, the better the stability of the recommendation system and the better the application effect.

[0152] In the embodiment of the present application, there is no restriction on the specific data form and content of the stability test result, which can use text, numerical values ​​or characters to represent the predetermined meaning. Specifically, the stability test result can include two situations: pass and fail. If the stability test result corresponding to the recommendation system is pass, it means that the test believes that the stability of the recommendation system is good and can meet the application requirements; conversely, if the stability test result corresponding to the recommendation system is fail, it means that the test believes that the stability of the recommendation system is poor and does not meet the application requirements, and the recommendation system needs to be further updated and optimized.

[0153] In the embodiments of the present application, there is no limitation on the specific relationship between the second estimated deviation and the stability test result, which can be flexibly set according to the requirements. For example, in some embodiments, a threshold value for comparison with the second estimated deviation can be set. If the second estimated deviation is greater than or equal to the threshold value, it can be considered that the stability of the current recommendation system is poor, and the obtained stability test result is not passed; if the second estimated deviation is less than the threshold value, it can be considered that the stability of the current recommendation system is good, and the obtained stability test result is passed.

[0154] In this step, the specific calculation method of the second estimated deviation can be determined according to the data form of the second estimated result and the third estimated result. Exemplarily, when the second estimated result and the third estimated result are numerical results, in some embodiments, the difference between the second estimated result and the third estimated result can be directly calculated as the second estimated deviation; in some embodiments, the difference between the second estimated result and the third estimated result can be calculated first, and then the proportion of the difference to the second estimated result or the third estimated result can be calculated, and the obtained proportion value can be used as the second estimated deviation. When the second estimated result and the third estimated result are not numerical results, the second estimated result and the third estimated result can be compared to see if they are the same. If the second estimated result and the third estimated result are the same, then the obtained second estimated deviation can be considered to be 0; relatively, if the second estimated result and the third estimated result are not the same, then the second estimated deviation can be determined as a specified value that is not 0. Of course, it can be understood that the above is only used to illustrate some implementation methods of calculating the second estimated deviation by way of example, and does not mean that it is limited to its specific implementation. In practical applications, other methods may also be used to calculate the second estimated deviation, and this application does not impose any limitation on this.

[0155] It is understandable that the stability detection method of the recommendation system in the embodiment of the present application determines the deviation of the estimated results obtained by the recommendation system for the same target object on two similar advertising information to achieve the stability detection of the recommendation system. On the one hand, the detection method does not need to collect the real results of the target object feedback, which can reduce the time cost required for detection, improve the lag problem existing in the current detection method, and is suitable for newly developed recommendation systems; on the other hand, the detection method can effectively detect the stability of the recommendation system in the dimensions of two samples, which can improve the granularity of the detected samples and facilitate the acquisition of more accurate and reliable stability detection results.

[0156] Specifically, in a possible implementation manner, obtaining the first advertisement information and the second advertisement information includes:

[0157] Obtaining an advertisement information queue; the advertisement information queue includes a plurality of candidate advertisement information;

[0158] Dividing the advertisement information queue to obtain at least one advertisement information set; wherein the advertisement information set includes at least two candidate advertisement information;

[0159] The first advertisement information and the second advertisement information are determined from candidate advertisement information in the same advertisement information set.

[0160] In the embodiment of the present application, the first advertisement information and the second advertisement information can be obtained from a pre-divided advertisement information set. Specifically, an advertisement information queue can be obtained first, and the advertisement information queue can include multiple (i.e., at least two) advertisement information, and these advertisement information are recorded as candidate advertisement information. The meaning of the candidate advertisement information is similar to that of the first advertisement information and the second advertisement information, and will not be repeated here.

[0161] After obtaining the advertisement information queue, the advertisement information queue can be divided, and the candidate advertisement information can be divided into an advertisement information set, wherein the candidate advertisement information divided into the same advertisement information set belongs to similar advertisement information. Here, the similarity between the candidate advertisement information in the same advertisement information set (recorded as the second similarity) can be constrained to be greater than or equal to the first threshold. The specific data format and calculation method of the second similarity can be implemented with reference to the aforementioned first similarity, and will not be described in detail here.

[0162] It should be noted that in the embodiments of the present application, in some cases, the first advertising information and the second advertising information may be advertising information with the same content. In the application scenario of advertising push, advertisers may place the same advertising information on multiple platforms. For these advertising information, although their content is the same, due to the differences in audiences or display styles that may exist on different platforms, the recommendation system generally takes into account the environmental differences brought by the platform when predicting the target object's interest in advertising information, and predicts the estimated results corresponding to these advertising information respectively, which can be regarded as performing multiple prediction tasks and obtaining multiple samples. Therefore, in the embodiments of the present application, when advertising information with the same content and involving different environments can also be selected as the first advertising information and the second advertising information, the present application does not limit this. In other words, the above-mentioned advertising information queue may include some candidate advertising information with the same content and involving different environments. When dividing the advertising information set, these candidate advertising information can be divided into one advertising information set.

[0163] Specifically, in the embodiment of the present application, when dividing the advertisement information queue, a corresponding division strategy can be selected according to actual needs, and the present application does not impose any restrictions on this.

[0164] For example, in some embodiments, fingerprint information corresponding to each candidate advertisement information may be extracted, and the advertisement information queue may be divided according to the fingerprint information. Here, the fingerprint information is a unique identifier of the candidate advertisement information, which may be obtained by mapping the candidate advertisement information. In the advertisement information queue, the candidate advertisement information may be divided according to the fingerprint information, and the same candidate advertisement information may be divided into the same advertisement information set.

[0165] In some embodiments, feature information corresponding to each candidate advertising information can be extracted and recorded as first feature information, and then the advertising information queue can be divided according to the first feature information. Here, the feature information can be extracted by a machine learning model or a deep learning model. The first feature information can reflect the correlation between different candidate advertising information in the feature space. In the same feature space, the first feature information corresponding to similar candidate advertising information is closer. Therefore, the first feature information can be used to realize the division of the advertising information queue, and similar (or identical) candidate advertising information can be divided into the same advertising information set.

[0166] In some embodiments, the industry category information corresponding to each candidate advertising information can be obtained, and the advertising information queue can be divided according to the industry category information. It can be understood that the industry category information is the classification information of the industry or field to which the advertising information belongs, and the candidate advertising information corresponding to the same industry category information can be considered to belong to similar advertising information to a certain extent. Therefore, in the embodiment of the present application, the advertising information queue can be divided by obtaining the industry category information corresponding to the candidate advertising information.

[0167] In some embodiments, the standardized product unit information corresponding to each candidate advertising information can also be obtained, and the advertising information queue can be divided according to the standardized product unit information. Standardized product unit information is a standardized description of advertising information. For resources in the form of commodities, if the standardized product unit information corresponding to two resources is the same, it means that the two resources belong to the same commodity. Therefore, the candidate advertising information corresponding to the same standardized product unit information can be considered to be similar advertising information to a certain extent. In the embodiment of the present application, the advertising information queue can be divided by obtaining the standardized product unit information corresponding to the candidate advertising information.

[0168] Of course, it can be understood that the above implementation method of dividing the advertisement information queue is only described as an example, and the present application does not limit its specific implementation process.

[0169] Specifically, in a possible implementation, the fingerprint information includes an information summary value; extracting the fingerprint information of each candidate advertisement information, and dividing the advertisement information queue according to the fingerprint information, including:

[0170] Extracting information summary values ​​corresponding to each candidate advertisement information;

[0171] The information digest values ​​are compared, and the candidate advertisement information with the same corresponding information digest values ​​are grouped into the same advertisement information set.

[0172] In the embodiment of the present application, when the advertising information queue is divided according to the fingerprint information, in some cases, the information digest algorithm can be used to extract the fingerprint information of each candidate advertising information, that is, the information digest value can be used as the fingerprint information. Here, the information digest algorithm is an algorithm that converts data of any length into a summary value of a fixed length. It generates a unique summary value by performing a series of mathematical operations and logical operations on the input data. The information digest algorithm is usually one-way, that is, the original data cannot be obtained by inverting the summary value.

[0173] In an embodiment of the present application, an information summary value of each candidate advertising information can be extracted through an information summary algorithm, and then each information summary value can be compared. If the information summary values ​​corresponding to multiple candidate advertising information are the same, it means that their contents are the same, and these candidate advertising information can be divided into the same advertising information set; if the information summary values ​​corresponding to two candidate advertising information are different, it can be further determined whether they belong to similar advertising information, which is not elaborated in the present application.

[0174] In an embodiment of the present application, when extracting the information digest value corresponding to the candidate advertising information, there is no restriction on the specific information digest algorithm used. Exemplarily, in some embodiments, the information digest algorithm used may be the MD5 (Message Digest Algorithm 5) algorithm. The MD5 algorithm can convert input data of any length into a 128-bit digest value. In other embodiments, the information digest algorithm used may be the SHA-256 (Secure Hash Algorithm 256-bit) algorithm. The SHA-256 algorithm can convert input data of any length into a 256-bit digest value. In an embodiment of the present application, when extracting the information digest value corresponding to the candidate advertising information, if the candidate advertising information includes multiple pictures, the information digest value of each picture can be extracted, and then the various information digest values ​​can be merged as the information digest value corresponding to the candidate advertising information.

[0175] Specifically, in a possible implementation, extracting first characteristic information of each candidate advertisement information, and dividing the advertisement information queue according to the first characteristic information includes:

[0176] Taking the first characteristic information corresponding to any candidate advertisement information as the initial cluster center, establishing a cluster;

[0177] Calculate the third similarity between each feature information to be clustered and the initial cluster center; the feature information to be clustered is the first feature information that has not been added to the clustering cluster;

[0178] When the third similarity between the feature information to be clustered and the initial cluster center is greater than or equal to the second threshold, the feature information to be clustered is added to the cluster corresponding to the initial cluster center, and the position of the initial cluster center is updated; or, when the third similarity between the feature information to be clustered and any initial cluster center is less than the second threshold, the feature information to be clustered is used as the initial cluster center, and a new cluster is additionally established;

[0179] The candidate advertisement information corresponding to the first characteristic information belonging to the same cluster is divided into the same advertisement information set.

[0180] In the embodiment of the present application, when the advertising information queue is divided according to the first characteristic information, in some cases, a streaming clustering algorithm can be used to implement a specific division process. Specifically, in the embodiment of the present application, first, the first characteristic information corresponding to each candidate advertising information can be extracted. The data format of the first characteristic information can be a numerical value, a vector, or a matrix. In the embodiment of the present application, the first characteristic information in the form of a vector is used for explanation.

[0181] Specifically, after obtaining the first characteristic information corresponding to each candidate advertising information, the first characteristic information corresponding to at least one candidate advertising information can be selected as the initial cluster center to establish several cluster clusters. Then, for each candidate advertising information whose cluster category has not been determined, its corresponding first characteristic information is recorded as the characteristic information to be clustered. Based on the characteristic information to be clustered, the similarity between the characteristic information to be clustered and the initial cluster center can be determined. In the embodiment of the present application, it is recorded as the third similarity. It can be understood that the third similarity between the characteristic information to be clustered and the initial cluster center can reflect the similarity between the two candidate advertising information. The larger the third similarity, the closer the candidate advertising information corresponding to the characteristic information to be clustered and the candidate advertising information corresponding to the cluster cluster are. Therefore, in the embodiment of the present application, a similarity threshold can be set in advance, recorded as the second threshold. If the third similarity between the feature information to be clustered and a certain initial cluster center is greater than or equal to the second threshold, it means that the candidate advertising information corresponding to the feature information to be clustered and the candidate advertising information corresponding to the cluster to which the initial cluster center belongs are sufficiently similar. Therefore, the feature information to be clustered can be added to the cluster to which the initial cluster center corresponds, and the cluster center coordinates of the cluster are updated once. On the contrary, if the third similarity between the feature information to be clustered and all the established initial cluster centers are less than the second threshold, it means that the candidate advertising information corresponding to the feature information to be clustered and the candidate advertising information corresponding to the cluster to which each initial cluster center belongs are not similar. At this time, the candidate advertising information corresponding to the feature information to be clustered needs to be additionally divided into one category, that is, the feature information to be clustered is used as the initial cluster center, and a new cluster is additionally established. This cycle is repeated until the first feature information corresponding to all the candidate advertising information is divided into clusters, and the current clustering process is completed.

[0182] After determining the clusters corresponding to each first characteristic information, the candidate advertising information can be divided according to each cluster, that is, the candidate advertising information corresponding to the first characteristic information belonging to the same cluster can be divided into the same advertising information set, thereby completing the division of similar (or identical) candidate advertising information. In the embodiment of the present application, since the first characteristic information can reflect the similarity between each candidate advertising information, the third similarity actually has a certain correlation with the second similarity. Therefore, when setting the second threshold, it can be set with reference to the previous first threshold, and the present application does not limit the specific size of the second threshold.

[0183] In the above-mentioned embodiment, the implementation method of detecting the stability of the recommendation system based on the first advertising information and the second advertising information is introduced. In practical applications, in order to improve the accuracy of the stability detection of the recommendation system, batches of advertising information can be used for detection, that is, more than two pieces of advertising information can be used to determine the stability detection result of the recommendation system. In this way, the problem of large randomness and easy fluctuation of the detection results caused by a very small number of samples can be improved. This is introduced and explained in conjunction with specific embodiments below:

[0184] like Figure 5 As shown, in an embodiment of the present application, another method for detecting the stability of a recommendation system is provided. The method for detecting the stability of a recommendation system can also be applied to the above-mentioned recommendation system 210, client 220 or performance detection system 240, and can also be executed on any other software program or hardware device with similar data processing capabilities.

[0185] Reference Figure 5 The stability detection method of the recommendation system provided in the embodiment of the present application specifically includes but is not limited to steps 510 to 550:

[0186] Step 510: Obtain an advertisement information queue; the advertisement information queue includes a plurality of candidate advertisement information;

[0187] Step 520: Divide the advertisement information queue to obtain at least two advertisement information sets; wherein the advertisement information set includes at least two candidate advertisement information;

[0188] Step 530: predict the interest tendency of the target object to each candidate advertisement information through the recommendation system, and obtain a first estimation result corresponding to each candidate advertisement information;

[0189] Step 540: determining a first estimation deviation between first estimation results corresponding to candidate advertisement information in each advertisement information set;

[0190] Step 550: Obtain a stability test result of the recommendation system according to the first estimated deviation corresponding to each advertisement information set.

[0191] In the embodiment of the present application, similar to the above embodiment, when the stability of the recommendation system is detected, an advertisement information queue can be obtained, and the advertisement information queue can include multiple (i.e., at least two) candidate advertisement information. In the embodiment of the present application, the advertisement information queue can be divided to obtain at least two advertisement information sets. Here, the specific method of dividing the advertisement information set can be implemented with reference to the above embodiment, and will not be described in detail here.

[0192] It should be noted that, in the embodiment of the present application, when the candidate advertising information in the advertising information queue is divided, there may be some sets that have only one candidate advertising information. Since when there is only one candidate advertising information in the set, the first estimated deviation corresponding to the set cannot be determined, and therefore, these sets will not be used as advertising information sets later. Exemplarily, when dividing advertising information sets, after all candidate advertising information in the advertising information queue are divided, the obtained sets can be first recorded as candidate information sets, and then the number of candidate advertising information in each candidate information set can be counted. If the candidate information set includes only one candidate advertising information, the candidate information set will be shelved and will not be used as an advertising information set; if the candidate information set includes two or more candidate advertising information, the candidate information set is determined to be an advertising information set.

[0193] After the advertising information sets are divided, for each advertising information set, the recommendation system can be used to predict the interest tendency of the target object in each candidate advertising information, and obtain the estimated result corresponding to each candidate advertising information, which is recorded as the first estimated result in the embodiment of the present application. The process of obtaining the first estimated result, the data format and content of the first estimated result can be implemented with reference to the aforementioned second estimated result.

[0194] In the embodiment of the present application, for each set of advertising information, the deviation between the first estimated results corresponding to each candidate advertising information can be determined, which is recorded as the first estimated deviation. It can be understood that in the embodiment of the present application, the number of candidate advertising information included in the advertising information set may be more than two, so the first estimated deviation can be determined using data indicators such as standard deviation or variance that are suitable for statistically analyzing the distribution of multiple results, and the present application does not limit this.

[0195] After obtaining the first estimated deviation corresponding to each advertising information set, the stability test result of the recommendation system can be determined based on the data of the first estimated deviation. Exemplarily, in some embodiments, the first estimated deviation can be compared with the pre-set threshold in turn to determine whether the recommendation system can achieve relatively accurate and stable predictions in the dimension of the advertising information set. If a first estimated deviation is found to be greater than or equal to the pre-set threshold, the current recommendation system stability test result can be considered to have failed; when the first estimated deviations corresponding to all advertising information sets are less than the pre-set threshold, the current recommendation system stability test result can be considered to have passed. In other embodiments, all the first estimated deviations can also be combined to make a judgment. For example, the mean of the first estimated deviations can be calculated, and the obtained mean can be compared with the pre-set threshold to determine the stability test result of the recommendation system.

[0196] It should be noted that although the number of samples used in the embodiments of the present application is generally multiple, it can obtain the first estimated deviation that can be used to determine the stability test result of the recommendation system in each smaller sample dimension (advertising information set). Compared with the traditional detection method, it can still greatly reduce the number of samples required for detection, and can appropriately improve the sample granularity of the detection. Moreover, compared with the application of using only two advertising information for detection, the embodiments of the present application can improve the problem of large randomness and easy fluctuation of the test results caused by a very small number of samples, and facilitate the acquisition of more accurate and reliable stability test results.

[0197] Specifically, in a possible implementation, the first estimation result includes a click rate estimation result or a conversion rate estimation result of the target object for the candidate advertisement information; the first estimation result corresponding to each candidate advertisement information is obtained by predicting the interest tendency of the target object for each candidate advertisement information through the recommendation system, including:

[0198] Acquire first associated information of the target object;

[0199] Based on the candidate advertisement information and the first associated information, a click rate estimation result or a conversion rate estimation result of the target object for the candidate advertisement information is predicted by the recommendation system.

[0200] In the embodiment of the present application, as mentioned above, the second estimation result may include the target object's click rate estimation result or conversion rate estimation result for the advertisement information. Similarly, the first estimation result may also include the same data type.

[0201] In an embodiment of the present application, when the target object's interest in candidate advertising information is predicted by the recommendation system, information related to the target object can be obtained and recorded as the first associated information. Here, the first associated information may include, but is not limited to, attribute information, historical behavior information, and context information of the target object; wherein the attribute information of the target object may include basic information such as the target object's gender, age, and work. Historical behavior information may include the target object's historical click records, purchase records, and browsing records. Context information may include environmental information related to the target object, such as the network information, location information, current time information, etc. used by the target object. This information can provide background information related to the target object, which is convenient for predicting the target object's click rate estimate or conversion rate estimate for advertising information.

[0202] In an embodiment of the present application, a prediction of click-through rate prediction results or conversion rate prediction results can be achieved through a recommendation system based on the candidate advertising information and the first associated information. The specific recommendation algorithm used here can be implemented based on the introduction of the recommendation system in the aforementioned embodiment, which will not be elaborated here.

[0203] Specifically, in a possible implementation manner, determining a first estimation deviation between first estimation results corresponding to candidate advertisement information in each advertisement information set includes:

[0204] Determine an estimated mean and an estimated standard deviation of the first estimated results corresponding to each candidate advertisement information in the advertisement information set;

[0205] A first estimated deviation corresponding to the advertisement information set is obtained according to the quotient of the estimated standard deviation and the estimated mean.

[0206] In an embodiment of the present application, when determining the first estimated deviation corresponding to a set of advertising information, it is necessary to take into account the situation that different types of advertising information may correspond to relatively large differences in the estimated results as a whole. For example, there may be a part of the advertising information that has a generally high click-through rate, such as at a level of about 10%, while the click-through rate of advertising information in another part of the field is generally low, such as at a level of about 1%. In this way, when calculating the first estimated deviation corresponding to the advertising information set in the two fields, if standard deviation data or variance data is used, due to the natural dimensional difference, the first estimated deviation obtained will also be quite different, which cannot well reflect the stability of the recommendation system. Therefore, in an embodiment of the present application, a technical solution is provided to improve this problem by using the coefficient of variation.

[0207] Specifically, in an embodiment of the present application, when determining the first estimated deviation corresponding to the advertising information set, first, the estimated mean and estimated standard deviation of the first estimated results corresponding to each candidate advertising information in the advertising information set can be determined, and then the quotient of the estimated standard deviation and the estimated mean corresponding to the advertising information set can be calculated, and this quotient value is used as the first estimated deviation corresponding to the advertising information set, that is, CV = σ / μ, where CV represents the first estimated deviation, σ represents the estimated standard deviation of the first estimated results corresponding to each candidate advertising information in the advertising information set, and μ represents the estimated mean of the first estimated results corresponding to each candidate advertising information in the advertising information set. In this way, the natural dimensional difference of the first estimated result can be avoided, and a more practical first estimated deviation can be obtained.

[0208] Specifically, in a possible implementation, obtaining a stability detection result of the recommendation system according to the first estimated deviation corresponding to each advertisement information set includes:

[0209] Counting the number of candidate advertisement information in each advertisement information set;

[0210] Determine the weight corresponding to each set of advertising information according to the number;

[0211] According to the weighted weights, weighted summation is performed on the first estimated deviations corresponding to the respective advertising information sets to obtain a third estimated deviation;

[0212] Determining a stability test result of the recommendation system according to the third estimated deviation;

[0213] The weighted weight corresponding to the advertisement information set is positively correlated with the number of candidate advertisement information in the advertisement information set.

[0214] As mentioned above, after obtaining the first estimated deviation corresponding to each advertising information set, all the first estimated deviations can be combined to determine the stability test result of the recommendation system. In the embodiment of the present application, an implementation scheme that comprehensively considers each first estimated deviation is proposed. Specifically, it can be understood that each advertising information set includes multiple candidate advertising information. Relatively speaking, the more candidate advertising information there are in the advertising information set, the more representative its corresponding first estimated deviation is. Therefore, in the embodiment of the present application, the first estimated deviations corresponding to the advertising information set can be weighted and summed based on the number of candidate advertising information in each advertising information set, so as to determine the stability test result of the recommendation system.

[0215] Specifically, in an embodiment of the present application, the number of candidate advertising information in each advertising information set can be counted, and the weighted weight corresponding to each advertising information set can be determined based on the number. The weighted weight and the number here can be positively correlated, and the present application does not limit the specific functional relationship between the two. After obtaining the weighted weight data, the first estimated deviation corresponding to each advertising information set can be weighted and summed, and the obtained value can be recorded as the third estimated deviation. Then, the stability test result of the recommendation system can be determined based on the relationship between the third estimated deviation and a pre-set threshold. In this way, the results of each first estimated deviation can be better integrated to obtain a more accurate and reliable stability test result.

[0216] Specifically, in a possible implementation manner, obtaining a stability detection result of the recommendation system according to the third estimated deviation includes:

[0217] Obtaining third estimated deviations corresponding to multiple target objects;

[0218] Performing weighted summation on the third estimated deviations corresponding to each target object to obtain a fourth estimated deviation;

[0219] A stability detection result of the recommendation system is determined according to the fourth estimated deviation.

[0220] In the embodiments of the present application, in addition to comprehensively considering multiple advertising information to improve the accuracy of the stability detection results, in some embodiments, the recommendation system can also be used to make predictions on multiple target objects, using the same method as the aforementioned embodiments to obtain the third estimated deviation corresponding to each target object.

[0221] After obtaining the third estimated deviation corresponding to each target object, these third estimated deviations can be weighted and summed, and the obtained value can be recorded as the fourth estimated deviation. Then, the stability test result of the recommendation system can be determined based on the relationship between the fourth estimated deviation and the preset threshold value. Here, the threshold can be recorded as the deviation threshold. Specifically, if the fourth estimated deviation is greater than or equal to the preset deviation threshold, it can be determined that the stability test result of the recommendation system is failed; conversely, if the fourth estimated deviation is less than the preset deviation threshold, it can be determined that the stability test result of the recommendation system is passed. In this way, the performance test of the recommendation system can be implemented on multiple target objects to obtain more accurate and reliable stability test results.

[0222] It should be noted that here, the first estimated deviation or the second estimated deviation corresponding to each target object can also be obtained, and then the estimated deviations corresponding to multiple target objects are weighted to comprehensively determine the stability detection result of the recommendation system. This application does not elaborate on this.

[0223] Specifically, in a possible implementation, a recommendation model is configured in the recommendation system; the recommendation system predicts the interest tendency of the target object for each of the candidate advertisement information, and obtains a first estimation result corresponding to each of the candidate advertisement information, including:

[0224] Acquire first associated information of the target object;

[0225] Extracting first feature information corresponding to the candidate advertisement information and second feature information corresponding to the first associated information through the recommendation model;

[0226] According to the first feature information and the second feature information, a first estimation result is obtained by prediction through a recommendation model.

[0227] In an embodiment of the present application, a method for predicting a first estimated result by using a recommendation model is provided. Here, the recommendation model can be built based on any machine learning algorithm. After the recommendation model is trained, it can be configured in the recommendation system to predict the interest tendency of the target object in advertising information.

[0228] Specifically, in an embodiment of the present application, when the recommendation model is used to predict the first estimated result, the first associated information of the target object can be obtained. Then, the first feature information corresponding to the candidate advertising information can be extracted through the recommendation model, and the feature information of the first associated information can be extracted, which is recorded as the second feature information. After obtaining the first feature information and the second feature information, the recommendation model can be used to predict the interest tendency of the target object in the candidate advertising information based on these feature information, and obtain the corresponding first estimated result. Specifically, the recommendation model can splice or fuse the first feature information and the second feature information, and then implement the prediction output of the first estimated result based on the obtained spliced ​​feature information or fused feature information. The present application does not limit the specific form of the first estimated result. In some embodiments, the first estimated result predicted by the recommendation model can be a classification result, such as a classification result that the target object is interested in or not interested in the candidate advertising information, or a classification result that the candidate advertising information is recommended to the target object or not recommended to the target object; in some embodiments, the first estimated result predicted by the recommendation model can also be a numerical result, such as a click-through rate estimation result or a conversion rate estimation result of the target object for the candidate advertising information, and the present application does not limit this.

[0229] Exemplarily, in some embodiments, the recommendation model of the present application may include a feature extractor and a classifier, wherein the feature extractor can be used to extract feature information, and the classifier can be used to process the feature information to obtain the corresponding estimated results. Specifically, the feature extractor can be implemented using a recurrent neural network (RNN), such as the commonly used long-short term memory network (LSTM); the classifier can be implemented using logistic regression, support vector machine, decision tree, etc., and the present application does not limit this.

[0230] Specifically, in a possible implementation, the recommendation model is trained through the following steps:

[0231] Acquire sample advertising information, second associated information of the sample object, and label information; the label information is used to represent the actual result of the sample object's interest tendency in the sample advertising information;

[0232] extracting the third characteristic information of the sample advertisement information and the fourth characteristic information of the second related information through the recommendation model;

[0233] According to the third feature information and the fourth feature information, predicting the interest tendency of the sample object to the sample advertising information through the recommendation model, and obtaining a fourth estimation result;

[0234] Determine the loss value of training according to the fourth estimation result and label information;

[0235] The parameters in the recommendation model are updated according to the loss value to obtain a trained recommendation model.

[0236] Before the recommendation model is actually applied, it is necessary to optimize its performance through training. In the embodiment of the present application, a corresponding method for training a recommendation model is provided. Specifically, when training the recommendation model, sample advertising information, second associated information of the sample object and label information can be obtained. Here, the sample advertising information and the aforementioned candidate advertising information are similar in meaning, the sample object and the aforementioned target object are similar in meaning, and the second associated information is the associated information of the sample object, which is similar in meaning to the aforementioned first associated information, and will not be elaborated here. During the training process, the recommendation model can be used to predict the interest tendency of the sample object in the sample advertising information; the label information is the actual result of the interest tendency of the sample object in the sample advertising information.

[0237] After obtaining the training data, the sample advertising information and the second associated information can be input into the recommendation model, and the feature information of the sample advertising information can be extracted through the recommendation model, which is recorded as the third feature information, and the feature information of the second associated information can be recorded as the fourth feature information. Then, based on the third feature information and the fourth feature information, the recommendation model can be used to predict the interest tendency of the sample object in the sample advertising information, and an estimated result can be obtained, which is recorded as the fourth estimated result. After obtaining the fourth estimated result, the prediction accuracy of the recommendation model can be evaluated based on the label information and the fourth estimated result, so as to facilitate the iterative update of the parameters of the recommendation model. Specifically, the loss value of the training can be determined based on the label information and the fourth estimated result. After obtaining the loss value, the recommendation model can be back-propagated and trained to update the relevant parameters inside it.

[0238] Specifically, for various models in the field of artificial intelligence, the accuracy of its prediction can be measured by a loss function, which is defined on a single training data and is used to measure the prediction error of a training data. Specifically, the loss value of the training data is determined by the label of a single training data and the prediction result of the model for the training data. During actual training, a training data set includes a lot of training data, so a cost function is generally used to measure the overall error of the training data set. The cost function is defined on the entire training data set and is used to calculate the average value of the prediction error of all training data, which can better measure the prediction effect of the model. For a general machine learning model, based on the aforementioned cost function, plus a regularization term that measures the complexity of the model, it can be used as the objective function of the training, and the loss value of the entire training data set can be calculated based on the objective function. There are many types of commonly used loss functions, such as 0-1 loss function, square loss function, absolute loss function, logarithmic loss function, cross entropy loss function, etc., which can all be used as loss functions of machine learning models, which will not be described one by one here. In the embodiment of the present application, one loss function can be selected to determine the loss value of the training, that is, the loss value between the label information and the fourth estimated result.

[0239] Specifically, in a possible implementation manner, the method further includes:

[0240] If the stability test result of the recommendation system fails, an alarm message is generated;

[0241] Display the warning information, or send the warning information to the associated object of the recommendation system.

[0242] In the embodiment of the present application, when the stability test result of the recommendation system fails, an alarm message may be generated accordingly, and the alarm message may be displayed on the relevant device interface, or sent to the associated objects of the recommendation system, such as the developer and tester of the recommendation system or the service provider of the recommendation system, to remind them to optimize or adjust the recommendation system accordingly. The present application does not limit the specific data form and content of the alarm message.

[0243] like Figure 6 As shown, in an embodiment of the present application, a method for recommending advertising information is also provided. The method for recommending advertising information can be applied to the above-mentioned recommendation system 210, and can also be executed on any other software program or hardware device with similar data processing capabilities.

[0244] Reference Figure 6 The method for recommending advertising information provided in the embodiment of the present application specifically includes but is not limited to steps 610 to 620:

[0245] Step 610: Detect the recommendation system using the recommendation system stability detection method to obtain a recommendation system stability detection result;

[0246] Step 620: When the stability test result of the recommendation system is passed, the recommendation system is used to recommend advertising information.

[0247] In the embodiment of the present application, a method for recommending advertising information is provided, and the method can be implemented based on the stability detection method of the recommendation system provided in the embodiment of the present application. Therefore, it can be understood that the contents of the stability detection method of the recommendation system provided in the embodiment of the present application are applicable to the embodiment of the recommendation method of advertising information, and the beneficial effects achieved by the stability detection method of the recommendation system are also applicable to the embodiment of the recommendation method of advertising information.

[0248] Specifically, in an embodiment of the present application, when the recommendation system needs to be deployed online, the recommendation system can be tested by the stability detection method of the recommendation system to determine its corresponding stability detection result. Based on the introduction of the stability detection method of the recommendation system in the aforementioned embodiment, it can be understood that in an embodiment of the present application, if the stability detection result of the recommendation system is passed, it means that its application effect is good and it can continuously and smoothly output relatively accurate estimated results; if the stability detection result of the recommendation system is not passed, it means that its application effect is poor and may affect the recommendation performance of advertising information. Therefore, in an embodiment of the present application, if the stability detection result of the recommendation system is passed, the recommendation system can be deployed normally, and the recommendation system can be used to recommend advertising information. Relatively speaking, when the stability detection result of the recommendation system is not passed, the deployment can be suspended, and after it is updated and optimized, it can be deployed online and put into use.

[0249] The advertising information recommendation method provided in the embodiments of the present application can implement business adjustments to the recommendation system based on accurate and reliable stability detection results, which is beneficial to improving the application effect of the recommendation system and improving the recommendation performance of advertising information.

[0250] Below, in conjunction with the specific application implementation process, the stability detection method of the recommendation system and the recommendation method of advertising information provided in this application are introduced and explained in detail.

[0251] The stability detection method of the recommendation system provided in the embodiment of the present application can be applied to the advertising push scenario. Currently, in most common applications, there is advertising information delivery. For example, referring to Figure 7 , Figure 7 A schematic diagram of displaying advertising information on an application interface is shown. Generally speaking, a corresponding advertising display area 710 is set on the display interface of the application, and advertisers can pay to display advertising information. In the application, the behavior chain corresponding to the advertising information can generally be summarized as display-click-conversion, where display specifies that the advertising information is displayed in the interface of the application, click refers to the target object (user of the application) clicking on the advertising information, and conversion is the business process executed after the target object clicks on the advertising information, such as registering an account, downloading the application, following a public account, filling out a form, etc. This application does not limit this.

[0252] In the above process, the recommendation system can predict the probability of the target object performing related behaviors, such as the probability of the target object performing click behavior from the display of advertising information, which is the click-through rate prediction result (pCTR), and can also predict the probability of the target object performing click behavior to the target object performing conversion behavior, which is the conversion rate prediction result (pCVR). The click-through rate prediction results and conversion rate prediction results are of great significance for the delivery of advertising information. For the target object, predicting the click-through rate prediction results and conversion rate prediction results can determine the content that the target object is interested in, which is convenient for targeted delivery, and can facilitate the target object to find the desired content from the massive amount of information, saving the time and energy consumed by information search; for advertisers, based on the click-through rate prediction results and conversion rate prediction results, accurate delivery of advertising information can be achieved, which is conducive to the promotion of related products or services and the increase of revenue and profit. In addition, the deduction of some advertising information is also related to the click-through rate and conversion rate. Therefore, predicting the click-through rate prediction results and conversion rate prediction results through the recommendation system is very important in the scenario of advertising push.

[0253] Of course, it is understandable that in the scenario of advertising push, the estimated results output by the recommendation system are not limited to the click-through rate estimation results and the conversion rate estimation results. For example, in some embodiments, the estimated results output by the recommendation system may also be the ranking value (Learning To Rank) of the advertising information in the recall and rough sorting stages. This indicator can usually be used to characterize the degree of interest of the object in the advertising information. Relatively speaking, the higher the degree of interest, the higher the ranking of the advertising information, and the easier it is to be pushed to the object by the recommendation system for display later. The specific relationship between the ranking value and the degree of interest is not limited in this application. In some embodiments, the estimated results output by the recommendation system may also be the product of the click-through rate and the conversion rate, which is not limited in this application.

[0254] In an embodiment of the present application, a method for detecting the stability of a recommendation system and a method for recommending advertising information are provided, which can help evaluate the stability of the recommendation system in an advertising push scenario, and is conducive to improving the application effect of the recommendation system and improving the recommendation performance of advertising information.

[0255] Reference Figure 8 Specifically, in the embodiment of the present application, when the recommendation system is detected, a candidate advertisement queue (i.e., advertisement information queue) can be obtained, and the queue can include multiple advertisement information (i.e., candidate advertisement information). Then, the fingerprint information of these advertisement information can be extracted by the fingerprint extraction system, and the candidate advertisement queue can be divided according to the fingerprint information, and the advertisement information with the same content can be divided into the same advertisement information set to obtain multiple advertisement information sets. Figure 8In the above, the obtained advertising information set can correspond to the fingerprint information one by one. These fingerprint information can carry label information. According to the label information, the fingerprint information can be recorded as fingerprint 1, fingerprint 2, fingerprint 3... fingerprint N (N is a positive integer). Correspondingly, the advertising information set can include the advertising information set corresponding to fingerprint 1, the advertising information set corresponding to fingerprint 2, etc.

[0256] Reference Fig. 9 In an embodiment of the present application, when dividing the advertising information set, for each advertising information, if the extracted fingerprint information is the same as the existing fingerprint information, it will be added to the advertising information set corresponding to the fingerprint information; if the extracted fingerprint information is different from the existing fingerprint information, a new advertising information set corresponding to the fingerprint information can be created, and the current advertising information can be added to the set. Specifically, in an embodiment of the present application, the fingerprint information can be sorted according to the order in which the different fingerprint information is extracted, and the corresponding label information can be determined, so that each fingerprint information can be recorded as fingerprint 1, fingerprint 2, fingerprint 3... fingerprint N (N is a positive integer) in turn. Each fingerprint information corresponds to an advertising information set, and the advertising information set can include one or more advertising information. In an embodiment of the present application, for the advertising information corresponding to each fingerprint information, it can be sorted in combination with the serial number of the fingerprint information. For example, referring to Fig. 9 , each advertisement information corresponding to fingerprint 1 can be assigned serial numbers id11, id12, and id13 in sequence. The advertisement information can be subsequently divided based on the corresponding serial numbers. For example, the advertisement information with serial numbers id11, id12, and id13 can be divided into the first advertisement information set 910 corresponding to fingerprint 1. In this way, multiple advertisement information sets can be obtained.

[0257] In the embodiment of the present application, for each set of advertising information, a recommendation system can be used to predict the click rate estimation result (or conversion rate estimation result) of each advertising information therein. According to the click rate estimation result (or conversion rate estimation result) predicted by the recommendation system, the estimation deviation (i.e., the first estimation deviation) corresponding to each set of advertising information can be determined by the coefficient of variation (CV). Figure 8 Based on the advertising information set corresponding to fingerprint 1, a first estimated deviation, namely, coefficient of variation CV1, can be determined; based on the advertising information set corresponding to fingerprint 2, another first estimated deviation, namely, coefficient of variation CV2, can be determined. Similarly, after obtaining the first estimated deviations corresponding to each advertising information set, these first estimated deviations can be aggregated, such as by taking the mean or weighting to obtain the overall estimated deviation (i.e., the third estimated deviation).

[0258] For example, there are currently three sets of advertising information, and the corresponding coefficients of variation are CV1, CV2, and CV3, respectively. Then, the mean of these coefficients of variation, i.e. (CV1+CV2+CV3) / 3, can be taken as the overall estimated deviation; or a weighted average method can be used, for example, the number of advertisements in the three sets of advertising information accounts for the total number of advertisements in the entire candidate advertising queue in proportions of x, y, and z, respectively. The estimated deviation after weighted average can be expressed as x*CV1+y*CV2+z*CV3. It can be understood that the smaller the estimated deviation value here, the higher the stability of the recommendation system and the better the application effect.

[0259] It should be noted here that if an advertising information set only includes one advertising information, then the first estimated deviation corresponding to the advertising information set will not be calculated subsequently. Therefore, the advertising information set can be ignored, and only the advertising information set including multiple advertising information can be used to participate in the stability detection of the recommendation system.

[0260] It is understandable that in the foregoing embodiments, the implementation principle of the embodiments of the present application is described at the level of a single target object. In actual application, the estimated deviations corresponding to multiple target objects can be calculated together, and the estimated deviations corresponding to multiple target objects can be aggregated as an indicator value for evaluating the stability of the recommendation system. Fig.10 In the embodiment of the present application, it is assumed that a total of T (T is a positive integer) target objects corresponding to the estimated deviations are processed, and T estimated deviation data are obtained, which are respectively recorded as avg_cv1, avg_cv2...avg_cvT. The estimated deviations of these T target objects can be averaged (or weighted summed) to obtain the estimated deviations at the multi-object level. Then, the obtained estimated deviation is compared with the pre-set deviation threshold. If it is greater than or equal to the deviation threshold, it means that the detection result is not passed, and an alarm message can be generated to remind the recommendation system that further optimization is needed; on the contrary, if it is less than the deviation threshold, it means that the detection result is passed, and it can be considered that the recommendation system performs normally and has high stability and can be put into use.

[0261] The stability detection method for the recommendation system provided in the embodiment of the present application, when applied in the advertising push scenario, does not need to collect the actual feedback operation results of the object on the advertising information, can reduce the time cost required for detection, can improve the lag problem existing in the current detection method, and is suitable for newly developed recommendation systems; moreover, the detection method does not rely on a large amount of sample data to effectively detect the stability of the recommendation system, can reduce the amount of data processing, improve the sample granularity of the detection, and facilitate obtaining more accurate and reliable stability detection results.

[0262] The method for recommending advertising information provided in the embodiments of the present application applies the recommendation system based on the stability detection result of the recommendation system. When the stability detection result of the recommendation system is passed, the recommendation system can be used to recommend advertising information, which can improve the application effect of the recommendation system and enhance the recommendation performance of advertising information.

[0263] In an embodiment of the present application, another method for detecting the stability of a recommendation system is provided. The method for detecting the stability of a recommendation system can also be applied to the above-mentioned recommendation system 210, client 220 or performance detection system 240, and can also be executed on any other software program or hardware device with similar data processing capabilities.

[0264] The stability detection method of the recommendation system provided in the embodiment of the present application specifically includes but is not limited to the following steps:

[0265] Obtaining first advertisement information;

[0266] Predicting the first subject's interest tendency in the first advertisement information by the recommendation system to obtain a fifth estimation result;

[0267] Predicting the interest tendency of the second object in the first advertisement information through the recommendation system to obtain a sixth estimation result; wherein the fourth similarity between the first object and the second object is greater than or equal to the third threshold;

[0268] A fifth estimation deviation between the fifth estimation result and the sixth estimation result is determined, and a stability detection result of the recommendation system is obtained according to the fifth estimation deviation.

[0269] In the present application embodiment, and Figure 4 Similarly, another stability detection method for recommendation systems is provided. Figure 4 The stability detection method shown, in the embodiment of the present application, obtains the first advertising information, and predicts the interest tendency of two similar objects in the same first advertising information through the recommendation system. Here, the similar objects are recorded as the first object and the second object, and the interest tendency of the first object in the first advertising information is predicted by the recommendation system, and the estimated result is recorded as the fifth estimated result; the interest tendency of the second object in the first advertising information is predicted by the recommendation system, and the estimated result is recorded as the sixth estimated result. Comparing the fifth estimated result and the sixth estimated result, the estimated deviation between the two can be obtained, which is recorded as the fifth estimated deviation. It can be understood that since the same first advertising information is presented to the similar first object and the second object, the fifth estimated result and the sixth estimated result should be relatively close, so the size of the fifth estimated deviation can reflect the stability of the recommendation system to a certain extent, and the stability detection result of the recommendation system can be determined according to the fifth estimated deviation.

[0270] In the embodiments of the present application, various estimated results, estimated deviations, and a method for determining stability detection results based on the estimated deviations can be implemented with reference to the aforementioned embodiments and will not be elaborated herein.

[0271] Similarly, in the embodiment of the present application, the similarity relationship between the first object and the second object can be constrained by similarity, and the similarity between the first object and the second object is recorded as the fourth similarity. A threshold can be set, recorded as the third threshold, and the fourth similarity is limited to be greater than or equal to the third threshold, so that the first object and the second object are within a similar range. The specific size of the third threshold is not limited in the present application.

[0272] In an embodiment of the present application, another method for detecting the stability of a recommendation system is provided. The method for detecting the stability of a recommendation system can also be applied to the above-mentioned recommendation system 210, client 220 or performance detection system 240, and can also be executed on any other software program or hardware device with similar data processing capabilities.

[0273] The stability detection method of the recommendation system provided in the embodiment of the present application specifically includes but is not limited to the following steps:

[0274] Acquire first advertisement information and second advertisement information; a first similarity between the first advertisement information and the second advertisement information is greater than or equal to a first threshold;

[0275] Predicting the first subject's interest tendency in the first advertisement information by the recommendation system to obtain a fifth estimation result;

[0276] Predicting the interest tendency of the second object in the second advertisement information by the recommendation system to obtain a seventh estimation result; wherein the fourth similarity between the first object and the second object is greater than or equal to the third threshold;

[0277] A sixth estimated deviation between the fifth estimated result and the seventh estimated result is determined, and a stability detection result of the recommendation system is obtained according to the sixth estimated deviation.

[0278] In the present application embodiment, and Figure 4 Similarly, another stability detection method for recommendation systems is provided. Figure 4The stability detection method shown, in the embodiment of the present application, obtains similar first advertising information and second advertising information, and predicts the interest tendency of two similar objects in similar advertising information through the recommendation system. Here, the two similar objects are the first object and the second object, which can be determined with reference to the aforementioned embodiment. The recommendation system predicts the interest tendency of the first object in the first advertising information, and obtains the fifth estimated result; the recommendation system predicts the interest tendency of the second object in the second advertising information, and the obtained estimated result is recorded as the seventh estimated result. Comparing the fifth estimated result and the seventh estimated result, the estimated deviation between the two can be obtained, which is recorded as the sixth estimated deviation. It can be understood that since the similar first advertising information and the second advertising information are presented to the similar first object and the second object, the fifth estimated result and the seventh estimated result should be relatively close, so the size of the sixth estimated deviation can reflect the stability of the recommendation system to a certain extent, and the stability detection result of the recommendation system can be determined according to the fifth estimated deviation.

[0279] In the embodiments of the present application, various estimated results, estimated deviations, and a method for determining stability detection results based on the estimated deviations can be implemented with reference to the aforementioned embodiments and will not be elaborated herein.

[0280] Understandably, Figure 4 The beneficial effects achieved by the illustrated embodiment of the method for detecting stability of a recommendation system are substantially the same as the beneficial effects achieved by the other two embodiments of the method for detecting stability of a recommendation system given in the present application, and the embodiments of the present application will not be described one by one here.

[0281] Reference Fig.11 In an embodiment of the present application, a device for recommending advertising information is also provided, and the device for recommending advertising information includes:

[0282] The acquisition unit 1110 is used to acquire an advertisement information queue; the advertisement information queue includes a plurality of candidate advertisement information;

[0283] The division unit 1120 is used to divide the advertisement information queue to obtain at least two advertisement information sets; wherein the advertisement information set includes at least two candidate advertisement information;

[0284] An estimation unit 1130 is used to predict the interest tendency of the target object to each candidate advertisement information through a recommendation system, and obtain a first estimation result corresponding to each candidate advertisement information;

[0285] A comparison unit 1140, configured to determine a first estimation deviation between first estimation results corresponding to candidate advertisement information in each advertisement information set;

[0286] A processing unit 1150 is used to obtain a stability detection result of the recommendation system according to the first estimated deviation corresponding to each advertisement information set;

[0287] The recommendation unit 1160 is configured to recommend advertisement information based on the recommendation system when the stability detection result of the recommendation system is passed.

[0288] Optionally, the division unit is specifically used for:

[0289] Extracting fingerprint information corresponding to each candidate advertisement information, and dividing the advertisement information queue according to the fingerprint information;

[0290] Alternatively, extracting first characteristic information corresponding to each candidate advertisement information, and dividing the advertisement information queue according to the first characteristic information;

[0291] Alternatively, the industry category information corresponding to each candidate advertisement information is obtained, and the advertisement information queue is divided according to the industry category information;

[0292] Alternatively, the standardized product unit information corresponding to each candidate advertisement information is obtained, and the advertisement information queue is divided according to the standardized product unit information.

[0293] Optionally, the fingerprint information includes an information summary value; and the dividing unit is specifically used for:

[0294] Extracting information summary values ​​corresponding to each candidate advertisement information;

[0295] The information digest values ​​are compared, and the candidate advertisement information with the same corresponding information digest values ​​are grouped into the same advertisement information set.

[0296] Optionally, the division unit is specifically used for:

[0297] Extracting first feature information corresponding to each candidate advertisement information;

[0298] Taking the first characteristic information corresponding to any candidate advertisement information as the initial cluster center, establishing a cluster;

[0299] Calculate the third similarity between each feature information to be clustered and the initial cluster center; the feature information to be clustered is the first feature information that has not been added to the clustering cluster;

[0300] When the third similarity between the feature information to be clustered and the initial cluster center is greater than or equal to the second threshold, the feature information to be clustered is added to the cluster corresponding to the initial cluster center, and the position of the initial cluster center is updated; or, when the third similarity between the feature information to be clustered and any initial cluster center is less than the second threshold, the feature information to be clustered is used as the initial cluster center, and a new cluster is additionally established;

[0301] The candidate advertisement information corresponding to the first characteristic information belonging to the same cluster is divided into the same advertisement information set.

[0302] Optionally, the first estimation result includes an estimation result of a click rate or a conversion rate of the target object for the candidate advertisement information; the estimation unit is specifically used for:

[0303] Acquire first associated information of the target object;

[0304] Based on the candidate advertisement information and the first associated information, a click rate estimation result or a conversion rate estimation result of the target object for the candidate advertisement information is predicted by the recommendation system.

[0305] Optionally, the comparison unit is specifically used for:

[0306] Determine an estimated mean and an estimated standard deviation of the first estimated results corresponding to each candidate advertisement information in the advertisement information set;

[0307] A first estimated deviation corresponding to the advertisement information set is obtained according to the quotient of the estimated standard deviation and the estimated mean.

[0308] Optionally, the processing unit is specifically configured to:

[0309] Counting the number of candidate advertisement information in each advertisement information set;

[0310] Determine the weight corresponding to each set of advertising information according to the number;

[0311] According to the weighted weights, weighted summation is performed on the first estimated deviations corresponding to the respective advertising information sets to obtain a third estimated deviation;

[0312] Determining a stability test result of the recommendation system according to the third estimated deviation;

[0313] The weighted weight corresponding to the advertisement information set is positively correlated with the number of candidate advertisement information in the advertisement information set.

[0314] Optionally, the processing unit is specifically configured to:

[0315] Obtaining third estimated deviations corresponding to multiple target objects;

[0316] Performing weighted summation on the third estimated deviations corresponding to each target object to obtain a fourth estimated deviation;

[0317] A stability detection result of the recommendation system is determined according to the fourth estimated deviation.

[0318] Optionally, the processing unit is specifically configured to:

[0319] comparing the fourth estimated deviation with a predetermined deviation threshold;

[0320] If the fourth estimated deviation is greater than or equal to the deviation threshold, it is determined that the stability test result of the recommendation system is failed; or if the fourth estimated deviation is less than the deviation threshold, it is determined that the stability test result of the recommendation system is passed.

[0321] Optionally, a recommendation model is configured in the recommendation system; the estimation unit is specifically used for:

[0322] Acquire first associated information of the target object;

[0323] Extracting first feature information corresponding to the candidate advertisement information and second feature information corresponding to the first associated information through the recommendation model;

[0324] According to the first feature information and the second feature information, a first estimation result is obtained by prediction through a recommendation model.

[0325] Optionally, the recommendation device further includes a training unit, and the training unit is specifically used for:

[0326] Acquire sample advertising information, second associated information of the sample object, and label information; wherein the label information is used to represent the actual result of the sample object's interest tendency toward the sample advertising information;

[0327] extracting the third characteristic information of the sample advertisement information and the fourth characteristic information of the second related information through the recommendation model;

[0328] According to the third feature information and the fourth feature information, predicting the interest tendency of the sample object to the sample advertising information through the recommendation model, and obtaining a fourth estimation result;

[0329] Determine the loss value of training according to the fourth estimation result and label information;

[0330] The parameters in the recommendation model are updated according to the loss value to obtain a trained recommendation model.

[0331] Optionally, the recommendation device further includes an alarm unit, which is specifically used for:

[0332] If the stability test result of the recommendation system fails, an alarm message is generated;

[0333] Display the warning information, or send the warning information to the associated object of the recommendation system.

[0334] It is understandable that if Figure 6 The contents of the advertising information recommendation method embodiment shown in FIG. 1 are applicable to the advertising information recommendation device embodiment of the present invention. The functions specifically implemented by the advertising information recommendation device embodiment of the present invention are similar to those of the present invention. Figure 6 The advertising information recommendation method embodiment shown is the same as that shown in FIG. Figure 6 The beneficial effects achieved by the illustrated advertising information recommendation method embodiment are also the same.

[0335] In an embodiment of the present application, an electronic device is also disclosed, including:

[0336] at least one processor;

[0337] at least one memory for storing at least one program;

[0338] When at least one program is executed by at least one processor, the at least one processor implements Figure 4 , Figure 5 The stability detection method embodiment of the recommendation system shown or Figure 6 An embodiment of a method for recommending advertising information is shown.

[0339] It is understandable that if Figure 4 , Figure 5 The stability detection method embodiment of the recommendation system shown or Figure 6 The contents of the advertising information recommendation method embodiment shown in the figure are applicable to the electronic device embodiment. The functions specifically implemented by the electronic device embodiment are similar to those in the example shown in the figure. Figure 4 , Figure 5 The stability detection method embodiment of the recommendation system shown or Figure 6 The advertising information recommendation method embodiment shown is the same as that shown in FIG. Figure 4 , Figure 5 The stability detection method embodiment of the recommendation system shown or Figure 6 The beneficial effects achieved by the illustrated advertising information recommendation method embodiment are also the same.

[0340] The electronic device of the embodiment of the present application may be a terminal device, a computer device or a server device.

[0341] For example, refer to Fig.12 , Fig.12 Schematic diagram of the structure of an electronic device provided in an embodiment of the present application. Taking the electronic device as a terminal device as an example, Fig.12 In the embodiment, the terminal device 1200 may include an RF (Radio Frequency) circuit 1210, a memory 1220 including one or more computer-readable storage media, an input unit 1230, a display unit 1240, a sensor 1250, an audio circuit 1260, a short-range wireless transmission module 1270, a processor 1280 including one or more processing cores, and a power supply 1290. Those skilled in the art will appreciate that Fig.12The device structure shown in the figure does not constitute a limitation on the terminal device, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0342] The RF circuit 1210 can be used for receiving and sending signals during information transmission or calls. In particular, after receiving the downlink information of the base station, it is handed over to one or more processors 1280 for processing; in addition, the data related to the uplink is sent to the base station. Generally, the RF circuit 1210 includes but is not limited to an antenna, at least one amplifier, a tuner, one or more oscillators, a subject identity module (SIM) card, a transceiver, a coupler, an LNA (Low Noise Amplifier), a duplexer, etc. In addition, the RF circuit 1210 can also communicate with the network and other devices through wireless communication. Wireless communication can use any communication standard or protocol, including but not limited to GSM (Global System of Mobile communication), GPRS (General Packet Radio Service), CDMA (Code Division Multiple Access), WCDMA (Wideband Code Division Multiple Access), LTE (Long Term Evolution), email, SMS (Short Messaging Service), etc.

[0343] The memory 1220 can be used to store software programs and modules (or units). The processor 1280 executes various functional applications and data processing by running the software programs and modules (or units) stored in the memory 1220. The memory 1220 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function), etc.; the data storage area may store data created according to the use of the terminal device 1200 (such as audio data, a phone book), etc. In addition, the memory 1220 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage devices. Accordingly, the memory 1220 may also include a memory controller to provide the processor 1280 and the input unit 1230 with access to the memory 1220. Although Fig.12 The RF circuit 1210 is shown, but it is understandable that it is not a necessary component of the terminal device 1200 and can be omitted as needed without changing the essence of the invention.

[0344] The input unit 1230 can be used to receive input digital or character information, and generate keyboard, mouse, joystick, optical or trackball signal input related to object setting and function control. Specifically, the input unit 1230 may include a touch-sensitive surface 1231 and other input devices 1232. The touch-sensitive surface 1231, also known as a touch display screen or a touch pad, can collect touch operations of an object on or near it (such as operations of an object using a finger, a stylus or any other suitable object or accessory on or near the touch-sensitive surface 1231), and drive the corresponding connection device according to a pre-set program. Optionally, the touch-sensitive surface 1231 may include a touch detection device and a touch controller. Among them, the touch detection device detects the touch position of the object, detects the signal brought by the touch operation, and transmits the signal to the touch controller; the touch controller receives the touch information from the touch detection device, converts it into touch point coordinates, and then sends it to the processor 1280, and can receive instructions sent by the processor 1280 and execute them. In addition, the touch-sensitive surface 1231 may be implemented in various types such as resistive, capacitive, infrared, and surface acoustic wave. In addition to the touch-sensitive surface 1231, the input unit 1230 may further include other input devices 1232. Specifically, the other input devices 1232 may include, but are not limited to, one or more of a physical keyboard, a function key (such as a volume control key, a switch key, etc.), a trackball, a mouse, a joystick, and the like.

[0345] The display unit 1240 can be used to display information input by an object or information provided to an object and to control various graphic object interfaces of the terminal device 1200. These graphic object interfaces can be composed of graphics, texts, icons, videos and any combination thereof. The display unit 1240 may include a display panel 1241. Optionally, the display panel 1241 may be configured in the form of an LCD (Liquid Crystal Display), an OLED (Organic Light-Emitting Diode), or the like. Further, the touch-sensitive surface 1231 may be covered on the display panel 1241. When the touch-sensitive surface 1231 detects a touch operation on or near it, it is transmitted to the processor 1280 to determine the type of touch event. Subsequently, the processor 1280 provides corresponding visual output on the display panel 1241 according to the type of touch event. Although in Fig.12 In the embodiment, the touch-sensitive surface 1231 and the display panel 1241 are implemented as two independent components to implement input and output functions, but in some embodiments, the touch-sensitive surface 1231 and the display panel 1241 can be integrated to implement input and output functions.

[0346] The terminal device 1200 may also include at least one sensor 1250, such as a light sensor, a motion sensor, and other sensors. Specifically, the light sensor may include an ambient light sensor and a proximity sensor, wherein the ambient light sensor may adjust the brightness of the display panel 1241 according to the brightness of the ambient light, and the proximity sensor may turn off the display panel 1241 or the backlight when the terminal device 1200 is moved to the ear. As a type of motion sensor, the gravity acceleration sensor can detect the magnitude of acceleration in each direction (generally three axes), and can detect the magnitude and direction of gravity when stationary, which can be used for applications that identify the posture of the mobile phone (such as horizontal and vertical screen switching, related games, magnetometer posture calibration), vibration recognition related functions (such as pedometer, tapping), etc.; as for other sensors such as gyroscopes, barometers, hygrometers, thermometers, infrared sensors, etc. that can also be configured in the terminal device 1200, they will not be repeated here.

[0347] The audio circuit 1260, the speaker 1261, and the microphone 1262 can provide an audio interface between the object and the terminal device 1200. The audio circuit 1260 can transmit the electrical signal converted from the received audio data to the speaker 1261, which is converted into a sound signal for output; on the other hand, the microphone 1262 converts the collected sound signal into an electrical signal, which is received by the audio circuit 1260 and converted into audio data, and then the audio data is output to the processor 1280 for processing, and then sent to another electronic device through the RF circuit 1210, or the audio data is output to the memory 1220 for further processing. The audio circuit 1260 may also include an earplug jack to provide communication between an external headset and the terminal device 1200.

[0348] The short-range wireless transmission module 1270 may be a WIFI (wireless fidelity) module, a Bluetooth module, an infrared module, etc. The terminal device 1200 may transmit information with wireless transmission modules provided on other devices through the short-range wireless transmission module 1270 .

[0349] The processor 1280 is the control center of the terminal device 1200. It uses various interfaces and lines to connect various parts of the entire device. By running or executing software programs or modules stored in the memory 1220, and calling data stored in the memory 1220, it executes various functions of the terminal device 1200 and processes data, thereby controlling the device as a whole. Optionally, the processor 1280 may include one or more processing cores; optionally, the processor 1280 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, object interface and application programs, etc., and the modem processor mainly processes wireless communications. It is understandable that the above-mentioned modem processor may not be integrated into the processor 1280.

[0350] The terminal device 1200 also includes a power supply 1290 (such as a battery) for supplying power to various components. Optionally, the power supply 1290 can be logically connected to the processor 1280 through a power management system, so as to manage charging, discharging, and power consumption through the power management system. The power supply 1290 can also include any components such as one or more DC or AC power supplies, recharging systems, power failure detection circuits, power converters or inverters, and power status indicators.

[0351] Although not shown, the terminal device 1200 may also include a camera, a Bluetooth module, etc., which will not be described in detail here.

[0352] For example, taking the electronic device as a server device as an example, referring to Fig.13 The server device 1300 may have relatively large differences due to different configurations or performances, and may include one or more central processing units 1310 (abbreviated as CPU, Central Processing Units) and memory 1360, and one or more storage media 1330 (such as one or more mass storage devices) storing application programs 1333 or data 1332. Among them, the memory 1360 and the storage medium 1330 can be temporary storage or permanent storage. The program stored in the storage medium 1330 may include one or more units or modules, and each unit or module may include a series of operating instructions in the server device 1300. Furthermore, the central processing unit 1310 can be configured to communicate with the storage medium 1330 and execute a series of operating instructions in the storage medium 1330 on the server device 1300.

[0353] The server device 1300 may further include one or more power supplies 1320 , one or more wired or wireless network interfaces 1340 , one or more input and output interfaces 1350 , and one or more operating systems 1331 .

[0354] The CPU 1310 in the server device 1300 may be used to execute the following Figure 4 , Figure 5 The stability detection method embodiment of the recommendation system shown or Figure 6 An embodiment of a method for recommending advertising information is shown.

[0355] The present application also discloses a computer-readable storage medium in which a program executable by a processor is stored. When the program executable by the processor is executed by the processor, it is used to implement the following Figure 4 , Figure 5 The stability detection method embodiment of the recommendation system shown or Figure 6 An embodiment of a method for recommending advertising information is shown.

[0356] Understandably, Figure 4 , Figure 5 The stability detection method embodiment of the recommendation system shown or Figure 6 The contents of the advertising information recommendation method embodiment shown in the figure are applicable to the computer-readable storage medium embodiment. The functions specifically implemented by the computer-readable storage medium embodiment are similar to those of the embodiment shown in the figure. Figure 4 , Figure 5 The stability detection method embodiment of the recommendation system shown or Figure 6 The advertising information recommendation method embodiment shown is the same as that of the embodiment shown in FIG. Figure 4 , Figure 5 The stability detection method embodiment of the recommendation system shown or Figure 6 The beneficial effects achieved by the illustrated advertising information recommendation method embodiment are also the same.

[0357] The embodiment of the present application further discloses a computer program product or a computer program, the computer program product or the computer program comprising computer instructions, the computer instructions being stored in the above-mentioned computer-readable storage medium; Fig.12 , Fig.13 The processor of the electronic device shown can read the computer instruction from the above-mentioned computer readable storage medium, and the processor executes the computer instruction so that the computer device executes Figure 4 , Figure 5 The stability detection method embodiment of the recommendation system shown or Figure 6 An embodiment of a method for recommending advertising information is shown.

[0358] Understandably, Figure 4 , Figure 5 The stability detection method embodiment of the recommendation system shown or Figure 6 The contents of the advertising information recommendation method embodiment shown in the figure are applicable to the computer program product or computer program embodiment, and the functions specifically implemented by the computer program product or computer program embodiment are the same as those in the embodiment of the present invention. Figure 4 , Figure 5 The stability detection method embodiment of the recommendation system shown or Figure 6 The advertising information recommendation method embodiment shown is the same as that of the embodiment shown in FIG. Figure 4 , Figure 5 The stability detection method embodiment of the recommendation system shown or Figure 6 The beneficial effects achieved by the illustrated advertising information recommendation method embodiment are also the same.

[0359] In some selectable embodiments, the function / operation mentioned in the block diagram may not occur in the order mentioned in the operation diagram. For example, depending on the function / operation involved, the two boxes shown in succession can actually be executed substantially simultaneously or the boxes can sometimes be executed in reverse order. In addition, the embodiment presented and described in the flow chart of the application is provided by way of example, for the purpose of providing a more comprehensive understanding of technology. The disclosed method is not limited to the operation and logic flow presented herein. Selectable embodiments are expected, wherein the order of various operations is changed and the sub-operation of a part described as a larger operation is performed independently.

[0360] In addition, although the present application is described in the context of functional modules, it should be understood that, unless otherwise specified, one or more of the functions and / or features can be integrated into a single physical device and / or software module, or one or more functions and / or features can be implemented in a separate physical device or software module. It is also understood that a detailed discussion of the actual implementation of each module is unnecessary for understanding the present application. More specifically, in view of the properties, functions, and internal relationships of the various functional modules in the device disclosed herein, the actual implementation of the module will be understood within the conventional techniques of the engineer. Therefore, those skilled in the art can implement the present application set forth in the claims without excessive experimentation using ordinary techniques. It is also understood that the specific concepts disclosed are merely illustrative and are not intended to limit the scope of the present application, which is determined by the full scope of the attached claims and their equivalents.

[0361] If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, and other media that can store program codes.

[0362] The logic and / or steps represented in the flowchart or otherwise described herein, for example, may be considered as an ordered list of executable instructions for implementing logical functions, and may be embodied in any computer-readable storage medium for use by an instruction execution system, apparatus, or device (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, apparatus, or device and execute instructions), or in conjunction with such instruction execution system, apparatus, or device. For purposes of this specification, a "computer-readable storage medium" may be any device that can contain, store, communicate, propagate, or transmit a program for use by an instruction execution system, apparatus, or device, or in conjunction with such instruction execution system, apparatus, or device.

[0363] It should be understood that the various parts of the present application can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0364] In the above description of this specification, the description with reference to the terms "one embodiment / example", "another embodiment / example" or "certain embodiments / examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0365] Although the embodiments of the present application have been shown and described, those skilled in the art will appreciate that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present application, and that the scope of the present application is defined by the claims and their equivalents.

[0366] The above is a specific description of the preferred implementation of the present application, but the present application is not limited to the embodiments. Technical personnel familiar with the field can make various equivalent modifications or substitutions without violating the spirit of the present application. These equivalent modifications or substitutions are all included in the scope defined by the claims of the present application.

Claims

1. A method for recommending advertising information, characterized in that: The method comprises: Acquire an advertisement information queue; the advertisement information queue includes a plurality of candidate advertisement information; Dividing the advertisement information queue to obtain at least two advertisement information sets; wherein the advertisement information set includes at least two of the candidate advertisement information; Predicting the interest tendency of the target object to each of the candidate advertising information through the recommendation system, and obtaining a first estimation result corresponding to each of the candidate advertising information; Determine a first estimation deviation between first estimation results corresponding to the candidate advertisement information in each of the advertisement information sets; Obtaining a stability detection result of the recommendation system according to the first estimated deviation corresponding to each of the advertising information sets; When the stability detection result of the recommendation system is passed, advertising information is recommended based on the recommendation system.

2. The method for recommending advertising information according to claim 1, characterized in that: The dividing the advertisement information queue comprises at least one of the following steps: Extracting fingerprint information corresponding to each of the candidate advertisement information, and dividing the advertisement information queue according to the fingerprint information; Alternatively, extracting first characteristic information corresponding to each of the candidate advertisement information, and dividing the advertisement information queue according to the first characteristic information; Alternatively, obtaining industry category information corresponding to each of the candidate advertisement information, and dividing the advertisement information queue according to the industry category information; Alternatively, standardized product unit information corresponding to each of the candidate advertisement information is obtained, and the advertisement information queue is divided according to the standardized product unit information.

3. The method for recommending advertising information according to claim 2, characterized in that: The fingerprint information includes an information summary value; the extracting the fingerprint information corresponding to each of the candidate advertisement information and dividing the advertisement information queue according to the fingerprint information includes: Extracting information summary values ​​corresponding to each of the candidate advertisement information; The information summary values ​​are compared, and the candidate advertisement information with the same corresponding information summary values ​​are grouped into the same advertisement information set.

4. The method for recommending advertising information according to claim 2, characterized in that: The extracting first characteristic information corresponding to each of the candidate advertisement information, and dividing the advertisement information queue according to the first characteristic information, includes: Extracting first feature information corresponding to each of the candidate advertisement information; Taking the first characteristic information corresponding to any of the candidate advertisement information as the initial cluster center, establishing a cluster; Calculating the third similarity between each feature information to be clustered and the initial cluster center; the feature information to be clustered is the first feature information that has not been added to the clustering cluster; When the third similarity between the feature information to be clustered and the initial cluster center is greater than or equal to the second threshold, the feature information to be clustered is added to the cluster corresponding to the initial cluster center, and the position of the initial cluster center is updated; or, when the third similarity between the feature information to be clustered and any of the initial cluster centers is less than the second threshold, the feature information to be clustered is used as the initial cluster center, and a new cluster is additionally established; The candidate advertisement information corresponding to the first characteristic information belonging to the same cluster is divided into the same advertisement information set.

5. The method for recommending advertising information according to claim 1, characterized in that: The first estimation result includes the target object's click rate estimation result or conversion rate estimation result for the candidate advertisement information; the first estimation result corresponding to each candidate advertisement information is obtained by predicting the target object's interest tendency for each candidate advertisement information through the recommendation system, including: Acquire first associated information of the target object; The recommendation system predicts, based on the candidate advertisement information and the first associated information, a click rate estimate result or a conversion rate estimate result of the target object for the candidate advertisement information.

6. The method for recommending advertising information according to claim 1, characterized in that: The determining of a first estimation deviation between first estimation results corresponding to the candidate advertisement information in each advertisement information set includes: Determine an estimated mean and an estimated standard deviation of the first estimated results corresponding to each of the candidate advertisement information in the advertisement information set; A first estimated deviation corresponding to the advertisement information set is obtained according to a quotient of the estimated standard deviation and the estimated mean.

7. The method for recommending advertising information according to claim 1, characterized in that: The step of obtaining the stability detection result of the recommendation system according to the first estimated deviation corresponding to each of the advertisement information sets includes: Counting the number of the candidate advertisement information in each of the advertisement information sets; Determining, according to the number, a weighted weight corresponding to each of the advertising information sets; According to the weighted weights, weighted summation is performed on the first estimated deviations corresponding to the respective advertising information sets to obtain a third estimated deviation; determining a stability detection result of the recommendation system according to the third estimated deviation; The weighted weight corresponding to the advertisement information set is positively correlated with the number of candidate advertisement information in the advertisement information set.

8. The method for recommending advertising information according to claim 7, characterized in that: The step of determining a stability detection result of the recommendation system according to the third estimated deviation includes: Obtaining third estimated deviations corresponding to the plurality of target objects; Performing weighted summation on the third estimated deviations corresponding to the target objects to obtain a fourth estimated deviation; A stability detection result of the recommendation system is determined according to the fourth estimated deviation.

9. The method for recommending advertising information according to claim 8, characterized in that: The step of determining a stability detection result of the recommendation system according to the fourth estimated deviation includes: comparing the fourth estimated deviation with a preset deviation threshold; If the fourth estimated deviation is greater than or equal to the deviation threshold, it is determined that the stability test result of the recommendation system is failed; or, if the fourth estimated deviation is less than the deviation threshold, it is determined that the stability test result of the recommendation system is passed.

10. The method for recommending advertising information according to any one of claims 1 to 9, characterized in that: The recommendation system is configured with a recommendation model; the recommendation system predicts the interest tendency of the target object for each of the candidate advertisement information to obtain the first estimated result corresponding to each of the candidate advertisement information, including: Acquire first associated information of the target object; extracting, by the recommendation model, first feature information corresponding to the candidate advertisement information and second feature information corresponding to the first associated information; A first estimation result is obtained by predicting the recommendation model according to the first feature information and the second feature information.

11. The method for recommending advertising information according to claim 10, characterized in that: The recommendation model is trained by the following steps: Acquire sample advertising information, second associated information of the sample object, and label information; wherein the label information is used to represent the actual result of the sample object's interest tendency in the sample advertising information; extracting the third characteristic information of the sample advertisement information and the fourth characteristic information of the second associated information through a recommendation model; According to the third feature information and the fourth feature information, predicting the interest tendency of the sample object to the sample advertising information by the recommendation model to obtain a fourth estimation result; Determining a loss value of training according to the fourth estimation result and the label information; The parameters in the recommendation model are updated according to the loss value to obtain a trained recommendation model.

12. The method for recommending advertising information according to any one of claims 1 to 9, characterized in that: The method further comprises: If the stability test result of the recommendation system fails, an alarm message is generated; The warning information is displayed, or the warning information is sent to an associated object of the recommendation system.

13. A device for recommending advertising information, characterized in that: The device comprises: An acquisition unit, used for acquiring an advertisement information queue; the advertisement information queue includes a plurality of candidate advertisement information; A division unit, configured to divide the advertisement information queue to obtain at least two advertisement information sets; wherein the advertisement information set includes at least two of the candidate advertisement information; An estimation unit, used for predicting the interest tendency of the target object to each of the candidate advertisement information through a recommendation system, and obtaining a first estimation result corresponding to each of the candidate advertisement information; a comparing unit, configured to determine a first estimation deviation between first estimation results corresponding to the candidate advertisement information in each of the advertisement information sets; A processing unit, configured to obtain a stability detection result of the recommendation system according to the first estimated deviation corresponding to each of the advertising information sets; The recommendation unit is used to recommend advertising information based on the recommendation system when the stability detection result of the recommendation system is passed.

14. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the method for recommending advertising information according to any one of claims 1 to 12 is implemented.

15. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method for recommending advertising information according to any one of claims 1 to 12 is implemented.

16. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the method for recommending advertising information according to any one of claims 1 to 12 is implemented.

Images