Method and apparatus for determining user feedback on information, device, medium, and product

Abstract

Embodiments of the present disclosure relate to a method and apparatus for determining user feedback on information, a device, a medium, and a program product. The method comprises: on the basis of a first user feature of a user and a first contextual feature, determining a first feedback probability of the user for information. The method also comprises: on the basis of a second user feature of the user, a second contextual feature, and a first information feature of first information, determining a second feedback probability of the user for the first information. In addition, the method further comprises: on the basis of the first feedback probability, adjusting the second feedback probability of the user for the first information. According to the embodiments of the present disclosure, by implementing a user-level debiased feedback method, more accurate user feedback on information can be realized. Additionally, the method takes into account the personalized requirements and preferences of users, improving the information placement effect, and eliminates various influencing factors that may distort the determination of user feedback, improving the user experience.

Description

[0001] The method, device, equipment, medium and product technology field for determining user feedback on information The present disclosure generally relates to the field of computers, and more particularly, to a method, device, equipment, medium and program product for determining user feedback on information. Background Information delivery refers to the process of delivering information content or information to target audiences through specific channels or media to achieve the purpose of promotion. Traditional information delivery mainly relies on traditional media such as television, radio, newspapers, etc., and the audience positioning is relatively broad. With the rapid development of the Internet and the increasing intelligence of mobile devices, people's way of obtaining and consuming information has changed, providing new channels and opportunities for information delivery. The interactivity of Internet information is an important difference from traditional information. Users are no longer passive recipients, but can actively participate and feedback. Through clicking, commenting, sharing, etc., users can express their views and feelings on information in real time, which provides valuable feedback data for information deliverers. Online feedback from users has become an important part of data-driven decision-making, helping information deliverers better understand user needs and adjust the content and delivery methods of information. Invention The embodiments of the present disclosure provide a method, device, electronic equipment and product for determining user feedback on information. According to the first aspect of the disclosure, a method for determining user feedback on information is provided. The method includes determining a first feedback probability of a user on information based on first user features and first context features of the user. The method also includes determining a second feedback probability of the user on the first information based on second user features, second context features of the user, and first information features of the first information. The method further includes adjusting the second feedback probability of the user on the first information based on the first feedback probability. In the second aspect of the disclosure, a device for determining user feedback on information is provided. The device includes a first feedback probability determination module configured to determine a first feedback probability of a user on information based on first user features and first context features of the user. The device also includes a second feedback probability determination module configured to determine a second feedback probability of the user on the first information based on second user features, second context features of the user, and first information features of the first information. The device further includes a second feedback probability adjustment module configured to adjust the second feedback probability of the user on the first information based on the first feedback probability. In the third aspect of the present disclosure, an electronic device is provided. The electronic device includes a processor and a memory coupled to the processor, the memory having stored therein instructions that, when executed by the processor, cause the electronic device to perform the method according to the first aspect. In the fourth aspect of the present disclosure, a computer-readable storage medium is provided.The computer readable storage medium stores computer executable instructions, wherein the computer executable instructions are executed by the processor to implement the method according to the first aspect. In a fifth aspect of the present disclosure, a computer program product is provided, which stores computer executable instructions including the computer executable instructions are executed by the processor to implement the method of the first aspect. The summary is intended to introduce some selected concepts of the concept in a simplified form, which will be further described in the detailed description below. The summary does not intend to identify key or critical features of the claimed subject matter or to delineate the scope of the claimed subject matter. The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent as various embodiments thereof are described in conjunction with the accompanying drawings, in which like reference numerals refer to like elements, and in which: FIG. 1 shows a schematic diagram of an example environment in which some embodiments of the present disclosure can be implemented; FIG. 2 shows a flowchart of a method for determining feedback of a user for information according to some embodiments of the present disclosure; FIG. 3 shows a schematic diagram for determining ranking of information according to feedback of a user for the information according to some embodiments of the present disclosure; FIG. 4 shows a schematic diagram of an architecture for training a model for determining feedback of a user for information according to some embodiments of the present disclosure; FIG. 5 shows a schematic diagram of applying a model for determining feedback of a user for information according to some embodiments of the present disclosure; FIG. 6 shows a block diagram of an apparatus for determining feedback of a user for information according to some embodiments of the present disclosure; and FIG. 7 shows a block diagram of an electronic device according to some embodiments of the present disclosure. In all the drawings, the same or similar reference numerals refer to the same or similar elements. It can be understood that the data involved in the technical solution of the present disclosure (including but not limited to the data itself, the acquisition or use of the data) should comply with the requirements of the relevant laws and regulations and relevant provisions. It can be understood that before using the technical solution disclosed in the embodiments of the present disclosure, the type of personal information involved in the present disclosure, the scope of use, the scene of use, etc. should be informed to the user and the authorization of the user should be obtained in a proper manner according to the relevant laws and regulations. For example, when receiving the active request of the user, the prompt information is sent to the user to clearly prompt the user that the operation requested to be executed will need to obtain and use the personal information of the user. Thus, the user can voluntarily choose whether to provide the personal information to the electronic device, application program, server or storage medium, etc. software or hardware that executes the operation of the technical solution of the present disclosure according to the prompt information. As an optional but non-limiting implementation manner, in response to receiving the active request of the user, the manner of sending the prompt information to the user may, for example, be the manner of pop-up window, and the prompt information may, for example, be presented in the form of text in the pop-up window.In addition, the pop-up window can also carry a selection control for the user to select "agree" or "disagree" to provide personal information to the electronic device. It can be understood that the above notification and obtaining user authorization process is only illustrative, and does not limit the implementation of the present disclosure, and other ways that meet the relevant laws and regulations can also be applied to the implementation of the present disclosure. Embodiments of the present disclosure will be described in more detail below with reference to the drawings. Although some embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure can be implemented in various forms, and should not be interpreted as being limited to the embodiments set forth herein, on the contrary, these embodiments are provided to more thoroughly and completely understand the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes, and are not intended to limit the scope of protection of the present disclosure. In the description of embodiments of the present disclosure, the term "comprising" and similar terms should be understood as open-ended inclusion, i.e., "including but not limited to". The term "based on" should be understood as "at least partially based on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The terms "first", "second", and the like can refer to different or the same objects unless otherwise specified. The following can also include other explicit and implicit definitions. Information delivery refers to delivering information content or information to target audiences through specific channels or media. With the popularity of the Internet and the intelligentization of mobile devices, the user base of digital media such as search engines, social media and video sharing platforms is expanding, which provides new channels and opportunities for information delivery. In related technologies, the delivery platform usually determines the ranking of the delivered information on the delivery platform according to the feedback of the users on the delivered information. However, in actual situations, there is a large deviation in the feedback of users on information. Some users are used to giving negative feedback on information, while other users do not like or are indifferent to the information, but also do not give feedback on the information. The negative feedback data of these users who give negative feedback occupies a high proportion in the whole data, which leads the platform to pay too much attention to these negative feedback when encountering such a situation, and may give the information an unfair negative evaluation. In addition, if the information is only devalued according to the data of the negative feedback of some users who are used to giving negative feedback, it may affect the experience of other potential audience users of the information.According to embodiments of the present disclosure, in addition to evaluating the feedback probability of a user for a specific information based on some basic features of the user such as user behavior habits and context features such as the user's interactive environment background and information features such as information classification, the feedback probability of the user for the entire information system is preliminarily evaluated based on some filtered basic features of the user such as user habits and filtered context features such as the user's interactive environment background. Based on the feedback probability of the user for the entire information system, the feedback probability of the user for a specific information can be accurately evaluated. By implementing a user-level debiasing strategy, more accurate information quality evaluation can be achieved. This strategy not only considers the personalized needs and preferences of users, but also eliminates various factors that may lead to distorted judgments. At the same time, based on this debiased evaluation, the information quality evaluation feedback mechanism is also optimized, more accurately reflecting the actual quality of the information. Such improvements not only enhance the user experience, enabling users to access more relevant and high-quality information content, but also improve the effectiveness of information delivery, thereby promoting the healthy development of the entire information ecosystem. It can be understood that the user mentioned here can be a vague user or a user in a user group without referring to a specific user identity. FIG. 1 shows a schematic diagram of an example environment 100 in which some embodiments of the present disclosure can be implemented. As shown in FIG. 1, the example environment 100 at least includes a content service 110, an information service 120, a user information feedback model service 122 included in the information service 120, and a hybrid ranking service 130. Among them, the content service 110 involves the creation, delivery, and a series of services related to content. For example, taking a video platform as an example, the content service 110 can be a service involving the delivery, pushing, and promotion of user short videos. The information service 120 is a service for providing information delivery for content creation that hopes to further expand its influence, and these information can be information content in various forms such as videos, texts, and pictures. Combined with the content service 110 and the information service 120, the specific ranking of a specific information being delivered can be determined in the hybrid ranking service 130. Continuing to refer to FIG. 1, in order to more accurately or accurately deliver information and optimize the effectiveness of information delivery, in the information service 120, the user information feedback model service 122 can be used to predict the feedback of the user for the information, so as to adjust the ranking of the information being delivered when displayed on the user side. In some embodiments, the user information feedback model 122 can have a bias correction mechanism and be able to predict the feedback attitude of a user without influence factors for a specific information.In some embodiments, the feedback attitude of a user refers to whether the user likes or dislikes the information. For example, by applying a user information feedback model service with a rectification function, feedbacks of all users for a certain information can be collected, and if the feedbacks of users for a certain information are mostly dislikes, the delivery weight of the information can be reduced. If the feedbacks of users for a certain information are mostly likes, the delivery weight of the information can be increased. In some embodiments, the user information feedback model service 122 can be a feedback service model with a double-tower structure. In some embodiments, the feedback of a user also includes feedback operations such as surveys, reports, etc. It can be understood that the content service 110 and the information service 120 (including the user information feedback model service 122) or the hybrid ranking service 130 can operate on different servers. The servers can be cloud servers or local servers, and the servers can be computing systems, single servers, distributed servers, etc. It should be understood that the architecture and functions in the example environment 100 are described for illustrative purposes only, and do not imply any limitation on the scope of the present disclosure. Embodiments of the present disclosure can also be applied to other environments with different structures and / or functions. The processes according to embodiments of the present disclosure will be described in detail below with reference to FIGS. 2-7. For ease of understanding, the specific data mentioned in the following description are exemplary and are not used to limit the protection scope of the present disclosure. It can be understood that the embodiments described below can also include additional actions not shown or can omit the actions shown, and the scope of the present disclosure is not limited in this respect. FIG. 2 shows a flowchart of a method 200 for determining feedback of a user for information according to some embodiments of the present disclosure. The user information feedback model service 122 shown in FIG. 1 is the execution subject of the method 200. At block 202, a first feedback probability of a user for information is determined based on first user features of the user and first context features. In some embodiments, the first user features can be filtered features of the user related to past interactions with content, or filtered demographic features of the user or content-side features, etc. In some embodiments, the first context features can be aggregated features of a user session filtered or recent content consumption behavior features of the user filtered. In some embodiments, the feedback probability of a user for information can be determined by a first prediction model capable of predicting feedback probabilities of a user for all information. In some embodiments, the first feedback probability of a user for all information refers to a probability of the user providing feedback for the information. It can be understood that the user mentioned here refers to an ambiguous user or a user in a user group without referring to a specific user identity.At block 204, a second feedback probability of the user for the first information is determined based on second user features of the user, second context features, first information features of the first information. In some embodiments, the second user features can be features related to past interactions of the user with content, or demographic features of the user, or content-side features, etc. In some embodiments, the second context features can be user session aggregation features or recent content consumption behavior features of the user. In some embodiments, the first information features can be aggregation features or specific attribute features at the information creative level, which can include information such as the classification of the information, etc. In some embodiments, the first user features are selected from the second user features, and the first context features are selected from the second context features. In some embodiments, the feedback probability of the user for a specific information can be determined by a second prediction model capable of predicting the feedback probability of the user for the specific information. At block 206, the second feedback probability of the user for the first information is adjusted based on the first feedback probability. In some embodiments, the feedback probability of the user for a specific information can be adjusted by a probability adjustment model, and the feedback probability adjusted by the probability adjustment model can more accurately reflect the quality of the information. In some embodiments, the feedback attitude of the user for a specific information can be accurately determined based on the feedback probability of the user for the specific information and the feedback probability of the user for all information. In this embodiment, in addition to evaluating the feedback probability of the user for a specific information based on user features of the user such as user behavior habits and context features such as the interactive environment background of the user, and information features of the specific information such as information classification, the feedback probability of the user for the entire information system can also be preliminarily evaluated based on selected basic features of the user such as user habits and selected context features such as the interactive environment background of the user. Based on the feedback probability of the user for the entire information system, the feedback probability of the user for a specific information can be accurately evaluated. By implementing a user-level debiasing strategy, more accurate information quality evaluation can be achieved. This strategy not only considers the personalized needs and preferences of the user, but also eliminates various factors that may cause distorted judgments. At the same time, based on this debiased evaluation, the information quality evaluation feedback mechanism is also optimized, and the actual quality of the information is more accurately reflected. Such improvements not only enhance the user experience, enabling the user to access more relevant and high-quality information content, but also improve the effectiveness of the information, thereby promoting the healthy development of the entire information ecosystem. FIG. 3 shows a schematic diagram of determining information ranking 300 according to feedback of the user for information, according to some embodiments of the present disclosure.Referring to FIG. 3, the recall 321, the coarse ranking 322, and the fine ranking 323 in the information service 320 can be some specific steps in the information service 120 in FIG. 1. The user information feedback model process 310 in FIG. 3 can be the process of the user information feedback model service 122 in FIG. 1. The content 330 in FIG. 3 can be the content service 110 in FIG. 1. The hybrid ranking 340 in FIG. 3 can be some steps of the hybrid ranking service 130 in FIG. 1. Referring to FIG. 3, in combination with the content 330 and the information service 320, the hybrid ranking 340 of a certain information in a content set and an information set can be determined. Specifically, in the information service 320, a number of candidate information in the order of millions that can be matched with a user can be selected from an information library based on the user features and the context features of the user in the recall 321. For example, if a user's interest is fitness or the user has interacted with the content related to health and fitness recently, a number of information related to fitness and bodybuilding can be recalled from the information library for the user. It can be understood that the user mentioned here is a fuzzy user or a user in a user group without referring to a specific user identity. Referring to FIG. 3, after a number of information are recalled from the information library, in order to filter out the information with high relevance to the user, the information can be roughly ranked first, and then the information that has been coarsely ranked can be further ranked more finely according to more personalized factors. In the coarse ranking 322, a number of candidate information can be filtered out from the number of candidate information in the order of millions that are recalled for the user according to the relevant characteristics or preference rules of the user. For example, if the user prefers fitness equipment in the content related to fitness and bodybuilding, 300 candidate information can be filtered out and roughly ranked according to the fitness equipment. Then, in the fine ranking 323, the 300 candidate information can be ranked more finely by considering more dimensional features and personalized factors. As shown in FIG. 3, the further fine ranking in the fine ranking 323 can be implemented according to the user information feedback model process 310. Specifically, the above-mentioned user and a certain information B are taken as an example.In the user information feedback model flow 310, by inputting the user features 311 related to the user, the context features 312, the information features 313 of the information B, and the posterior feedback 314 of the user to the information B into the user feedback model 315, and then via the information model prediction 316, the predicted score 317 of the information B by the information feedback model 316 can be obtained. In some embodiments, the predicted score 317 of the information feedback model 316 can be used as the adjusted feedback probability of the user to the information B. It can be understood that the user mentioned here is a vague user or a user in a user group, not a specific user identity. Referring to FIG. 3, after obtaining the predicted score 317 of the information B by the information feedback model 316, the implicit quality feedback 318 of the user to the information B can be obtained. Then, multiple implicit quality feedbacks 318 of multiple users to the information B can be collected to determine the comprehensive indicator 319 of the information B. Specifically, referring to formula (1).

[0002] Sorted_eCPM = eCPM + £ Hidden_qualityi (1) where Sorted_eCPM is the comprehensive index, eCPM can be an evaluation index of a product of information per thousand times of delivery, and Hidden_quality is the implicit quality feedback. In some embodiments, eCPM can be an index of a product of any type of information per thousand times of delivery, and for a cost per mille (CPM) index, eCPM can be a competitive index compared to CPM, because both involve evaluation of the number of information displays. For a cost per click (CPC) and a cost per view (CPV), eCPM is a competitive index eCTR * 1000, i.e., a comprehensive evaluation can be made by considering the click rate of a user on information and the number of displays after the information is delivered. For a cost per action (CPA) and an optimized CPM (oCPM), eCPM can also be a competitive index. With reference to FIG. 3, in the process of fine ranking 323, the user information feedback model process 310 is performed on the screened candidate information, for example, 300 candidate information, so that the ranking of the 300 candidate information can be determined according to the determined comprehensive index 319 of each information. Then, in combination with the content side 330, a comprehensive ranking result about the information set and the content can be obtained in the mixed ranking 340, and the related content or information can be pushed to the user according to the comprehensive ranking result. Alternatively, in the process of fine ranking 323, the comprehensive index 319 can also be obtained according to the following formula. Specifically, the normfinal can be obtained according to formula (2). coef * shk coef * uav * (org score + org offset ) * thd (2) where normcoef is a fixed value or function for adjusting the weight of the index, shk coefFor scaling up or down the metrics based on certain conditions, the uav is related to the user's activity or engagement, the orgscore is a score derived based on the user feedback model 315, the orgoffs et is used to adjust the orgscore to calibrate or normalize it to get the predicted score 317, the thd is a boolean value that is 1 (or true) when the orgscore exceeds a certain threshold, otherwise it is 0 (or false). FIG. 4 illustrates a schematic diagram of a model 400 for training to determine user feedback on information according to some embodiments of the present disclosure. Referring to FIG. 4, in some embodiments, the first feedback probability model 413 and the second feedback probability model 424 are trained separately or in stages. In some embodiments, the training user features 411 and the training context features 412 are input to the first feedback probability model 413, which can result in a training user feedback probability on information 414. In some embodiments, a loss is calculated at 415, which can adjust the parameters of the first feedback probability model 413 based on the loss between the resulting training user feedback probability on information 414 and the true feedback probability. In some embodiments, the loss function can be calculated using a backpropagation algorithm to compute the gradient of the loss function with respect to the parameters of the first feedback probability model, and an optimizer can be used to update the parameters. In some embodiments, the parameter update can be performed using Follow-the-Regularized-Leader (FTRL) or LossAwareFTRL optimizer with group sparsity. With continued reference to FIG. 4, in some embodiments, the training user features 421, the training context features 422, and the training information features 423 are input to the second feedback probability model architecture 424, which can result in a raw predicted value 425 (which can be an intermediate quantity) from the second feedback probability model 424 on the input, resulting in a training user feedback probability on specific information 426. In some embodiments, the second feedback probability model 424 can be adjusted at 427 based on the loss between the training user feedback probability on specific information 426 and the true feedback probability. In some embodiments, the parameters of the first feedback probability model 413 are kept constant during the process of computing the loss to adjust the parameters of the second feedback probability model 424. In some embodiments, the parameters of the second feedback probability model 424 can be updated using a backpropagation algorithm. In some embodiments, a stop gradient (stop_gradient) through 451 can be used on the output of the first feedback probability model 413, such that when the loss 427 is computed and backpropagated, the gradient only propagates in the second feedback probability model 424 and does not affect the parameters of the first feedback probability model 413.In some embodiments, the training user features 411 are filtered from the training user features 421, and the training context features 412 are filtered from the training context features 422. Continuing to refer to FIG. 4, the probability adjustment model can be trained with the training adjusted feedback probabilities 432 and the training score output 441. In some embodiments, the compensation bias scalar 431 is also adjusted when adjusting the parameters of the probability adjustment model. In some embodiments, the compensation bias scalar 431 can be used to adjust the training output 441. In some embodiments, the parameters of the probability adjustment model can be updated using a backpropagation algorithm. In some embodiments, a stop gradient (stop_gradient) through 452 can be used on the output of the second feedback probability model 424, such that when computing the loss 433 and backpropagating, the gradient only propagates in the probability adjustment model and does not affect the parameters of the second feedback probability model 424. With this stop gradient training method, the model training can be made more flexible, allowing different parts of the model to be optimized in stages. Continuing to refer to FIG. 4, in some embodiments, the first feedback probability model 413 can be composed of two layers of neural networks, the first layer can have 256 neurons, and the second layer can have 128 neurons. In some embodiments, the first feedback probability model 413 can be composed of three layers of neural networks, the first layer can have 2048 neurons, the second layer can have 512 neurons, and the third layer can have 128 neurons. In some embodiments, the neural dimensions of the second feedback probability model 424 can be composed of 5 layers of neural networks, the first layer can have 2048 neurons, the second layer can have 1024 neurons, the third layer can have 1024 neurons, the fourth layer can have 512 neurons, and the fifth layer can have 128 neurons. Each layer reduces the number of neurons, which helps the model gradually abstract higher-level features from the raw data, while also reducing the number of parameters and the complexity of the computation. The following will describe the flow of the model of determining user feedback for information according to some embodiments of the present disclosure with reference to FIG. 5. FIG. 5 illustrates a schematic diagram of a flow 500 of a model of determining user feedback for information according to some embodiments of the present disclosure. Referring to FIG. 5, the model architecture shown in FIG. 5 is a composite architecture of the user feedback model 315 and the information model prediction 316 shown in FIG. 3. The user feedback model 315 shown in FIG. 3 includes at least the architecture 510 of the first feedback probability model and the architecture 520 of the second feedback probability model shown in FIG. 5. The information model prediction 316 shown in FIG. 3 is as shown in the architecture 530.In some embodiments, the model architecture 500 shown in FIG. 5 is a two-tower model structure. With reference to FIG. 5, in some embodiments, the first feedback probability model 513 can be a single tower neural network structure model based on a Factorization-Machine (FM). In some embodiments, applying the user features 511 and the context features 512 to the first feedback probability model 513 can obtain the feedback probability 514o of the user on the information. With reference to FIG. 5, in some embodiments, the user features 511 at least include a plurality of user demographic features, a plurality of user content aspect aggregation features (such as the categories of videos watched by the user, the tags and the number of sharing or forwarding, etc.), and a plurality of user information aspect aggregation features (such as the types of information clicked by the user, the completion rate of videos watched by the user, etc.). In some embodiments, the context features 512 at least include a plurality of user session sequences and aggregation features, a plurality of user recent content and information attribute features (such as the types of information recently clicked by the user, etc.), and a plurality of user recent content aggregation features (such as the number of videos recently collected or liked by the user, the change of the user's preference for a certain type of content, etc.). In some embodiments, the user features 511 are filtered from the user features 521, and the context features 512 are filtered from the context features 522. In this way, the relevance of predicting the feedback probability of the user on all information and predicting the feedback probability of the user on specific information can be ensured. As shown in FIG. 5, taking a user as an example, applying the user features 511 and the context features 512 of the user to the first feedback probability model 513 can obtain the probability 514 of the user giving feedback on all information. For example, it can be predicted that the posterior feedback probability of the user on the information is 50% (only for illustration). In some embodiments, these feedbacks refer to the user's like, dislike, report, or investigation, etc. on all information. In some embodiments, the user features contain feature information such as the user's purchase history. In some embodiments, the context features contain information such as the content (video or text, etc.) being browsed by the user. It can be understood that the user mentioned here is not a specific user identity, but a fuzzy user or a user in a user group. With reference to FIG. 5, in some embodiments, applying the user features 521, the context features 522, and the information features 523 to the second feedback probability model architecture 524 can obtain the feedback probability 525 of the user on specific information. The information features 523 include information such as the classification of the information.In some embodiments, the information features 523 at least include aggregated features of information creative level (e.g., the number of times the information creative is shared or forwarded, the view completion rate of the information creative, etc.) and information creative attribute features (e.g., the product display manner in the information creative, the type of the information creative, etc.). Continuing with the example of the user and information B, applying the user features 521, the context features 522 of the user, and the information features 523 of information B to the second feedback probability model 524 can obtain the feedback probability 525 of the user to information B. For example, information B involves the attribute of adult education that the user does not like, and it is predicted that the probability of the user’s feedback (e.g., disliking or reporting, etc.) to information B is 80% (only for example, the probability can also be other forms of numerical values). Continuing to refer to FIG. 5, in order to determine the accurate feedback probability of the user to information B, the information compensation bias scalar 531 and the probability adjustment model can be used to adjust the unbiased feedback probability of the user to information B. In combination with FIG. 3, the probability adjustment model is the information prediction model 316o shown in FIG. 3. In some embodiments, the information compensation bias scalar 531 is used to correct the score output 541. In some embodiments, the feedback probability 532 adjusted by the probability adjustment model is the feedback probability of the user to information B on the basis of the same feedback probability of the user to all information. Then, according to the feedback probability of the user to the adjusted information B, a posteriori prediction score output 541 to information B can be output. In some embodiments, the value of the adjusted feedback probability of the user to information B can be determined or converted from the feedback probability of the user to all information output by the first probability detection model 513 via the probability adjustment model, so that the value of the adjusted feedback probability of the user to information B is close to a normal distribution. By implementing the user-level unbiased feedback method, it can ensure that the feedback of the user to the information is more objective and accurate. This method not only considers the personalized needs and preferences of the user, thereby optimizing the effect of information delivery, but also eliminates various bias factors that may cause distortion of the user feedback judgment, thereby improving the user experience. FIG. 6 shows a block diagram of an apparatus 600 for determining the feedback of a user to information according to some embodiments of the present disclosure. As shown in FIG. 6, the apparatus 600 includes a first feedback probability determination module 602 configured to determine a first feedback probability of a user to information based on first user features and first context features of the user. The apparatus 600 also includes a second feedback probability determination module 604 configured to determine a second feedback probability of the user to the first information based on second user features, second context features, and first information features of the first information of the user.Further, the apparatus 600 further includes a second feedback probability adjustment module 606 configured to adjust a second feedback probability of the user for the first information based on the first feedback probability. In some embodiments, the first feedback probability determination module 602 includes a first feedback probability module configured to determine, by a first feedback probability model, the first feedback probability of the user for the information based on first user features of the user and first context features. In some embodiments, the second feedback probability determination module 604 includes a second feedback probability module configured to determine, by a second feedback probability model, the second feedback probability of the user for the first information based on second user features of the user, second context features, and first information features of the first information. In some embodiments, the second feedback probability adjustment module 606 includes a probability adjustment module configured to determine, by a probability adjustment model, an adjusted second feedback probability based on the first feedback probability, the second feedback probability, and a compensation scalar, wherein the compensation scalar is used to correct the adjusted second feedback probability. In some embodiments, the second feedback probability adjustment module 606 further includes a first feedback probability model training module configured to determine, by the first feedback probability model, a first training feedback probability based on first training user features and first training context features, and a first feedback probability model adjustment module configured to adjust the first feedback probability model based on a loss of the first training feedback probability from a true feedback probability. In some embodiments, the second feedback probability adjustment module 606 includes a second feedback probability model training module configured to determine, by the second feedback probability model, a second training feedback probability based on second training user features, second training context features, and first training information features, and a second feedback probability model adjustment module configured to adjust the second feedback probability model based on a loss of the second training feedback probability from a true training feedback probability, wherein the first feedback probability model is not required to be adjusted inversely at the same time as the second feedback probability model is adjusted. In some embodiments, the second feedback probability adjustment module 606 further includes a probability adjustment model training module configured to determine, by the probability adjustment model and the compensation scalar, a training adjusted second feedback probability based on a first training feedback probability, a second training feedback probability, and a training compensation scalar, and a probability adjustment model adjustment module configured to adjust the probability adjustment model and the compensation scalar based on the training adjusted second feedback probability, wherein the second feedback probability model is not required to be adjusted inversely at the same time as the probability adjustment model is adjusted. In some embodiments, the apparatus 600 further includes a feedback probability determination module configured to determine a feedback probability of the user for the information based on the first feedback probability and the second feedback probability.Also included is an implicit quality feedback score determination module configured to determine an implicit quality feedback score of the first information based on the adjusted second feedback probability, the adjusted second feedback probability indicating that the user likes or dislikes the first information. In some embodiments, the implicit quality feedback score determination module further includes a first ranking module configured to determine a ranking of the first information in the set of information based on the implicit quality feedback score, wherein the mixed ranking is used to recommend information to the user or content provided by a content service. In some embodiments, the implicit quality feedback score determination module further includes a second ranking module configured to determine a mixed ranking of the first information based on the content service and the ranking of the first information in the set of information. FIG. 7 illustrates a block diagram of an electronic device 700 of some embodiments of the present disclosure, which can be the devices or apparatuses described in embodiments of the present disclosure. As shown in FIG. 7, the device 700 includes a central processing unit (CPU) and / or a graphics processing unit (GPU) 701, which can perform various appropriate actions and processes according to computer program instructions stored in a read-only memory (ROM) 702 or loaded from a storage unit 708 into a random access memory (RAM) 703. Various programs and data required by the device 700 to operate can also be stored in the RAM 703. The CPU / GPU 701, the ROM 702, and the RAM 703 are connected to each other through a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704. Although not shown in FIG. 7, the device 700 can also include a co-processor. A plurality of components in the device 700 are connected to the I / O interface 705, including: an input unit 706, such as a keyboard, a mouse, etc.; an output unit 707, such as various types of displays, speakers, etc.; the storage unit 708, such as a magnetic disk, an optical disk, etc.; and a communication unit 709, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 709 allows the device 700 to exchange information / data with other devices through a computer network, such as the Internet, and / or various telecommunication networks. The various methods or processes described above can be performed by the CPU / GPU 701. For example, in some embodiments, the methods can be implemented as a computer software program tangibly embodied in a machine-readable medium, such as the storage unit 708. In some embodiments, part or all of the computer program can be loaded and / or installed on the device 700 via the ROM 702 and / or the communication unit 709.When the computer program is loaded into the RM 703 and executed by the CPU / GPU 701, one or more steps or actions in the methods or processes described above can be performed. In some embodiments, the methods and processes described above can be implemented as a computer program product. The computer program product can include a computer-readable storage medium having stored thereon computer-readable program instructions that, when executed by a computing device, implement various aspects of the present disclosure. The computer-readable storage medium can be a tangible device that can retain and store instructions for use by an instruction execution device. The computer-readable storage medium can be, for example, but is not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of the computer-readable storage medium include the following: a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanically encoded device such as punch-cards or punched tape, a holographic storage medium, or any suitable combination of the foregoing. Computer-readable storage media, as used herein, is not to be construed as being transitory signals per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through a waveguide or other transmission media (e.g., light pulses passing through a fiber-optic cable), or electrical signals transmitted through a wire. Computer-readable program instructions described herein can be downloaded to respective computing / processing devices from a computer-readable storage medium or to an external computer or external storage device via a network, for example, the Internet, a local area network, a wide area network and / or a wireless network. The network can comprise copper transmission cables, optical transmission fibers, wireless transmission, routers, firewalls, switches, gateway computers and / or edge servers. A network adapter card or network interface in each computing / processing device receives computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in a computer- readable storage medium within the respective computing / processing device. Computer-readable program instructions for carrying out operations of the present disclosure can be assembler instructions, instruction-set-architecture (ISA) instructions, machine instructions, machine dependent instructions, microcode, firmware instructions, state-setting data, or either source code or object code written in any combination of one or more programming languages, including an object-oriented programming language and conventional procedural programming languages.Computer readable program instructions can be implemented in a global system for mobile communications (GSM), a code division multiple access (CDMA) system, a widefield system (WAP), a universal mobile telecommunications system (UMTS), a long term evolution (LTE) system, a 5th generation (5G) system, a new radio (NR) system, a code division multiple access 2000 (CDMA2000) system, a global platform for application (GPA) system, a Bluetooth® system, a Zigbee® system, a wireless local area network (WLAN) system, an Internet of Things (IoT) system, an Internet of Everything (IoE) system, a radio frequency identification (RFID) system, a near-field communication (NFC) system, a global navigation satellite system (GNSS) system, a 5G vehicle-to-everything (V2X) system, a 5G vehicle-to-infrastructure (V2I) system, a 5G vehicle-to-network (V2N) system, a 5G vehicle-to-pedestrian (V2P) system, a 5G networked control system (NC) system, a 5G networked industrial automation system, a 5G networked health care system, a 5G networked public safety system, a 5G networked smart grid system, a 5G networked environmental monitoring system, a 5G networked connected city system, a 5G networked connected home system, a 5G networked emergency system, a 5G networked connected office system, a 5G networked logistics system, a 5G networked connected retail system, a 5G networked connected factory system, a 5G networked connected ship system, a 5G networked connected aircraft system, a 5G networked connected automobile system, a 5G networked connected space system, a 5G networked connected medical system, a 5G networked connected security system, a 5G networked connected transportation system, a 5G networked connected energy system, a 5G networked connected agricultural system, a 5G networked connected water system, a 5G networked connected mining system, a 5G networked connected construction system, a 5G networked connected military system, a 5G networked connected satellite system, a 5G networked connected space system, a 5G networked connected navigation system, a 5G networked connected meteorological system, a 5G networked connected meteorological system, a 5G networked connected weather system, a 5G networked connected disaster prevention system, a 5G networked connected disaster relief system, a 5G networked connected rescue system, a 5G networked connected fire prevention system, a 5G networked connected fire control system, a 5G networked connected environmental protection system, a 5G networked connected ecological system, a 5G networked connected energy saving system, a 5G networked connected energy management system, a 5G networked connected energy utilization system, a 5G networked connected energy efficiency system, a 5G networked connected energy conservation system, a 5G networked connected energy recycling system, a 5G networked connected energy development system, a 5G networked connected energy production system, a 5G networked connected energy distribution system, a 5G networked connected energy transmission system, a 5G networked connected energy storage system, a 5G networked connected energy supply system, a 5G networked connected energy utilization system, a 5G networked connected energy system, a 5G networked connected energy system, a 5G networked connected energy system, a 5G networked connected energy system, a 5G networked connected energy system, a 5G networked connected energy system, a 5G networked connected energy system, a 5G networked connected energy system, a 5G networked connected energy system, a 5G networked connected energy system, a 5G networked connected energy system, a 5G networked connected energy system, a 5G networked connected energy system, a 5G networked connected energy system, a 5G networked connected energy system, a 5G networked connected energy system, a 5G networked connected energy system, a 5G networked connected energy system, a 5G networked connected energy system, a 5G networked connected energy system, a 5G networked connected energy system, a 5G networked connected energy system, a 5G networked connected energy system, a 5G networked connected energy system, a 5G networked connected energy system, a 5G networked connected energy system, a 5G networked connected energy system, a 5G networked connected energy system, a 5G networked connected energy system, a 5G networked connected energy system, a 5G networked connected energy system, a 5G networked connected energy system, a 5G networked connected energy system, a 5G networked connected energy system, a 5G networked connected energy system, a 5G networked connected energy system, a 5G networked connected energy system, a 5G networked connected energy system, a 5G networked connected energy system, a 5G networked connected energy system, a 5G networked connected energy system, a 5G networked connected energy system, a 5G networked connected energy system, a 5G networked connected energy system, a 5G networked connected energy system, a 5G networked connected energy system, a 5G networked connected energy system, a 5G networked connected energy system, a 5G networked connected energy system, a 5G networked connected energy system, a 5G networked connected energy system, a 5G networked connected energy system, a 5G networked connected energy system, a 5G networked connected energy system, a 5G networked connected energy system, a 5G networked connected energy system, a 5G networked connected energy system, a 5G networked connected energy system, a 5G networked connected energy system, a 5G networked connected energy system, a 5G networked connected energy system, a 5G networked connected energy system, a 5G networked connected energy system, a 5G networked connected energy system, a 5G networked connected energy system, a 5G networked connected energy system, a 5G networked connected energy system, a 5G networked connected energy system, a 5G networked connected energy system, a 5G networked connected energy system, a 5G networked connected energy system, a 5G networked connected energy system, a 5G networked connected energy system, a 5G networked connected energy system, a 5G networked connected energy system, a 5G networked connected energy system, a 5G networked connected energy system, a 5G networked connected energy system, a 5G networked connected energy system, a 5G networked connected energy system, a 5G networked connected energy system, a 5G networked connected energy system, a 5G networked connected energy system, a 5G networked connected energy system, a 5G networked connected energy system, a 5G networked connected energy system, a 5G networked connected energy system, a 5G networked connected energy system, a 5G networked connected energy system, a 5G networked connected energy system, a 5G networked connected energy system, a 5G networked connected energy system, a 5G networked connected energy system, a 5G networked connected energy system, a 5G networked connected energy system, a 5G networked connected energy system, a 5G networked connected energy system, a 5G networked connected energy system, a 5G networked connected energy system, a 5G networked connected energy system, a 5G networked connected energy system, a 5G networked connected energy system, a 5G networked connected energy system, a 5G networked connected energy system, a 5G networked connected energy system, a 5G networked connected energy system, a 5G networked connected energy system, a 5G networked connected energy system, a 5G networked connected energy system, a 5G networked connected energy system, a 5G networked connected energy system, a 5G networked connected energy system, a 5G networked connected energy system, a 5G networked connected energy system, a 5G networked connected energy system, a 5G networked connected energy system, a 5G networked connected energy system, a 5G networked connected energy system, a 5G networked connected energy system, a 5G networked connected energy system, a 5G networked connected energy system, a 5G networked connected energy system, a 5G networked connected energy system, a 5G networked connected energy system, a 5G networked connected energy system, a 5G networked connected energy system, a 5G networked connected energy system, aIt is also important to note that each block of the flowchart and / or block diagram illustrations, and combinations of blocks in the flowchart and / or block diagrams, can be implemented by dedicated-function hardware-based systems that perform the specified functions or acts, or combinations of dedicated-function hardware and computer instructions. Having thus described the embodiments of the present disclosure in detail, it will be apparent to those skilled in the relevant technical fields that many modifications and variations to the embodiments described herein are possible. All such modifications and variations are intended to be within the scope of the disclosed embodiments. The foregoing description of various embodiments of the technology will be understood to be illustrative only. Thus, no limitation is implied based on the description(s) as described. Further, any steps that can be performed in either order or concurrently with one another are not described as sequential or parallel processes, as some example embodiments can perform steps described in a different order or concurrently with one another. It will thus be appreciated that those skilled in the art will be able to devise numerous systems and methods to implement the various embodiments of the technology from the description that is provided herein, without departing from the spirit and scope of the technology. The language used in the specification is expressly intended to be illustrative only and not limiting, and any use of an optional feature will be specifically indicated as such.

Claims

CLAIM 1. A method for determining feedback of a user for information, comprising: determine a first feedback probability of the user for the information based on first user features of the user and first context features; determine a second feedback probability of the user for the first information based on second user features of the user, second context features, and first information features of the first information; and adjust the second feedback probability of the user for the first information based on the first feedback probability.

2. The method of claim 1, wherein determining the first feedback probability of the user for information based on the first user features and the first context features of the user comprises: determine the first feedback probability of the user for the information based on the first user features of the user and the first context features by a first feedback probability model.

3. The method of claim 2, wherein determining a second feedback probability of the user for the first information based on a second user feature of the user, a second context feature, a first information feature of the first information comprises: determining a second feedback probability of the user for the first information based on a second user feature of the user, a second context feature, a first information feature of the first information. determine the second feedback probability of the user for the first information based on the second user features of the user, the second context features, and the first information features of the first information by a second feedback probability model. ​ 4. The method of claim 3, wherein adjusting, based on the first feedback probability, a second feedback probability of the user for the first information comprises: determine an adjusted second feedback probability based on the first feedback probability, the second feedback probability, and a compensation scalar by a probability adjustment model, the compensation scalar being used to correct the adjusted second feedback probability.

5. The method of claim 4, further comprising: determine a first training feedback probability based on first training user features and training first context features by the first feedback probability model; and adjust the first feedback probability model based on a loss of the first training feedback probability and a real feedback probability.

6. The method of claim 4, further comprising: determine a second training feedback probability based on second training user features, second training context features, and first training information features by the second feedback probability model; and adjust the second feedback probability model based on a loss of the second training feedback probability and the real feedback probability, wherein the first feedback probability model does not need to be adjusted reversely at the same time when adjusting the second feedback probability model.

7. The method of claim 4, further comprising: determine a training adjusted second feedback probability based on a first training feedback probability, a second training feedback probability, and a training compensation scalar by the probability adjustment model and the compensation scalar; and adjust the probability adjustment model and the compensation scalar based on the training adjusted second feedback probability, wherein the second feedback probability model does not need to be adjusted reversely at the same time when adjusting the probability adjustment model.

8. The method of claim 1, further comprising: determine an implicit quality feedback score of the first information based on the adjusted second feedback probability, the adjusted second feedback probability indicating that the user likes or dislikes the first information.

9. The method of claim 8, further comprising: determine a ranking of the first information in an information set based on the implicit quality feedback score.

10. The method of claim 9, further comprising: determine a hybrid ranking of the first information based on a content service and the ranking of the first information in the information set, the hybrid ranking being used to recommend the information or content provided by the content service to the user.

11. An apparatus for determining feedback of a user for information, comprising: a first feedback probability determining module configured to determine a first feedback probability of a user for information based on first user features of the user and first context features; The second feedback probability determination module is configured to determine a second feedback probability of the user for the first information based on a second user feature of the user, a second context feature, and a first information feature of the first information. The second feedback probability adjustment module is configured to adjust the second feedback probability of the user for the first information based on the first feedback probability.

12. An electronic device, comprising: a processor; and a memory coupled with the processor, the memory having stored within it instructions that, when executed by the processor, cause the electronic device to perform the method according to any one of claims 1-10.

13. A computer program product comprising computer-executable instructions, wherein the computer-executable instructions are executed by a processor to implement the method according to any one of claims 1-10.

Images