Method, apparatus, electronic device, and storage medium for obtaining feedback information

Abstract

The present disclosure relates to a method, apparatus, electronic device, and storage medium for obtaining feedback information, belonging to the field of network technologies. The present disclosure obtains an experience prediction value of a user for a played content item based on the user information of the user and the content item information of the played content item, and obtains the prediction probabilities of various feedback behaviors of the user based on the user information, the content item information, and the experience prediction value. When the prediction probabilities meet the target probability conditions, a target page resource is sent to the terminal, so that the target page resource triggers the terminal to return the collected feedback information. This collection method that does not rely on random sampling and periodic deduplication can accurately control when to display the collection window and to which users, and can greatly improve the acquisition efficiency and accuracy of the feedback information.

Description

Technical Field

[0001] The present disclosure relates to the field of network technologies, and in particular, to a method, apparatus, electronic device, and storage medium for obtaining feedback information. Background Art

[0002] With the development of network technologies and the diversification of terminal functions, users can browse multimedia resources on terminals anytime and anywhere, and advertisers can also cooperate with platform operators to deliver content items such as advertisements to terminals. Currently, platform operators can collect users' feedback information through pop-up windows. By analyzing the collected feedback information, it can be used to indicate the subsequent delivery methods and delivery densities of content items.

[0003] In the above process of collecting feedback information, a method of combining random sampling and periodic duplicate removal is usually adopted to screen out some users, and pop-up windows are displayed on the terminals of these users. Usually, a questionnaire designed by business personnel is displayed in the pop-up window, prompting these users to input corresponding feedback information for the preset questions in the questionnaire. Among them, random sampling means randomly selecting some sample users for pop-up window display, and periodic duplicate removal means that the same user can receive at most one pop-up window display within a specified period.

[0004] The above method for collecting feedback information has great limitations for some sparse user behaviors. Taking the negative feedback behavior of advertisements (referring to the feedback behavior generated by users due to negative experiences after browsing advertisements) as an example, assuming that the sampling ratio is α, the proportion of users covered by advertisements is β, and the negative feedback behavior rate of advertisements is γ, the negative feedback behavior rate of advertisements collected is αβγ. Since both the sampling ratio α and the negative feedback behavior rate γ of advertisements are extremely small in actual scenarios, the finally collected negative feedback behavior rate αβγ of advertisements is extremely low, making it difficult to ensure that the feedback information collected by the method of random sampling and periodic duplicate removal can represent the overall users, that is, the acquisition efficiency and accuracy of feedback information are relatively low. Summary of the Invention

[0005] The present disclosure provides a method, apparatus, electronic device, and storage medium for obtaining feedback information, which can improve the acquisition efficiency and accuracy of feedback information. The technical solutions of the present disclosure are as follows:

[0006] According to a first aspect of an embodiment of the present disclosure, there is provided a method for obtaining feedback information, including:

[0007] Obtaining an experience prediction value of a user for at least one played content item according to the user information of the user and the content item information of the at least one played content item, where one experience prediction value is used to characterize the viewing experience of the user for one played content item;

[0008] Obtain the predicted probability that the user generates at least one feedback behavior according to the user information, the content item information, and the experience prediction value, where one predicted probability is used to characterize the possibility that the user generates a feedback behavior.

[0009] In response to the predicted probability meeting the target probability condition, send the target page resource to the terminal corresponding to the user, where the target page resource is used to trigger the terminal to return the collected feedback information after collecting the feedback information.

[0010] In a possible implementation manner, the obtaining the predicted probability that the user generates at least one feedback behavior according to the user information, the content item information, and the experience prediction value includes:

[0011] Input the user information, the content item information, and the experience prediction value into the feedback behavior model, and perform weighted processing on the user, the content item information, and the experience prediction value through the feedback behavior model to respectively obtain the predicted probabilities that the user generates the at least one feedback behavior.

[0012] In a possible implementation manner, the at least one feedback behavior includes at least one of normal subsequent viewing, reduced subsequent viewing, or exiting the application; the target probability condition is that the weighted sum value of the predicted probabilities of the at least one feedback behavior is greater than the target probability threshold.

[0013] In a possible implementation manner, the method further includes:

[0014] Obtain the sample user information of the sample user, the sample content item information of at least one played sample content item, the sample experience prediction value of the sample user for the at least one sample content item, and the historical behavior.

[0015] Train the initial behavior model based on the sample user information, the sample content item information, the sample experience prediction value, and the historical behavior to obtain the feedback behavior model.

[0016] In a possible implementation manner, the obtaining the experience prediction value of the user for the at least one content item according to the user information of the user and the content item information of at least one played content item includes:

[0017] Input the user information and the content item information into the viewing experience model, and perform weighted processing on the user information and the content item information through the viewing experience model to respectively predict the playback duration ratio of the user for the at least one content item, and determine the playback duration ratio as the experience prediction value, where one playback duration ratio is used to represent the ratio between the expected playback duration of the user for a content item and the total duration of the content item.

[0018] In a possible implementation manner, the method further includes:

[0019] Obtain sample user information of a sample user and sample content item information of at least one played sample content item;

[0020] From the at least one sample content item, screen out positive sample content items with a playback duration ratio greater than a first target threshold and negative sample content items with a playback duration ratio less than a second target threshold, where the first target threshold is greater than or equal to the second target threshold;

[0021] Train an initial experience model based on the sample user information, the sample content item information of the positive sample content items, and the sample content item information of the negative sample content items to obtain the viewing experience model.

[0022] In a possible implementation manner, before the step of sending the target page resource to the terminal corresponding to the user in response to the prediction probability meeting the target probability condition, the method further includes:

[0023] Determine at least one research question of the user according to the weekly activity information of the user;

[0024] Add the at least one research question to the target page resource.

[0025] In a possible implementation manner, the step of determining at least one research question of the user according to the weekly activity information of the user includes:

[0026] Input the weekly activity information of the user into a multi-classification model, and predict the feedback problem category to which the user belongs through the multi-classification model, where the feedback problem category is used to characterize the category of harm suffered by the user experience during the viewing of the content item;

[0027] Determine at least one research question under the feedback problem category based on the predicted feedback problem category.

[0028] According to a second aspect of the embodiments of the present disclosure, a method for obtaining feedback information is provided, which is applied to a terminal and includes:

[0029] Receive a target page resource sent by a server when the prediction probability meets the target probability condition, where the prediction probability is obtained by the server according to the user information of the user corresponding to the terminal, the content item information of at least one played content item, and the experience prediction value of the user for the at least one content item;

[0030] Based on the target page resource, display a collection window for collecting feedback information;

[0031] Collect the feedback information of the user corresponding to the terminal based on the collection window;

[0032] Send the collected feedback information to the server.

[0033] According to the third aspect of the embodiments of the present disclosure, there is provided a device for obtaining feedback information, including:

[0034] A first obtaining unit configured to execute obtaining an experience prediction value of the user for at least one content item based on the user information of the user and the content item information of at least one played content item, where an experience prediction value is used to characterize the viewing experience of the user for one played content item;

[0035] A second obtaining unit configured to execute obtaining a prediction probability that the user generates at least one feedback behavior based on the user information, the content item information, and the experience prediction value, where a prediction probability is used to characterize the possibility that the user generates a feedback behavior;

[0036] A sending unit configured to execute sending a target page resource to the terminal corresponding to the user in response to the prediction probability meeting a target probability condition, where the target page resource is used to trigger the terminal to return the collected feedback information after collecting the feedback information.

[0037] In a possible implementation manner, the second obtaining unit is configured to execute:

[0038] Input the user information, the content item information, and the experience prediction value into a feedback behavior model, and perform weighted processing on the user, the content item information, and the experience prediction value through the feedback behavior model to respectively obtain the prediction probabilities that the user generates the at least one feedback behavior.

[0039] In a possible implementation manner, the at least one feedback behavior includes at least one of normal subsequent viewing, reduced subsequent viewing, or exiting the application; the target probability condition is that the weighted sum value of the prediction probabilities of the at least one feedback behavior is greater than a target probability threshold.

[0040] In a possible implementation manner, the device further includes:

[0041] A first training unit configured to execute obtaining sample user information of a sample user, sample content item information of at least one played sample content item, sample experience prediction values of the sample user for the at least one sample content item, and historical behaviors; training an initial behavior model based on the sample user information, the sample content item information, the sample experience prediction values, and the historical behaviors to obtain the feedback behavior model.

[0042] In a possible implementation manner, the first obtaining unit is configured to perform:

[0043] Input the user information and the content item information into a viewing experience model, perform weighted processing on the user information and the content item information through the viewing experience model, respectively predict the proportion of the playing duration of the user for the at least one content item, and determine the proportion of the playing duration as the experience prediction value, where a proportion of the playing duration is used to represent the ratio between the predicted playing duration of the user for a content item and the total duration of the content item.

[0044] In a possible implementation manner, the device further includes:

[0045] A second training unit, configured to obtain sample user information of a sample user and sample content item information of at least one played sample content item; screen out positive sample content items with a proportion of playing duration greater than a first target threshold and negative sample content items with a proportion of playing duration less than a second target threshold from the at least one sample content item, where the first target threshold is greater than or equal to the second target threshold; train an initial experience model based on the sample user information, the sample content item information of the positive sample content items, and the sample content item information of the negative sample content items to obtain the viewing experience model.

[0046] In a possible implementation manner, the device further includes:

[0047] A determination unit, configured to determine at least one research question of the user according to the weekly activity information of the user;

[0048] An adding unit, configured to add the at least one research question to the target page resource.

[0049] In a possible implementation manner, the determination unit is configured to perform:

[0050] Input the weekly activity information of the user into a multi-classification model, and predict, through the multi-classification model, the feedback problem category to which the user belongs, where the feedback problem category is used to characterize the category of harm suffered by the user experience during the viewing of the content item;

[0051] Based on the predicted feedback problem category, determine at least one research question under the feedback problem category.

[0052] According to a fourth aspect of the embodiments of the present disclosure, there is provided a device for obtaining feedback information, including:

[0053] A receiving unit, configured to receive a target page resource sent by a server when a prediction probability meets a target probability condition, where the prediction probability is obtained by the server based on user information of the user corresponding to the terminal, content item information of at least one played content item, and an experience prediction value of the user for the at least one content item;

[0054] A display unit, configured to display a collection window for collecting feedback information based on the target page resource;

[0055] A collection unit, configured to collect feedback information of the user corresponding to the terminal based on the collection window;

[0056] A sending unit, configured to send the collected feedback information to the server.

[0057] According to a fifth aspect of the embodiments of the present disclosure, there is provided an electronic device, including:

[0058] One or more processors;

[0059] One or more memories for storing executable instructions of the one or more processors;

[0060] Wherein, the one or more processors are configured to execute the feedback information acquisition method according to any one of the first aspect and the possible implementation manners of the first aspect; or execute the feedback information acquisition method according to any one of the second aspect and the possible implementation manners of the second aspect.

[0061] According to a sixth aspect of the embodiments of the present disclosure, there is provided a storage medium, when at least one instruction in the storage medium is executed by one or more processors of an electronic device, enabling the electronic device to execute the feedback information acquisition method according to any one of the first aspect and the possible implementation manners of the first aspect; or execute the feedback information acquisition method according to any one of the second aspect and the possible implementation manners of the second aspect.

[0062] According to a seventh aspect of the embodiments of the present disclosure, there is provided a computer program product, including one or more instructions, where the one or more instructions can be executed by one or more processors of an electronic device, enabling the electronic device to execute the feedback information acquisition method according to any one of the first aspect and the possible implementation manners of the first aspect; or execute the feedback information acquisition method according to any one of the second aspect and the possible implementation manners of the second aspect.

[0063] The technical solutions provided by the embodiments of the present disclosure at least bring the following beneficial effects:

[0064] Based on the user information of the user and the content item information of the played content item, obtain the predicted value of the user's experience with the played content item. According to the user information, the content item information, and the predicted value of the experience, obtain the predicted probabilities of various feedback behaviors of the user. When the predicted probabilities meet the target probability conditions, send the target page resource to the terminal, triggering the terminal to return the collected feedback information after receiving the target page resource. By controlling different target probability conditions, the timing of sending the target page resource to the terminal is controlled, which is equivalent to controlling the pop-up conditions of the collection window for the feedback information on the terminal side. This collection method that does not rely on random sampling and periodic deduplication can automatically, accurately, and intelligently control when to send the target page resource to the terminals corresponding to which users, so that after the terminal collects the feedback information based on the target page resource, the collected feedback information is returned to the electronic device. Therefore, the acquisition efficiency and accuracy of the feedback information can be greatly improved.

[0065] It should be understood that the above general description and the following detailed description are only exemplary and explanatory, and do not limit the present disclosure. Brief Description of the Drawings

[0066] The drawings here are incorporated into the specification and form a part of this specification, showing the embodiments consistent with the present disclosure, and are used together with the specification to explain the principles of the present disclosure, and do not constitute an improper limitation of the present disclosure.

[0067] Figure 1 It is a schematic diagram of the implementation environment of a method for obtaining feedback information provided by an embodiment of the present disclosure;

[0068] Figure 2 It is a flowchart of a method for obtaining feedback information shown according to an exemplary embodiment;

[0069] Figure 3 It is an interaction flowchart of a method for obtaining feedback information shown according to an exemplary embodiment;

[0070] Figure 4 It is a logical structure block diagram of a device for obtaining feedback information shown according to an exemplary embodiment;

[0071] Figure 5 It is a logical structure block diagram of a device for obtaining feedback information shown according to an exemplary embodiment;

[0072] Figure 6 It shows a structural block diagram of a terminal provided by an exemplary embodiment of the present disclosure;

[0073] Figure 7 It is a schematic structural diagram of a server provided by an embodiment of the present disclosure. Detailed Embodiments

[0074] To enable those of ordinary skill in the art to better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below with reference to the accompanying drawings.

[0075] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and do not necessarily need to describe a specific order or sequence. It should be understood that such used data can be interchanged under appropriate circumstances so that the embodiments of the present disclosure described here can be implemented in an order different from those illustrated or described here. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. On the contrary, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0076] The user information involved in the present disclosure may be information authorized by the user or fully authorized by all parties.

[0077] Figure 1 It is a schematic diagram of the implementation environment of a method for obtaining feedback information provided by an embodiment of the present disclosure. Refer to Figure 1 , in this implementation environment, it includes a terminal 101 and a server 102. The terminal 101 and the server 102 are connected through a wireless network or a wired network. The terminal 101 and the server 102 are both examples of electronic devices.

[0078] Among them, the terminal 101 is used to browse each content item put by the server 102 and collect feedback information of the user for each put content item. After the collection is completed, the terminal 101 can report the collected feedback information to the server 102.

[0079] Optionally, the content item may be formed by combining various types of multimedia resources. For example, the content item may be a video advertisement or a graphic advertisement. An application program supporting the content item placement service may be installed on the terminal 101, so that the user can browse the content item by starting the application program. The application program may be at least one of a live broadcast application program, a short video application program, a shopping application program, a food delivery application program, a travel application program, a game application program, or a social application program.

[0080] Optionally, the feedback information may include positive feedback information and negative feedback information. Positive feedback information refers to the feedback information input by the user based on a positive viewing experience (that is, no harmful experience) after browsing each content item, while negative feedback information refers to the feedback information input by the user based on a negative viewing experience (that is, a harmful experience) after browsing each content item.

[0081] Among them, the server 102 is used to deliver content items to the terminal 101 and collect feedback information reported by the terminal 101. The server 102 may include at least one of a single server, multiple servers, a cloud computing platform, or a virtualization center. Optionally, the server 102 may undertake the main computing work, and the terminal 101 may undertake the secondary computing work; or, the server 102 undertakes the secondary computing work, and the terminal 101 undertakes the main computing work; or, a distributed computing architecture is adopted between the server 102 and the terminal 101 for collaborative computing.

[0082] In an exemplary scenario, the content item may be a video advertisement. Taking the installation of a short video application on the terminal 101 as an example, the server 102 provides a short video viewing platform for the terminal 101 through the short video application. The server 102 delivers various video advertisements to the terminal 101. After the user browses any video advertisement on the terminal 101, the server 102, based on the feedback information acquisition method provided in the embodiments of the present disclosure, under the condition of meeting the target probability condition, issues a target page resource to the terminal 101, so that the terminal 101 displays a collection window for collecting feedback information based on the target page resource. The user can input feedback information for the browsed video advertisement in the collection window. After the terminal 101 collects the feedback information based on the collection window, the collected feedback information is reported to the server 102.

[0083] It should be noted that the terminal 101 may generally refer to one of multiple terminals. The device types of the terminal 101 may include at least one of a smart phone, a tablet computer, an e-book reader, an MP3 (Moving Picture Experts Group Audio Layer III) player, an MP4 (Moving Picture Experts Group Audio Layer IV) player, a laptop computer, or a desktop computer. For example, the terminal 101 may be a smart phone or other handheld portable electronic device. In the following embodiments, the terminal is exemplified by including a smart phone.

[0084] Those skilled in the art can know that the number of the above terminals may be more or less. For example, the above terminal may be only one, or the above terminals may be dozens or hundreds, or a larger number. The embodiments of the present disclosure do not limit the number and device types of the terminals.

[0085] Figure 2 is a flowchart of a method for obtaining feedback information shown according to an exemplary embodiment. Refer to Figure 2, The method for obtaining the feedback information is applied to an electronic device, which may be the server 102 in the above implementation environment. The following is an explanation.

[0086] In step 201, the electronic device obtains an experience prediction value of the user for the at least one content item according to the user information of the user and the content item information of the at least one played content item. An experience prediction value is used to represent the viewing experience of the user for one played content item.

[0087] In step 202, the electronic device obtains a prediction probability that the user generates at least one feedback behavior according to the user information, the content item information, and the experience prediction value. A prediction probability is used to represent the possibility that the user generates a feedback behavior.

[0088] In step 203, in response to the prediction probability meeting the target probability condition, the electronic device sends target page resources to the terminal corresponding to the user. The target page resources are used to trigger the terminal to return the collected feedback information after collecting the feedback information.

[0089] The method provided by the embodiments of the present disclosure obtains an experience prediction value of the user for the played content item through the user information of the user and the content item information of the played content item, obtains a prediction probability that the user generates various feedback behaviors according to the user information, the content item information, and the experience prediction value, and when the prediction probability meets the target probability condition, sends target page resources to the terminal to trigger the terminal to return the collected feedback information after receiving the target page resources. By controlling different target probability conditions, the timing of sending the target page resources to the terminal is controlled, which is equivalent to controlling the pop-up condition of the feedback information collection window on the terminal side. This collection method that does not rely on random sampling and periodic deduplication can automatically, accurately, and intelligently control when and to which terminals corresponding to which users the target page resources are sent, so that after the terminal collects the feedback information according to the target page resources and returns the collected feedback information to the electronic device, the efficiency and accuracy of obtaining the feedback information can be greatly improved.

[0090] In a possible implementation manner, obtaining a prediction probability that the user generates at least one feedback behavior according to the user information, the content item information, and the experience prediction value includes:

[0091] Input the user information, the content item information, and the experience prediction value into a feedback behavior model, and perform weighted processing on the user, the content item information, and the experience prediction value through the feedback behavior model to respectively obtain the prediction probability that the user generates the at least one feedback behavior.

[0092] In a possible implementation, the at least one feedback behavior includes at least one of normal subsequent viewing, reduced subsequent viewing, or exiting the application; the target probability condition is that the weighted sum value of the prediction probabilities of the at least one feedback behavior is greater than the target probability threshold.

[0093] In a possible implementation, the method further includes:

[0094] Obtain the sample user information of the sample user, the sample content item information of at least one played sample content item, the sample experience prediction value of the sample user for the at least one sample content item, and the historical behavior.

[0095] Based on the sample user information, the sample content item information, the sample experience prediction value, and the historical behavior, train the initial behavior model to obtain the feedback behavior model.

[0096] In a possible implementation, obtaining the experience prediction value of the user for the at least one content item according to the user information of the user and the content item information of at least one played content item includes:

[0097] Input the user information and the content item information into the viewing experience model, perform weighted processing on the user information and the content item information through the viewing experience model, respectively predict the proportion of the playing duration of the at least one content item for the user, and determine the proportion of the playing duration as the experience prediction value. One proportion of the playing duration is used to represent the ratio between the expected playing duration of the user for a content item and the total duration of the content item.

[0098] In a possible implementation, the method further includes:

[0099] Obtain the sample user information of the sample user and the sample content item information of at least one played sample content item.

[0100] From the at least one sample content item, screen out positive sample content items with a playing duration proportion greater than the first target threshold and negative sample content items with a playing duration proportion less than the second target threshold, where the first target threshold is greater than or equal to the second target threshold.

[0101] Based on the sample user information, the sample content item information of the positive sample content items, and the sample content item information of the negative sample content items, train the initial experience model to obtain the viewing experience model.

[0102] In a possible implementation, before sending the target page resource to the terminal corresponding to the user in response to the prediction probability meeting the target probability condition, the method further includes:

[0103] Determine at least one research question for the user according to the weekly active information of the user.

[0104] Add the at least one research question to the target page resource.

[0105] In a possible implementation, determining at least one research question of the user according to the user's weekly activity information includes:

[0106] Input the user's weekly activity information into a multi-classification model, and predict the feedback problem category to which the user belongs through the multi-classification model. The feedback problem category is used to characterize the category of harm suffered by the user experience during the viewing of the content item;

[0107] Based on the predicted feedback problem category, determine at least one research question under the feedback problem category.

[0108] All the above optional technical solutions can be combined arbitrarily to form optional embodiments of the present disclosure, which will not be elaborated here one by one.

[0109] Figure 3 It is an interaction flowchart of a method for obtaining feedback information shown according to an exemplary embodiment. As Figure 3 shown, the method for obtaining feedback information is applied to the interaction process between a terminal and a server. Both the terminal and the server are examples of electronic devices. This embodiment includes the following steps.

[0110] In step 301, the server inputs the user information of the user and the content item information of at least one played content item into a viewing experience model, and performs weighted processing on the user information and the content item information through the viewing experience model, and respectively predicts the playback duration ratio of the user for the at least one content item, and determines the playback duration ratio as the experience prediction value of the user for the at least one content item.

[0111] Among them, a playback duration ratio is used to represent the ratio between the predicted playback duration of the user for a content item and the total duration of the content item. Taking the content item as a video advertisement as an example, the playback duration ratio refers to the ratio of the duration of the user watching the video advertisement to the total duration of the video advertisement.

[0112] Among them, an experience prediction value is used to characterize the viewing experience of the user for a played content item. The higher the experience prediction value, the lower the experience harm suffered and the better the viewing experience. On the contrary, the lower the experience prediction value, the higher the experience harm suffered and the worse the viewing experience.

[0113] Optionally, the user information may include user attribute information and user behavior information. The user attribute information is used to represent the basic attributes of the user, and may include at least one of the user's gender, age, nickname, occupation, or geographical location. The user behavior information is used to represent the historical behavior of the user on the application, and may include at least one of the historical browsing behavior of a certain content item or the historical consumption behavior of the products recommended by a certain content item. It should be noted that the user information involved in the embodiments of the present disclosure is all information authorized by the user or fully authorized by all parties.

[0114] In the above process, the user information may be stored in the local database or downloaded from the cloud database. Of course, the user information may also be sent from the terminal to the server. In some embodiments, the server may store the user attribute information submitted by the user when registering the account in the local database and obtain the user behavior information from the terminal in real time.

[0115] Optionally, the content item information may include the material information of the content item. The material types may include at least one of video, copywriting, cover, or sticker. After different types of materials are combined in different combination ways, different content items can be formed. For example, for the same video material, the same cover material, and the same sticker material, after they are combined with different copywriting materials, different content items can be obtained. Optionally, the content item may be of the video type or the graphic type, that is to say, not all types of materials need to be included in each content item. Taking the content item as an advertisement as an example, the video advertisement may include video, copywriting, cover, advertisement position, sticker, and advertisement slogan. The graphic advertisement may not include video but only carry pictures, copywriting, advertisement position, sticker, and advertisement slogan. The embodiments of the present disclosure do not specifically limit the type of the content item.

[0116] In the above process, the content item information may be stored in the local database or downloaded from the cloud database. Of course, the content item information may also be uploaded by the advertiser to the server.

[0117] In step 301 above, after obtaining the user information and the content item information, the server may input the user information and the content item information into the viewing experience model. The viewing experience model extracts features from the user information to obtain user features, extracts features from the content item information to obtain content item features, performs weighted processing on the user features and the content item features, respectively predicts the proportion of the playback duration of each content item for the user, and obtains the experience prediction value of the user for each content item as the proportion of the playback duration of each content item for the user.

[0118] In the above process, the server can first extract features from the user information and the content item information respectively through the viewing experience model, and then obtain the respective experience prediction values based on the extracted user features and content item features. At this time, it can be considered that the viewing experience model includes a feature extraction part and a prediction part. The above two parts can adopt independent machine learning sub-models respectively, or can adopt an overall encoder-decoder model (where the encoding part is the feature extraction part and the decoding part is the prediction part). The embodiments of the present disclosure do not specifically limit the type of the viewing experience model. For example, the viewing experience model can be a CNN (Convolutional Neural Networks), a DNN (Deep Neural Networks), an LSTM (Long Short-Term Memory), etc.

[0119] In the above process, the server obtains the experience prediction values of the user for the at least one content item according to the user information of the user and the content item information of the at least one played content item. In some embodiments, the feature extraction part can also be used as a pre-network of the viewing experience model instead of a sub-model of the viewing experience model. At this time, after the server obtains the user information and the content item information, it extracts features from the user information through the user feature extraction network to obtain user features, extracts features from the content item information through the content item feature extraction network to obtain content item features, inputs the user features and the content item features into the viewing experience model, and the viewing experience model performs weighted processing on the user features and the content item features to respectively predict the proportion of the playing duration of each content item for the user, and obtains the proportion of the playing duration of each content item for the user as the experience prediction value of the user for each content item. Optionally, the user feature extraction network or the content item feature extraction network can be any machine learning model, such as CNN, DNN, LSTM, etc., which will not be elaborated here.

[0120] In some embodiments, in addition to using a machine learning model to extract user features and content item features, the one-hot encoding method can also be used to extract at least one of the user features or the content item features. Of course, the embedding method can also be used to extract at least one of the user features or the content item features. The embodiments of the present disclosure do not limit the specific feature extraction method.

[0121] It should be noted that before the server calls the viewing experience model to obtain the experience prediction value, the viewing experience model can be trained in the following way: obtain the sample user information of the sample users and the sample content item information of at least one played sample content item; from the at least one sample content item, screen out the positive sample content items with a playback duration ratio greater than the first target threshold and the negative sample content items with a playback duration ratio less than the second target threshold, where the first target threshold is greater than or equal to the second target threshold; based on the sample user information, the sample content item information of the positive sample content items, and the sample content item information of the negative sample content items, train the initial experience model to obtain the viewing experience model. Among them, the first target threshold is any value greater than or equal to 0 and less than or equal to 1, and the second target threshold is any value greater than or equal to 0 and less than or equal to 1.

[0122] In the above process, after the server collects the sample user information and the sample content item information, it is necessary to screen the sample content items to screen out the representative positive sample content items (referred to as "positive samples" for short) and negative sample content items (referred to as "negative samples" for short). Based on the sample user information, the screened positive samples, and the negative samples, iterate and train the initial experience model. When the stop training condition is not met, adjust the parameters of the initial experience model and iterate to execute the training steps until the stop training condition is met, and the viewing experience model is obtained.

[0123] Optionally, the stop training condition can be that the loss function value is less than the third target threshold, or the stop training condition can be that the number of iterations is greater than the first target number. The third target threshold can be any value greater than or equal to 0 and less than or equal to 1, and the first target number can be any integer greater than or equal to 1.

[0124] In an exemplary scenario, assume that the first target threshold is 0.8 and the second target threshold is 0.2. At this time, according to the playback duration ratio of each sample content item by the user, the sample content items with a playback duration ratio greater than 0.8 can be classified as positive sample content items, and the corresponding label of such positive sample content items can be set to "long playback", and the sample content items with a playback duration ratio less than 0.2 can be classified as negative sample content items, and the corresponding label of such negative sample content items can be set to "short playback", so as to train a binary classification model as the viewing experience model. This binary classification model is used to identify whether the content item to be predicted belongs to the "short playback" label or the "long playback" label.

[0125] In the above process, since the proportion of the user's playback duration can intuitively reflect the user's viewing experience, the longer the proportion of the playback duration of a certain content item by the user, the better the user's viewing experience of this content item can be considered, and the greater its experience prediction value should be. By training a viewing experience model, the experience prediction value of the user for any content item can be accurately predicted, so as to complete the quantification of the harm to the user experience.

[0126] In other words, the index that can most intuitively reflect the user's viewing experience is the proportion of the user's playback duration, and the proportion of the user's playback duration can be quantified in different dimensions. For example, in the time dimension, it can be quantified to the granularity of days, weeks, and months. For example, in the behavior dimension, it can be quantified to each login, each refresh, and each viewing. And exactly the viewing experience model can predict the user experience in the dimension of "each viewing", so as to timely perceive whether the user experience has been damaged. For example, assume that most users have watched a video advertisement for T (T≥0) seconds, and assume that there is a target user whose viewing duration of this video advertisement is much less than T seconds. Then, it is very likely that the viewing experience of this target user has been damaged. If the damage to the user's viewing experience accumulates continuously, then ultimately it will cause the user to exit the application program and even directly result in user churn. Therefore, it is necessary to timely collect the user's feedback information (negative feedback information is particularly important), and execute step 302 below.

[0127] In step 302, the server inputs the user information, the content item information, and the experience prediction value into a feedback behavior model, and performs weighted processing on the user, the content item information, and the experience prediction value through this feedback behavior model to respectively obtain the prediction probabilities of the user generating at least one feedback behavior.

[0128] Among them, one prediction probability is used to represent the possibility that the user generates a feedback behavior.

[0129] Optionally, the at least one feedback behavior includes at least one of normal subsequent viewing, reduced subsequent viewing, or exiting the application program. Among them, normal subsequent viewing means that the number of videos expected to be watched by the user is not significantly less than that before watching the content item, reduced subsequent viewing means that the number of videos expected to be watched by the user is significantly less than that before watching the content item, and exiting the application program means that the user is expected to immediately exit the application program after watching the content item.

[0130] Since content items are usually interspersed and played in videos, the user's feedback behavior can be measured by comparing the change in the number of videos expected to be watched by the user before and after watching the content item. Optionally, when the difference between the number of videos before watching the content item and the number of videos expected to be watched by the user is greater than the video quantity threshold, it is determined that there is a significant decrease. The video quantity threshold is any integer greater than or equal to 1.

[0131] In the above step 302, the server may input the user information, content item information, and experience prediction value into a feedback behavior model. The feedback behavior model extracts features from the user information to obtain user features, extracts features from the content item information to obtain content item features, extracts features from the experience prediction value to obtain viewing experience features, and performs weighted processing on the user features, content item features, and viewing experience features to respectively obtain the prediction probabilities of the user generating each feedback behavior.

[0132] In the above process, the server may first extract features from the user information, content item information, and experience prediction value respectively through the feedback behavior model, and then obtain each prediction probability based on the extracted user features, content item features, and viewing experience features. At this time, it can be considered that the feedback behavior model includes a feature extraction part and a prediction part. The above two parts may respectively adopt independent machine learning sub-models, or may adopt an overall encoder-decoder model (where the encoding part is the feature extraction part and the decoding part is the prediction part). The embodiments of the present disclosure do not specifically limit the type of the feedback behavior model. For example, the feedback behavior model may be a CNN, DNN, LSTM, etc.

[0133] In the above process, the server obtains the prediction probabilities of the user generating at least one feedback behavior according to the user information, the content item information, and the experience prediction value. In some embodiments, the server may also directly use the user features and content item features extracted in the above step 301, and only needs to extract viewing experience features for each experience prediction value, which can save the computational amount of the prediction process.

[0134] It should be noted that before the server calls the feedback behavior model to obtain the prediction probabilities, the feedback behavior model may be trained in the following manner: obtaining the sample user information of the sample user, the sample content item information of at least one played sample content item, the sample experience prediction value of the sample user for the at least one sample content item, and the historical behavior; training an initial behavior model based on the sample user information, the sample content item information, the sample experience prediction value, and the historical behavior to obtain the feedback behavior model.

[0135] In the above process, after the server collects the sample user information and sample content item information, first, it inputs the sample user information and sample content item information into the viewing experience model, and through this viewing experience model, it outputs the sample experience prediction values of the sample user for each sample content item. Secondly, it selects the historical session data of the sample user and labels the historical behaviors of the sample user in the historical session. For example, after playing a certain content item, if the amount of video data transmitted in the historical session significantly decreases (the difference in data volume before and after watching the content item is greater than the data threshold), then the historical behavior can be labeled as "reduce subsequent viewing". Another example is that after playing a certain content item, if the video data stream transmitted in the historical session does not significantly decrease (the difference in data volume before and after watching the content item is less than or equal to the data threshold), then the historical behavior can be labeled as "normal subsequent viewing". Another example is that after playing a certain content item, the historical session is directly disconnected, then the historical behavior can be labeled as "exit the application".

[0136] Further, the server iteratively trains the initial behavior model based on the sample user information, sample content item information, sample experience prediction values, and the labeled historical behaviors. When the stop training condition is not met, it adjusts the parameters of the initial behavior model and iteratively executes the training steps until the stop training condition is met, and then obtains the feedback behavior model.

[0137] Optionally, the stop training condition can be that the loss function value is less than the fourth target threshold, or the number of iterations is greater than the second target number. The fourth target threshold can be any value greater than or equal to 0 and less than or equal to 1, and the second target number can be any integer greater than or equal to 1.

[0138] In step 303, in response to the prediction probability meeting the target probability condition, the server inputs the weekly active information of the user into the multi-classification model, and through this multi-classification model, it predicts the feedback problem category to which the user belongs.

[0139] Among them, the target probability condition is that the weighted sum value of the prediction probabilities of the at least one feedback behavior is greater than the target probability threshold. Optionally, the weights of each feedback behavior and the target probability threshold can be preset by technicians.

[0140] In some embodiments, the target probability threshold can be set with different levels of values, so as to be able to adaptively adjust the level dynamically according to the platform activity. For example, when the overall platform activity is relatively high, the target probability threshold is adjusted to a relatively high-level value, and when the overall platform activity is relatively low, the target probability threshold is adjusted to a relatively low-level value.

[0141] Among them, the feedback problem category is used to characterize the category of harm suffered by the user experience during the viewing of the content item. For example, the feedback problem category can be that there are too many content items being delivered, not interested in the content item, the visual experience of the content item is not good, clickbait titles, etc.

[0142] In an exemplary scenario, assuming the content item is a video advertisement, its feedback problem category can be divided into 4 categories, namely "Problem Classification 1: There are too many platform advertisements", "Problem Classification 2: Not interested in the content", "Problem Classification 3: The video visual experience is not good", and "Problem Classification 4: Clickbait titles".

[0143] Among them, the weekly active information of the user is used to represent the activity information of the user in the most recent week. For example, the weekly active information can include the active situation of the user in the most recent week (such as online duration), the advertisement density of the user in the most recent week, the short video ratio of the user in the most recent week (short video means short-time playback), and the distribution of video understanding results of the user in the most recent week (video understanding can include low visual experience, clickbait, title party, fraud, etc.).

[0144] In step 303 above, the server can obtain the weights of each feedback behavior, add up the products obtained by multiplying the predicted probability of each feedback behavior by the weight of each feedback behavior itself, to get the weighted sum value of each predicted probability. When the weighted sum value is greater than the target probability threshold, it is determined that a pop-up window for collecting feedback information needs to be triggered. At this time, the weekly active information of the user is input into the multi-classification model, and the multi-classification model performs weighted processing on the weekly active information and outputs the feedback problem category corresponding to the user.

[0145] Optionally, the multi-classification model can be GB (Gradient Boosting), GBDT (Gradient Boosting Decision Tree), XGBoost (eXtreme Gradient Boosting), etc.

[0146] In an exemplary scenario, for 3 different feedback behaviors, namely "Normal subsequent viewing", "Reduce subsequent viewing", and "Exit the application", their corresponding weights and predicted probabilities are shown in Table 1 below.

[0147] Table 1

[0148] Feedback behavior Number Weight Predicted probability Normal subsequent viewing 1 W1 P1 Reduced subsequent viewing 2 W2 P2 Exit the application 3 W3 P3

[0149] Assuming the target probability threshold is η, then the target probability condition can be expressed by the following formula:

[0150]

[0151] Among them, W k represents the weight of the k-th feedback behavior, and P k represents the predicted probability of the k-th feedback behavior, where 1 ≤ k ≤ 3.

[0152] In the above process, since the problems that damage the user experience felt by different users are not the same, by using this multi-classification model to predict the feedback problem category that best matches the user, the most appropriate research question can be selected according to the feedback problem category, so as to display the research question in the pop-up window (i.e., the feedback information collection window), and targeted pop-up questions can be asked for the problems that the user is most likely to encounter, so that the most effective feedback information can be collected.

[0153] It should be noted that before the server calls the multi-classification model to classify the feedback problem category, the multi-classification model can be trained in the following way: in the early stage, a large amount of feedback information of users is collected by randomly distributing questionnaires, and users are prompted in the questionnaires to select the problem category with the most serious problem according to their recent subjective feelings. Based on the questionnaire feedback results, the weekly active information of each user and the problem category selected by each user in the questionnaire are statistically analyzed, and a set of training samples can be obtained, denoted as {X i , y i}, 1≤i≤N where X i represents the weekly active information of the i-th user, and y i represents the problem category selected by the i-th user in the questionnaire, N is the sample size of the users, and the above multi-classification model can be trained based on the statistically obtained training samples.

[0154] In step 304, the server determines at least one research question under the feedback problem category based on the predicted feedback problem category, and adds the at least one research question to the target page resource.

[0155] Among them, at least one research question can correspond to each feedback problem category, and the feedback problem category and the research question can be stored in the database correspondingly according to the mapping relationship between the feedback problem category and the research question.

[0156] Among them, the target page resource is used to provide the page layout information of the collection window for collecting feedback information and the at least one research question.

[0157] In the above process, the server takes the predicted feedback problem category as an index, queries the database to check whether there is index content corresponding to the index, and can obtain some or all of the research questions in the index content as the at least one research question, and add the at least one research question to the target page resource.

[0158] Optionally, all the research questions can be obtained as the at least one research question, or, alternatively, a target number of research questions can be randomly selected from all the research questions and obtained as the at least one research question. The embodiments of the present disclosure do not specifically limit the manner of obtaining the research questions.

[0159] In the above steps 303-304, the server determines at least one research question for the user according to the user's weekly activity information, and then adds the at least one research question to the target page resource. In some embodiments, after training the viewing experience model, the feedback behavior model, and the multi-classification model, the server can also send the viewing experience model, the feedback behavior model, and the multi-classification model to the terminal, and the terminal automatically determines whether the target probability condition is reached and obtains the research questions from the cloud database, thereby reducing the load on the server.

[0160] In step 305, the server sends the target page resource to the terminal corresponding to the user, and the target page resource is used to trigger the terminal to feedback the collected feedback information after collecting the feedback information.

[0161] In the above process, after obtaining at least one research question based on the above step 304, the server can also obtain the page layout information of the collection window for collecting feedback information. The page layout information can be randomly selected from one or more pre-stored template groups in the server, such as vertical arrangement, horizontal arrangement, etc. After obtaining the at least one research question and the page layout information, the at least one research question and the page layout information are encapsulated as the target page resource.

[0162] In step 306, the terminal receives the target page resource sent by the server when the predicted probability meets the target probability condition.

[0163] The predicted probability is obtained by the server according to the user information of the user corresponding to the terminal, the content item information of at least one played content item, and the experience prediction value of the user for the at least one content item.

[0164] In the above process, when the terminal receives the target page resource, it can parse the target page resource to obtain the page layout information and the at least one research question.

[0165] In step 307, the terminal displays a collection window for collecting feedback information based on the target page resource.

[0166] The layout of the collection window is determined based on the page layout information, and the at least one research question is displayed in the collection window.

[0167] In the above process, after receiving the target page resource, the terminal parses to obtain the page layout information and the at least one research question. It can determine the layout of the collection window according to the page layout information, determine the target area for displaying the research question in the layout, and then pop up a collection window for display based on this layout in the application, and display the at least one research question in the target area of the collection window, so as to be able to interact with the user in real time in the form of a pop-up window.

[0168] In some embodiments, the terminal can also automatically jump to the feedback information collection window after the application finishes playing the current video, and display the at least one research question in the target area of the collection window, so as to avoid disturbing the video being watched by the user and avoid the experience damage caused by collecting feedback information.

[0169] In step 308, the terminal collects the feedback information of the user corresponding to the terminal based on the collection window.

[0170] In the above process, the user can input feedback information in the collection window. For example, the user clicks on the feedback options for each research question in the collection window. After the user finishes answering, the terminal completes the collection of the feedback information. Thereafter, the following step 309 can be executed to send the collected feedback information to the server for unified data analysis.

[0171] In step 309, the terminal sends the collected feedback information to the server.

[0172] In the above process, the terminal can compress, encrypt, and package the collected feedback information into a resource package and send the resource package to the server.

[0173] In the related art, usually when the application is first launched by the user every day, a randomly sampled user is triggered to display a pop-up window. For the negative feedback behavior of advertisements, when the user just feels the experience damage, they can express their negative feedback information most clearly. However, this kind of pop-up window triggered in the mode of specific time and specific behavior obviously cannot capture the moment when the user's negative feedback willingness is the strongest in time, and the research questions displayed in the pop-up window are also the same and cannot be customized for different users with personalized research questions.

[0174] In the embodiments of the present disclosure, instead of adopting the acquisition method combining random sampling and periodic duplicate removal, among all users, users who need to pop up a window are selected according to the target probability condition. The weighted sum value of the prediction probabilities of these users is greater than the target probability threshold, which means that these users are usually users whose user experience has been damaged or users who may be lost due to experience damage. These users are the ones who urgently need to collect feedback information. By quantifying the user's experience damage (quantified by the proportion of playback duration), the probability of user loss can be considered. When the possibility of a certain degree of loss or the experience damage reaches a certain threshold, a pop-up window is automatically displayed, and based on a multi-classification model, the category of feedback problems is determined, and preset research questions under the corresponding feedback problem category are displayed in the pop-up window. In this way, the three major problems of "which people to select for negative feedback pop-up window", "when to perform negative feedback pop-up window", and "what problems to display in the negative feedback pop-up window" can be solved, achieving high-efficiency and high-accuracy acquisition of feedback information.

[0175] The method provided by the embodiments of the present disclosure obtains the experience prediction value of the user for the played content item through the user information of the user and the content item information of the played content item. According to the user information, the content item information, and the experience prediction value, the prediction probability of the user generating various feedback behaviors is obtained. When the prediction probability meets the target probability condition, a target page resource is sent to the terminal, triggering the terminal to return the collected feedback information after receiving the target page resource. By controlling different target probability conditions, the timing of sending the target page resource to the terminal is controlled, which is equivalent to controlling the pop-up condition of the feedback information acquisition window on the terminal side. This acquisition method that does not rely on random sampling and periodic duplicate removal can automatically, accurately, and intelligently control when to send the target page resource to the terminals corresponding to which users, so that after the terminal collects feedback information according to the target page resource and returns the collected feedback information to the electronic device, the acquisition efficiency and accuracy of the feedback information can be greatly improved.

[0176] Figure 4 It is a logical structural block diagram of a feedback information acquisition device shown according to an exemplary embodiment. Refer to Figure 4 and the device includes a first acquisition unit 401, a second acquisition unit 402, and a sending unit 403.

[0177] The first acquisition unit 401 is configured to execute obtaining the experience prediction value of the user for the at least one played content item according to the user information of the user and the content item information of the at least one played content item. An experience prediction value is used to represent the viewing experience of the user for one played content item.

[0178] The second acquisition unit 402 is configured to obtain a predicted probability that the user generates at least one feedback behavior according to the user information, the content item information, and the experience prediction value, where a predicted probability is used to characterize the possibility that the user generates a feedback behavior;

[0179] The sending unit 403 is configured to, in response to the predicted probability meeting the target probability condition, send a target page resource to the terminal corresponding to the user, where the target page resource is used to trigger the terminal to return the collected feedback information after collecting the feedback information.

[0180] The device provided in the embodiment of the present disclosure obtains an experience prediction value of the user for the played content item through the user information of the user and the content item information of the played content item, obtains the predicted probability that the user generates various feedback behaviors according to the user information, the content item information, and the experience prediction value, and when the predicted probability meets the target probability condition, sends a target page resource to the terminal, triggering the terminal to return the collected feedback information after receiving the target page resource. By controlling different target probability conditions, the timing of sending the target page resource to the terminal is controlled, which is equivalent to controlling the pop-up condition of the feedback information collection window on the terminal side. This collection method that does not rely on random sampling and periodic deduplication can automatically, accurately, and intelligently control when to send the target page resource to the terminals corresponding to which users, so that after the terminal collects the feedback information according to the target page resource and returns the collected feedback information to the electronic device. Therefore, the acquisition efficiency and accuracy of the feedback information can be greatly improved.

[0181] In a possible implementation manner, the second acquisition unit 402 is configured to perform:

[0182] Input the user information, the content item information, and the experience prediction value into a feedback behavior model, and perform weighted processing on the user, the content item information, and the experience prediction value through the feedback behavior model to respectively obtain the predicted probability that the user generates the at least one feedback behavior.

[0183] In a possible implementation manner, the at least one feedback behavior includes at least one of normal subsequent viewing, reduced subsequent viewing, or exiting the application; the target probability condition is that the weighted sum value of the predicted probabilities of the at least one feedback behavior is greater than a target probability threshold.

[0184] In a possible implementation manner, based on Figure 4 the composition of the device, the device further includes:

[0185] The first training unit is configured to execute: obtaining sample user information of a sample user, sample content item information of at least one played sample content item, sample experience prediction values of the sample user for the at least one sample content item, and historical behaviors; training an initial behavior model based on the sample user information, the sample content item information, the sample experience prediction values, and the historical behaviors to obtain the feedback behavior model.

[0186] In a possible implementation manner, the first obtaining unit 401 is configured to execute:

[0187] Input the user information and the content item information into a viewing experience model, perform weighted processing on the user information and the content item information through the viewing experience model, respectively predict the proportion of the playing duration of the user for the at least one content item, and determine the proportion of the playing duration as the experience prediction value, where a proportion of the playing duration is used to represent the ratio between the predicted playing duration of the user for a content item and the total duration of the content item.

[0188] In a possible implementation manner, based on Figure 4 the device composition, the device further includes:

[0189] The second training unit is configured to execute: obtaining sample user information of a sample user and sample content item information of at least one played sample content item; screening out positive sample content items with a proportion of playing duration greater than a first target threshold and negative sample content items with a proportion of playing duration less than a second target threshold from the at least one sample content item, where the first target threshold is greater than or equal to the second target threshold; training an initial experience model based on the sample user information, the sample content item information of the positive sample content items, and the sample content item information of the negative sample content items to obtain the viewing experience model.

[0190] In a possible implementation manner, based on Figure 4 the device composition, the device further includes:

[0191] The determination unit is configured to execute: determining at least one research question of the user according to the weekly active information of the user;

[0192] The addition unit is configured to execute: adding the at least one research question to the target page resource.

[0193] In a possible implementation manner, the determination unit is configured to execute:

[0194] Input the weekly active information of the user into a multi-classification model, and predict the feedback question category to which the user belongs through the multi-classification model, where the feedback question category is used to characterize the category of harm suffered by the user experience during the viewing of the content item;

[0195] Based on the predicted feedback problem categories, determine at least one research question under the feedback problem category.

[0196] Regarding the device in the above embodiments, the specific manner in which each unit performs operations has been described in detail in the embodiments of the method for obtaining the feedback information, and will not be elaborated here.

[0197] Figure 5 It is a logical structural block diagram of a device for obtaining feedback information shown according to an exemplary embodiment. Refer to Figure 5 , the device includes a receiving unit 501, a display unit 502, a collection unit 503, and a sending unit 504.

[0198] The receiving unit 501 is configured to receive the target page resource sent by the server when the prediction probability meets the target probability condition, and the prediction probability is obtained by the server according to the user information of the user corresponding to the terminal, the content item information of at least one played content item, and the experience prediction value of the user for the at least one content item;

[0199] The display unit 502 is configured to display a collection window for collecting feedback information based on the target page resource;

[0200] The collection unit 503 is configured to collect the feedback information of the user corresponding to the terminal based on the collection window;

[0201] The sending unit 504 is configured to send the collected feedback information to the server.

[0202] The device provided by the embodiments of the present disclosure receives the target page resource sent by the server when the prediction probability meets the target probability condition. Since the prediction probability is obtained by the server based on the user information of the user corresponding to the terminal, the content item information of at least one played content item, and the experience prediction value of the user for the at least one content item, that is to say, the server can control different target probability conditions, thereby controlling the timing of sending the target page resource to the terminal, which is equivalent to controlling the pop-up condition of the acquisition window for collecting feedback information on the terminal side. The terminal displays an acquisition window for collecting feedback information based on the target page resource, collects the feedback information of the user corresponding to the terminal based on the acquisition window, and sends the collected feedback information to the server. This acquisition method that is not based on random sampling and periodic deduplication can automatically, accurately, and intelligently control the server to select when and to which terminals corresponding to users to send the target page resource, that is, control when the terminal will receive the target page resource, so that after the terminal collects feedback information based on the target page resource, the collected feedback information is returned to the server. Therefore, the acquisition efficiency and accuracy of the feedback information can be greatly improved.

[0203] Regarding the device in the above embodiment, the specific manner in which each unit performs operations has been described in detail in the embodiment of the method for obtaining the feedback information, and will not be elaborated here.

[0204] Figure 6 The block diagram of a terminal provided by an exemplary embodiment of the present disclosure is shown. The terminal is also an example of an electronic device. The terminal 600 may be: a smart phone, a tablet computer, an MP3 player (Moving Picture Experts Group Audio Layer III), an MP4 (Moving Picture Experts Group Audio Layer IV) player, a laptop computer, or a desktop computer. The terminal 600 may also be referred to by other names such as user equipment, portable terminal, laptop terminal, desktop terminal, etc.

[0205] Generally, the terminal 600 includes: a processor 601 and a memory 602.

[0206] The processor 601 may include one or more processing cores, such as a quad-core processor, an octa-core processor, etc. The processor 601 may be implemented in at least one hardware form of DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), or PLA (Programmable Logic Array). The processor 601 may also include a main processor and a coprocessor. The main processor is a processor used to process data in the wake state, also known as the CPU (Central Processing Unit); the coprocessor is a low-power processor used to process data in the standby state. In some embodiments, the processor 601 may be integrated with a GPU (Graphics Processing Unit), and the GPU is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 601 may further include an AI (Artificial Intelligence) processor, and the AI processor is used to process computational operations related to machine learning.

[0207] The memory 602 may include one or more computer-readable storage media, and the computer-readable storage media may be non-transitory. The memory 602 may further include high-speed random access memory and non-volatile memory, such as one or more disk storage devices and flash storage devices. In some embodiments, the non-transitory computer-readable storage media in the memory 602 is used to store at least one instruction, and the at least one instruction is used to be executed by the processor 601 to implement the method for obtaining feedback information provided in various embodiments of the present disclosure.

[0208] In some embodiments, the terminal 600 may further optionally include: a peripheral device interface 603 and at least one peripheral device. The processor 601, the memory 602, and the peripheral device interface 603 may be connected through a bus or signal lines. Each peripheral device may be connected to the peripheral device interface 603 through a bus, signal lines, or a circuit board. Specifically, the peripheral devices include at least one of a radio frequency circuit 604, a touch display screen 605, a camera assembly 606, an audio circuit 607, a positioning assembly 608, and a power supply 609.

[0209] The peripheral device interface 603 can be used to connect at least one I / O (Input / Output) related peripheral device to the processor 601 and the memory 602. In some embodiments, the processor 601, the memory 602, and the peripheral device interface 603 are integrated on the same chip or circuit board; in some other embodiments, any one or two of the processor 601, the memory 602, and the peripheral device interface 603 can be implemented on separate chips or circuit boards, and this embodiment does not limit this.

[0210] The radio frequency circuit 604 is used to receive and transmit RF (Radio Frequency) signals, also known as electromagnetic signals. The radio frequency circuit 604 communicates with the communication network and other communication devices through electromagnetic signals. The radio frequency circuit 604 converts an electrical signal into an electromagnetic signal for transmission, or converts the received electromagnetic signal into an electrical signal. Optionally, the radio frequency circuit 604 includes: an antenna system, an RF transceiver, one or more amplifiers, a tuner, an oscillator, a digital signal processor, a codec chipset, a subscriber identity module card, and so on. The radio frequency circuit 604 can communicate with other terminals through at least one wireless communication protocol. The wireless communication protocol includes but is not limited to: metropolitan area network, generations of mobile communication networks (2G, 3G, 4G, and 5G), wireless local area network, and / or WiFi (Wireless Fidelity) network. In some embodiments, the radio frequency circuit 604 may further include a circuit related to NFC (Near Field Communication), and this disclosure does not limit this.

[0211] The display screen 605 is used to display the UI (User Interface). The UI may include graphics, text, icons, videos, and any combination thereof. When the display screen 605 is a touch display screen, the display screen 605 also has the ability to collect touch signals on or above the surface of the display screen 605. The touch signals can be input as control signals to the processor 601 for processing. At this time, the display screen 605 can also be used to provide virtual buttons and / or a virtual keyboard, also known as soft buttons and / or a soft keyboard. In some embodiments, there may be one display screen 605, which is provided on the front panel of the terminal 600; in other embodiments, there may be at least two display screens 605, which are respectively provided on different surfaces of the terminal 600 or are in a foldable design; in still other embodiments, the display screen 605 may be a flexible display screen, which is provided on the curved surface or the folding surface of the terminal 600. Even more, the display screen 605 can also be set to an irregular non-rectangular shape, that is, an irregular-shaped screen. The display screen 605 can be prepared using materials such as LCD (Liquid Crystal Display) and OLED (Organic Light-Emitting Diode).

[0212] The camera module 606 is used to capture images or videos. Optionally, the camera module 606 includes a front camera and a rear camera. Generally, the front camera is provided on the front panel of the terminal, and the rear camera is provided on the back of the terminal. In some embodiments, there are at least two rear cameras, which are any one of a main camera, a depth-of-field camera, a wide-angle camera, and a telephoto camera, to achieve functions such as background blurring by fusing the main camera and the depth-of-field camera, panoramic shooting by fusing the main camera and the wide-angle camera, and VR (Virtual Reality) shooting function or other fused shooting functions. In some embodiments, the camera module 606 may also include a flash. The flash can be a single-color temperature flash or a two-color temperature flash. A two-color temperature flash refers to a combination of a warm light flash and a cold light flash, which can be used for light compensation under different color temperatures.

[0213] The audio circuit 607 may include a microphone and a speaker. The microphone is used to collect sound waves of the user and the environment, and convert the sound waves into electrical signals for input to the processor 601 for processing, or input to the radio frequency circuit 604 to achieve voice communication. For the purpose of stereo collection or noise reduction, there may be multiple microphones, which are respectively arranged at different parts of the terminal 600. The microphone may also be an array microphone or an omnidirectional collection microphone. The speaker is used to convert the electrical signal from the processor 601 or the radio frequency circuit 604 into sound waves. The speaker may be a traditional thin film speaker or a piezoelectric ceramic speaker. When the speaker is a piezoelectric ceramic speaker, it can not only convert the electrical signal into sound waves audible to humans, but also convert the electrical signal into sound waves inaudible to humans for uses such as ranging. In some embodiments, the audio circuit 607 may further include a headphone jack.

[0214] The positioning component 608 is used to locate the current geographical location of the terminal 600 to achieve navigation or LBS (Location Based Service). The positioning component 608 may be a positioning component based on the GPS (Global Positioning System) of the United States, the Beidou system of China, the GLONASS system of Russia, or the Galileo system of the European Union.

[0215] The power supply 609 is used to supply power to each component in the terminal 600. The power supply 609 may be alternating current, direct current, a disposable battery, or a rechargeable battery. When the power supply 609 includes a rechargeable battery, the rechargeable battery may support wired charging or wireless charging. The rechargeable battery may also be used to support fast charging technology.

[0216] In some embodiments, the terminal 600 further includes one or more sensors 610. The one or more sensors 610 include but are not limited to: an acceleration sensor 611, a gyroscope sensor 612, a pressure sensor 613, a fingerprint sensor 614, an optical sensor 615, and a proximity sensor 616.

[0217] The acceleration sensor 611 can detect the magnitude of acceleration on the three coordinate axes of the coordinate system established with the terminal 600. For example, the acceleration sensor 611 can be used to detect the components of the gravitational acceleration on the three coordinate axes. The processor 601 can control the touch display screen 605 to display the user interface in a landscape view or a portrait view according to the gravitational acceleration signal collected by the acceleration sensor 611. The acceleration sensor 611 can also be used for collecting game or user's motion data.

[0218] The gyroscope sensor 612 can detect the body direction and rotation angle of the terminal 600. The gyroscope sensor 612 can cooperate with the acceleration sensor 611 to collect the 3D actions of the user on the terminal 600. Based on the data collected by the gyroscope sensor 612, the processor 601 can implement the following functions: motion sensing (such as changing the UI according to the user's tilting operation), image stabilization during shooting, game control, and inertial navigation.

[0219] The pressure sensor 613 can be disposed on the side frame of the terminal 600 and / or the lower layer of the touch display screen 605. When the pressure sensor 613 is disposed on the side frame of the terminal 600, it can detect the holding signal of the user on the terminal 600, and the processor 601 can perform left / right hand recognition or shortcut operations according to the holding signal collected by the pressure sensor 613. When the pressure sensor 613 is disposed on the lower layer of the touch display screen 605, the processor 601 can control the operable controls on the UI interface according to the pressure operation of the user on the touch display screen 605. The operable controls include at least one of button controls, scroll bar controls, icon controls, and menu controls.

[0220] The fingerprint sensor 614 is used to collect the fingerprint of the user. The processor 601 can identify the user's identity according to the fingerprint collected by the fingerprint sensor 614, or the fingerprint sensor 614 can identify the user's identity according to the collected fingerprint. When the identity of the user is identified as a trusted identity, the processor 601 authorizes the user to perform relevant sensitive operations, and the sensitive operations include unlocking the screen, viewing encrypted information, downloading software, making payments, and changing settings, etc. The fingerprint sensor 614 can be disposed on the front, back, or side of the terminal 600. When there are physical buttons or manufacturer logos on the terminal 600, the fingerprint sensor 614 can be integrated with the physical buttons or manufacturer logos.

[0221] The optical sensor 615 is used to collect the ambient light intensity. In one embodiment, the processor 601 can control the display brightness of the touch display screen 605 according to the ambient light intensity collected by the optical sensor 615. Specifically, when the ambient light intensity is high, the display brightness of the touch display screen 605 is increased; when the ambient light intensity is low, the display brightness of the touch display screen 605 is decreased. In another embodiment, the processor 601 can also dynamically adjust the shooting parameters of the camera module 606 according to the ambient light intensity collected by the optical sensor 615.

[0222] The proximity sensor 616, also known as the distance sensor, is typically disposed on the front panel of the terminal 600. The proximity sensor 616 is used to collect the distance between the user and the front of the terminal 600. In one embodiment, when the proximity sensor 616 detects that the distance between the user and the front of the terminal 600 is gradually decreasing, the touch display screen 605 is controlled by the processor 601 to switch from the lit screen state to the off-screen state; when the proximity sensor 616 detects that the distance between the user and the front of the terminal 600 is gradually increasing, the touch display screen 605 is controlled by the processor 601 to switch from the off-screen state to the lit screen state.

[0223] Those skilled in the art can understand that Figure 5 the structure shown in does not constitute a limitation on the terminal 600, and may include more or fewer components than shown in the figure, or combine certain components, or adopt different component arrangements.

[0224] Figure 7 is a schematic structural diagram of a server provided by an embodiment of the present disclosure. The server is also an example of an electronic device. The server 700 may vary greatly due to different configurations or performances, and may include one or more processors (Central Processing Units, CPUs) 701 and one or more memories 702. Among them, at least one program code is stored in the memory 702, and the at least one program code is loaded and executed by the processor 701 to implement the method for obtaining feedback information provided in the above various embodiments. Of course, the server 700 may also have components such as a wired or wireless network interface, a keyboard, and an input / output interface for input / output. The server 700 may also include other components for implementing the functions of the device, which will not be elaborated here.

[0225] In an exemplary embodiment, a storage medium including at least one instruction is also provided, such as a memory including at least one instruction. The at least one instruction can be executed by a processor in an electronic device to complete the method for obtaining feedback information in the above embodiments. Optionally, the above storage medium may be a non-transitory computer-readable storage medium. For example, the non-transitory computer-readable storage medium may include a ROM (Read-Only Memory), a RAM (Random-Access Memory), a CD-ROM (Compact Disc Read-Only Memory), a magnetic tape, a floppy disk, and an optical data storage device, etc.

[0226] In an exemplary embodiment, there is also provided a computer program product including one or more instructions that can be executed by a processor of an electronic device to implement the method for obtaining feedback information provided in each of the foregoing embodiments.

[0227] Those skilled in the art will readily conceive of other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. The present disclosure is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include known common general knowledge or conventional technical means in the technical field not disclosed by the present disclosure. The specification and embodiments are only to be regarded as exemplary, and the true scope and spirit of the present disclosure are pointed out by the following claims.

[0228] It should be understood that the present disclosure is not limited to the exact structures described above and shown in the drawings, and various modifications and changes can be made without departing from its scope. The scope of the present disclosure is only limited by the appended claims.

Claims

1. A method for obtaining feedback information, characterized in that, comprising: By using a viewing experience model to perform weighted processing on the user characteristics of the user and the content item characteristics of the content item information of at least one played content item, respectively predicting the proportion of the playback duration of the at least one content item by the user, and determining the proportion of the playback duration as an experience prediction value. A proportion of the playback duration is used to represent the ratio between the predicted playback duration of the user for a content item and the total duration of the content item, and an experience prediction value is used to characterize the viewing experience of the user for a played content item. The viewing experience model is obtained by the server iteratively training an initial experience model based on positive sample content items and negative sample content items; According to the user characteristics, the content item characteristics of the content item information, and the viewing experience characteristics of the experience prediction value, obtaining the prediction probability that the user generates at least one feedback behavior. A prediction probability is used to characterize the possibility that the user generates a feedback behavior; In response to the prediction probability meeting the target probability condition, predicting the feedback problem category to which the user belongs through a multi-classification model. The feedback problem category is used to characterize the category of the user's negative viewing experience during the viewing of the content item; based on the predicted feedback problem category, determining at least one research question under the feedback problem category; Adding the at least one research question to a target page resource, and sending the target page resource to the terminal corresponding to the user. The target page resource is used to trigger the terminal to return the collected feedback information after collecting the feedback information. The target probability condition is that the weighted sum value of the prediction probabilities of the at least one feedback behavior is greater than a target probability threshold, and different levels of numerical values are set for the target probability threshold.

2. The method for obtaining feedback information according to claim 1, characterized in that, The obtaining the prediction probability that the user generates at least one feedback behavior according to the user characteristics, the content item characteristics of the content item information, and the viewing experience characteristics of the experience prediction value includes: Inputting the user characteristics, the content item characteristics of the content item information, and the viewing experience characteristics of the experience prediction value into a feedback behavior model, and performing weighted processing on the user characteristics, the content item characteristics of the content item information, and the viewing experience characteristics of the experience prediction value through the feedback behavior model, respectively obtaining the prediction probability that the user generates the at least one feedback behavior.

3. The method for obtaining feedback information according to claim 1 or 2, characterized in that, The at least one feedback behavior includes at least one of normal subsequent viewing, reduced subsequent viewing, or exiting the application.

4. The method for obtaining feedback information according to claim 1 or 2, characterized in that, The method further includes: Obtaining the sample user information of the sample user, the sample content item information of at least one played sample content item, the sample experience prediction value of the sample user for the at least one sample content item, and the historical behavior; Train an initial behavior model based on the sample user information, the sample content item information, the sample experience prediction value, and the historical behavior to obtain a feedback behavior model.

5. The method for obtaining feedback information according to claim 1, wherein, the method further includes: Obtain the sample user information of the sample user and the sample content item information of at least one played sample content item; From the at least one sample content item, screen out positive sample content items with a playback duration ratio greater than a first target threshold and negative sample content items with a playback duration ratio less than a second target threshold, where the first target threshold is greater than or equal to the second target threshold; Train an initial experience model based on the sample user information, the sample content item information of the positive sample content items, and the sample content item information of the negative sample content items to obtain the viewing experience model.

6. A method for obtaining feedback information, wherein, applied to a terminal, includes: Receive a target page resource sent by the server when the prediction probability meets the target probability condition, where the prediction probability is obtained by the server based on the user information of the user corresponding to the terminal, the content item information of at least one played content item, and the experience prediction value of the user for the at least one content item; Based on the target page resource, display a collection window for collecting feedback information, where the target page resource includes at least one research question under the feedback problem category to which the user belongs, and the feedback problem category is predicted based on a multi-classification model, and the feedback problem category is used to characterize the category of the user's negative viewing experience during the viewing of the content item; Collect the feedback information of the user corresponding to the terminal based on the collection window; Send the collected feedback information to the server; wherein, the experience prediction value is the playback duration ratio of the user for at least one content item respectively predicted by the viewing experience model through weighted processing of the user characteristics of the user and the content type characteristics of the content item information of at least one played content item, and one playback duration ratio is used to represent the ratio of the predicted playback duration of the user for a content item to the total duration of the content item; the viewing experience model is iteratively trained by the server based on the positive sample content items and the negative sample content items for the initial experience model; the target probability condition is that the weighted sum value of the prediction probabilities of at least one feedback behavior is greater than the target probability threshold, and different levels of values are set for the target probability threshold.

7. A device for obtaining feedback information, wherein, includes: A first acquisition unit, configured to perform weighted processing on the user characteristics of a user and the content item characteristics of at least one played content item through a viewing experience model, respectively predict the proportion of the playback duration of the at least one content item by the user, and determine the proportion of the playback duration as an experience prediction value. A proportion of the playback duration is used to represent the ratio between the predicted playback duration of a content item by the user and the total duration of the content item, and an experience prediction value is used to characterize the viewing experience of the user for a played content item. The viewing experience model is obtained by the server through iterative training of an initial experience model based on positive sample content items and negative sample content items; A second acquisition unit, configured to perform obtaining the prediction probability of the user generating at least one feedback behavior according to the user characteristics, the content item characteristics of the content item information, and the viewing experience characteristics of the experience prediction value. A prediction probability is used to characterize the possibility of the user generating a feedback behavior; A determination unit, configured to perform predicting the feedback problem category to which the user belongs through a multi-classification model. The feedback problem category is used to characterize the category of the user's negative viewing experience during the viewing of the content item; based on the predicted feedback problem category, determining at least one research question under the feedback problem category; An addition unit, configured to perform adding the at least one research question to a target page resource; A sending unit, configured to perform sending the target page resource to the terminal corresponding to the user in response to the prediction probability meeting a target probability condition. The target page resource is used to trigger the terminal to return the collected feedback information after collecting the feedback information. The target probability condition is that the weighted sum value of the prediction probabilities of the at least one feedback behavior is greater than a target probability threshold, and the target probability threshold is set with numerical values at different levels.

8. The feedback information acquisition device according to claim 7, wherein, the second acquisition unit is configured to perform: Inputting the user information, the content item information, and the experience prediction value into a feedback behavior model, and performing weighted processing on the user, the content item information, and the experience prediction value through the feedback behavior model to respectively obtain the prediction probability of the user generating the at least one feedback behavior.

9. The feedback information acquisition device according to claim 7 or 8, wherein, the at least one feedback behavior includes at least one of normal subsequent viewing, reduced subsequent viewing, or exiting the application; and the target probability condition is that the weighted sum value of the prediction probabilities of the at least one feedback behavior is greater than a target probability threshold.

10. The feedback information acquisition device according to claim 7 or 8, wherein, the device further includes: The first training unit is configured to obtain sample user information of a sample user, sample content item information of at least one played sample content item, sample experience prediction values of the sample user for the at least one sample content item, and historical behaviors; and train an initial behavior model based on the sample user information, the sample content item information, the sample experience prediction values, and the historical behaviors to obtain a feedback behavior model.

11. The feedback information acquisition device according to claim 7, wherein, the device further includes: A second training unit configured to obtain sample user information of a sample user and sample content item information of at least one played sample content item; screen out positive sample content items with a playback duration ratio greater than a first target threshold and negative sample content items with a playback duration ratio less than a second target threshold from the at least one sample content item, where the first target threshold is greater than or equal to the second target threshold; and train an initial experience model based on the sample user information, the sample content item information of the positive sample content items, and the sample content item information of the negative sample content items to obtain the viewing experience model.

12. A feedback information acquisition device, wherein, it includes: A receiving unit configured to receive a target page resource sent by a server when a prediction probability meets a target probability condition, where the prediction probability is obtained by the server based on user information of a user corresponding to the terminal, content item information of at least one played content item, and experience prediction values of the user for the at least one content item; A display unit configured to display a collection window for collecting feedback information based on the target page resource, where the target page resource includes at least one research question under a feedback problem category to which the user belongs, and the feedback problem category is predicted based on a multi-classification model and is used to characterize the category of the user's negative viewing experience during the viewing of the content item; A collection unit configured to collect feedback information of the user corresponding to the terminal based on the collection window; A sending unit configured to send the collected feedback information to the server; wherein, the experience prediction value is a playback duration ratio of the user for at least one content item respectively predicted by a viewing experience model through weighted processing of user characteristics of the user and content type characteristics of content item information of at least one played content item, and one playback duration ratio is used to represent the ratio of the expected playback duration of the user for a content item to the total duration of the content item; The viewing experience model is obtained by the server through iterative training of an initial experience model based on positive sample content items and negative sample content items; the target probability condition is that the weighted sum value of the prediction probabilities of at least one feedback behavior is greater than a target probability threshold, and different levels of values are set for the target probability threshold.

13. An electronic device, wherein, it includes: One or more processors; One or more memories for storing executable instructions of the one or more processors; Wherein, the one or more processors are configured to execute the instructions to implement the method for obtaining feedback information as described in any one of claims 1 to 5 or claim 6.

14. A storage medium, characterized in that when at least one instruction in the storage medium is executed by one or more processors of an electronic device, the electronic device is enabled to execute the method for obtaining feedback information as described in any one of claims 1 to 5 or claim 6.

15. A computer program product, characterized in that it includes at least one instruction, and the at least one instruction is executed by one or more processors of an electronic device, so that the electronic device is enabled to execute the method for obtaining feedback information as described in any one of claims 1 to 5 or claim 6.

Images