Processing method for performing end-side control on recommended content and electronic equipment

Abstract

The embodiment of the invention discloses a processing method for performing end-side control on recommended content and electronic equipment, and the method comprises the steps: obtaining an end-side signal generated by terminal equipment in a user interaction process in a process of orderly playing the recommended content; according to the end side signal, predicting real-time emotion of the user in the process of watching the recommended content and / or behaviors possibly executed on the current recommended content; and performing recommendation control on the recommended content according to the predicted real-time emotion and / or behavior. According to the embodiment of the invention, end-side intelligent intervention on the recommended content based on the end-side signal can be realized.

Description

Technical Field

[0001] The present application relates to the technical field of content recommendation, and in particular to a processing method and electronic device for performing terminal-side control of recommended content. Background Art

[0002] In the content service system, content can be recommended to users. In a typical scenario, the specific content may be short video content. During the playback process, each short video can be played in full screen. After a short video is played, the next short video can be played automatically. Or, during the playback of a short video, if the user performs operations such as sliding up and down, the next short video can be switched to play, and so on.

[0003] In the above content recommendation process, the traditional method is usually to extract user characteristics and content characteristics, select appropriate recommendation models (such as collaborative filtering and neural networks, etc.) for modeling, and then train the model on massive user historical behavior data to determine the match between users and content. Among them, the perception of users is mainly based on labels such as user age, gender, location and historical interests. Summary of the invention

[0004] The present application provides a processing method and electronic device for performing terminal-side control of recommended content, which can implement terminal-side intelligent intervention on recommended content based on terminal-side signals.

[0005] This application provides the following solutions:

[0006] A method for performing terminal-side control on recommended content includes:

[0007] In the process of orderly playing the recommended content, obtaining the terminal side signal generated by the terminal device during the user interaction process;

[0008] Predicting, based on the terminal-side signal, the real-time emotions of the user during viewing the recommended content and / or the possible behaviors performed on the current recommended content;

[0009] The recommended content is controlled based on the predicted real-time emotion and / or behavior.

[0010] Among them, the terminal side signal includes physical parameter information of user gesture signals, heart rate signals, breathing signals, skin electricity signals, eye contact signals, myoelectric signals, skin temperature signals, electroencephalogram signals, and electrocardiogram signals captured by the terminal device sensor, which are used to input into the perception model, and the perception model predicts the user's real-time emotions and / or behaviors.

[0011] Among them, it also includes:

[0012] Based on the terminal side signal generated by the terminal device sensor, the user's spatiotemporal environment context is perceived, and the perceived spatiotemporal environment context information is input into the perception model for predicting the real-time emotion and / or behavior.

[0013] Among them, it also includes:

[0014] Based on the terminal-side signal generated by the terminal device sensor, the gesture category and / or body posture category performed by the user when playing the content is perceived, and the perceived gesture category and / or body posture category information is input into the perception model for predicting the real-time emotion and / or behavior.

[0015] The predicting of the emotions of the user during viewing the recommended content and / or the possible behaviors of the user on the current recommended content includes:

[0016] The emotions and behaviors are predicted using a perception model of a multi-task classification structure, wherein the perception model of the multi-task classification framework is generated by jointly training an emotion prediction task and a behavior prediction task.

[0017] In the process of predicting the emotions and behaviors using the perception model of the multi-task classification structure, the intermediate state prediction information of the real-time emotions and behaviors is input into the perception model so that the prediction information of emotions and behaviors affects each other's prediction results.

[0018] The recommended content includes: a mixed ordered set determined based on multiple short video contents pushed by the server to the client, wherein the mixed ordered set includes multiple short video contents;

[0019] The controlling the recommended content according to the predicted user emotion and / or behavior includes:

[0020] According to the predicted user emotion and / or behavior, the short video contents in the mixed ordered set are reordered, some short video contents are deleted from the mixed ordered set, and / or short video contents are inserted into the mixed ordered set.

[0021] The predicted emotions include emotion categories and emotion intensity, and the emotion categories include positive emotions or negative emotions;

[0022] The playing control of the recommended content according to the predicted user emotion and / or behavior includes:

[0023] If the predicted user emotion is negative, then based on the intensity and / or intensity change of the predicted user emotion and the emotion label of the currently playing content, determine whether the currently playing content induces or strengthens the user's emotion. If so, cancel or reduce the playback of recommended content with the same emotion label.

[0024] Among them, the predicted behavior categories include: swiping away the recommended content, finishing, rewatching, agreeing, disagreeing, reading comments, posting comments, collecting or sharing;

[0025] The playing control of the recommended content according to the predicted user emotion and / or behavior includes:

[0026] Based on the predicted behavior category and the corresponding recommended content category, it is determined whether the user is addicted to the same category of content. If so, the recommendation of the same category of recommended content is canceled or reduced.

[0027] Among them, it also includes:

[0028] According to the feedback information of the user's interest in the played content and the corresponding category of the played content, the played content of the category that the user is not interested in is deleted or reduced, or the played content of the category that the user is interested in is inserted.

[0029] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of any of the methods described above.

[0030] An electronic device, comprising:

[0031] one or more processors; and

[0032] A memory associated with the one or more processors, the memory being used to store program instructions, wherein the program instructions, when read and executed by the one or more processors, execute the steps of any of the aforementioned methods.

[0033] A computer program product comprises a computer program / computer executable instructions, wherein the computer program / computer executable instructions implement the steps of any of the aforementioned methods when executed by a processor in an electronic device.

[0034] According to the specific embodiments provided in this application, this application discloses the following technical effects:

[0035] Through the embodiments of the present application, in the process of orderly playing the recommended content, the terminal side signal generated by the terminal device during the user interaction process can be obtained, and then, based on the terminal side signal, the real-time emotions of the user in the process of watching the recommended content and / or the possible behaviors performed on the current recommended content can be predicted; thereafter, the playback of the recommended content can be controlled based on the predicted real-time emotions and / or behaviors. In this way, terminal-side intelligent intervention of the recommended content based on the terminal side signal can be achieved, and because the user's real-time emotions, behaviors, etc. are first predicted based on the terminal signal, and then the playback of the recommended content is intervened based on the emotion and behavior prediction results, the terminal side signal is no longer added in a "black box" manner, so that the application method of the terminal signal has stronger explainability, which is convenient for better control, thereby effectively improving the intervention effect during the playback of the recommended content.

[0036] Of course, any product implementing the present application does not necessarily need to achieve all of the advantages described above at the same time. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0038] Figure 1 It is a schematic diagram of the system architecture provided by the embodiment of the present application;

[0039] Figure 2 is a flow chart of the method provided in the embodiment of the present application;

[0040] Figure 3 is a schematic diagram of a device provided in an embodiment of the present application;

[0041] Figure 4 It is a schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0042] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments in the present application belong to the scope of protection of this application.

[0043] First of all, it should be noted that the inventor of the present application found in the process of implementing the present application that when users watch content on mobile terminal devices such as mobile phones, they usually perform various gesture operations on the mobile device, such as zooming in and out, sliding left and right, sliding up and down, clicking, pressing, liking, and collecting. In addition, sensors (such as gyroscopes, light sensors, etc.) on mobile terminal devices can also perceive the physical parameters of user behavior, the environment in which they are located, etc., for example, the pressure and duration of the user's click on the screen, etc., or it can also detect whether the user is running or lying down, whether the light in the environment where the mobile phone is located is sufficient, etc. However, in the existing content recommendation system, model training is mainly based on massive data, and these real-time and diverse terminal signals have not been effectively utilized, and their value for the recommendation system has not been fully explored. Although some content recommendation systems introduce terminal signal data as features, how these terminal signals specifically work is in a "black box" state, with poor interpretability, difficult to control, and very limited improvement effect. Therefore, this implementation method does not make good use of the real-time feedback behavior data of users on these terminals.

[0044] Based on the above situation, a corresponding solution is proposed in the embodiment of the present application. In this solution, considering that when a user uses a mobile terminal device to watch content, some signals perceived by the terminal device (including information related to user interaction behavior, information about the time and space environment, etc.) can actually reflect the user's emotions at the time (for example, positive or negative, happy or sad, the intensity of emotions, etc.), and what kind of behavior may be performed on the currently watched content (for example, swiping away, watching to the end, watching again, liking, liking, reading comments, posting comments, collecting, sharing, etc.). Therefore, in this solution, based on the signals perceived by the above terminal device, the user's emotions during the process of watching the recommended content and / or the behavior that may be performed on the current recommended content can be predicted, and then, based on this real-time emotion and / or behavior prediction result, the playback of the recommended content can be controlled. In this way, the on-end signal can be applied to the content recommendation system. The specific application method is to first predict the user's real-time emotions, behaviors, etc. based on the on-end signal, and then intervene in the playback of the recommended content based on the emotion and behavior prediction results. Therefore, the application method of the on-end signal has stronger interpretability, which facilitates better control and effectively improves the intervention effect during the playback of recommended content.

[0045] For example, in a typical application scenario, the specific recommended content may be short video content. For this type of content, in order to avoid freezing when the client switches between different short videos, the server usually pushes content to the client in units of short video sets, that is, pushes multiple short videos to the client each time, and then the client can form a mixed ordered set of these short videos and play these short videos in sequence. In this case, the specific playback control of the recommended content can be that the client controls the playback of the short video content in the mixed ordered set based on the predicted real-time emotions, behaviors and other information. For example, multiple short videos in the mixed ordered set can be rearranged (reordered), or one or more short video contents can be deleted or inserted from the mixed ordered set, etc. For example, if it is found that the user's mood has been relatively low during the playback process, and most of the content watched is also likely to induce or strengthen this negative emotion, then this type of content can be deleted from the mixed ordered set to reduce further impact on the user's mood.

[0046] Regarding inserting short video content into a mixed ordered set, it can be that when it is found that the user is more interested in a certain type of content, and this type of content will not have a negative impact on the user's emotions or bad guidance, more of this type of content can be inserted into the current mixed ordered set. Regarding the source of the "inserted" content, there can be multiple ways in the specific implementation. For example, in one way, the number of short videos sent by the server to the client each time may be more than the number of short videos in the mixed ordered set. For example, assuming that the server sends 20 short videos to the client each time, but the client may only select 10 short videos from these to add to the mixed ordered set, and so on. In the process of intervening according to the user's real-time emotion and behavior prediction results, suitable short video content can be selected from the short videos that have been sent to the client but have not yet been added to the mixed ordered set, and inserted into the current mixed ordered set. Alternatively, the client can also request the server to obtain a short video of a certain type or with a certain tag for insertion into the current mixed ordered set, and so on.

[0047] From the perspective of system architecture, Figure 1As shown, the embodiment of the present application mainly predicts the user's real-time emotions, behaviors, etc. by perceiving the end-side signals on the client side of the content recommendation system, and then implements end-side intelligent intervention on the playback of recommended content through the prediction results. Specifically, when predicting the user's real-time emotions, behaviors, etc. based on the end-side signals, it can be achieved through a pre-established perception model. Among them, after obtaining the end-side signals, some of the signals (including physical parameter information of user gesture signals captured by the terminal device sensor, such as finger movement distance, speed, x-axis coordinates, y-axis coordinates, z-axis coordinates, pressing force, pressing duration, etc.) can be directly input into the perception model; in addition, the end-side signals can be used for some preprocessing to obtain higher-order semantic information (including spatiotemporal environment context information, user gesture categories, etc.), and then input into the perception model, so that the perception model can obtain richer multi-order information to obtain more accurate prediction results. Among them, when predicting real-time emotions, behaviors, etc., the prediction results of emotions and behaviors can be output through the same perception model, that is, the perception model can be a model of a multi-task classification structure, and during training, the emotion prediction task and the behavior prediction task can also be trained together. In an optional way, the intermediate state emotion prediction and behavior prediction results can also be input back into the perception model, so that the intermediate state results of emotion prediction and behavior prediction can also participate in the prediction of the perception model and affect each other's prediction results, that is, emotion prediction can affect behavior prediction results, behavior prediction can also affect emotion prediction results, and so on.

[0048] After obtaining the real-time emotion and / or behavior prediction results, the playback control of the recommended content can be intervened. There can be many specific intervention methods, for example, social value intervention, on-end re-arrangement strategy, real-time interaction strategy (intervention with user feedback such as "like", "dislike", "thumbs-up", "thumbs-up", etc.), etc. After hitting one of the strategies, a variety of different intervention operations can be performed on the end side. For example, for short video content recommendation scenarios, since there is a mixed ordered set on the client side, specific operations that can be performed may include re-arranging, deleting, inserting, and so on, the short video content in the mixed ordered set.

[0049] The specific implementation scheme provided in the embodiments of the present application is described in detail below.

[0050] First, the embodiment of the present application provides a processing method for end-side playback control of recommended content, wherein each step of the method can be executed by the client, or can also be executed by the server, or can also be partially executed by the client and partially executed by the server. For example, the step of identifying emotions and behaviors can be executed on the server, and the acquisition of end-side signals and subsequent intervention can be executed on the client, etc. For details, see Figure 2 , the method may include:

[0051] S201: In the process of orderly playing recommended content, a terminal side signal generated by a terminal device during a user interaction process is obtained.

[0052] Among them, the recommended content can be a variety of different types such as short videos, pictures and texts. Among them, in the process of playing such recommended content, the user can perform interactive operations, including clicking, long pressing, swiping away, etc., and may also use certain gestures or body postures, and the body posture may also change, for example, from lying to sitting, or from running to walking, etc. In addition, the time and space environment in which the user is located will also have changes in light and darkness. Furthermore, the user may also have accelerated heartbeats, accelerated breathing, sighs, laughs, etc. in the process of watching recommended content. All of the above can be captured by various sensors in the terminal device, and corresponding terminal-side signals are generated accordingly. It should be noted here that, regarding the sensors of terminal devices, in addition to the common gyroscopes, light sensors, distance sensors, acceleration sensors, gravity sensors, position sensors, etc., some terminal devices may also include heart rate sensors, electrocardiogram sensors (electrocardiogram is a physiological signal that can interpret the electrical activity of the heart in real time and non-invasively), electroencephalogram sensors (electroencephalogram is a non-invasive electrophysiological technology that records the electrical activity of the human brain), skin electrode signal sensors (skin electrode signal is also called skin conductivity or electrical activity. It is a physiological signal collection method for continuously measuring the electrical signals on the surface of the skin), electrooculogram sensors (electrooculogram is a weak surface bioelectric signal generated with the movement of the eyeball), electromyogram sensors (electromyogram is the comprehensive effect of electrical activity on the superficial muscles and nerve trunks on the skin surface, which reflects the activity of the nerves and muscles to a certain extent), skin temperature sensors (skin temperature signals generally show a decrease due to the contraction of blood vessels when the sympathetic nerves are activated), breathing sensors (the breathing signal is the number of breaths per minute, and the number of breaths per minute is obtained by calculating the time interval between the peaks of each respiratory waveform), and so on.

[0053] The embodiment of the present application can obtain these terminal side signals for subsequent terminal side intervention of the recommended content. Specifically, the operation of obtaining the terminal side signals can be performed in real time, that is, new signals are continuously generated in the terminal device, so these terminal side signals can also be continuously obtained.

[0054] S202: Predicting, based on the terminal-side signal, the real-time emotions of the user while viewing the recommended content and / or the possible behaviors performed on the current recommended content.

[0055] After obtaining the terminal side signal, the user's real-time emotions and / or possible behaviors for the current recommended content can be predicted. In specific implementation, the prediction step can be completed by a pre-trained perception model, wherein, in a specific implementation mode, when predicting emotions and / or behaviors through the perception model, the data input into the perception model can be the following three types: the first is to directly input part of the terminal side signal into the perception model, which can be the physical parameter information of the user gesture signal captured by the terminal device sensor, for example, it can specifically include: finger movement distance, speed, x-axis coordinate, y-axis coordinate, z-axis coordinate, pressing force, pressing duration, etc. In addition, if the terminal device can detect heart rate signals, breathing signals, skin electricity signals, eyeball signals, myoelectric signals, skin temperature signals, electroencephalogram signals, electrocardiogram signals, etc., these signals can also be directly input into the perception model for prediction. Among them, the above-mentioned parameter information will be helpful for predicting specific emotional intensity, etc. For example, if it is found that the user's finger presses with great force, fast speed, heart rate and breathing are also accelerated, etc., it proves that the emotion may be more excited, etc.

[0056] The second input data of the perception model can be the context information of the user's spatiotemporal environment perceived based on the terminal side signal, that is, the characteristics of the user's spatiotemporal environment can be determined based on the terminal side signal, including whether it is an indoor or outdoor environment, whether there are lights indoors (which can be determined based on the signals captured by the ambient light sensor, microphone, etc.), the specific location, time, etc. Then, the perceived spatiotemporal environment context information can be input into the perception model to predict real-time emotions and / or behaviors.

[0057] The third input data of the perception model may be: information such as user gesture categories and / or body posture categories perceived based on the end-side signals, that is, after acquiring the end-side signals, these signals may be used to perceive the user's gesture categories, body posture categories, etc., and the perceived gesture categories and / or body posture category information may be input into the perception model for predicting the real-time emotions and / or behaviors. For example, the perceived gesture categories may include tapping, pressing, sweeping, scrolling, zooming, etc., and the perceived body posture categories may include lying, standing, sitting, leaning, walking, running, and so on.

[0058] Of course, the above only introduces one specific implementation method. In other methods, one or part of the above-mentioned various information can be input into the perception model for prediction, or other types of high-order semantic information can be perceived using terminal-side signals and input into the perception model. The above methods all fall within the scope of protection of this application and will not be described in detail here.

[0059] After the above data is input into the perception model, the perception model can predict the user's emotions and / or possible behaviors for the recommended content currently being played. Among them, the prediction of emotions and behaviors can be performed separately, that is, two independent perception models complete the emotion prediction task and the behavior prediction task respectively, or, in another way, considering that there is an intrinsic connection between emotions and behaviors, when users have certain emotions, they may be more likely to have certain behaviors, and so on. Therefore, in a preferred implementation method, a model with a multi-task classification structure can also be used to jointly predict emotions and behaviors, that is, the same perception model can have two kinds of output information, namely, emotion prediction results and behavior prediction results. In addition, the model of the specific multi-task classification framework can also be generated by jointly training the emotion prediction task and the behavior prediction task.

[0060] In a more preferred manner, in the process of predicting the emotions and behaviors using the model of the multi-task classification structure, the intermediate state prediction information of the real-time emotions and behaviors can also be input into the perception model, so that the prediction information of emotions and behaviors can affect each other's prediction results. In other words, the prediction results of emotions can affect the prediction results of behaviors, and correspondingly, the prediction results of behaviors can also affect the prediction results of emotions. In this way, the prediction structures of emotions and behaviors can be made more accurate.

[0061] Among them, the specific predicted real-time emotions of users can be expressed in a variety of ways. For example, one of the ways can include high arousal, medium arousal, low arousal; high valence, medium valence, low valence; anger, disgust, fear, happiness, sadness, surprise, etc. Among them, arousal means the intensity of the emotion felt, very low means almost no emotion, the current mood is as calm as water, that is, the current video does not induce emotions; very high means that the current video induces particularly strong emotions, such as being very angry or extremely happy. Valence indicates the positive / negative feeling of emotions, that is, positive emotions and negative emotions, negative emotions are negative and unpleasant emotions, such as depression, sadness, and anger; positive emotions are positive and pleasant emotions, such as happiness and excitement.

[0062] Specific predicted behaviors that users may perform on the current recommended content may include: swiping away, finishing watching, rewatching, agreeing (clicking "like"), disagreeing (clicking "dislike"), reading comments, posting comments, collecting, sharing, etc.

[0063] S203: Controlling the playing of the recommended content according to the predicted real-time emotion and / or behavior.

[0064] After predicting the user's real-time emotions and / or behaviors, the recommended content can be intelligently intervened on the terminal side. There can be many specific intervention methods. For example, as mentioned above, it can include social value intervention, terminal re-arrangement strategy, real-time interaction strategy, etc.

[0065] Among them, the so-called social value intervention mainly refers to the judgment of whether the recommended content will induce or aggravate the user's negative emotions, whether it will cause the user to be addicted to a certain category of content, etc. If so, the playback of such recommended content can be deleted or reduced. Specifically, in the embodiment of the present application, since the predicted emotions can specifically include emotion categories and emotion intensity, the emotion categories include positive emotions or negative emotions. More specifically, they can be subdivided into anger, disgust, fear, happiness, sadness, surprise and other different types. In addition, for specific recommended content, emotional labels can also be added to the specific content in advance based on the keywords of the content, the voice features in the content, etc. The emotional labels of the content may cause the user to have certain emotions. For example, a certain content may make people feel very happy, so positive emotional labels such as "happy" can be added. Another content is easy to make people feel depressed, so negative emotional labels such as "depressed" can be added, and so on.

[0066] Based on the above situation, when performing end-side intervention, if the predicted user emotion is found to be negative, it is also possible to determine whether the currently played content induces or strengthens the user's emotion based on the predicted intensity and / or intensity change of the user emotion and the emotion label of the currently played content. If so, the play of recommended content with the same emotion label can be canceled or reduced. For example, if it is determined that the user has been watching negative content and is in a low mood, the recommendation of such content can be reduced (to prevent the user from being induced to do some extreme behavior).

[0067] Alternatively, since the predicted behavior categories include: swiping away the recommended content, finishing, re-watching, agreeing, disagreeing, reading comments, posting comments, collecting or sharing, etc., it is also possible to determine whether the user is addicted to the same category of content based on the predicted behavior categories and the corresponding recommended content categories. If so, the playback of the same category of recommended content can be canceled or reduced. For example, assuming that the user will finish watching any content that contains some bad social habits, etc., the user may be addicted to such content. Since such content may have some bad guidance, the playback of such content can be reduced to prevent addiction, etc. Alternatively, assuming that an advertisement is recommended to the user, and then it is predicted that the user is about to swipe away, the content that the user is really interested in can be pushed immediately, etc.

[0068] Regarding the on-end reordering strategy, that is, reordering the recommended content that the client is about to play. For example, based on the real-time perceived user emotions, the content that the user is currently more interested in and does not have any negative connotations can be adjusted to a more forward position for priority playback, and so on.

[0069] Regarding the real-time interaction strategy, that is, based on the user's feedback information on the interest in the played content and the corresponding category of the played content, the played content of the category that the user is not interested in can be deleted or reduced, or the played content of the category that the user is interested in can be inserted. For example, if it is found through user feedback that the user does not like a certain tag, the content containing the tag can be deleted or reduced.

[0070] After a specific intervention strategy is hit, the corresponding operation can be performed on the terminal side, and the specific operation can include the aforementioned rearrangement, deletion, insertion, etc. Among them, as mentioned above, the specific recommended content can be a mixed ordered set determined based on multiple short video contents pushed by the server to the client, and the mixed ordered set includes multiple short video contents. Then, according to the predicted user emotions and / or behaviors, the short video contents in the mixed ordered set can be reordered, some short video contents can be deleted from the mixed ordered set, and / or short video contents can be inserted into the mixed ordered set.

[0071] In summary, through the embodiments of the present application, in the process of orderly playing the recommended content, the terminal side signal generated by the terminal device during the user interaction process can be obtained, and then, based on the terminal side signal, the real-time emotions of the user in the process of watching the recommended content and / or the possible behaviors performed on the current recommended content can be predicted; thereafter, the playback of the recommended content can be controlled based on the predicted real-time emotions and / or behaviors. In this way, terminal-side intelligent intervention of the recommended content based on the terminal side signal can be achieved, and because the user's real-time emotions, behaviors, etc. are first predicted based on the terminal signal, and then the playback of the recommended content is intervened based on the emotion and behavior prediction results, the terminal side signal is no longer added in a "black box" manner, so that the application method of the terminal signal has stronger explainability, which is convenient for better control, thereby effectively improving the intervention effect in the playback process of the recommended content.

[0072] It should be noted that the embodiments of the present application may involve the use of user data. In actual applications, user-specific personal data can be used in the scheme described herein within the scope permitted by applicable laws and regulations, subject to the requirements of applicable laws and regulations of the country where the user is located (for example, with the user's explicit consent, effective notification to the user, etc.).

[0073] Corresponding to the above method embodiment, the present application embodiment also provides a processing device for controlling the recommended content on the terminal side, see Figure 3 , the device may include:

[0074] The terminal side signal acquisition unit 301 is used to acquire the terminal side signal generated by the terminal device during the user interaction process during the orderly playback of the recommended content;

[0075] A prediction unit 302, configured to predict, based on the terminal-side signal, the real-time emotion of the user during viewing the recommended content and / or the possible behavior performed on the current recommended content;

[0076] The recommendation control unit 303 is used to control the recommendation content according to the predicted real-time emotions and / or behaviors.

[0077] Among them, the terminal side signal includes physical parameter information of user gesture signals, heart rate signals, breathing signals, skin electricity signals, eye contact signals, myoelectric signals, skin temperature signals, electroencephalogram signals, and electrocardiogram signals captured by the terminal device sensor, which are used to input into the perception model, and the perception model predicts the user's real-time emotions and / or behaviors.

[0078] Alternatively, in a specific implementation, the device may further include:

[0079] The environmental context perception unit is used to perceive the user's spatiotemporal environmental context based on the terminal side signal generated by the terminal device sensor, and input the perceived spatiotemporal environmental context information into the perception model for predicting the real-time emotion and / or behavior.

[0080] A gesture and body posture perception unit is used to perceive the gesture category and / or body posture category performed by the user when playing the content based on the terminal side signal generated by the terminal device sensor, and input the perceived gesture category and / or body posture category information into the perception model for predicting the real-time emotion and / or behavior.

[0081] Specifically, the prediction unit can be used for:

[0082] The emotions and behaviors are predicted using a perception model of a multi-task classification structure, wherein the perception model of the multi-task classification framework is generated by jointly training an emotion prediction task and a behavior prediction task.

[0083] Specifically, in the process of predicting the emotions and behaviors using the perception model of the multi-task classification structure, the intermediate state prediction information of the real-time emotions and behaviors can be input into the perception model so that the prediction information of emotions and behaviors can influence each other's prediction results.

[0084] The recommended content includes: a mixed ordered set determined based on multiple short video contents pushed by the server to the client, wherein the mixed ordered set includes multiple short video contents;

[0085] At this point, the recommended controls can be used specifically for:

[0086] According to the predicted user emotion and / or behavior, the short video contents in the mixed ordered set are reordered, some short video contents are deleted from the mixed ordered set, and / or short video contents are inserted into the mixed ordered set.

[0087] Alternatively, the predicted emotion includes an emotion category and an emotion intensity, wherein the emotion category includes a positive emotion or a negative emotion;

[0088] At this point, the recommended controls can be used specifically for:

[0089] If the predicted user emotion is negative, then based on the intensity and / or intensity change of the predicted user emotion and the emotion label of the currently playing content, determine whether the currently playing content induces or strengthens the user's emotion. If so, cancel or reduce the playback of recommended content with the same emotion label.

[0090] Alternatively, the predicted behavior categories include: swiping away the recommended content, finishing watching, rewatching, agreeing, disagreeing, reading comments, posting comments, collecting, or sharing;

[0091] At this point, the recommended controls can be used specifically for:

[0092] Based on the predicted behavior category and the corresponding recommended content category, it is determined whether the user is addicted to the same category of content. If so, the recommendation of the same category of recommended content is canceled or reduced.

[0093] Additionally, this recommended control can be used to:

[0094] According to the feedback information of the user's interest in the played content and the corresponding category of the played content, the played content of the category that the user is not interested in is deleted or reduced, or the played content of the category that the user is interested in is inserted.

[0095] In addition, an embodiment of the present application further provides a computer-readable storage medium on which a computer program is stored, and when the program is executed by a processor, the steps of any one of the methods in the aforementioned method embodiments are implemented.

[0096] And an electronic device, comprising:

[0097] one or more processors; and

[0098] A memory associated with the one or more processors, the memory being used to store program instructions, wherein the program instructions, when read and executed by the one or more processors, execute the steps of the method described in any one of the aforementioned method embodiments.

[0099] A computer program product includes a computer program / computer executable instructions, which, when executed by a processor in an electronic device, implement the steps of the method described in the aforementioned method embodiment.

[0100] in, Figure 4 The architecture of an electronic device is exemplarily shown. For example, device 400 can be a mobile phone, a computer, a digital broadcast terminal, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, an aircraft, etc.

[0101] Reference Figure 4 , device 400 may include one or more of the following components: a processing component 402 , a memory 404 , a power component 406 , a multimedia component 408 , an audio component 410 , an input / output (I / O) interface 412 , a sensor component 414 , and a communication component 416 .

[0102] The processing component 402 generally controls the overall operation of the device 400, such as operations associated with display, phone calls, data communications, camera operations, and recording operations. The processing component 402 may include one or more processors 420 to execute instructions to complete all or part of the steps of the method provided by the technical solution of the present disclosure. In addition, the processing component 402 may include one or more modules to facilitate the interaction between the processing component 402 and other components. For example, the processing component 402 may include a multimedia module to facilitate the interaction between the multimedia component 408 and the processing component 402.

[0103] The memory 404 is configured to store various types of data to support operations on the device 400. Examples of such data include instructions for any application or method operating on the device 400, contact data, phone book data, messages, pictures, videos, etc. The memory 404 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0104] The power supply component 406 provides power to the various components of the device 400. The power supply component 406 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the device 400.

[0105] The multimedia component 408 includes a screen that provides an output interface between the device 400 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundaries of the touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 408 includes a front camera and / or a rear camera. When the device 400 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera may receive external multimedia data. Each front camera and rear camera may be a fixed optical lens system or have a focal length and optical zoom capability.

[0106] The audio component 410 is configured to output and / or input audio signals. For example, the audio component 410 includes a microphone (MIC), and when the device 400 is in an operating mode, such as a call mode, a recording mode, and a speech recognition mode, the microphone is configured to receive an external audio signal. The received audio signal can be further stored in the memory 404 or sent via the communication component 416. In some embodiments, the audio component 410 also includes a speaker for outputting audio signals.

[0107] I / O interface 412 provides an interface between processing component 402 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include but are not limited to: a home button, a volume button, a start button, and a lock button.

[0108] The sensor assembly 414 includes one or more sensors for providing various aspects of status assessment for the device 400. For example, the sensor assembly 414 can detect the open / closed state of the device 400, the relative positioning of components, such as the display and keypad of the device 400, and the sensor assembly 414 can also detect the position change of the device 400 or a component of the device 400, the presence or absence of user contact with the device 400, the orientation or acceleration / deceleration of the device 400, and the temperature change of the device 400. The sensor assembly 414 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. The sensor assembly 414 may also include an optical sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 414 may also include an accelerometer, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0109] The communication component 416 is configured to facilitate wired or wireless communication between the device 400 and other devices. The device 400 can access a wireless network based on a communication standard, such as WiFi, or a mobile communication network such as 2G, 3G, 4G / LTE, 5G, etc. In an exemplary embodiment, the communication component 416 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 416 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0110] In an exemplary embodiment, the device 400 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the above methods.

[0111] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 404 including instructions, and the above instructions can be executed by a processor 420 of a device 400 to complete the method provided by the technical solution of the present disclosure. For example, the non-transitory computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0112] It can be known from the description of the above implementation methods that those skilled in the art can clearly understand that the present application can be implemented by means of software plus a necessary general hardware platform. Based on such an understanding, the technical solution of the present application can be essentially or partly contributed to the prior art in the form of a software product, which can be stored in a storage medium such as ROM / RAM, a magnetic disk, an optical disk, etc., and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in the various embodiments of the present application or certain parts of the embodiments.

[0113] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can refer to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system or system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can refer to the partial description of the method embodiment. The system and system embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without creative work.

[0114] The above is a detailed introduction to the processing method and electronic device for end-side control of recommended content provided by this application. This article uses specific examples to illustrate the principles and implementation methods of this application. The description of the above embodiments is only used to help understand the method and core ideas of this application. At the same time, for those skilled in the art, according to the ideas of this application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting this application.

Claims

1. A method for controlling recommended content on a terminal side, characterized in that: include: In the process of orderly playing the recommended content, obtaining the terminal side signal generated by the terminal device during the user interaction process; Predicting, based on the terminal-side signal, the real-time emotions of the user during viewing the recommended content and / or the possible behaviors performed on the current recommended content; The recommended content is controlled based on the predicted real-time emotion and / or behavior.

2. The method according to claim 1, characterized in that The terminal side signal includes physical parameter information of user gesture signals, heart rate signals, breathing signals, skin electricity signals, eye contact signals, myoelectric signals, skin temperature signals, electroencephalogram signals, and electrocardiogram signals captured by terminal device sensors, for input into the perception model, and the perception model predicts the user's real-time emotions and / or behaviors.

3. The method according to claim 1 or 2, characterized in that: Also includes: Based on the terminal side signal generated by the terminal device sensor, the user's spatiotemporal environment context is perceived, and the perceived spatiotemporal environment context information is input into the perception model for predicting the real-time emotion and / or behavior.

4. The method according to claim 1 or 2, characterized in that: Also includes: Based on the terminal-side signal generated by the terminal device sensor, the gesture category and / or body posture category performed by the user when playing the content is perceived, and the perceived gesture category and / or body posture category information is input into the perception model for predicting the real-time emotion and / or behavior.

5. The method according to claim 1, characterized in that The predicting of the emotions of the user during viewing the recommended content and / or the possible actions of the user on the current recommended content includes: The emotions and behaviors are predicted using a perception model of a multi-task classification structure, wherein the perception model of the multi-task classification framework is generated by jointly training an emotion prediction task and a behavior prediction task.

6. The method according to claim 5, characterized in that In the process of predicting the emotions and behaviors using the perception model of the multi-task classification structure, the intermediate state prediction information of the real-time emotions and behaviors is input into the perception model so that the prediction information of emotions and behaviors influence each other's prediction results.

7. The method according to claim 1, characterized in that The recommended content includes: a mixed ordered set determined based on multiple short video contents pushed by the server to the client, wherein the mixed ordered set includes multiple short video contents; The controlling the recommended content according to the predicted user emotion and / or behavior includes: According to the predicted user emotion and / or behavior, the short video contents in the mixed ordered set are reordered, some short video contents are deleted from the mixed ordered set, and / or short video contents are inserted into the mixed ordered set.

8. The method according to claim 1, characterized in that The predicted emotions include emotion categories and emotion intensity, wherein the emotion categories include positive emotions or negative emotions; The playing control of the recommended content according to the predicted user emotion and / or behavior includes: If the predicted user emotion is negative, then based on the intensity and / or intensity change of the predicted user emotion and the emotion label of the currently playing content, determine whether the currently playing content induces or strengthens the user's emotion. If so, cancel or reduce the playback of recommended content with the same emotion label.

9. The method according to claim 1, characterized in that: The predicted behavior categories include: swiping away the recommended content, finishing it, rewatching it, agreeing, disagreeing, reading comments, posting comments, collecting it, or sharing it; The playing control of the recommended content according to the predicted user emotion and / or behavior includes: Based on the predicted behavior category and the corresponding recommended content category, it is determined whether the user is addicted to the same category of content. If so, the recommendation of the same category of recommended content is canceled or reduced.

10. The method according to claim 1, characterized in that Also includes: According to the feedback information of the user's interest in the played content and the corresponding category of the played content, the played content of the category that the user is not interested in is deleted or reduced, or the played content of the category that the user is interested in is inserted.

11. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the method described in any one of claims 1 to 10 are implemented.

12. An electronic device, characterized in that: include: one or more processors; as well as A memory associated with the one or more processors, the memory being used to store program instructions, wherein when the program instructions are read and executed by the one or more processors, the steps of the method according to any one of claims 1 to 10 are executed.

13. A computer program product comprising a computer program / computer executable instructions, characterized in that: When the computer program / computer executable instructions are executed by a processor in an electronic device, the steps of the method according to any one of claims 1 to 10 are implemented.

Images