Video recommendation method and device based on multi-modal feature recognition and storage medium

Abstract

The invention discloses a video recommendation method and device based on multi-modal feature recognition, a storage medium and computer equipment. The method comprises the following steps: collecting and analyzing facial images and physiological signal data of a user respectively collected by a camera device and intelligent monitoring equipment when a target user browses a video on a video platform, and recording watching information of the user at the same time; extracting facial features, physiological features and viewing features from the collected data; emotion recognition is carried out in combination with the multi-modal features, and emotion information of the user on the video is determined; and determining emotional preferences of the user according to an emotional recognition result, and recommending personalized video contents to the user based on the preferences. Through multi-modal feature recognition, the scheme can more comprehensively capture emotional changes and interest points of the user in the video watching process, so that the preferences and preferences of the user are more accurately judged, and the recommendation accuracy is improved.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a video recommendation method, device, storage medium and computer equipment based on multimodal feature recognition. Background Art

[0002] With the rapid development of Internet technology, video platforms have become an indispensable part of people's daily lives. Users consume a wide variety of content on these platforms, ranging from entertainment, education to news and information. However, faced with massive video resources, how to efficiently recommend content that matches users' interests and preferences has become a key issue that video platforms need to solve. Traditional video recommendation systems mostly rely on users' explicit feedback (such as ratings and likes) or implicit feedback (such as viewing time and click behavior) to build user portraits and then recommend content. However, this approach has many limitations, such as sparse user feedback and difficulty in capturing changes in interests, which results in recommendations that are often not accurate and personalized enough. Summary of the invention

[0003] In view of this, the embodiments of the present application provide a video recommendation method, apparatus, storage medium and computer equipment based on multimodal feature recognition. Through multimodal feature recognition, the solution can more comprehensively capture the user's emotional changes and interests during the video watching process, thereby more accurately judging the user's likes and preferences and improving the accuracy of recommendations.

[0004] According to one aspect of the present application, a video recommendation method based on multimodal feature recognition is provided, the method comprising:

[0005] Acquire facial images of a target user when browsing a video on a video platform captured by a camera device and physiological signal data collected by an intelligent monitoring device worn by the target user, and acquire viewing information of the target user on the video;

[0006] Extracting facial features corresponding to the facial image, physiological features corresponding to the physiological signal data, and viewing features corresponding to the viewing information;

[0007] Performing emotion recognition on the facial features, the physiological features, and the viewing features to determine the target user's emotion information on the video;

[0008] The target user's emotional preference for the video is determined according to the emotional information, and a video is recommended to the target user according to the emotional preference.

[0009] In an embodiment of the present application, optionally, the facial features include position features and morphological features of facial expression feature points, the physiological signal data include heart rate features and blood pressure features, and the viewing features include the played time period, total playback time, played percentage, and end video frame content features when the viewing of the video is finished, wherein the played time period is the video frame period in which all video clips that the target user has watched in the video are located, the total playback time is the total time the target user has watched the video, the played percentage is the video playback progress when the target user finishes viewing the video, and the end video frame content features are content features corresponding to the video frame where the target user ends viewing the video;

[0010] Performing emotion recognition on the facial features, the physiological features, and the viewing features to determine the target user's emotion information on the video includes:

[0011] Performing feature fusion on the position features of the facial expression feature points, the morphological features of the facial expression feature points, the heart rate features, and the blood pressure features to obtain user features, performing emotion recognition on the user features through a pre-trained emotion recognition model, and determining the first emotion information of the target user on the video;

[0012] Determining the second emotion information of the target user towards the video according to the played time period, the total playing time and the played percentage;

[0013] Determine the target user's emotional information about the video according to the first emotional information and the second emotional information.

[0014] In the embodiment of the present application, optionally, determining the second emotion information of the target user towards the video according to the played time period, the total playing time and the played percentage includes:

[0015] Obtaining a key frame period, a reference playback duration, and a total video duration corresponding to the video, wherein the key frame period is a video frame period where at least one preset key segment in the video is located, and the reference playback duration indicates the playback duration of the video that has achieved an expected playback effect;

[0016] If the played time is greater than the reference played time, the ratio of the duration of at least part of the key frame period included in the played time period to the total duration of the key frame period is used as the first played time period score, and the first preset weight is determined as the first played time period weight; the first ratio of the difference between the played time and the key frame duration to the key frame duration is calculated, the sum of the first ratio and the preset score is determined as the first played time score, and the second preset weight is determined as the first played time weight; the played percentage is determined as the first played percentage score, and the first played percentage weight is determined according to the first played time period weight and the first played time weight; according to the first played time period weight, the first played time weight and the first played percentage weight, the first played time score, the first played time score and the first played percentage score are weighted and summed to determine the second emotional information of the target user towards the video;

[0017] If the played time is less than or equal to the reference played time, the ratio of the duration of at least part of the key frame period included in the played time period to the total duration of the key frame period is used as the first played time period score, and the first preset weight is determined as the second played time period weight; the second ratio of the difference between the reference played time and the played time and the key frame time is calculated, the difference between the preset score and the second ratio is used as the second played time score, and the third preset weight is determined as the second played time weight; the played percentage is determined as the second played percentage score, and the second played percentage weight is determined according to the second played time period weight and the second played time weight; according to the second played time period weight, the second played time weight and the second played percentage weight, the second played time score, the second played time score and the second played percentage score are weighted and summed to determine the second emotional information of the target user towards the video, wherein the second preset weight is less than the third preset weight.

[0018] In the embodiment of the present application, optionally, the position feature of the facial expression feature point, the morphological feature of the facial expression feature point, the heart rate feature and the blood pressure feature are subjected to feature fusion to obtain the user feature, including:

[0019] Determining the change amplitudes of the position features of the facial expression feature points, the morphological features of the facial expression feature points, the heart rate features, and the blood pressure features of the target user when browsing the video, and determining the facial feature change amplitude according to the position feature change amplitude and the morphological feature change amplitude;

[0020] According to the facial feature change amplitude, the heart rate feature change amplitude and the blood pressure feature change amplitude, the facial feature fusion weight, the heart rate feature fusion weight and the blood pressure feature fusion weight are determined, and based on the facial feature fusion weight, the heart rate feature fusion weight and the blood pressure feature fusion weight, the facial features, the heart rate features and the blood pressure features are fused to obtain the user features.

[0021] In the embodiment of the present application, optionally, determining the target user's emotional information about the video according to the first emotional information and the second emotional information includes:

[0022] Comparing and analyzing the first emotional information and the second emotional information, according to the difference between the second emotional information and the first emotional information;

[0023] If the difference is within a preset threshold, determining the average of the first emotion information and the second emotion information as the emotion state information;

[0024] If the difference is not within the preset threshold, then calculate the third ratio of the difference between the difference and the preset threshold and the preset threshold, determine the difference between 0.5 and the third ratio as the fusion weight of the first emotional information, determine the sum of 0.5 and the third ratio as the fusion weight of the second emotional information, and perform weighted fusion of the first emotional information and the second emotional information to determine the emotional information.

[0025] In the embodiment of the present application, optionally, recommending a video to the target user according to the emotional preference includes:

[0026] If the target user has a positive emotional preference for the video, recommending a video of the same type as the video to the target user;

[0027] If the target user's emotional preference for the video is neutral, recommending videos of similar types corresponding to the video to the target user;

[0028] If the target user's emotional preference for the video is negative, a video of a different type and a different type than the video is recommended to the target user.

[0029] In the embodiment of the present application, optionally, if the target user's emotional preference for the video is negative, recommending a video of a different type and a different type corresponding to the video to the target user includes:

[0030] If the target user has a negative emotional preference for the video, candidate videos of a non-similar and non-identical type to the video are determined, and the target user's uninterested content is determined based on the content features of the end video frame, and video selection and recommendation are performed among the candidate videos based on the uninterested content.

[0031] According to another aspect of the present application, a video recommendation device based on multimodal feature recognition is provided, the device comprising:

[0032] An acquisition module, used to acquire a facial image of a target user when browsing a video on a video platform captured by a camera device and physiological signal data collected by an intelligent monitoring device worn by the target user, and to acquire viewing information of the target user on the video;

[0033] A feature extraction module, used to extract facial features corresponding to the facial image, physiological features corresponding to the physiological signal data, and viewing features corresponding to the viewing information;

[0034] A recognition module, used for performing emotion recognition on the facial features, the physiological features and the viewing features, and determining the target user's emotion information on the video;

[0035] The recommendation module is used to determine the target user's emotional preference for the video according to the emotional information, and recommend the video to the target user according to the emotional preference.

[0036] In an embodiment of the present application, optionally, the facial features include position features and morphological features of facial expression feature points, the physiological signal data include heart rate features and blood pressure features, and the viewing features include the played time period, total playback time, played percentage, and end video frame content features when the viewing of the video is finished, wherein the played time period is the video frame period in which all video clips that the target user has watched in the video are located, the total playback time is the total time the target user has watched the video, the played percentage is the video playback progress when the target user finishes viewing the video, and the end video frame content features are content features corresponding to the video frame where the target user ends viewing the video;

[0037] The identification module is further used for:

[0038] Performing feature fusion on the position features of the facial expression feature points, the morphological features of the facial expression feature points, the heart rate features, and the blood pressure features to obtain user features, performing emotion recognition on the user features through a pre-trained emotion recognition model, and determining the first emotion information of the target user on the video;

[0039] Determining the second emotion information of the target user towards the video according to the played time period, the total playing time and the played percentage;

[0040] Determine the target user's emotional information about the video according to the first emotional information and the second emotional information.

[0041] In the embodiment of the present application, optionally, the identification module is further used to:

[0042] Obtaining a key frame period, a reference playback duration, and a total video duration corresponding to the video, wherein the key frame period is a video frame period where at least one preset key segment in the video is located, and the reference playback duration indicates the playback duration of the video that has achieved an expected playback effect;

[0043] If the played time is greater than the reference played time, the ratio of the duration of at least part of the key frame period included in the played time period to the total duration of the key frame period is used as the first played time period score, and the first preset weight is determined as the first played time period weight; the first ratio of the difference between the played time and the key frame duration to the key frame duration is calculated, the sum of the first ratio and the preset score is determined as the first played time score, and the second preset weight is determined as the first played time weight; the played percentage is determined as the first played percentage score, and the first played percentage weight is determined according to the first played time period weight and the first played time weight; according to the first played time period weight, the first played time weight and the first played percentage weight, the first played time score, the first played time score and the first played percentage score are weighted and summed to determine the second emotional information of the target user towards the video;

[0044] If the played time is less than or equal to the reference played time, the ratio of the duration of at least part of the key frame period included in the played time period to the total duration of the key frame period is used as the first played time period score, and the first preset weight is determined as the second played time period weight; the second ratio of the difference between the reference played time and the played time and the key frame time is calculated, the difference between the preset score and the second ratio is used as the second played time score, and the third preset weight is determined as the second played time weight; the played percentage is determined as the second played percentage score, and the second played percentage weight is determined according to the second played time period weight and the second played time weight; according to the second played time period weight, the second played time weight and the second played percentage weight, the second played time score, the second played time score and the second played percentage score are weighted and summed to determine the second emotional information of the target user towards the video, wherein the second preset weight is less than the third preset weight.

[0045] In the embodiment of the present application, optionally, the identification module is further used to:

[0046] The user features are obtained by fusing the position features of the facial expression feature points, the morphological features of the facial expression feature points, the heart rate features, and the blood pressure features, including:

[0047] Determining the change amplitudes of the position features of the facial expression feature points, the morphological features of the facial expression feature points, the heart rate features, and the blood pressure features of the target user when browsing the video, and determining the facial feature change amplitude according to the position feature change amplitude and the morphological feature change amplitude;

[0048] According to the facial feature change amplitude, the heart rate feature change amplitude and the blood pressure feature change amplitude, the facial feature fusion weight, the heart rate feature fusion weight and the blood pressure feature fusion weight are determined, and based on the facial feature fusion weight, the heart rate feature fusion weight and the blood pressure feature fusion weight, the facial features, the heart rate features and the blood pressure features are fused to obtain the user features.

[0049] In the embodiment of the present application, optionally, the identification module is further used to:

[0050] Comparing and analyzing the first emotional information and the second emotional information, according to the difference between the second emotional information and the first emotional information;

[0051] If the difference is within a preset threshold, determining the average of the first emotion information and the second emotion information as the emotion state information;

[0052] If the difference is not within the preset threshold, then calculate the third ratio of the difference between the difference and the preset threshold and the preset threshold, determine the difference between 0.5 and the third ratio as the fusion weight of the first emotional information, determine the sum of 0.5 and the third ratio as the fusion weight of the second emotional information, and perform weighted fusion of the first emotional information and the second emotional information to determine the emotional information.

[0053] In the embodiment of the present application, optionally, the recommendation module is further used to:

[0054] If the target user has a positive emotional preference for the video, recommending a video of the same type as the video to the target user;

[0055] If the target user's emotional preference for the video is neutral, recommending videos of similar types corresponding to the video to the target user;

[0056] If the target user's emotional preference for the video is negative, a video of a different type and a different type than the video is recommended to the target user.

[0057] In the embodiment of the present application, optionally, the recommendation module is further used to:

[0058] If the target user has a negative emotional preference for the video, candidate videos of a non-similar and non-identical type to the video are determined, and the target user's uninterested content is determined based on the content features of the end video frame, and video selection and recommendation are performed among the candidate videos based on the uninterested content.

[0059] According to another aspect of the present application, a storage medium is provided, on which a computer program is stored, and when the program is executed by a processor, the above-mentioned video recommendation method based on multimodal feature recognition is implemented.

[0060] According to another aspect of the present application, a computer device is provided, including a storage medium, a processor, and a computer program stored on the storage medium and executable on the processor, wherein the processor implements the above-mentioned video recommendation method based on multimodal feature recognition when executing the program.

[0061] Through the above technical scheme, the embodiments of the present application provide a video recommendation method, device, storage medium and computer equipment based on multimodal feature recognition. Based on multimodal feature recognition technology, the method collects and analyzes the facial image and physiological signal data of the target user when browsing videos on the video platform using a camera device and an intelligent monitoring device, and records the user's viewing information at the same time; extracts facial features, physiological features and viewing features from the collected data; performs emotion recognition in combination with multimodal features to determine the user's emotional information about the video; determines the user's emotional preference based on the result of emotion recognition, and recommends personalized video content to the user based on these preferences. The embodiment of the present application uses multimodal feature recognition, and the solution can more comprehensively capture the user's emotional changes and interests during the video viewing process, so as to more accurately judge the user's likes and preferences and improve the accuracy of recommendations; personalized video recommendations can better meet the user's viewing needs, improve user satisfaction and stickiness, and at the same time, reduce the time wasted by users on content that they are not interested in, thereby improving the overall user experience; in addition, for video platforms, the solution helps to optimize content distribution strategies, push more video content that meets user interests to users, increase content exposure and user activity, and thus increase the commercial value of the platform.

[0062] The above description is only an overview of the technical solution of the present application. In order to more clearly understand the technical means of the present application, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present application more obvious and easy to understand, the specific implementation methods of the present application are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0064] Figure 1 A schematic diagram of a process of a video recommendation method based on multimodal feature recognition provided by an embodiment of the present application is shown;

[0065] Figure 2 A schematic diagram of the structure of a video recommendation device based on multimodal feature recognition provided by an embodiment of the present application is shown. DETAILED DESCRIPTION

[0066] The present application will be described in detail below with reference to the accompanying drawings and in combination with embodiments. It should be noted that the embodiments and features in the embodiments of the present application can be combined with each other without conflict.

[0067] In this embodiment, a video recommendation method based on multimodal feature recognition is provided. Figure 1As shown, the method includes:

[0068] Step 101, obtaining a facial image of a target user when browsing a video on a video platform captured by a camera device and physiological signal data collected by a smart monitoring device worn by the target user, and obtaining viewing information of the target user on the video.

[0069] The embodiment of the present application can accurately evaluate the user's emotional preference for video content by comprehensively analyzing the user's facial expressions, physiological reactions and specific viewing behaviors when watching videos, and make personalized video recommendations accordingly. First, when the target user browses videos on the video platform, with the permission of the target user, multimodal data collection can be performed on the target user. Specifically, data collection can be performed for ordinary videos browsed by the user (such as short videos) to recommend video content to the user, and data collection can also be performed for advertising videos browsed by the user to achieve accurate advertising to the user. Real-time facial images of the target user when watching videos are captured by a high-definition camera device, which contain changes in the user's expression, and physiological signals of the user, such as heart rate and blood pressure data, are collected using intelligent monitoring devices (such as smart watches). These data reflect the user's physiological reactions during the viewing process. In addition, the specific behavior of the user watching the video will also be recorded, including viewing time, percentage of playback, and content features of the video frame where the user ends viewing. This information helps to understand the user's viewing habits and interests.

[0070] Step 102: extracting facial features corresponding to the facial image, physiological features corresponding to the physiological signal data, and viewing features corresponding to the viewing information.

[0071] Among them, the facial features include position features and morphological features of facial expression feature points, the physiological signal data include heart rate features and blood pressure features, and the viewing features include the played time period, total playback time, played percentage and end video frame content features when the viewing of the video is finished, wherein the played time period is the video frame period in which all video clips that the target user has watched in the video are located, the total playback time is the total time the target user has watched the video, the played percentage is the video playback progress when the target user finishes watching the video, and the end video frame content features are content features corresponding to the video frame when the target user finishes watching the video.

[0072] In the embodiment of the present application, after obtaining the multimodal data, feature extraction of the multimodal data is performed. Using image processing technology and machine learning algorithms, the positions (such as the corners of the eyes, the corners of the mouth, etc.) and morphological features (such as wrinkles, muscle tension, etc.) of facial expression feature points are extracted from the facial image. These features can reflect the emotional state of the user. The heart rate and blood pressure data are processed to extract physiological features such as heart rate variability and blood pressure fluctuations. These features can reveal the physiological states of the user such as tension, relaxation or excitement when watching the video. The played period refers to the video frame period corresponding to all the video clips that the target user has watched in the video. This is not just a time point, but a period or multiple time ranges, covering all the video content that the user has actually watched. Through the played period, you can understand which parts of the video the user has watched and the position of these parts in the overall video. The total playback time refers to the total time that the target user has spent from the beginning of watching the video to the end of watching it. This is a single time point difference value, which indicates the overall length of time the user watches the video. The total playback time is the actual length of time the video is played, regardless of whether it is fast-forwarded or not. If the video is fast-forwarded, the length of time used for fast-forwarding is the total playback time. The total playback time reflects the user's overall involvement in the video content. The percentage played refers to the video progress when the user finishes watching the video. The end video frame content features refer to the content features corresponding to the video frame where the target user finishes watching the video. These features may include information conveyed by images, sounds, text or other multimedia elements in the video frame. In the process of emotion recognition and personalized recommendation, these viewing features can be combined with features of other modalities (such as facial expressions and physiological signals) to more comprehensively understand the user's emotional attitude and interest preferences towards the video. By comprehensively analyzing these features, it is possible to more accurately judge the user's preference for the video content and make personalized video recommendations accordingly.

[0073] Step 103: Perform emotion recognition on the facial features, the physiological features, and the viewing features to determine the target user's emotion information on the video.

[0074] After extracting multimodal features in the embodiment of the present application, emotion recognition is performed using an emotional computing or machine learning model in combination with the extracted facial features, physiological features, and viewing features. By comprehensively analyzing these multimodal information, it is possible to more accurately determine the emotional changes of users during video viewing, such as from calm to pleasure, from pleasure to boredom, etc.

[0075] Step 104: determining the target user's emotional preference for the video based on the emotional information, and recommending the video to the target user based on the emotional preference.

[0076] After analyzing the target user's emotional preference for the video in the embodiment of the present application, it is possible to predict whether the target user is interested in such videos based on the user's emotional preference for the video, thereby recommending videos to the target user. Specifically, based on the result of emotion recognition, the target user's emotional preference for the current video is determined. For example, if the user shows a high level of pleasure and a long viewing time during the viewing process, it can be considered that the user has a high preference for the video. Based on the user's emotional preference, combined with the content library and recommendation algorithm of the video platform, video content with similar emotional characteristics or in line with the user's interests is recommended to the user. This can improve the accuracy of recommended videos and user satisfaction. Thus, the video recommendation method based on multimodal feature recognition achieves a more accurate and personalized video recommendation service by comprehensively considering the user's various information sources when watching videos.

[0077] By applying the technical solution of this embodiment, based on multimodal feature recognition technology, by collecting and analyzing the target user's facial image and physiological signal data when browsing videos on the video platform using a camera device and an intelligent monitoring device, the user's viewing information is recorded at the same time; facial features, physiological features and viewing features are extracted from the collected data; emotion recognition is performed in combination with multimodal features to determine the user's emotional information on the video; the user's emotional preference is determined according to the result of emotion recognition, and personalized video content is recommended to the user based on these preferences. The embodiment of the present application uses multimodal feature recognition, and the solution can more comprehensively capture the user's emotional changes and points of interest during the video viewing process, so as to more accurately judge the user's preferences and preferences and improve the accuracy of the recommendation; personalized video recommendations can better meet the user's viewing needs, improve user satisfaction and stickiness, and at the same time, reduce the time wasted by users on content that they are not interested in, and improve the overall user experience; in addition, for video platforms, the solution helps to optimize content distribution strategies, push more video content that meets user interests to users, improve content exposure and user activity, and thus increase the commercial value of the platform.

[0078] In the embodiment of the present application, optionally, step 103 includes:

[0079] Step 103-1, performing feature fusion on the position features of the facial expression feature points, the morphological features of the facial expression feature points, the heart rate features, and the blood pressure features to obtain user features, performing emotion recognition on the user features through a pre-trained emotion recognition model, and determining the first emotion information of the target user for the video;

[0080] Step 103-2, determining the second emotion information of the target user towards the video according to the played time period, the total playing time and the played percentage;

[0081] Step 103-3, determining the target user's emotional information about the video based on the first emotional information and the second emotional information.

[0082] In this embodiment, emotion recognition is first performed based on multimodal features. Specifically, the position features, morphological features, heart rate features and blood pressure features of facial expression feature points can be fused based on a pre-trained emotion recognition model, and then emotion recognition is performed on the fused user features. This model can be built based on deep learning, machine learning or other artificial intelligence technologies, and has been trained on a data set containing multiple emotion tags. Through the reasoning process of the model, the first emotional information of the target user on the video can be output, which is usually expressed in the form of emotional categories (such as happiness, sadness, surprise, etc.) or emotional intensity (such as positivity, negativity). Through multimodal feature fusion, the user's emotional state can be captured more comprehensively and the accuracy of emotion recognition can be improved. Then, emotion inference is performed based on viewing behavior features. Specifically, the viewing behavior pattern of the target user is analyzed based on viewing behavior features such as the playback period, total playback time and playback percentage. These features reflect the user's involvement in the video content, interest points and viewing habits. Based on the analysis results of the viewing behavior, certain inference rules or algorithms are used to infer the user's second emotional information on the video. For example, a longer total playback time and a higher percentage of playback may indicate that the user has a positive attitude towards the video; while a shorter total playback time and a lower percentage of playback may indicate that the user is not interested in or is disappointed with the video. By analyzing viewing behavior characteristics, it is possible to more directly understand the user's actual feedback on the video content. Emotional inference provides an additional source of information for emotion recognition, which helps to improve the comprehensiveness and accuracy of emotion recognition. Finally, a comprehensive determination of the emotional information is performed. Specifically, the first emotional information and the second emotional information are fused. For example, weighted fusion is performed to ensure the reasonable contribution of the two emotional information in the final decision. Therefore, based on the fused emotional information, a final decision is made on the user's emotional attitude towards the video. By integrating multiple sources of emotional information, the user's emotional attitude towards the video can be grasped more accurately. Emotional decisions provide strong support for subsequent personalized recommendations, which helps to improve user experience and satisfaction.

[0083] In an embodiment of the present application, optionally, step 103-1 includes: respectively determining the change amplitudes of the position characteristics of the facial expression feature points, the morphological characteristics of the facial expression feature points, the heart rate characteristics, and the blood pressure characteristics when the target user browses the video, and determining the facial feature change amplitude based on the position feature change amplitude and the morphological feature change amplitude; determining the facial feature fusion weight, the heart rate feature fusion weight, and the blood pressure feature fusion weight based on the facial feature change amplitude, the heart rate feature change amplitude, and the blood pressure feature change amplitude, and based on the facial feature fusion weight, the heart rate feature fusion weight, and the blood pressure feature fusion weight, feature fusion of the facial features, the heart rate features, and the blood pressure features is performed to obtain the user features.

[0084] In this embodiment, first, the position and shape of the facial expression feature points (such as key points of eyes, mouth, etc.) of the target user during the video viewing process are analyzed to see how they change over time. These changes can be quantified by calculating the difference (i.e., the amplitude of change) in the position or shape of the feature points between adjacent frames. Similarly, the heart rate and blood pressure data of the target user are monitored and recorded, and their amplitude of change over time is calculated. The changes in these physiological indicators can reflect the user's emotional fluctuations. Further, based on the amplitude of change of the facial position feature and the amplitude of change of the morphological feature, the overall amplitude of change of the facial features can be comprehensively calculated to reflect the overall dynamics of the facial expression. Based on the amplitude of change of the facial feature, the amplitude of change of the heart rate feature, and the amplitude of change of the blood pressure feature, the relative importance of each in emotion recognition is determined, that is, the weight of each is determined, and the size of the weight reflects the contribution of different features to emotion recognition. For example, if the amplitude of change of the facial feature is large and is highly correlated with the target emotional state, the corresponding facial feature fusion weight will also be correspondingly high. Finally, based on the determined facial feature fusion weight, heart rate feature fusion weight, and blood pressure feature fusion weight, the original facial features, heart rate features, and blood pressure features are weighted fused to generate a user feature vector that integrates multiple feature information. The embodiment of the present application analyzes the feature change amplitude and determines the feature fusion weight. This step can more accurately evaluate the importance of different features in emotion recognition, thereby improving the accuracy and robustness of emotion recognition. The feature fusion process integrates information from multiple modalities to form a comprehensive user feature representation, which helps to more comprehensively understand the user's emotional state.

[0085] In the embodiment of the present application, optionally, step 103-2 includes:

[0086] Obtaining a key frame period, a reference playback duration, and a total video duration corresponding to the video, wherein the key frame period is a video frame period where at least one preset key segment in the video is located, and the reference playback duration indicates the playback duration of the video that has achieved an expected playback effect;

[0087] If the played time is greater than the reference played time, the ratio of the duration of at least part of the key frame period included in the played time period to the total duration of the key frame period is used as the first played time period score, and the first preset weight is determined as the first played time period weight; the first ratio of the difference between the played time and the key frame duration to the key frame duration is calculated, the sum of the first ratio and the preset score is determined as the first played time score, and the second preset weight is determined as the first played time weight; the played percentage is determined as the first played percentage score, and the first played percentage weight is determined according to the first played time period weight and the first played time weight; according to the first played time period weight, the first played time weight and the first played percentage weight, the first played time score, the first played time score and the first played percentage score are weighted and summed to determine the second emotional information of the target user towards the video;

[0088] If the played time is less than or equal to the reference played time, the ratio of the duration of at least part of the key frame period included in the played time period to the total duration of the key frame period is used as the first played time period score, and the first preset weight is determined as the second played time period weight; the second ratio of the difference between the reference played time and the played time and the key frame time is calculated, the difference between the preset score and the second ratio is used as the second played time score, and the third preset weight is determined as the second played time weight; the played percentage is determined as the second played percentage score, and the second played percentage weight is determined according to the second played time period weight and the second played time weight; according to the second played time period weight, the second played time weight and the second played percentage weight, the second played time score, the second played time score and the second played percentage score are weighted and summed to determine the second emotional information of the target user towards the video, wherein the second preset weight is less than the third preset weight.

[0089] In this embodiment, when the played time is greater than the reference played time, the first played time score and weight are calculated first: the ratio of the duration of at least part of the key frame period contained in the played time to the total duration of the key frame period is evaluated as the first played time score, which reflects how much key content the user has watched, and a preset weight (first preset weight) is assigned to the first played time score; secondly, the first played time score and weight are calculated: the ratio of the difference between the played time and the key frame time to the key frame time (first ratio) is calculated, and then this ratio is added to the preset score to obtain the first played time score, which takes into account the user's viewing experience. The part of the user's viewing time that exceeds the expected time, and another preset weight (second preset weight) is assigned to the first playback time score; then the first playback percentage score and weight are determined: the playback percentage is directly used as the first playback percentage score, and the first playback percentage weight is determined according to the first playback time period weight and the first playback time weight, for example 1-first playback time period weight-first playback time weight; finally, the second emotional information is determined by weighted summation: the first playback time period score, the first playback time score and the first playback percentage score are weightedly summed using the above weights to obtain the target user's second emotional information about the video.

[0090] When the played time is less than or equal to the reference played time, the score and weight of the second played time period are calculated first: the same as above, the proportion of the duration of at least part of the key frame period contained in the played time period to the total duration of the key frame period is evaluated as the second played time period score, and the first preset weight is used as the second played time period weight; secondly, the second played time score and weight are calculated: the proportion of the difference between the reference played time and the played time to the key frame time (the second ratio) is calculated, and then this ratio is subtracted from the preset score to obtain the second played time score. This score takes into account the part of the user's viewing time that does not meet the expected time, and assigns the third preset weight (third preset weight) to the second played time score. Note that the third preset weight here is usually greater than the second preset weight in the second case to reflect the importance of the viewing time not meeting expectations; then determine the second played percentage score and weight: Similarly, directly use the played percentage as the second played percentage score, and determine the second played percentage weight based on the second played time period weight and the second played time weight; finally, weighted sum to determine the second emotional information: Use the above weights to weight the second played time period score, the second played time score, and the second played percentage score to obtain the target user's second emotional information about the video. In this way, it is possible to comprehensively consider different aspects of the user's video viewing (such as the amount of key content viewed, the total viewing time, and the viewing percentage), so as to more accurately evaluate the user's emotional attitude towards the video.

[0091] In the embodiment of the present application, optionally, step 103-3 includes:

[0092] Comparing and analyzing the first emotional information and the second emotional information, according to the difference between the second emotional information and the first emotional information;

[0093] If the difference is within a preset threshold, determining the average of the first emotion information and the second emotion information as the emotion state information;

[0094] If the difference is not within the preset threshold, then calculate the third ratio of the difference between the difference and the preset threshold and the preset threshold, determine the difference between 0.5 and the third ratio as the fusion weight of the first emotional information, determine the sum of 0.5 and the third ratio as the fusion weight of the second emotional information, and perform weighted fusion of the first emotional information and the second emotional information to determine the emotional information.

[0095] In this embodiment, the final emotional state information is determined by combining the first emotional information (emotion based on multimodal feature recognition) and the second emotional information (emotion based on viewing behavior analysis). First, the first emotional information and the second emotional information are compared and analyzed, and the difference between them is calculated. This difference reflects the degree of difference between the results obtained by the two emotion recognition methods. Next, it is determined whether the difference falls within the preset threshold range. The preset threshold is set according to the actual application scenario and requirements, and is used to determine whether the two emotional information are close enough to be considered consistent or similar. If the difference is within the preset threshold, it means that the first emotional information and the second emotional information are relatively consistent, and their mean can be directly used as the final emotional state information. This method is simple and direct, and is suitable for situations where the difference between the two emotional information is not large. If the difference is not within the preset threshold, it means that there is a large difference between the first emotional information and the second emotional information, and more complex fusion processing is required. Specifically, the ratio of the difference between the difference and the preset threshold to the preset threshold (the third ratio) is calculated. This ratio reflects the degree of deviation of the difference relative to the preset threshold, and then the fusion weight of the first emotional information and the second emotional information is adjusted according to the third ratio. Among them, the difference between 0.5 (i.e., initial equal weight) and the third ratio is used as the fusion weight of the first emotional information, and the sum of 0.5 and the third ratio is used as the fusion weight of the second emotional information. In this way, when the difference deviates from the preset threshold by a large amount, the emotional information with a larger difference will obtain a lower weight, while the emotional information with a smaller difference will obtain a higher weight. Finally, the first emotional information and the second emotional information are weightedly fused according to the determined fusion weight to obtain the final emotional information. This method can handle the difference between the two emotional information more flexibly, making the final emotional information more accurate and reliable. Through the above steps, the degree of difference between the first emotional information and the second emotional information is comprehensively considered, and an appropriate fusion strategy is adopted to determine the final emotional state information, thereby improving the accuracy and robustness of emotion recognition.

[0096] In the embodiment of the present application, optionally, step 104 includes:

[0097] Step 104-1, if the target user's emotional preference for the video is positive, recommending a video of the same type as the video to the target user;

[0098] Step 104-2: If the target user's emotional preference for the video is neutral, recommending a video of a similar type to the video to the target user;

[0099] Step 104 - 3 : If the target user has a negative emotional preference for the video, a video of a different type and a different type than the video is recommended to the target user.

[0100] In this embodiment, when the target user's emotional preference for the video is determined to be positive, that is, the user likes the video, and the user may also be interested in the same type of content, the system recommends videos of the same type as the video to the target user, so as to meet the user's interest needs by providing videos of the same type, and enhance the user's viewing experience and satisfaction. When the target user's emotional preference for the video is determined to be neutral, that is, the user does not show obvious likes or dislikes for the video, the system recommends videos of similar types to the user. Compared with videos of the same type, similar types of videos may be similar in some aspects (such as themes, styles, actors, etc.), but may be different in other aspects, which helps to further explore the user's interests and stimulate the user's viewing interest, while avoiding repeated recommendations that may make the user feel bored. When the target user's emotional preference for the video is determined to be negative, that is, the user does not like the video, the system recommends videos of non-similar and non-similar types to the user. This strategy is intended to avoid showing users content types that they know they don't like, so as to reduce unpleasant viewing experiences, and to help users discover new points of interest by providing new content that does not conflict with the user's known preferences, while protecting the user's viewing experience from negative impacts. In addition, if the target user's emotional preference for the recommended video is negative for a certain number of consecutive times, the recommendation process can be reset to randomly recommend videos to the user, and return to step 101 to repeat the video recommendation process. In summary, through a refined recommendation strategy, the recommended content is customized according to the target user's different emotional preferences for videos, aiming to improve the user's viewing satisfaction and the platform's usage experience. At the same time, the strategy also takes into account the diversity and variability of user interests, and meets the diverse needs of users by recommending videos of different and similar types.

[0101] In addition, for each video in the video platform, video classification can be performed in advance. Specifically, computer vision and machine learning technologies can be used to analyze the image, audio, and text information in the video to identify the subject and category of the video. For example, extract key frames in the video and use image recognition technology to determine the image content, such as people, scenes, objects, etc.; analyze the audio information in the video to identify background music, human voice dialogue, etc., to assist in determining the type of video; if the video contains text descriptions in subtitles or metadata, text analysis technology can be used to extract keywords to help with classification.

[0102] In an embodiment of the present application, optionally, step 104-3 includes: if the target user has a negative emotional preference for the video, determining candidate videos of a non-similar type and non-similar type corresponding to the video, and determining the content of no interest to the target user based on the content features of the end video frame, and selecting and recommending videos from the candidate videos based on the content of no interest.

[0103] In this embodiment, when the system identifies that the target user has a negative emotional preference for a certain video, it will first look for candidate videos that are not of the same type or similar to the video. These candidate videos should ensure that they are significantly different from the videos that the target user has expressed dislike in terms of content, style, theme, etc. In this step, the system will pay special attention to the content features of the last few frames (i.e., the end video frames) when the user watches the video. These features may include objects, colors, textures, scenes, character expressions, etc. in the image, which can reflect the content information that the user is exposed to before the end of the viewing. By analyzing the content features of the end video frame, the system can infer specific content elements or types that the user may not be interested in. For example, if the user ends the video viewing after watching a fierce fighting scene, the system may think that the user is not interested in violent content. After determining the candidate video list and the user's judgment on the content that is not interesting, the system will further screen the candidate videos based on this information. Specifically, the system will exclude videos that contain content that the user has expressed no interest in, ensuring that the recommended videos will not touch the user's negative preferences. Next, the system will select the most suitable videos to recommend to the target user from the remaining candidate videos. Specifically, it can recommend videos to the target user based on a comprehensive consideration of multiple factors such as video quality, popularity, and user reviews. Finally, the system will recommend these carefully selected videos to the target user in order to provide new content that does not conflict with the user's known preferences and may attract the user. This process not only avoids recommending to users the type of videos they know they don't like, but also further considers the user's specific feelings during the viewing of a specific video, thereby achieving more personalized and accurate recommendations.

[0104] Further, as Figure 1 The specific implementation of the method, the embodiment of the present application provides a video recommendation device based on multimodal feature recognition, such as Figure 2 As shown, the device comprises:

[0105] An acquisition module, used to acquire a facial image of a target user when browsing a video on a video platform captured by a camera device and physiological signal data collected by an intelligent monitoring device worn by the target user, and to acquire viewing information of the target user on the video;

[0106] A feature extraction module, used to extract facial features corresponding to the facial image, physiological features corresponding to the physiological signal data, and viewing features corresponding to the viewing information;

[0107] A recognition module, used for performing emotion recognition on the facial features, the physiological features and the viewing features, and determining the target user's emotion information on the video;

[0108] The recommendation module is used to determine the target user's emotional preference for the video according to the emotional information, and recommend the video to the target user according to the emotional preference.

[0109] In an embodiment of the present application, optionally, the facial features include position features and morphological features of facial expression feature points, the physiological signal data include heart rate features and blood pressure features, and the viewing features include the played time period, total playback time, played percentage, and end video frame content features when the viewing of the video is finished, wherein the played time period is the video frame period in which all video clips that the target user has watched in the video are located, the total playback time is the total time the target user has watched the video, the played percentage is the video playback progress when the target user finishes viewing the video, and the end video frame content features are content features corresponding to the video frame where the target user ends viewing the video;

[0110] The identification module is further used for:

[0111] Performing feature fusion on the position features of the facial expression feature points, the morphological features of the facial expression feature points, the heart rate features, and the blood pressure features to obtain user features, performing emotion recognition on the user features through a pre-trained emotion recognition model, and determining the first emotion information of the target user on the video;

[0112] Determining the second emotion information of the target user towards the video according to the played time period, the total playing time and the played percentage;

[0113] Determine the target user's emotional information about the video according to the first emotional information and the second emotional information.

[0114] In the embodiment of the present application, optionally, the identification module is further used to:

[0115] Obtaining a key frame period, a reference playback duration, and a total video duration corresponding to the video, wherein the key frame period is a video frame period where at least one preset key segment in the video is located, and the reference playback duration indicates the playback duration of the video that has achieved an expected playback effect;

[0116] If the played time is greater than the reference played time, the ratio of the duration of at least part of the key frame period included in the played time period to the total duration of the key frame period is used as the first played time period score, and the first preset weight is determined as the first played time period weight; the first ratio of the difference between the played time and the key frame duration to the key frame duration is calculated, the sum of the first ratio and the preset score is determined as the first played time score, and the second preset weight is determined as the first played time weight; the played percentage is determined as the first played percentage score, and the first played percentage weight is determined according to the first played time period weight and the first played time weight; according to the first played time period weight, the first played time weight and the first played percentage weight, the first played time score, the first played time score and the first played percentage score are weighted and summed to determine the second emotional information of the target user towards the video;

[0117] If the played time is less than or equal to the reference played time, the ratio of the duration of at least part of the key frame period included in the played time period to the total duration of the key frame period is used as the first played time period score, and the first preset weight is determined as the second played time period weight; the second ratio of the difference between the reference played time and the played time and the key frame time is calculated, the difference between the preset score and the second ratio is used as the second played time score, and the third preset weight is determined as the second played time weight; the played percentage is determined as the second played percentage score, and the second played percentage weight is determined according to the second played time period weight and the second played time weight; according to the second played time period weight, the second played time weight and the second played percentage weight, the second played time score, the second played time score and the second played percentage score are weighted and summed to determine the second emotional information of the target user towards the video, wherein the second preset weight is less than the third preset weight.

[0118] In the embodiment of the present application, optionally, the identification module is further used to:

[0119] The user features are obtained by fusing the position features of the facial expression feature points, the morphological features of the facial expression feature points, the heart rate features, and the blood pressure features, including:

[0120] Determining the change amplitudes of the position features of the facial expression feature points, the morphological features of the facial expression feature points, the heart rate features, and the blood pressure features of the target user when browsing the video, and determining the facial feature change amplitude according to the position feature change amplitude and the morphological feature change amplitude;

[0121] According to the facial feature change amplitude, the heart rate feature change amplitude and the blood pressure feature change amplitude, the facial feature fusion weight, the heart rate feature fusion weight and the blood pressure feature fusion weight are determined, and based on the facial feature fusion weight, the heart rate feature fusion weight and the blood pressure feature fusion weight, the facial features, the heart rate features and the blood pressure features are fused to obtain the user features.

[0122] In the embodiment of the present application, optionally, the identification module is further used to:

[0123] Comparing and analyzing the first emotional information and the second emotional information, according to the difference between the second emotional information and the first emotional information;

[0124] If the difference is within a preset threshold, determining the average of the first emotion information and the second emotion information as the emotion state information;

[0125] If the difference is not within the preset threshold, then calculate the third ratio of the difference between the difference and the preset threshold and the preset threshold, determine the difference between 0.5 and the third ratio as the fusion weight of the first emotional information, determine the sum of 0.5 and the third ratio as the fusion weight of the second emotional information, and perform weighted fusion of the first emotional information and the second emotional information to determine the emotional information.

[0126] In the embodiment of the present application, optionally, the recommendation module is further used to:

[0127] If the target user has a positive emotional preference for the video, recommending a video of the same type as the video to the target user;

[0128] If the target user's emotional preference for the video is neutral, recommending videos of similar types corresponding to the video to the target user;

[0129] If the target user's emotional preference for the video is negative, a video of a different type and a different type than the video is recommended to the target user.

[0130] In the embodiment of the present application, optionally, the recommendation module is further used to:

[0131] If the target user has a negative emotional preference for the video, candidate videos of a non-similar and non-identical type to the video are determined, and the target user's uninterested content is determined based on the content features of the end video frame, and video selection and recommendation are performed among the candidate videos based on the uninterested content.

[0132] It should be noted that for other corresponding descriptions of the functional units involved in the video recommendation device based on multimodal feature recognition provided in the embodiment of the present application, reference can be made to Figure 1 to Figure 2 The corresponding description in the method will not be repeated here.

[0133] The embodiment of the present application also provides a computer device, which can be a personal computer, a server, a network device, etc. The computer device includes a bus, a processor, a memory and a communication interface, and can also include an input and output interface and a display device. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store location information. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, the steps in each method embodiment are implemented.

[0134] Those skilled in the art will appreciate that the structure of the above-mentioned computer device is only a partial structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components, or combine certain components, or have a different arrangement of components.

[0135] In one embodiment, a computer-readable storage medium is provided. The computer-readable storage medium may be non-volatile or volatile, and stores a computer program thereon. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0136] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0137] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties.

[0138] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.

[0139] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0140] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be construed as limiting the scope of the present application. It should be noted that, for a person of ordinary skill in the art, several modifications and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A video recommendation method based on multimodal feature recognition, characterized in that: The method comprises: Acquire facial images of a target user captured by a camera device when the target user browses a video on a video platform and physiological signal data collected by a smart monitoring device worn by the target user, and acquire viewing information of the target user on the video; Extracting facial features corresponding to the facial image, physiological features corresponding to the physiological signal data, and viewing features corresponding to the viewing information; Performing emotion recognition on the facial features, the physiological features, and the viewing features to determine the target user's emotion information on the video; The target user's emotional preference for the video is determined according to the emotional information, and a video is recommended to the target user according to the emotional preference.

2. The method according to claim 1, characterized in that The facial features include positional features and morphological features of facial expression feature points, the physiological signal data include heart rate features and blood pressure features, and the viewing features include the played time period, total playback time, played percentage, and end video frame content features when the video is finished being viewed, wherein the played time period is the video frame period where all video clips that the target user has watched in the video are located, the total playback time is the total time the target user has watched the video, the played percentage is the video playback progress when the target user finishes viewing the video, and the end video frame content features are content features corresponding to the video frame where the target user ends viewing the video; Performing emotion recognition on the facial features, the physiological features, and the viewing features to determine the target user's emotion information on the video includes: Performing feature fusion on the position features of the facial expression feature points, the morphological features of the facial expression feature points, the heart rate features, and the blood pressure features to obtain user features, performing emotion recognition on the user features through a pre-trained emotion recognition model, and determining the first emotion information of the target user on the video; Determining the second emotion information of the target user towards the video according to the played time period, the total playing time and the played percentage; Determine the target user's emotional information about the video according to the first emotional information and the second emotional information.

3. The method according to claim 2, characterized in that Determining the second emotion information of the target user towards the video according to the played time period, the total playing time and the played percentage, including: Obtaining a key frame period, a reference playback duration, and a total video duration corresponding to the video, wherein the key frame period is a video frame period where at least one preset key segment in the video is located, and the reference playback duration indicates the playback duration of the video that has achieved an expected playback effect; If the played time is greater than the reference played time, the ratio of the duration of at least part of the key frame period included in the played time period to the total duration of the key frame period is used as the first played time period score, and the first preset weight is determined as the first played time period weight; the first ratio of the difference between the played time and the key frame duration to the key frame duration is calculated, the sum of the first ratio and the preset score is determined as the first played time score, and the second preset weight is determined as the first played time weight; the played percentage is determined as the first played percentage score, and the first played percentage weight is determined according to the first played time period weight and the first played time weight; according to the first played time period weight, the first played time weight and the first played percentage weight, the first played time score, the first played time score and the first played percentage score are weighted and summed to determine the second emotional information of the target user towards the video; If the played time is less than or equal to the reference played time, the ratio of the duration of at least part of the key frame period included in the played time period to the total duration of the key frame period is used as the first played time period score, and the first preset weight is determined as the second played time period weight; the second ratio of the difference between the reference played time and the played time and the key frame time is calculated, the difference between the preset score and the second ratio is used as the second played time score, and the third preset weight is determined as the second played time weight; the played percentage is determined as the second played percentage score, and the second played percentage weight is determined according to the second played time period weight and the second played time weight; according to the second played time period weight, the second played time weight and the second played percentage weight, the second played time score, the second played time score and the second played percentage score are weighted and summed to determine the second emotional information of the target user towards the video, wherein the second preset weight is less than the third preset weight.

4. The method according to claim 2, characterized in that: The user features are obtained by fusing the position features of the facial expression feature points, the morphological features of the facial expression feature points, the heart rate features, and the blood pressure features, including: Determining the change amplitudes of the position features of the facial expression feature points, the morphological features of the facial expression feature points, the heart rate features, and the blood pressure features of the target user when browsing the video, and determining the facial feature change amplitude according to the position feature change amplitude and the morphological feature change amplitude; According to the facial feature change amplitude, the heart rate feature change amplitude and the blood pressure feature change amplitude, the facial feature fusion weight, the heart rate feature fusion weight and the blood pressure feature fusion weight are determined, and based on the facial feature fusion weight, the heart rate feature fusion weight and the blood pressure feature fusion weight, the facial features, the heart rate features and the blood pressure features are fused to obtain the user features.

5. The method according to claim 2, characterized in that: Determining the target user's emotional information about the video according to the first emotional information and the second emotional information includes: Comparing and analyzing the first emotional information and the second emotional information, according to the difference between the second emotional information and the first emotional information; If the difference is within a preset threshold, determining the average of the first emotion information and the second emotion information as the emotion state information; If the difference is not within the preset threshold, then calculate the third ratio of the difference between the difference and the preset threshold and the preset threshold, determine the difference between 0.5 and the third ratio as the fusion weight of the first emotional information, determine the sum of 0.5 and the third ratio as the fusion weight of the second emotional information, and perform weighted fusion of the first emotional information and the second emotional information to determine the emotional information.

6. The method according to any one of claims 2 to 5, characterized in that Recommending videos to the target user according to the emotional preference includes: If the target user has a positive emotional preference for the video, recommending a video of the same type as the video to the target user; If the target user's emotional preference for the video is neutral, recommending videos of similar types corresponding to the video to the target user; If the target user's emotional preference for the video is negative, a video of a different type and a different type than the video is recommended to the target user.

7. The method according to claim 6, characterized in that If the target user's emotional preference for the video is negative, recommending a video of a different type and a different type corresponding to the video to the target user, including: If the target user has a negative emotional preference for the video, candidate videos of a non-similar and non-identical type to the video are determined, and the target user's uninterested content is determined based on the content features of the end video frame, and video selection and recommendation are performed among the candidate videos based on the uninterested content.

8. A video recommendation device based on multimodal feature recognition, characterized in that: The device comprises: An acquisition module, used to acquire a facial image of a target user when browsing a video on a video platform, acquired by a camera device, and physiological signal data acquired by an intelligent monitoring device worn by the target user, and to acquire viewing information of the target user on the video; A feature extraction module, used to extract facial features corresponding to the facial image, physiological features corresponding to the physiological signal data, and viewing features corresponding to the viewing information; A recognition module, used for performing emotion recognition on the facial features, the physiological features and the viewing features, and determining the target user's emotion information on the video; The recommendation module is used to determine the target user's emotional preference for the video according to the emotional information, and recommend the video to the target user according to the emotional preference.

9. A storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

10. A computer device comprising a storage medium, a processor, and a computer program stored in the storage medium and executable on the processor, characterized in that: When the processor executes the computer program, the method according to any one of claims 1 to 7 is implemented.

Images