Multimedia content recommendation method and device, electronic equipment, vehicle and medium

By acquiring and analyzing users' visual and voice information, determining users' emotions, and pushing multimedia content that matches those emotions, the problem of in-vehicle device recommendations lacking humanization is solved, achieving more intelligent multimedia content recommendations.

CN118939816APending Publication Date: 2024-11-12BEIJING CO WHEELS TECH CO LTD
View PDF 5 Cites 0 Cited by

Patent Information

Application Number
CN202310524941.4
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2023-05-10
Publication Date
2024-11-12

AI Technical Summary

Technical Problem

In existing technologies, in-vehicle devices lack a human touch when recommending multimedia content and cannot intelligently push content based on the user's emotions.

Method used

By acquiring visual and voice information of the target user, and using visual and voice emotion analysis models, visual and voice emotion scores are calculated and weighted to determine the user's emotion, thereby controlling the in-vehicle device to push multimedia content that matches the user's emotion.

Benefits of technology

It improves the humanization of multimedia content recommendations, enhances the emotional connection between vehicles and users, and increases users' happiness index.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118939816A_ABST
    Figure CN118939816A_ABST
Patent Text Reader

Abstract

The invention provides a multimedia content recommendation method and device, electronic equipment, a vehicle and a medium, and the method comprises the steps: obtaining visual information and voice information of a target user, the visual information comprises at least one of a face, eyes, a mouth and a posture, and the voice information comprises at least one of semantics, mood, intonation and volume; then performing visual analysis on the visual information to obtain visual emotion scores corresponding to the emotion types, performing voice analysis on the voice information to obtain voice emotion scores corresponding to the emotion types, and performing weighted calculation on the corresponding visual emotion scores and voice emotion scores for each emotion type to obtain a voice emotion score corresponding to each emotion type; and finally, the vehicle-mounted equipment is controlled to intelligently and actively recommend at least one multimedia content conforming to the emotion of the target user to the target user, so that the humanization of multimedia content recommendation is improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of vehicle technology, and in particular to a method, device, electronic equipment, vehicle and medium for recommending multimedia content. Background Art

[0002] With the development of artificial intelligence technology, the intelligence of automobiles has been paid more and more attention. Users not only have high requirements for driving safety performance, but also attach great importance to the humanization of automobiles during driving. At present, automobiles are usually equipped with on-board devices for users to watch or listen to multimedia content, which includes video files or audio files. In related technologies, users can control on-board devices to play multimedia content through voice control commands. At this time, the on-board devices will recommend multimedia content to users based on the user's historical playback data. This recommendation method is less humanized.

[0003] Therefore, how to improve the humanization of multimedia content recommendations is an urgent problem to be solved. Summary of the invention

[0004] The present invention aims to solve the technical problems in the related art to a certain extent.

[0005] To this end, the first purpose of the present invention is to propose a method for recommending multimedia content. The method first determines the emotions of the target user based on the visual information and voice information of the target user, and then controls the vehicle-mounted device to intelligently and actively push at least one multimedia content that matches the emotions of the target user to the target user, thereby improving the humanization of multimedia content recommendation.

[0006] A second objective of the present invention is to provide a multimedia content recommendation device.

[0007] A third objective of the present invention is to provide an electronic device.

[0008] A fourth object of the present invention is to provide a vehicle.

[0009] A fifth object of the present invention is to provide a computer-readable storage medium.

[0010] A sixth object of the present invention is to provide a computer program product.

[0011] To achieve the above-mentioned purpose, the first aspect of the present invention proposes a method for recommending multimedia content, comprising the following steps: obtaining visual information and voice information of a target user; wherein the visual information includes at least one of face, eyes, mouth and posture, and the voice information includes at least one of semantics, tone, intonation and volume; performing visual analysis on the visual information to obtain a visual emotion score corresponding to each emotion type, and performing voice analysis on the voice information to obtain a voice emotion score corresponding to each emotion type; for each emotion type, performing weighted calculation on the corresponding visual emotion score and voice emotion score to determine the emotion of the target user; and controlling the vehicle-mounted device to recommend to the target user at least one multimedia content that matches the emotion of the target user.

[0012] According to the multimedia content recommendation method of the embodiment of the present invention, the visual information and voice information of the target user are first obtained, and then the visual information is visually analyzed to obtain the visual emotion score corresponding to each emotion type, and the voice information is voice analyzed to obtain the voice emotion score corresponding to each emotion type, and for each emotion type, the corresponding visual emotion score and voice emotion score are weighted to determine the emotion of the target user, and finally the vehicle-mounted device is controlled to recommend at least one multimedia content that matches the emotion of the target user to the target user. Therefore, the method first determines the emotion of the target user based on the visual information and voice information of the target user, and then controls the vehicle-mounted device to intelligently and actively push at least one multimedia content that matches the emotion of the target user to the target user, thereby improving the humanization of multimedia content recommendation.

[0013] In addition, the multimedia content recommendation method proposed in the first aspect of the present invention may also have the following additional technical features:

[0014] According to an embodiment of the present invention, for each emotion type, weighted calculation is performed on the corresponding visual emotion score and the voice emotion score to determine the emotion of the target user, including:

[0015] Performing weighted calculation on the visual emotion score of each emotion type and the speech emotion score corresponding to the emotion type to obtain a weighted score corresponding to each emotion type;

[0016] Sorting the plurality of weighted scores according to size to obtain sorting results of the plurality of emotion types;

[0017] The emotion of the target user is determined from the ranking results.

[0018] According to an embodiment of the present invention, determining the emotion of the target user from the ranking results includes:

[0019] The first two emotion types with the highest weighted scores are obtained as the first emotion and the second emotion of the target user respectively;

[0020] The emotion types of the first emotion and the second emotion are compared to determine the emotion of the target user.

[0021] According to an embodiment of the present invention, comparing the emotion types of the first emotion and the second emotion to determine the emotion of the target user includes:

[0022] In a case where the emotion types of the first emotion and the second emotion are contradictory, determining that the emotion of the target user is the first emotion;

[0023] In a case where the emotion types of the first emotion and the second emotion are not contradictory, it is determined that the emotion of the target user includes the first emotion and the second emotion.

[0024] According to an embodiment of the present invention, controlling the vehicle-mounted device to recommend at least one multimedia content that matches the target user's emotions to the target user includes:

[0025] Determining at least one emotion tag corresponding to the emotion of the target user;

[0026] Acquire at least one of the multimedia contents related to each of the emotion tags;

[0027] Obtaining the corresponding matching degree between each of the multimedia contents and the emotion of the target user;

[0028] Sorting the plurality of multimedia contents according to the magnitude relationship between the matching degrees to obtain a sorting result of the multimedia contents recommended to the target user;

[0029] The in-vehicle device is controlled to recommend the multimedia content to the target user according to the ranking result.

[0030] According to an embodiment of the present invention, before acquiring the visual information and voice information of the target user, the method further includes:

[0031] Acquire the identity information of the current user; wherein the identity information is used to characterize the biometric characteristics of the current user;

[0032] Matching the identity information of the current user with the set identity information;

[0033] When the identity information of the current user successfully matches the set identity information, the current user is determined to be the target user.

[0034] To achieve the above-mentioned purpose, the second aspect of the present invention proposes a multimedia content recommendation device, including: an acquisition module, used to acquire visual information and voice information of a target user; wherein the visual information includes at least one of face, eyes, mouth and posture, and the voice information includes at least one of semantics, tone, intonation and volume; a visual analysis module, used to perform visual analysis on the visual information to obtain a visual emotion score corresponding to each emotion type; a voice analysis module, used to perform voice analysis on the voice information to obtain a voice emotion score corresponding to each emotion type; a determination module, used to perform weighted calculation on the corresponding visual emotion score and the voice emotion score for each emotion type to determine the emotion of the target user; and a control module, used to control the vehicle-mounted device to recommend to the target user at least one multimedia content that matches the emotion of the target user.

[0035] According to the multimedia content recommendation device of the embodiment of the present invention, the acquisition module acquires the visual information and voice information of the target user, the visual analysis module performs visual analysis on the visual information to obtain the visual emotion score corresponding to each emotion type, the voice analysis module performs voice analysis on the voice information to obtain the voice emotion score corresponding to each emotion type, and the determination module performs weighted calculation on the corresponding visual emotion score and voice emotion score for each emotion type to determine the emotion of the target user, so that the control module controls the vehicle-mounted device to recommend at least one multimedia content that matches the emotion of the target user to the target user. Thus, the device determines the emotion of the target user based on the visual information and voice information of the target user, and then controls the vehicle-mounted device to intelligently and actively push at least one multimedia content that matches the emotion of the target user to the target user, thereby improving the humanization of multimedia content recommendation.

[0036] In addition, the multimedia content recommendation device provided in the second aspect of the present invention may also have the following additional technical features:

[0037] According to an embodiment of the present invention, the determination module is used to perform weighted calculation on the corresponding visual emotion score and the voice emotion score for each emotion type to determine the emotion of the target user, including:

[0038] Performing weighted calculation on the visual emotion score of each emotion type and the speech emotion score corresponding to the emotion type to obtain a weighted score corresponding to each emotion type;

[0039] Sorting the plurality of weighted scores according to size to obtain sorting results of the plurality of emotion types;

[0040] The emotion of the target user is determined from the ranking results.

[0041] According to an embodiment of the present invention, when the determination module is used to determine the emotion of the target user from the sorting result, it includes:

[0042] The first two emotion types with the highest weighted scores are obtained as the first emotion and the second emotion of the target user respectively;

[0043] The emotion types of the first emotion and the second emotion are compared to determine the emotion of the target user.

[0044] According to an embodiment of the present invention, when the determination module is used to compare the emotion types of the first emotion and the second emotion to determine the emotion of the target user, it includes:

[0045] In a case where the emotion types of the first emotion and the second emotion are contradictory, determining that the emotion of the target user is the first emotion;

[0046] In a case where the emotion types of the first emotion and the second emotion are not contradictory, it is determined that the emotion of the target user includes the first emotion and the second emotion.

[0047] According to an embodiment of the present invention, when the control module is used to control the vehicle-mounted device to recommend at least one multimedia content that matches the emotion of the target user to the target user, the control module includes:

[0048] Determining at least one emotion tag corresponding to the emotion of the target user;

[0049] Acquire at least one of the multimedia contents related to each of the emotion tags;

[0050] Obtaining the corresponding matching degree between each of the multimedia contents and the emotion of the target user;

[0051] Sorting the plurality of multimedia contents according to the magnitude relationship between the matching degrees to obtain a sorting result of the multimedia contents recommended to the target user;

[0052] The in-vehicle device is controlled to recommend the multimedia content to the target user according to the ranking result.

[0053] According to an embodiment of the present invention, before the acquisition module is used to acquire the visual information and voice information of the target user, it is also used to:

[0054] Acquire the identity information of the current user; wherein the identity information is used to characterize the biometric characteristics of the current user;

[0055] Matching the identity information of the current user with the set identity information;

[0056] When the identity information of the current user successfully matches the set identity information, the current user is determined to be the target user.

[0057] To achieve the above-mentioned purpose, the third aspect of the present invention proposes an electronic device, comprising: a processor and a memory; wherein the processor runs a program corresponding to the executable program code by reading the executable program code stored in the memory, so as to implement the multimedia content recommendation method of the first aspect of the embodiment.

[0058] The electronic device of the embodiment of the present invention, by executing the above-mentioned multimedia content recommendation method, first determines the target user's emotions based on the target user's visual information and voice information, and then controls the vehicle-mounted device to intelligently and actively push at least one multimedia content that matches the target user's emotions to the target user, thereby improving the humanization of multimedia content recommendation.

[0059] To achieve the above-mentioned object, a fourth aspect of the present invention provides a vehicle, which includes the above-mentioned electronic device.

[0060] The vehicle of the embodiment of the present invention, by using the above-mentioned electronic device, can determine the emotion of the target user based on the visual information and voice information of the target user, and then control the on-board equipment to intelligently and actively push at least one multimedia content that matches the emotion of the target user to the target user, thereby improving the humanization of the multimedia content recommendation.

[0061] To achieve the above-mentioned purpose, a fifth aspect of the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method for recommending multimedia content of the first aspect of the present invention.

[0062] The computer-readable storage medium of an embodiment of the present invention can determine the target user's emotions based on the target user's visual information and voice information by executing the above-mentioned multimedia content recommendation method, and then control the vehicle-mounted device to intelligently and actively push at least one multimedia content that matches the target user's emotions to the target user, thereby improving the humanization of multimedia content recommendation.

[0063] To achieve the above-mentioned purpose, a sixth aspect of the present invention provides a computer program product. When an instruction processor in the computer program product executes, the method for recommending multimedia content of the first aspect of the present invention is executed.

[0064] The computer program product of an embodiment of the present invention can determine the target user's emotions based on the target user's visual information and voice information by executing the above-mentioned multimedia content recommendation method, and then control the vehicle-mounted device to intelligently and actively push at least one multimedia content that matches the target user's emotions to the target user, thereby improving the humanization of multimedia content recommendation.

[0065] Additional aspects and advantages of the present invention will be given in part in the following description and in part will be obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] The above and / or additional aspects and advantages of the present invention will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0067] Figure 1 is a flow chart of a method for recommending multimedia content according to an embodiment of the present invention;

[0068] Figure 2 is a flow chart of a method for recommending multimedia content according to an embodiment of the present invention;

[0069] Figure 3 is a flow chart of a method for recommending multimedia content according to another embodiment of the present invention;

[0070] Figure 4 is a flow chart of a method for recommending multimedia content according to yet another embodiment of the present invention;

[0071] Figure 5 4 is a block diagram of a multimedia content recommendation device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0072] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present invention, and should not be construed as limiting the present invention.

[0073] The following describes a method, apparatus, electronic device, vehicle, and medium for recommending multimedia content according to embodiments of the present invention with reference to the accompanying drawings.

[0074] Figure 1 is a flowchart of a method for recommending multimedia content according to an embodiment of the present invention.

[0075] It should be noted that the executor of the multimedia content recommendation method of the embodiment of the present invention is the multimedia content recommendation device of the embodiment of the present invention. The multimedia content recommendation device of the embodiment of the present invention can be set in an electronic device, for example, it can be an electronic device on the vehicle side such as a vehicle-mounted device, or it can be an electronic device on the cloud such as a server. The specific setting location is not limited in the present invention.

[0076] like Figure 1 As shown, the method for recommending multimedia content according to an embodiment of the present invention comprises the following steps:

[0077] S101, obtaining visual information and voice information of a target user; wherein the visual information includes at least one of face, eyes, mouth and posture, and the voice information includes at least one of semantics, tone, intonation and volume.

[0078] For example, the target user can be tracked in real time by a vehicle camera installed in the vehicle to obtain visual information and voice information of the target user, wherein the visual information includes at least one of face, eyes, mouth and posture, and the voice information includes at least one of semantics, tone, intonation and volume.

[0079] S102, performing visual analysis on the visual information to obtain visual emotion scores corresponding to each emotion type, and performing voice analysis on the voice information to obtain voice emotion scores corresponding to each emotion type.

[0080] The emotion type includes at least one of the following: happiness, excitement, anger, fear, sadness, and quietness.

[0081] In step S102, after obtaining the visual information and the voice information, the visual information is input into a pre-trained visual emotion analysis model to extract the image information such as the face, eyes, mouth and expression of the target user, and the face, eyes, mouth and expression of the target user are analyzed to obtain the visual emotion scores corresponding to each emotion type; at the same time, the voice information is input into a pre-trained voice emotion analysis model to extract the semantics, tone, intonation and volume of the target user's voice, and the semantics, tone, intonation and volume of the target user's voice are analyzed to obtain the voice emotion scores corresponding to each emotion type.

[0082] It should be noted that the visual emotion analysis model and the speech emotion analysis model need to be pre-trained. The training of the model requires obtaining multiple visual information samples and multiple speech information samples, and training the multiple visual information samples and multiple speech information samples respectively to obtain the corresponding visual emotion analysis model and speech emotion analysis model.

[0083] S103, for each emotion type, weighted calculation is performed on the corresponding visual emotion score and voice emotion score to determine the emotion of the target user.

[0084] In step S103, for each emotion type, the corresponding visual emotion score and voice emotion score are weightedly calculated to obtain a weighted score corresponding to each emotion type. The weighted scores corresponding to the various emotion types are then sorted according to size, for example, in descending order, to obtain sorting results of multiple emotion types, and the emotions of the target user are determined from the sorting results.

[0085] Assume that after analyzing the visual information and voice information of the target user respectively, the visual emotion scores and voice emotion scores corresponding to each emotion type are obtained as follows: the visual emotion score D11 of happiness is 70, and the voice emotion score D12 is 80; the visual emotion score D21 of excitement is 50, and the voice emotion score D22 is 70; the visual emotion score D31 of anger is 10, and the voice emotion score D32 is 10; the visual emotion score D41 of fear is 0, and the voice emotion score is 0; the visual emotion score D51 of sadness is 60, and the voice emotion score D52 is 70; the visual emotion score D61 of quietness is 40, and the voice emotion score D62 is 50. In addition, the weight Q1 of the visual emotion can be set to 0.6 in advance, and the weight Q2 of the voice emotion can be set to 0.4. It should be noted that the user can adjust the weight Q1 of the visual emotion and the weight Q2 of the voice emotion according to their own needs. For example, when there is only one person in the vehicle, the weight Q1 of the visual emotion can be adjusted to 0.9, and the weight Q2 of the voice emotion can be adjusted to 0.1.

[0086] Then, according to the visual emotion score and voice emotion score of each emotion type obtained above, the weighted scores corresponding to various emotion types are calculated respectively: the weighted score of happiness D1=D11*Q1+D12*Q2=70*0.6+80*0.4=74; the weighted score of excitement D2=D21*Q1+D22*Q2=50*0.6+70*0.4=58; the weighted score of anger D3=D31*Q1+D32*Q2=10*0.6+10*0.4=10; the weighted score of fear D4=D41*Q1+D42*Q2=0; the weighted score of sadness D5=D51*Q1+D52*Q2=60*0.6+70*0.4=64; the weighted score of quietness D6=D61*Q1+D62*Q2=40*0.6+50*0.4=44.

[0087] After obtaining the weighted scores corresponding to happiness, excitement, anger, fear, sadness, and quietness respectively, the ranking results are obtained in descending order: the weighted score of happiness is D1, the weighted score of sadness is D5, the weighted score of excitement is D2, the weighted score of quietness is D6, the weighted score of anger is D3, and the weighted score of fear is D4.

[0088] As a feasible process for determining the emotion of the target user from the sorting results, the process includes: obtaining the emotion type with the first weighted score from the sorting results as the emotion of the target user.

[0089] For example, when the ranking results are: weighted score D1 for happiness, weighted score D5 for sadness, weighted score D2 for excitement, weighted score D6 for quietness, weighted score D3 for anger, and weighted score D4 for fear, happiness is taken as the emotion of the target user.

[0090] As another achievable process for determining the target user's emotion from the sorting results, the process includes: obtaining the first two emotion types with the highest weighted scores from the sorting results, respectively as the first emotion and the second emotion of the target user, and then comparing the emotion types of the first emotion and the second emotion to determine the target user's emotion. Wherein, when the emotion types of the first emotion and the second emotion are contradictory, the target user's emotion is determined to be the first emotion; when the emotion types of the first emotion and the second emotion are not contradictory, the target user's emotion is determined to include the first emotion and the second emotion.

[0091] Let’s first give examples of contradictory emotion combinations and non-contradictory emotion combinations. For example, contradictory emotion combinations include: happiness and anger, excitement and sadness, fear and happiness, etc.; non-contradictory emotion combinations include: happiness and excitement, anger and fear, fear and sadness, sadness and quietness, etc.

[0092] For example, when the sorting results are: weighted score D1 for happiness, weighted score D5 for sadness, weighted score D2 for excitement, weighted score D6 for quietness, weighted score D3 for anger, and weighted score D4 for fear, the first emotion is happiness and the second emotion is sadness, but because the first emotion and the second emotion are a contradictory combination of emotions, the emotion of the target user is determined to be the first emotion, i.e., happiness.

[0093] When the sorting results are: weighted score D1 for happiness, weighted score D6 for quietness, weighted score D5 for sadness, weighted score D2 for excitement, weighted score D3 for anger, and weighted score D4 for fear, the first emotion is happiness and the second emotion is quietness. Since the first emotion and the second emotion are non-contradictory emotion combinations, it is determined that the target user's emotions include happiness and quietness.

[0094] S104, controlling the in-vehicle device to recommend at least one multimedia content that matches the target user's emotions to the target user.

[0095] As an implementable method of step S104, it includes: after obtaining the emotion of the target user, determining at least one emotion tag corresponding to the emotion of the target user, obtaining at least one multimedia content related to each emotion tag, and controlling the vehicle-mounted device to recommend the multimedia content to the target user.

[0096] Specifically, the multimedia content automatic recognition system can be used to recognize pictures and voices in different multimedia materials (videos, music, etc.) to obtain the emotional tags to which the multimedia belongs. In this example, when the target user's emotion is determined to be happy, the emotional tag corresponding to happiness is determined to be happy, and then at least one multimedia content matching happiness is obtained, such as "Change Yourself", "Only", "Loving You Is Equal to Loving Yourself", "Forever's First Day", etc.; when the target user's emotion is determined to be sad, the emotional tag corresponding to the target emotion is sad, and then at least one multimedia content matching sadness is obtained, such as "Gentle Mercy", "Ignition Point", "Passing", "Reason"; when the target user's emotion includes happiness and quietness, the emotional tags corresponding to the target user's emotion include joy and quietness, and then the multimedia content ranked in the top three of joy and quietness is obtained as the multimedia content to be played, such as Canon Pure Music, "Kiss the Rain", etc.

[0097] Among them, it should be noted that, when the target user has one emotion, the order of playing the multimedia content can be random, can be a single loop, or can be played in sequence, and there is no specific restriction here; when the target user has two emotions, the order of playing the multimedia content can be to play each song one by one.

[0098] Therefore, the multimedia content recommendation method of the embodiment of the present invention determines the target user's emotions based on the target user's visual information and voice information, and then controls the vehicle-mounted device to intelligently and actively push at least one multimedia content that matches the target user's emotions to the target user, thereby improving the humanization of multimedia content recommendation, allowing the car to better understand the user's heart, strengthening the emotional communication between the vehicle and the user, and improving the user's happiness index.

[0099] Figure 2 is a flowchart of a method for recommending multimedia content according to an embodiment of the present invention.

[0100] like Figure 2 As shown, the method for recommending multimedia content according to an embodiment of the present invention comprises the following steps:

[0101] S201, obtaining visual information and voice information of a target user; wherein the visual information includes at least one of face, eyes, mouth and posture, and the voice information includes at least one of semantics, tone, intonation and volume.

[0102] S202, performing visual analysis on the visual information to obtain visual emotion scores corresponding to each emotion type, and performing voice analysis on the voice information to obtain voice emotion scores corresponding to each emotion type.

[0103] S203 , for each emotion type, weighted calculation is performed on the corresponding visual emotion score and voice emotion score to determine the emotion of the target user.

[0104] It should be noted that, for the execution process of step S201 to step S203 in the embodiment of the present invention, please refer to the execution process of step S101 to step S103 in the embodiment of the present invention, and the details will not be repeated here.

[0105] S204: Determine at least one emotion tag corresponding to the emotion of the target user.

[0106] For example, when the target user's emotion is happy, the emotion label is happy; when the target user's emotion includes happy and quiet, the emotion label includes happy and quiet.

[0107] S205: Acquire at least one multimedia content related to each emotion tag.

[0108] For example, when it is determined that the emotion label is happiness, at least one multimedia content related to happiness is obtained as the multimedia content to be played; when it is determined that the emotion label includes happiness and quiet, at least one multimedia content related to happiness and at least one multimedia content related to quiet are obtained as the multimedia content to be played.

[0109] S206: Obtain the corresponding matching degree between each multimedia content and the emotion of the target user.

[0110] It should be noted that the multimedia content automatic recognition system pre-processes different types of multimedia content (video, music, etc.), such as converting the voice of the multimedia content into text, and performing semantic analysis on the text to obtain the emotional tags to which the multimedia content belongs. It also calculates and generates a table of scores between the multimedia content and each emotional tag based on the frequency of occurrence of different keywords in the multimedia content, and stores it in the vehicle-mounted device for subsequent use.

[0111] In this step, after the multimedia content to be played is obtained, the score between each multimedia content and each emotion tag is obtained by calling the lookup table, and then, according to the obtained score between each multimedia content and each emotion tag, the corresponding matching degree between each multimedia content and the emotion of the target user is calculated. For example, when the emotion of the target user is happy, the score between each multimedia content and happiness is used as the matching degree between the corresponding multimedia content and happiness; when the emotion of the target user includes happiness and quietness, the score between each multimedia content and happiness is multiplied by the first proportion of 60% to obtain a first product, and the score between each multimedia content and quietness is multiplied by the second proportion of 40% to obtain a second product, and then the sum of the first product and the second product is calculated as the corresponding matching degree between each multimedia content and the emotion of the target user.

[0112] S207: sorting the plurality of multimedia contents according to the magnitude relationship between the matching degrees to obtain a sorting result of the multimedia contents recommended to the target user.

[0113] In this step, after obtaining the corresponding matching degree between each multimedia content and the emotion of the target user, the plurality of multimedia contents are sorted according to the matching degree from large to small to obtain a sorting result.

[0114] For example, the sorting results of multimedia content matching happiness are: "The Only One", "Loving You is Loving Myself", "Change Myself", "The First Day of Forever", etc.; at least one multimedia content matching happiness and quietness includes: "Loving You is Loving Myself", Canon Pure Music, "Kiss the Rain", "The First Day of Forever", etc.

[0115] S208, controlling the in-vehicle device to recommend multimedia content to the target user according to the ranking result.

[0116] In this step, after the sorting result is obtained, the sorting result is presented to the target user through the central control screen so that the target user can understand the order of playing the multimedia content at the current time.

[0117] Therefore, the multimedia content recommendation method of the embodiment of the present invention, when pushing at least one multimedia content that matches the target user's emotions to the target user, obtains the corresponding matching degree between each multimedia content and the target user's emotions, and sorts the multimedia content according to each matching degree to obtain the recommendation result of the multimedia content recommended to the target user. Therefore, the recommendation method of the present invention can further improve the relevance of the recommended multimedia content and the target user's emotions, further improve the humanization of multimedia content recommendation, let the car better understand the user's heart, further strengthen the emotional communication between the vehicle and the user, and further improve the user's happiness index.

[0118] Figure 3 is a flowchart of a method for recommending multimedia content according to another embodiment of the present invention.

[0119] like Figure 3 As shown, the method for recommending multimedia content according to an embodiment of the present invention comprises the following steps:

[0120] S301, obtaining identity information of the current user; wherein the identity information is used to represent the biometric characteristics of the current user.

[0121] The biometric feature may be at least one of fingerprint, face, iris and voiceprint.

[0122] For example, fingerprints can be obtained through fingerprint sensors; faces and irises can be obtained through vehicle-mounted cameras; and voiceprints can be obtained through voiceprint sensors.

[0123] S302: Match the current user's identity information with the set identity information.

[0124] The set identity information may be identity information pre-stored in the cloud or on the vehicle side for matching with the uploaded identity information of the current user.

[0125] S303: When the identity information of the current user successfully matches the set identity information, determine that the current user is the target user.

[0126] In this step, if the current user's identity information matches the set identity information successfully, the current user is determined to be the target user, and the following steps S304-S307 are executed; if the current user's identity information does not match the set identity information successfully, the current user is determined not to be the target user, and the process ends.

[0127] S304, obtaining visual information and voice information of the target user; wherein the visual information includes at least one of face, eyes, mouth and posture, and the voice information includes at least one of semantics, tone, intonation and volume.

[0128] S305, performing visual analysis on the visual information to obtain visual emotion scores corresponding to each emotion type, and performing voice analysis on the voice information to obtain voice emotion scores corresponding to each emotion type.

[0129] S306 , for each emotion type, weighted calculation is performed on the corresponding visual emotion score and voice emotion score to determine the emotion of the target user.

[0130] S307: Control the in-vehicle device to recommend at least one multimedia content that matches the target user's emotions to the target user.

[0131] It should be noted that the execution process of the above steps S304 to S307 can refer to the process described in the above steps S101 to S104, and the details will not be repeated here.

[0132] Therefore, the multimedia content recommendation method of the embodiment of the present invention first determines whether the current user is the target user, and if so, determines the target user's emotions based on the target user's visual information and voice information, and then controls the vehicle-mounted device to intelligently and actively push at least one multimedia content that matches the target user's emotions to the target user. In other words, the present invention sets permissions for the multimedia content recommendation function, and only when the current user is the target user, the function is enabled.

[0133] In order to make those skilled in the art understand the present invention more clearly, the following Figure 4 The method for recommending multimedia content according to an embodiment of the present invention is described as follows. Figure 4 As shown, the method for recommending multimedia content according to an embodiment of the present invention comprises the following steps:

[0134] S401, determining that a user enters the vehicle.

[0135] S402, collecting the identity information of the current user.

[0136] S403, identifying whether the current user is the target user. If yes, executing step S404; if no, ending.

[0137] S404, collecting visual information and voice information of the target user.

[0138] S405: Analyze the visual information and the voice information through the emotion analysis system to obtain the target emotion of the target user. It should be noted that the emotion analysis system includes a visual emotion analysis model and a voice emotion analysis model. The visual information is analyzed by the visual emotion analysis model, and the voice information is analyzed by the voice emotion analysis model.

[0139] S406, performing semantic analysis on the multimedia content through the multimedia content recognition system to obtain emotion tags, and calculating the matching degree between each multimedia content and the emotion tag.

[0140] S407, obtaining the emotion tag corresponding to the target emotion through the multimedia intelligent push system, and obtaining at least one multimedia content matching the emotion tag. In the process of executing step S407, the matching degree corresponding to each multimedia content can also be obtained, and the multimedia can be sorted according to each matching degree.

[0141] S408: Push the multimedia content to the target user. It should be noted that the order of pushing can be random, or can be pushed according to the order result obtained in step S407.

[0142] S409, determining that the target user's emotion has changed.

[0143] S410, the multimedia intelligent push system adjusts the pushed multimedia content.

[0144] It should be noted that after executing step S410, the process returns to executing step S404 to collect the visual information and voice information of the target user in real time and obtain the target emotion of the target user, and then adjust the pushed multimedia content in real time according to the target emotion.

[0145] In summary, according to the multimedia content recommendation method of the embodiment of the present invention, the visual information and voice information of the target user are first obtained, and then the visual information is visually analyzed to obtain the visual emotion score corresponding to each emotion type, and the voice information is voice analyzed to obtain the voice emotion score corresponding to each emotion type, and for each emotion type, the corresponding visual emotion score and voice emotion score are weightedly calculated to determine the emotion of the target user, and finally the vehicle-mounted device is controlled to recommend at least one multimedia content that matches the emotion of the target user to the target user. Therefore, the method first determines the emotion of the target user based on the visual information and voice information of the target user, and then controls the vehicle-mounted device to intelligently and actively push at least one multimedia content that matches the emotion of the target user to the target user, thereby improving the humanization of multimedia content recommendation.

[0146] Figure 5 4 is a block diagram of a multimedia content recommendation device according to an embodiment of the present invention.

[0147] like Figure 5 As shown, the multimedia content recommendation device 500 according to the embodiment of the present invention includes: an acquisition module 510 , a visual analysis module 520 , a speech analysis module 530 , a determination module 540 and a control module 550 .

[0148] Among them, the acquisition module 510 is used to obtain the visual information and voice information of the target user; wherein the visual information includes at least one of face, eyes, mouth and posture, and the voice information includes at least one of semantics, tone, intonation and volume. The visual analysis module 520 is used to perform visual analysis on the visual information to obtain the visual emotion score corresponding to each emotion type. The voice analysis module 530 is used to perform voice analysis on the voice information to obtain the voice emotion score corresponding to each emotion type. The determination module 540 is used to perform weighted calculation on the corresponding visual emotion score and voice emotion score for each emotion type to determine the emotion of the target user. The control module 550 is used to control the vehicle-mounted device to recommend at least one multimedia content that matches the emotion of the target user to the target user.

[0149] According to an embodiment of the present invention, the determination module 540 is used to perform weighted calculation on the corresponding visual emotion score and voice emotion score for each emotion type to determine the emotion of the target user, including:

[0150] The visual emotion score of each emotion type and the speech emotion score of the corresponding emotion type are weighted and calculated to obtain a weighted score corresponding to each emotion type;

[0151] Sort multiple weighted scores by size to obtain sorting results of multiple emotion types;

[0152] Determine the target user's sentiment from the ranking results.

[0153] According to an embodiment of the present invention, when the determination module 540 is used to determine the emotion of the target user from the sorting results, it includes:

[0154] The top two emotion types with the highest weighted scores are obtained and used as the first emotion and the second emotion of the target user respectively;

[0155] The emotion types of the first emotion and the second emotion are compared to determine the emotion of the target user.

[0156] According to an embodiment of the present invention, when the determination module 540 is used to compare the emotion types of the first emotion and the second emotion to determine the emotion of the target user, it includes:

[0157] When the emotion types of the first emotion and the second emotion are contradictory, determining the emotion of the target user to be the first emotion;

[0158] In a case where the emotion types of the first emotion and the second emotion are not contradictory, it is determined that the emotion of the target user includes the first emotion and the second emotion.

[0159] According to an embodiment of the present invention, the control module 550 is used to control the vehicle-mounted device to recommend at least one multimedia content that matches the emotion of the target user to the target user, including:

[0160] determining at least one emotion tag corresponding to an emotion of the target user;

[0161] Acquire at least one multimedia content related to each emotion tag;

[0162] Obtaining the corresponding matching degree between each multimedia content and the emotion of the target user;

[0163] Sorting the plurality of multimedia contents according to the magnitude relationship between the respective matching degrees to obtain a ranking result of the multimedia contents recommended to the target user;

[0164] The vehicle-mounted device is controlled to recommend multimedia content to target users according to the sorting results.

[0165] According to an embodiment of the present invention, before the acquisition module 510 is used to acquire the visual information and voice information of the target user, it is also used to:

[0166] Obtaining the identity information of the current user; wherein the identity information is used to characterize the biometric characteristics of the current user;

[0167] Match the current user's identity information with the set identity information;

[0168] When the identity information of the current user successfully matches the set identity information, the current user is determined to be the target user.

[0169] It should be noted that for details not disclosed in the multimedia content recommendation device of the embodiment of the present invention, please refer to the details disclosed in the multimedia content recommendation method of the embodiment of the present invention, and no further details will be given here.

[0170] According to the multimedia content recommendation device of the embodiment of the present invention, the acquisition module acquires the visual information and voice information of the target user, the visual analysis module performs visual analysis on the visual information to obtain the visual emotion score corresponding to each emotion type, the voice analysis module performs voice analysis on the voice information to obtain the voice emotion score corresponding to each emotion type, and the determination module performs weighted calculation on the corresponding visual emotion score and voice emotion score for each emotion type to determine the emotion of the target user, so that the control module controls the vehicle-mounted device to recommend at least one multimedia content that matches the emotion of the target user to the target user. Thus, the device determines the emotion of the target user based on the visual information and voice information of the target user, and then controls the vehicle-mounted device to intelligently and actively push at least one multimedia content that matches the emotion of the target user to the target user, thereby improving the humanization of multimedia content recommendation.

[0171] Based on the above embodiments, the present invention further provides an electronic device.

[0172] The electronic device of the embodiment of the present invention comprises: a processor and a memory; wherein the processor runs a program corresponding to the executable program code by reading the executable program code stored in the memory, so as to implement the above-mentioned multimedia content recommendation method.

[0173] The electronic device of the embodiment of the present invention, by executing the above-mentioned multimedia content recommendation method, first determines the target user's emotions based on the target user's visual information and voice information, and then controls the vehicle-mounted device to intelligently and actively push at least one multimedia content that matches the target user's emotions to the target user, thereby improving the humanization of multimedia content recommendation.

[0174] Based on the above embodiments, the present invention also provides a vehicle.

[0175] The vehicle according to the embodiment of the present invention includes the above-mentioned electronic device.

[0176] The vehicle of the embodiment of the present invention, by using the above-mentioned electronic device, can determine the emotion of the target user based on the visual information and voice information of the target user, and then control the on-board equipment to intelligently and actively push at least one multimedia content that matches the emotion of the target user to the target user, thereby improving the humanization of the multimedia content recommendation.

[0177] Based on the above embodiments, the present invention also proposes a computer-readable storage medium.

[0178] A computer program is stored on a computer-readable storage medium in an embodiment of the present invention. When the computer program is executed by a processor, the method for recommending multimedia content in the embodiment of the first aspect is implemented.

[0179] The computer-readable storage medium of an embodiment of the present invention can determine the target user's emotions based on the target user's visual information and voice information by executing the above-mentioned multimedia content recommendation method, and then control the vehicle-mounted device to intelligently and actively push at least one multimedia content that matches the target user's emotions to the target user, thereby improving the humanization of multimedia content recommendation.

[0180] Based on the above embodiments, the present invention also proposes a computer program product.

[0181] When the instruction processor in the computer program product of the embodiment of the present invention is executed, the above-mentioned method for recommending multimedia content is performed.

[0182] The computer program product of an embodiment of the present invention can determine the target user's emotions based on the target user's visual information and voice information by executing the above-mentioned multimedia content recommendation method, and then control the vehicle-mounted device to intelligently and actively push at least one multimedia content that matches the target user's emotions to the target user, thereby improving the humanization of multimedia content recommendation.

[0183] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0184] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In the description of the present invention, the meaning of "plurality" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0185] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, segment or portion of code comprising one or more executable instructions for implementing the steps of a custom logical function or process, and the scope of the preferred embodiments of the present invention includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in reverse order depending on the functions involved, which should be understood by those skilled in the art to which the embodiments of the present invention belong.

[0186] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in combination with these instruction execution systems, devices or apparatuses. For the purpose of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in combination with these instruction execution systems, devices or apparatuses. More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or more wires (electronic device), a portable computer disk box (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium and then editing, interpreting or processing in other suitable ways if necessary, and then stored in a computer memory.

[0187] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, a plurality of steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0188] A person skilled in the art may understand that all or part of the steps in the method for implementing the above-mentioned embodiment may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiment.

[0189] In addition, each functional unit in each embodiment of the present invention may be integrated into one module, or each unit may exist physically separately, or two or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.

[0190] The storage medium mentioned above may be a read-only memory, a magnetic disk or an optical disk, etc. Although the embodiments of the present invention have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limiting the present invention. A person of ordinary skill in the art may change, modify, replace and modify the above embodiments within the scope of the present invention.

Claims

1. A method for recommending multimedia content, characterized in that: The following steps are involved: Acquire visual information and voice information of the target user; wherein the visual information includes at least one of face, eyes, mouth and posture, and the voice information includes at least one of semantics, tone, intonation and volume; Performing visual analysis on the visual information to obtain visual emotion scores corresponding to each emotion type, and performing voice analysis on the voice information to obtain voice emotion scores corresponding to each emotion type; For each emotion type, weighted calculation is performed on the corresponding visual emotion score and the voice emotion score to determine the emotion of the target user; The in-vehicle device is controlled to recommend at least one multimedia content that matches the emotion of the target user to the target user.

2. The method according to claim 1, characterized in that For each emotion type, weighted calculation is performed on the corresponding visual emotion score and the voice emotion score to determine the emotion of the target user, including: Performing weighted calculation on the visual emotion score of each emotion type and the speech emotion score corresponding to the emotion type to obtain a weighted score corresponding to each emotion type; Sorting the plurality of weighted scores according to size to obtain sorting results of the plurality of emotion types; The emotion of the target user is determined from the ranking results.

3. The method according to claim 2, characterized in that Determining the emotion of the target user from the sorting results includes: The first two emotion types with the highest weighted scores are obtained as the first emotion and the second emotion of the target user respectively; The emotion types of the first emotion and the second emotion are compared to determine the emotion of the target user.

4. The method according to claim 3, characterized in that The comparing the emotion types of the first emotion and the second emotion to determine the emotion of the target user includes: In a case where the emotion types of the first emotion and the second emotion are contradictory, determining that the emotion of the target user is the first emotion; In a case where the emotion types of the first emotion and the second emotion are not contradictory, it is determined that the emotion of the target user includes the first emotion and the second emotion.

5. The method according to claim 1, characterized in that The controlling the vehicle-mounted device to recommend at least one multimedia content that matches the target user's emotions to the target user comprises: Determining at least one emotion tag corresponding to the emotion of the target user; Acquire at least one of the multimedia contents related to each of the emotion tags; Obtaining the corresponding matching degree between each of the multimedia contents and the emotion of the target user; Sorting the plurality of multimedia contents according to the magnitude relationship between the matching degrees to obtain a sorting result of the multimedia contents recommended to the target user; The in-vehicle device is controlled to recommend the multimedia content to the target user according to the ranking result.

6. The method according to claim 1, characterized in that Before obtaining the visual information and voice information of the target user, the method further includes: Acquire the identity information of the current user; wherein the identity information is used to characterize the biometric characteristics of the current user; Matching the identity information of the current user with the set identity information; When the identity information of the current user successfully matches the set identity information, the current user is determined to be the target user.

7. A multimedia content recommendation device, characterized in that: include: An acquisition module, used to acquire visual information and voice information of a target user; wherein the visual information includes at least one of face, eyes, mouth and posture, and the voice information includes at least one of semantics, tone, intonation and volume; A visual analysis module, used for performing visual analysis on the visual information to obtain a visual emotion score corresponding to each emotion type; A speech analysis module, used for performing speech analysis on the speech information to obtain a speech emotion score corresponding to each emotion type; A determination module, configured to perform weighted calculation on the corresponding visual emotion score and the voice emotion score for each emotion type to determine the emotion of the target user; The control module is used to control the vehicle-mounted device to recommend at least one multimedia content that matches the emotion of the target user to the target user.

8. An electronic device, characterized in that: include: Processor and memory; The processor runs a program corresponding to the executable program code by reading the executable program code stored in the memory, so as to implement the method for recommending multimedia content according to any one of claims 1 to 6.

9. A vehicle, characterized in that: include: The electronic device as claimed in claim 8.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method for recommending multimedia content according to any one of claims 1 to 6 is implemented.

Citation Information

Patent Citations

  • Media file recommending method and system based on vehicle-mounted terminal and vehicle-mounted terminal

    CN106649843A

  • Emotion recognition method and device based on multi-feature fusion and storage medium

    CN110110653A

  • Multimedia recommendation method, device and equipment and storage medium

    CN110175245A

  • Human-vehicle interaction method and system, vehicle and computer readable storage medium

    CN112927721A

  • Facial emotion recognition model training method, emotion recognition method and related equipment

    CN114202791A