Emotion pacifying method for vehicle user, controller, vehicle, medium and product

Abstract

The invention relates to an emotion pacifying method for a vehicle user, a controller, a vehicle, a medium and a product. The method comprises the following steps: displaying a 3D virtual object in the vehicle; identifying emotion information of a vehicle user, wherein the vehicle user comprises a driver of the vehicle; and controlling the 3D virtual object according to the emotion information so as to pacify the emotion of the vehicle user. Since the vehicle user comprises the driver of the vehicle, emotion pacification can be performed on the driver through the displayed 3D virtual object, and the emotion of the driver returns to the emotion interval suitable for driving, so that the driving risk caused by the emotion is reduced, and traffic accidents caused by the emotion problem of the driver are reduced. Besides, the displayed 3D virtual object is controlled to pacify the mood of the vehicle user, so that the man-machine interaction mode is more natural and visual, the mood pacifying effect on the driver is improved, and traffic accidents caused by the mood problem of the driver are further reduced.

Description

Technical Field

[0001] The present disclosure relates to the field of vehicle technology, and in particular, to an emotion soothing method for a vehicle user, a controller, a vehicle, a medium, and a product. Background Art

[0002] With the continuous advancement of vehicle technology, vehicles have become an important tool for people's daily travel. Correspondingly, the incidence of traffic accidents has also increased. Studies have shown that the driver's emotional state has a significant impact on the occurrence of traffic accidents.

[0003] In the related art, in order to reduce traffic accidents caused by driver's emotional problems, when the driver is detected to be in a bad mood, the vehicle-mounted system can output corresponding information to remind the driver to pay attention to driving safety, etc., but there are problems such as poor effect. Summary of the invention

[0004] The purpose of the present disclosure is to provide a method, controller, vehicle, medium and product for calming the emotions of a vehicle user to solve the above-mentioned technical problems.

[0005] In order to achieve the above-mentioned object, the present disclosure provides, in a first aspect, a method for calming the emotions of a vehicle user, comprising: Display 3D virtual objects in the vehicle; identifying emotional information of a vehicle user, wherein the vehicle user includes a driver of the vehicle; The 3D virtual object is controlled according to the emotional information to calm the vehicle user.

[0006] Optionally, controlling the 3D virtual object according to the emotional information to calm the vehicle user emotionally includes: Based on the emotional information, control the display action, display expression and / or conversation content of the 3D virtual object; and / or, based on the emotional information, control the 3D virtual object to trigger the vehicle's onboard device to emotionally soothe the vehicle user through the onboard device.

[0007] Optionally, controlling the display action and / or display expression of the 3D virtual object according to the emotion information includes: Determining a first display action and / or a first display expression of the 3D virtual object according to the emotion information and a first preset correspondence relationship, wherein the first preset correspondence relationship is used to represent a correspondence relationship between the emotion information and the display action of the 3D virtual object and / or a correspondence relationship between the emotion information and the display expression of the 3D virtual object; The 3D virtual object is controlled to perform the first display action and / or display the first display expression.

[0008] Optionally, controlling the conversation content of the 3D virtual object according to the emotion information includes: Determining the conversation content of the 3D virtual object according to the emotion information and the conversation model, wherein the conversation model is used to output corresponding conversation content according to the input emotion information; The 3D virtual object is controlled to output the conversation content.

[0009] Optionally, the 3D virtual object outputs the conversation content in the form of audio, and the method further includes: Determining vehicle user information, and determining audio information of the conversation content according to the vehicle user information; The controlling the 3D virtual object to output the conversation content includes: Based on the audio information, the 3D virtual object is controlled to output the conversation content.

[0010] Optionally, the conversation model is trained in the following manner: Obtaining sample psychological question data and sample psychological answer data; Inputting the sample psychological question data and the sample psychological answer data into a prompt word generation model to obtain a sample prompt word; The conversation model is trained according to the sample prompt words and the sample psychological answer data.

[0011] Optionally, controlling the 3D virtual object to trigger an onboard device of the vehicle according to the emotional information, so as to emotionally soothe the vehicle user through the onboard device, includes: According to the emotional information, controlling the 3D virtual object to trigger a music playing device of the vehicle, so as to play music through the music playing device to soothe the emotions of the vehicle user; and / or, According to the emotional information, controlling the 3D virtual object to trigger the ambient light device of the vehicle, so as to turn on the ambient light through the ambient light device to soothe the vehicle user; and / or, According to the emotional information, the 3D virtual object is controlled to trigger a fragrance device of the vehicle, so as to soothe the emotions of the vehicle user by emitting a scent through the fragrance device.

[0012] Optionally, the emotion information includes an emotion category, and the controlling the 3D virtual object according to the emotion information to soothe the vehicle user emotionally includes: When the emotion category is a first category emotion, the 3D virtual object is controlled according to the emotion information to emotionally soothe the vehicle user, the emotion activation level of the first category emotion is outside a preset emotion activation level range, and / or the emotion pleasure level of the first category emotion is outside a preset emotion pleasure level range.

[0013] Optionally, it also includes: When the emotion category is the second category emotion, the driving state of the vehicle is determined, and the 3D virtual object is controlled according to the driving state, wherein the emotion activation degree of the second category emotion is within the preset emotion activation degree range and the emotion pleasure degree of the second category emotion is within the preset emotion pleasure degree range.

[0014] Optionally, the controlling the 3D virtual object according to the driving state includes: Determining a second display action and / or a second display expression of the 3D virtual object according to the driving state and a second preset correspondence relationship, wherein the second preset correspondence relationship is used to characterize the driving state and the display action of the 3D virtual object and / or is used to characterize the correspondence relationship between the driving state and the display expression of the 3D virtual object; The 3D virtual object is controlled to perform the second display action and / or display the second display expression.

[0015] Optionally, the identifying the emotion information of the vehicle user includes: Acquiring facial image and voice data of the vehicle user; The first emotion information is obtained through a facial information recognition model and the facial image, and the second emotion information is obtained through a speech recognition model and speech data, wherein the facial information recognition model is used to output the corresponding emotion information according to the input facial image, and the speech recognition model is used to output the corresponding emotion information according to the input speech data; Obtaining fused emotion information according to the first emotion information, the second emotion information and a multimodal emotion recognition model, wherein the multimodal emotion recognition model is used to output corresponding fused emotion information according to the input emotion information; The emotion information of the vehicle user is obtained according to the fused emotion information and the emotion classification model, wherein the emotion classification model is used to output corresponding emotion information according to the input fused emotion information.

[0016] A second aspect of the present disclosure provides a controller, the controller comprising: processor; a memory for storing processor-executable instructions; Wherein, the processor is configured to: execute the steps of any method described in the first aspect.

[0017] A third aspect of the present disclosure provides a vehicle, comprising the controller described in the second aspect.

[0018] A fourth aspect of the present disclosure provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of any one of the methods described in the first aspect.

[0019] A fifth aspect of the present disclosure provides a computer program product, including a computer program, which, when executed by a processor, implements the steps of any one of the methods in the first aspect.

[0020] Through the above technical solution, 3D virtual objects can be displayed in the vehicle, and the emotional information of the vehicle user can be identified, and the 3D virtual objects can be controlled according to the emotional information of the vehicle user, so as to calm the emotions of the vehicle user. Since the vehicle user includes the driver of the vehicle, the driver can be calmed by the displayed 3D virtual object, so that the driver's emotions return to an emotional range suitable for driving, thereby reducing the driving risk caused by emotions, and further reducing traffic accidents caused by emotional problems of the driver. In addition, by controlling the displayed 3D virtual object to calm the emotions of the vehicle user, the human-computer interaction method can be made more natural and intuitive, thereby improving the emotional calming effect on the driver and further reducing traffic accidents caused by emotional problems of the driver.

[0021] Other features and advantages of the present disclosure will be described in detail in the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] The accompanying drawings are used to provide a further understanding of the present disclosure and constitute a part of the specification. Together with the following specific embodiments, they are used to explain the present disclosure but do not constitute a limitation of the present disclosure. In the accompanying drawings: Figure 1 What is shown is a schematic diagram of dividing emotions according to two dimensions of emotional pleasure and emotional activation in the related art; Figure 2 What is shown is a schematic diagram of the distribution of emotion intervals suitable for driving; Figure 3 Shown is a flowchart of a related technology for reducing traffic accidents caused by emotional problems of vehicle users; Figure 4 is a flow chart of a method for calming the emotions of a vehicle user according to an exemplary embodiment of the present disclosure; Figure 5is a flow chart showing another method for calming the emotions of a vehicle user according to an exemplary embodiment of the present disclosure; Figure 6 is a block diagram of a vehicle user's emotion soothing method and device according to an exemplary embodiment of the present disclosure; Figure 7 The figure is a block diagram of another method and device for calming emotions of a vehicle user according to an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION

[0023] The specific implementation of the present disclosure is described in detail below in conjunction with the accompanying drawings. It should be understood that the specific implementation described herein is only used to illustrate and explain the present disclosure, and is not used to limit the present disclosure.

[0024] In the present disclosure, unless otherwise stated, directional words such as "upper" and "lower" are defined according to the actual arrangement of the battery assembly, and directional words such as "inside" and "outside" are defined with respect to the outline of the corresponding component. The terms "first, second" and the like are used for the purpose of distinguishing different components and do not have order or importance. In addition, in the following description, when referring to the drawings, unless otherwise explained, the same figure numbers in different drawings represent the same or similar elements.

[0025] According to the cyclical model based on subjective feelings proposed by psychologist James Russell in 1980, emotions can be divided into two dimensions: pleasure and activation, such as Figure 1 At the same time, according to the "Driver Emotion-Driving Risk Mechanism Analysis" of the Journal of Mechanical Engineering, under dimensional emotions, when the driver is in the emotional range of medium pleasure and medium activation, the impact of emotions on driving safety risks is the lowest, so it is better for the driver's emotions to be maintained in the emotional range of medium pleasure and medium activation (i.e., emotions close to the origin); under discrete emotions, the probability ranking of high driving safety risks is: neutral < happy < disgust < sad < fear < surprise, so positive emotions correspond to lower driving risks than negative emotions.

[0026] Therefore, on the whole, the optimal driving emotion range is positive emotions with medium pleasure and medium activation, such as pleasure, neutrality, and Figure 2 As shown in Figure 2, other emotional states of pleasure and / or activation are likely to increase driving safety risks.

[0027] Therefore, in the related art, in order to reduce traffic accidents caused by driver's emotional problems, Figure 3As shown, by collecting user facial features, user actions and user voice information, and transmitting the collected information to the data processing module, the data processing module analyzes and calculates the degree of quarrel, and outputs corresponding control signals to the vehicle system according to the degree of quarrel, and the vehicle system outputs corresponding control. For example, the corresponding control reminder is output through the vehicle central control screen to remind the driver to pay attention to driving safety, and / or the driving state of the vehicle is controlled by the control system, such as controlling the vehicle to turn on the double flash or control the vehicle to park on the side of the road, etc., so as to reduce traffic accidents caused by driver emotional problems.

[0028] However, in this process, since the corresponding control reminder is output by the on-board central control screen, the human-computer interaction is not vivid and natural, which affects the reminder effect of the control reminder. On the other hand, since this method cannot relieve the driver's driving emotions, it cannot effectively avoid traffic accidents caused by the driver's emotional problems.

[0029] In view of this, the embodiments of the present disclosure provide a method for calming the emotions of a vehicle user, a controller, a vehicle, a medium and a product to solve the above-mentioned technical problems.

[0030] The embodiments of the present disclosure are further explained below with reference to the accompanying drawings.

[0031] Figure 4 is a flowchart of a method for calming the emotions of a vehicle user according to an exemplary embodiment of the present disclosure, referring to Figure 4 , the method may include the following steps: S401: Displaying a 3D virtual object in a vehicle.

[0032] In this embodiment, the 3D virtual object can be generated based on the aerial imaging technology. For example, an aerial imaging device can be installed in a vehicle, and after the aerial imaging device is turned on, the divergent light can be re-converged in the air through a negative refraction flat lens and the light field reconstruction principle, thereby forming a real image that does not require a medium to carry, that is, a 3D virtual object.

[0033] In this embodiment, the number of 3D virtual objects displayed in the vehicle may be one or more. When the number of 3D virtual objects displayed in the vehicle is one, in order to prevent the driver from increasing the risk of driving safety when viewing the 3D virtual object, the 3D virtual object may be displayed at a preset position around the central control screen of the vehicle. For example, the 3D virtual object may be displayed above the central control screen. When the number of 3D virtual objects displayed in the vehicle is multiple, a 3D virtual object may be displayed at a preset position around each seat of the vehicle so that vehicle users at different seats can view the 3D virtual object.

[0034] S402: Identify emotional information of a vehicle user, where the vehicle user includes a driver of the vehicle.

[0035] For example, an image acquisition device for acquiring a facial image of a vehicle user may be installed in the vehicle, and then emotion recognition may be performed on the acquired facial image to obtain the emotion information of the vehicle user. Alternatively, a speech recognition device for acquiring voice data of the vehicle user may be installed in the vehicle, and then emotion recognition may be performed on the acquired voice data to obtain the emotion information of the vehicle user.

[0036] In a possible way, in order to improve the recognition accuracy of emotional information, the emotional information obtained based on facial image recognition and the emotional information obtained based on voice data recognition can be input into the multimodal model together, so that the facial image and voice data of the vehicle user can be combined for emotion recognition, thereby obtaining more accurate emotional information. Therefore, when controlling the 3D virtual object according to the emotional information, the vehicle user can be more effectively emotionally appeased, thereby reducing the risk of accidents caused by the vehicle user's emotions, thereby improving driving safety.

[0037] That is to say, among possible ways, identifying the emotional information of the vehicle user may include: Acquire a facial image and voice data of a vehicle user; obtain first emotion information through a facial information recognition model and a facial image, and obtain second emotion information through a voice recognition model and voice data, wherein the facial information recognition model is used to output corresponding emotion information according to the input facial image, and the voice recognition model is used to output corresponding emotion information according to the input voice data; obtain fused emotion information according to the first emotion information, the second emotion information and a multimodal emotion recognition model, wherein the multimodal emotion recognition model is used to output corresponding fused emotion information according to the input emotion information; obtain emotion information of the vehicle user according to the fused emotion information and a sentiment classification model, wherein the sentiment classification model is used to output corresponding emotion information according to the input fused emotion information.

[0038] For example, an information collection device and an emotion processing device may be provided in the vehicle, wherein the information collection device may include a camera and a microphone. The emotion processing device may include a facial information recognition module, a language information recognition module, a multimodal emotion processing module, and an emotion classification processing module. Thus, when the vehicle is powered on, the camera may collect a user image of the vehicle user in real time, and transmit the user image to the facial information recognition module. The microphone may collect voice data of the vehicle user in real time, and transmit the voice data to the voice information recognition module.

[0039] When the facial information recognition module receives the user image captured by the camera, it can capture the user's facial image from the user image, pre-process the user's facial image, extract the user's facial features, and then input the user's facial features into a trained facial expression recognition classifier to obtain the first emotion information, and send the first emotion information to the multimodal emotion processing module.

[0040] When the voice information recognition module receives the voice data collected by the microphone, it can pre-process the voice data, then use a convolutional neural network to analyze the user's voice features, and input the user's voice features into a trained voice recognition emotion classifier to obtain second emotion information, and send the second emotion information to the multimodal emotion processing module.

[0041] The multimodal emotion processing module receives the first emotion information and the second emotion information, performs decision-level fusion by weighted summation to obtain fused emotion information, and sends the fused emotion information to the emotion classification processing module.

[0042] After receiving the fused emotional information, the emotion classification processing module uses multimodal emotion recognition technology to identify the vehicle user's emotional information, such as whether the vehicle user's emotions are excitement, sadness, anger, fatigue or neutral.

[0043] Through the above method, multimodal technology and large model technology can be combined to identify the emotional information of vehicle users, so that 3D virtual objects can be controlled more accurately based on the emotional information, thereby more effectively calming the emotions of vehicle users, thereby reducing the risk of accidents caused by driver emotions and improving driving safety.

[0044] S403: Controlling the 3D virtual object according to the emotional information to calm the vehicle user.

[0045] In this embodiment, the vehicle user may include one or more than one vehicle user. Similarly, the 3D virtual object may include one or more than one 3D virtual object, and this embodiment does not impose any limitation on this.

[0046] For example, when both the vehicle user and the 3D virtual object include one, the vehicle user at this time is the driver, and thus the 3D virtual object can be controlled to calm the driver's emotions.

[0047] For example, when there are multiple vehicle users and one 3D virtual object, the 3D virtual object can be controlled based on the driver's emotional information. When the driver does not need to be emotionally appeased, the remaining vehicle users can be emotionally appeased. Alternatively, an emotional appeasing sequence can be pre-set, whereby the object to be appeased can be determined based on the emotional appeasing sequence and the vehicle user's emotional information, and then the 3D virtual object can be controlled based on the emotional information of the object to be appeased to emotionally appease the object to be appeased.

[0048] For example, when there are multiple vehicle users and 3D virtual objects, a 3D virtual object can be controlled to soothe the emotions according to the emotional information of each vehicle user.

[0049] Through the above technical solution, 3D virtual objects can be displayed in the vehicle, and the emotional information of the vehicle user can be identified, and the 3D virtual objects can be controlled according to the emotional information of the vehicle user, so as to calm the emotions of the vehicle user. Since the vehicle user includes the driver of the vehicle, the driver can be calmed by the displayed 3D virtual object, so that the driver's emotions return to an emotional range suitable for driving, thereby reducing the driving risk caused by emotions, and further reducing traffic accidents caused by emotional problems of the driver. In addition, by controlling the displayed 3D virtual object to calm the emotions of the vehicle user, the human-computer interaction method can be made more natural and intuitive, thereby improving the emotional calming effect on the driver and further reducing traffic accidents caused by emotional problems of the driver.

[0050] To facilitate understanding of the method for calming vehicle users' emotions provided by the present disclosure, possible implementation methods of the present disclosure are described below.

[0051] In a possible manner, controlling the 3D virtual object according to the emotional information to comfort the vehicle user emotionally may include: Based on the emotional information, the display actions, expressions and / or conversation contents of the 3D virtual object are controlled; and / or, based on the emotional information, the 3D virtual object is controlled to trigger the vehicle's onboard equipment to emotionally soothe the vehicle user through the onboard equipment.

[0052] In this embodiment, according to the emotional information, the display action, display expression and / or conversation content of the 3D virtual object can be controlled by: presetting the corresponding relationship between the emotional information, the display action, the display expression and the conversation content. Thus, the display action, display expression and / or conversation content of the 3D virtual object can be controlled according to the emotional information and the corresponding relationship. It can also be that the display action, display expression and / or conversation content of the 3D virtual object is randomly controlled according to the emotional information. Of course, the display action, display expression and / or conversation content of the 3D virtual object can also be controlled according to the emotional information in other ways, and the real-time example disclosed in this disclosure does not impose any restrictions on this.

[0053] For example, the action library, expression library and / or language library of the 3D virtual object can be preset, wherein the action library can include multiple preset actions, such as dancing, walking, running and / or jumping, etc. The expression library can include multiple preset expressions, such as laughing, frowning, blinking and / or showing off, etc. The language library can include multiple chat contents, such as jokes, stories and / or news, etc. Thus, an action, expression and / or chat content can be randomly selected from the action library, expression library and / or language library according to the emotional information to control the 3D virtual object to perform the selected action, display the selected expression and / or input the selected chat content, etc.

[0054] The vehicle-mounted device in this embodiment can be an emotion regulation device, such as a music playing device, an ambient light device or a fragrance device. Thus, the emotion of the vehicle user can be regulated by the emotion regulation device, thereby reducing accidents caused by the driver's emotions and improving driving safety.

[0055] Through the above method, the 3D virtual object can be controlled to use a variety of different methods to comfort the vehicle user's emotions, thereby comforting the vehicle user's emotions from multiple different dimensions, thereby quickly alleviating the vehicle user's emotions and reducing traffic accidents caused by emotional problems of the vehicle user.

[0056] In a possible manner, controlling the display action and / or display expression of the 3D virtual object according to the emotion information may include: According to the emotional information and a first preset correspondence, determine a first display action and / or a first display expression of the 3D virtual object, wherein the first preset correspondence is used to characterize the correspondence between the emotional information and the display action of the 3D virtual object and / or to characterize the correspondence between the emotional information and the display expression of the 3D virtual object; control the 3D virtual object to perform the first display action and / or display the first display expression.

[0057] For example, the emotion information may include first emotion information, second emotion information, and third emotion information, the display action of the 3D virtual object may include a first action, a second action, and a second action, the display expression of the 3D virtual object may include a first expression, a second expression, and a third expression, and the first preset correspondence between the emotion information, the display action of the 3D virtual object, and the display expression of the 3D virtual object may be as shown in Table 1: Table 1 First preset correspondence

[0058] Therefore, when the emotional information is the first emotional information, the 3D virtual object can be controlled to perform the first action and display the second expression. When the emotional information is the second emotional information, the 3D virtual object can be controlled to perform the third action and display the first expression. When the emotional information is the third emotional information, the 3D virtual object can be controlled to perform the second action and display the third expression.

[0059] Possible ways to control the conversation content of a 3D virtual object based on the emotional information include: The conversation content of the 3D virtual object is determined according to the emotion information and the conversation model, wherein the conversation model is used to output the corresponding conversation content according to the input emotion information; and the 3D virtual object is controlled to output the conversation content.

[0060] For example, after obtaining the driver's emotional information, a prompt word text may be generated according to the emotional information, and then the prompt word text may be input into a conversation model to obtain conversation content for the emotional information, and the 3D virtual object may be controlled to output the conversation content.

[0061] In a possible manner, after the 3D virtual object outputs the conversation content, it is also possible to detect in real time whether the vehicle user provides feedback on the conversation content. After detecting that the vehicle user provides feedback on the conversation content, the vehicle user's feedback content can be output to the conversation model to obtain new conversation content for the feedback content, and the 3D virtual object can be controlled to output the new conversation content. The above steps are repeated, so that the vehicle user's emotions can be alleviated by chatting with the vehicle user.

[0062] In this way, the emotional information can be input into the conversation model, thereby obtaining the conversation content for the emotional information. Thus, the conversation can be carried out in a targeted manner according to different emotional information, thereby more effectively soothing the vehicle user, improving the soothing efficiency and user experience.

[0063] In a possible way, in order to make the conversation between the vehicle user and the 3D virtual object more natural, and to enable the 3D virtual object to more effectively soothe the vehicle user's emotions. When training the conversation model, real psychological question and answer data, i.e., real psychological question data and real psychological answer data, can be collected, and then psychological soothing conversation data can be generated through large model prompt word engineering, and finally the conversation model can be trained based on the psychological soothing conversation data. That is to say, in a possible way, the conversation model can be trained in the following way: Obtain sample psychological question data and sample psychological answer data; input the sample psychological question data and the sample psychological answer data into a prompt word generation model to obtain sample prompt words; and train the conversation model based on the sample prompt words and the sample psychological answer data.

[0064] For example, real psychological question and answer data can be crawled through web crawler technology to obtain sample psychological question data and sample psychological answer data, and then the sample psychological question data and sample psychological answer data are input into large language models (LLMs) to obtain sample prompt words. Finally, the sample prompt words and sample psychological answer data are used as psychological comfort dialogue data to train the conversation model until the conversation model converges.

[0065] In a possible manner, the 3D virtual object outputs the conversation content in the form of audio. Accordingly, the method for calming the vehicle user's emotions may further include: Determine vehicle user information, and determine audio information of conversation content according to the vehicle user information; Accordingly, controlling the 3D virtual object to output the conversation content may include: Based on the audio information, the 3D virtual object is controlled to output the conversation content.

[0066] In this embodiment, the vehicle user information may be the gender information, age group information and / or preference information pre-configured by the vehicle user, etc. Thus, determining the vehicle user information may be determining the vehicle user information by using the vehicle user image captured by the image acquisition device, for example, determining the gender information and / or age group information of the vehicle user. Alternatively, the personal information file pre-entered by the vehicle user may be read, and then the gender information, age group information and / or preference information of the vehicle user may be obtained. In this embodiment, the audio information may be timbre information and / or language type information. Thus, determining the audio information of the conversation content according to the vehicle user information may include determining the timbre information, such as a child's voice, a male voice or a female voice, according to the gender information, age group information and / or preference information pre-configured by the vehicle user, etc., and / or determining the language type information, such as English, Chinese or a dialect, according to the gender information, age group information and / or preference information pre-configured by the vehicle user, etc.

[0067] After the audio information is determined, the 3D virtual object can be controlled to output the conversation content based on the audio information.

[0068] For example, if the timbre information is determined to be male and the language type information is Chinese according to the vehicle user information, the 3D virtual object can be controlled to output the conversation content in Chinese with the timbre of a male voice.

[0069] In the above manner, the audio information of the conversation content can be determined according to the user information, so as to control the 3D virtual object to output the conversation content based on the audio information. Thus, the output form of the conversation content can be determined according to the user information, so that the 3D virtual object can provide a personalized interaction method to the vehicle user, further improving the user experience.

[0070] In a possible manner, according to the emotional information, controlling the 3D virtual object to trigger the vehicle's onboard device to soothe the vehicle user's emotions through the onboard device may include: Based on the emotional information, the 3D virtual object is controlled to trigger the vehicle's music playback device to play music through the music playback device to emotionally soothe the vehicle user; and / or, based on the emotional information, the 3D virtual object is controlled to trigger the vehicle's ambient light device to emotionally soothe the vehicle user by turning on the ambient light through the ambient light device; and / or, based on the emotional information, the 3D virtual object is controlled to trigger the vehicle's fragrance device to emotionally soothe the vehicle user by emitting scent through the fragrance device.

[0071] In this embodiment, according to the emotional information, controlling the 3D virtual object to trigger the music playback device of the vehicle can be: according to the emotional information, the music playback device is controlled to play songs that the vehicle user is interested in or songs that meet the user's preferences, or the corresponding relationship between the emotional information and the song category is pre-set, so that according to the emotional information, the music playback device can be controlled to play the corresponding category of music. For example, when the vehicle user is in an excited mood, classical music can be played; when the vehicle user is in a sad, angry, or tired mood, pop music with positive emotions can be played. In a possible way, in order to further relieve the emotions of the vehicle user, when controlling the music playback device to play music, the 3D virtual object can also be controlled to dance to the music.

[0072] Similarly, according to the emotional information, the 3D virtual object can be controlled to trigger the vehicle's fragrance device to emit the fragrance that the vehicle user likes according to the emotional information, or the corresponding relationship between the emotional information and the fragrance can be pre-set, so that the fragrance device can be controlled to emit the corresponding category of fragrance according to the emotional information. For example, when the vehicle user is in an excited mood, some fragrances that can stabilize the mood can be emitted, such as lavender, cedar or sandalwood; when the vehicle user is in a sad or angry mood, some fragrances that can make the mood positive can be emitted, such as rose, jasmine or orange; when the vehicle user is in a tired mood or tired state, some fragrances that can help people sort out their thoughts can be transmitted, such as grass, lemon or green tea.

[0073] Similarly, according to the emotional information, controlling the 3D virtual object to trigger the vehicle's atmosphere light device can be: controlling the atmosphere light device to turn on the atmosphere light that the vehicle user is interested in according to the emotional information, or the correspondence between the emotional information and the atmosphere light can be pre-set, so that the atmosphere light device can be controlled to turn on the corresponding atmosphere light according to the emotional information. For example, when the vehicle user is in an excited mood, the lights with color values ​​of 53F9FF, 00C2C9 and 2F67A9 can be controlled to be constantly on, with an interval interpolation of 100ms; when the user is in an angry mood, the light with a color value of FFEF05 can be controlled to flash three times to indicate recognition, and then the lights with color values ​​of 53F9FF and 00C2C9 can be controlled to be constantly on, with an interval difference of 80ms; when the user is in a sad mood, the lights with color values ​​of FE9406 and FF0000 can be controlled to flash three times alternately, and then the lights with color values ​​of FFEF05 and FFFF00 can be controlled to be constantly on, with an interval interpolation of 100ms; when the user is in a tired mood or tired state, the light with a color value of 7CDB05 can be controlled to light up slowly, and then the lights with color values ​​of FFEF05 and FE9406 can be controlled to be constantly on, with an interval interpolation of 2000ms.

[0074] Among them, 53F9FF represents light blue, 00C2C9 represents cyan, 2F67A9 represents dark blue, FFEF05 represents bright yellow, FE9406 represents orange, FF0000 represents red, FFFF00 represents yellow, and 7CDB05 represents bright green.

[0075] In a possible manner, the emotion information may include an emotion category. Accordingly, controlling the 3D virtual object according to the emotion information to comfort the vehicle user may include: When the emotion category is the first category emotion, the 3D virtual object is controlled according to the emotion information to emotionally comfort the vehicle user, the emotion activation level of the first category emotion is outside the preset emotion activation level range, and / or the emotion pleasure level of the first category emotion is outside the preset emotion pleasure level range.

[0076] In this embodiment, the preset emotional activation interval and the preset emotional pleasure interval can be determined according to actual conditions, and the present disclosure embodiment does not impose any restrictions on this. For example, the preset emotional activation interval can be an emotional activation interval that has a smaller impact on driving safety, for example, Figure 2 The preset emotional pleasure interval may be an emotional pleasure interval that has a small impact on driving safety, for example, Figure 2 The medium pleasure range shown.

[0077] Through the above method, when the emotion category is the first category of emotion, such as sadness, anger, excitement or fatigue, the vehicle user can be emotionally soothed by controlling the 3D virtual object, thereby reducing traffic accidents caused by emotional problems of the vehicle user.

[0078] Possible approaches may also include: When the emotion category is the second category emotion, the driving state of the vehicle is determined, and the 3D virtual object is controlled according to the driving state, wherein the emotion activation degree of the second category emotion is within a preset emotion activation degree range and the emotion pleasure degree of the second category emotion is within a preset emotion pleasure degree range.

[0079] Through the above method, when the emotion category is the second category emotion, such as neutral emotion, the 3D virtual object can be controlled according to the driving status, thereby increasing the interaction between the 3D virtual object and the vehicle user, thereby increasing driving pleasure and enhancing the entertainment experience of the vehicle user.

[0080] Among possible ways, controlling the 3D virtual object according to the driving state may include: According to the driving state and the second preset correspondence, determine the second display action and / or the second display expression of the 3D virtual object, wherein the second preset correspondence is used to characterize the driving state and the display action of the 3D virtual object and / or to characterize the correspondence between the driving state and the display expression of the 3D virtual object; control the 3D virtual object to perform the second display action and / or display the second display expression.

[0081] In this embodiment, the driving state and the second preset relationship can be determined according to actual conditions, and the embodiment of the present disclosure does not impose any restrictions on this. For example, the driving state may include driving at a constant speed, accelerating, driving and playing music, etc. Displaying actions may include dancing, walking, and running, etc. Displaying expressions may include smiling, blinking, and laughing, etc. Therefore, the second preset corresponding relationship may be as shown in Table 2: Table 2 Second preset correspondence

[0082] Therefore, when the driving state is constant speed driving, the 3D virtual object can be controlled to perform walking movements and show a laughing expression; when the driving state is accelerating, the 3D virtual object can be controlled to perform running movements and show a blinking expression; when the driving state is driving and playing music, the 3D virtual object can be controlled to perform dancing movements and show a smiling expression.

[0083] Through the above method, the 3D virtual object can be controlled to perform different actions and / or display different expressions according to different driving conditions, thereby increasing driving pleasure and enhancing the entertainment experience of vehicle users.

[0084] To further understand the method for calming the emotions of vehicle users provided by the present disclosure, a possible implementation of the present disclosure is described below in combination with various steps: For example, Figure 5As shown, a camera for capturing the driver's image can be installed on the A-pillar of the vehicle, and the driver's image is transmitted to the facial information recognition module. The facial information recognition module captures the driver's facial image from the driver's image, extracts the feature points of the driver's eyebrows, mouth, and chin using a convolutional neural network, and inputs them into a facial expression recognition classifier that has been trained using the dataset ImageNet. The facial expression recognition classifier sends the recognition result to the multimodal emotion processing module. A microphone for collecting the driver's voice data can also be installed on the A-pillar of the vehicle, and the voice data is transmitted to the voice recognition module. The voice recognition module extracts voice features using a convolutional neural network, and inputs the voice features into a voice recognition model that has been trained using the dataset RAVDESS. The voice recognition model sends the result to the multimodal emotion processing module. The multimodal emotion processing module receives the recognition results of the facial expression recognition classifier and the voice recognition model, performs decision-layer fusion using a weighted summation method, and sends the result to the emotion classification model. If the emotion classification model recognizes that the driver is in the first category of emotions, such as sadness, on the one hand, the 3D virtual object can be controlled to tell jokes to the driver to alleviate the driver's sadness; on the other hand, the 3D virtual object can be controlled to trigger the ambient light device, the music player device and the fragrance device, so that the ambient light device controls the lights with color values ​​of FE9406 and FF0000 to flash alternately 3 times, and then controls the lights with color values ​​of FFEF05 and FFFF00 to be on for a long time, with an interval interpolation of 100ms; so that the music player device can play music with positive emotions, and the fragrance device can emit fragrance with jasmine scent, so as to regulate the driver's sad emotions, thereby reducing accidents caused by the driver's emotions and improving driving safety.

[0085] Based on the same concept, the embodiment of the present disclosure also provides an emotional soothing device for a vehicle user, such as Figure 6 As shown, the emotional soothing device may include: The information collection device 601 includes a camera and a microphone installed on the vehicle, and is used to collect facial images and voice information of the vehicle user; The emotion processing device 602 is used to receive the collected facial image and voice information, process the facial image and voice information, obtain the emotion information, and send the emotion information to the aerial imaging device; The aerial imaging device 603 includes a voice module and a 3D virtual object. The voice module is used to generate conversation content according to the emotional information and output the conversation content through the 3D virtual object. The 3D virtual object is also used to perform corresponding actions and / or display expressions according to the emotional information.

[0086] The emotion regulation device 604 includes a music module, an atmosphere light module and a fragrance module, and is used to perform different emotion regulation according to different emotions.

[0087] Based on the same concept, the embodiment of the present disclosure also provides an emotional soothing device for a vehicle user, such as Figure 7 As shown, the vehicle user's emotion soothing device 700 may include: Display module 701, used to display 3D virtual objects in the vehicle; The recognition module 702 is used to recognize the emotion information of the vehicle user, and the vehicle user includes the driver of the vehicle; The control module 703 is used to control the 3D virtual object according to the emotional information to calm the vehicle user.

[0088] Through the above-mentioned vehicle user emotion soothing device 700, 3D virtual objects can be displayed in the vehicle, and the vehicle user's emotional information can be identified, and the 3D virtual objects can be controlled according to the vehicle user's emotional information, so as to soothe the vehicle user's emotions. Since the vehicle user includes the driver of the vehicle, the driver's emotions can be soothed by the displayed 3D virtual objects, so that the driver's emotions return to an emotional range suitable for driving, thereby reducing the driving risk caused by emotions, and further reducing traffic accidents caused by the driver's emotional problems. In addition, by controlling the displayed 3D virtual objects to soothe the vehicle user's emotions, the human-computer interaction method can be made more natural and intuitive, thereby enhancing the emotional soothing effect on the driver and further reducing traffic accidents caused by the driver's emotional problems.

[0089] In a possible manner, the control module 703 can be used to control the display actions, display expressions and / or conversation content of the 3D virtual object according to the emotional information; and / or, according to the emotional information, control the 3D virtual object to trigger the vehicle's on-board device to emotionally soothe the vehicle user through the on-board device.

[0090] In a possible manner, the control module 703 may include: A first determining unit, configured to determine a first display action and / or a first display expression of the 3D virtual object according to the emotion information and a first preset correspondence relationship, wherein the first preset correspondence relationship is used to characterize a correspondence relationship between the emotion information and the display action of the 3D virtual object and / or a correspondence relationship between the emotion information and the display expression of the 3D virtual object; The first control unit is used to control the 3D virtual object to perform a first display action and / or display a first display expression.

[0091] In a possible manner, the control module 703 may include: A second determining unit is used to determine the conversation content of the 3D virtual object according to the emotion information and the conversation model, wherein the conversation model is used to output the corresponding conversation content according to the input emotion information; The second control unit is used to control the 3D virtual object to output the conversation content.

[0092] In a possible manner, the 3D virtual object outputs the conversation content in the form of audio. Accordingly, the control module 703 may further include: a third determining unit, configured to determine vehicle user information, and determine audio information of the conversation content according to the vehicle user information; Correspondingly, the second control unit is used to control the 3D virtual object to output the conversation content based on the audio information.

[0093] In a possible way, the conversation model is trained as follows: Obtaining sample psychological question data and sample psychological answer data; Inputting sample psychological question data and sample psychological answer data into a prompt word generation model to obtain sample prompt words; The conversation model is trained based on sample prompt words and sample psychological response data.

[0094] In a possible manner, the control module 703 may include: A third control unit is used to control the 3D virtual object to trigger the music playing device of the vehicle according to the emotional information, so as to play music through the music playing device to soothe the emotions of the vehicle user; and / or, a fourth control unit, configured to control the 3D virtual object to trigger the ambient light device of the vehicle according to the emotional information, so as to turn on the ambient light of the vehicle user through the ambient light device to soothe the emotions of the vehicle user; and / or, The fifth control unit is used to control the 3D virtual object to trigger the vehicle's fragrance device according to the emotional information, so as to soothe the vehicle user's emotions by emitting odor through the fragrance device.

[0095] In a possible manner, the emotion information includes an emotion category, and accordingly, the control module 703 can be used to control the 3D virtual object according to the emotion information to emotionally comfort the vehicle user when the emotion category is a first category emotion, the emotion activation level of the first category emotion is outside a preset emotion activation level range, and / or the emotion pleasure level of the first category emotion is outside a preset emotion pleasure level range.

[0096] In a possible manner, the control module 703 can also be used to determine the driving status of the vehicle and control the 3D virtual object according to the driving status when the emotion category is a second category emotion, wherein the emotion activation level of the second category emotion is within a preset emotion activation level range and the emotion pleasure level of the second category emotion is within a preset emotion pleasure level range.

[0097] In a possible manner, the control module 703 may include: a fourth determining unit, configured to determine a second display action and / or a second display expression of the 3D virtual object according to the driving state and a second preset corresponding relationship, wherein the second preset corresponding relationship is used to characterize the driving state and the display action of the 3D virtual object and / or the corresponding relationship between the driving state and the display expression of the 3D virtual object; A sixth control unit is used to control the 3D virtual object to perform a second display action and / or display a second display expression.

[0098] In a possible manner, the identification module 702 may include: An acquisition unit, used to acquire a facial image and voice data of a vehicle user; a fifth determining unit, configured to obtain first emotion information through a facial information recognition model and a facial image, and to obtain second emotion information through a speech recognition model and speech data, wherein the facial information recognition model is configured to output corresponding emotion information according to an input facial image, and the speech recognition model is configured to output corresponding emotion information according to an input speech data; a sixth determining unit, configured to obtain fused emotion information according to the first emotion information, the second emotion information and the multimodal emotion recognition model, wherein the multimodal emotion recognition model is configured to output corresponding fused emotion information according to the input emotion information; The seventh determination unit is used to obtain the emotion information of the vehicle user according to the fused emotion information and the emotion classification model, wherein the emotion classification model is used to output corresponding emotion information according to the input fused emotion information.

[0099] Regarding the vehicle user's emotion soothing device 700 in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0100] Based on the same concept, the embodiment of the present disclosure further provides a controller, the controller comprising: processor; a memory for storing processor-executable instructions; Wherein, the processor is configured to: execute the steps of the above-mentioned method for calming the emotions of the vehicle user.

[0101] Based on the same concept, an embodiment of the present disclosure also provides a vehicle, comprising the above-mentioned controller.

[0102] Based on the same concept, an embodiment of the present disclosure also provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of the above-mentioned method for calming the emotions of a vehicle user are implemented.

[0103] Based on the same concept, an embodiment of the present disclosure also provides a computer program product, including a computer program, which, when executed by a processor, implements the steps of the above-mentioned method for calming the emotions of a vehicle user.

[0104] The preferred embodiments of the present disclosure are described in detail above in conjunction with the accompanying drawings; however, the present disclosure is not limited to the specific details in the above embodiments. Within the technical concept of the present disclosure, a variety of simple modifications can be made to the technical solution of the present disclosure, and these simple modifications all fall within the protection scope of the present disclosure.

[0105] It should also be noted that the various specific technical features described in the above specific embodiments can be combined in any suitable manner without contradiction. In order to avoid unnecessary repetition, the present disclosure will not further describe various possible combinations.

[0106] In addition, various embodiments of the present disclosure may be arbitrarily combined, and as long as they do not violate the concept of the present disclosure, they should also be regarded as the contents disclosed by the present disclosure.

Claims

1. A method for calming the emotions of a vehicle user, characterized in that: include: Display 3D virtual objects in the vehicle; identifying emotional information of a vehicle user, wherein the vehicle user includes a driver of the vehicle; The 3D virtual object is controlled according to the emotional information to calm the vehicle user.

2. The method for calming the emotions of a vehicle user according to claim 1, characterized in that: The controlling the 3D virtual object according to the emotion information to comfort the vehicle user includes: Based on the emotional information, control the display action, display expression and / or conversation content of the 3D virtual object; and / or, based on the emotional information, control the 3D virtual object to trigger the vehicle's onboard device to emotionally soothe the vehicle user through the onboard device.

3. The method for calming the emotions of a vehicle user according to claim 2, characterized in that: The controlling the display action and / or display expression of the 3D virtual object according to the emotion information includes: Determining a first display action and / or a first display expression of the 3D virtual object according to the emotion information and a first preset correspondence relationship, wherein the first preset correspondence relationship is used to represent a correspondence relationship between the emotion information and the display action of the 3D virtual object and / or a correspondence relationship between the emotion information and the display expression of the 3D virtual object; The 3D virtual object is controlled to perform the first display action and / or display the first display expression.

4. The method for calming the emotions of a vehicle user according to claim 2, characterized in that: The controlling the conversation content of the 3D virtual object according to the emotion information includes: Determining the conversation content of the 3D virtual object according to the emotion information and the conversation model, wherein the conversation model is used to output corresponding conversation content according to the input emotion information; The 3D virtual object is controlled to output the conversation content.

5. The method for calming the emotions of a vehicle user according to claim 4, characterized in that: The 3D virtual object outputs the conversation content in the form of audio, and the method further includes: Determining vehicle user information, and determining audio information of the conversation content according to the vehicle user information; The controlling the 3D virtual object to output the conversation content includes: Based on the audio information, the 3D virtual object is controlled to output the conversation content.

6. The method for calming the emotions of a vehicle user according to claim 4, characterized in that: The conversation model is trained in the following way: Obtaining sample psychological question data and sample psychological answer data; Inputting the sample psychological question data and the sample psychological answer data into a prompt word generation model to obtain a sample prompt word; The conversation model is trained according to the sample prompt words and the sample psychological answer data.

7. The method for calming the emotions of a vehicle user according to claim 2, characterized in that: The controlling the 3D virtual object to trigger an onboard device of the vehicle according to the emotion information, so as to comfort the vehicle user through the onboard device, comprises: According to the emotional information, controlling the 3D virtual object to trigger a music playing device of the vehicle, so as to play music through the music playing device to soothe the emotions of the vehicle user; and / or, According to the emotional information, controlling the 3D virtual object to trigger the ambient light device of the vehicle, so as to turn on the ambient light through the ambient light device to soothe the vehicle user; and / or, According to the emotional information, the 3D virtual object is controlled to trigger a fragrance device of the vehicle, so as to soothe the emotions of the vehicle user by emitting a scent through the fragrance device.

8. The method for calming the emotions of a vehicle user according to any one of claims 1 to 7, characterized in that: The emotional information includes an emotional category, and the controlling the 3D virtual object according to the emotional information to comfort the vehicle user includes: When the emotion category is a first category emotion, the 3D virtual object is controlled according to the emotion information to emotionally soothe the vehicle user, the emotion activation level of the first category emotion is outside a preset emotion activation level range, and / or the emotion pleasure level of the first category emotion is outside a preset emotion pleasure level range.

9. The method for calming the emotions of a vehicle user according to claim 8, characterized in that: Also includes: When the emotion category is the second category emotion, the driving state of the vehicle is determined, and the 3D virtual object is controlled according to the driving state, wherein the emotion activation degree of the second category emotion is within the preset emotion activation degree range and the emotion pleasure degree of the second category emotion is within the preset emotion pleasure degree range.

10. The method for calming the emotions of a vehicle user according to claim 9, characterized in that: The controlling the 3D virtual object according to the driving state includes: Determining a second display action and / or a second display expression of the 3D virtual object according to the driving state and a second preset correspondence relationship, wherein the second preset correspondence relationship is used to characterize the driving state and the display action of the 3D virtual object and / or is used to characterize the correspondence relationship between the driving state and the display expression of the 3D virtual object; The 3D virtual object is controlled to perform the second display action and / or display the second display expression.

11. The method for calming the emotions of a vehicle user according to any one of claims 1 to 7, characterized in that: The identifying the emotion information of the vehicle user includes: Acquiring facial image and voice data of the vehicle user; The first emotion information is obtained through a facial information recognition model and the facial image, and the second emotion information is obtained through a speech recognition model and speech data, wherein the facial information recognition model is used to output the corresponding emotion information according to the input facial image, and the speech recognition model is used to output the corresponding emotion information according to the input speech data; Obtaining fused emotion information according to the first emotion information, the second emotion information and a multimodal emotion recognition model, wherein the multimodal emotion recognition model is used to output corresponding fused emotion information according to the input emotion information; The emotion information of the vehicle user is obtained according to the fused emotion information and the emotion classification model, wherein the emotion classification model is used to output corresponding emotion information according to the input fused emotion information.

12. A controller, characterized in that: The controller comprises: processor; a memory for storing processor-executable instructions; The processor is configured to execute the steps of the method according to any one of claims 1 to 11.

13. A vehicle, characterized in that: Comprising the controller as claimed in claim 12.

14. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 11 are implemented.

15. A computer program product, characterized in that The invention comprises a computer program, which, when executed by a processor, implements the steps of the method according to any one of claims 1 to 11.

Images