Interaction method and device for emotional companion, vehicle and storage medium

Abstract

The application relates to the technical field of intelligent automobiles, and discloses an interactive method and device for emotional accompaniment, a vehicle and a storage medium, which are applied to a vehicle terminal and comprise the following steps: acquiring facial data and / or voice data of a user; inputting the facial data and / or voice data into a language interaction model that has been trained to obtain emotional accompaniment text data; and interacting with the user by taking the emotional accompaniment text data as voice information. In this way, the user can better understand and complete artificial language or instructions, imitate an artificial person, provide coherent and logical voice information, and provide high-quality emotional accompaniment for the driver.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of language interaction, for example, to a training method and device for emotional interaction, a vehicle and a storage medium. BACKGROUND

[0002] With the rapid development of China's economy, people's material needs have been greatly met. However, according to research, some people have varying degrees of psychological problems, such as feeling lonely, not knowing where to pour out their inner distress, and being unable to get the recognition and understanding of others. Therefore, spiritual and emotional needs cannot be met.

[0003] Among them, in order to meet the spiritual and emotional needs of people, sound boxes and car machines with interactive functions have been developed at home and in cars. However, in the car machine in the car, the interactive function of the car machine is mostly a tool attribute, mainly to solve user requests. For example, data analysis of various electronic control units and sensors in the car; limited by external data collection and sharing restrictions, the service response provided for users is mechanical and process-oriented, and lacks humanization and individualization. SUMMARY

[0004] The present application provides an interactive method, device, vehicle and storage medium for emotional accompaniment, to solve the problem that the vehicle cannot provide users with humanized and individualized spiritual and emotional needs in related technologies.

[0005] The first aspect of the present application provides an interactive method for emotional accompaniment, applied to a car machine end, comprising:

[0006] Obtaining facial data and / or voice data of a user;

[0007] Inputting the facial data and / or voice data into a language interaction model that has been trained to obtain emotional accompaniment text data;

[0008] Interacting with the user by taking the emotional accompaniment text data as voice information.

[0009] Optionally, in an embodiment of the present application, obtaining facial data and / or voice data of a user comprises:

[0010] In the case of obtaining a facial image of a user;

[0011] Performing image enhancement feature processing on the facial image to obtain facial data of the user;

[0012] In the case of obtaining voice information of a user;

[0013] Performing noise reduction and feature extraction processing on the voice information to obtain voice data of the user;

[0014] Optionally, in an embodiment of the present application, the language interaction model is trained in the following manner:

[0015] A language data set is collected;

[0016] The dialogue training samples in the language data set are input into the training supervised strategy model to obtain a corresponding language reply training data set;

[0017] The dialogue training samples and the language reply training data set are input into the training reward model to obtain a score of the dialogue training samples and the language reply training data;

[0018] The dialogue samples are randomly extracted and input into the reinforcement learning model to obtain corresponding language reply data, and a score of the language reply data is obtained according to the training reward model to determine the emotional companion text data.

[0019] Optionally, in an embodiment of the present application, the language data set is collected;

[0020] The dialogue training samples in the language data set are randomly extracted;

[0021] The dialogue training samples are input into the training supervised strategy model to obtain a corresponding language reply training data set, including:

[0022] The multiple language reply training data are classified and sorted in a marked manner;

[0023] The classified and sorted language reply training data are integrated into the language reply training data set.

[0024] Optionally, in an embodiment of the present application, the dialogue training samples and the language reply training data set are input into the training reward model to obtain a score of the dialogue training samples and the language reply training data, including:

[0025] The dialogue training samples and the language reply training data in the language reply training data set are correspondingly matched;

[0026] The language reply training data and the dialogue training samples are correspondingly matched according to the sorting result;

[0027] The dialogue training samples and the language reply training data are matched;

[0028] A score of the training data pair is determined according to the evaluation index.

[0029] Optionally, in an embodiment of the present application, the language reply data is obtained according to the training reward model to determine the emotional companion text data, including:

[0030] inputting the dialogue sample and the language reply data into the training reward model to obtain a score of the training data pair;

[0031] taking the language reply data in the training data pair with the largest score as the emotional companion text data.

[0032] The second aspect embodiment of the present application provides an interactive device for emotional companionship, comprising:

[0033] The acquisition module is configured to acquire facial data and / or voice data of the user;

[0034] The determination module is configured to input the facial data and / or the voice data into the language interaction model that has been trained to obtain the emotional companion text data;

[0035] The output module is configured to interact with the user by taking the emotional companion text data as voice information.

[0036] The third aspect embodiment of the present application provides an interactive device for emotional companionship, comprising a processor and a memory storing program instructions, wherein the processor is configured to execute the interactive method for emotional companionship as described in the foregoing embodiments when running the program instructions.

[0037] The fourth aspect embodiment of the present application provides a vehicle comprising the interactive device for emotional companionship as described in the foregoing embodiments.

[0038] The fourth aspect embodiment of the present application provides a storage medium storing program instructions, wherein the program instructions execute the interactive method for emotional companionship as described in the foregoing embodiments when running.

[0039] The interactive method, device, vehicle and storage medium for emotional companionship provided by the embodiments of the present disclosure can achieve the following technical effects:

[0040] The facial data and / or the voice data of the user are acquired, and the facial data and / or the voice data are input into the language interaction model that has been trained to obtain the emotional companion text data. The emotional companion text data is taken as voice information to interact with the user. In this way, the artificial language or instruction can be better understood and completed, the artificial is imitated, the coherent and logical voice information is provided, and the high-quality emotional companionship is provided for the driver.

[0041] Additional aspects and advantages of the present application will be in part apparent and in part pointed out hereinafter. BRIEF DESCRIPTION OF DRAWINGS

[0042] The above and / or additional aspects and advantages of the present application will become apparent and more readily appreciated from the following description, taken in conjunction with the following drawings of exemplary embodiments of the present application, wherein:

[0043] Figure 1 is a flowchart of an interaction method for emotional accompaniment provided by an embodiment of the present disclosure;

[0044] Figure 2 is a schematic diagram of an interaction device for emotional accompaniment provided by an embodiment of the present disclosure;

[0045] Figure 3 is a schematic diagram of an interaction device for emotional accompaniment provided by an embodiment of the present disclosure. DETAILED DESCRIPTION

[0046] The embodiments of the present application are described in detail below with reference to the accompanying drawings, wherein the same or similar components are denoted by the same or similar reference numerals throughout the drawings. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to explain the present application, and cannot be understood as a limitation of the present application.

[0047] In conjunction with Figure 1 The first aspect of the present application provides an interaction method for emotional accompaniment, applied to a car machine end, comprising:

[0048] S101, acquiring facial data and / or voice data of a user;

[0049] S102, inputting the facial data and / or voice data into a language interaction model that has been trained to obtain emotional accompaniment text data;

[0050] S103, interacting with the user by taking the emotional accompaniment text data as voice information.

[0051] The interaction method for emotional accompaniment provided by the embodiments of the present disclosure acquires facial data and / or voice data of a user, inputs the facial data and / or voice data into a language interaction model that has been trained to obtain emotional accompaniment text data, and interacts with the user by taking the emotional accompaniment text data as voice information. In this way, it can better understand and complete artificial language or instructions, imitate artificial, provide coherent and logical voice information, and provide high-quality emotional accompaniment for the driver.

[0052] Optionally, in an embodiment of the present application, acquiring facial data and / or voice data of a user comprises:

[0053] In the case of acquiring facial images of a user;

[0054] Performing image enhancement feature processing on the facial images to obtain facial data of the user;

[0055] In the case of obtaining the voice information of the user;

[0056] The voice information is denoised and extracted for feature processing to obtain the voice data of the user.

[0057] In the embodiment, the driver facial expression features are collected by a driver monitoring system (DMS) camera, the facial image of the driver is obtained, and the collected facial image is processed by data processing. The facial image is image enhanced, including gray scale transformation, image sharpening, smoothing, etc. The image is transmitted to a language interaction model that has been trained, so that the data can be better recognized by the model, and the voice data of the driver is replied with more effective voice information.

[0058] In addition, the facial image features can also determine the mood of the driver at this time. Specifically, the received facial image features are image enhanced by gray scale transformation, image preprocessed, and then two-dimensional Gabor wavelet transform is performed on the image region. The feature vector of the expression is extracted according to the shape and position of each part of the human face, wherein the feature vector represents the geometric features of the face, and different expressions are recognized according to different feature vectors. Here, the facial expression recognition is defined as a plurality of expressions, including some or all of happy, surprised, sad, angry, disgusted, and frightened.

[0059] In the embodiment, the voice data of the driver is obtained by the vehicle-mounted intelligent voice assistant. The noise is reduced by using the spectral subtraction method, and then the active voice data recognition is performed. The voice data is preprocessed, feature extracted, and model recognized and matched. The voice data of the user is converted into understood text and transmitted to a language interaction model that has been trained. So that the data can be better recognized by the model, and the voice data of the driver is replied with more effective voice information.

[0060] Optionally, in an embodiment of the present application, the language interaction model is trained in the following manner:

[0061] Collecting a language data set;

[0062] Inputting the dialogue training samples in the language data set into a training supervised strategy model to obtain a corresponding language reply training data set;

[0063] Inputting the dialogue training samples and the language reply training data set into a training reward model to obtain the score of the dialogue training sample and the language reply training data;

[0064] The dialogue sample is randomly extracted and input into the reinforcement learning model to obtain corresponding language reply data, and the score of the language reply data is obtained according to the training reward model to determine the emotional accompanying text data. Optionally, in an embodiment of the present application, the dialogue training sample in the language data set is input into the training supervised strategy model to obtain corresponding language reply training data set, including:

[0065] The dialogue training sample is randomly extracted in the language data set;

[0066] The dialogue training sample is input into the training supervised strategy model respectively to obtain a plurality of language reply training data corresponding to the dialogue training sample;

[0067] The plurality of language reply training data are classified and sorted by marking;

[0068] The classified and sorted language reply training data are integrated into the language reply training data set.

[0069] In the embodiment, in order to enable the training supervised strategy model to initially understand the instructions, firstly, the dialogue training sample is randomly extracted in the language data set, the user and the artificial intelligence assistant of the dialogue are played by the artificial marker to provide the dialogue sample, the training supervised strategy model generates some replies, that is, the language reply training data; secondly, the marker scores and ranks the reply options, and feeds back the better results to the training supervised strategy model; finally, the marker gives a high-quality answer, and the high-quality answer can be selected according to the requirements of the task to measure the quality of the answer.

[0070] Optionally, in an embodiment of the present application, the dialogue training sample and the language reply training data set are input into the training reward model to obtain the score of the dialogue training sample and the language reply training data, including:

[0071] The dialogue training sample and the language reply training data in the language reply training data set are correspondingly matched;

[0072] The language reply training data and the dialogue training sample are correspondingly matched according to the sorting result;

[0073] The dialogue training sample and the language reply training data are composed;

[0074] The score of the training data pair is determined according to the evaluation index.

[0075] In this embodiment, the training reward model is a TAMER architecture, and a human annotator gives a ranking order by comprehensively considering these results, selects the optimal answer for labeling, and uses this ranking result data to train the reward model (RM). Here, the dialogue training samples and language reply training data are combined two by two to form multiple training data pairs. Each training data pair is scored according to the evaluation indicators, thereby obtaining the score of each training data pair. Here, if multiple evaluation indicators are used, the scores can be weighted or combined.

[0076] Optionally, in an embodiment of the present application, the score of the language reply data obtained according to the training reward model is used to determine the emotional companion text data, including:

[0077] The dialogue sample and the language reply data are input into the training reward model to obtain the score of the training data pair;

[0078] The language reply data in the training data pair with a larger score is used as the emotional companion text data.

[0079] In this embodiment, dialogue text is randomly extracted from the language data set, language reply data is generated using a reinforcement learning model (Proximal Policy Optimization, PPO), and the corresponding score is given by the training reward model, so that the language reply data in the training data pair with a larger score is used as the emotional companion text data.

[0080] In this embodiment, the dialogue text is input into the training reward model and the current reinforcement learning model to obtain output text y1 and output text y2, respectively. The two output texts y1 and y2 are compared, and the language reply data in the training data pair with a larger score is used as the emotional companion text data, thereby ensuring that the output is a reasonable and coherent emotional companion text data.

[0081] In this embodiment, the language interaction model described above will not only give feedback on the driver's voice information, but also propose an opening line that is most suitable for the current situation according to the driver's facial expression.

[0082] Specifically, the driver's mood is determined according to the recognized facial image features of the driver, and an opening line suitable for the driver's current mood is selected according to the driver's mood; including recognizing that the driver's expression is happy, and the opening line is "I feel incredibly happy to see you so happy"; or "Today is a meeting full of joy, let's share this joy together."

[0083] In the embodiment, the voice data of the driver is acquired by the in-vehicle intelligent voice assistant in the case that the driver communicates with the in-vehicle intelligent voice assistant again, the voice data is converted into text information by the voice data processing module and then transmitted to the corresponding model again, the model processes the received text information through data processing, learning, reasoning and decision-making to reply, and then the text information is transmitted to the voice output module.

[0084] In combination with Figure 2 As shown in the second aspect of the application, an interactive device for emotional accompaniment is provided, which includes: an acquisition module 10 configured to acquire facial data and / or voice data of a user; a determination module 20 configured to input the facial data and / or voice data into a trained language interaction model to obtain emotional accompaniment text data; and an output module 30 configured to interact with the user by taking the emotional accompaniment text data as voice information.

[0085] Figure 3 A structural schematic diagram of a vehicle is provided for the embodiments of the application. The vehicle can include:

[0086] The memory 701, the processor 702, and the computer program stored in the memory 701 and executable on the processor 702. When the processor 702 executes the program, the interactive device for emotional accompaniment provided in the above embodiments is implemented.

[0087] Further, the vehicle further includes a communication interface 703 for communication between the memory 701 and the processor 702.

[0088] The memory 701 is used to store the computer program executable on the processor 702.

[0089] The memory 701 can include a high-speed RAM memory, and can also include a non-volatile memory, such as at least one disk memory.

[0090] If the memory 701, the processor 702 and the communication interface 703 are independently implemented, the communication interface 703, the memory 701 and the processor 702 can be connected to each other through a bus and complete the communication between each other. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For the convenience of representation, Figure 3Only one bus or only one type of bus can exist, however.

[0091] Optionally, in a specific implementation, if the memory 701, the processor 702 and the communication interface 703 are integrated on a chip, the memory 701, the processor 702 and the communication interface 703 can complete the communication among each other through an internal interface.

[0092] The processor 702 can be a central processing unit (CPU) or an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement one or more embodiments of the present application.

[0093] The embodiment of the present disclosure provides a vehicle, comprising: a vehicle body, and the interactive device for emotional accompaniment described above. The interactive device for emotional accompaniment is installed on the vehicle body. The installation relationship described herein is not limited to being placed in the interior of the product body, but also includes installation connection with other components of the vehicle body, including but not limited to physical connection, electrical connection or signal transmission connection, etc. Those skilled in the art can understand that the interactive device for emotional accompaniment can be adapted to a feasible vehicle body, and thus realize other feasible embodiments.

[0094] The embodiment also provides a storage medium, which stores program instructions. When the program instructions are executed, the interactive method for emotional accompaniment described in the foregoing embodiments is executed.

[0095] In the description of the present specification, the description of the terms "one embodiment", "some embodiments", "an example", "a specific example", or "some examples" means that the specific features, structures, materials or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present application. In the present specification, the illustrative description of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or N embodiments or examples in a suitable manner. In addition, the skilled in the art can combine and combine the different embodiments or examples described in the present specification and the features of the different embodiments or examples without contradiction.

[0096] Moreover, the terms "first", "second", "third", etc. are used herein only to describe different steps or claims and do not imply or imply a relative importance or an order of magnitude. Thus, features defined with "first", "second" can explicitly or implicitly include at least one of the features. In the description of the present application, the meaning of "N" is at least two, for example, two, three, etc., unless otherwise explicitly specified.

[0097] Any process or method descriptions or descriptions of the flow diagrams described herein or otherwise described herein can be understood as representing the modules, segments, or portions of code that include executable instructions for implementing custom logic functions or processes, and the scope of preferred embodiments of the present application includes additional implementations in which the order of execution is different from that which is shown or discussed, including that the functions can be executed in substantially simultaneous or in reverse order, as will be understood by those skilled in the art of the embodiments of the present application.

[0098] The logic and / or steps represented in the flow diagrams or otherwise described herein, for example, can be considered as a list of executable instructions for implementing the logic function, which can be specifically embodied in any computer-readable medium for use by or in connection with an instruction execution system, apparatus, or device, such as a computer-based system, processor- based system, or other system that can fetch the instructions from the instruction execution system, apparatus, or device and execute the instructions, or in conjunction with such an instruction execution system, apparatus, or device. For the purposes of this specification, a "computer-readable medium" can be any apparatus that can contain, store, communicate, propagate, or transport the program for use by or in connection with the instruction execution system, apparatus, or device, or in conjunction with such an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of the computer-readable medium include the following: an electrical connection having one or more wires (electronic devices), a portable computer diskette (magnetic devices), a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disc read-only memory (CDROM). In addition, the computer-readable medium can even be paper or other suitable medium on which the program can be printed, as the program can be electronically obtained from the paper or other suitable medium, by optically scanning the paper or other suitable medium, then editing, interpreting, or otherwise processing the electronically obtained program to store it in a computer memory.

[0099] It should be understood that portions of the application can be realized with a combination of hardware, software, firmware, or their combination. In the above-described embodiments, the N steps or methods can be realized with software or firmware stored in a memory and executed by a suitable instruction execution system. As in another embodiment, if realized with hardware, any one or their combination of the following technologies known in the art can be used: discrete logic circuit with logic gate circuit for implementing logic functions on data signals, application specific integrated circuit with suitable combination logic gate circuit, programmable gate array (PGA), field programmable gate array (FPGA), etc.

[0100] Those skilled in the art of the present technology can understand that all or part of the steps carried out by the above-mentioned embodiment method can be completed by a program instructing the relevant hardware, and the program can be stored in a computer readable storage medium. When the program is executed, it includes one of the steps of the method embodiment or a combination thereof.

[0101] In addition, the functional units in each embodiment of the present application can be integrated into one processing module, or each unit can be physically present separately, or two or more units can be integrated into one module. The above-mentioned integrated module can be realized in the form of hardware or in the form of a software function module. The integrated module, if realized in the form of a software function module and sold or used as an independent product, can also be stored in a computer readable storage medium.

[0102] The above-mentioned storage medium can be a read-only memory, a magnetic disk or an optical disk, etc. Although the embodiments of the present application have been shown and described above, it should be understood that the above-mentioned embodiments are exemplary and cannot be understood as limiting the present application, and those skilled in the art can make changes, modifications, replacements and variations to the above-mentioned embodiments within the scope of the present application.

Claims

1. An interactive method for emotional companionship, applied to a car machine end, characterized in that, The method comprises the following steps: obtaining facial data and / or voice data of a user; inputting the facial data and / or voice data into a language interaction model that has been trained to obtain emotional care text data; interacting with the user by using the emotional care text data as voice information; wherein the language interaction model is trained in the following manner: collecting a language data set; inputting dialogue training samples in the language data set into a training supervised strategy model to obtain a corresponding language reply training data set; inputting the dialogue training samples and the language reply training data set into a training reward model to obtain a score of the dialogue training samples and the language reply training data; randomly inputting dialogue samples into a reinforcement learning model to obtain corresponding language reply data, and obtaining a score of the language reply data according to the training reward model to determine the emotional care text data; wherein, the inputting of the dialogue training samples in the language data set into the training supervised strategy model to obtain the corresponding language reply training data set further comprises the following steps: randomly inputting dialogue training samples in the language data set; inputting the dialogue training samples into the training supervised strategy model respectively to obtain a plurality of language reply training data corresponding to the dialogue training samples; sorting and classifying the plurality of language reply training data by marking; integrating the sorted and classified language reply training data into the language reply training data set; the inputting of the dialogue training samples and the language reply training data set into the training reward model to obtain the score of the dialogue training samples and the language reply training data further comprises the following steps: correspondingly matching the dialogue training samples with language reply training data in the language reply training data set; correspondingly matching the language reply training data with the dialogue training samples according to the sorting result; composing training data pairs by the dialogue training samples and the language reply training data; determining the score of the training data pairs according to evaluation indexes; the obtaining of the score of the language reply data according to the training reward model to determine the emotional care text data further comprises the following steps: inputting the dialogue samples and the language reply data into the training reward model to obtain the score of the training data pairs; taking the language reply data in the training data pair with the higher score as the emotional care text data.

2. The interaction method of claim 1, wherein, The method comprises the following steps: in the case of obtaining a facial image of a user; performing image enhancement feature processing on the facial image to obtain facial data of the user; in the case of obtaining voice information of a user; performing noise reduction feature processing on the voice information to obtain voice data of the user.

3. An interactive device for emotional companionship, characterized in that, The method comprises the following steps: an obtaining module configured to obtain facial data and / or voice data of a user; a determining module configured to input the facial data and / or voice data into a language interaction model that has been trained to obtain emotional care text data; an output module configured to interact with the user by using the emotional care text data as voice information; wherein, in the determining module, the language interaction model is trained in the following manner: collecting a language data set; inputting dialogue training samples in the language data set into a training supervised strategy model to obtain a corresponding language reply training data set; inputting the dialogue training sample and the language reply training data set into the training reward model to obtain a score of the dialogue training sample and the language reply training data; randomly inputting the dialogue sample into the reinforcement learning model to obtain corresponding language reply data, and obtaining a score of the language reply data according to the training reward model to determine the emotional accompanying text data; wherein, the inputting the dialogue training sample in the language data set into the training supervised strategy model to obtain corresponding language reply training data set further comprises: randomly inputting the dialogue training sample in the language data set; inputting the dialogue training sample into the training supervised strategy model to obtain a plurality of language reply training data corresponding to the dialogue training sample; sorting the plurality of language reply training data by marking; integrating the sorted language reply training data into the language reply training data set; the inputting the dialogue training sample and the language reply training data set into the training reward model to obtain a score of the dialogue training sample and the language reply training data further comprises: correspondingly matching the dialogue training sample with the language reply training data in the language reply training data set; correspondingly matching the language reply training data with the dialogue training sample according to the sorting result; the dialogue training sample and the language reply training data form a training data pair; determining a score of the training data pair according to the evaluation index; the inputting the dialogue sample and the language reply data into the training reward model to obtain a score of the training data pair; taking the language reply data in the training data pair with the larger score as the emotional accompanying text data. The processor is configured to execute the interactive method for emotional accompanying when running the program instructions.

4. An interactive device for emotional companionship, comprising a processor and a memory having stored therein program instructions, characterized in that, The interactive device for emotional accompanying.

5. A vehicle characterized by comprising: The program instructions execute the interactive method for emotional accompanying when running.

6. A storage medium storing program instructions, characterized in that, The program instructions execute the interactive method for emotional accompanying when running.

Images