Method, apparatus and medium for generating prompt audio

Abstract

The application discloses a prompt audio generation method, device, equipment and medium, and relates to the technical field of automobiles. The method is executed by a vehicle terminal of a first vehicle and comprises the following steps: collecting voice data of a first subject, the first subject being located in the first vehicle; analyzing the voice data to obtain emotional state information of the first subject, the emotional state information being used for indicating the emotional state of the first subject when expressing the voice data; generating corresponding prompt information based on the emotional state information, the prompt information being used for indicating emotional feedback provided to the first subject; converting the prompt information into prompt audio, and playing the prompt audio. The voice data of the driver can be collected, the current emotional state of the driver can be analyzed, and the accuracy of the recognition result can be improved. The corresponding emotional feedback is provided to the driver, the driver can adjust the mood in time, and the driving experience of the driver and the safety in the driving process can be improved.

Description

TECHNICAL FIELD

[0001] Embodiments of the present application relate to the technical field of automobiles, and in particular to a method and device for generating prompt audio, an apparatus, and a medium. BACKGROUND

[0002] In the field of modern intelligent automobiles, the development of vehicle-mounted systems is moving towards greater personalization and intelligence. Intelligent automobiles that can identify the emotions of drivers in a timely manner and provide corresponding comfort measures can enhance the emotional connection between drivers and vehicles and improve the overall driving experience and safety during travel.

[0003] In related technologies, facial expressions of drivers are analyzed to identify emotions through image processing and machine learning algorithms, and personalized comfort measures are provided accordingly.

[0004] However, changes in environmental light, the complexity of facial expressions, and high requirements for real-time processing speed can all affect the accuracy of emotion recognition and the comfort effect of the vehicle on the driver. SUMMARY

[0005] Embodiments of the present application provide a method and device for generating prompt audio, which can identify the emotional state of a driver in a timely manner and generate prompt audio for emotional feedback to the driver. The technical solution is as follows:

[0006] On the one hand, a method for generating prompt audio is provided, which is executed by a vehicle-mounted terminal of a first vehicle. The method includes:

[0007] Collecting voice data of a first subject, the first subject being located in the first vehicle;

[0008] Analyzing the voice data to obtain emotional state information of the first subject, the emotional state information being used to indicate the emotional state of the first subject when expressing the voice data;

[0009] Generating corresponding prompt information based on the emotional state information, the prompt information being used to indicate emotional feedback provided to the first subject;

[0010] Converting the prompt information into prompt audio and playing the prompt audio.

[0011] On the other hand, a device for generating prompt audio is provided, which includes:

[0012] A collecting module configured to collect voice data of a first subject, the first subject being located in the first vehicle;

[0013] an analysis module configured to analyze the voice data to obtain emotional state information of the first subject, the emotional state information being indicative of an emotional state of the first subject when expressing the voice data;

[0014] a generation module configured to generate corresponding prompt information based on the emotional state information, the prompt information being indicative of providing emotional feedback to the first subject;

[0015] a conversion module configured to convert the prompt information into prompt audio and play the prompt audio.

[0016] In an optional embodiment, the analysis module is further configured to input the voice data into a pre-trained deep learning model, perform emotional analysis on the voice data by using the deep learning model, and output an analysis result, the analysis result including at least one candidate emotional state; collect driving information of the first vehicle, the driving information of the first vehicle including vehicle operation information of the first vehicle and environmental information of an environment in which the first vehicle is located; and determine the emotional state information of the first subject from the candidate emotional states based on the analysis result and the driving information.

[0017] In an optional embodiment, the analysis module is further configured to obtain an emotional state reference table, the emotional state reference table including a corresponding relationship between the driving information of the first vehicle and at least one emotional state; determine a first corresponding relationship matching the driving information from the emotional state reference table based on the driving information; and determine the emotional state information of the first subject from the candidate emotional states based on an emotional state indicated by the first corresponding relationship.

[0018] In an optional embodiment, before the analysis module, the apparatus further includes:

[0019] a training module configured to obtain sample voice data, the sample voice data being labeled with an emotional label, the emotional label being indicative of an emotional state expressed by the sample voice data; train a pre-trained model based on the sample voice data to obtain the pre-trained deep learning model.

[0020] In an optional embodiment, the training module is further configured to model a time sequence of the sample voice data by using the pre-trained model, determine context information and a time dependency of the sample voice data, the time dependency being a characteristic of the sample voice data in a time sequence; output a predicted emotional state based on the context information and the time dependency of the sample voice data; and adjust the pre-trained model based on a training result and the emotional label to obtain the pre-trained deep learning model.

[0021] In an optional embodiment, the generation module is further configured to input the emotional state information into a pre-trained natural language model, analyze the emotional state information through the natural language model, and determine the emotional type of the feedback of the vehicle-mounted terminal to the first subject; generate text prompt information conforming to the emotional type, wherein the text prompt information contains emotional feedback content in the form of text.

[0022] In an optional embodiment, the generation module is further configured to obtain historical text prompt information generated by the natural language model in a historical time period, wherein the historical text prompt information is used to indicate emotional feedback content provided by the vehicle-mounted terminal to the first subject in the historical time period; and filter information conforming to the emotional type from the historical text prompt information based on the emotional type as the text prompt information.

[0023] In another aspect, a computer device is provided, which includes a processor and a memory, and the memory stores at least one instruction, at least one program, a code set or an instruction set, which are loaded and executed by the processor to implement the method for generating prompt audio according to any of the above embodiments.

[0024] In another aspect, a computer readable storage medium is provided, which stores at least one instruction, at least one program, a code set or an instruction set, which are loaded and executed by a processor to implement the method for generating prompt audio according to any of the above embodiments.

[0025] In another aspect, a computer program product or computer program is provided, which includes computer instructions stored in a computer readable storage medium. A processor of a computer device reads the computer instructions from the computer readable storage medium, and the processor executes the computer instructions to make the computer device perform the method for generating prompt audio according to any of the above embodiments.

[0026] The technical scheme provided by the embodiments of the present application has at least the following beneficial effects:

[0027] By collecting voice data of the driver, analyzing the current emotional state of the driver, corresponding emotional feedback can be provided to the driver, helping the driver to adjust the mood in time, improving the driving experience of the driver and the safety in the driving process. Since the voice has continuity, with the continuous change of pitch, volume and the like in the voice data, the real emotions of the driver can be better reflected. Compared with the related art, the way of collecting the facial expressions of the driver to recognize the emotions of the driver, the emotional recognition of the driver based on the voice data can improve the accuracy of the recognition result, and then provide more suitable emotional feedback. BRIEF DESCRIPTION OF DRAWINGS

[0028] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed to be used in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0029] Figure 1 is a schematic diagram of a prompt audio generation system provided by an exemplary embodiment of the present application;

[0030] Figure 2 is a schematic diagram of a prompt audio generation link provided by an exemplary embodiment of the present application;

[0031] Figure 3 is a flowchart of a prompt audio generation method provided by an exemplary embodiment of the present application;

[0032] Figure 4 is a structural block diagram of a prompt audio generation device provided by an exemplary embodiment of the present application;

[0033] Figure 5 is a structural block diagram of a prompt audio generation device provided by another exemplary embodiment of the present application;

[0034] Figure 6 is a structural block diagram of a computer device provided by an exemplary embodiment of the present application. DETAILED DESCRIPTION

[0035] In order to make the purpose, technical solutions and advantages of the present application more clear, the embodiments of the present application will be further described in detail below with reference to the drawings.

[0036] The exemplary embodiments will be described in detail herein with reference to the attached drawings. The following description is made with reference to the accompanying drawings in which like reference numerals refer to like elements, unless the context of use indicates otherwise. The following description of exemplary embodiments is not representative of all embodiments consistent with the present application. Rather, it is merely an example of apparatus and methods consistent with some aspects of the present application as detailed in the appended claims.

[0037] The terminology used in this application is for the purpose of describing particular embodiments only and is not intended to be limiting of the present application. As used in this application and the appended claims, the singular forms "a," "an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will also be understood that the term "and / or" as used herein refers to and encompasses any and all possible combinations of one or more of the associated listed items.

[0038] It should be noted that the information and data involved in the present application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data need to comply with relevant laws, regulations and standards of relevant countries and regions.

[0039] It should be understood that although the terms first, second, etc. can be used herein to describe various information, these information should not be limited to these terms. These terms are only used to distinguish one piece of information from another piece of information of the same type. For example, a first parameter can also be referred to as a second parameter without departing from the scope of the present application, and similarly, a second parameter can also be referred to as a first parameter. Depending on the context, the word "if" as used herein can be interpreted as "when" or "upon determination" or "in response to a determination".

[0040] First, a brief introduction to the terms involved in the embodiments of the present application is given:

[0041] Recurrent Neural Network (RNN): A type of artificial neural network suitable for processing sequential data. RNNs can handle dynamic features in sequences, i.e. they can remember information from previous inputs and use this information to influence the current output. For example, RNNs can convert speech signals into text, learn the temporal features of speech signals, and predict the corresponding sequence of words. By analyzing the pitch, volume, and speed of speech, RNNs can also be used to identify the emotional state of the speaker, such as happiness, sadness, anger, etc.

[0042] Loss function: also known as cost function or objective function, is a function that measures the difference between the predicted value of the model and the actual value. In machine learning and deep learning, the loss function is a key part of the training process, which guides the learning direction of the model, that is, by minimizing the loss function to optimize the model parameters.

[0043] Among them, the type of loss function is diverse, such as cross-entropy loss function, and the actual application scene can be selected according to the corresponding loss function to train the model.

[0044] Optimizer: in machine learning, it is an algorithm used to adjust model parameters, whose purpose is to minimize the loss function. The optimizer updates the parameters of the model through iteration, so that the model performs best on the training data.

[0045] In the modern automotive industry, the innovation of in-vehicle systems is constantly moving towards personalization and intelligence. Intelligent cars can recognize the emotional state of the driver through advanced technology, and provide customized comfort measures accordingly to improve the driving experience and driving safety of the driver.

[0046] When the car recognizes the driver's emotions, it usually relies on image processing and machine learning algorithms to collect the driver's facial images, etc., and analyzes the driver's facial expressions to recognize their emotional state. For example, if the system identifies that the driver is nervous or anxious, it may provide some soothing measures, such as adjusting the air conditioning temperature or playing soft natural sounds.

[0047] However, the above-mentioned method faces a series of challenges in practical application. Changes in environmental light may affect the accuracy of facial expression recognition. In strong light or weak light conditions, facial features may be difficult to capture, leading to misjudgment of the emotional recognition system. And human emotional expression is diverse, sometimes subtle changes in expression may be difficult for algorithms to accurately capture.

[0048] The present application provides a method for generating prompt audio, which can analyze the emotional state of the driver based on the voice data of the driver, improve the accuracy of the emotional recognition result, and generate corresponding prompt audio to provide emotional feedback to the driver, avoiding accidents caused by the emotional state of the driver in the process of controlling the vehicle to travel.

[0049] Secondly, the generation system of the prompt audio involved in the embodiment of the present application is described, and schematically, please refer to Figure 1 The first vehicle 100, the server 120 are connected through the communication network 140.

[0050] The first vehicle 100 is deployed with a vehicle terminal and a voice collection component. The voice collection component is used to collect voice data of the driver under the condition that the driver authorizes to start. The voice data of the driver includes the voice of the driver. After the voice data is collected, the voice data is sent to the server 120 through the communication network 140 by the vehicle terminal. The server 120 analyzes and processes the voice data to identify the emotional state of the driver reflected by the voice data, and generates a prompt audio corresponding to the emotional state information and returns it to the vehicle terminal. The prompt audio is used to provide emotional feedback to the driver, for example, to comfort the driver and make the driver calm down.

[0051] The server 120 includes a deep learning model. The voice data is input into the deep learning model, and the corresponding emotional state information is output. The emotional state information is used to indicate the emotional state of the driver.

[0052] The server 120 generates corresponding prompt information based on the emotional state information, synthesizes the prompt information into a prompt audio, and returns it to the vehicle terminal.

[0053] The vehicle terminal plays the prompt audio in time to provide emotional feedback to the driver. For example, when the emotional state information indicates that the driver is in a sad state, the prompt audio is used to comfort the driver.

[0054] In some embodiments, the above process can be regarded as a plurality of processing links, which are executed by the vehicle terminal and the server together.

[0055] Optionally, as shown in FIG. 2, the process includes the following links: a voice data collection link 210, an emotional recognition link 220, an emotional analysis link 230, an audio synthesis link 240, and an audio output link 250. Figure 2

[0056] The voice data collection link 210 is used to collect voice data of the driver in the car or during driving. It can be realized by a vehicle microphone or other voice collection devices. In some embodiments, this link also includes a pre-processing step of the voice data, such as removing environmental noise and determining valid voice segments. The valid voice segment refers to a voice segment containing the voice of the driver.

[0057] The emotional recognition link 220 is used to identify and classify the emotional state of the driver. A deep learning model is usually used for emotional recognition, such as a recurrent neural network or a convolutional neural network. The input of this link is the voice data, and the output is the corresponding emotional state information, such as happy, frustrated, anxious, etc.

[0058] ​Emotion analysis link 230: Further analysis and processing is performed on the driver emotional state information identified by the emotion recognition link 220. In some embodiments, this link combines other information (such as vehicle speed, traffic conditions of the environment in which the vehicle is located, etc.) to comprehensively analyze the emotional state of the driver.

[0059] Audio synthesis link 240: Based on the results of emotion recognition and analysis, appropriate voice comfort content is selected, and corresponding prompt audio is generated using voice synthesis technology. This link can convert text into natural and fluent speech to provide specific voice feedback such as comfort, encouragement, and reminders.

[0060] Audio output link 250: Responsible for outputting the generated voice synthesis content to the driver, which can be output through the car audio system or the earphones worn by the driver to achieve the function of emotional comfort and guidance for the driver.

[0061] It is worth noting that the above-mentioned server can be a standalone physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms, etc. Basic cloud computing services.

[0062] Among them, cloud technology refers to a kind of hosting technology that unifies a series of resources such as hardware, software, network in a wide area network or local area network to realize the calculation, storage, processing and sharing of data. Cloud technology is a general term for network technology, information technology, integration technology, management platform technology, application technology, etc. based on cloud computing business model application, which can form a resource pool, use on demand, and is flexible and convenient. Cloud computing technology will become an important support. The background service of the technical network system needs a large amount of calculation and storage resources, such as video websites, picture websites and more portal websites. With the high development and application of the Internet industry, every item in the future may have its own identification mark and needs to be transmitted to the background system for logical processing. Different levels of data will be processed separately, and various industry data will need strong system support, which can only be realized through cloud computing.

[0063] In some embodiments, the above-mentioned server can also be implemented as a node in a blockchain system.

[0064] With the above-mentioned name introduction and application scenarios, the method for generating prompt audio provided in the present application is described. The method can be executed by a server or a vehicle terminal, or can be executed by a server and a vehicle terminal together. In the present embodiment, the method is executed by the vehicle terminal of the first vehicle, as shown in Figure 3 Figure 3 is a flowchart of the method for generating prompt audio provided in an exemplary embodiment of the present application. The method includes the following steps.

[0065] Step 310: Collecting voice data of the first subject.

[0066] The first subject is located in the first vehicle, and the first subject can be a driver or a passenger in the first vehicle.

[0067] Optionally, the first vehicle is internally configured with a component capable of collecting vehicle internal audio data, wherein the audio data includes voice data when the subject in the first vehicle speaks. For example, the first vehicle is internally configured with a microphone array composed of multiple microphones, which can capture the sound in the first vehicle and determine the source of the sound based on sound positioning technology.

[0068] The starting mode of the voice collection component includes but is not limited to: (1) the first subject operates the switch of the voice collection component to start; (2) the first subject uses other terminals or devices having a binding relationship with the voice collection component to start; (3) the first subject starts through a voice instruction, for example, the voice collection component has a voice control function.

[0069] The voice data refers to the data obtained by the voice collection component collecting the voice signal existing in the vehicle interior when the subject in the first vehicle starts the voice collection component function. When the voice collection component is not started, the voice signal in the first vehicle cannot be collected.

[0070] In some embodiments, the voice collection component continuously maintains a starting state, and when there is a subject in the first vehicle, the voice collection component actively plays a prompt voice to seek voice collection permission from the subject in the first vehicle.

[0071] For example, there is a sensor in the first vehicle, which can detect whether there is a first subject in the first vehicle. When it is detected that a subject has entered the first vehicle, the vehicle terminal automatically controls the voice collection component to play a prompt voice: “whether to start voice collection permission”. If the subject in the vehicle agrees to start the voice collection permission through a voice instruction or other means, the voice collection component continues to maintain the starting state. If the subject in the vehicle refuses to start the voice collection permission through a voice instruction or other means, the voice collection component is automatically turned off.

[0072] ​Optionally, taking the first subject as a driver as an example. The voice data of the first subject contains the voice of the first subject, i.e., the language sound produced by the vocal cords of the first subject, such as speaking, reading, singing, etc., and the voice data can reflect the tone, volume, speed and continuity of the first subject when speaking.

[0073] Optionally, the voice data also includes other sounds emitted by the first subject, such as laughter, crying, sighing, coughing, etc., which can reflect the physical health and emotional state of the first subject.

[0074] In some embodiments, the voice data collected by the voice collection component may also include sounds emitted by other sources in addition to the sounds emitted by the first subject, such as environmental noise in the environment where the first vehicle is located, and sounds emitted by electronic devices operating inside the first vehicle.

[0075] When performing voice analysis, such as emotional analysis, removing noise helps to more accurately capture key features of the first subject in the voice data. In order to highlight the sounds in the voice data that directly come from the first subject and improve the accuracy of subsequent identification and analysis of the voice data, the vehicle terminal performs noise reduction processing on the voice data after the voice collection component has completed collection, to remove noise in the environment and determine the effective voice segment in the voice data.

[0076] Step 320, analyzing the voice data to obtain the emotional state information of the first subject.

[0077] The emotional state information is used to indicate the emotional state of the first subject when expressing the voice data.

[0078] For example, the voice data is collected when the first subject sings, and the emotional state information obtained by analyzing the voice data is used to indicate the emotional state of the first subject when singing, such as excitement, sadness, excitement, joy, etc.

[0079] Optionally, the voice data is input into a pre-trained deep learning model, and the voice data is analyzed by the deep learning model to obtain and output an analysis result, the analysis result containing at least one candidate emotional state.

[0080] For example, the deep learning model refers to a recurrent neural network model, and the structure of the recurrent neural network model includes an input layer, a hidden layer and an output layer.

[0081] The input layer is the first layer of the neural network receiving the original data. After the voice data is input into the recurrent neural network model, the voice data first enters the input layer, which maps the voice data to corresponding numerical features for the recurrent neural network to process the data. The hidden layer is used for feature extraction of the input data, combining simple features into more complex features to help the neural network understand the input data, learn the importance of each feature in the input data, and adjust the neural network to achieve the purpose of optimizing performance, and pass the processed data to the next layer. The output layer converts the information passed by the hidden layer into the final prediction result of the model.

[0082] That is, the deep learning model processes the voice data through at least the above three structure layers, and the prediction data obtained by output is the analysis result.

[0083] For example, the data collected when the voice data is the first subject of the voice call is taken as an example.

[0084] The call content of the first subject is collected after the first subject authorizes the voice collection component, such as: "The road is too congested, I should have been home in 20 minutes, but now it looks like at least two hours late".

[0085] Among them, the voice data reflects that the volume of the first subject is high, the pitch is high, and the speech speed is fast. Then, the voice data is input into the pre-trained deep learning model, the deep learning model converts the call content in the voice data into text content, and analyzes the sentiment tendency of the text content by natural language processing. The pitch reflected by the voice data is used to identify the high pitch change of the first subject, and the volume reflected by the voice data is used to evaluate whether the emotion of the first subject is high. Combined with the text content, the emotional state of the first subject is analyzed, and at least one candidate emotion is obtained.

[0086] Through the above analysis, it can be known that the volume of the first subject is significantly increased, the speech speed is accelerated, the pitch is sharp, and the intonation fluctuation is large, and the text content contains words expressing dissatisfaction and anxiety.

[0087] From the text content, the first subject is in a congested road condition and is anxious about the possible late return. For example, the analysis result output by the deep learning model includes the following candidate emotional states: anxiety, anger, anger, fatigue, and impatience.

[0088] Optionally, the driving information of the first vehicle is collected, and the driving information of the first vehicle includes vehicle running information of the first vehicle and environmental information of an environment in which the first vehicle is located.

[0089] Based on the analysis result and the driving information, the emotional state information of the first subject is determined from the candidate emotional states.

[0090] For example, the environmental information indicates that the road where the first vehicle is located is congested, there are many vehicles on the road, and the following distance between the first vehicle and other vehicles is short. The vehicle operation information indicates that the first vehicle is in the off state for 5 minutes.

[0091] Therefore, the driving information indicates that the road section where the first vehicle is located remains stationary for a short time, and the first vehicle is in the off state to save vehicle energy consumption elements. Therefore, the analysis result and the driving information both indicate that the emotional state of the first subject is impatience and anger.

[0092] For example, an emotional state reference table is obtained, which includes a corresponding relationship between the driving information of the first vehicle and at least one emotional state.

[0093] Based on the driving information, a first corresponding relationship matching the driving information is determined from the emotional state reference table.

[0094] Based on the emotional state indicated by the first corresponding relationship, the emotional state information of the first subject is determined from the candidate emotional states.

[0095] For example, as shown in Table 1 below, Table 1 is an example of an emotional state reference table for indicating the corresponding relationship between the driving information and the emotional state, wherein the driving information is used to indicate the situation / road condition encountered by the driver when driving the first vehicle.

[0096] Table 1

[0097]

[0098] For example, the current driving information indicates that the first vehicle remains stationary for 5 minutes, which indicates that the road condition at this time belongs to "traffic jam", and the analysis process of the voice data determines that the first subject's tone is raised, so the emotional state information of the first subject is most likely to be "explosive anger". Therefore, based on the driving information, a first corresponding relationship matching the driving information is determined from the emotional state reference table as follows: traffic jam-explosive anger.

[0099] Among the at least one candidate emotional state, the following emotional states are included: anxiety, anger, anger, fatigue, impatience, and based on the first corresponding relationship, "anger" is selected from the candidate emotional states as the emotional state information of the first subject.

[0100] The types of deep learning models include but are not limited to recurrent neural networks and convolutional neural networks. Taking a recurrent neural network as an example, the training process of the model is as follows: obtaining sample voice data, the sample voice data is labeled with an emotional label, and the emotional label is used to indicate the emotional state expressed by the sample voice data. Based on the sample voice data, the pre-trained model is trained to obtain a pre-trained deep learning model.

[0101] For example, the sample voice data contains multiple sub-data, each of which corresponds to a different emotion label, including a first sub-data.

[0102] For example, the first sub-data is the conversation content when the user encounters an elevator failure and calls the staff for help through the telephone in the elevator, and the emotion label marked by the first sub-data is: nervousness and fear.

[0103] It is worth noting that since the emotional state of a human being is not single in most cases, each sub-data can correspond to at least one emotion label, so that the deep learning model can learn the human emotion more comprehensively and improve the accuracy of analysis and recognition. When training the model based on the first sub-data, multiple sub-data that jointly contain the same emotion label can be combined to specifically train the model to recognize the emotion indicated by the emotion label, for example, sub-data 1 contains labels A, B, and C, and sub-data 2 contains labels A and D. Sub-data 1 and sub-data 2 are used as input data to train the model to recognize the emotion label A. Different emotion labels can also be marked on the same sub-data, and the model is trained to recognize the emotion indicated by one of the emotion labels based on only one sub-data each time. This embodiment is not limited in this regard.

[0104] Optionally, the time sequence of the sample voice data is modeled by the pre-trained model to determine the context information and time dependency of the sample voice data. The time dependency refers to the characteristics of the sample voice data in the time sequence.

[0105] The predicted emotion state is output based on the context information and time dependency of the sample voice data.

[0106] The pre-trained model is adjusted based on the training results and emotion labels to obtain a pre-trained deep learning model.

[0107] For example, sample voice data with emotion labels is prepared in advance, which includes voice samples of different emotional states. A recurrent neural network model is constructed to capture the time dependency in the sample voice data through the ability of sequence learning.

[0108] The time-dependent relationship of the voice data refers to the correlation of the voice signal in the time sequence, which is embodied in multiple aspects, for example: (1) phoneme dependence: phonemes (such as consonants and vowels) in the voice usually do not exist in isolation, and the pronunciation of a phoneme is often affected by the previous and subsequent phonemes, which is called coarticulation or assimilation; (2) phrase and sentence dependence: at the phrase or sentence level, the intonation, rhythm and intensity of the voice are affected by the sentence structure and semantic content, showing continuity and variability in time; (3) emotional expression: the emotional state in the voice, such as happiness, sadness, anger, etc., will be expressed through the volume, tone, speed, etc. of the voice in a period of time, showing continuity in time; (4) long-term dependence: in a longer voice paragraph, such as a speech or a conversation, some characteristics of the voice may remain consistent or gradually change throughout the paragraph, showing long-term time dependence.

[0109] The structure of the recurrent neural network includes an input layer, a hidden layer and an output layer. In the training process, the recurrent neural network models the time sequence of the voice data, helping the model to capture the context information and long-term dependence in the sample voice data.

[0110] The recurrent neural network model is trained using sample voice data labeled with emotional labels, and appropriate loss functions (such as cross-entropy loss function) and optimizers (such as Adam optimizer) are used for parameter optimization. The model parameters are constantly updated through the backpropagation algorithm to minimize the value of the loss function. Finally, the trained model is evaluated by the test set to determine the classification performance of the model in the emotion classification task. The test set also contains multiple sample voice data with pre-known emotional states / labels.

[0111] Step 330, generating corresponding prompt information based on the emotional state information.

[0112] The prompt information is used to indicate the emotional feedback to the first subject.

[0113] Exemplarily, the types of prompt information include but are not limited to: text prompt information, voice prompt information, animation prompt information, picture prompt information, etc. The embodiment takes the text prompt information as an example for illustration.

[0114] The emotional state information is input into the pre-trained natural language model, and the natural language model is used to identify and analyze the emotional state information to determine the emotional type of the feedback to the first subject by the vehicle-mounted terminal.

[0115] Text prompt information conforming to the emotional type is generated, and the text prompt information contains emotional feedback content in the form of text.

[0116] The natural language model (NLM) is a computational model used to process, understand, and generate human language, i.e., natural language. These models aim to simulate the way humans understand and use language, enabling computers to perform various language-related tasks. Natural language models can understand the structure and meaning of text content, perform sentiment analysis on text content, identify sentiment tendencies in text content, such as positive, negative, or neutral, and generate natural and fluent language text as feedback.

[0117] In some embodiments, the text prompt information can be pre-prepared text content that has a corresponding relationship with different sentiment state information. For example, when the sentiment state information indicates that the first subject is in a state of depression or sadness, the text prompt information corresponds to text content that can give the first subject comfort and encouragement.

[0118] Alternatively, the text prompt information can also be determined from the text prompt information generated by the natural language model in the historical time period. Directly using the prompt information generated in the historical time period according to the sentiment state information can improve the efficiency of the natural language model in generating feedback content.

[0119] Optionally, the historical text prompt information generated by the natural language model in the historical time period is obtained, and the historical text prompt information is used to indicate the emotional feedback content provided by the vehicle terminal to the first subject in the historical time period.

[0120] The historical text prompt information is filtered based on the sentiment type to obtain the text prompt information that conforms to the sentiment type.

[0121] In some embodiments, the vehicle terminal has network connection capability and can access and obtain multimedia content that has been disclosed on an Internet platform. After inputting the sentiment state information into the pre-trained natural language model and identifying and analyzing the sentiment state information by the natural language model to determine the sentiment type of the feedback provided by the vehicle terminal to the first subject, the vehicle terminal automatically indexes the Internet platform based on the feedback sentiment type, collects text content such as stories, poems, scripts, and news that conforms to the feedback sentiment type, and obtains the text prompt information based on the text content. For example, after summarizing the text content, the text prompt information is obtained; or after extracting keywords from the text content, the text prompt information is automatically generated based on the extracted keywords, wherein the keywords can be words with a high frequency of occurrence in the text content, or words that summarize the text content.

[0122] In some embodiments, the first subject logs in a social account on the in-vehicle terminal of the first vehicle, the social account being an account associated with the information system, service or application of the first vehicle, and the first subject can invoke services and functions inside the first vehicle by logging in the social account.

[0123] Optionally, the social account of the first subject is associated with other social accounts, for example, the other social accounts are accounts logged in by the first subject on an Internet application platform, or the other social accounts are accounts logged in by a second subject on an Internet application platform or other vehicles, and the other social accounts and the social account of the first subject are friend accounts of each other.

[0124] Taking the second subject logging in a friend account as an example, after the deep learning model outputs the emotional state information of the first subject, the emotional state information is sent to the friend account, and the second subject edits prompt information according to the emotional state information and feeds back the prompt information to the social account of the first subject through the friend account. The in-vehicle terminal receives the prompt information fed back by the second subject.

[0125] For example, the emotional state information indicates that the first subject is in a depressed state, and the prompt information fed back by the second subject through the social account is an audio with a comforting emotional feature, and the audio contains the voice of the second subject.

[0126] In some embodiments, after the deep learning model outputs the emotional state information of the first subject, the in-vehicle terminal generates adjustment suggestions for the first vehicle based on the collected vehicle running information as prompt information. The vehicle running information further includes the air temperature and humidity inside the first vehicle, the working state of various in-vehicle devices in the first vehicle, etc., such as the brightness of the in-vehicle lighting device, whether the in-vehicle air conditioner is turned on, whether the window is opened, etc.

[0127] For example, the emotional state information indicates that the first subject is in an angry state due to traffic congestion, and the vehicle running information indicates that the air temperature inside the first vehicle is 30°C, the in-vehicle air conditioner is turned off, and the window is opened. Then the in-vehicle terminal automatically generates the following suggestions: close the window and turn on the in-vehicle air conditioner. Based on the above suggestions, the prompt information is as follows: the road is congested, the noise outside the vehicle is large, please close the window, and the temperature inside the vehicle is high, please turn on the in-vehicle air conditioner.

[0128] Step 340, converting the prompt information into prompt audio and playing the prompt audio.

[0129] Wherein, taking the prompt information as a text prompt information as an example, the process of converting the text content into audio content is usually called text-to-speech (TTS) conversion, which mainly includes the following steps: (1) Text preprocessing: word segmentation of the text; (2) Voice selection: selecting a pre-recorded voice or a synthesized voice, which can be male or female, or have different age characteristics; (3) Phoneme synthesis: decomposing the text into phonemes or phonetic symbols; (4) Rhythm generation: determining the intonation, rhythm, intensity, etc. of the sentence; (5) Speech synthesis: converting phonemes into audio signals using a TTS engine;

[0130] (6) Output audio: output the processed audio in a common audio format.

[0131] In some embodiments, the emotional state of the first subject can change in a short time, therefore, after generating the prompt audio, the voice data of the first subject is collected in real time according to the voice collection component and emotional analysis, which can reflect the emotional changes of the first subject, if the emotional state information of the first subject at the current time still meets the generated prompt audio, the prompt audio can be played directly. If the emotional state information of the first subject at the current time has changed, the corresponding prompt audio can be re-generated in combination with the change of the emotional state information of the first subject at the current time and the historical time.

[0132] In some embodiments, when there are multiple subjects in the first vehicle, the voice data collected by the voice collection component is likely to contain the conversation content between multiple subjects in the vehicle, and then according to the difference of the conversation content, the voice data can be analyzed by emotional analysis to generate prompt audio suitable for emotional feedback to multiple subjects and play the prompt audio.

[0133] For example, after the pre-trained deep learning model analyzes the voice data and determines that there are two subjects in the first vehicle who are arguing, the prompt audio played by the vehicle audio should be neutral, peaceful and helpful to ease the tense atmosphere. For example, the calm reminder audio: “Try deep breathing, calm down and continue the conversation”; the audio emphasizing safety: “Safety driving is the most important, if the current emotion affects the driving, please consider stopping the car for a rest first”.

[0134] It is worth noting that when emotional feedback is given to different situations, appropriate audio should be selected in combination with the specific scene. In addition to generating corresponding audio after analyzing the collected voice data, playing locally stored music can also achieve the effect of comforting the subjects in the vehicle. For example, playing slow-paced music to help the subjects in the vehicle calm down and remain calm, etc.

[0135] In summary, the application provides a method for generating prompt audio. By collecting voice data of a driver and analyzing the current emotional state of the driver, corresponding emotional feedback can be provided to the driver to help the driver adjust their mood in a timely manner and improve the driving experience and safety of the driver. Since voice is continuous, as the pitch and volume in the voice data change constantly, the real emotions of the driver can be better reflected. Compared with the related art, which collects facial expressions of the driver to identify the emotions of the driver, the emotional recognition of the driver based on voice data can improve the accuracy of the recognition result and thus provide more suitable emotional feedback.

[0136] Figure 4 FIG. 1 is a structural block diagram of a device for generating prompt audio according to an example embodiment of the application, as shown in the figure, the device comprises the following parts. Figure 4

[0137] The collection module 410 is configured to collect voice data of a first subject in a first vehicle.

[0138] The analysis module 420 is configured to analyze the voice data to obtain emotional state information of the first subject, the emotional state information being used to indicate the emotional state of the first subject when expressing the voice data.

[0139] The generation module 430 is configured to generate corresponding prompt information based on the emotional state information, the prompt information being used to indicate emotional feedback provided to the first subject.

[0140] The conversion module 440 is configured to convert the prompt information into prompt audio and play the prompt audio.

[0141] In an optional embodiment, the analysis module 420 is further configured to input the voice data into a pre-trained deep learning model, perform emotional analysis on the voice data through the deep learning model, obtain and output an analysis result, and the analysis result contains at least one candidate emotional state; collect driving information of the first vehicle, the driving information of the first vehicle including vehicle operation information of the first vehicle and environmental information of an environment in which the first vehicle is located; and determine the emotional state information of the first subject from the candidate emotional states based on the analysis result and the driving information.

[0142] ​In an optional embodiment, the analysis module 420 is further configured to obtain an emotion state reference table, the emotion state reference table comprising a correspondence between driving information of the first vehicle and at least one emotion state; determine a first correspondence matching the driving information from the emotion state reference table based on the driving information; and determine the emotion state information of the first subject from the candidate emotion states based on an emotion state indicated by the first correspondence.

[0143] In an optional embodiment, the analysis module 420 is further configured to obtain an emotion state reference table, the emotion state reference table comprising a correspondence between driving information of the first vehicle and at least one emotion state; determine a first correspondence matching the driving information from the emotion state reference table based on the driving information; and determine the emotion state information of the first subject from the candidate emotion states based on an emotion state indicated by the first correspondence. Figure 5 The apparatus further comprises:

[0144] The training module 450 is configured to obtain sample voice data, the sample voice data being labeled with an emotion label, the emotion label being used to indicate an emotion state expressed by the sample voice data; and train a pre-trained model based on the sample voice data to obtain the pre-trained deep learning model.

[0145] In an optional embodiment, the training module 450 is further configured to model a time sequence of the sample voice data by the pre-trained model, determine context information and time dependency of the sample voice data, the time dependency being a characteristic of the sample voice data in a time sequence; output a predicted emotion state based on the context information and the time dependency of the sample voice data; and adjust the pre-trained model based on a training result and the emotion label to obtain the pre-trained deep learning model.

[0146] In an optional embodiment, the generation module 430 is further configured to input the emotion state information into a pre-trained natural language model, identify and analyze the emotion state information by the natural language model, determine an emotion type of the first subject when the vehicle-mounted terminal provides feedback to the first subject, and generate text prompt information conforming to the emotion type, the text prompt information comprising emotion feedback content in a text form.

[0147] In an optional embodiment, the generation module 430 is further configured to obtain historical text prompt information generated by the natural language model in a historical time period, the historical text prompt information being used to indicate emotion feedback content provided by the vehicle-mounted terminal to the first subject in the historical time period; and filter information conforming to the emotion type from the historical text prompt information as the text prompt information based on the emotion type.

[0148] To sum up, the prompt audio generation device provided in the application can collect voice data of the driver, analyze the current emotional state of the driver, provide corresponding emotional feedback to the driver, help the driver adjust the mood in time, and improve the driving experience and safety of the driver during driving. Since voice has continuity, with the continuous change of pitch, volume and the like in the voice data, the real emotions of the driver can be better reflected. Compared with the way of collecting facial expressions of the driver to identify the emotions of the driver in the related art, the driver emotional recognition based on voice data can improve the accuracy of the recognition result, and further provide more suitable emotional feedback.

[0149] It should be noted that: the prompt audio generation device provided in the above embodiments is only exemplified by the division of the above functional modules. In actual application, the above functions can be completed by different functional modules according to needs, that is, the internal structure of the device is divided into different functional modules to complete all or part of the above described functions. In addition, the prompt audio generation device and the prompt audio generation method provided in the above embodiments belong to the same concept, and the specific implementation process is described in the method embodiments, which will not be repeated here.

[0150] Figure 6 The structural block diagram of the computer device 600 provided in an example embodiment of the application is shown. The computer device 600 can be: a smart phone, a tablet computer, an MP3 player (Moving Picture Experts Group Audio Layer III), an MP4 (Moving Picture Experts Group Audio Layer IV) player, a notebook computer or a desktop computer. The computer device 600 can also be referred to as a user device, a portable terminal, a laptop terminal, a desktop terminal and other names.

[0151] Generally, the computer device 600 includes a processor 601 and a memory 602.

[0152] The processor 601 can include one or more processing cores, such as a 4-core processor, an 8-core processor, and the like. The processor 601 can be implemented in at least one of a hardware form of a DSP (Digital Signal Processing), an FPGA (Field-Programmable Gate Array), a PLA (Programmable Logic Array). The processor 601 can also include a main processor and a coprocessor, the main processor being a processor for processing data in an awake state, also known as a CPU (Central Processing Unit), and the coprocessor being a low-power processor for processing data in a standby state. In some embodiments, the processor 601 can be integrated with a GPU (Graphics Processing Unit) that is responsible for rendering and drawing the content required to be displayed by the display screen. In some embodiments, the processor 601 can also include an AI (Artificial Intelligence) processor for processing machine learning-related computing operations.

[0153] The memory 602 can include one or more computer-readable storage media that can be non-transitory. The memory 602 can also include a high-speed random access memory, and a nonvolatile memory such as one or more disk storage devices, flash storage devices. In some embodiments, the non-transitory computer-readable storage medium in the memory 602 is used to store at least one instruction for being executed by the processor 601 to implement the method for generating prompt audio provided by the method embodiments in the present application.

[0154] In some embodiments, the computer device 600 further includes some other components 603, and the type and number of the other components 603 can be selected based on the functional needs of the computer device 600. Those skilled in the art can understand that the structure shown in the figure does not constitute a limitation on the computer device 600, and can include more or fewer components than shown, or combine certain components, or use different arrangements of components. Figure 6 The structure shown in the figure does not constitute a limitation on the computer device 600, and can include more or fewer components than shown, or combine certain components, or use different arrangements of components.

[0155] Optionally, the computer readable storage medium can include a read only memory (ROM), a random access memory (RAM), a solid state disk (SSD), an optical disk, etc. The random access memory can include a resistance random access memory (ReRAM) and a dynamic random access memory (DRAM). The above-mentioned application embodiment numbers are only for description, and do not represent the advantages and disadvantages of the embodiments.

[0156] The application further provides a computer device, including a processor and a memory, the memory storing at least one instruction, at least one program, a code set or an instruction set, the at least one instruction, the at least one program, the code set or the instruction set being loaded and executed by the processor to implement the prompt audio generation method according to any one of the above-mentioned embodiments of the application.

[0157] The application further provides a computer readable storage medium, the storage medium storing at least one instruction, at least one program, a code set or an instruction set, the at least one instruction, the at least one program, the code set or the instruction set being loaded and executed by a processor to implement the prompt audio generation method according to any one of the above-mentioned embodiments of the application.

[0158] The application further provides a computer program product or a computer program, the computer program product or the computer program including computer instructions stored in a computer readable storage medium. A processor of a computer device reads the computer instructions from the computer readable storage medium, and the processor executes the computer instructions to make the computer device execute the prompt audio generation method according to any one of the above-mentioned embodiments.

[0159] Those skilled in the art can understand that all or part of the steps of the above-mentioned embodiments can be completed by hardware, or by a program instructing related hardware to complete, and the program can be stored in a computer readable storage medium. The storage medium mentioned above can be a read only memory, a magnetic disk or an optical disk, etc.

[0160] The above-mentioned is only the optional embodiment of the application, and does not limit the application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the application shall be included in the protection scope of the application.

Claims

1. A method of generating a prompt tone, characterized by, The method is executed by a terminal of a first vehicle, and comprises the following steps: collecting voice data of a first subject, the first subject being located in the first vehicle; inputting the voice data into a pre-trained deep learning model, performing sentiment analysis on the voice data by using the deep learning model, obtaining and outputting an analysis result, the analysis result including at least one candidate sentiment state; collecting driving information of the first vehicle, the driving information of the first vehicle including vehicle operation information of the first vehicle and environmental information of an environment in which the first vehicle is located; determining sentiment state information of the first subject from the candidate sentiment states based on the analysis result and the driving information, the sentiment state information being used to indicate a sentiment state of the first subject when expressing the voice data; inputting the sentiment state information into a pre-trained natural language model, performing recognition analysis on the sentiment state information by using the natural language model, and determining a sentiment type of the terminal when providing feedback to the first subject; generating text prompt information conforming to the sentiment type, the text prompt information including sentiment feedback content in a text form, the prompt information being used to indicate to provide sentiment feedback to the first subject; converting the prompt information into prompt audio, and playing the prompt audio.

2. The method of claim 1, wherein, The method further comprises the following steps before the step of inputting the voice data into the pre-trained deep learning model: obtaining sample voice data, the sample voice data being labeled with a sentiment label, the sentiment label being used to indicate a sentiment state expressed by the sample voice data; training a pre-training model based on the sample voice data, to obtain the pre-trained deep learning model. The method further comprises the following steps before the step of inputting the voice data into the pre-trained deep learning model:

3. The method of claim 1, wherein, obtaining sample voice data, the sample voice data being labeled with a sentiment label, the sentiment label being used to indicate a sentiment state expressed by the sample voice data; training a pre-training model based on the sample voice data, to obtain the pre-trained deep learning model. The method further comprises the following steps before the step of inputting the voice data into the pre-trained deep learning model:

4. The method of claim 3, wherein, modeling a time sequence of the sample voice data by using the pre-training model, to determine context information and a time dependency of the sample voice data, the time dependency being a characteristic of the sample voice data in the time sequence; outputting a predicted sentiment state based on the context information and the time dependency of the sample voice data; adjusting the pre-training model based on a training result and the sentiment label, to obtain the pre-trained deep learning model. ​ 5. The method of claim 1, wherein, The method further comprises: obtaining historical text prompt information generated by the natural language model in a historical time period, the historical text prompt information being used to indicate emotional feedback content provided by the vehicle terminal to the first subject in the historical time period; filtering information conforming to the emotional type from the historical text prompt information as the text prompt information based on the emotional type.

6. An apparatus for generating a prompt tone, the apparatus comprising: The device comprises: a collection module configured to collect voice data of a first subject, the first subject being located in a first vehicle; an analysis module configured to input the voice data into a pre-trained deep learning model, perform emotional analysis on the voice data by the deep learning model, and obtain and output an analysis result, the analysis result containing at least one candidate emotional state; collect driving information of the first vehicle, the driving information of the first vehicle including vehicle operation information of the first vehicle and environmental information of an environment in which the first vehicle is located; determine emotional state information of the first subject from the candidate emotional state based on the analysis result and the driving information, the emotional state information being used to indicate an emotional state of the first subject when expressing the voice data; a generation module configured to input the emotional state information into a pre-trained natural language model, perform recognition analysis on the emotional state information by the natural language model, and determine an emotional type of the vehicle terminal when providing feedback to the first subject; generate text prompt information conforming to the emotional type, the text prompt information containing emotional feedback content in a text form, the prompt information being used to indicate providing emotional feedback to the first subject; a conversion module configured to convert the prompt information into prompt audio and play the prompt audio.

7. A computer device, characterized by The computer device comprises a processor and a memory, the memory storing at least one program, the at least one program being loaded and executed by the processor to implement the method for generating prompt audio according to any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, The storage medium stores at least one program, the at least one program being loaded and executed by the processor to implement the method for generating prompt audio according to any one of claims 1 to 5.

Images