A call processing method, device, apparatus and computer readable storage medium

Abstract

Embodiments of the present application disclose a call processing method, device and equipment and a computer readable storage medium. The method comprises: obtaining first video data and first voice data of a driver when a mobile communication device receives an incoming call; determining a target emotion category of the driver based on the first video data, the first voice data and a preset emotion classification model; and determining whether to answer the incoming call based on the target emotion category. Thus, the emotion category of the driver for the incoming call can be determined according to the video data and the voice data of the driver obtained when the incoming call is received and the preset emotion classification model, and whether to answer the incoming call is automatically determined according to the emotion category, without manual operation, thereby improving driving safety.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a method, apparatus, device and computer-readable storage medium for handling incoming calls. Background Technology

[0002] With the continuous development of communication technology, drivers can answer calls through in-vehicle systems. Currently, in-vehicle systems on the market require manual operation to answer or block incoming calls. However, manual operation can distract drivers and affect driving safety. Summary of the Invention

[0003] In view of this, embodiments of this application provide a method for handling incoming calls, which can automatically determine whether to answer the call based on the driver's emotional state, thereby improving the driver's driving safety.

[0004] The technical solution of this application is implemented as follows:

[0005] This application provides a call processing method applied to an in-vehicle terminal device, including:

[0006] When a mobile communication device receives an incoming call, it acquires the driver's first video data and first voice data.

[0007] Based on the first video data, the first voice data, and a preset emotion classification model, the driver's target emotion category is determined, and the target emotion category represents the driver's emotional inclination towards the incoming call.

[0008] Whether to answer the call is determined based on the target emotion category.

[0009] This application provides a call processing device, including:

[0010] The acquisition module is used to acquire the driver's first video data and first voice data when the mobile communication device receives an incoming call;

[0011] The first determining module is used to determine the driver's target emotion category based on the first video data, the first voice data and a preset emotion classification model, wherein the target emotion category represents the driver's emotional orientation towards the incoming call;

[0012] The second determining module is used to determine whether to answer the incoming call based on the target emotion category.

[0013] This application provides a call processing device, including:

[0014] Memory, used to store executable incoming call processing instructions;

[0015] The processor, when executing executable incoming call processing instructions stored in the memory, implements the method provided in the embodiments of this application.

[0016] This application provides a computer-readable storage medium storing computer-executable instructions configured to perform the above-described incoming call processing steps.

[0017] This application provides a method, apparatus, device, and computer-readable storage medium for handling incoming calls. Using this technical solution, firstly, when a mobile communication device receives an incoming call, it acquires first video data and first voice data of the driver; then, based on the first video data, the first voice data, and a preset emotion classification model, it determines the driver's target emotion category; finally, it determines whether to answer the call based on the target emotion category. Thus, based on the driver's video and audio data acquired during the incoming call, and the preset emotion classification model, the driver's emotion category in response to the call can be determined, and the decision to answer the call can be automatically made based on this emotion category without manual operation, improving driving safety. Attached Figure Description

[0018] Figure 1 A flowchart illustrating an incoming call processing method provided in an embodiment of this application;

[0019] Figure 2 A flowchart illustrating a method for obtaining a driver's target emotion category, provided in an embodiment of this application;

[0020] Figure 3 A flowchart illustrating another method for obtaining the target emotion category of a driver, provided in an embodiment of this application;

[0021] Figure 4 A flowchart illustrating a phone call blocking method based on video and voice emotion recognition, provided for an embodiment of this application;

[0022] Figure 5 This is a schematic diagram of the composition structure of an incoming call processing device provided in an embodiment of this application;

[0023] Figure 6 This is a schematic diagram of the composition structure of a call processing device provided in an embodiment of this application. Detailed Implementation

[0024] The technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings.

[0025] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described below in conjunction with the accompanying drawings. The described embodiments should not be regarded as limitations on this application. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0026] In the following description, references to “some embodiments” or “other embodiments” describe a subset of all possible embodiments. However, it is understood that “some embodiments” or “other embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0027] In the following description, the terms "first" and "second" are used merely to distinguish similar objects and do not represent a specific ordering of objects. It is understood that "first" and "second" may be interchanged in a specific order or sequence where permitted, so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.

[0028] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0029] Based on the problems existing in related technologies, this application provides a method for handling incoming calls. This method can automatically determine whether to answer an incoming call based on the driver's emotional state, thereby improving the driver's driving safety. Figure 1 The diagram shown is a flowchart illustrating a call processing method provided in an embodiment of this application. The method includes the following steps:

[0030] S101. When the mobile communication device receives an incoming call, it acquires the driver's first video data and first voice data.

[0031] It should be noted that the mobile communication device can be a mobile phone, tablet computer, or other communication device. The mobile communication device can establish a communication connection with the vehicle terminal, for example, via Bluetooth. The first video data can be video of the driver captured by image acquisition devices (such as cameras or webcams), and the first audio data can be voice of the driver captured by audio acquisition devices (such as recorders).

[0032] In some embodiments, after a communication connection is established between the mobile communication device and the vehicle terminal, the mobile communication device can display an incoming call on the corresponding display screen of the vehicle terminal. Upon detecting that the mobile communication device has received an incoming call, the image acquisition device can be controlled to collect video data of the driver, and simultaneously the audio acquisition device can be controlled to collect the driver's voice data. In some embodiments, the first video data includes at least facial image data of the driver, such as eyes, eyebrows, and mouth.

[0033] S102. Based on the first video data, the first voice data, and the preset emotion classification model, determine the target emotion category of the driver.

[0034] In some embodiments, the preset emotion classification model can be a pre-established emotion model. For example, it can be an emotion model established based on the initial video data and initial voice data of the driver obtained after the driver gets into the vehicle. The preset emotion classification model includes the correspondence between initial facial emotion features, initial audio emotion features and emotion categories. The initial facial emotion features are obtained by feature extraction from the initial video data, and the initial audio emotion features are obtained by feature extraction from the initial audio emotion features.

[0035] In some embodiments, feature extraction can be performed on the first video data to obtain a first facial emotion feature, and feature extraction can be performed on the first voice data to obtain a first audio emotion feature. Then, the first facial emotion feature, the first audio emotion feature, and a preset emotion classification model are matched to determine the driver's target emotion category. In implementation, reference facial emotion features that satisfy a first preset condition and reference audio emotion features that satisfy a second preset condition in the preset emotion classification model can be identified first. Then, the emotion category corresponding to the reference facial emotion feature and the reference audio emotion feature is determined as the driver's target emotion category.

[0036] In other embodiments, besides matching the first facial emotion features extracted from the first video data and the first audio emotion features extracted from the first voice data with a preset emotion classification model, the first facial emotion features and the first audio emotion features can also be matched with other emotion models pre-stored in a database. During matching, the first facial emotion features are compared with the facial emotion features stored in the database, and the first audio emotion features are matched with the audio emotion features stored in the database. If a match is successful, the driver's target emotion category can be obtained. Unlike the preset emotion classification model, the emotion models stored in the database can be built based on the facial and audio emotion features of multiple individuals, but do not include the current driver's facial and audio emotion features.

[0037] In other embodiments, the preset emotion classification model (or other emotion models stored in a database) can also be a trained neural network model, such as a long short-term memory network model. This preset emotion classification model can perform emotion recognition on facial emotion features and audio emotion features to obtain the emotion category. The preset emotion classification model can be trained based on reference facial emotion features and reference audio emotion features, and obtained after training. The reference facial emotion features and reference audio emotion features can be obtained by feature extraction from the initial monitoring data obtained after the driver gets into the vehicle. By inputting the first facial emotion feature and the first audio emotion feature into the preset emotion classification model, the driver's target emotion category can be obtained.

[0038] In some embodiments, the target emotion category represents the driver's emotional orientation toward the incoming call, which can indicate whether the driver is willing to answer the call. The target emotion category can be confusion, stress, anger, etc. For example, if the target emotion category is confusion, it can indicate that the driver considers the incoming call to be a nuisance call and is unwilling to answer the call.

[0039] S103. Determine whether to answer the call based on the target emotion category.

[0040] In some embodiments, a driver's willingness to answer an incoming call can be determined based on the target emotion category. For example, when the target emotion category is happiness, excitement, etc., it means that the driver is willing to answer the call; when the target emotion category is stress, anger, etc., it means that the driver believes the call is intended to be a nuisance and is unwilling to answer the call.

[0041] In some embodiments, if it is determined based on the target emotion category that the driver does not want to answer the call, the call can be blocked via a Bluetooth calling application, the call forwarding function of the Bluetooth calling application can be enabled to block the call, or the call can be hidden from display on the in-vehicle terminal's screen. Blocking calls via a Bluetooth calling application can achieve automatic blocking without requiring manual operation by the driver, thus improving driving safety.

[0042] In this embodiment, firstly, when the mobile communication device receives an incoming call, it acquires the driver's first video data and first voice data; then, based on the first video data, the first voice data, and a preset emotion classification model, it determines the driver's target emotion category; finally, it determines whether to answer the call based on the target emotion category. Thus, by using the driver's video and audio data acquired during the incoming call, along with the preset emotion classification model, the driver's emotion category in response to the call can be determined, and the decision to answer the call can be automatically made based on this emotion category, without requiring manual operation, thereby improving driving safety.

[0043] In some embodiments of this application, the target emotion category of the driver is determined based on the first video data, the first voice data and the preset emotion classification model. That is, step S102 can be implemented by the following steps S1021 to S1022. Each step is described below.

[0044] S1021. Perform facial feature extraction on the first video data to obtain facial emotion features, and perform audio feature extraction on the first speech data to obtain audio emotion features.

[0045] In some embodiments, by segmenting the first video data into frames, image frames corresponding to the first video data can be obtained. Then, facial emotion features of the driver in the image frames can be extracted. Facial emotion features can be the states of the eyebrows, eyes, mouth, etc., such as frowning or raising the corners of the mouth. In practice, facial emotion features in the first video data can be extracted using principal component analysis, optical flow, or feature extraction methods based on deep learning models.

[0046] In some embodiments, audio emotion features can be voiceprint features. When extracting audio features from the first speech data to obtain audio emotion features, the first speech data can first be converted into speech frequency, and the tone, speech rate and other information of the speech frequency can be determined. Based on the tone, speech rate and other information, the corresponding voiceprint features can be determined. The voiceprint features can include calmness, impatience, coherence and so on.

[0047] S1022. Match and analyze facial and audio emotion features with a preset emotion classification model to obtain the driver's target emotion category.

[0048] In some embodiments, the preset emotion classification model includes initial facial emotion features, initial audio emotion features, emotion categories corresponding to the initial facial emotion features, emotion categories corresponding to the initial audio emotion features, and emotion categories corresponding to both the initial facial emotion features and the initial audio emotion features. By matching facial emotion features with the initial facial emotion features, the emotion category corresponding to the facial emotion features can be determined; by matching audio emotion features with the initial audio emotion features, the emotion category corresponding to the audio emotion features can be determined; by matching facial emotion features, audio emotion features, and the initial facial emotion features and initial audio emotion features, the emotion categories corresponding to the facial emotion features and audio emotion features can be determined.

[0049] In some embodiments, the emotion categories corresponding to facial emotion features and audio emotion features can be fused. For example, the emotion categories corresponding to facial emotion features and audio emotion features can be compared. When the corresponding emotion categories are the same, the driver's target emotion category is obtained. In other embodiments, facial emotion features and audio emotion features can be fused, and the driver's target emotion category can be determined based on the fused emotion features. In implementation, the emotion categories corresponding to the facial emotion features and audio emotion features can be determined, and this emotion category can be identified as the driver's target emotion category. The methods provided in the above two embodiments combine two emotion features, thereby making the determined emotion category more accurate.

[0050] Understandably, after obtaining facial and audio emotion features, the facial and audio emotion features are fused together, which combines the two emotion features; or, the emotion categories corresponding to the facial emotion features and the emotion categories corresponding to the audio emotion features are fused together, which combines the emotion recognition results of the two emotion features, making the determined target emotion category more accurate.

[0051] In some embodiments of this application, the preset emotion classification model includes a correspondence between reference facial emotion features, reference audio emotion features, and reference emotion categories. For example, if the reference facial emotion feature is an upturned mouth and the reference audio feature is sweet, then the corresponding reference emotion category is happy. Based on this, the facial emotion features and audio emotion features are matched and analyzed with the preset emotion classification model to obtain the driver's target emotion category. That is, step S1022 can be implemented through the following steps S201A to S202A, which will be described below. Figure 2 Let's take an example to explain each step.

[0052] S201A: Obtain candidate facial emotion features that satisfy the first preset condition from the reference facial emotion features, and obtain candidate audio emotion features that satisfy the second preset condition from the reference audio emotion features.

[0053] In some embodiments, the reference facial emotion feature, the reference audio emotion feature, and the reference emotion category each include at least one. The candidate facial emotion feature that satisfies the first preset condition with the facial emotion feature can be the same as the facial emotion feature among multiple reference facial emotion features, or it can be a reference facial emotion feature similar to the facial emotion feature among multiple reference facial emotion features. For example, if the facial emotion feature is an upturned corner of the mouth, then the candidate facial emotion feature is also an upturned corner of the mouth, but the arc of the upturned corner of the mouth is different for the two.

[0054] In some embodiments, a candidate audio emotion feature that satisfies the second preset condition with the audio emotion feature can be a reference speech emotion feature that is the same as the speech emotion feature among a plurality of reference speech emotion features, or it can be a reference facial emotion feature that is similar to the audio emotion feature among a plurality of reference audio emotion features. For example, if the audio emotion feature is a characteristic voiceprint, then the reference audio emotion feature can be a characteristic voiceprint that is similar to or similar to the characteristic voiceprint.

[0055] S202A: The reference emotion category corresponding to the candidate facial emotion feature and the candidate audio emotion feature is determined as the driver's target emotion category.

[0056] In some embodiments, after determining candidate facial emotion features that satisfy a first preset condition and candidate audio emotion features that satisfy a second preset condition, the reference emotion categories corresponding to the candidate facial emotion features and the candidate audio emotion features can be determined as the driver's target emotion category.

[0057] Understandably, by determining the reference emotion categories corresponding to candidate facial emotion features and candidate audio emotion features as the driver's target emotion category, i.e., the target emotion category is determined based on both facial emotion features and audio emotion features, the determined target emotion category can be more accurate.

[0058] In some embodiments of this application, the preset emotion classification model includes a correspondence between reference facial emotion features and reference emotion categories, as well as a correspondence between reference audio emotion features and reference emotion categories. For example, if the reference facial emotion feature is frowning, the corresponding reference emotion category could be confusion; if the reference audio emotion feature is irritability, the corresponding reference emotion category could be stress. Based on this, matching and analyzing the facial emotion features and audio emotion features with the preset emotion classification model to obtain the driver's target emotion category, i.e., step S1022, can also be implemented through the following steps S201B to S203B, which will be described below. Figure 3 Let's take an example to explain each step.

[0059] S201B: Obtain candidate facial emotion features from the reference facial emotion features that satisfy the first preset condition, and determine the reference emotion category corresponding to the candidate facial emotion features as the first emotion category.

[0060] In some embodiments, the first emotion category may be an emotion category corresponding to facial emotion features. By first determining candidate facial emotion features that satisfy a first preset condition, and then obtaining a reference emotion category corresponding to the candidate facial emotion features, the emotion features corresponding to the facial emotion features can be determined.

[0061] S202B: Obtain candidate audio emotion features from the reference audio emotion features that satisfy the first preset condition, and determine the reference emotion category corresponding to the candidate audio emotion features as the second emotion category.

[0062] In some embodiments, the second emotion category may be an emotion category corresponding to the audio emotion feature. By first determining candidate audio emotion features that satisfy the first preset condition and then obtaining the reference emotion category corresponding to the candidate audio emotion feature, the emotion feature corresponding to the audio emotion feature can be determined.

[0063] S203B: If the first emotion category and the second emotion category are the same, the first emotion category or the second emotion category shall be used as the driver's target emotion category.

[0064] In some embodiments, the first emotion category determined based on facial emotion features and the second emotion category determined based on audio emotion features may be the same or different. If the first emotion category and the second emotion category are the same, it means that the emotion categories determined based on facial emotion features and audio emotion features are consistent, and the first emotion category or the second emotion category can be determined as the driver's target emotion category. If the first emotion category and the second emotion category are different, the facial emotion features or audio emotion features can be updated, for example, the facial emotion features or audio emotion features can be extracted again until the emotion categories determined by the two are the same.

[0065] Understandably, by comparing the first emotion category determined by facial emotion features with the second emotion category determined by audio emotion features, and when the first and second emotion categories are the same, determining the first or second emotion category as the driver's target emotion category, the driver's emotional inclination towards the incoming call can be more accurately determined.

[0066] In some embodiments of this application, before "when the mobile communication device receives an incoming call, it acquires the driver's first video data and first voice data", i.e., step S101, the following steps S301 to S303 can also be performed. Each step is described below.

[0067] S301. After confirming that the driver has boarded the vehicle, obtain the driver's initial monitoring data.

[0068] In some embodiments, the initial monitoring data may be video data acquired through an image acquisition device or voice data acquired through an audio acquisition device. The initial monitoring data and the first video data and first audio data are acquired at different times. The initial monitoring data is acquired as soon as the driver gets into the vehicle, while the first video data and first audio data are acquired after receiving an incoming call from the driver.

[0069] S302. Extract features from the initial monitoring data to obtain at least one reference emotion feature, and determine at least one reference emotion category corresponding to each reference emotion feature.

[0070] In some embodiments, when the initial monitoring data is video data, the corresponding reference emotional features may be facial emotional features such as frowning or upturned corners of the mouth; when the initial monitoring data is audio data, the corresponding reference emotional features may be audio emotional features such as calmness or coherence. In some embodiments, the initial monitoring data may include both video data and audio data, in which case the corresponding reference emotional features include both facial emotional features and audio emotional features.

[0071] In some embodiments, after obtaining each reference emotion feature, the reference emotion category corresponding to the reference emotion feature can be determined. In some embodiments, a reference emotion category can be determined based on a reference emotion feature, which can be a reference facial emotion feature or a reference audio emotion feature; in other embodiments, a reference emotion category can also be determined based on multiple reference emotion features, which can all be reference facial emotion features, all be reference audio emotion features, or at least one reference facial emotion feature and at least one audio emotion feature.

[0072] In some embodiments, determining the reference emotion category corresponding to the reference emotion feature can be achieved by using an emotion recognition model such as a deep learning network (e.g., a convolutional neural network) to perform emotion recognition on the reference emotion feature. In implementation, a portion of the reference emotion features can be used as a training set to train the emotion recognition model. After training, the trained emotion recognition model can be used to identify the reference emotion feature and obtain the reference emotion category corresponding to the reference emotion feature.

[0073] For example, if the reference emotional features are a frown and a closed mouth, the corresponding reference emotional category could be stress; if the reference emotional features are a frown and an open mouth, the corresponding reference emotional category could be confusion; if the reference emotional features are a smile and a sweet voice, the corresponding reference emotional category could be happiness. The reference emotional features and corresponding reference emotional categories described here are merely illustrative and are not intended to limit the scope of this application.

[0074] S303. Based on each reference emotion feature and each reference emotion category, establish a preset emotion classification model.

[0075] In some embodiments, one or more reference emotion features and reference emotion categories have a corresponding relationship. Therefore, after obtaining each reference emotion feature and the reference emotion category corresponding to each reference emotion feature, a preset model can be established based on the correspondence between the reference emotion features and the reference emotion categories.

[0076] In some embodiments of this application, the reference emotion features include reference facial emotion features and reference voice emotion features. Based on this, a preset emotion classification model is established based on each reference emotion feature and each reference emotion category. That is, step S303 can be implemented by the following steps S3031 to S3032. Each step is described below.

[0077] S3031. Determine at least one reference emotion category corresponding to each reference facial emotion feature and each reference audio emotion feature.

[0078] In some embodiments, a corresponding reference emotion category can be determined based on reference facial emotion features combined with reference audio emotion features. Multiple reference emotion categories can be obtained by performing emotion recognition on each reference facial emotion feature and each reference audio emotion feature. In practice, emotion recognition models can be used, such as a trained neural network model, to determine the reference emotion categories corresponding to the reference facial emotion features and reference audio emotion features. By inputting the reference facial emotion features and reference audio emotion recognition features into the trained neural network model, the reference emotion categories can be obtained.

[0079] In some embodiments, determining at least one reference emotion category corresponding to each reference facial emotion feature and each reference voice emotion feature may include four cases: one reference facial emotion feature and one reference audio emotion feature correspond to one reference emotion category; multiple reference facial emotion features and one reference audio emotion feature correspond to one reference emotion category; one reference facial emotion feature and multiple reference audio emotion features correspond to one reference emotion category; multiple reference facial emotion features and multiple reference audio emotion features correspond to one reference emotion category.

[0080] S3032. Establish a preset emotion classification model based on each reference facial emotion feature, each reference audio emotion feature, and at least one reference emotion category corresponding to each reference facial emotion feature and each reference audio emotion feature.

[0081] In some embodiments, after obtaining each reference facial emotion feature, each reference audio emotion feature, and each reference emotion category, a preset emotion classification model can be established based on the reference facial emotion features, reference audio emotion features, and the corresponding reference emotion categories.

[0082] It is understandable that by establishing a preset emotion classification model based on the obtained reference facial emotion features, reference audio emotion features, and reference emotion categories corresponding to each reference facial emotion feature and each reference audio emotion feature, it is possible to match the obtained driver's emotion features with the preset model after receiving an incoming call, thereby obtaining the driver's emotion recognition result for the incoming call more quickly and accurately.

[0083] In this embodiment, firstly, when the mobile communication device receives an incoming call, it acquires the driver's first video data and first voice data; then, based on the first video data, the first voice data, and a preset emotion classification model, it determines the driver's target emotion category; finally, it determines whether to answer the call based on the target emotion category. Thus, by using the driver's video and audio data acquired during the incoming call, along with the preset emotion classification model, the driver's emotion category in response to the call can be determined, and the decision to answer the call can be automatically made based on this emotion category, without requiring manual operation, thereby improving driving safety.

[0084] The implementation process of the application embodiments in practical application scenarios is described below.

[0085] In some embodiments, such as Figure 4 The diagram shown is a flowchart of a telephone blocking method based on video and voice emotion recognition provided in an embodiment of this application. The method can be implemented through the following steps S401 to S406, and each step is described below.

[0086] S401. After recognizing that the main driver (equivalent to "driver" in other embodiments) has entered the vehicle, the information collection device is used to obtain the main driver's video information and audio information (equivalent to "initial monitoring data" in other embodiments). Feature extraction is performed on the video information and audio information respectively to obtain video features and audio features (equivalent to "reference facial emotion features and reference audio emotion features" in other embodiments).

[0087] In some embodiments, the driver can be the driver of the vehicle, and the identity of the driver can be identified by the Driver Monitor System (DMS) camera to determine whether the driver has boarded the vehicle. The information acquisition device can include image acquisition devices and audio acquisition devices; the image acquisition device can be a DMS camera, etc., and the audio acquisition device can be a recorder, etc. By extracting features from the video information, the driver's facial features can be obtained, and by extracting features from the audio information, the driver's audio features can be obtained.

[0088] In some embodiments, once it is determined that the driver has boarded the vehicle, the driver can be continuously monitored. The video and audio information of the driver can be continuously acquired through information acquisition devices, thereby obtaining multiple video and audio features corresponding to the driver.

[0089] S402. Based on the correlation between video features and audio features, emotion recognition is performed on the video features and audio features respectively to obtain emotion recognition results. The emotion recognition results are then fused to obtain the emotion classification of the driver user (equivalent to "emotion category" in other embodiments).

[0090] In some embodiments, correlation analysis can be performed on video features and audio features obtained at the same time or within the same time period. For example, the emotion recognition results identified from video features can be compared with those identified from audio features. When the emotion recognition results determined by both are the same, the emotion recognition result can be determined as the emotion category of the driver. In practice, trained neural networks and other deep learning models can be used to perform emotion recognition on video features and audio features, thereby obtaining the emotion recognition results corresponding to each video feature and audio feature.

[0091] In some embodiments, fusing the emotion recognition results corresponding to video features and the emotion recognition results corresponding to audio features can be achieved by analyzing the similarity between the two emotion recognition results. When the two are completely identical, the emotion classification of the driver can be obtained (equivalent to "if the first emotion category and the second emotion category are the same, the first emotion category or the second emotion category is taken as the driver's target emotion category" in other embodiments).

[0092] In other embodiments, after obtaining audio and video features, the audio and video features can be modified separately to obtain modified audio features and modified video features. Then, the modified audio and video features are fused for emotion recognition to obtain the driver's emotion classification. Furthermore, text with clear emotional inclinations can be extracted from the audio information to obtain text features. Combining these text features with the corresponding emotional inclinations yields more accurate emotion classification results.

[0093] Understandably, by fusing the emotion recognition results corresponding to video features and the emotion recognition results corresponding to audio features, that is, by combining the video and audio features of the driver when identifying the driver's emotions, the determined emotion classification of the driver can be more accurate.

[0094] S403. Establish a reference emotion classification model based on video features, audio features, and emotion classification.

[0095] In some embodiments, since the driver is continuously monitored after uploading, multiple video features and multiple audio features can be obtained. Then, a reference emotion classification model can be established based on the video features, audio features, and emotion classifications. This reference emotion classification model includes the correspondence between video features, audio features, and emotion classifications. Because the emotions exhibited by the driver may differ at different times or in different time periods, the reference emotion classification model may include multiple sets of correspondences between video features, audio features, and emotion classifications.

[0096] In some embodiments, the emotion recognition processor provided by the vehicle-mounted continuous damping control (CDC) can process any step of the above-described emotion recognition method.

[0097] S404. When an incoming call is received, the first video information and the first audio information of the driver user are obtained (equivalent to "first video data and first audio data" in other embodiments). Feature extraction is performed on the first video information and the first audio information respectively to obtain the first video feature and the first audio feature (equivalent to "facial emotion feature and audio emotion feature" in other embodiments).

[0098] It should be noted that the first video information and the first audio information are monitoring data of the driver user obtained after receiving the incoming call. By extracting features from the first video information and the first audio information, the emotional characteristics of the driver user can be obtained, namely the first video features and the first audio features.

[0099] In some embodiments, the first video feature and the first audio feature can reflect the driver's intention to answer the call. The first video feature can be the driver's eyebrows, eyes, mouth, etc., and the first audio feature can be the driver's voiceprint.

[0100] S405. Match and analyze the first video features and the first audio features with the reference emotion classification model to obtain the target emotion classification of the driver user (equivalent to the "target emotion category" in other embodiments).

[0101] In some embodiments, the reference emotion classification model includes a correspondence between video features, audio features, and the reference emotion model. Therefore, the first video features are matched with other video features, and video features that are the same as or similar to the first video are used as reference video features; the first audio features are matched with other audio features, and audio features that are the same as or similar to the first audio features are used as reference audio features. The reference emotion classifications corresponding to the reference video features and the reference audio features are determined to be the target emotion classification for the driver.

[0102] S406. When the target emotion category is the preset emotion category, determine that the incoming call is a nuisance call and block the incoming call.

[0103] In some embodiments, the preset emotion category can be stress, confusion, etc. When the driver's emotion is stress, confusion, etc., it means that the driver considers the incoming call to be a nuisance call. At this time, the incoming call can be blocked. In implementation, the call forwarding function of the Bluetooth phone APP can be called to request the blocking of the incoming call.

[0104] It is understood that, in this embodiment of the application, video and audio features corresponding to the driver are continuously acquired after the driver gets into the vehicle; emotion recognition is performed separately based on the video and audio features, and the emotion recognition results of the video and audio features are fused to obtain an emotion classification; a reference emotion classification model is established based on the video features, audio features, and emotion classification; by matching the first video features and first audio features obtained when receiving an incoming call with the reference emotion classification model, the driver's target emotion classification for the incoming call can be accurately and quickly determined, and the incoming call can be identified as a nuisance call based on the target emotion classification, thus seamlessly blocking the incoming call without requiring manual operation by the driver, thereby improving driving safety.

[0105] This application also provides a call processing device. Figure 5 This is a schematic diagram of the composition structure of a call processing device provided in an embodiment of this application, as shown below. Figure 5 As shown, the incoming call processing device 500 includes:

[0106] The acquisition module 501 is used to acquire the driver's first video data and first voice data when the mobile communication device receives an incoming call;

[0107] The first determining module 502 is used to determine the target emotion category of the driver based on the first video data, the first voice data and a preset emotion classification model, wherein the target emotion category represents the driver's emotional orientation towards the incoming call;

[0108] The second determining module 503 is used to determine whether to answer the incoming call based on the target emotion category.

[0109] In some embodiments, the first determining module 502 includes:

[0110] The feature extraction submodule is used to extract facial features from the first video data to obtain facial emotion features, and to extract audio features from the first speech data to obtain audio emotion features.

[0111] The matching submodule is used to perform matching analysis between the facial emotion features and the audio emotion features and the preset emotion classification model to obtain the target emotion category of the driver.

[0112] In some embodiments, the preset emotion classification model includes a correspondence between reference facial emotion features, reference audio emotion features, and reference emotion categories; the matching submodule includes:

[0113] The first acquisition unit is used to acquire candidate facial emotion features that satisfy a first preset condition from the reference facial emotion features, and to acquire candidate audio emotion features that satisfy a second preset condition from the reference audio emotion features.

[0114] The first determining unit is used to determine the reference emotion category corresponding to the candidate facial emotion feature and the candidate audio emotion feature as the target emotion category of the driver.

[0115] In some embodiments, the preset emotion classification model includes a correspondence between reference facial emotion features and reference emotion categories, and a correspondence between reference audio emotion features and reference emotion categories; the matching submodule further includes:

[0116] The second determining unit is used to obtain candidate facial emotion features that satisfy the first preset condition from the reference facial emotion features, and determine the reference emotion category corresponding to the candidate facial emotion features as the first emotion category.

[0117] The third determining unit is used to obtain candidate audio emotion features that satisfy the second preset condition from the reference audio emotion features, and to determine the reference emotion category corresponding to the candidate audio emotion features as the second emotion category.

[0118] The fourth determining unit is configured to, if the first emotion category and the second emotion category are the same, use the first emotion category or the second emotion category as the target emotion category of the driver.

[0119] In some embodiments, the call processing device 500 further includes:

[0120] The second acquisition module is used to acquire the initial monitoring data of the driver after the driver gets into the vehicle;

[0121] The third determining module is used to extract features from the initial monitoring data, obtain at least one reference emotion feature, and determine at least one reference emotion category corresponding to each reference emotion feature.

[0122] A module is created to establish the preset emotion classification model based on the various reference emotion features and various reference emotion categories.

[0123] In some embodiments, the reference emotion features include reference facial emotion features and reference voice emotion features; the creation module includes:

[0124] A determination submodule is used to determine at least one reference emotion category corresponding to each of the reference facial emotion features and each of the reference audio emotion features;

[0125] A submodule is created to establish the preset emotion classification model based on the various reference facial emotion features, the various reference audio emotion features, and at least one reference emotion category corresponding to the various reference facial emotion features and the various reference audio emotion features.

[0126] In some embodiments, the call processing device 500 further includes:

[0127] A blocking module is used to block incoming calls via a Bluetooth calling application after determining that the call will not be answered based on the target emotion category.

[0128] It should be noted that the description of the call processing device in this application is similar to the description of the method embodiment described above, and has similar beneficial effects as the method embodiment; therefore, it will not be repeated. For technical details not disclosed in this device embodiment, please refer to the description of the method embodiment of this application for understanding.

[0129] It should be noted that, in the embodiments of this application, if the above-described call processing method is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiments of this application, or the part that contributes to related solutions, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), magnetic disks, or optical disks. Thus, the embodiments of this application are not limited to any specific hardware and software combination.

[0130] Accordingly, embodiments of this application provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the incoming call processing method provided in the above embodiments.

[0131] This application also provides a call processing device. Figure 6 This is a schematic diagram of the composition structure of a call processing device provided in an embodiment of this application, as shown below. Figure 6 As shown, the incoming call processing device 600 includes: a memory 601, a processor 602, a communication interface 603, and a communication bus 604. The memory 601 stores executable incoming call processing instructions; the processor 602 executes the executable incoming call processing instructions stored in the memory to implement the incoming call processing method provided in the above embodiment.

[0132] The descriptions of the call processing device and storage medium embodiments above are similar to those of the method embodiments above, and have similar beneficial effects. For technical details not disclosed in the call processing device and storage medium embodiments of this application, please refer to the descriptions of the method embodiments of this application for understanding.

[0133] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising at least one…" does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0134] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods, such as: multiple units or components can be combined, or integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the various components shown or discussed can be through some interfaces, and the indirect coupling or communication connection between devices or units can be electrical, mechanical, or other forms.

[0135] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units. They may be located in one place or distributed across multiple network units. Some or all of the units may be selected to achieve the purpose of this embodiment according to actual needs.

[0136] In addition, each functional unit in the various embodiments of this application can be integrated into one processing unit, or each unit can be a separate unit, or two or more units can be integrated into one unit; the integrated unit can be implemented in hardware or in the form of hardware plus software functional units.

[0137] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, ROMs, magnetic disks, or optical disks.

[0138] Alternatively, if the integrated units described above are implemented as software functional modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the embodiments of this application, or the parts that contribute to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a product to execute all or part of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, ROMs, magnetic disks, or optical disks.

[0139] The above description is merely an embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for processing an incoming call, applied to a vehicle terminal device, characterized in that, The method comprises the steps of: When a mobile communication device receives an incoming call, first video data and first voice data of a driver are acquired; Based on a preset emotion classification model, facial feature extraction is performed on the first video data to obtain facial emotion features, and audio feature extraction is performed on the first voice data to obtain audio emotion features; the preset emotion classification model comprises a corresponding relationship between reference facial emotion features and reference emotion categories, and a corresponding relationship between reference audio emotion features and reference emotion categories; From the reference facial emotion features, a candidate facial emotion feature that satisfies a first preset condition with the facial emotion features is acquired, and a reference emotion category corresponding to the candidate facial emotion feature is determined as a first emotion category; From the reference audio emotion features, a candidate audio emotion feature that satisfies a second preset condition with the audio emotion features is acquired, and a reference emotion category corresponding to the candidate audio emotion feature is determined as a second emotion category; If the first emotion category and the second emotion category are the same, the first emotion category or the second emotion category is taken as a target emotion category of the driver, and the target emotion category represents an emotion orientation of the driver to the incoming call; Based on the target emotion category, it is determined whether to answer the incoming call.

2. The method of claim 1, wherein, The preset emotion classification model comprises a corresponding relationship between reference facial emotion features, reference audio emotion features and reference emotion categories; The matching analysis of the facial emotion features and the audio emotion features with the preset emotion classification model to obtain the target emotion category of the driver comprises the steps of: From the reference facial emotion features, a candidate facial emotion feature that satisfies a first preset condition with the facial emotion features is acquired, and from the reference audio emotion features, a candidate audio emotion feature that satisfies a second preset condition with the audio emotion features is acquired; The reference emotion category corresponding to the candidate facial emotion feature and the candidate audio emotion feature is determined as the target emotion category of the driver.

3. The method according to any one of claims 1 to 2, characterized in that, Further comprising: After determining that the driver gets on the vehicle, initial monitoring data of the driver is acquired; Feature extraction is performed on the initial monitoring data to obtain at least one reference emotion feature, and at least one reference emotion category corresponding to each reference emotion feature is determined; Based on the each reference emotion feature and each reference emotion category, the preset emotion classification model is established.

4. The method of claim 3, wherein, The reference emotion features comprise reference facial emotion features and reference voice emotion features; Based on the each reference emotion feature and each reference emotion category, the preset emotion classification model is established, which comprises the steps of: At least one reference emotion category corresponding to the each reference facial emotion feature and the each reference audio emotion feature is determined; Based on the each reference facial emotion feature, the each reference audio emotion feature, and at least one reference emotion category corresponding to the each reference facial emotion feature and the each reference audio emotion feature, the preset emotion classification model is established.

5. The method according to any one of claims 1 to 2, characterized in that, Further comprising: After determining not to answer the incoming call based on the target emotion category, the incoming call is shielded through a Bluetooth call application.

6. An incoming call processing apparatus, comprising: a first obtaining module configured to obtain first video data and first voice data of a driver when a mobile communication device receives an incoming call; a feature extraction submodule configured to perform facial feature extraction on the first video data based on a preset emotion classification model to obtain facial emotion features, and perform audio feature extraction on the first voice data to obtain audio emotion features, wherein the preset emotion classification model comprises a corresponding relationship between reference facial emotion features and reference emotion categories, and a corresponding relationship between reference audio emotion features and reference emotion categories; a second determination unit configured to obtain a candidate facial emotion feature from the reference facial emotion features that satisfies a first preset condition with the facial emotion features, and determine a reference emotion category corresponding to the candidate facial emotion feature as a first emotion category; a third determination unit configured to obtain a candidate audio emotion feature from the reference audio emotion features that satisfies a second preset condition with the audio emotion features, and determine a reference emotion category corresponding to the candidate audio emotion feature as a second emotion category; a fourth determination unit configured to determine a target emotion category of the driver as the first emotion category or the second emotion category if the first emotion category and the second emotion category are the same, wherein the target emotion category represents an emotion orientation of the driver to the incoming call; a second determination module configured to determine whether to answer the incoming call based on the target emotion category.

7. An incoming call processing device, comprising: a memory configured to store executable incoming call processing instructions; a processor configured to execute the executable incoming call processing instructions stored in the memory to implement the method of any one of claims 1 to 5.

8. A computer readable storage medium storing incoming call processing instructions configured to cause a processor to implement the method of any one of claims 1 to 5 when executed.

Images