Audio conversion method and device, computer device and storage medium

Abstract

The application relates to a conversion method and device of audio, a computer device, a storage medium and a computer program product, and relates to the technical field of artificial intelligence. The method comprises the following steps: acquiring an audio feature and a face image feature of a target user when the target user operates a business page of a business device; inputting the audio feature and the face image feature into an emotion recognition model to obtain a target emotion recognition result corresponding to the target user; determining a target sound conversion strategy matched with the target emotion recognition result from each preset sound conversion strategy; and determining to-be-output audio of the business device according to the target sound conversion strategy. The scheme can make the to-be-output audio match real-time emotions of the target user, thereby improving the interaction effect.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to an audio conversion method, apparatus, computer device, and storage medium. Background Technology

[0002] Currently, to facilitate user understanding of business operations, when users interact with the business page of a business device, the device uses text on the page and transaction audio to facilitate human-computer interaction. The business device pre-stores audio to be output. Upon responding to a user's business operation request, the business device outputs the transaction audio based on the pre-stored audio.

[0003] However, the audio parameters (such as pitch, volume, and timbre) of the pre-stored audio to be output are fixed, resulting in relatively simple transaction audio output based on the audio to be output and poor interactive effects. Summary of the Invention

[0004] Therefore, it is necessary to provide an audio conversion method, apparatus, computer device, computer-readable storage medium, and computer program product that can match the real-time emotions of the target user to the output audio and improve the interactive effect, in order to address the above-mentioned technical problems.

[0005] Firstly, this application provides an audio conversion method. The method includes:

[0006] Acquire audio and facial image features of the target user when operating the business page of the business device;

[0007] The audio features and the facial image features are input into the emotion recognition model to obtain the target emotion recognition result corresponding to the target user;

[0008] Among the preset voice conversion strategies, a target voice conversion strategy that matches the target emotion recognition result is determined;

[0009] Based on the target sound conversion strategy, the audio to be output by the service device is determined.

[0010] In one embodiment, the emotion recognition model includes a face image recognition sub-model, an audio recognition sub-model, and a fusion recognition sub-model; the step of inputting the audio features and the face image features into the emotion recognition model to obtain the target emotion recognition result corresponding to the target user includes:

[0011] The audio features are input into the audio recognition sub-model to obtain the audio emotion recognition result;

[0012] The facial image features are input into the facial image recognition sub-model to obtain the image emotion recognition result;

[0013] The audio emotion recognition result and the image emotion recognition result are input into the fusion recognition sub-model to obtain the target emotion recognition result.

[0014] In one embodiment, the face image recognition sub-model includes a residual network, multiple first dual attention mechanisms, a first long short-term memory network, a first splicing layer, and a first emotion recognition network; the step of inputting the face image features into the face image recognition sub-model to obtain the image emotion recognition result includes:

[0015] The facial image features are input into the residual network, which contains multiple residual subnetworks, to obtain the image semantic features output by each residual subnetwork and the target image semantic features output by the residual network; wherein, the output of the residual subnetwork except the last residual subnetwork is the input of the next residual subnetwork.

[0016] For each of the image semantic features, the image semantic features are input into the first dual attention mechanism to obtain the image weighted features corresponding to the image semantic features;

[0017] Each of the image weighted features is input into the corresponding input neurons in the first long short-term memory network to obtain image context features;

[0018] The image context features and the target image semantic features are input into the first stitching layer to obtain image fusion features;

[0019] The image fusion features are input into the first emotion recognition network to obtain the image emotion recognition result.

[0020] In one embodiment, the audio recognition sub-model includes a second dual attention mechanism, a second long short-term memory network, and a second emotion recognition network; the step of inputting the audio features into the audio recognition sub-model to obtain the audio emotion recognition result includes:

[0021] The audio features are input into the second dual attention mechanism to obtain the audio weighted features;

[0022] The audio weighted features are input into the second long short-term memory network to obtain audio context features;

[0023] The audio context features are input into the second emotion recognition network to obtain the audio emotion recognition result.

[0024] In one embodiment, obtaining the audio and facial image features of the target user when operating the business page of the business device includes:

[0025] When the triggering conditions for feature extraction are met, the audio features and facial image features of the target user when operating the business page of the business device are obtained.

[0026] In one embodiment, obtaining the audio features and facial image features of the target user operating the business page of the business device when the feature extraction trigger condition is met includes:

[0027] Obtain the number of times the audio to be output from the business device has been determined;

[0028] If the determined number of times falls within a first preset range, acquire the audio features and facial image features of the target user when operating the business page of the business device;

[0029] When the determined number of times falls within a preset second interval and the current target voice conversion strategy is different from the previous target voice conversion strategy, the audio features and facial image features of the target user when operating the business page of the business device are obtained; wherein, the upper limit of the first preset interval is less than the lower limit of the second preset interval.

[0030] In one embodiment, obtaining the audio and facial image features of the target user when operating the business page of the business device includes:

[0031] Acquire audio data and facial images of the target user when operating the business page of the business device;

[0032] The audio features are obtained by determining the Mel-frequency cepstral coefficients of the audio data;

[0033] According to a preset segmentation strategy, the face image is segmented to obtain each face sub-image, and according to a preset scaling strategy, the face sub-images are scaled to obtain face image features.

[0034] In one embodiment, the target emotion recognition result includes a positive emotion probability and a negative emotion probability; determining the target voice conversion strategy that matches the target emotion recognition result among various preset voice conversion strategies includes:

[0035] Calculate the difference between the probability of positive emotion and the probability of negative emotion;

[0036] If the difference is greater than or equal to a preset difference threshold, and if the probability of negative emotion is greater than the probability of positive emotion, then the voice conversion strategy corresponding to severe negative emotion will be used as the target voice conversion strategy.

[0037] If the difference is less than the preset difference threshold, a target voice conversion strategy that matches the positive and negative emotion probabilities contained in the target emotion recognition result is determined based on the mapping relationship between the negative emotion probability, the positive emotion probability, and the voice conversion strategy.

[0038] In one embodiment, the method further includes:

[0039] If the difference is greater than or equal to a preset difference threshold, and the probability of positive emotion is greater than the probability of negative emotion, then the current voice conversion strategy remains unchanged.

[0040] Secondly, this application also provides an audio conversion apparatus. The apparatus includes:

[0041] The first acquisition module is used to acquire the audio features and facial image features of the target user when operating the business page of the business device;

[0042] The first determining module is used to input the audio features and the facial image features into the emotion recognition model to obtain the target emotion recognition result corresponding to the target user;

[0043] The second determining module is used to determine, among the preset voice conversion strategies, the target voice conversion strategy that matches the target emotion recognition result;

[0044] The third determining module is used to determine the audio to be output by the service device based on the target sound conversion strategy.

[0045] In one embodiment, the emotion recognition model includes a face image recognition sub-model, an audio recognition sub-model, and a fusion recognition sub-model; the first determining module is specifically used for:

[0046] The audio features are input into the audio recognition sub-model to obtain the audio emotion recognition result;

[0047] The facial image features are input into the facial image recognition sub-model to obtain the image emotion recognition result;

[0048] The audio emotion recognition result and the image emotion recognition result are input into the fusion recognition sub-model to obtain the target emotion recognition result.

[0049] In one embodiment, the face image recognition sub-model includes a residual network, multiple first dual attention mechanisms, a first long short-term memory network, a first splicing layer, and a first emotion recognition network; the first determining module is specifically used for:

[0050] The facial image features are input into the residual network, which contains multiple residual subnetworks, to obtain the image semantic features output by each residual subnetwork and the target image semantic features output by the residual network; wherein, the output of the residual subnetwork except the last residual subnetwork is the input of the next residual subnetwork.

[0051] For each of the image semantic features, the image semantic features are input into the first dual attention mechanism to obtain the image weighted features corresponding to the image semantic features;

[0052] Each of the image weighted features is input into the corresponding input neurons in the first long short-term memory network to obtain image context features;

[0053] The image context features and the target image semantic features are input into the first stitching layer to obtain image fusion features;

[0054] The image fusion features are input into the first emotion recognition network to obtain the image emotion recognition result.

[0055] In one embodiment, the audio recognition sub-model includes a second dual attention mechanism, a second long short-term memory network, and a second emotion recognition network; the first determining module is specifically used for:

[0056] The audio features are input into the second dual attention mechanism to obtain the audio weighted features;

[0057] The audio weighted features are input into the second long short-term memory network to obtain audio context features;

[0058] The audio context features are input into the second emotion recognition network to obtain the audio emotion recognition result.

[0059] In one embodiment, the first acquisition module is specifically used for:

[0060] When the triggering conditions for feature extraction are met, the audio features and facial image features of the target user when operating the business page of the business device are obtained.

[0061] In one embodiment, the first acquisition module is specifically used for:

[0062] Obtain the number of times the audio to be output from the business device has been determined;

[0063] If the determined number of times falls within a first preset range, acquire the audio features and facial image features of the target user when operating the business page of the business device;

[0064] When the determined number of times falls within a preset second interval and the current target voice conversion strategy is different from the previous target voice conversion strategy, the audio features and facial image features of the target user when operating the business page of the business device are obtained; wherein, the upper limit of the first preset interval is less than the lower limit of the second preset interval.

[0065] In one embodiment, the first acquisition module is specifically used for:

[0066] Acquire audio data and facial images of the target user when operating the business page of the business device;

[0067] The audio features are obtained by determining the Mel-frequency cepstral coefficients of the audio data;

[0068] According to a preset segmentation strategy, the face image is segmented to obtain each face sub-image, and according to a preset scaling strategy, the face sub-images are scaled to obtain face image features.

[0069] In one embodiment, the target emotion recognition result includes a positive emotion probability and a negative emotion probability; the second determining module is specifically used for:

[0070] Calculate the difference between the probability of positive emotion and the probability of negative emotion;

[0071] If the difference is greater than or equal to a preset difference threshold, and if the probability of negative emotion is greater than the probability of positive emotion, then the voice conversion strategy corresponding to severe negative emotion will be used as the target voice conversion strategy.

[0072] If the difference is less than the preset difference threshold, a target voice conversion strategy that matches the positive and negative emotion probabilities contained in the target emotion recognition result is determined based on the mapping relationship between the negative emotion probability, the positive emotion probability, and the voice conversion strategy.

[0073] In one embodiment, the second determining module is further configured to:

[0074] If the difference is greater than or equal to a preset difference threshold, and the probability of positive emotion is greater than the probability of negative emotion, then the current voice conversion strategy remains unchanged.

[0075] Thirdly, this application also provides a computer device. The computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the steps described in the first aspect.

[0076] Fourthly, this application also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program thereon, which, when executed by a processor, implements the steps described in the first aspect.

[0077] Fifthly, this application also provides a computer program product. The computer program product includes a computer program that, when executed by a processor, performs the steps described in the first aspect.

[0078] The aforementioned audio conversion method, apparatus, computer equipment, storage medium, and computer program product acquire audio features and facial image features of a target user operating a business page on a business device; input the audio features and facial image features into an emotion recognition model to obtain a target emotion recognition result corresponding to the target user; determine a target voice conversion strategy that matches the target emotion recognition result from various preset voice conversion strategies; and determine the audio to be output by the business device based on the target voice conversion strategy. In this scheme, the target voice conversion strategy is determined based on audio features, facial image features, an emotion recognition model, and various preset voice conversion strategies, and the audio to be output by the business device is determined based on this target voice conversion strategy. In other words, this scheme can convert the audio to be output by the business device in real time based on the audio features and facial image features of the target user operating the business page of the business device. Therefore, the audio to be output matches the real-time emotion of the target user, resulting in good interactive effects. Attached Figure Description

[0079] Figure 1 This is a flowchart illustrating an audio conversion method in one embodiment;

[0080] Figure 2 This is a schematic diagram of the structure of an emotion recognition model in one embodiment;

[0081] Figure 3 This is a flowchart illustrating a method for determining image emotion recognition results in one embodiment;

[0082] Figure 4 This is a schematic diagram of the structure of a face image recognition sub-model in one embodiment;

[0083] Figure 5 This is a schematic diagram of the residual block in one embodiment;

[0084] Figure 6 This is a schematic diagram of the structure of an audio recognition sub-model in one embodiment;

[0085] Figure 7 This is a flowchart illustrating a method for determining a target sound conversion strategy in one embodiment;

[0086] Figure 8This is a flowchart illustrating the audio conversion method in another embodiment;

[0087] Figure 9 This is a structural block diagram of an audio conversion device in one embodiment;

[0088] Figure 10 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0089] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0090] In one embodiment, such as Figure 1 As shown, an audio conversion method is provided. This embodiment illustrates the application of this method to a terminal. It is understood that this method can also be applied to a server, and further to a system including both a terminal and a server, and is implemented through interaction between the terminal and the server. In this embodiment, the method includes the following steps:

[0091] Step 102: Obtain the audio features and facial image features of the target user when operating the business page of the business device.

[0092] In this embodiment, the terminal acquires audio data and facial images of a target user operating a business page on a business device. The audio data is the audio generated when the target user interacts with the business device in the time domain. The terminal preprocesses the audio data according to a preset audio preprocessing strategy to obtain audio features. In one embodiment, the audio features are Mel-Frequency Cepstral Coefficients (MFCCs) of the audio data. The terminal preprocesses the facial image according to a preset image preprocessing strategy to obtain facial image features.

[0093] Step 104: Input the audio features and facial image features into the emotion recognition model to obtain the target emotion recognition result corresponding to the target user.

[0094] The emotion recognition model includes a face image recognition sub-model, an audio recognition sub-model, and a fusion recognition sub-model. The face image recognition sub-model includes a residual network, multiple first dual attention mechanisms, a first long short-term memory network, a first splicing layer, and a first emotion recognition network. The audio recognition sub-model includes a second dual attention mechanism, a second long short-term memory network, and a second emotion recognition network.

[0095] In the embodiments of this application, such as Figure 2 As shown, the terminal inputs audio features into the audio recognition sub-model of the emotion recognition model and facial image features into the facial image recognition sub-model of the emotion recognition model to obtain the target emotion recognition result corresponding to the target user output by the emotion recognition model. In one embodiment, the target emotion recognition result includes positive emotion probability and negative emotion probability. The positive emotion probability ranges from [0,1], and the negative emotion probability is [0,1].

[0096] Step 106: Among the preset voice conversion strategies, determine the target voice conversion strategy that matches the target emotion recognition result.

[0097] In this embodiment, the terminal determines a target voice conversion strategy that matches the target emotion recognition result from among various preset voice conversion strategies. Optionally, the voice conversion strategy can be a corpus dataset or an audio parameter adjustment strategy. In one embodiment, the terminal matches the target voice conversion strategy corresponding to the target emotion recognition result based on the target emotion recognition result and a preset first correspondence between emotion recognition results and voice conversion strategies. In one embodiment, the first correspondence can be stored in list form, such as a first correspondence table. The first correspondence table includes multiple key-value pairs, each consisting of an emotion recognition result and the voice conversion strategy corresponding to that emotion recognition result. Specifically, the terminal searches the first correspondence table based on the target emotion recognition result to obtain key-value pairs containing the target emotion recognition result, and uses the voice conversion strategy in that key-value pair as the target voice conversion strategy corresponding to the target emotion recognition result. In one embodiment, the terminal determines the target voice conversion strategy that matches the target emotion recognition result based on the target emotion recognition result and a preset first mapping relationship between emotion recognition results and voice conversion strategies. Specifically, the terminal calculates the difference between the probabilities of positive and negative emotions in the target emotion recognition result, and retrieves the second mapping relationship corresponding to the difference based on this difference and a preset correspondence between the difference and a second mapping relationship. The second mapping relationship characterizes the mapping relationship between the probability of negative emotion, the probability of positive emotion, and the voice conversion strategy. Based on the probability of positive and negative emotions in the target emotion recognition result and the second mapping relationship corresponding to the difference, the terminal maps a target voice conversion strategy that matches the target emotion recognition result.

[0098] Step 108: Determine the audio to be output by the service device according to the target audio conversion strategy.

[0099] In this embodiment, the terminal determines the audio to be output by the service device according to the target audio conversion strategy. When the target audio conversion strategy is a target corpus dataset, the terminal uses the corpus data from the target corpus dataset as the audio to be output by the service device. When the target audio conversion strategy is a target audio parameter adjustment strategy, the terminal adjusts the audio parameters of the original audio to be output according to the target audio parameter adjustment strategy to obtain the audio to be output by the service device.

[0100] In the aforementioned audio conversion method and scheme, a target audio conversion strategy is determined based on audio features, facial image features, an emotion recognition model, and various preset audio conversion strategies. The audio to be output by the business device is then determined according to this target audio conversion strategy. In other words, this scheme can convert the audio to be output by the business device in real time based on the audio features and facial image features of the target user operating the business page. Therefore, the output audio matches the target user's real-time emotion, resulting in a good interactive effect.

[0101] In one embodiment, the emotion recognition model includes a face image recognition sub-model, an audio recognition sub-model, and a fusion recognition sub-model; inputting audio features and face image features into the emotion recognition model to obtain the target emotion recognition result corresponding to the target user includes:

[0102] The audio features are input into the audio recognition sub-model to obtain the audio emotion recognition result; the facial image features are input into the facial image recognition sub-model to obtain the image emotion recognition result; the audio emotion recognition result and the image emotion recognition result are input into the fusion recognition sub-model to obtain the target emotion recognition result.

[0103] In the embodiments of this application, such as Figure 2 As shown, the terminal inputs audio features into the audio recognition sub-model and outputs an audio emotion recognition result. In one embodiment, the audio emotion recognition result includes the probability of positive audio emotion and the probability of negative audio emotion. The range of the positive audio emotion probability is [0,1], and the range of the negative audio emotion probability is [0,1]. The terminal inputs facial image features into the facial image recognition sub-model and outputs an image emotion recognition result. In one embodiment, the image emotion recognition result includes the probability of positive image emotion and the probability of negative image emotion. The range of the positive image emotion probability is [0,1], and the range of the negative image emotion probability is [0,1]. The terminal inputs the audio emotion recognition result and the image emotion recognition result into the fusion recognition sub-model and outputs a target emotion recognition result.

[0104] In this embodiment, emotion recognition is performed on audio features based on an audio recognition sub-model to obtain an audio emotion recognition result; emotion recognition is performed on facial images based on a facial image recognition sub-model to obtain an image emotion recognition result; then, based on the audio emotion recognition result and the image emotion recognition result, the target emotion recognition result is determined. Therefore, the target emotion recognition result obtained by this scheme is determined based on audio features and facial image features, determining the target user's emotion from multiple dimensions and improving the accuracy of emotion recognition.

[0105] In one embodiment, such as Figure 3 As shown. The face image recognition sub-model includes a residual network, multiple first dual attention mechanisms, a first long short-term memory network, a first splicing layer, and a first emotion recognition network; inputting face image features into the face image recognition sub-model yields image emotion recognition results including:

[0106] Step 302: Input the face image features into a residual network containing multiple residual subnetworks to obtain the image semantic features output by each residual subnetwork and the target image semantic features output by the residual network.

[0107] The residual network consists of convolutional layers, multiple residual subnetworks, and pooling layers. In the residual network, the output of each residual subnetwork (except the last one) becomes the input of the next residual subnetwork, and the output of the last residual subnetwork becomes the input of the pooling layer.

[0108] In the embodiments of this application, such as Figure 4As shown, the terminal inputs the facial image features into the convolutional layer of the residual network to obtain the output of the convolutional layer. The terminal then inputs the output of the convolutional layer into the first residual sub-network of the residual network to obtain the image semantic features output by the first residual sub-network. The terminal inputs the image semantic features output by the first residual sub-network into the second residual sub-network of the same residual network to obtain the image semantic features output by the second residual sub-network. This process continues until the terminal inputs the image semantic features output by the second-to-last residual sub-network into the last residual sub-network of the same residual network to obtain the image semantic features output by the last residual sub-network. Finally, the terminal inputs the image semantic features output by the last residual sub-network into the pooling layer of the residual network to output the target image semantic features. Optionally, the pooling layer can be an average pooling layer or a max pooling layer. For ease of distinction, the image semantic feature output by the Nth residual sub-network is referred to as the Nth image semantic feature, where N is a positive integer. In one embodiment, N is 4. Each residual sub-network includes at least one residual block. The number of residual blocks contained in each residual subnetwork can be the same or different, but the structure of each residual block is the same. In one embodiment, the residual network includes one convolutional layer, four residual subnetworks, and one average pooling layer. Specifically, the first residual subnetwork consists of 3 residual blocks, the second consists of 4 residual blocks, the third consists of 6 residual blocks, and the fourth consists of 3 residual blocks, with each residual block having the same structure. In one embodiment, the residual blocks in the residual subnetworks are composed of three residual convolutional layers. The first residual convolutional layer has a 1x1 kernel with 64 channels, the second has a 3x3 kernel with 64 channels, and the third has a 1x1 kernel with 256 channels. Specifically, as shown... Figure 5 As shown. Step 304: For each image semantic feature, input the image semantic feature into the first dual attention mechanism to obtain the image weighted feature corresponding to the image semantic feature.

[0109] In this embodiment, for each image semantic feature, the terminal inputs the image semantic feature into a first dual attention mechanism and outputs the corresponding image weighted feature. The first dual attention mechanism includes a channel attention mechanism and a spatial attention mechanism. Specifically, for each image semantic feature, the terminal inputs the image semantic feature into the channel attention mechanism of the first dual attention mechanism and outputs the initial image weighted feature. The terminal inputs the image channel semantic features into the spatial attention mechanism of the first dual attention mechanism and outputs the image weighted feature. It can be understood that the number of first dual attention mechanisms is equal to the number of residual subnetworks.

[0110] Step 306: Input the weighted features of each image into the corresponding input neurons in the first long short-term memory network to obtain the image context features.

[0111] In this embodiment, the terminal inputs the weighted features of each image into the corresponding input neurons in the first long short-term memory network and outputs the image context features. The first long short-term memory network is a Bidirectional Long Short-Term Memory (BiLSTM) network.

[0112] Step 308: Input the image context features and the target image semantic features into the first stitching layer to obtain the image fusion features.

[0113] In this embodiment, the terminal inputs the image context features output by the first long short-term memory network and the target image semantic features output by the pooling layer of the residual network into the first concatenation layer, and outputs image fusion features. The first concatenation layer is a concat layer.

[0114] Step 310: Input the image fusion features into the first emotion recognition network to obtain the image emotion recognition result.

[0115] In this embodiment, the terminal inputs image fusion features into a first emotion recognition network and outputs an image emotion recognition result. The first emotion recognition network includes a fully connected layer and a sigmoid function. Specifically, the terminal inputs image fusion features into the fully connected layer of the first emotion recognition network, outputs an initial image emotion recognition result, and then inputs the initial image emotion recognition result into the sigmoid function of the first emotion recognition network to output the final image emotion recognition result.

[0116] In this embodiment, the facial image features are processed for emotion recognition by a facial image recognition sub-model to obtain the image emotion recognition result. Therefore, the current emotion of the target user can be determined from the dimension of the facial image, thus providing a premise for determining the current emotion of the target user from multiple dimensions in the future.

[0117] In one embodiment, the audio recognition sub-model includes a second dual attention mechanism, a second long short-term memory network, and a second emotion recognition network; inputting audio features into the audio recognition sub-model yields audio emotion recognition results including:

[0118] The audio features are input into the second dual attention mechanism to obtain audio weighted features; the audio weighted features are input into the second long short-term memory network to obtain audio context features; the audio context features are input into the second emotion recognition network to obtain the audio emotion recognition result.

[0119] In the embodiments of this application, such as Figure 6 As shown, the terminal inputs audio features into a second dual attention mechanism and outputs audio weighted features corresponding to the audio features. The second dual attention mechanism includes a channel attention mechanism and a spatial attention mechanism. Specifically, the terminal inputs audio features into the channel attention mechanism of the second dual attention mechanism and outputs initial audio weighted features. The terminal inputs the initial audio weighted features into the spatial attention mechanism of the second dual attention mechanism and outputs audio weighted features. The terminal inputs the audio weighted features into a second long short-term memory network and outputs audio context features. In one embodiment, the second long short-term memory network is a Bidirectional Long Short-Term Memory (BiLSTM) network. The terminal inputs the audio context features into a second emotion recognition network to obtain the audio emotion recognition result. In one embodiment, the second emotion recognition network includes fully connected layers and a sigmoid function.

[0120] In this embodiment, the audio features are processed by an audio recognition sub-model to obtain the audio emotion recognition result. Therefore, the current emotion of the target user can be determined from the audio dimension, thus providing a basis for determining the current emotion of the target user from multiple dimensions in the future.

[0121] In one embodiment, obtaining the audio and facial image features of a target user while operating a business page on a business device includes:

[0122] When the triggering conditions for feature extraction are met, the audio features and facial image features of the target user when operating the business page of the business device are obtained.

[0123] In this embodiment, when the triggering condition for feature extraction is met, the terminal acquires audio data and facial images of the target user while operating the business page of the business device. Specifically, the terminal responds to the first operation request initiated by the target user through the business device while operating the business page and acquires the initial request initiation time. The terminal calculates the data acquisition time based on the initial request initiation time and a preset data acquisition strategy. Specifically, the terminal calculates the data acquisition time based on the initial request initiation time and a preset data acquisition period. When the current time reaches the data acquisition time, the terminal acquires audio data and facial images of the target user while operating the business page of the business device. The terminal preprocesses the audio data according to a preset audio preprocessing strategy to obtain audio features. The terminal preprocesses the facial images according to a preset image preprocessing strategy to obtain facial image features.

[0124] In this embodiment, when the triggering condition for feature extraction is met, the terminal acquires the target user's audio features and facial image features, providing a prerequisite for subsequent emotion recognition based on audio features and facial image features.

[0125] In one embodiment, such as Figure 6 As shown, when the feature extraction trigger condition is met, the audio features and facial image features obtained when the target user operates the business page of the business device include:

[0126] Step 602: Obtain the number of times the audio to be output by the service device has been determined.

[0127] In this embodiment, the terminal obtains the number of times the audio to be output by the service device has been determined. The number of times determined refers to the number of times step 108 has been executed. Specifically, after each execution of step 108, the terminal adds 1 to the original number of determined results to obtain a new number of determined results. It can be understood that the initial value of the number of determined results is 0.

[0128] Step 604: If the number of times has been determined to be within the first preset range, obtain the audio features and facial image features of the target user when operating the business page of the business device.

[0129] In this embodiment, when the number of occurrences is determined to fall within a first preset interval, the terminal acquires the audio and facial image features of the target user while operating the business page of the business device. The lower limit of the first preset interval is 0, and the upper limit is a positive number. Optionally, the upper limit of the first preset interval can be 1, 2, or 3. Specifically, when the data collection time is reached at the current moment and the number of occurrences is determined to fall within the first preset interval, the terminal acquires the audio and facial image features of the target user while operating the business page of the business device.

[0130] Step 606: If the number of times has been determined to be within the preset second interval and the current target voice conversion strategy is different from the previous target voice conversion strategy, obtain the audio features and facial image features of the target user when operating the business page of the business device.

[0131] The upper limit of the first preset interval is less than the lower limit of the second preset interval.

[0132] In this embodiment, when the number of occurrences has been determined to fall within a preset second interval, and the current target voice conversion strategy is different from the previous target voice conversion strategy, the terminal acquires the audio features and facial image features of the target user when operating the business page of the business device. In one embodiment, the union of the preset first interval and the preset second interval is an interval with a lower limit of 0 and an upper limit of the upper limit of the preset second interval. For example, the preset first interval is [0, 1] and the preset second interval is (1, 2], or the preset first interval is [0, 2) and the preset second interval is [2, 3). Specifically, when the current time reaches the data acquisition time, the number of occurrences has been determined to fall within the preset second interval, and the current target voice conversion strategy is different from the previous target voice conversion strategy, the terminal acquires the audio features and facial image features of the target user when operating the business page of the business device.

[0133] In this embodiment, the specific triggering condition for feature extraction can be that the number of times the target audio conversion has been determined falls within a first preset interval, or that the number of times the target audio conversion has been determined falls within a preset second interval, and the current target audio conversion strategy is different from the previous target audio conversion strategy. Therefore, the triggering condition for feature extraction can be used to determine whether audio features and facial image features need to be acquired.

[0134] In one embodiment, obtaining the audio and facial image features of a target user while operating a business page on a business device includes:

[0135] Acquire audio data and facial images of the target user when operating the business page of the business device; determine the Mel frequency cepstral coefficients of the audio data to obtain audio features; segment the facial image according to a preset segmentation strategy to obtain each facial sub-image, and scale the facial sub-images according to a preset scaling strategy to obtain facial image features.

[0136] In this embodiment, the terminal acquires audio data and facial images of a target user operating a business page on a business device. Based on the audio data, the terminal determines the Mel-frequency cepstral coefficients (MFCCs) of the audio data and uses these MFCCs as audio features. The terminal segments the facial image according to a preset segmentation strategy, obtaining segmented facial sub-images. In one embodiment, the size of each facial sub-image is 256*256. The terminal scales each facial sub-image according to a preset scaling strategy, obtaining scaled facial images, and uses each scaled facial image as a facial image feature. In one embodiment, the size of each scaled facial image is 254*254.

[0137] In this embodiment, the terminal uses the Mel-frequency cepstral coefficients of the audio data as the audio feature of the audio data; and obtains the facial image feature by segmenting and scaling the facial image. Therefore, this solution can obtain audio features and facial image features based on audio data and facial images. Audio features and facial image features are more categorical than audio data and facial images themselves, thereby improving the accuracy of emotion recognition based on audio features and facial image features.

[0138] In one embodiment, such as Figure 7 As shown, the target emotion recognition result includes the probability of positive emotion and the probability of negative emotion; among the preset voice conversion strategies, the target voice conversion strategies that match the target emotion recognition result include:

[0139] Step 702: Calculate the difference between the probability of positive emotion and the probability of negative emotion.

[0140] In this embodiment, the terminal calculates the difference between the probabilities of positive and negative emotions based on the probabilities of positive and negative emotions. Specifically, the terminal calculates the absolute value of the difference between the probabilities of positive and negative emotions to obtain the difference between the probabilities of positive and negative emotions.

[0141] Step 704: If the difference is greater than or equal to the preset difference threshold, and the probability of negative emotion is greater than the probability of positive emotion, then the voice conversion strategy corresponding to severe negative emotion is taken as the target voice conversion strategy.

[0142] In this embodiment, when the difference is greater than or equal to a preset difference threshold, the terminal determines the magnitude of the probability of negative emotion and the probability of positive emotion. If the probability of negative emotion is greater than the probability of positive emotion, the terminal uses the voice conversion strategy corresponding to severe negative emotion as the target voice conversion strategy. In one embodiment, the preset difference threshold is 0.2.

[0143] Step 706: If the difference is less than a preset difference threshold, based on the mapping relationship between the negative emotion probability, the positive emotion probability, and the voice conversion strategy, determine the target voice conversion strategy that matches the positive and negative emotion probabilities contained in the target emotion recognition result.

[0144] In this embodiment, when the difference is less than a preset difference threshold, the terminal determines whether the probability of negative emotion and the probability of positive emotion belong to the same preset probability interval. It can be understood that the union of the preset probability intervals is [0, 1]. In one embodiment, the preset probability interval includes a first preset probability interval and a second preset probability interval. The first preset probability interval is [0, 0.5], and the second preset probability interval is (0.5, 1). If the probability of negative emotion and the probability of positive emotion belong to the same preset probability interval, the terminal uses the voice conversion strategy corresponding to a neutral emotion as the target voice conversion strategy. If the probability of negative emotion and the probability of positive emotion belong to different preset probability intervals, the terminal determines the magnitude of the negative emotion probability and the positive emotion probability. If the probability of negative emotion is greater than the probability of positive emotion, the terminal uses the voice conversion strategy corresponding to a mild negative emotion as the target voice conversion strategy. If the probability of negative emotion is less than or equal to the probability of positive emotion, the terminal uses the voice conversion strategy corresponding to a mild positive emotion as the target voice conversion strategy.

[0145] In this embodiment, the target voice conversion strategy is determined based on the probability of positive emotion, the probability of negative emotion, and the degree of difference. Therefore, this scheme can determine the target voice conversion strategy based on the target emotion recognition results, thereby achieving audio conversion.

[0146] In one embodiment, the audio conversion method further includes:

[0147] If the difference is greater than or equal to the preset difference threshold, and the probability of positive emotion is greater than the probability of negative emotion, then the current voice conversion strategy remains unchanged.

[0148] In this embodiment, when the difference is greater than or equal to a preset difference threshold, the terminal determines the magnitude of the probability of negative emotion and the probability of positive emotion. If the probability of positive emotion is greater than the probability of negative emotion, the terminal maintains the current voice conversion strategy unchanged. Specifically, when the difference is greater than or equal to the preset difference threshold and the probability of positive emotion is greater than the probability of negative emotion, the terminal uses the current voice conversion strategy as the target voice conversion strategy.

[0149] For example, steps 702 to 706, and the judgment conditions for determining the target voice conversion strategy in this application embodiment, are shown in Table 1 below. Where x is the probability of positive emotion, y is the probability of negative emotion, |xy| is the difference between the probabilities of positive and negative emotions, 0.2 is a preset difference threshold, [0, 0.5] is the first preset probability interval, and (0.5, 1] ​​is the second preset probability interval.

[0150] Table 1

[0151]

[0152] In this embodiment, when the difference is greater than or equal to a preset difference threshold and the probability of positive emotion is greater than the probability of negative emotion, the target user's emotion can be considered to be strongly positive. In other words, the target user is relatively satisfied with the audio currently output by the service device, so audio conversion is unnecessary. Therefore, the output audio of this solution matches the target user's real-time emotion, resulting in good interactive performance.

[0153] In one embodiment, such as Figure 8 As shown, an example of an audio conversion method is also provided, which includes the following steps:

[0154] Step 802: Obtain the number of times the audio to be output by the service device has been determined.

[0155] Step 804: If the triggering condition for feature extraction is met, acquire the audio data and facial image of the target user when operating the business page of the business device.

[0156] Step 806: Determine the Mel frequency cepstral coefficients of the audio data to obtain the audio features.

[0157] Step 808: Segment the face image according to the preset segmentation strategy to obtain each face sub-image, and scale the face sub-images according to the preset scaling strategy to obtain face image features.

[0158] Step 810: Input the audio features and facial image features into the emotion recognition model to obtain the target emotion recognition result corresponding to the target user. The target emotion recognition result includes the probability of positive emotion and the probability of negative emotion.

[0159] Step 812: Calculate the difference between the probability of positive emotion and the probability of negative emotion.

[0160] Step 814: Determine whether the difference is greater than or equal to the preset difference threshold. If yes, proceed to step 816; otherwise, proceed to step 822.

[0161] Step 816: Determine whether the probability of negative emotion is greater than the probability of positive emotion. If yes, proceed to step 818; otherwise, proceed to step 820.

[0162] Step 818: Use the voice conversion strategy corresponding to severe negative emotions as the target voice conversion strategy and execute step 832.

[0163] Step 820: Keep the current voice conversion strategy unchanged and proceed to step 832.

[0164] Step 822: Determine whether the probability of negative emotion and the probability of positive emotion belong to the same preset probability interval. If yes, proceed to step 824; otherwise, proceed to step 826.

[0165] Step 824: Select the voice conversion strategy corresponding to the neutral emotion as the target voice conversion strategy and execute step 832.

[0166] Step 826: Determine whether the probability of negative emotion is greater than the probability of positive emotion. If yes, proceed to step 828; otherwise, proceed to step 830.

[0167] Step 828: Select the voice conversion strategy corresponding to mild negative emotions as the target voice conversion strategy and execute step 832.

[0168] Step 830: Use the voice conversion strategy corresponding to mild positive emotions as the target voice conversion strategy, and execute step 832.

[0169] Step 832: Determine the audio to be output by the service device according to the target audio conversion strategy, and return to step 802.

[0170] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0171] Based on the same inventive concept, this application also provides an audio conversion apparatus for implementing the audio conversion method described above. The solution provided by this apparatus is similar to the implementation described in the above method; therefore, the specific limitations of one or more audio conversion apparatus embodiments provided below can be found in the limitations of the audio conversion method described above, and will not be repeated here.

[0172] In one embodiment, such as Figure 9 As shown, an audio conversion device is provided, comprising:

[0173] The first acquisition module 902 is used to acquire the audio features and facial image features of the target user when operating the business page of the business device;

[0174] The first determining module 904 is used to input audio features and facial image features into the emotion recognition model to obtain the target emotion recognition result corresponding to the target user;

[0175] The second determining module 906 is used to determine the target voice conversion strategy that matches the target emotion recognition result among the preset voice conversion strategies;

[0176] The third determination module 908 is used to determine the audio to be output by the service device based on the target sound conversion strategy.

[0177] In one embodiment, the emotion recognition model includes a face image recognition sub-model, an audio recognition sub-model, and a fusion recognition sub-model; the first determining module 904 is specifically used for:

[0178] The audio features are input into the audio recognition sub-model to obtain the audio emotion recognition result;

[0179] The facial image features are input into the facial image recognition sub-model to obtain the image emotion recognition result;

[0180] The audio emotion recognition results and the image emotion recognition results are input into the fusion recognition sub-model to obtain the target emotion recognition result.

[0181] In one embodiment, the face image recognition sub-model includes a residual network, multiple first dual attention mechanisms, a first long short-term memory network, a first splicing layer, and a first emotion recognition network; the first determining module 904 is specifically used for:

[0182] The facial image features are input into a residual network containing multiple residual subnetworks to obtain the image semantic features output by each residual subnetwork and the target image semantic features output by the residual network. The output of each residual subnetwork except the last one is the input of the next residual subnetwork.

[0183] For each image semantic feature, the image semantic feature is input into the first dual attention mechanism to obtain the image weighted feature corresponding to the image semantic feature;

[0184] Each image weighted feature is input into the corresponding input neuron in the first long short-term memory network to obtain the image context features;

[0185] Image context features and target image semantic features are input into the first concatenation layer to obtain image fusion features;

[0186] The image fusion features are input into the first emotion recognition network to obtain the image emotion recognition result.

[0187] In one embodiment, the audio recognition sub-model includes a second dual attention mechanism, a second long short-term memory network, and a second emotion recognition network; the first determining module 904 is specifically used for:

[0188] The audio features are input into a second dual attention mechanism to obtain audio weighted features;

[0189] The audio weighted features are input into the second long short-term memory network to obtain the audio context features;

[0190] The audio context features are input into the second emotion recognition network to obtain the audio emotion recognition result.

[0191] In one embodiment, the first acquisition module 902 is specifically used for:

[0192] When the triggering conditions for feature extraction are met, the audio features and facial image features of the target user when operating the business page of the business device are obtained.

[0193] In one embodiment, the first acquisition module 902 is specifically used for:

[0194] Obtain the number of times the audio to be output from the business device has been determined;

[0195] If the number of times has been determined to be within the first preset range, obtain the audio features and facial image features of the target user when operating the business page of the business device;

[0196] If the number of times has been determined to be within the preset second interval, and the current target voice conversion strategy is different from the previous target voice conversion strategy, the audio features and facial image features of the target user when operating the business page of the business device are obtained; wherein, the upper limit of the first preset interval is less than the lower limit of the second preset interval.

[0197] In one embodiment, the first acquisition module 902 is specifically used for:

[0198] Acquire audio data and facial images of the target user when operating the business page of the business device;

[0199] Determine the Mel-frequency cepstral coefficients of the audio data to obtain the audio characteristics;

[0200] According to the preset segmentation strategy, the face image is segmented to obtain each face sub-image, and according to the preset scaling strategy, the face sub-images are scaled to obtain face image features.

[0201] In one embodiment, the target emotion recognition result includes a positive emotion probability and a negative emotion probability; the second determining module 906 is specifically used for:

[0202] Calculate the difference between the probabilities of positive and negative emotions;

[0203] If the probability of negative emotion is greater than or equal to the preset difference threshold, and the probability of positive emotion is greater than the probability of negative emotion, then the voice conversion strategy corresponding to severe negative emotion will be used as the target voice conversion strategy.

[0204] If the difference is less than a preset difference threshold, a target voice conversion strategy that matches the positive and negative emotion probabilities contained in the target emotion recognition result is determined based on the mapping relationship between the negative emotion probability, the positive emotion probability, and the voice conversion strategy.

[0205] In one embodiment, the second determining module 906 is further configured to:

[0206] If the difference is greater than or equal to the preset difference threshold, and the probability of positive emotion is greater than the probability of negative emotion, then the current voice conversion strategy remains unchanged.

[0207] The modules in the aforementioned audio conversion device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the operations corresponding to each module.

[0208] In one embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 10As shown, the computer device includes a processor, memory, input / output interfaces, a communication interface, a display unit, and an input device. The processor, memory, and input / output interfaces are connected via a system bus, and the communication interface, display unit, and input device are also connected to the system bus via the input / output interfaces. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The input / output interfaces are used for exchanging information between the processor and external devices. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, NFC (Near Field Communication), or other technologies. When executed by the processor, the computer program implements an audio conversion method. The display unit is used to form a visually visible image and can be a display screen, a projection device, or a virtual reality imaging device. The display screen can be an LCD screen or an e-ink screen. The input device of the computer device can be a touch layer covering the display screen, or buttons, trackballs, or touchpads set on the casing of the computer device, or external keyboards, touchpads, or mice, etc.

[0209] Those skilled in the art will understand that Figure 10 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0210] In one embodiment, a computer device is also provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above method embodiments.

[0211] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, implements the steps in the above method embodiments.

[0212] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0213] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0214] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0215] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0216] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. An audio conversion method, characterized in that, The method includes: Obtain the number of times the audio to be output from the business device has been determined; Based on the determined number of times, when the trigger condition for feature extraction is reached, the audio features and facial image features of the target user when operating the business page of the business device are obtained. The trigger condition is that the determined number of times belongs to a first preset interval, or the determined number of times belongs to a second preset interval and the current target voice conversion strategy is different from the previous target voice conversion strategy. The upper limit of the first preset interval is less than the lower limit of the second preset interval. The audio features and the facial image features are input into the emotion recognition model to obtain the target emotion recognition result corresponding to the target user; the target emotion recognition result includes the probability of positive emotion and the probability of negative emotion. Calculate the difference between the probability of positive emotion and the probability of negative emotion; If the difference is greater than or equal to a preset difference threshold, and the probability of negative emotion is greater than the probability of positive emotion, then the voice conversion strategy corresponding to severe negative emotion is used as the target voice conversion strategy that matches the target emotion recognition result; if the probability of positive emotion is greater than the probability of negative emotion, then the current voice conversion strategy remains unchanged. If the difference is less than the preset difference threshold, determine whether the probability of negative emotion and the probability of positive emotion belong to the same preset probability interval, and the union of each preset probability interval is [0, 1]. If the probability of negative emotion and the probability of positive emotion belong to the same preset probability range, then the voice conversion strategy corresponding to the neutral emotion is taken as the target voice conversion strategy. If the probability of negative emotion and the probability of positive emotion belong to different preset probability intervals, then the magnitude of the probability of negative emotion and the probability of positive emotion are determined; when the probability of negative emotion is greater than the probability of positive emotion, the voice conversion strategy corresponding to mild negative emotion is taken as the target voice conversion strategy; when the probability of negative emotion is less than or equal to the probability of positive emotion, the voice conversion strategy corresponding to mild positive emotion is taken as the target voice conversion strategy. Based on the target sound conversion strategy, the audio to be output by the service device is determined.

2. The method according to claim 1, characterized in that, The emotion recognition model includes a face image recognition sub-model, an audio recognition sub-model, and a fusion recognition sub-model; the step of inputting the audio features and the face image features into the emotion recognition model to obtain the target emotion recognition result corresponding to the target user includes: The audio features are input into the audio recognition sub-model to obtain the audio emotion recognition result; The facial image features are input into the facial image recognition sub-model to obtain the image emotion recognition result; The audio emotion recognition result and the image emotion recognition result are input into the fusion recognition sub-model to obtain the target emotion recognition result.

3. The method according to claim 2, characterized in that, The face image recognition sub-model includes a residual network, multiple first dual attention mechanisms, a first long short-term memory network, a first splicing layer, and a first emotion recognition network; the step of inputting the face image features into the face image recognition sub-model to obtain the image emotion recognition result includes: The facial image features are input into the residual network, which contains multiple residual subnetworks, to obtain the image semantic features output by each residual subnetwork and the target image semantic features output by the residual network; wherein, the output of the residual subnetwork except the last residual subnetwork is the input of the next residual subnetwork. For each of the image semantic features, the image semantic features are input into the first dual attention mechanism to obtain the image weighted features corresponding to the image semantic features; Each of the image weighted features is input into the corresponding input neurons in the first long short-term memory network to obtain image context features; The image context features and the target image semantic features are input into the first stitching layer to obtain image fusion features; The image fusion features are input into the first emotion recognition network to obtain the image emotion recognition result.

4. The method according to claim 2, characterized in that, The audio recognition sub-model includes a second dual attention mechanism, a second long short-term memory network, and a second emotion recognition network; the step of inputting the audio features into the audio recognition sub-model to obtain the audio emotion recognition result includes: The audio features are input into the second dual attention mechanism to obtain audio weighted features; The audio weighted features are input into the second long short-term memory network to obtain audio context features; The audio context features are input into the second emotion recognition network to obtain the audio emotion recognition result.

5. The method according to claim 1, characterized in that, The acquisition of audio and facial image features of the target user when operating the business page of the business device includes: Acquire audio data and facial images of the target user when operating the business page of the business device; The audio features are obtained by determining the Mel-frequency cepstral coefficients of the audio data; The face image is segmented according to a preset segmentation strategy to obtain individual face sub-images, and the face sub-images are scaled according to a preset scaling strategy to obtain face image features.

6. An audio conversion device, characterized in that, The device includes: The first acquisition module is used to acquire the number of times the audio to be output by the business device has been determined; based on the number of times, when the trigger condition for feature extraction is determined, the audio features and facial image features of the target user when operating the business page of the business device are acquired. The trigger condition is that the number of times is determined to be within a first preset interval, or the number of times is determined to be within a second preset interval and the current target voice conversion strategy is different from the previous target voice conversion strategy, and the upper limit of the first preset interval is less than the lower limit of the second preset interval. The first determining module is used to input the audio features and the facial image features into the emotion recognition model to obtain the target emotion recognition result corresponding to the target user; the target emotion recognition result includes the probability of positive emotion and the probability of negative emotion. The second determining module is used to calculate the difference between the positive emotion probability and the negative emotion probability; If the difference is greater than or equal to a preset difference threshold, and the probability of negative emotion is greater than the probability of positive emotion, then the voice conversion strategy corresponding to severe negative emotion is used as the target voice conversion strategy that matches the target emotion recognition result; if the probability of positive emotion is greater than the probability of negative emotion, then the current voice conversion strategy remains unchanged. If the difference is less than the preset difference threshold, determine whether the probability of negative emotion and the probability of positive emotion belong to the same preset probability interval, and the union of each preset probability interval is [0, 1]. If the probability of negative emotion and the probability of positive emotion belong to the same preset probability range, then the voice conversion strategy corresponding to the neutral emotion is taken as the target voice conversion strategy. If the probability of negative emotion and the probability of positive emotion belong to different preset probability intervals, then the magnitude of the probability of negative emotion and the probability of positive emotion are determined; when the probability of negative emotion is greater than the probability of positive emotion, the voice conversion strategy corresponding to mild negative emotion is taken as the target voice conversion strategy; when the probability of negative emotion is less than or equal to the probability of positive emotion, the voice conversion strategy corresponding to mild positive emotion is taken as the target voice conversion strategy. The third determining module is used to determine the audio to be output by the service device based on the target sound conversion strategy.

7. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 5.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 5.

9. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 5.

Images