Method and system for generating real-time audio based on dialogue content and electronic equipment

Abstract

The embodiment of the invention provides a method and system for generating real-time audio based on dialogue content and electronic equipment, and the method comprises the steps: obtaining an input voice signal, and determining audio related parameters, such as a theme type, an emotional tendency, an interaction position, interaction time and an interaction event type, corresponding to the voice signal; and generating and playing an audio matched with the audio based on the determined audio correlation parameter. By selecting the embodiment of the invention, the matched audio can be generated and played according to specific audio correlation parameters such as the theme type, the emotional tendency, the interaction position, the interaction time, the interaction event type and the like of the human-computer interaction voice signal in real time, and the sound effect matched with the current dialogue scene and atmosphere can be generated; and for some scenes requiring interactive experience such as amusement parks, theme parks, exhibition halls and the like, more immersive use experience and interactive experience can be provided for the user by selecting the embodiment of the invention.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a method, system and electronic device for generating real-time audio based on conversation content. Background Art

[0002] With the development of artificial intelligence technology, the application scenarios of human-computer dialogue have been widely used in all aspects of daily life. Existing human-computer dialogue scenarios mainly focus on generating and displaying text content based on the content of human-computer dialogue, but lack the ability to generate sound effects that match the dialogue scene and atmosphere based on the content of human-computer dialogue, and cannot provide users with a more immersive use experience and interactive experience. Summary of the invention

[0003] In view of this, the embodiments of the present application provide a method, system and electronic device for generating real-time audio based on conversation content, so as to realize real-time generation of adaptive sound effects according to human-computer interaction scenarios and interaction content, so as to provide users with a more immersive usage experience and interaction experience.

[0004] In a first aspect, an embodiment of the present application provides a method for generating real-time audio based on conversation content, wherein the method comprises:

[0005] Acquire an input voice signal, and determine audio-related parameters corresponding to the voice signal, wherein the audio-related parameters include: topic type, emotional tendency, interaction position, interaction time, and interaction event type; wherein the interaction event type is an interaction task pre-set according to the interaction position;

[0006] Based on the audio association parameters, audio matching the audio association parameters is generated and played.

[0007] In some possible embodiments, the step of generating and playing audio matching the audio association parameter based on the audio association parameter includes:

[0008] Inputting the audio-related parameters into a preset background music generation algorithm model, and generating a plurality of initial music clips by the preset background music generation algorithm model;

[0009] Based on the user's identity tag information, an initial music clip corresponding to the user's identity tag information is determined as the initial music.

[0010] In some possible embodiments, the method further includes:

[0011] According to the preset sampling intervals, the input voice signal is obtained and analyzed to determine the audio correlation coefficient corresponding to each sampling interval;

[0012] Based on the audio correlation coefficient corresponding to each sampling interval and in combination with a preset process music generation algorithm model, process music clips corresponding to different sampling intervals are generated;

[0013] According to the sampling interval sequence, each of the process music clips is connected front to back to obtain process music.

[0014] In some possible embodiments, the method further includes:

[0015] Determine a matching sound effect sound combination based on a pre-trained sound effect matching model, wherein the sound effect sound combination includes a plurality of sound effects, wherein each sound effect in the sound effect sound combination is associated with the interactive event type;

[0016] During the process of continuously playing the initial music and the process music, a target sound effect matching the interactive event type is embedded and played in the gaps between different process music clips.

[0017] In some possible embodiments, the method further includes:

[0018] The sound intensity of the input voice signal is obtained in real time. If the sound intensity is lower than a preset sound intensity threshold, the sound intensity of the played audio is reduced according to the preset sound intensity reduction ratio.

[0019] In some possible embodiments, the method further includes:

[0020] If the interactive task is changed, the audio associated with the current interactive task is terminated, and the audio corresponding to the changed new interactive task is switched and played.

[0021] In some possible embodiments, the topic type is determined in the following manner:

[0022] According to a preset speech-to-text function, converting the input speech signal into target text content;

[0023] Performing semantic analysis based on the target text content to extract associated words related to a preset topic type contained in the target text content;

[0024] According to the associated vocabulary, determining the topic type with the highest degree of association with the associated vocabulary as the topic type corresponding to the speech signal;

[0025] The emotional tendency is determined in the following way:

[0026] Acquire audio features of the speech signal, the audio features including: speaking speed, volume, and pitch;

[0027] Based on a reference speech rate, a reference volume, and a reference pitch, the speech feature is normalized to obtain a normalized audio coefficient;

[0028] According to the audio coefficient combined with preset emotional tendency label information, a target emotional tendency score is calculated, and the emotional tendency corresponding to the speech signal is determined based on the target emotional tendency score.

[0029] In a second aspect, the present application provides a system for generating real-time audio based on conversation content, wherein the system comprises:

[0030] An acquisition module, used to acquire an input voice signal and determine audio-related parameters corresponding to the voice signal, wherein the audio-related parameters include: topic type, emotional tendency, interaction position, interaction time, and interaction event type; wherein the interaction event type is an interactive task pre-set according to the interaction position;

[0031] The audio generation and playback module is used to generate and play audio matching the audio association parameters based on the audio association parameters.

[0032] In a third aspect, an embodiment of the present application provides an electronic device, wherein the electronic device comprises: a processor; and a memory for storing programs; wherein the program comprises instructions, which, when executed by the processor, cause the processor to execute the method for generating real-time audio based on conversation content as described in the first aspect.

[0033] In a fourth aspect, an embodiment of the present application provides a non-transitory computer-readable storage medium storing computer instructions, characterized in that the computer instructions are used to enable a computer to execute the method for generating real-time audio based on conversation content described in the first aspect.

[0034] Beneficial effects of this application:

[0035] The embodiments of the present application provide a method, system and electronic device for generating real-time audio based on the content of a conversation, wherein the method obtains an input voice signal and determines the audio-related parameters such as the subject type, emotional tendency, interaction position, interaction time, and interaction event type corresponding to the voice signal, and then generates and plays audio that matches the audio based on the determined audio-related parameters. By selecting the embodiments of the present application, matching audio can be generated and played in real time according to the specific audio-related parameters such as the subject type, emotional tendency, interaction position, interaction time, and interaction event type of the voice signal of human-computer interaction, and sound effects that match the current conversation scene and atmosphere can be generated. For some scenes that require interactive experience such as amusement parks, theme parks, and exhibition halls, selecting the embodiments of the present application can provide users with a more immersive use experience and interactive experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Further details, features and advantages of the present application are disclosed in the following description of exemplary embodiments in conjunction with the accompanying drawings, in which:

[0037] Figure 1 A schematic diagram of a process of generating real-time audio based on conversation content provided by an embodiment of the present application is shown;

[0038] Figure 2 A schematic diagram of a system architecture of a system for generating real-time audio based on conversation content provided by an embodiment of the present application is shown;

[0039] Figure 3 Another schematic diagram of the system architecture of the system for generating real-time audio based on conversation content provided by an embodiment of the present application is shown;

[0040] Figure 4 A structural block diagram of an exemplary electronic device that can be used to implement an embodiment of the present application is shown. DETAILED DESCRIPTION

[0041] The embodiments of the present application will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present application are shown in the accompanying drawings, it should be understood that the present application can be implemented in various forms and should not be construed as being limited to the embodiments described herein. Instead, these embodiments are provided to provide a more thorough and complete understanding of the present application. It should be understood that the drawings and embodiments of the present application are only for exemplary purposes and are not intended to limit the scope of protection of the present application.

[0042] It should be understood that the various steps described in the method implementation of the present application can be performed in different orders and / or performed in parallel. In addition, the method implementation may include additional steps and / or omit the steps shown. The scope of the present application is not limited in this respect.

[0043] The term "including" and its variations used in this document are open inclusions, that is, "including but not limited to". The term "based on" means "based at least in part on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one other embodiment"; the term "some embodiments" means "at least some embodiments". Relevant definitions of other terms will be given in the description below. It should be noted that the concepts of "first", "second", etc. mentioned in this application are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0044] It should be noted that the modifications of "one" and "plurality" mentioned in the present application are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0045] In order to achieve real-time generation of adaptive sound effects based on the content of human-computer dialogue, so as to provide users with a more immersive use experience and interactive experience. The present application provides a method, system and electronic device for generating real-time audio based on dialogue content, wherein the method for generating real-time audio based on dialogue content provided by the present application can be applied to any electronic device capable of generating real-time audio based on dialogue content. Similarly, the system for generating real-time audio based on dialogue content provided by the present application can be deployed in the electronic device capable of generating real-time audio based on dialogue content. The type of the electronic device may vary according to the application scenario, including but not limited to: personal mobile terminal device, computer or server, etc.

[0046] As an example, the electronic device may be a trigger device in an amusement theme park for providing interactive services to visitors. The trigger device may be a portable interactive device or a fixed interactive device installed in the amusement park.

[0047] In the first aspect, the present application provides a method for generating real-time audio based on conversation content, which can be as follows: Figure 1 As shown, the method comprises the following steps:

[0048] S11, obtaining an input voice signal, and determining audio-related parameters corresponding to the voice signal, wherein the audio-related parameters include: topic type, emotional tendency, interaction position, interaction time, and interaction event type; wherein the interaction event type is an interaction task pre-set according to the interaction position;

[0049] S12. Based on the audio association parameters, generate and play audio that matches the audio association parameters.

[0050] The embodiment of the present application obtains an input voice signal and determines audio-related parameters such as the topic type, emotional tendency, interaction position, interaction time, interaction event type, etc. corresponding to the voice signal, and then generates and plays audio that matches the audio based on the determined audio-related parameters.

[0051] By selecting the embodiment of the present application, it is possible to generate and play matching audio in real time based on specific audio-related parameters such as the subject type, emotional tendency, interaction position, interaction time, and interaction event type of the voice signal of human-computer interaction, and to generate sound effects that match the current dialogue scene and atmosphere. For some scenes that focus on interactive experience, such as amusement parks, theme parks, and exhibition halls, selecting the embodiment of the present application can provide users with a more immersive usage and interaction experience.

[0052] In order to facilitate understanding of the beneficial effects of the embodiments of the present application, the present invention takes the method provided by the embodiments of the present application as an example of applying the method to an amusement park. Assuming that there are several attractions in an amusement park, different attractions form a complete tour route. If a tourist arrives at an attraction on the tour route, he can interact with the interactive device in the attraction to complete the interactive mini-game specially designed for the attraction. There is continuity between the plots of the interactive mini-games between the various attractions. The interactive mini-games are interactive events. The types of interactive events of different attractions are different, and the interactive events of various attractions form a complete story. The interactive device, as the execution subject of the embodiments of the present application, can determine how many interactive events the current tourist has completed, the interactive events that need to be completed, the theme corresponding to the current attraction, the emotional tendency of the tourist at this time, and the tourist's response based on the type of interactive event, and generate audio that matches the current attraction, the current theme, the emotional tendency of the current tourist, and the type of interactive event to enhance the tourist's play experience.

[0053] The following will describe the above steps S11 to S12 in detail with reference to specific examples:

[0054] In step S11, the input voice signal is obtained, which may be a voice signal collected by a microphone module inside the acquisition device. For example, the execution subject of the present application is a portable interactive device, and the input voice signal is a voice signal collected by the microphone of the portable interactive device. As another implementation, the input voice signal may be obtained by acquiring a voice signal collected by an external microphone device and sent to the execution subject, wherein the external sound pickup device and the execution subject of the method of the embodiment of the present application are not the same device, and the two devices are connected by near field communication NFC, radio frequency identification RFID, Bluetooth, etc.

[0055] As an example, if the fixed interactive device in the amusement park is connected to the microphone on the tourist via Bluetooth, the tour guide robot is the executor of the method of the embodiment of the present application, and the microphone on the tourist is a specific sound pickup device. In the embodiment of the present application, the device capable of collecting voice signals is referred to as a sound pickup device. Based on this, in some embodiments, since the application scenario of the method provided in the embodiment of the present application may be a large public place, the place may include other sounds besides the dialogue users, that is, there will be noise interference. In order to ensure the accuracy of the generated sound effect, in the process of executing the above step S11, the method for generating real-time audio based on the dialogue content provided in the embodiment of the present application may also include the following steps:

[0056] S11-1. Acquire a voice signal collected by a target sound pickup device, and add identity tag information to the voice signal.

[0057] S11-2. Based on a preset noise reduction algorithm, the voice signal of the target sound pickup device is subjected to noise reduction processing to obtain a noise-reduced voice signal to be analyzed.

[0058] Among them, when executing step S11-1, the source device of the voice signal, that is, the sound pickup device that inputs the voice signal (referred to as the target sound pickup device in this article), can be used, and then the input voice signal can be encoded according to the input time of the voice signal and the device number of the voice signal acquisition device, and identification information that can distinguish different voice signals can be added, and this identification information is the identity tag information. As another embodiment, the identity tags of different users can be determined by interacting with other identity recognition devices in the application scenario, so that matching audio can be generated according to the identity tag later. Exemplarily, the identity tags of different users can be determined by a face recognition camera. In the embodiment of the present application, the identity tag information can be set to I. There will be specific instructions for using identity tag I below, which will not be explained here.

[0059] As another possible implementation, when a user uses the execution subject of the method of the embodiment of the present application, the user can register to use the service, and the execution subject can generate user tag information based on the user's registration information, and then when the user decides to use the function of automatically generating sound effects, the identity tag information is automatically added to the voice signal emitted by the user based on the user's user tag information. The user here can be the owner of the device or the user of the device, and the user of the device can also be determined as a participant in this article. As another possible implementation, when the user restarts the function of automatically generating sound effects, the participant manually inputs an audio tag, and the audio tag manually input by the participant is determined as the identity tag information of the voice signal.

[0060] When executing step S11-2, the AI ​​noise reduction algorithm in the pre-set execution body is used to filter out other irrelevant sounds in the noisy environment to ensure that only the voice of one participant is recognized. voice (t), t represents time. Through the preset AI noise reduction algorithm, S voice (t) is converted into the noise-reduced speech signal S' to be analyzed voice (t).

[0061] Furthermore, the parsed speech signal S' voice (t) Analyze and determine the speech signal S' to be analyzed voice (t) Corresponding topic type and emotional tendency. As an implementation method, the topic type corresponding to the speech signal to be processed can be determined by the following steps:

[0062] S11-3, converting the input voice signal into target text content according to a preset voice-to-text function;

[0063] S11-4, performing semantic analysis based on the target text content, extracting associated words related to a preset topic type contained in the target text content;

[0064] S11-5. Determine, based on the associated vocabulary, the topic type with the highest degree of association with the associated vocabulary as the topic type corresponding to the speech signal.

[0065] In step S11-3, the preset speech-to-text function can be any algorithm function with speech-to-text function. For example, a trained speech-to-text neural network model can be used to convert the input speech signal into the target text content, or to convert the input speech signal to be analyzed into the target text content. Assume that the preset speech-to-text function is T trans () then the speech signal to be analyzed S' voice (t) is converted into the corresponding target text content W, where W = T trans (S' voice (t)).

[0066] Further, step S11-4 is performed as an implementation method, and semantic analysis is performed on the target text content W through a preset natural language processing tool or model to extract the nouns and emotion-related auxiliary words contained in the target text content W, so as to subsequently determine the corresponding topic type C based on the extracted nouns and emotion-related auxiliary words. As another implementation method, the possible topic types can be pre-set based on the applicable scenarios of the method provided in the embodiments of the present application. Such pre-set topic types are preset topic types. Exemplarily, taking the aforementioned amusement park as an example, assuming that the specific application scenarios can be subdivided into technology, forests, farms, oceans, etc., then the extracted nouns are matched with the set vocabulary, and the associated related vocabulary contained in the target text content is extracted to further determine which topic is currently being discussed.

[0067] As another implementation, a language text recognition and classification model may be pre-trained based on a specific application scenario, such as a text classification model based on word vectors and deep learning, to determine the topic type with the highest degree of association with the conversation content. Then, step S11-5 is executed to determine the topic type with the degree of association with the associated vocabulary based on different associated vocabulary as the topic type corresponding to the voice signal. Exemplarily, taking the aforementioned amusement park as an example of an application scenario, assuming that the amusement park is divided into different plot areas, and different plot areas correspond to a topic type, the vocabulary of which specific plot is contained in the target text can be determined based on the vocabulary associated with the amusement park plot contained in the current target text content, and the plot with the highest degree of association is determined as the topic type corresponding to the current voice signal.

[0068] The specific algorithm implementation of the above steps S11-4 and S11-5 can be understood as follows: Assume that a topic classification function is obtained based on actual application scenario training: F topic (), then input the target text content into the F topic () function, the corresponding subject type can be obtained, that is, C = F topic (W), W = T trans (S' voice (t)).

[0069] As an implementation method, semantic analysis can be performed based on the target text content, and then the modal particles contained in the target text content can be extracted to further assist in determining the current emotions and emotional tendencies of the participants. As another implementation method, step S11 can determine the emotional tendency corresponding to the voice signal through the following steps:

[0070] S11-6, obtaining audio features of the speech signal, the audio features including: speech rate v, volume l, and pitch p;

[0071] S11-7, based on the reference speech speed, reference volume, and reference pitch, normalizing the speech features to obtain normalized audio coefficients;

[0072] S11-8. Calculate the target emotional tendency score according to the audio coefficient combined with the emotional tendency label information obtained by preprocessing, and determine the emotional tendency corresponding to the speech signal based on the target emotional tendency score.

[0073] When executing step S11-6, as an implementation method, the number of words or syllables per unit time in the speech signal can be calculated as the speech rate v of the speech signal. Specifically, by identifying the word or syllable boundaries in the speech signal, then measuring the time intervals between the word or syllable boundaries, and then calculating the number of words or syllables per unit time.

[0074] As an implementation method, audio feature extraction can be performed on the speech signal to extract the amplitude of the speech signal as the volume l of the speech signal. Specifically, the corresponding volume can be calculated by calculating the sum of the amplitudes of all sampling points in a sound frame, or by calculating the sum of the squares of the amplitudes of all sampling points and then taking the logarithm as the volume of the speech signal.

[0075] As an implementation method, the basic frequency of the speech signal can be extracted by performing spectrum analysis on the speech signal through methods such as short-time Fourier transform, and then the basic frequency is determined as the pitch p of the speech signal.

[0076] In step S11-7, the reference speech speed v0, the reference volume l0, and the reference pitch p0 are used to measure the speech speed, volume, and pitch of the speech signal. By comparing the speech speed v of the speech signal with the reference speech speed v0, the speed of the participant's speech can be determined. Similarly, by comparing with the reference volume, the volume of the participant's speech can be determined, and by comparing with the reference pitch, the pitch of the participant's speech can be determined. Since the speech speed, volume, and pitch are data in different dimensions, in order to facilitate unified calculation, in the embodiment of the present application, the actual speech features of the participant are normalized based on the reference amount to obtain a normalized audio coefficient, wherein, corresponding to the audio features of the speech, the audio coefficient also includes three categories: speech speed audio coefficient, volume audio coefficient, and pitch audio coefficient.

[0077] In the embodiment of the present application, the normalization process can be performed by obtaining the ratio between the actual audio feature and the reference value as the audio coefficient. Specifically, taking speech rate as an example, the speech rate audio coefficient k v = actual speech rate v of speech signal / reference speech rate v0, and so on, volume audio coefficient k l = Actual volume of speech signal l / reference volume l0, pitch audio coefficient k p= actual pitch p of the speech signal / reference pitch p0. As an implementation method, a comprehensive audio coefficient K = k can be obtained by multiplying the speech rate audio coefficient, the volume audio coefficient, and the pitch audio coefficient. v *k l *k p , that is, K = actual speaking speed v of the speech signal / reference speaking speed v0*actual volume l of the speech signal / reference volume l0*actual pitch p of the speech signal / reference pitch p0.

[0078] Further, step S11-8 is performed, firstly using a sentiment analysis algorithm (such as a support vector machine (SVM) algorithm) to perform real-time sentiment analysis on the text of each human-computer dialogue to obtain the sentiment tendency label information E0 of the dialogue text. Then, combined with the comprehensive audio coefficient K representing the speech audio characteristics of this dialogue, the actual sentiment tendency score E=E0*K of this dialogue is calculated.

[0079] Among them, the level corresponding to the emotional tendency label of the dialogue text can be a simple three-level negative, neutral, and positive level, or it can be pessimistic, relatively pessimistic, negative, relatively negative, neutral, relatively positive, positive, relatively optimistic, and optimistic, with a total of 9 levels. Different levels correspond to different emotional tendency label values, that is, different E0 values. As an implementation method, when a participant first uses the service of generating real-time audio based on dialogue content provided by the embodiment of the present application, the emotional tendency score of the participant can be calculated by executing the above steps S11-6 to S11-8.

[0080] As another implementation method, with multiple human-computer interactions of the participant, the participant's speaking speed v', volume l' and pitch p' at different times can be compared to learn the participant's voice characteristics and gradually perform normalization processing, so that the calculation of the above-mentioned comprehensive audio coefficient K can be more in line with the participant's conventional voice characteristics, avoiding the comprehensive audio coefficient obtained by a single calculation from excessively affecting the result of the sentiment tendency analysis, and finally updating the sentiment tendency score E and the corresponding sentiment tendency category in the above manner.

[0081] Specifically, the volume, speaking rate and pitch values ​​used to determine emotional tendencies can be updated by adopting statistical methods such as mean and standard deviation. For example, the mean value can be used to calculate the mean value, the mean value of the volume, the mean value of the speaking rate and the mean value of the pitch, and then the final comprehensive audio coefficient K can be calculated with the help of the mean value of the volume, the mean value of the speaking rate and the mean value of the pitch, so that the result of K is more reasonable.

[0082] Based on the execution result of the above step S11, the topic type of the participant and the emotional tendency of the participant can be basically determined. Further, the execution of step S12 can be simply based on the topic type and emotional tendency, and generate and play audio that matches the topic type and emotional tendency. Exemplary, for example, if the current theme of the amusement is a quiz, it can be determined based on the participant's voice signal whether the participant's answer is correct. If the answer is wrong, a sadder audio is generated and played. If the participant's voice signal is a correct answer, a happier audio can be generated and played. In an embodiment of the present application, the audio is subdivided into music and sound effects, wherein music is continuously played audio, and sound effects are non-continuously played audio embedded in the music playback process. Among them, as an implementation method, step S12 can be implemented by the following steps:

[0083] S12-1, inputting the audio-related parameters into a preset background music generation algorithm model, and generating a plurality of initial music clips by the preset background music generation algorithm model;

[0084] S12-2. Based on the user's identity tag information, determine the initial music clip corresponding to the user's identity tag information as the initial music.

[0085] Among them, in step S12-1, the preset background music generation algorithm model can be any type of algorithm model that can generate corresponding audio based on the input text content. The preset process music generation algorithm hereinafter is the same, which can be any type of algorithm model that can generate corresponding audio based on the input text content. The two music generation algorithm models can be the same or different. As an example, the preset background music generation algorithm model or the preset process music generation algorithm model can be an algorithm model such as MusicGen, MusicLM, etc., which is used to generate music clips that match audio-related parameters.

[0086] Specifically, the initial music clip M can be generated based on the information such as the participant's interactive position P, the participant's interactive role R, the interactive time T, the basic event O, etc., where the basic event is the interactive event, that is, the interactive task set by the party providing the audio at the corresponding interactive position P according to the application scenario. This information, the theme type C, and the emotional tendency E are used as input parameters of the music generation algorithm model. initial .

[0087] Among them, the interactive role R can be flexibly set according to the scene plot. For example, in the application scenario of the above-mentioned amusement park, the interactive role can be a magician or an elf. The specific type of interactive role can be flexibly set. The interactive position P can be set according to the location where the interactive behavior occurs. Specifically, it can be the location obtained by GPS positioning within the execution subject, or the location of the device interacting with it. The interaction time can be determined according to the device system time, and the basic event can also be flexibly set according to the scene plot. It is mainly set by the scene organizer according to actual needs. Taking the above-mentioned amusement park as an example, the basic event can be an interactive task that tourists need to complete at the current scenic spot.

[0088] As an implementation method, multiple corresponding initial music clips may be pre-generated for different user identity tag information I, and then in the actual interaction process, step S12-2 is executed, and the initial music clip corresponding to the user's identity tag information may be determined as background music based on the user's identity tag information. For example, taking the above-mentioned amusement park as an example, since different tourists have different identity tags, or different tourists have different progress in completing the script, the background music corresponding to different tourists at the same tourist attraction should be different. The initial music that suits the tourist can be determined based on the tourist's identity tag to provide the tourist with an immersive play experience.

[0089] Specifically, it is assumed that the preset music generation algorithm model includes a background music generation algorithm, and the background music generation algorithm is G music (), generate parameter prompt set B = {P, R, T, O, E, K, C}, and obtain the initial music segment M initial =G music (B), different identity tag information I i Corresponding to different initial music sets M initial (I i ), the actual background music played is M play =M initial (I i ).

[0090] Furthermore, during the execution of the above step S12, the sound effect may be generated in the following manner:

[0091] S12-3. According to the preset sampling interval, the input voice signal is obtained and analyzed to determine the audio correlation coefficient corresponding to each sampling interval; that is, the generation parameter prompt set B = {P, R, T, O, E, K, C} under different sampling intervals is determined.

[0092] S12-4, based on the audio correlation coefficient corresponding to each sampling interval, combined with the preset process music generation algorithm model, process music clips corresponding to different sampling intervals are generated, and each of the process music clips is connected to obtain process music according to the sampling interval sequence. Based on the generation parameter prompt set B={P, R, T, O, E, K, C} under different sampling intervals, combined with the preset process music generation algorithm model, process music under different sampling intervals is generated. Finally, according to the sampling interval sequence, the process music clips corresponding to each sampling interval are connected to each other, which can also be understood as splicing, to obtain process music.

[0093] S12-5, determining a matching sound effect sound combination based on a pre-trained sound effect matching model, wherein the sound effect sound combination includes a plurality of sound effects, wherein each sound effect in the sound effect sound combination is associated with the interactive event type;

[0094] S12-6. While playing the initial music and the process music continuously without pause, embed and play a target sound effect matching the interactive event type in the gaps between different process music clips.

[0095] Specifically, in step S12-3 and step S12-4, the preset sampling interval is determined according to the sampling time interval set by the device, wherein the sampling time interval can be flexibly set according to the sound effect accuracy requirements of the actual application scenario, and this application does not strictly limit it. As an example, the preset music generation algorithm may also include a process music generation algorithm G process (), the sampling time interval can be set to t interval = 15s, sample and analyze the human-computer dialogue according to the sampling time of 15 seconds each time, and obtain the topic type C of the dialogue in the current sampling period current and emotional tendency E current , passing the theme type and emotional tendency as input parameters to the process music generation algorithm G process (), generate the corresponding background music clip M segment By connecting the process music clips corresponding to different sampling time intervals, the background music can be generated in real time in an accompanying manner.

[0096] The specific algorithm implementation process can be understood as follows: at each sampling time t n (n represents the sampling number), by the process music generation algorithm G process () Based on the topic type C determined within the current sampling interval current 、Emotional tendency current Output the corresponding music clips M in different sampling intervals segment (t n )=G process (Ccurrent (t n ),E current (t n )), and then connect each music clip in the order of sampling: M process =∑ n M segment (t n ).

[0097] Further, step S12-5 is executed to determine a matching sound effect sound combination based on the pre-trained sound effect matching model, wherein the sound effect sound combination includes multiple sound effects. Specifically, a sound effect library S including various sounds can be produced based on the interactive content and the environment corresponding to the current location. Among them, various sounds can refer to animal sounds, machine sounds, natural sounds, human sounds, etc., or they can be sound effects specially set according to actual scenes. For example, in the theme scene of war in an amusement park, different shooting sound effects can be set for the theme.

[0098] Then, the sound effects in the sound effect library S are used as training sample data to train the sound effect matching model, and a sound effect matching model M is obtained by training to generate matching effects based on interactive content and the overall environment. match , using the sound matching model M match A many-to-many matching relationship is constructed between each sound effect and the dialogue topic type C and the emotional tendency E. Then, according to step S11, the current dialogue topic type C corresponding to the currently input voice signal is obtained. current 、Emotional tendency current , input to the sound matching model M match In the process, determine the sound effect combination that matches the current dialogue topic type and emotional tendency, and set the sound effect combination as S combination Among them, the sound effect sound combination is a set of multiple sound effects, namely S combination ={s1,s2,...,s m}, where m is the number of matched sound effects.

[0099] Further, step S12-6 is executed. During the playing of background music and process music, since the process music is composed of multiple music clips, the above-mentioned sound effect combination S is embedded in the gaps between different music clips. combination ={s1,s2,...,s m}, so that the background music plays without interruption, and the sound effects in the sound effect combination are superimposed on the background music and process music.

[0100] In some embodiments, the method for generating real-time audio based on conversation content provided by the embodiment of the present application further includes the following steps:

[0101] The sound intensity of the input voice signal is obtained in real time. If the sound intensity is lower than a preset sound intensity threshold, the sound intensity of the played audio is reduced according to the preset sound intensity reduction ratio.

[0102] Specifically, the sound intensity L of the human-computer dialogue sound of the participant can be detected in real time by the above-mentioned method for determining the volume. voice , and then according to the predefined sound intensity threshold L threshold , to determine the volume of the conversation of the current participant. Among them, the predefined sound intensity threshold is the above-mentioned preset sound intensity threshold, and the sound intensity threshold can be flexibly set according to different actual application scenarios, and this application does not strictly limit it.

[0103] If L voice <L threshold , the sound intensity of the audio can be reduced by lowering the volume of the audio being played. voice <L threshold The corresponding sound intensity reduction ratio is determined by the range of, for example, if it is less than 10%, the volume of the audio playback is reduced by 10%, and if it is less than 20%, the volume of the audio playback is reduced by 20%. The specific sound intensity reduction ratio can also be flexibly set according to the actual application scenario, and this application does not strictly limit it.

[0104] In this way, the volume level of the music and sound effects can be automatically adjusted to ensure that the sound effects do not interfere with the clarity of the conversation.

[0105] In some possible embodiments, the method for generating real-time audio based on conversation content provided in the embodiments of the present application may also include:

[0106] If the interactive task is changed, the audio associated with the current interactive task is terminated, and the audio corresponding to the changed new interactive task is switched and played.

[0107] Specifically, by monitoring the task switching instructions, if the monitoring result is that the task is switched, based on the label information of the switched task and the theme type and emotional tendency corresponding to the input new voice signal, generate and play audio that matches the label information, new theme type, and new emotional tendency of the new task.

[0108] Alternatively, when receiving a task switching button or interface triggered by the user, it can be determined that there is a task switching instruction and the task is switched. At this time, based on the label information of the interactive task after switching, combined with the input new voice signal, steps S11 and S12 can be executed to generate and play the initial music, process music, and target sound effect corresponding to the new interactive task.

[0109] The above process can be specifically implemented by the following algorithm:

[0110] Assume that the mixed audio signal is A(t). At time t, if t is in the background music playing stage and not in the gap of embedded sound effects, let A(t) = M(t); if t is in the gap between music clips, let:

[0111]

[0112] Among them, M(t) is the background music corresponding to time t, m is the number of sound effects, and s i (t) is the sound effect combination S combination The signal of the i-th sound effect at time t.

[0113] Then, the adjustment function F based on the sound intensity can be adjust , then the adjusted sound intensity of the music played is L music =F adjust (L voice ,L threshold )*L original_music , where L original_music is the sound intensity of the currently playing music, and the sound intensity of the adjusted playing sound effect L effect =F adjust (L voice ,L threshold )*L original_effect , where L original_effect The sound intensity of the sound effect currently being played.

[0114] When the task is switched, assume that the background music and sound effect termination function associated with the current task is F terminate (M current ,S current ), the starting function of the background music and sound effects corresponding to the new task is F start (M new ,S new ), then execute F first terminate (M current ,S current ), then execute F start (M new ,S new ), where M current For the music currently being played, S current is the sound effect currently being played, M new For the music corresponding to the new task, S new The sound effects corresponding to the new tasks.

[0115] By selecting the embodiment of the present application, an overall experience of background music and sound effects for human-computer dialogue can be intelligently created, starting with personalized music that combines the environment and the task, and in the process, matching music and sound effects are generated around the human-computer dialogue between the participant and the AI. In addition, emotional tendency analysis is performed on the participants, and the voice characteristics of their dialogue are included in the analysis and application, which can make the generated audio more fault-tolerant, achieving a good user experience and not prone to obvious errors and the serious problems caused by them.

[0116] In the embodiment of the present application, the audio can be played by the execution subject itself, or by an external audio device controlled by the execution subject. As an example, after the execution subject determines the corresponding audio in step S12, the audio is played through the audio device in the current interactive position in the playground.

[0117] Based on the method of the first aspect, in a second aspect, an embodiment of the present application provides a system for generating real-time audio based on conversation content, wherein the system can be as follows: Figure 2 As shown, the system 20 includes:

[0118] The acquisition module 201 is used to acquire an input voice signal and determine audio-related parameters corresponding to the voice signal, wherein the audio-related parameters include: topic type, emotional tendency, interaction position, interaction time, and interaction event type; wherein the interaction event type is an interaction task pre-set according to the interaction position;

[0119] The audio generation and playback module 202 is used to generate and play audio matching the audio association parameters based on the audio association parameters. As an implementation method, the system can be as follows Figure 3 As shown, it is divided into the following modules:

[0120] The user identity information identification module 21 is used to identify the identity of the participant by executing step S11 - 1 in the above method.

[0121] The speech-to-text and target participant voice recognition module 22 is used to convert the interactive speech signal of the participant into text content and perform noise reduction processing by executing steps S11 - 2 to S11 - 5 in the above method.

[0122] The dialogue topic analysis and dialogue sentiment tendency analysis module 23 is used to execute steps S11-6 to S11-8 to analyze the dialogue topic and sentiment tendency of the participants' dialogue speech.

[0123] The initial music generation and process music generation module 24 is used to execute the above steps S12-1 to S12-4 to generate corresponding initial music and process music.

[0124] The sound effect synthesis and audio mixing and playing module 25 is used to execute the above steps S12-5 to S12-6, synthesize the sound effects and embed the sound effects to obtain mixed audio, and finally output the mixed audio and play it.

[0125] By selecting the embodiment of the present application, the overall experience of the background music and sound effects of the human-computer dialogue can be intelligently created, starting from the personalized music that combines the environment and the task, and in the process, the matching music and sound effects are generated in a companion manner around the human-computer dialogue between the participant and the AI. In addition, the emotional tendency analysis of the participants is carried out, and the voice characteristics of their dialogue are included in the analysis and application, which can make the generated audio more fault-tolerant, and can achieve a good user experience without being prone to obvious errors and the serious problems caused by them. It can generate music and sound effects around comprehensive information such as interaction location, interaction role, interaction time, basic events, etc., and realize the deep integration of interactive content and overall environment customization training sound effect generation model.

[0126] Among them, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved in this application are in compliance with the relevant laws and regulations and do not violate public order and good morals.

[0127] The names of the messages or information exchanged between multiple devices in the embodiments of the present application are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0128] In a third aspect, the exemplary embodiments of the present application further provide an electronic device, comprising: at least one processor; and a memory connected to the at least one processor in communication. The memory stores a computer program that can be executed by the at least one processor, and the computer program is used to enable the electronic device to perform a method according to an embodiment of the present application when executed by the at least one processor.

[0129] The exemplary embodiment of the present application further provides a non-transitory computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor of a computer, is used to cause the computer to perform a method according to an embodiment of the present application.

[0130] The exemplary embodiments of the present application further provide a computer program product, including a computer program, wherein when the computer program is executed by a processor of a computer, it is used to enable the computer to execute the method according to the embodiment of the present application.

[0131] refer to Figure 4, the structural block diagram of the electronic device 400 that can be used as the server or client of the present application will now be described, which is an example of the hardware device that can be applied to various aspects of the present application. The electronic device is intended to represent various forms of digital electronic computer equipment, such as laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices and other similar computing devices. The components shown herein, their connections and relationships, and their functions are only examples, and are not intended to limit the implementation of the present application described and / or required herein.

[0132] like Figure 4 As shown, the electronic device 400 includes a computing unit 401, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM 402) or a computer program loaded from a storage unit 408 into a random access memory (RAM 403). In RAM 403, various programs and data required for the operation of the electronic device 400 can also be stored. The computing unit 401, ROM 402 and RAM 403 are connected to each other via a bus 404. An input / output interface (I / O interface 405) is also connected to the bus 404.

[0133] A plurality of components in the electronic device 400 are connected to the I / O interface 405, including: an input unit 406, an output unit 407, a storage unit 408, and a communication unit 409. The input unit 406 may be any type of device capable of inputting information to the electronic device 400, and the input unit 406 may receive input digital or character information, and generate key signal inputs related to user settings and / or function control of the electronic device. The output unit 407 may be any type of device capable of presenting information, and may include, but is not limited to, a display, a speaker, a video / audio output terminal, a vibrator, and / or a printer. The storage unit 408 may include, but is not limited to, a disk, an optical disk. The communication unit 409 allows the electronic device 400 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks, and may include, but is not limited to, a modem, a network card, an infrared communication device, a wireless communication transceiver, and / or a chipset, such as a Bluetooth™ device, a WiFi device, a WiMax device, a cellular communication device, and / or the like.

[0134] The computing unit 401 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 401 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 401 performs the various methods and processes described above. For example, in some embodiments, the aforementioned method for generating real-time audio based on the content of the conversation may be implemented as a computer software program, which is tangibly included in a machine-readable medium, such as a storage unit 408. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 400 via the ROM 402 and / or the communication unit 409. In some embodiments, the computing unit 401 may be configured to perform the aforementioned method for generating real-time audio based on the content of the conversation by any other appropriate means (e.g., by means of firmware).

[0135] The program code for implementing the method of the present application can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, implements the functions / operations specified in the flow chart and / or block diagram. The program code can be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0136] In the context of the present application, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0137] As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, apparatus, and / or device (e.g., disk, optical disk, memory, programmable logic device (PLD)) for providing machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal for providing machine instructions and / or data to a programmable processor.

[0138] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0139] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0140] A computer system may include clients and servers. Clients and servers are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship to each other.

Claims

1. A method for generating real-time audio based on conversation content, characterized in that: The method comprises: Acquire an input voice signal, and determine audio-related parameters corresponding to the voice signal, wherein the audio-related parameters include: topic type, emotional tendency, interaction position, interaction time, and interaction event type; wherein the interaction event type is an interaction task pre-set according to the interaction position; Based on the audio association parameters, audio matching the audio association parameters is generated and played.

2. The method according to claim 1, characterized in that The step of generating and playing audio that matches the audio association parameter based on the audio association parameter includes: Inputting the audio-related parameters into a preset background music generation algorithm model, and generating a plurality of initial music clips by the preset background music generation algorithm model; Based on the user's identity tag information, an initial music clip corresponding to the user's identity tag information is determined as the initial music.

3. The method according to claim 2, characterized in that The method further comprises: According to the preset sampling intervals, the input voice signal is obtained and analyzed to determine the audio correlation coefficient corresponding to each sampling interval; Based on the audio correlation coefficient corresponding to each sampling interval and in combination with a preset process music generation algorithm model, process music clips corresponding to different sampling intervals are generated; According to the sampling interval sequence, each of the process music clips is connected front to back to obtain process music.

4. The method according to claim 3, characterized in that The method further comprises: Determine a matching sound effect sound combination based on a pre-trained sound effect matching model, wherein the sound effect sound combination includes a plurality of sound effects, wherein each sound effect in the sound effect sound combination is associated with the interactive event type; During the process of continuously playing the initial music and the process music, a target sound effect matching the interactive event type is embedded and played in the gaps between different process music clips.

5. The method according to claim 1, characterized in that The method further comprises: The sound intensity of the input voice signal is obtained in real time. If the sound intensity is lower than a preset sound intensity threshold, the sound intensity of the played audio is reduced according to the preset sound intensity reduction ratio.

6. The method according to claim 1, characterized in that The method further comprises: If the interactive task is changed, the audio associated with the current interactive task is terminated, and the audio corresponding to the changed new interactive task is switched and played.

7. The method according to claim 1, characterized in that The subject type is determined as follows: According to a preset speech-to-text function, converting the input speech signal into target text content; Performing semantic analysis based on the target text content to extract associated words related to a preset topic type contained in the target text content; According to the associated vocabulary, determining the topic type with the highest degree of association with the associated vocabulary as the topic type corresponding to the speech signal; The emotional tendency is determined in the following way: Acquire audio features of the speech signal, the audio features including: speaking speed, volume, and pitch; Based on a reference speech rate, a reference volume, and a reference pitch, the speech feature is normalized to obtain a normalized audio coefficient; According to the audio coefficient combined with preset emotional tendency label information, a target emotional tendency score is calculated, and the emotional tendency corresponding to the speech signal is determined based on the target emotional tendency score.

8. A system for generating real-time audio based on conversation content, characterized in that: The system comprises: An acquisition module, used to acquire an input voice signal and determine audio-related parameters corresponding to the voice signal, wherein the audio-related parameters include: topic type, emotional tendency, interaction position, interaction time, and interaction event type; wherein the interaction event type is an interactive task pre-set according to the interaction position; The audio generation and playback module is used to generate and play audio matching the audio association parameters based on the audio association parameters.

9. An electronic device, characterized in that: The electronic device comprises: a processor and a memory storing a program; wherein the program comprises instructions, and when the instructions are executed by the processor, the processor executes the method according to any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium storing computer instructions, characterized in that: The computer instructions are used to make a computer execute the method according to any one of claims 1-7.

Images