A method and system for generating and adjusting virtual human context sound effects

Abstract

Some embodiments of the present disclosure provide a method and system for generating and adjusting virtual human situational sound effects. The method comprises obtaining information to be output by a virtual human and environmental information, the information to be output including at least one of text information that has not been actually presented, speech information that has not been output, and action information that has not been executed; determining a virtual human emotion parameter based on the information to be output and the environmental information; determining a sound effect adjustment parameter corresponding to the virtual human emotion parameter; and generating target sound effect data based on the sound effect adjustment parameter.

Description

Technical Field

[0001] This manual relates to the field of virtual human context sound effects generation and adjustment technology, and in particular to methods and systems for generating and adjusting virtual human context sound effects. Background Technology

[0002] With the widespread application of virtual digital humans (hereinafter referred to as virtual humans) in scenarios such as video games, virtual reality, and online customer service, contextual sound effects can express the environment, psychological state, and behavior of virtual humans through sound changes during the presentation of virtual humans. Users have put forward higher requirements for the contextual sound effects of virtual humans.

[0003] Virtual human output typically possesses strong real-time and interactive characteristics. For example, during user interaction, the virtual human's text, voice, actions, and environmental information may all change in real time as the interaction progresses. Correspondingly, the virtual human's contextual sound effects also need to be dynamically adjusted according to the virtual human's emotions, behaviors, and environmental changes to ensure the sound performance remains consistent with the virtual human's current context. However, current virtual human contextual sound effects usually rely primarily on playing fixed preset audio files or pre-produced audio data, often only triggering fixed sound effects within preset scenarios. It is difficult to generate or flexibly adjust sound effects in real time based on the virtual human's text, voice, actions, and environmental information during the current interaction.

[0004] Therefore, it is necessary to provide a method and system for generating and adjusting virtual human contextual sound effects, so as to improve the virtual human's contextual expressiveness, emotional expression effect and interactive immersion. Summary of the Invention

[0005] This specification provides one or more embodiments of a method for generating and adjusting virtual human contextual sound effects. The method includes: acquiring virtual human output information and environmental information, wherein the output information includes at least one of text information not yet presented, speech information not yet output, and action information not yet executed; determining virtual human emotional parameters based on the output information and the environmental information; determining sound effect adjustment parameters corresponding to the virtual human emotional parameters; and generating target sound effect data based on the sound effect adjustment parameters.

[0006] This specification provides one or more embodiments of a virtual human contextual sound effect generation and adjustment system. The system includes an acquisition module, a first determination module, a second determination module, and a generation module. The acquisition module is configured to acquire the virtual human's output information and environmental information. The output information includes at least one of text information that has not yet been actually presented, speech information that has not yet been output, and action information that has not yet been executed. The first determination module is configured to determine the virtual human's emotional parameters based on the output information and the environmental information. The second determination module is configured to determine the sound effect adjustment parameters corresponding to the virtual human's emotional parameters. The generation module is configured to generate target sound effect data based on the sound effect adjustment parameters.

[0007] This specification provides one or more embodiments of a device for generating and adjusting virtual human contextual sound effects, including at least one processor and at least one memory; the at least one memory is used to store computer instructions; the at least one processor is used to execute a method for generating and adjusting virtual human contextual sound effects.

[0008] This specification provides one or more embodiments of a computer-readable storage medium that stores computer instructions. When a computer reads the computer instructions in the storage medium, the computer executes a method for generating and adjusting virtual human context sound effects. Attached Figure Description

[0009] This specification will be further described by way of exemplary embodiments, which will be described in detail with reference to the accompanying drawings. These embodiments are not limiting; in these embodiments, the same reference numerals denote the same structures, wherein:

[0010] Figure 1 This is a schematic diagram illustrating the application scenario of the virtual human context sound effect generation and adjustment system according to some embodiments of this specification; Figure 2 This is an exemplary block diagram of a virtual human context sound effect generation and adjustment system according to some embodiments of this specification; Figure 3 This is an exemplary flowchart of a method for generating and adjusting virtual human context sound effects according to some embodiments of this specification; Figure 4 These are exemplary schematic diagrams illustrating the determination of target sound effect data according to some embodiments of this specification; Figure 5 This is an exemplary flowchart illustrating the application of interpolated acoustic parameters to the output of ambient sound effects in virtual space, according to some embodiments of this specification. Detailed Implementation

[0011] To more clearly illustrate the technical solutions of some embodiments of this specification, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are merely some examples or embodiments of this specification. For those skilled in the art, these drawings can be applied to other similar scenarios without creative effort. Unless obvious from the context or otherwise specified, the same reference numerals in the drawings represent the same structures or operations.

[0012] It should be understood that the terms “system,” “device,” “unit,” and / or “module” used herein are one way to distinguish different components, elements, parts, sections, or assemblies at different levels. However, if other terms can achieve the same purpose, they may be replaced by other expressions.

[0013] As indicated in this specification and claims, unless the context clearly indicates otherwise, the words "a," "an," "an," and / or "the" do not specifically refer to the singular and may also include the plural. Generally speaking, the terms "comprising" and "including" only indicate the inclusion of expressly identified steps and elements, which do not constitute an exclusive list, and the method or apparatus may also include other steps or elements.

[0014] Flowcharts are used in this specification to illustrate the operations performed by the system according to embodiments of this specification. It should be understood that the preceding or following operations are not necessarily performed in exact order. Instead, the steps can be processed in reverse order or simultaneously. Furthermore, other operations can be added to these processes, or one or more steps can be removed from them.

[0015] Figure 1 This is a schematic diagram illustrating application scenarios of a virtual human context sound effect generation and adjustment system, as shown in some embodiments of this specification. For example... Figure 1 As shown, application scenario 100 of the virtual human context sound effect generation and adjustment system may include processing device 110, network 120, terminal 130, and storage device 140. In some embodiments, application scenario 100 may be used to implement the methods and / or processes disclosed in this specification.

[0016] It should be understood that Figure 1The application scenario 100 shown is merely illustrative and does not constitute a limitation on the implementation methods described in this specification. In other embodiments, the number of devices and functional divisions included in application scenario 100 can be adjusted according to actual application needs. For example, application scenario 100 can be at least one of the following: virtual digital human interaction scenario, game scenario, online customer service scenario, live streaming companionship scenario, education and training scenario, virtual exhibition hall scenario, or metaverse interaction scenario. In the above scenarios, the virtual human can generate and adjust sound effects output in real time to adapt to the current context based on user input, environmental changes, and action changes.

[0017] As an example, in a virtual human interaction scenario, a user can input the voice message "How's the weather today?" through terminal 130. Terminal 130 can then transmit this voice message to processing device 110 via network 120. Processing device 110 can generate the virtual human's response text, corresponding voice information, and action information based on the voice message. It can also determine the virtual human's emotional parameters, sound effect adjustment parameters, and target sound effect data by combining the environmental information of the virtual human's environment. Afterward, processing device 110 can transmit the generated target sound effect data to terminal 130 for output, thereby achieving the synchronous presentation of the virtual human's text, voice, actions, and contextual sound effects.

[0018] The processing device 110 can be used to process data and / or information from various components and / or external data sources of the application scenario 100 of the virtual human contextual sound effect generation and adjustment system. The processing device 110 can execute program instructions based on this data, information, and / or processing results, thereby performing one or more functions described in this specification. For example, the processing device 110 can acquire the virtual human's output information and environmental information; determine the virtual human's emotional parameters based on the output information and environmental information; determine the sound effect adjustment parameters corresponding to the virtual human's emotional parameters; and generate target sound effect data based on the sound effect adjustment parameters.

[0019] In some embodiments, the processing device 110 may be a single server, a group of servers, a local device, a remote device, or implemented on a cloud platform.

[0020] In some embodiments, the processing device 110 may include an acquisition module 210, a first determination module 220, a second determination module 230, a generation module 240, and a region processing module 250. For more details, please refer to... Figure 2 And its explanation.

[0021] Network 120 may include any suitable network that facilitates the exchange of information and / or data. In some embodiments, one or more components of application scenario 100 of the virtual human contextual sound effects generation and adjustment system may exchange information and / or data via network 120. For example, processing device 110 may acquire the virtual human's output information and environmental information via network 120. Network 120 may include a local area network (LAN), a wide area network (WAN), a wired network, a wireless network, or any combination thereof.

[0022] Terminal 130 refers to one or more terminal devices used by a user. In some embodiments, the user includes video game users, virtual reality users, online customer service users, etc. In some embodiments, terminal device 130 may include mobile phone 131, tablet 132, and computer 133, etc.

[0023] In some embodiments, terminal 130 may present a virtual human to the user. A virtual human refers to a digital character, digital image, or intelligent interactive object that interacts with the user. After processing device 110 generates target sound effect data, terminal 130 may play the target sound effect data while presenting the virtual human.

[0024] Storage device 140 can store data or information generated by other devices. In some embodiments, storage device 140 can store the virtual human's output information and environmental information. In some embodiments, storage device 140 can store data and / or information processed by processing device 110, such as virtual human emotional parameters. Storage device 140 may include one or more storage device components, each of which can be a separate device or part of another device. Storage device 140 can be local or implemented via the cloud. In some embodiments, storage device 140 can be implemented on a cloud platform.

[0025] For example, the processing device 110 can obtain the virtual human's output information and environmental information through the network 120, and the storage device 140 can store the virtual human's output information and environmental information. The processing device 110 can also determine the virtual human's emotional parameters based on the output information and environmental information, and then determine the sound effect adjustment parameters, and generate target sound effect data based on the sound effect adjustment parameters. After that, the processing device 110 can transmit the generated target sound effect data to the terminal 130 so that the terminal 130 outputs sound effects that are compatible with the virtual human's current situation, thereby improving the naturalness, immersion and emotional expressiveness of the virtual human's interaction process.

[0026] Figure 2 This is an exemplary block diagram of a virtual human context sound effect generation and adjustment system according to some embodiments of this specification. In some embodiments, such as Figure 2As shown, the virtual human context sound effect generation and adjustment system 200 may include an acquisition module 210, a first determination module 220, a second determination module 230, a generation module 240, and a region processing module 250. In some embodiments, one or more of the above modules may be integrated into the processing device 110.

[0027] In some embodiments, the acquisition module 210 can be configured to acquire the virtual human's output information and environmental information.

[0028] In some embodiments, the first determining module 220 is configured to determine the virtual human's emotional parameters based on the information to be output and environmental information. The information to be output includes at least one of text information that has not yet been actually presented, voice information that has not yet been output, and action information that has not yet been performed.

[0029] In some embodiments, the first determining module 220 is further configured to obtain modal sentiment parameters and the confidence levels corresponding to the modal sentiment parameters. The modal sentiment parameters include at least one of sentiment parameters corresponding to text modality, sentiment parameters corresponding to speech modality, sentiment parameters corresponding to action modality, and sentiment parameters corresponding to environment modality. Based on the confidence levels of the modal sentiment parameters and the changes in historical sentiment parameters, the weights corresponding to the modal sentiment parameters are determined. Based on the modal sentiment parameters and their corresponding weights, the virtual human sentiment parameters are obtained.

[0030] In some embodiments, the second determining module 230 is configured to determine the sound effect adjustment parameters corresponding to the virtual human's emotional parameters.

[0031] In some embodiments, the generation module 240 is configured to generate target sound effect data based on sound effect adjustment parameters.

[0032] In some embodiments, the generation module 240 is further configured to: determine the virtual human's corresponding action state information, historical state information, and actual action events based on the virtual human's action information that has not yet been performed; determine the virtual human's candidate event types for future time windows based on the action state information and historical state information; generate basic sound effect data corresponding to the candidate event types based on the candidate event types, and store the basic sound effect data in a cache; in response to the virtual human's actual action events matching the candidate event types, retrieve the corresponding basic sound effect data from the cache; and adjust the retrieved basic sound effect data based on sound effect adjustment parameters to generate target sound effect data.

[0033] In some embodiments, the region processing module 250 is configured to divide the virtual space in which the virtual person is located into regions to form region boundaries between different regions; set corresponding acoustic parameters for different regions in the virtual space; determine interpolation weights based on the spatial distance information of the virtual person relative to the crossed region boundary in response to the virtual person's coordinates crossing the region boundary; and perform smooth interpolation on the acoustic parameters corresponding to the regions on both sides of the crossed region boundary based on the interpolation weights, and apply the interpolated acoustic parameters to the environmental sound effect output in the virtual space.

[0034] It should be understood that Figure 2 The system and its modules shown can be implemented in various ways. It should be noted that the above description of the virtual human context sound effect generation and adjustment system and its modules is for convenience only and should not limit this specification to the scope of the illustrated embodiments. It is understood that those skilled in the art, after understanding the principle of the system, may arbitrarily combine the various modules or construct subsystems connected to other modules without departing from this principle. In some embodiments, Figure 2 The acquisition module 210, the first determination module 220, the second determination module 230, the generation module 240, and the region processing module 250 disclosed herein can be different modules within a single system, or a single module can implement the functions of two or more of the aforementioned modules. For example, the modules can share a single storage module, or each module can have its own separate storage module. Such variations are all within the scope of protection of this specification.

[0035] Figure 3 This is an exemplary flowchart of a method for generating and adjusting virtual human contextual sound effects according to some embodiments of this specification. In some embodiments, process 300 may be executed by a processing device.

[0036] Step 310: Obtain the virtual human's output information and environmental information.

[0037] Information to be output refers to information that has not yet been actually presented, output, or executed by the virtual human during the interaction process.

[0038] In some embodiments, the information to be output may include at least one of text information that has not yet been actually presented, voice information that has not yet been output, and action information that has not yet been executed. Specifically, text information that has not yet been actually presented includes text content that has not yet been sent to the terminal, or that has not yet been voice-broadcast or presented as subtitles; voice information that has not yet been output includes voice waveforms, voice parameters, or voice synthesis control parameters that have not yet been output and played; and action information that has not yet been executed includes action plans, action segments, or control instructions that have not yet been used to drive the virtual human skeleton, expression controller, or behavior tree.

[0039] Environmental information refers to information related to the environment in which the virtual human is currently located in the virtual space. Environmental information may include multiple environmental parameters, such as at least one of ambient lighting parameters, weather condition parameters, and background noise parameters. Among them, the background noise parameter can be represented by the ambient noise floor value in decibels or other noise characterization parameters.

[0040] In some embodiments, environmental information may further include environmental status markers. Environmental status markers are information markers that characterize environmental categories, regional attributes, or scene states. Environmental status markers may include at least one of weather status markers, region markers, and scene type markers. For example, weather status markers may include "sunny" and "heavy rain," region markers may include "indoor room," "street," "corridor," and "square," and scene type markers may include "quiet environment" and "noisy environment."

[0041] In some embodiments, the processing device can generate text information, voice information, and action information corresponding to the virtual human based on the interaction between the user and the virtual human; before the text information, voice information, and action information are actually presented, broadcast, or executed, at least one of them can be acquired in real time as information to be output. For example, when a user inputs the voice content "How's the weather today?" through a terminal, the processing device can generate the virtual human's reply text, the corresponding voice waveform or voice synthesis control parameters, and the action plan or action control instruction corresponding to the reply text; before it is displayed, broadcast, played, or executed, at least one of them can be acquired in real time as information to be output.

[0042] In some embodiments, the processing device can pre-set ambient lighting parameters, weather state parameters, background noise parameters, and environmental state markers for the environment in the virtual space. During the interaction between the user and the virtual human, the processing device reads the aforementioned ambient lighting parameters, weather state parameters, background noise parameters, and environmental state markers based on the virtual space where the virtual human is currently located, thereby obtaining environmental information. It should be understood that the methods for obtaining the information to be output and the environmental information are not limited to the above embodiments. In some embodiments, the processing device can obtain the information to be output and / or environmental information through at least one of the following methods: local generation, real-time reading, receiving from a terminal, obtaining from a storage device, or obtaining from other devices or servers via a network. The information to be output may originate from at least one of the text information, voice information, and action information generated by the processing device for output by the virtual human; the environmental information may originate from scene configuration data, environmental parameter configuration data, state marker data, or other data used to characterize the environmental state of the virtual human.

[0043] Step 320: Determine the virtual human's emotional parameters based on the information to be output and the environmental information.

[0044] Virtual human emotional parameters refer to parameters that characterize the current emotional state of a virtual human. These parameters can be represented by an emotional tendency vector, which indicates the distribution of the virtual human's current emotional state across multiple preset emotional categories. The emotional tendency vector can include multiple preset emotional categories and their corresponding scores. These preset emotional categories can be pre-set based on prior knowledge; for example, they can include at least one of the following: joy, sadness, tension, anger, and calmness.

[0045] The scores corresponding to each preset emotion category can represent the degree to which the virtual human's current emotional state tends to the corresponding preset emotion category. The higher the emotion score, the more the virtual human's current emotional state is biased towards the corresponding preset emotion category.

[0046] In some embodiments, the processing device can determine corresponding emotion scores for multiple preset emotion categories based on the information to be output and environmental information, according to an emotion processing method, and construct an emotion tendency vector based on the emotion scores corresponding to each preset emotion category to determine the virtual human's emotion parameters. The emotion processing method can be pre-set based on prior knowledge. The emotion scores corresponding to each preset emotion category can be determined by weighted fusion based on multiple modal emotion parameters and the weights corresponding to each modal emotion parameter.

[0047] For example, the virtual human's emotional parameters can be represented by an emotional tendency vector as [pleasure 0.6, sadness 0.1, tension 0.2]. Here, 0.6, 0.1, and 0.2 can be the result of weighted summation and normalization of the scores of the corresponding preset emotional categories in each modality. 0.6, 0.1, and 0.2 respectively represent the degree of virtual human's current emotional state towards the three preset emotional categories of pleasure, sadness, and tension. Since pleasure has the highest score, it indicates that the virtual human's current emotional state is biased towards pleasure, accompanied by a certain degree of tension and a small amount of sadness.

[0048] In some embodiments, the virtual human emotion parameters are determined based on the information to be output and environmental information, including: obtaining modal emotion parameters and the confidence levels corresponding to the modal emotion parameters; determining the weights corresponding to the modal emotion parameters based on the confidence levels of the modal emotion parameters and the changes in historical emotion parameters; and obtaining the virtual human emotion parameters based on the modal emotion parameters and their corresponding weights.

[0049] Modal sentiment parameters refer to the parameters of the emotional state corresponding to different modalities. In some embodiments, modal sentiment parameters may include at least one of the following: sentiment parameters corresponding to text modality, sentiment parameters corresponding to speech modality, sentiment parameters corresponding to action modality, and sentiment parameters corresponding to environment modality.

[0050] A modality refers to different types of information or different channels of information acquisition. For example, a modality can include at least one of text modality, speech modality, action modality, and environment modality. Different modalities correspond to different types of data information. For example, text modality corresponds to text information, speech modality corresponds to speech information, action modality corresponds to action feature information, and environment modality corresponds to environmental state information.

[0051] In some embodiments, modal sentiment parameters can also be represented by sentiment tendency vectors. For example, when the text information corresponding to the text modality is "I'm a little nervous, but I'm really happy!", the sentiment parameters corresponding to the text modality can be represented by sentiment tendency vectors as [pleasure 0.76, nervousness 0.24].

[0052] Confidence level refers to the quantitative representation of the reliability of sentiment parameters.

[0053] In some embodiments, the confidence level of a modal sentiment parameter can reflect the strength of support for sentiment judgment from information corresponding to different modalities. The higher the confidence level of a modal sentiment parameter, the higher its reliability.

[0054] For example, the confidence level of the sentiment parameter corresponding to the text modality can be 0.9, the confidence level of the sentiment parameter corresponding to the action modality can be 0.4, the confidence level of the sentiment parameter corresponding to the speech modality can be 0.7, and the confidence level of the sentiment parameter corresponding to the environment modality can be 0.5. This shows that the reliability of the sentiment parameter corresponding to different modalities can be different.

[0055] In some embodiments, the processing device may use an emotion processing method adapted to different modalities to determine the emotion parameters corresponding to each modality and the confidence level corresponding to the emotion parameters.

[0056] Taking text modality as an example, the processing device can identify one or more sentiment words in the text information corresponding to the text modality, and determine the preset sentiment category and basic intensity score corresponding to each of the one or more sentiment words through a sentiment dictionary; then, it can correct the basic intensity score by combining preset correction rules corresponding to degree adverbs, negation words, transition words and / or punctuation marks in the text information; based on the corrected basic intensity score, it can determine the text sentiment score corresponding to each preset sentiment category; and it can normalize the text sentiment scores corresponding to each preset sentiment category to construct the sentiment tendency vector corresponding to the text modality, and use this sentiment tendency vector as the sentiment parameter corresponding to the text modality. The sentiment dictionary stores the correspondence between sentiment words, preset sentiment categories and basic intensity scores; the preset correction rules can set corresponding correction methods for different rule types. The sentiment dictionary and preset correction rules can be pre-constructed based on historical data or prior knowledge.

[0057] The preset correction rules for degree adverbs may include: when the text information includes degree adverbs such as "very," "especially," or "extremely," the base intensity score corresponding to the sentiment word may be multiplied by a first preset correction coefficient; when the text information includes degree adverbs such as "relatively" or "quite," the base intensity score corresponding to the sentiment word may be multiplied by a second preset correction coefficient; when the text information includes degree adverbs such as "somewhat" or "slightly," the base intensity score corresponding to the sentiment word may be multiplied by a third preset correction coefficient. In some embodiments, the first preset correction coefficient may be greater than the second preset correction coefficient, and the second preset correction coefficient may be greater than 1; the third preset correction coefficient may be less than 1. As an example only, the first preset correction coefficient may be 1.5, the second preset correction coefficient may be 1.2, and the third preset correction coefficient may be 0.7.

[0058] The preset correction rules corresponding to negative words may include: when the text information includes negative words such as "not", "no", or "not so", the polarity of the basic intensity score corresponding to the emotion word is weakened or reversed. Here, polarity weakening means reducing the basic intensity score corresponding to the preset emotion category according to the preset weakening coefficient, and polarity reversal means adjusting the basic intensity score corresponding to the preset emotion category to an emotion category with the opposite polarity to the preset emotion category.

[0059] The preset correction rules for transition words may include: when the text information includes transition words such as "but", "however", or "however", the weight of the basic intensity score of the emotional words in the sentence after the transition is increased according to a preset enhancement coefficient.

[0060] The preset correction rules for punctuation marks may include: multiplying the base intensity score of the sentiment words in the corresponding clause by a fourth preset correction coefficient when the text information includes the punctuation mark “!”; multiplying the base intensity score of the sentiment words in the corresponding clause by a fifth preset correction coefficient when the text information includes the punctuation mark “!!”; and reducing the emotional clarity when the text information includes “…”, and adjusting it downwards when determining the confidence level of the sentiment parameters corresponding to the text modality. In some embodiments, the fourth preset correction coefficient may be less than the fifth preset correction coefficient, and both the fourth and fifth preset correction coefficients are greater than 1.

[0061] In some embodiments, in response to the simultaneous presence of emotional words in the text information that point to preset emotional categories with opposite sentiment tendencies (e.g., when the text information contains both "happy" and "sad," with "happy" corresponding to a pleasant emotion and "sad" corresponding to a sad emotion), the processing device can accumulate the corrected base intensity score corresponding to each emotional word to its respective preset emotional category to obtain text sentiment scores corresponding to multiple preset emotional categories; based on the text sentiment scores corresponding to multiple preset emotional categories, a sentiment tendency vector is formed, and the sentiment tendency vector is used as the sentiment parameter corresponding to the text modality.

[0062] The processing device can determine the confidence level of the sentiment parameters corresponding to a text modality based on the number of sentiment words in the text information, the concentration of the text sentiment score in each preset sentiment category, and the clarity of sentence expression. For example, the more sentiment words in the text information or the more concentrated the text sentiment score is in a certain preset sentiment category, the higher the confidence level of the sentiment parameters corresponding to the text modality; the more negative expressions, contrastive expressions, rhetorical questions, and / or parallel conflict expressions in the text information, or the shorter the text length and / or the more ambiguous the sentence expression, the lower the confidence level of the sentiment parameters corresponding to the text modality.

[0063] Taking speech modality as an example, the processing device can extract one or more acoustic features, such as fundamental frequency, short-time energy, speech rate, and speech signal-to-noise ratio, based on the speech information corresponding to the speech modality. Based on preset acoustic emotion mapping rules, it determines one or more preset emotion categories corresponding to the one or more acoustic features, and the speech emotion scores corresponding to each of the one or more preset emotion categories. The speech emotion scores corresponding to the one or more preset emotion categories are then normalized to construct an emotion tendency vector corresponding to the speech modality, and this emotion tendency vector is used as the emotion parameter corresponding to the speech modality. The preset acoustic emotion mapping rules store the correspondence between acoustic features, preset emotion categories, and speech emotion scores. For example, when the mean fundamental frequency in the speech information increases and the short-time energy surges, the processing device can determine the corresponding preset emotion category as anger and / or excitement using the preset acoustic emotion mapping rules; when the fundamental frequency is relatively flat, the speech rate slows down, and the short-time energy decreases, the corresponding preset emotion category can be determined as sadness and / or calm using the preset acoustic emotion mapping rules. A short-time energy surge refers to a significant increase in the energy value of the speech signal within a short time window compared to the previous time window. For example, a shorter time window can be 20 to 40 milliseconds; when the energy value increases beyond a preset percentage threshold within the shorter time window, a short-term energy surge can be identified. The preset percentage threshold can be pre-set based on prior knowledge; for example, the preset percentage threshold can be 20%.

[0064] The processing device can determine the speech signal-to-noise ratio (SNR) based on the ratio between effective speech components and background noise components in the speech information. Based on the SNR, and through a pre-set correspondence between the SNR and confidence levels, it can determine the confidence level corresponding to the SNR, and use this confidence level as the confidence level of the emotion parameter corresponding to the speech modality. The correspondence between the SNR and confidence levels can be pre-set based on prior knowledge.

[0065] Taking action modality as an example, the processing device can acquire kinematic data of virtual human skeleton nodes based on the action information corresponding to the action modality, and extract at least one kinematic feature from the instantaneous velocity, acceleration, and body bounding box expansion of the end effector; and determine the mapping relationship between one or more kinematic features and multiple preset emotion categories, as well as the action emotion scores corresponding to each of the multiple preset emotion categories, based on preset action emotion mapping rules; normalize the action emotion scores corresponding to the multiple preset emotion categories to construct an emotion tendency vector corresponding to the action modality, and use this emotion tendency vector as the emotion parameter corresponding to the action modality. The preset action emotion mapping rules store the correspondence between kinematic features, preset emotion categories, and action emotion scores.

[0066] The processing device can determine the amplitude, duration, and direction of a virtual human's movements based on the movement information corresponding to a movement modality; and determine the confidence level of the emotional parameters corresponding to the movement modality based on the amplitude, duration, and direction of the movements. For example, if the amplitude of the virtual human's movements is lower than a preset movement threshold and the duration is lower than a preset movement duration, the confidence level of the emotional parameters corresponding to the movement modality can be determined to be low; if the amplitude of the movements is higher than the preset movement threshold, the duration is higher than the preset movement duration, and the direction of the movements is clear, the confidence level of the emotional parameters corresponding to the movement modality can be determined to be high. The preset movement threshold and preset movement duration can be preset based on prior knowledge.

[0067] Taking environmental modalities as an example, the processing device can determine environmental parameters such as ambient lighting parameters, weather state parameters, background noise parameters, and environmental state markers based on the environmental information corresponding to the environmental modality. Then, based on preset environmental sentiment mapping rules, it determines the mapping relationship between multiple environmental parameters and multiple preset sentiment categories, as well as the environmental sentiment scores corresponding to each preset sentiment category. The environmental sentiment scores corresponding to each preset sentiment category are normalized to construct a sentiment tendency vector corresponding to the environmental modality, and this sentiment tendency vector is used as the sentiment parameter corresponding to the environmental modality. The preset environmental sentiment mapping rules store the correspondence between environmental parameters, preset sentiment categories, and environmental sentiment scores.

[0068] The processing device can determine the confidence level of the sentiment parameter corresponding to an environmental modality based on the degree of change of environmental parameters in the environmental information. For example, when at least one of the environmental illumination parameters, weather state parameters, background noise parameters, and environmental state markers changes significantly, the confidence level of the sentiment parameter corresponding to the environmental modality can be determined to be high; when at least one of the above changes is small, the confidence level of the sentiment parameter corresponding to the environmental modality is low. A significant change refers to at least one environmental parameter in the environmental information changing by a magnitude reaching a preset change threshold relative to its corresponding parameter value at a previous moment or time period, and / or a clear switch in the environmental state marker; a small change refers to at least one environmental parameter in the environmental information changing by a magnitude not reaching the preset change threshold relative to its corresponding parameter value at a previous moment or time period, and no clear switch in the environmental state marker. The preset change threshold can be pre-set based on prior knowledge.

[0069] In some embodiments, the processing device can iteratively update the emotion processing method according to a preset cycle, wherein each preset cycle corresponds to one round of update iteration, including: in each round of update iteration, determining basic emotion parameters based on historical modal information of the previous preset cycle through an emotion parameter model, wherein the emotion parameter model is a machine learning model; determining historical emotion parameters based on historical modal information; determining the error between the historical emotion parameters and the basic emotion parameters; determining the item to be corrected that causes the error in response to the error continuously exceeding a preset threshold for a preset number of times; and correcting the item to be corrected based on the basic emotion parameters to update the emotion processing method.

[0070] The preset cycle refers to the time period for checking and updating the rules and corresponding relationships within the emotion processing method. The preset cycle can be set by day, week, or a fixed number of interaction rounds.

[0071] Historical modal information refers to modal information collected and stored within the previous preset period. For example, historical modal information may include at least one of historical text information, historical voice information, historical action information, and historical environmental information.

[0072] Baseline sentiment parameters refer to the reference parameters used to calibrate sentiment parameters. Baseline sentiment parameters can include at least one of the following: baseline sentiment parameters corresponding to text information, baseline sentiment parameters corresponding to speech information, baseline sentiment parameters corresponding to action information, and baseline sentiment parameters corresponding to environmental information. Baseline sentiment parameters can also be represented by sentiment tendency vectors.

[0073] In some embodiments, the processing device may determine basic sentiment parameters based on historical modality information through a sentiment parameter model, wherein the sentiment parameter model is a machine learning model.

[0074] In some embodiments, the sentiment parameter model can be a machine learning model, such as a neural network (NN).

[0075] In some embodiments, the input to the sentiment parameter model may include historical text information, historical speech information, historical action information, and historical environment information at the same historical moment, and the output of the sentiment parameter model may be the basic sentiment parameters corresponding to the aforementioned historical text information, historical speech information, historical action information, and historical environment information, respectively.

[0076] In some embodiments, the processing device can acquire multiple labeled training samples to form a training sample set, and perform multiple rounds of iteration based on the training sample set. Specifically, the processing device can extract multiple historical interaction segments from historical interaction data as multiple training samples. Each training sample includes historical text information, historical speech information, historical action information, and historical environment information corresponding to the same historical moment or historical interaction segment. The labels corresponding to the training samples are the basic sentiment parameters corresponding to the historical text information, the historical speech information, the historical action information, and the historical environment information. The labels corresponding to the training samples can be manually acquired and labeled based on historical data.

[0077] At least one iteration includes: selecting one or more training samples from the training sample set; inputting the one or more training samples into the initial sentiment parameter model to obtain the model prediction output corresponding to the one or more training samples; substituting the model prediction output corresponding to the one or more training samples, and the labels corresponding to the one or more training samples, into the formula of a predefined loss function to calculate the value of the loss function; iteratively updating the model parameters in the initial sentiment parameter model based on the value of the loss function until the iteration termination condition is met, at which point the iteration ends, and the trained sentiment parameter model is obtained. The iterative updating of the model parameters of the initial sentiment parameter model can be performed using various methods, such as gradient descent.

[0078] Historical sentiment parameters refer to the modal sentiment parameters corresponding to a modality within a historical time window. In some embodiments, the processing device can determine the historical sentiment parameters corresponding to a modality within a historical time window based on historical text information, historical speech information, historical action information, and historical environmental information. The method for determining the historical sentiment parameters corresponding to a modality within a historical time window is similar to the process of obtaining modal sentiment parameters described above; relevant technical details can be found above and will not be repeated here.

[0079] In some embodiments, the processing device can determine an error value based on the difference between the sentiment tendency vector corresponding to historical sentiment parameters and the sentiment tendency vector corresponding to basic sentiment parameters. The difference can be calculated using Euclidean distance, cosine difference, and / or one-dimensional absolute difference.

[0080] The terms to be corrected refer to the rule content in the emotion processing method that needs to be adjusted or updated. For example, terms to be corrected may include rule items in the text modality used to determine the preset emotion category, base intensity score, preset correction rule, and text modality confidence level for emotion words; rule items in the speech modality used to determine one or more acoustic features corresponding to one or more preset emotion categories, speech emotion score, and speech modality confidence level; rule items in the action modality used to determine kinematic features corresponding to preset emotion categories, action emotion score, preset action threshold, preset action duration, and action modality confidence level; rule items in the environment modality used to determine environmental parameters corresponding to preset emotion categories, environment emotion score, preset environment emotion mapping rule, and environment modality confidence level; and rule items used to determine the weights of each modality. Here, a rule item refers to the specific rule content or rule unit in the emotion processing method. For example, a rule item may include word mapping relationships, preset correction rules, emotion mapping rules, confidence level determination rules, and / or weight determination rules in the emotion dictionary.

[0081] In some embodiments, in response to the error between historical sentiment parameters and basic sentiment parameters exceeding a preset threshold for a preset number of consecutive times, a set of rules that are actually hit in the sentiment processing method used to determine historical sentiment parameters is determined; and each rule item in the rule set is tentatively adjusted to obtain the adjustment result corresponding to each rule item; based on the adjustment result corresponding to each rule item, the adjusted historical sentiment parameters are re-determined, and the error between the adjusted historical sentiment parameters and basic sentiment parameters is determined; if the error decreases the most after adjusting a certain rule item, then the rule item is determined as the item to be corrected that causes the error. Here, the preset number of times refers to the number of times the error between historical sentiment parameters and basic sentiment parameters exceeds a preset threshold consecutively; each error judgment can be based on historical text information, historical voice information, historical action information, and historical environment information corresponding to a historical moment, or it can be based on historical text information, historical voice information, historical action information, and historical environment information corresponding to a historical interaction segment.

[0082] Among them, the preset threshold and preset number of times can be preset based on prior knowledge. For example, the preset threshold can be 0.4 and the preset number of times can be 3. The set of rules that are actually hit refers to the set of rule items that are actually called or applied in the emotion processing method during the process of determining historical emotion parameters. The tentative adjustment refers to temporarily deleting, replacing and / or adjusting the parameters of a single rule item to observe the impact of the change of the rule item on the error.

[0083] Taking text modality as an example, the set of rules actually applied can include the matched sentiment dictionary entries, rules corresponding to degree adverbs, rules corresponding to negation words, and rules corresponding to transition words. The processing device can make tentative adjustments to each rule item in the rule set and redetermine the historical sentiment parameters based on the sentiment processing method after the tentative adjustments; if the error between the historical sentiment parameters and the basic sentiment parameters decreases the most after making tentative adjustments to the preset sentiment category mapping relationship and / or basic intensity score corresponding to a certain sentiment word, then the preset sentiment category mapping relationship and / or basic intensity score currently corresponding to that sentiment word is determined as the item to be corrected.

[0084] In some embodiments, the processing device can correct the item to be corrected based on the basic sentiment parameters to reduce the error between the historical sentiment parameters and the basic sentiment parameters. The correction may include updating the preset sentiment category mapping relationship, the basic intensity score, the threshold range, the confidence determination rule, the weight determination rule, and / or logical conditions.

[0085] Taking text modality as an example, if the preset emotion category corresponding to "happy" in the emotion dictionary is "joy," and the corresponding base intensity score is 0.60, for a certain historical text message "Although I'm a little nervous, I'm really happy!", the processing device can determine the historical emotion parameters corresponding to this historical text message as [joy 0.58, nervous 0.42] based on the current emotion processing method, while the base emotion parameters corresponding to the historical text message are [joy 0.78, nervous 0.22]. If adjusting the base intensity score corresponding to "happy" from 0.60 to 0.85 results in a greater decrease in error than the decrease in error after tentatively adjusting other rule items, then the current base intensity score of 0.60 corresponding to "happy" can be identified as a term to be corrected and updated to 0.85.

[0086] In some embodiments of this specification, by introducing basic emotional parameters as calibration benchmarks and correcting the terms to be corrected, the risk of mismatch, drift and aging after long-term use of fixed rules can be reduced, the long-term accuracy, stability and adaptability of the determination results of emotional parameters of each modality and the determination results of emotional parameters of virtual humans can be improved, and the target sound effect data generated subsequently can better fit the real emotional expression of virtual humans and actual interaction scenarios.

[0087] The changes in historical sentiment parameters refer to the degree of fluctuation in historical sentiment parameters over time.

[0088] In some embodiments, the processing device can extract historical sentiment parameters corresponding to multiple consecutive historical moments within a historical time window, and determine the degree of fluctuation of the historical sentiment parameters within the historical time window, so as to determine the changes in the historical sentiment parameters.

[0089] In some embodiments, the processing device can determine the degree of fluctuation of historical sentiment parameters within a historical time window in various ways. For example, the processing device can determine the degree of fluctuation of historical sentiment parameters within a historical time window based on the variance of the dominant sentiment category scores corresponding to multiple consecutive historical moments within the historical time window. Here, the dominant sentiment category refers to the preset sentiment category with the highest score in the sentiment tendency vector.

[0090] For example, at three historical moments within a historical time window, the historical sentiment parameters corresponding to a certain modality are represented as: [pleasure 0.76, tension 0.24], [pleasure 0.72, tension 0.28], and [pleasure 0.80, tension 0.20]. In these three historical moments, the dominant sentiment category is pleasure, with scores of 0.76, 0.72, and 0.80, respectively. The processing device can determine the variances of 0.76, 0.72, and 0.80 to characterize the degree of fluctuation of the historical sentiment parameters within the historical time window. A smaller variance indicates less change in the historical sentiment parameters within the historical time window; a larger variance indicates greater fluctuation in the historical sentiment parameters within the historical time window.

[0091] In some embodiments, the processing device can also determine the changes in historical sentiment parameters based on the gradient of change between sentiment tendency vectors corresponding to adjacent historical moments within a historical time window. For example, at two adjacent historical moments, the historical sentiment parameters corresponding to a certain modality are represented as [pleasure 0.76, tension 0.24] and [pleasure 0.72, tension 0.28], respectively. The processing device can determine the gradient of change between the two sentiment tendency vectors based on the difference between the scores corresponding to each preset sentiment category. When the difference is small, it can be determined that the degree of change in the historical sentiment parameters is small; when the difference is large, it can be determined that the degree of change in the historical sentiment parameters is large.

[0092] In some embodiments, the processing device can also count the number of times the dominant emotion category in the historical emotion parameters switches within a historical time window to determine the changes in the historical emotion parameters. For example, at three historical moments within a historical time window, the historical emotion parameters corresponding to a certain modality are represented as: [pleasure 0.76, tension 0.24], [tension 0.55, pleasure 0.45], and [sadness 0.60, pleasure 0.25, tension 0.15]. At these three historical moments, the dominant emotion categories are pleasure, tension, and sadness in sequence. The processing device can count two switches in the dominant emotion category and determine the changes in the historical emotion parameters based on the number of switches.

[0093] In some embodiments, the processing device can determine the basic weights of modal sentiment parameters based on the confidence level of the modal sentiment parameters; correct the basic weights based on the changes in historical sentiment parameters; and normalize the corrected basic weights to determine the weights corresponding to the modal sentiment parameters. The direction of the basic weight correction is inversely related to the degree of change in historical sentiment parameters, and the magnitude of the basic weight correction can be determined based on the degree of change in historical sentiment parameters according to a preset mapping relationship. For example, in response to a large change in the historical sentiment parameters of a certain modality within a historical time window, a larger reduction can be determined according to the preset mapping relationship to reduce the basic weight of the modal sentiment parameter corresponding to that modality; conversely, in response to a small change in the historical sentiment parameters of a certain modality within a historical time window, a larger increase can be determined according to the preset mapping relationship to increase the basic weight of the modal sentiment parameter corresponding to that modality. The preset mapping relationship can be determined based on prior knowledge.

[0094] In some embodiments, the processing device can determine the basic weights of modal sentiment parameters in various ways. For example, the processing device can use the confidence scores corresponding to the sentiment parameters of the text modality, the speech modality, the action modality, and the environment modality, respectively, as the basic weights of each modal sentiment parameter. Alternatively, the processing device can determine the basic weights of each modal sentiment parameter based on the confidence scores corresponding to the sentiment parameters of the text modality, the speech modality, the action modality, and the environment modality, through a weight mapping relationship. This weight mapping relationship includes the correspondence between the confidence scores of each modal sentiment parameter and the basic weights, and the weight mapping relationship can be predetermined through prior knowledge or historical data.

[0095] In some embodiments, the processing device can perform weighted fusion based on the sentiment parameters corresponding to the text modality, the speech modality, the action modality, and the environment modality, as well as the weights corresponding to the sentiment parameters of each modality, to determine the virtual human's sentiment parameters. Specifically, the processing device can perform weighted summation on the scores corresponding to the same preset sentiment category for the sentiment tendency vector in each modality's sentiment parameters according to their respective weights, and normalize the weighted summation result to form the sentiment tendency vector corresponding to the virtual human, and use the sentiment tendency vector corresponding to the virtual human as the virtual human's sentiment parameters.

[0096] In some embodiments of this specification, by determining the emotional parameters corresponding to the text modality, speech modality, action modality, and environment modality respectively, and obtaining the virtual human's emotional parameters based on the modal emotional parameters and their corresponding weights, it is possible to reduce the emotional judgment bias caused by single modality misjudgment or fixed weight fusion, improve the accuracy and robustness of the virtual human's emotional parameters, and make the subsequently generated sound effects more in line with the virtual human's real intentions.

[0097] Step 330: Determine the sound effect adjustment parameters corresponding to the virtual human's emotional parameters.

[0098] Sound effect adjustment parameters refer to parameter information that characterizes the direction, degree, or method of adjustment of basic sound effect data. For example, sound effect adjustment parameters may include at least one of the following: parameters for adjusting acoustic performance (such as volume, pitch, rhythm density, frequency band enhancement parameters, reverberation intensity, etc.), parameters for adjusting spatial performance (such as sound image position, spatial diffusion degree, etc.), and parameters for adjusting temporal performance (such as start trigger time, duration, etc.).

[0099] Basic sound effect data refers to the initial audio data of the virtual human's contextual sound effects. For example, basic sound effect data may include at least one of the following: action sound effects, environmental sound effects, emotional sound effects, and interactive sound effects corresponding to the virtual human's current context.

[0100] In some embodiments, the processing device can determine basic sound effect data based on text information that has not yet been actually presented, voice information that has not yet been output, action information that has not yet been executed, and environmental information in the information to be output. For example, in response to action information that has not yet been executed representing a foot contacting the ground, the basic sound effect data corresponding to the action information that has not yet been executed is determined to be a footstep sound or a landing sound; in response to action information that has not yet been executed representing an object contacting, the corresponding basic sound effect data is determined to be a collision sound; in response to the environmental state of the environmental information being marked as a rainy street scene, the basic sound effect data corresponding to the environmental information is determined to be rain sounds and street ambient sounds; in response to text information that has not yet been actually presented representing an interactive prompt, the corresponding basic sound effect data is determined to be a prompt tone or a confirmation tone.

[0101] In some embodiments, the processing device can determine the dominant emotion category of the virtual human based on the virtual human's emotion parameters; and based on the dominant emotion category and intensity of the virtual human, determine the corresponding sound effect adjustment parameters using a sound effect adjustment lookup table. The sound effect adjustment lookup table can include the correspondence between different emotion categories, different emotion intensities, and sound effect adjustment parameters. The sound effect adjustment lookup table can be pre-constructed based on historical data or prior knowledge.

[0102] For example, in response to the virtual human's emotional parameters indicating that the dominant emotional category is tension and the emotional intensity is high, the sound effect adjustment parameters can be determined as increasing the proportion of high-frequency components, accelerating the rhythm of pulse-like sound effects, and increasing the reverberation parameters of environmentally oppressive sound effects. In response to the virtual human's emotional parameters indicating that the dominant emotional category is calm and the emotional intensity is low, the sound effect adjustment parameters can be determined as reducing the density of action sound effects, reducing high-frequency stimulation, and enhancing soft environmental background sounds. The processing device can determine the emotional intensity corresponding to the dominant emotional category based on the score corresponding to the dominant emotional category in the emotional tendency vector. For example, when the virtual human's emotional parameters are represented by the emotional tendency vector as [tension 0.8, sadness 0.1], the dominant emotional category of the virtual human can be determined to be tension; and based on the score corresponding to tension, the emotional intensity corresponding to tension can be determined to be high.

[0103] It should be understood that the determination of the sound effect adjustment parameters corresponding to the virtual human's emotional parameters is not limited to using a sound effect adjustment lookup table. In some embodiments, the processing device can also determine the corresponding sound effect adjustment parameters based on preset rules and / or preset calculation methods, according to the virtual human's dominant emotional category and emotional intensity. For example, the processing device can determine the sound effect adjustment parameters based on the parameter adjustment direction corresponding to different emotional categories and the parameter adjustment degree corresponding to different emotional intensities.

[0104] Step 340: Generate target sound effect data based on sound effect adjustment parameters.

[0105] Target sound effect data refers to the audio data used to output contextual sound effects for the virtual human. Target sound effect data can be a complete audio stream, audio clips, parametric synthesis control sequences, or a combination thereof. For example, target sound effect data may include at least one of the following: motion sound effects corresponding to the virtual human's actual action events, environmental sound effects corresponding to the current context, emotional sound effects, and interactive sound effects. More information about actual action events can be found in the relevant descriptions later in this specification.

[0106] In some embodiments, the processing device can generate target sound effect data in a variety of ways. For example, the processing device can generate target sound effect data by using a sound effect synthesis engine to control the spatial sound emission direction, diffusion range, volume, and pitch of the basic sound effect data in real time, based on parameters for adjusting acoustic performance, spatial performance, and temporal performance in the sound effect adjustment parameters.

[0107] Figure 4 This is an exemplary schematic diagram illustrating the determination of target sound effect data according to some embodiments of this specification. In some embodiments, such as Figure 4 As shown, the processing device can also determine the virtual human's corresponding action state information 421, historical state information 422, and actual action event 423 based on the virtual human's action information 410 that has not yet been performed; determine the virtual human's candidate event type 430 in the future time window based on the action state information 421 and historical state information 422; generate basic sound effect data 440 corresponding to the candidate event type based on the candidate event type 430, and store the basic sound effect data 440 in the cache; in response to the virtual human's actual action event 423 matching the candidate event type 430, call the corresponding basic sound effect data 440 from the cache; and adjust the called basic sound effect data based on the sound effect adjustment parameters to generate target sound effect data 450.

[0108] Action state information refers to data that characterizes the execution state of a virtual human's actions at the current moment. For example, action state information may include the virtual human's skeletal node positions, displacement, orientation, velocity, and acceleration at the current moment.

[0109] Historical state information refers to historical data that characterizes the changes in the execution state of a virtual human within a historical time window. For example, historical state information may include the skeletal node positions, displacements, velocities, and accelerations of the virtual human at multiple consecutive historical moments within the historical time window.

[0110] Actual action events refer to the action events that occur when a virtual human is actually performing actions. Actual action events can include multiple events, such as contact events, interaction events, landing events, turning events, and collision events.

[0111] In some embodiments, the processing device can parse the action plan, action fragment, or control instruction contained in the action information that has not yet been executed to determine the action status information corresponding to the action information that has not yet been executed; and read and parse the action information that the virtual person has not yet executed within the historical time window to determine the historical status information.

[0112] In some embodiments, the processing device determines the actual action event based on action information that has not yet been executed, using a preset action type mapping table. The preset action type mapping table can include the correspondence between different action information and different actual action events, and can be pre-constructed based on prior knowledge. For example, in response to action information that has not yet been executed, representing foot contact with the ground, the processing device can determine that the actual action event corresponding to this action information is a contact event using the preset action type mapping table.

[0113] The candidate event type for the future time window refers to the action events that the virtual human may perform within the future time window. Candidate event types can include multiple event types, such as candidate contact events, candidate interaction events, candidate landing events, candidate turning events, and candidate collision events.

[0114] In some embodiments, the duration of the future time window is determined based on a reference rate.

[0115] Reference rate refers to rate information related to the degree of change in the virtual human's actions and the degree of change in the environment. In some embodiments, the reference rate may include one or more of the rate of change in the virtual human's action state and the rate of change in the environment.

[0116] The rate of change of motion state refers to the rate at which the motion state of a virtual human changes per unit time. For example, the rate of change of motion state may include at least one of the following: rate of change of velocity, rate of change of acceleration, rate of change of direction, and frequency of posture switching.

[0117] In some embodiments, the processing device can determine the rate of change of the action state based on the action state information. The rate of change of the action state can be determined based on the action state information corresponding to adjacent moments; adjacent moments include the current moment and the previous historical moment, as well as two consecutive historical moments.

[0118] For example, the rate of change of velocity can be determined by the ratio of the velocity difference between adjacent moments to the time interval; the rate of change of acceleration can be determined by the ratio of the acceleration difference between adjacent moments to the time interval; the rate of change of direction can be determined by the ratio of the orientation angle difference between adjacent moments to the time interval; and the attitude switching frequency can be determined by the number of times the action category or attitude state is switched within the historical time window.

[0119] The rate of environmental change refers to information that characterizes how quickly the environment in which a virtual human exists changes per unit of time. For example, the rate of environmental change may include at least one of the following: the rate of change in illuminance, the frequency of weather state switching, the rate of change in ambient noise level in decibels, and the frequency of area switching.

[0120] In some embodiments, the processing device may determine the rate of environmental change based on environmental information. The rate of environmental change may be determined based on environmental information corresponding to adjacent time points.

[0121] For example, the rate of change of illuminance can be determined based on the ratio of the difference in ambient illuminance parameters between adjacent moments to the time interval; the frequency of weather state switching can be determined based on the number of times the weather state markers in the environmental information change within adjacent moments; the rate of change of ambient background noise in decibels can be determined based on the ratio of the difference in background noise parameters between adjacent moments to the time interval; and the frequency of area switching can be determined based on the number of times the virtual human crosses different area boundaries within adjacent moments. More information on area boundaries can be found in [link to relevant documentation]. Figure 5 And its explanation.

[0122] In some embodiments, the processing device first determines an action correction coefficient and an environment correction coefficient based on the rate of change of the action state and the rate of change of the environment, respectively; and determines the duration of the future time window based on the product of the base prediction duration, the action correction coefficient, and the environment correction coefficient. As an example only, the duration of the future time window = base prediction duration × action correction coefficient × environment correction coefficient.

[0123] The base prediction duration refers to the baseline time length for predicting the future when both the action state and the environment state are relatively stable. The value of the base prediction duration can be predetermined based on the cycle length of common virtual human actions and the inherent delay of the sound effect generation link. For example, the base prediction duration can be set to 1.0s (seconds).

[0124] In some embodiments, the processing device can pre-set corresponding motion correction coefficients for different rates of change of motion states and corresponding environment correction coefficients for different rates of change of environment, based on prior knowledge. The faster the rate of change of motion state, the smaller the motion correction coefficient; the faster the rate of change of environment, the smaller the environment correction coefficient. For example, in response to the rate of change of motion state indicating that the virtual human is in a stationary state, the motion correction coefficient can be pre-set to 1.2; in response to the rate of change of motion state indicating that the virtual human is in a slow walking state, the motion correction coefficient can be pre-set to 1.0; in response to the rate of change of motion state indicating that the virtual human is in a running state, the motion correction coefficient can be pre-set to 0.6; in response to the rate of change of environment indicating that the environment is in a stable state, the environment correction coefficient can be pre-set to 1.2; in response to the rate of change of environment indicating that the environment is in a slowly changing state, the environment correction coefficient can be pre-set to 1.0.

[0125] In some embodiments of this specification, by determining the future time window based on a reference rate, the prediction range of action intent prediction can be adaptively adjusted according to changes in the virtual human's actions and the environment. When the virtual human's action state changes rapidly and / or the environment changes rapidly, the accuracy of candidate event type prediction can be improved by shortening the future time window, reducing the accumulation of errors caused by overly distant predictions. When the virtual human's action state and the environment state are relatively stable, the future time window can be extended to provide more preparation time for the pre-generation and caching of basic sound effect data. Thus, a more reasonable balance can be achieved between prediction accuracy and advance preparation time, improving cache hit rate and real-time response capability in dynamic scenes.

[0126] In some embodiments, the processing device can predict candidate event types for future time windows based on action state information and historical state information using an action intent prediction model. The action intent prediction model can be a machine learning model, such as a Long Short-Term Memory (LSTM) network.

[0127] In some embodiments, the input to the action intent prediction model can be a historical action state sequence, and the output of the action intent prediction model can include at least one candidate event type arranged chronologically within a future time window. The processing device can construct a historical action state sequence by arranging the action state information and historical state information corresponding to multiple consecutive historical moments within the historical time window in chronological order.

[0128] In some embodiments, the processing device can acquire multiple labeled training samples to form a training sample set, and perform multiple rounds of iteration based on the training sample set. The training samples include sequences of historical action states, and the labels corresponding to the training samples are candidate event types.

[0129] In some embodiments, the processing device can arrange historical state information in chronological order to construct a sample historical action state sequence; at the same time, it can obtain a future time window corresponding to the sample historical action state sequence and arrange the candidate event types within the future time window in chronological order as labels corresponding to the training samples.

[0130] At least one iteration includes: selecting one or more training samples from the training sample set; inputting the one or more training samples into the initial action intent prediction model to obtain the model prediction output corresponding to the one or more training samples; substituting the model prediction output corresponding to the one or more training samples, and the labels corresponding to the one or more training samples, into the formula of a predefined loss function to calculate the value of the loss function; iteratively updating the model parameters in the initial action intent prediction model based on the value of the loss function until the iteration termination condition is met, at which point the iteration ends, and the trained action intent prediction model is obtained. The iterative updating of the model parameters of the initial action intent prediction model can be performed using various methods, such as gradient descent.

[0131] In some embodiments, the processing device can determine the basic sound effect data in a variety of ways. For example, the processing device can generate basic sound effect data using sound effect generation methods of varying complexity based on the device's current processing capabilities and the importance of the sound sources corresponding to the candidate event types.

[0132] In some embodiments, the processing device may also set multiple sound effect generation levels, each corresponding to a different sound effect generation algorithm; assign corresponding sound effect generation levels to candidate event types based on the device's computing power load and the acoustic importance of the sound source; and generate basic sound effect data based on the sound effect generation algorithm corresponding to the sound effect generation level, and store the basic sound effect data in the cache area.

[0133] Sound effect generation level refers to the hierarchical information representing the complexity and processing precision of the sound effect generation algorithm used in the basic sound effect data generation process. For example, sound effect generation levels can include high-level, medium-level, and low-level. Among them, high-level refers to sound effect generation levels that use higher computational complexity, higher rendering precision, and higher spatial refinement; medium-level refers to sound effect generation levels that use medium computational complexity, medium rendering precision, and medium spatial refinement; and low-level refers to sound effect generation levels that use lower computational complexity, lower rendering precision, and lower spatial refinement.

[0134] In some embodiments, multiple sound effect generation levels correspond to different sound effect generation algorithms, and the distinction between high-level, mid-level, and low-level can be determined based on the differences in computational complexity, rendering accuracy, and spatial refinement of the processing methods covered by the corresponding sound effect generation algorithms. Specifically, the sound effect generation algorithms corresponding to high-level can include sound effect generation algorithms covering the range of full-parameter real-time sound effect synthesis, fine-grained spectrum shaping, and high-precision 3D reverberation rendering; the sound effect generation algorithms corresponding to mid-level can include sound effect generation algorithms covering the range of simplified acoustic models, basic spatial positioning, and limited parameter adjustment; and the sound effect generation algorithms corresponding to low-level can include sound effect generation algorithms covering the range of basic preset audio segments as well as volume adjustment, pitch adjustment, or simple equalization adjustment.

[0135] Device computing load refers to the information on the utilization of computing resources and the level of processing pressure during audio effects processing. For example, device computing load can include high load, low load, and / or medium load states. Device computing load can be comprehensively evaluated using indicators such as CPU utilization, GPU utilization, audio thread latency, memory usage, and audio buffer pressure.

[0136] In some embodiments, the processing device can acquire values ​​corresponding to various indicators of the device's computing load; when the values ​​of at least two indicators exceed their respective preset load thresholds, the device's computing load can be determined to be in a high-load state; when the values ​​of at least two indicators are lower than their respective preset load thresholds, the device's computing load can be determined to be in a low-load state; when the device's computing load neither meets the criteria for a high-load state nor the criteria for a low-load state, the device's computing load can be determined to be in a medium-load state. The preset load thresholds can be pre-constructed based on prior knowledge.

[0137] A sound source refers to the source corresponding to the sound output in the environment in which the virtual human exists. For example, the sound effect category corresponding to a sound source can include at least one of the following: sources corresponding to action sound effects, sources corresponding to environmental sound effects, sources corresponding to emotional sound effects, and sources corresponding to interactive sound effects. Among them, sources corresponding to action sound effects can include foot contact with the ground, waving actions, contact collisions, and clothing friction; sources corresponding to environmental sound effects can include wind, rain, street environments, and indoor air conditioning equipment; sources corresponding to emotional sound effects can include emotional expressions such as tension, pressure, and relaxation; and sources corresponding to interactive sound effects can include button clicks, prompt triggers, and confirmation operations.

[0138] The acoustic importance of a sound source refers to evaluation information characterizing the priority and processing sophistication of the sound source in the current environment. The acoustic importance of a sound source can be categorized as high, medium, and low. In some embodiments, the processing device can determine the acoustic importance of a sound source based on the spatial distance between the sound source and the virtual human and the sound effect category to which the sound source belongs. For example, in response to the spatial distance between the sound source and the virtual human being being relatively close, the sound source can be determined to have high acoustic importance; in response to the spatial distance between the sound source and the virtual human being being relatively far, and the sound effect category to which the sound source belongs being the source corresponding to an ambient sound effect, the sound source can be determined to have low acoustic importance.

[0139] In some embodiments, the processing device can determine the acoustic importance of the sound source corresponding to a candidate event type based on the candidate event type; then, based on the device's computing power load and acoustic importance, it can determine the sound effect generation level corresponding to the candidate event type by querying a sound effect generation level lookup table. The sound effect generation level lookup table can include the correspondence between the acoustic importance of the candidate event type, the device's computing power load, and the sound effect generation level. The sound effect generation level lookup table can be pre-constructed based on historical data or prior knowledge. For example, in response to a candidate event type being a candidate landing event or a candidate collision event, the acoustic importance of the candidate event type can be determined to be high, and a high level can be assigned to the candidate event type when the device's computing power load is normal or medium; in response to a high device computing power load, a medium level can be assigned to the candidate event type. Similarly, in response to a candidate event type having medium or low acoustic importance, the corresponding sound effect generation level can also be determined by combining the device's computing power load with the sound effect generation level lookup table.

[0140] In some embodiments, in response to a candidate event type being assigned to a high-level position, the processing device may invoke the sound effect generation algorithm corresponding to the high-level position to perform full-parameter real-time sound effect synthesis, fine-grained spectrum shaping, and high-precision 3D reverberation rendering on the initial audio corresponding to the candidate event type, thereby generating basic sound effect data corresponding to the candidate event type; in response to a candidate event type being assigned to a mid-level position, the processing device may invoke the sound effect generation algorithm corresponding to the mid-level position to perform simplified acoustic model processing, basic spatial positioning, and finite parameter adjustment on the initial audio corresponding to the candidate event type, thereby generating basic sound effect data corresponding to the candidate event type; in response to a candidate event type being assigned to a low-level position, the processing device may invoke the sound effect generation algorithm corresponding to the low-level position, using a preset audio segment as the initial audio corresponding to the candidate event type, and adjusting the volume, pitch, or simple equalization of the initial audio, thereby generating basic sound effect data corresponding to the candidate event type. Here, the initial audio corresponding to a candidate event type refers to the original audio segment corresponding to the candidate event type. The processing device may pre-generate corresponding initial audio for each candidate event type based on at least one candidate event type. The sound effect generation algorithms corresponding to high-level, mid-level, and low-level audio processing technologies can include existing audio processing algorithms used for sound effect synthesis, audio rendering, spatial positioning, and audio adjustment.

[0141] In some embodiments of this specification, by setting multiple sound effect generation levels and generating basic sound effect data corresponding to candidate event types based on the sound effect generation algorithms corresponding to the sound effect generation levels, low-level sound effect generation algorithms can be used for candidate event types with low acoustic importance when the device's computing power load is high, thereby reducing the computational pressure on the overall sound effect processing chain; at the same time, high-level sound effect generation algorithms can still be preferentially allocated to candidate event types with high acoustic importance, thereby ensuring the generation quality of basic sound effect data corresponding to key sound sources and important action events.

[0142] In some embodiments, the processing device can compare the actual action event of the virtual human with candidate event types within a future time window; in response to the actual action event of the virtual human matching one of the candidate event types, it retrieves the basic sound effect data corresponding to that candidate event type from the cache. For example, if the actual action event is a contact event, and the candidate event types include candidate contact events, then it is determined that the actual action event matches the candidate event type.

[0143] In some embodiments, such as Figure 4As shown, when the actual action event 423 of the virtual human does not match multiple events in the candidate event type 430, the basic sound effect data 440 corresponding to the candidate event type in the cache can be cleared, and the candidate event type 430 of the virtual human in the future time window can be re-determined. The basic sound effect data 440 corresponding to the candidate event type can be regenerated and stored in the cache. For more details on how to determine the candidate event type of the virtual human in the future time window and how to generate the basic sound effect data corresponding to the candidate event type, please refer to the relevant description above, which will not be repeated here.

[0144] In some embodiments, the processing device can adjust the called basic sound effect data based on sound effect adjustment parameters to generate target sound effect data. The processing device can adjust the volume and pitch of the basic sound effect data based on parameters that adjust acoustic performance, such as gaining or attenuating the basic sound effect data and raising or lowering its pitch; it can also adjust the start offset and duration of the basic sound effect data based on parameters that adjust timing performance, such as aligning the start position of the basic sound effect data with the triggering time of the actual action event by adjusting the start position of the sound effect data, and adjusting the playback duration to match the duration of the actual action event by truncating or fading out the tail of the basic sound effect data; it can also adjust the sound image position, spatial diffusion, and reverberation of the basic sound effect data based on parameters that adjust spatial performance, thereby generating target sound effect data corresponding to the actual action event.

[0145] In some embodiments of this specification, by predicting candidate event types within a future time window and pre-caching the basic sound effect data corresponding to the candidate event types, when a virtual human triggers an actual action event, it can match the candidate event type corresponding to the actual action event, directly call the corresponding basic sound effect data from the cache, and adjust the called basic sound effect data in conjunction with sound effect adjustment parameters to generate the target sound effect data. This shortens the response latency between the triggering of the actual action event and the output of the sound effect, reduces real-time computing pressure, and ensures consistency between the target sound effect data and the actual action event, thereby improving the audio-visual synchronization effect and overall immersion in fast-paced action scenes.

[0146] In some embodiments of this specification, by acquiring the virtual human's output information and environmental information in real time, determining the virtual human's emotional parameters based on the output information and environmental information, and further determining the corresponding sound effect adjustment parameters to generate target sound effect data, the sound effects can maintain a higher consistency with the virtual human's text information, voice information, action information, and environmental information in time, and be more coordinated in emotional expression, thereby improving the virtual human's situational expressiveness, interaction naturalness, and overall immersion.

[0147] Figure 5 This is an exemplary flowchart illustrating the application of interpolated acoustic parameters to the output of ambient sound effects in a virtual space, according to some embodiments of this specification. In some embodiments, process 500 may be executed by a processing device.

[0148] Step 510: Divide the virtual space where the virtual human is located into regions to form the boundaries between different regions.

[0149] Virtual space refers to a virtual scene space that carries virtual human activities and outputs environmental sound effects.

[0150] In some embodiments, the processing device can divide the virtual space into regions based on environmental status markers in the environmental information to form different regions. The environmental status markers can include region markers, such as "indoor room," "street," "corridor," and "square." The processing device can divide the virtual space into different regions according to the spatial range corresponding to different region markers. The spatial range corresponding to the region markers can be pre-set based on prior knowledge. For more information on environmental information and environmental status markers, please refer to [link to relevant documentation]. Figure 1 And its explanation. For more information on environmental status markings, please refer to... Figure 3 Step 310 and its explanation.

[0151] A region refers to a pre-defined spatial unit in a virtual space. For example, a region may include indoor rooms, corridors, streets, squares, etc.

[0152] A region boundary refers to the boundary used to separate different regions in virtual space. In some embodiments, when the spatial ranges corresponding to adjacent regions intersect, the boundary between the spatial ranges corresponding to adjacent regions can form a region boundary between different regions.

[0153] Step 520: Set the corresponding acoustic parameters for different areas in the virtual space.

[0154] Acoustic parameters refer to acoustic characteristics in a virtual space. For example, acoustic parameters may include reverberation time RT60, early reflection delay, damping parameters, echo density, spatial diffusion parameters, filtering parameters, and environmental attenuation parameters.

[0155] In some embodiments, the processing device can set different reverberation times, spatial diffusion levels, ambient noise intensity, and early reflection parameters for each region based on the corresponding environmental characteristics and acoustic effect requirements. The environmental characteristics and acoustic effect requirements for each region can be preset based on prior knowledge or historical data.

[0156] Taking an indoor room as an example, since its environmental characteristics are relatively enclosed space and low background noise, and the acoustic effect requirement is to present a quiet and close listening effect, acoustic parameters corresponding to short reverberation, low diffusion and low noise can be set for indoor rooms. Taking a street as an example, since streets have open space and high environmental noise, and need to present a strong sense of environmental immersion, acoustic parameters corresponding to strong environmental reflection, high environmental noise and wide spatial diffusion can be set for streets.

[0157] Step 530: In response to the virtual human's coordinates crossing the region boundary, determine the interpolation weights based on the spatial distance information of the virtual human relative to the crossed region boundary.

[0158] The virtual character's coordinates crossing the region boundary means that the virtual character's position coordinates cross the region boundary from the spatial range corresponding to the front region and enter the spatial range corresponding to the rear region. The front region refers to the first region on one side of the region boundary, and the rear region refers to the second region on the other side of the region boundary.

[0159] For example, when the virtual person's location coordinates move from the spatial range corresponding to the indoor room to the spatial range corresponding to the corridor, it can be determined that the virtual person's coordinates have crossed the boundary between the indoor room and the corridor.

[0160] Spatial distance information can include the distance between the virtual human's current location and the region boundary, as well as the distance between the virtual human's current location and the center locations of the regions on both sides of the boundary.

[0161] Interpolation weight refers to the proportional coefficient that represents the degree of bias of the current ambient sound effect between the regions on both sides of the boundary.

[0162] In some embodiments, the processing device can determine the proximity of the virtual human to the front and rear regions based on the distance between the virtual human's current position and the center positions of the front and rear regions; and determine interpolation weights based on the proximity of the virtual human to the front and rear regions. Specifically, the interpolation weights may include a first sub-weight and a second sub-weight corresponding to the front and rear regions, respectively. The first sub-weight serves as the weight for the acoustic parameters corresponding to the front region, and the second sub-weight serves as the weight for the acoustic parameters corresponding to the rear region, and the sum of the first and second sub-weights is 1. When the virtual human is closer to the front region, the first sub-weight is greater than the second sub-weight, thus making the interpolation weights more biased towards the acoustic parameters corresponding to the front region; when the virtual human is closer to the rear region, the second sub-weight is greater than the first sub-weight, thus making the interpolation weights more biased towards the acoustic parameters corresponding to the rear region.

[0163] In some embodiments, as the virtual human's current position changes relative to the front and rear regions, the first and second sub-weights can change continuously, thereby interpolating the acoustic parameters corresponding to the front and rear regions according to the continuously changing weights to achieve a smooth transition of environmental sound parameters near the region boundaries.

[0164] Step 540: Based on the interpolation weight, smoothly interpolate the acoustic parameters corresponding to the regions on both sides of the crossed region boundary, and apply the interpolated acoustic parameters to the environmental sound effect output in the virtual space.

[0165] Smooth interpolation can include at least one of linear interpolation, logarithmic interpolation, or easing-in / easing-out interpolation. In some embodiments, the processing device can perform linear interpolation on the acoustic parameters corresponding to the regions on both sides of the boundary of the crossed region. As an example only, the processing device can determine the interpolated acoustic parameters based on interpolation weights as follows: Interpolated acoustic parameters = Acoustic parameters corresponding to the front region × (1-t) + Acoustic parameters corresponding to the rear region × t. Wherein, t can be used as the second sub-weight in the interpolation weights, and 1-t can be used as the first sub-weight in the interpolation weights. The first and second sub-weights can be dynamically determined based on the proximity of the virtual human's current position to the front and rear regions; as the virtual human's position changes, the first and second sub-weights can change continuously, thereby continuously updating the interpolated acoustic parameters.

[0166] In some embodiments, the processing device may also perform logarithmic interpolation or easing-in / easing-out interpolation on the acoustic parameters corresponding to the regions on both sides of the crossed region boundary so that the interpolated acoustic parameters transition smoothly between the two sides of the region boundary.

[0167] In some embodiments, the processing device can apply interpolated acoustic parameters to the environmental sound output in the virtual space. The processing device can load the interpolated acoustic parameters into the digital signal processing link of the environmental sound in real time to adjust the reverberation effect, spatial sense, reflection characteristics and spectral characteristics of the environmental sound, thereby making the environmental sound output in the virtual space continuously change during the movement of the virtual human.

[0168] In some embodiments of this specification, by setting corresponding acoustic parameters for different areas in the virtual space and smoothly interpolating the acoustic parameters on both sides of the area boundary when the virtual person moves across areas, abrupt changes in environmental sound effects at area switching are avoided, so that reverberation, spatial sense and other environmental acoustic characteristics transition continuously with the change of the virtual person's position, thereby improving the naturalness, realism and immersion of environmental sound effect switching.

[0169] The basic concepts have been described above. Obviously, for those skilled in the art, the detailed disclosure above is merely illustrative and does not constitute a limitation of this specification. Although not explicitly stated herein, those skilled in the art may make various modifications, improvements, and corrections to this specification. Such modifications, improvements, and corrections are suggested in this specification and therefore remain within the spirit and scope of the exemplary embodiments described herein.

[0170] Furthermore, this specification uses specific terms to describe embodiments thereof. For example, "an embodiment," "one embodiment," and / or "some embodiments" refer to a particular feature, structure, or characteristic associated with at least one embodiment of this specification. Therefore, it should be emphasized and noted that references to "an embodiment," "one embodiment," or "an alternative embodiment" in different locations throughout this specification do not necessarily refer to the same embodiment. Moreover, certain features, structures, or characteristics in one or more embodiments of this specification can be appropriately combined.

[0171] Furthermore, unless expressly stated in the claims, the order of processing elements and sequences, the use of numbers and letters, or other names described in this specification are not intended to limit the order of the processes and methods described herein. Although various examples have been discussed in the foregoing disclosure of some embodiments of the invention that are currently considered useful, it should be understood that such details are for illustrative purposes only, and the appended claims are not limited to the disclosed embodiments. Rather, the claims are intended to cover all modifications and equivalent combinations that conform to the substance and scope of some embodiments of this description. For example, while the system components described above can be implemented using hardware devices, they can also be implemented solely using software solutions, such as installing the described system on existing servers or mobile devices.

[0172] Similarly, it should be noted that, in order to simplify the descriptions disclosed herein and thus aid in the understanding of one or more embodiments of the invention, the foregoing descriptions of some embodiments of this specification sometimes combine multiple features into a single embodiment, drawing, or description thereof. However, this method of disclosure does not imply that the subject matter of this specification requires more features than those mentioned in the claims. In fact, the embodiments contain fewer features than all the features of a single embodiment disclosed above.

[0173] In some embodiments, numbers describing the quantity of components and attributes are used. It should be understood that such numbers used in the description of embodiments are modified in some examples with the terms "approximately," "approximately," or "generally." Unless otherwise stated, "approximately," "approximately," or "generally" indicates that the numbers are allowed to vary by ±20%. Accordingly, in some embodiments, the numerical parameters used in the specification and claims are approximate values, which may be changed depending on the characteristics required by individual embodiments. In some embodiments, numerical parameters should take into account specified significant digits and employ a general method of digit reservation. Although the numerical ranges and parameters used to confirm their breadth of range in some embodiments of this specification are approximate values, in specific embodiments, such values ​​are set as precisely as feasible.

[0174] For each patent, patent application, patent application publication, and other material such as articles, books, specifications, publications, and documents referenced in this specification, the entire contents of which are incorporated herein by reference. This excludes historical application documents that are inconsistent with or conflict with the content of this specification, as well as documents that limit the broadest scope of the claims in this specification (currently or subsequently appended to this specification). It should be noted that in the event of any inconsistency or conflict between the descriptions, definitions, and / or terminology used in the supplementary materials to this specification and the content of this specification, the descriptions, definitions, and / or terminology used in this specification shall prevail.

[0175] Finally, it should be understood that the embodiments described in this specification are merely illustrative of the principles of some embodiments of this specification. Other variations may also fall within the scope of this specification. Therefore, alternative configurations of some embodiments of this specification are intended to be exemplified rather than limited, and are considered consistent with the teachings of this specification. Accordingly, the embodiments of this specification are not limited to those explicitly described and illustrated herein.

Claims

1. A method for generating and adjusting virtual human-like sound effects, characterized in that, include: The virtual human's output information and environmental information are obtained, wherein the output information includes at least one of text information that has not yet been actually presented, voice information that has not yet been output, and action information that has not yet been performed; Based on the information to be output and the environmental information, determine the virtual human's emotional parameters; Determine the sound effect adjustment parameters corresponding to the virtual human's emotional parameters; as well as Based on the aforementioned sound effect adjustment parameters, target sound effect data is generated.

2. The method according to claim 1, characterized in that, The process of determining the virtual human's emotional parameters based on the information to be output and the environmental information includes: Obtain modal sentiment parameters and the confidence levels corresponding to the modal sentiment parameters. The modal sentiment parameters include at least one of the following: sentiment parameters corresponding to text modality, sentiment parameters corresponding to speech modality, sentiment parameters corresponding to action modality, and sentiment parameters corresponding to environment modality. Based on the confidence level of the modal sentiment parameters and the changes in historical sentiment parameters, the weights corresponding to the modal sentiment parameters are determined; and The virtual human's emotional parameters are obtained based on the modal emotional parameters and their corresponding weights.

3. The method according to claim 1, characterized in that, The step of generating target sound effect data based on the sound effect adjustment parameters includes: Based on the action information that the virtual human has not yet performed, determine the action status information, historical status information and actual action events corresponding to the virtual human; Based on the action state information and the historical state information, the candidate event types for the virtual human in the future time window are determined; Based on the candidate event type, generate the basic sound effect data corresponding to the candidate event type, and store the basic sound effect data in the cache area; In response to the virtual human's actual action event matching the candidate event type, the corresponding basic sound effect data is retrieved from the cache; and Based on the aforementioned sound effect adjustment parameters, the called basic sound effect data is adjusted to generate the target sound effect data.

4. The method according to claim 1, characterized in that, Also includes: The virtual space where the virtual human resides is divided into regions to form regional boundaries between different regions; Set corresponding acoustic parameters for the different regions in the virtual space; In response to the virtual human's coordinates crossing the region boundary, interpolation weights are determined based on the spatial distance information of the virtual human relative to the crossed region boundary. as well as Based on the interpolation weights, the acoustic parameters corresponding to the regions on both sides of the crossed region boundary are smoothly interpolated, and the interpolated acoustic parameters are applied to the environmental sound effect output in the virtual space.

5. A system for generating and adjusting virtual human context sound effects, comprising an acquisition module, a first determining module, a second determining module, and a generation module; The acquisition module is configured to acquire the virtual human's output information and environmental information. The output information includes at least one of text information that has not yet been actually presented, voice information that has not yet been output, and action information that has not yet been executed. The first determining module is configured to determine the virtual human's emotional parameters based on the information to be output and the environmental information; The second determining module is configured to determine the sound effect adjustment parameters corresponding to the virtual human's emotional parameters; The generation module is configured to generate target sound effect data based on the sound effect adjustment parameters.

6. The system according to claim 5, characterized in that, The first determining module is further configured as follows: Obtain modal sentiment parameters and the confidence levels corresponding to the modal sentiment parameters. The modal sentiment parameters include at least one of the following: sentiment parameters corresponding to text modality, sentiment parameters corresponding to speech modality, sentiment parameters corresponding to action modality, and sentiment parameters corresponding to environment modality. Based on the confidence level of the modal sentiment parameter and the changes in historical sentiment parameters, the weights corresponding to the modal sentiment parameters are determined. as well as The virtual human's emotional parameters are obtained based on the modal emotional parameters and their corresponding weights.

7. The system according to claim 5, characterized in that, The generation module is further configured to: Based on the action information that the virtual human has not yet performed, determine the action status information, historical status information and actual action events corresponding to the virtual human; Based on the action state information and the historical state information, the candidate event types for the virtual human in the future time window are determined; Based on the candidate event type, generate the basic sound effect data corresponding to the candidate event type, and store the basic sound effect data in the cache area; In response to the actual action event of the virtual human matching the candidate event type, the corresponding basic sound effect data is retrieved from the cache. as well as Based on the aforementioned sound effect adjustment parameters, the called basic sound effect data is adjusted to generate the target sound effect data.

8. The system according to claim 5, characterized in that, It also includes a region processing module, which is configured to: The virtual space where the virtual human resides is divided into regions to form regional boundaries between different regions; Set corresponding acoustic parameters for the different regions in the virtual space; In response to the virtual human's coordinates crossing the region boundary, interpolation weights are determined based on the spatial distance information of the virtual human relative to the crossed region boundary. as well as Based on the interpolation weights, the acoustic parameters corresponding to the regions on both sides of the crossed region boundary are smoothly interpolated, and the interpolated acoustic parameters are applied to the environmental sound effect output in the virtual space.

9. A device for generating and adjusting virtual human-like sound effects, characterized in that, The apparatus includes at least one processor and at least one memory; the at least one memory is used to store computer instructions; the at least one processor is used to execute at least a portion of the computer instructions to implement the method as described in any one of claims 1 to 4.

10. A computer-readable storage medium, characterized in that, The storage medium stores computer instructions. When the computer reads the computer instructions from the storage medium, the computer executes the method as described in any one of claims 1 to 4.

Images