Virtual character interaction control method, apparatus and device, and computer program product

Abstract

The embodiment of the invention provides a virtual character interaction control method, device and equipment and a computer program product. The method comprises the steps that in response to interaction information for a virtual character, voice feedback information corresponding to the interaction information is generated, the voice feedback information corresponds to a label set, and the label set comprises at least one non-language interaction label used for limiting the virtual character and time sequence information corresponding to the non-language interaction label; determining a voice timestamp corresponding to a voice element in the voice feedback information; based on the voice feedback information, the tag set and the voice timestamp, determining a trigger time point corresponding to the non-language interaction tag; and based on the voice feedback information, the voice timestamp and the triggering time point corresponding to the non-language interaction tag, controlling the virtual character to perform an interaction operation. According to the scheme, the problem of separation between the voice and the expression is solved, and the natural degree of interaction between the user and the virtual character can be improved to a certain extent.

Description

Technical Field

[0001] The present invention relates to the field of virtual data technology, and in particular to a method, device, equipment and computer program product for interactive control of a virtual character. Background Art

[0002] With the rapid development of technologies such as virtual humans, 3D animation, and virtual assistants, the coordination of voice, action, and expression of 3D virtual characters has become crucial in user experience. However, for 3D virtual characters, the current voice generation operation and expression rendering operation are usually performed independently, which easily leads to the problem of voice and expression separation, and further makes it difficult for the expression of 3D virtual characters to synchronously reflect the voice emotions. Summary of the invention

[0003] The embodiments of the present invention provide a method, device, equipment and computer program product for interactive control of a virtual character, which can solve the problem that the voice and expression of a 3D virtual character are difficult to synchronously reflect emotions.

[0004] In a first aspect, an embodiment of the present invention provides a method for interactive control of a virtual character, comprising:

[0005] In response to interaction information for a virtual character, voice feedback information corresponding to the interaction information is generated, wherein the voice feedback information corresponds to a tag set, the tag set including at least one non-verbal interaction tag for defining the virtual character, and timing information corresponding to the non-verbal interaction tag;

[0006] Determining a speech timestamp corresponding to a speech element in the speech feedback information;

[0007] Determining a trigger time point corresponding to the non-verbal interaction tag based on the voice feedback information, the tag set and the voice timestamp;

[0008] Based on the voice feedback information, the voice timestamp and the triggering time point corresponding to the non-verbal interaction tag, the virtual character is controlled to perform an interactive operation.

[0009] In a second aspect, an embodiment of the present invention provides an interactive control device for a virtual character, comprising:

[0010] A first acquisition module, configured to generate, in response to interaction information for a virtual character, voice feedback information corresponding to the interaction information, wherein the voice feedback information corresponds to a tag set, the tag set including at least one non-verbal interaction tag for defining the virtual character, and timing information corresponding to the non-verbal interaction tag;

[0011] A first determining module, used to determine the speech timestamp corresponding to the speech element in the speech feedback information;

[0012] The first determination module is further configured to determine a trigger time point corresponding to the non-language interaction tag based on the voice feedback information, the tag set and the voice timestamp;

[0013] The first processing module is used to control the virtual character to perform interactive operations based on the voice feedback information, the voice timestamp and the trigger time point corresponding to the non-language interaction tag.

[0014] In a third aspect, an embodiment of the present invention provides an electronic device, comprising: a memory, and a processor; wherein the memory is used to store one or more computer instructions, wherein the one or more computer instructions, when executed by the processor, implement the interactive control method of the virtual character in the above-mentioned first aspect.

[0015] In a fourth aspect, an embodiment of the present invention provides a computer storage medium for storing a computer program, wherein the computer program enables a computer to implement the interactive control method of a virtual character in the first aspect when executed.

[0016] In a fifth aspect, an embodiment of the present invention provides a computer program product, comprising: a computer-readable storage medium storing computer instructions, which, when executed by one or more processors, causes the one or more processors to execute the steps in the method for interactive control of a virtual character shown in the first aspect above.

[0017] The interactive control method, apparatus, device and computer program product of a virtual character provided in the present embodiment can generate a voice timestamp corresponding to the voice element in the voice feedback information after acquiring the interactive information for the virtual character, and determine the trigger time point corresponding to the non-verbal interaction tag based on the tag set, the voice feedback information and the voice timestamp. Since the non-verbal interaction tag and the voice feedback information can be associated and coordinated in the time dimension, when the virtual character is controlled based on the voice feedback information, the voice timestamp and the trigger time point corresponding to the non-verbal interaction tag, the synchronous reflection operation of the non-verbal interaction behavior and the voice of the virtual character in the time dimension is effectively achieved. This not only solves the separation problem between voice and expression existing in the related art, but also can improve the naturalness of the interaction between the user and the virtual character to a certain extent, further improving the practicality of the method. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0019] Figure 1 A schematic diagram of a scenario of a method for interactive control of a virtual character provided by an embodiment of the present invention;

[0020] Figure 2 A schematic diagram of a flow chart of a method for interactive control of a virtual character provided by an embodiment of the present invention;

[0021] Figure 3 A schematic diagram of a process for generating voice feedback information corresponding to the interaction information provided by an embodiment of the present invention;

[0022] Figure 4 A schematic diagram of a process for determining a trigger time point corresponding to the non-verbal interaction tag based on the tag set, the voice feedback information and the voice timestamp provided in an embodiment of the present invention;

[0023] Figure 5 A schematic diagram of the principle of a 3D virtual character interactive control method provided by an application embodiment of the present invention;

[0024] Figure 6 A schematic diagram of the structure of a virtual character interactive control device provided by an embodiment of the present invention;

[0025] Figure 7 For Figure 6 A schematic diagram of the structure of an electronic device corresponding to the interactive control device for a virtual character provided by an embodiment is shown. DETAILED DESCRIPTION

[0026] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0027] The terms used in the embodiments of the present invention are only for the purpose of describing specific embodiments, and are not intended to limit the present invention. The singular forms "a", "an", and "the" used in the embodiments of the present invention and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings, and "multiple" generally includes at least two, but does not exclude the situation of including at least one.

[0028] It should be understood that the term "and / or" used in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship.

[0029] As used herein, the words "if" and "if" may be interpreted as "at the time of" or "when" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrases "if it is determined" or "if (stated condition or event) is detected" may be interpreted as "when it is determined" or "in response to determining" or "when detecting (stated condition or event)" or "in response to detecting (stated condition or event)", depending on the context.

[0030] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a product or system including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such a product or system. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the product or system including the elements.

[0031] In addition, the step sequence in the following method embodiments is only an example and not a strict limitation.

[0032] Definition of terms:

[0033] Multimodal synchronization technology: refers to the technology of precise synchronization and processing of multiple modal information (such as voice, action, expression, etc.) in the time dimension. This technology ensures the output consistency between different modal information and greatly improves the naturalness and fluency of interaction.

[0034] 3D virtual character's facial expressions: refers to the 3D virtual character expressing various emotions and reactions (such as smiling, frowning) through facial animation, which is used to enhance the character's expressiveness and interactive effects. The facial expressions are usually generated synchronously with the speech content and tone.

[0035] 3D virtual character's action and voice output: refers to the 3D virtual character's simulation of real interactive behavior through skeletal animation and TTS voice generation technology, combined with gestures, body movements and other performances; these actions and voices are coordinated and output through synchronization technology.

[0036] Phoneme synchronization algorithm: refers to generating accurate timestamps for each element based on the element characteristics of speech, which is used to align with the expression and action triggering time of the 3D virtual character. This algorithm is the core foundation of multimodal synchronization technology.

[0037] Virtual character: refers to a character driven by 3D models and algorithms in a virtual environment, which can interact with users through voice, actions and expressions; virtual characters are commonly found in application scenarios such as virtual humans, game characters and virtual assistants.

[0038] In order to facilitate understanding of the specific implementation process and implementation effect of the interactive control method, device, equipment and computer program product of the virtual character in this embodiment, the relevant technology is briefly described below:

[0039] With the rapid development of technologies such as virtual humans, 3D animation, and virtual assistants, the coordination of 3D characters' voices, movements, and expressions has become crucial for user experience. Since multimodal synchronization technology can synchronize data in different modalities, it has become the key to improving the naturalness and immersion of interactions. However, high-precision synchronization between multimodal data still faces many technical challenges:

[0040] (1) There is a disconnect between speech and expression: Since speech generation and expression rendering are usually performed independently, it is easy for the expression of the 3D virtual character to be unable to synchronously reflect the speech emotions.

[0041] (2) Inconsistency between the timing of actions and voice: The actions (such as gestures) of 3D virtual characters and the triggering time of voice content often lack precise alignment, which affects the user's interactive immersion.

[0042] (3) Insufficient emotional consistency: The tone of the voice content is inconsistent with the facial expressions and actions of the 3D virtual character, which can easily cause users to have a fragmented experience.

[0043] (4) Limited multi-language support: In multi-language scenarios such as Chinese, English, Japanese, and Korean, the rhythm differences between different types of speech and actions further increase the technical difficulty of synchronous operation of multi-modal information such as speech, expression, and action.

[0044] In view of the technical challenges existing in the above-mentioned related technologies, the related technologies provide several solutions. The following is a brief description of the related technologies:

[0045] Related technology 1, based on an immersive artificial intelligence (AI) content community, enables users to communicate with AI virtual characters and experience conversational interactive operations.

[0046] However, the conversational interaction in the above method is mainly concentrated between text information and voice information, and lacks the ability to generate expressions, actions and voice synchronously. As a result, expressions and voice cannot be accurately synchronized through timestamps, and it is easy for expression changes to lag behind voice content, and the interaction effect becomes unnatural.

[0047] Related technology 2: AI tools can be used to drive virtual character images to perform content generation operations, thereby achieving visual generation operations that focus on static characters.

[0048] However, the dialogue operation in the above method does not have the dynamic synchronization function between voice, expression and action. Therefore, the application scenarios are concentrated on static content creation, and do not support real-time interactive scenarios. It is difficult to apply to virtual humans, real-time animations or metaverse scenarios, and thus cannot cover the interactive scenarios of immersive 3D virtual characters.

[0049] Related technology 3: The large language model can realize natural language dialogue operations based on the input language information.

[0050] However, the above implementation lacks the ability to synchronize the expressions and actions of 3D virtual characters, and only supports simple conversion of single modality or text to speech; and does not provide high-precision modal collaboration technology based on timestamps, and cannot solve the inconsistency problem of speech, expression and action.

[0051] In summary, the solutions provided in the related art have the following defects:

[0052] (1) Low synchronization accuracy: The synchronization of speech, actions, and expressions relies on preset rules, which makes it difficult to achieve dynamic adjustment, and the synchronization accuracy is low.

[0053] (2) Weak cross-modal collaboration capabilities: Collaborative operations between different modal information are usually completed through simple timeline mapping, which is difficult to adapt to complex emotions and task requirements.

[0054] (3) Insufficient scalability: The adaptation cost of multiple languages ​​and multiple scenarios is high, and there is a lack of flexible dynamic adjustment mechanism.

[0055] In order to solve the technical problems existing in the related art, this embodiment provides a method, device, equipment and computer program product for interactive control of a virtual character. Figure 1As shown, the execution subject of the interactive control method of the virtual character can be an interactive control device 200 of the virtual character, and the interactive control device 200 of the virtual character can be implemented as a local server, a server in the cloud or a preset device. In the case where the interactive control device 200 of the virtual character is implemented as a server in the cloud, the interactive control method of the virtual character can be executed in the cloud, and several computing nodes (cloud servers) can be deployed in the cloud, and each computing node has computing, storage and other processing resources. In the cloud, multiple computing nodes can be organized to provide a certain service. Of course, a computing node can also provide one or more services. The way to provide the service in the cloud can be to provide a service interface to the outside, and the user calls the service interface to use the corresponding service. The service interface includes a software development kit (Software Development Kit, referred to as SDK), an application programming interface (Application Programming Interface, referred to as API), and the like.

[0056] The interactive control device 200 of the virtual character is connected to the client 100 in communication, wherein the client 100 is used for users to perform applications so as to trigger the interactive control operation of the virtual character. The above-mentioned client 100 can be any computing device with certain information interaction capabilities. When implemented specifically, the client 100 can be a mobile phone, a personal computer PC, a tablet computer, a setting application, etc. In addition, the basic structure of the client 100 may include: at least one processor. The number of processors depends on the configuration and type of the client. The client 100 may also include a memory, which may be volatile, such as a random access memory (RAM), or non-volatile, such as a read-only memory (ROM), flash memory, etc., or may include both types. The memory usually stores an operating system (OS), one or more application programs, and may also store program data, etc. In addition to the processing unit and the memory, the client 100 also includes some basic configurations, such as a network card chip, an IO bus, a display component, and some peripheral devices, etc. Optionally, some peripheral devices may include, for example, a keyboard, a mouse, an input pen, a printer, etc. Other peripheral devices are well known in the art and will not be described in detail here.

[0057] The interactive control device 200 of a virtual character refers to a device that can implement interactive control operations of a virtual character in a network virtual environment, and generally refers to a device that uses a network to perform information planning and interactive control operations of a virtual character. In physical implementation, the interactive control device 200 of a virtual character can be any device that can provide computing services and can perform interactive control operations of a corresponding virtual character, for example, a cluster server, a conventional server, a cloud server, a cloud host, a virtual center, etc. The interactive control device 200 of a virtual character mainly includes a processor, a hard disk, a memory, a system bus, etc., which is similar to a general computer architecture.

[0058] In the above embodiment, the client 100 is connected to the interactive control device 200 of the virtual character through a network, and the network connection can be a wireless or wired network connection. If the client 100 can be connected to the interactive control device 200 of the virtual character through communication, the network standard of the mobile network can be any one of 2G (Global System for Mobile Communications GSM), 2.5G (General Packet Radio Service GPRS), 3G (Wideband Code Division Multiple Access WCDMA, Time Division Synchronous Code Division Multiple Access TD-SCDMA, 4G (Long Term Evolution LTE), 4G+ (Enhanced Long Term Evolution LTE+), World Interoperability for Microwave Access WiMax, 5G, 6G, etc.

[0059] In the embodiment of the present application, the client 100 is used by the user to generate interactive information for the virtual character, wherein the interactive information of the virtual character can be generated by human-computer interaction operation and voice interaction operation. After acquiring the interactive information of the virtual character, the interactive information of the virtual character can be sent to the interactive control device 200 of the virtual character, so that the interactive control device 200 of the virtual character can perform the corresponding interactive control operation of the virtual character based on the interactive information of the virtual character.

[0060] The interactive control device 200 for a virtual character is used to obtain interactive information for the virtual character through the client 100. The interactive information can be implemented as text interactive information and / or voice interactive information, and then voice feedback information corresponding to the interactive information can be generated based on the interactive information, wherein the generated voice feedback information can correspond to a tag set, which can include at least one non-verbal interactive tag for limiting the virtual character and timing information corresponding to the non-verbal interactive tag. The above-mentioned non-verbal interactive tag can include at least one of the following: an expression tag and an action tag.

[0061] In order to ensure that the expression, action and voice of the virtual character can be consistently reflected when the virtual character performs interactive operations, the voice timestamp corresponding to the voice element in the voice feedback information can be determined first, and then the trigger time point corresponding to the non-verbal interaction tag can be determined based on the tag set, voice feedback information and voice timestamp, and the virtual character can be controlled to perform interactive operations based on the voice feedback information, voice timestamp and the trigger time point corresponding to the non-verbal interaction tag, so that the virtual character can perform natural interactive operations with the user based on the voice feedback information and non-verbal interaction tags.

[0062] In this embodiment, after the interaction information for the virtual character is obtained, the voice timestamp of the voice element in the voice feedback information is determined, and the trigger time point corresponding to the non-verbal interaction tag is determined based on the tag set, the voice feedback information and the voice timestamp, and then the virtual character is controlled to perform interactive operations based on the voice feedback information, the voice timestamp and the trigger time point corresponding to the non-verbal interaction tag. Since the non-verbal interaction tag and the voice feedback information are associated and coordinated in the time dimension, when the virtual character is controlled based on the voice feedback information, the voice timestamp and the trigger time point corresponding to the non-verbal interaction tag, the non-verbal interaction behavior and voice of the virtual character can be effectively synchronized in the time dimension. This not only solves the problem of separation between voice and expression in the related art, but also can improve the naturalness of the interaction between the user and the virtual character to a certain extent, further improving the practicality of the method.

[0063] Some embodiments of the present invention are described in detail below in conjunction with the accompanying drawings. In the case where there is no conflict between the embodiments, the following embodiments and the features in the embodiments can be combined with each other. In addition, the step sequence in the following method embodiments is only an example and not a strict limitation.

[0064] Figure 2 A flowchart of a method for interactive control of a virtual character provided by an embodiment of the present invention; Figure 2 As shown, this embodiment provides an interactive control method for a virtual character, the execution subject of the method is an interactive control device for a virtual character, wherein the interactive control device for a virtual character can be implemented as software, or a combination of software and hardware. When the interactive control device for a virtual character is implemented as hardware, it can be various electronic devices that can implement interactive control operations of virtual characters, including but not limited to personal computers, servers, etc. When the interactive control device for a virtual character is implemented as software, it can be installed in the electronic devices listed above. Based on the interactive control device for a virtual character, the interactive control operation of a virtual character can be implemented. Specifically, the interactive control method for a virtual character may include:

[0065] Step S201: In response to interaction information for a virtual character, voice feedback information corresponding to the interaction information is generated, wherein the voice feedback information corresponds to a tag set, the tag set includes at least one non-verbal interaction tag for defining the virtual character, and timing information corresponding to the non-verbal interaction tag.

[0066] Step S202: Determine the speech timestamp corresponding to the speech element in the speech feedback information.

[0067] Step S203: Determine the trigger time point corresponding to the non-verbal interaction tag based on the voice feedback information, the tag set and the voice timestamp.

[0068] Step S204: Based on the voice feedback information, the voice timestamp, and the trigger time point corresponding to the non-verbal interaction tag, the virtual character is controlled to perform an interactive operation.

[0069] The specific implementation process and effects of the above steps are described in detail below:

[0070] Step S201: In response to interaction information for a virtual character, voice feedback information corresponding to the interaction information is generated, wherein the voice feedback information corresponds to a tag set, the tag set includes at least one non-verbal interaction tag for defining the virtual character, and timing information corresponding to the non-verbal interaction tag.

[0071] Among them, at least one virtual character capable of interacting with the user is provided in the virtual scene, and the virtual character can be implemented as any one of the following: 2D virtual character, 3D virtual character, 4D virtual character, etc. When the user has an interactive demand for the virtual character, the interactive control device of the virtual character (hereinafter referred to as "interactive control device") can obtain interactive information for the virtual character, and the interactive information can be implemented as text interactive information and / or voice interactive information. Specifically, in a text interactive scene, the interactive information obtained can be implemented as text interactive information; in a voice interactive scene, the interactive information obtained can be implemented as voice interactive information; in a multimodal information interaction scene, the interactive information obtained can include text interactive information and voice interactive information.

[0072] In order to enable natural interaction between virtual users and users, after obtaining interaction information for the virtual user, voice feedback information corresponding to the interaction information can be generated. In some instances, the voice feedback information can be determined by analyzing and processing the interaction information through a large language model. At this time, generating voice feedback information corresponding to the interaction information may include: obtaining a large language model for realizing information interaction operations; inputting the interaction information for the virtual character into the large language model, and obtaining voice feedback information output by the large language model. This ensures the accuracy and reliability of the generation of voice feedback information to a certain extent.

[0073] In addition, for the voice feedback information, it may include the voice feedback content corresponding to the interactive information, and the voice feedback information corresponds to a tag set, which includes at least one non-verbal interactive tag for limiting the virtual character, and the non-verbal interactive tag may include at least one of the following: at least one expression tag, at least one action tag, at least one emotion tag, etc. The above-mentioned tag set will not only include at least one non-verbal interactive tag for limiting the virtual character, but also include timing information corresponding to the non-verbal interactive tag, for example: the non-verbal interactive tag includes action tag 1, action tag 2, action tag 3 and action tag 4, etc., and the timing information between the above-mentioned multiple action tags can be: action tag 2-action tag 1-action tag 4-action tag 3; the non-verbal interactive tag can include: expression tag 1, expression tag 2, expression tag 3, and the timing information between the above-mentioned multiple expression tags can be: expression tag 2-expression tag 3-expression tag 1.

[0074] Step S202: Determine the speech timestamp corresponding to the speech element in the speech feedback information.

[0075] When the user interacts with the virtual character, in order to achieve a high degree of consistency between the non-verbal interactive behavior and the voice of the virtual character in the time dimension, after generating the voice feedback information corresponding to the interactive information, the voice feedback information can be analyzed and processed to determine the voice timestamp corresponding to the voice element in the voice feedback information, wherein the voice element can be a basic component of the voice feedback information. In some examples, the voice element can be implemented as at least one of the following: a character element, a phrase element, a short sentence element, etc. The above-mentioned character element can refer to a single character used to constitute the voice feedback information, the phrase element can refer to at least two characters used to constitute the voice feedback information, and the short sentence element can be set to at least five characters used to constitute the voice feedback information. For example: when the voice feedback information is "Today's weather is fine", when the voice element is a character element, the character elements used to constitute the voice feedback information may include: "today", "day", "sky", "air", "clear", "bright"; when the voice element is a phrase element, the phrase element used to constitute the voice feedback information may include "today", "weather", "clear"; when the voice element is a short sentence element, the phrase element used to constitute the voice feedback information may include "Today's weather is fine".

[0076] In some instances, the voice timestamp corresponding to the voice element can be determined by the time length corresponding to the voice feedback information. At this time, determining the voice timestamp corresponding to the voice element in the voice feedback information may include: determining the time axis corresponding to the voice feedback information, the length of the time axis is used to reflect the time length required for broadcasting the voice feedback information; determining the voice elements included in the voice feedback information, the broadcasting order between the various voice elements, and the voice broadcasting interval between the voice elements; based on the voice broadcasting interval and the broadcasting order between the various voice elements, associating the time axis with the voice elements included in the voice feedback information, and determining the voice timestamp corresponding to the voice element in the voice feedback information, the voice timestamp is used to identify the broadcasting time point corresponding to the voice element when the voice feedback information is broadcast, thereby achieving stable reliability in determining the voice timestamp corresponding to the voice element to a certain extent.

[0077] In other examples, the speech timestamp corresponding to the speech element can be determined not only by the time length corresponding to the speech feedback information, but also because the speech feedback information corresponds to a tag set, in order to avoid the tag set affecting the analysis and processing of the speech feedback information, the speech feedback information and the tag set can be segmented first, and then the segmented speech feedback information can be analyzed and processed to determine the speech timestamp corresponding to the speech element. At this time, determining the speech timestamp corresponding to the speech element in the speech feedback information may include: segmenting the tag set and the speech feedback information to obtain the processed feedback information; determining the speech element in the processed feedback information, wherein the speech element includes at least one of the following: a character element and a phrase element; determining the start timestamp and the end timestamp corresponding to the speech element based on the processed feedback information; determining the start timestamp and the end timestamp corresponding to the speech element as the speech timestamp corresponding to the speech element.

[0078] After obtaining the voice feedback information corresponding to the tag set, in order to reduce the influence of the tag set on the voice feedback information, the tag set and the voice feedback information can be segmented first, so as to obtain processed feedback information that does not correspond to the tag set. After obtaining the processed feedback information, the processed feedback information can be semantically parsed, so as to obtain the voice elements in the processed feedback information, and the voice elements can include at least one of the following: character elements and phrase elements.

[0079] After determining the voice element in the processed feedback information, the start timestamp and the end timestamp corresponding to the voice element can be determined based on the processed feedback information, wherein the start timestamp and the end timestamp can be determined by analyzing and processing the processed feedback information through a preset information synchronization algorithm. For the start timestamp and the end timestamp, the start timestamp is used to identify the start time point of broadcasting the voice element, and the end timestamp is used to identify the end time point of broadcasting the voice element, that is, the start timestamp and the end timestamp are used to identify the time point of broadcasting the voice element, and then the start timestamp and the end timestamp corresponding to the voice element can be determined as the voice timestamp corresponding to the voice element, which not only realizes the flexibility and reliability of determining the voice timestamp, but also guarantees the accuracy of determining the voice timestamp to a certain extent.

[0080] Step S203: Determine the trigger time point corresponding to the non-verbal interaction tag based on the voice feedback information, the tag set and the voice timestamp.

[0081] Since the voice timestamp corresponding to the voice element can control the voice broadcast time when the virtual character interacts, when the virtual character interacts with the user, in order to make the interaction between the virtual character and the user natural, it is necessary to consider not only the voice communication function of the virtual character, but also the non-verbal interaction behavior of the virtual character, for example, the emotional effect, action effect and expression effect of the virtual character. Therefore, in order to make the non-verbal interaction behavior of the virtual character and the voice feedback information of the virtual character consistent, the non-verbal interaction tag and the voice element can be associated in the time dimension. At this time, after obtaining the voice feedback information corresponding to the label set and the voice timestamp corresponding to the voice element, the label set, the voice feedback information and the voice timestamp can be analyzed and processed to determine the trigger time point corresponding to the non-verbal interaction tag, which is used to identify the occurrence time point of the non-verbal interaction behavior corresponding to the non-verbal interaction tag.

[0082] In some instances, the trigger time point corresponding to the non-verbal interaction label can be determined by a pre-trained deep learning model. At this time, based on the voice feedback information, the label set and the voice timestamp, determining the trigger time point corresponding to the non-verbal interaction label can include: obtaining a pre-trained deep learning model; inputting the label set, the voice feedback information and the voice timestamp into the deep learning model for processing, and obtaining the trigger time point corresponding to the non-verbal interaction label output by the deep learning model, thereby ensuring the accuracy and reliability of the determination of the trigger time point to a certain extent.

[0083] Step S204: Based on the voice feedback information, the voice timestamp, and the trigger time point corresponding to the non-verbal interaction tag, the virtual character is controlled to perform an interactive operation.

[0084] After obtaining the voice feedback information, voice timestamp and trigger time point corresponding to the non-verbal interaction tag, the virtual character can be controlled to perform interactive operations based on the voice feedback information, voice timestamp and trigger time point corresponding to the non-verbal interaction tag. Since the voice timestamp is used to identify the time information corresponding to the voice element in the voice feedback information, and the trigger time point is used to identify the occurrence time point of the non-verbal interaction behavior corresponding to the non-verbal interaction tag, when the virtual character is controlled based on the voice feedback information, voice timestamp and trigger time point corresponding to the non-verbal interaction tag, the voice information and non-verbal interaction behavior of the virtual character can be associated and synchronized in the time dimension, so that the voice and non-verbal interaction behavior of the virtual character can be synchronously reflected in the time dimension, which effectively solves the problem that the voice and expression of the virtual character in the related technology are difficult to reflect synchronously, thereby significantly improving the naturalness of the interaction of the virtual character.

[0085] In addition, this embodiment does not limit the implementation method of controlling the virtual character to perform interactive operations. In some instances, the interactive operations of controlling the virtual character can be implemented through the generated interactive parameters. At this time, based on the voice feedback information, voice timestamp and trigger time points corresponding to the non-verbal interaction tags, controlling the virtual character to perform interactive operations may include: the interactive control device may generate interactive control parameters based on the voice feedback information, voice timestamp and trigger time points corresponding to the non-verbal interaction tags, wherein the interactive control parameters may include: voice control parameters and non-verbal interaction parameters in the time dimension; and then the interactive control device may control the virtual character to perform interactive operations based on the interactive control parameters, thereby ensuring the stability and reliability of the control of the virtual character to a certain extent.

[0086] In other instances, the interactive operations of controlling the virtual character can be achieved not only through the interactive control device, but also through the rendering engine. In this case, based on the voice feedback information, the voice timestamp, and the trigger time point corresponding to the non-verbal interaction tag, controlling the virtual character to perform interactive operations may include: based on the voice feedback information, the voice timestamp, and the trigger time point corresponding to the non-verbal interaction tag, generating non-verbal interaction data corresponding to the virtual character, the non-verbal interaction data including at least one of the following: expression data, action data; sending the voice feedback information and the non-verbal interaction data to the rendering engine, so that the rendering engine renders the virtual character based on the voice feedback information and the non-verbal interaction data, and controls the virtual character to perform interactive operations.

[0087] Specifically, after the interactive control device obtains the trigger time point corresponding to the voice feedback information, voice timestamp and non-verbal interaction label, the voice feedback information, voice timestamp and non-verbal interaction label can be analyzed and processed to generate non-verbal interaction data corresponding to the virtual character, wherein the non-verbal interaction data can be obtained by analyzing and processing the voice feedback information, voice timestamp and non-verbal interaction label through a preset algorithm or a pre-trained deep learning model, and the non-verbal interaction data obtained above may include at least one of the following: expression data for limiting the facial expressions and / or body emotions of the virtual character, and action data for controlling the virtual character to perform certain operations.

[0088] Among them, the interactive control device is communicatively connected to the rendering engine, and the rendering engine can be located on the client. In order to ensure the accuracy and reliability of controlling the virtual character to a certain extent, after the interactive control device generates non-verbal interaction data corresponding to the virtual character, the voice feedback information and the non-verbal interaction data are sent to the rendering engine, so that the rendering engine can render the virtual character based on the voice feedback information and the non-verbal interaction data, and control the virtual character to perform interactive operations, thereby effectively enabling the user to perform stable interactive operations with the rendered virtual character.

[0089] The interactive control method of a virtual character provided in the present embodiment can generate a voice timestamp corresponding to the voice element in the voice feedback information after acquiring the interactive information for the virtual character, and determine the trigger time point corresponding to the non-verbal interaction tag based on the tag set, the voice feedback information and the voice timestamp. Since the non-verbal interaction tag and the voice feedback information can be associated and coordinated in the time dimension, when the virtual character is controlled based on the voice feedback information, the voice timestamp and the trigger time point corresponding to the non-verbal interaction tag, the synchronous reflection operation of the non-verbal interactive behavior and voice of the virtual character in the time dimension is effectively achieved. This not only solves the separation problem between voice and expression existing in the related art, but also can improve the naturalness of the interaction between the user and the virtual character to a certain extent, further improving the practicality of the method.

[0090] Figure 3 A schematic diagram of a flow chart of generating voice feedback information corresponding to interactive information provided by an embodiment of the present invention; based on the above embodiment, refer to the attached Figure 3 As shown, for the voice feedback information corresponding to the interaction information, it can be determined not only by analyzing and processing the interaction information through a large language model, but also by the interaction intention and interaction emotion information of the interaction information. At this time, generating the voice feedback information corresponding to the interaction information may include:

[0091] Step S301: Acquire interaction intention and interaction emotion information corresponding to the interaction information.

[0092] Since the voice feedback information is the reply information (or answer information) when the virtual character interacts with the user, and different interactive information can generate different voice feedback information, in order to ensure the accuracy and reliability of the generation of voice feedback information, after obtaining the interactive information, the interactive information can be analyzed and processed to obtain the interactive intention and interactive emotion information corresponding to the interactive information, wherein the interactive emotion information can include at least one of the following: happy information, calm information, sad information, sad information, etc.

[0093] In some instances, the interaction intent can be determined based on the information type of the interaction information and the semantic analysis technology. At this time, obtaining the interaction intent corresponding to the interaction information may include: when the interaction information is text interaction information, processing the text interaction information based on the semantic analysis technology to obtain the interaction intent; or, when the interaction information is voice interaction information, using voice recognition technology to convert the voice interaction information into corresponding text content, and then performing semantic analysis on the text content to obtain the interaction intent, thereby effectively realizing the determination operation of the interaction intent.

[0094] Specifically, the interaction intention can be determined not only based on the information type of the interaction information and the semantic analysis technology, but also in combination with the triggering information user and / or context information. At this time, obtaining the interaction intention corresponding to the interaction information may include: obtaining the triggering user information and / or context information of the interaction information; based on the interaction information, the triggering user information and / or the context information, determining the interaction intention corresponding to the interaction information.

[0095] In addition, since the interaction intention is not only related to the current interaction information, but also to the type of the triggering user and / or the context information, in order to ensure the accuracy and reliability of determining the interaction intention, the triggering user information and / or the context information of the interaction information can be obtained, wherein the triggering user information can include at least one of the following: the preference information of the triggering user, the habit information of the triggering user, the user portrait information of the triggering user, etc., and the triggering user information can be determined by the identity of the triggering user; the context information can include the historical information located before the interaction information within a preset time period, and the context information can be obtained by querying the historical interaction database. After obtaining the triggering user information and / or the context information, the interaction intention corresponding to the interaction information can be determined based on the interaction information, the triggering user information and / or the context information. Specifically, the interaction intention can be determined by the following implementation methods: Implementation method 1: determining the interaction intention corresponding to the interaction information based on the interaction information and the triggering user information; Implementation method 2: determining the interaction intention corresponding to the interaction information based on the interaction information and the context information; Implementation method 3: determining the interaction intention corresponding to the interaction information based on the interaction information, the triggering user information and the context information, which effectively ensures the flexibility and reliability of determining the interaction intention.

[0096] Since the voice feedback information is not only related to the interaction intention of the interaction information, but also to the interaction emotion information corresponding to the interaction information, in addition to obtaining the interaction intention corresponding to the interaction information, it is also necessary to obtain the interaction emotion information corresponding to the interaction information. For the interaction emotion information, since the interaction information can correspond to different types, the interaction emotion information can be determined based on different information processing methods. At this time, obtaining the interaction emotion information corresponding to the interaction information may include: when the interaction information is implemented as text interaction information, obtaining at least one reference emotion information; determining a first vector representation of the interaction intention and a second vector representation of the reference emotion information; and determining the interaction emotion information corresponding to the interaction information based on the first vector representation and the second vector representation.

[0097] Specifically, when the interactive information is implemented as text interactive information, in order to accurately determine the interactive emotion information corresponding to the interactive information, at least one reference emotion information for determining the interactive emotion information can be obtained first. The at least one reference emotion information can be pre-configured information stored in a preset area. In some instances, the at least one reference emotion information can include at least one of the following: happy emotion information, sad emotion information, angry emotion information, fearful emotion information, surprised emotion information, disgusted emotion information, calm emotion information, sad emotion information, etc. When it is necessary to determine the interactive emotion information corresponding to the interactive information, at least one reference emotion information can be obtained by accessing the preset area.

[0098] After obtaining at least one reference emotion information, the reference emotion information can be analyzed and processed. Specifically, the reference emotion information can be encoded and processed using a preset mapping algorithm or encoder, so that the second vector representation of the reference emotion information can be determined. In order to accurately determine the interactive emotion information corresponding to the interactive information, it is necessary not only to determine the second vector representation of the reference emotion information, but also to analyze and process the interactive intent after obtaining the interactive intent to determine the first vector representation of the interactive intent, wherein the method for determining the first vector representation is similar to the method for determining the second vector representation, and the details can be referred to the above statements, which will not be repeated here.

[0099] After determining the first vector representation and the second vector representation, the first vector representation and the second vector representation can be analyzed and processed to determine the interactive emotion information corresponding to the interactive information. In some instances, the interactive emotion information can be determined by a pre-trained deep learning model. At this time, based on the first vector representation and the second vector representation, determining the interactive emotion information corresponding to the interactive information can include: obtaining a pre-trained deep learning model; inputting the first vector representation and the second vector representation corresponding to at least one reference emotion information into the deep learning model, and obtaining the interactive emotion information corresponding to the interactive information output by the deep learning model. This ensures the quality and efficiency of determining the interactive emotion information to a certain extent.

[0100] In other instances, the interactive emotion information can be determined not only by a pre-trained deep learning model, but also by calculating the similarity between the first vector representation and the second vector representation. At this time, based on the first vector representation and the second vector representation, determining the interactive emotion information corresponding to the interactive information may include: obtaining the similarity between the first vector representation and the second vector representation; determining the target vector representation corresponding to the maximum similarity among multiple second vector representations; and determining the reference emotion information corresponding to the target vector representation as the interactive emotion information corresponding to the interactive information.

[0101] Specifically, after obtaining the first vector representation and the second vector representation, the first vector representation and the second vector representation can be analyzed and calculated to obtain the similarity between the first vector representation and the second vector representation. Since the similarity between the first vector representation and the second vector representation is used to identify the degree of adaptation between the interactive information and the interactive emotion information, in order to accurately determine the interactive emotion information corresponding to the interactive information, the target vector representation corresponding to the maximum similarity can be determined among multiple second vector representations, and then the reference emotion information corresponding to the target vector representation can be determined as the interactive emotion information corresponding to the interactive information. This ensures the accuracy and reliability of the determination of the interactive emotion information to a certain extent.

[0102] For interactive emotion information, since different types of interactive information can use different implementation methods to determine the interactive emotion information corresponding to the interactive information, at this time, obtaining the interactive emotion information corresponding to the interactive information may include: when the interactive information is voice interactive information, obtaining at least one reference emotional information and a voice feature corresponding to the voice interactive information; determining a first vector representation of the interactive intention, a second vector representation of the reference emotional information, and a third vector representation corresponding to the voice feature; based on the first vector representation, the second vector representation, and the third vector representation, determining the interactive emotion information corresponding to the interactive information.

[0103] Specifically, when the interaction information is voice interaction information, in order to be able to determine the interaction emotion information corresponding to the interaction information to a certain extent, at least one reference emotion information and a voice feature corresponding to the voice interaction information can be first obtained, wherein the method for obtaining at least one reference emotion information in this embodiment is similar to the method for obtaining at least one reference emotion information in the above embodiment. For details, please refer to the above statements, which will not be repeated here.

[0104] Since reference emotion information is often text-type information, and voice interaction information is often voice-type information, that is, reference emotion information and voice interaction information belong to two different modalities of information, in order to accurately determine the interactive emotion information corresponding to the interactive information to a certain extent, after acquiring the voice interaction information, information extraction operations can be performed on the voice interaction information to obtain voice features corresponding to the voice interaction information, and then the interaction intention, reference emotion information and voice features can be encoded and processed to obtain a first vector representation of the interaction intention, a second vector representation of the reference emotion information and a third vector representation corresponding to the voice features, and the first vector representation, the second vector representation and the third vector representation are analyzed and processed, so that the interactive emotion information corresponding to the interactive information can be determined.

[0105] In some instances, the interactive emotion information can be determined by a pre-trained deep learning model. At this time, based on the first vector representation, the second vector representation, and the third vector representation, determining the interactive emotion information corresponding to the interactive information may include: obtaining a pre-trained deep learning model; inputting the first vector representation, the second vector representation, and the third vector representation into the deep learning model, and obtaining the interactive emotion information corresponding to the interactive information output by the deep learning model.

[0106] In other instances, the interactive emotion information can not only be determined by a pre-trained deep learning model, but also can be determined by analyzing and processing the first vector representation, the second vector representation and the third vector representation. At this time, based on the first vector representation, the second vector representation and the third vector representation, determining the interactive emotion information corresponding to the interactive information may include: obtaining a first similarity between the first vector representation and the second vector representation, and a second similarity between the third vector representation and the second vector representation; based on the first similarity and the second similarity, determining the fusion similarity between the voice interaction information and the reference emotion information; and determining the reference emotion information corresponding to the maximum fusion similarity as the interactive emotion information corresponding to the interactive information.

[0107] Specifically, after obtaining the first vector representation, the second vector representation and the third vector representation, the first vector representation, the second vector representation and the third vector representation can be analyzed and calculated to obtain a first similarity between the first vector representation and the second vector representation, and a second similarity between the third vector representation and the second vector representation. Since the first similarity is used to identify the degree of adaptation between the interaction intention and the reference emotional information, and the second similarity is used to identify the degree of adaptation between the voice interaction information and the reference emotional information, in order to accurately determine the interactive emotional information corresponding to the interactive information, the first similarity and the second similarity can be analyzed and processed to determine the fusion similarity between the voice interaction information and the reference emotional information. In some instances, the fusion similarity can be obtained by weighted summing the first similarity and the second similarity; or, the fusion similarity can be obtained by weighted averaging the first similarity and the second similarity; and then the reference emotional information corresponding to the maximum fusion similarity can be determined as the interactive emotional information corresponding to the interactive information, which ensures the accuracy and reliability of the determination of the interactive emotional information to a certain extent.

[0108] Step S302: Based on the interaction intention and the interaction emotion information, generate voice feedback information corresponding to the label set.

[0109] After the interaction intention and interaction emotion information are obtained, the interaction intention and interaction emotion information can be analyzed and processed to generate voice feedback information corresponding to the label set. In some instances, the voice feedback information corresponding to the label set can be determined by a pre-trained deep learning model. At this time, based on the interaction intention and interaction emotion information, generating the voice feedback information corresponding to the label set can include: obtaining a pre-trained deep learning model; inputting the interaction intention and interaction emotion information into the deep learning model for processing, and obtaining the voice feedback information corresponding to the label set output by the deep learning model.

[0110] In other instances, voice feedback information can be determined not only by a pre-trained deep learning model, but also by a character material library and a scene material library. At this time, based on the interaction intention and interaction emotion information, generating voice feedback information corresponding to a label set may include: obtaining a character material library and a scene material library corresponding to the virtual character, the character material library including expression data and action data that the virtual character can achieve, and the scene material library including scene feedback data that the virtual character can generate; based on the character material library, the scene material library, the interaction intention and the interaction emotion information, generating voice feedback information corresponding to the label set.

[0111] Among them, in order to ensure the accuracy and reliability of the generation of voice feedback information to a certain extent, the role material library and scene material library corresponding to the virtual character can be obtained first, wherein the role material library can include expression data and action data that the virtual character can achieve, and the action data can refer to data corresponding to a certain virtual action, such as: data corresponding to a virtual singing action, data corresponding to a virtual hosting action, data corresponding to a virtual opening item action, etc. The scene material library may include scene feedback data that can be generated by the virtual character, such as: scene feedback data corresponding to the virtual character in a virtual office scene, scene feedback data corresponding to the virtual character in a virtual competition scene, scene feedback data corresponding to the virtual character in a virtual battle scene, and so on.

[0112] Different virtual characters may correspond to different character material libraries, and virtual characters in different scenes may correspond to different scene material libraries. Therefore, the character material library and the scene material library may be determined based on the character identification and scene identification of the virtual character. At this time, obtaining the character material library and the scene material library corresponding to the virtual character may include: obtaining the character identification of the virtual character and the scene identification corresponding to the virtual character; based on the character identification, determining the character material library corresponding to the virtual character; based on the scene identification, determining the scene material library corresponding to the virtual character.

[0113] Specifically, in order to improve the accuracy of voice feedback information generation to a certain extent, after obtaining the virtual character, the image of the virtual character can be analyzed and processed to obtain the character identification of the virtual character; and the scene information corresponding to the virtual character can also be analyzed and processed to obtain the scene identification corresponding to the virtual character. Since different character identifications can correspond to different character material libraries, and different scene identifications can correspond to different scene material libraries, after obtaining the character identification, the character material library corresponding to the virtual character can be determined based on the character identification. Similarly, after obtaining the scene identification, the scene material library corresponding to the virtual character can be determined based on the scene identification. This ensures the accuracy and reliability of obtaining the character material library and the scene material library to a certain extent.

[0114] It should be noted that the character material library and the scene material library can not only be determined by analyzing and processing the character identification and the scene identification, but can also be determined by other methods. For example, obtaining the character material library and the scene material library corresponding to the virtual character can include: analyzing and identifying the virtual character, obtaining the virtual character data, and determining the character material library corresponding to the virtual character based on the virtual character data and the preset mapping relationship; identifying the scene information corresponding to the virtual character, obtaining the scene data, and then determining the scene material library corresponding to the virtual character based on the scene data and the preset mapping relationship. This also ensures the accuracy and reliability of the acquisition of the character material library and the scene material library.

[0115] After obtaining the character identification of the virtual character and the scene identification corresponding to the virtual character, the character material library, the scene material library, the interaction intention and the interaction emotion information can be analyzed and processed to generate voice feedback information corresponding to the label set. In some instances, the voice feedback information corresponding to the label set can be determined by a pre-trained deep learning model. At this time, based on the character material library, the scene material library, the interaction intention and the interaction emotion information, generating the voice feedback information corresponding to the label set can include: obtaining a pre-trained deep learning model; inputting the character material library, the scene material library, the interaction intention and the interaction emotion information into the deep learning model for processing, and obtaining the voice feedback information corresponding to the label set output by the deep learning model.

[0116] In this embodiment, by acquiring the interaction intention and interaction emotion information corresponding to the interaction information, and then generating voice feedback information corresponding to the label set based on the interaction intention and interaction emotion information, the voice feedback information generation operation is completed, and the accuracy and reliability of the acquisition of the voice feedback information is guaranteed to a certain extent.

[0117] Figure 4 A flowchart of determining a trigger time point corresponding to a non-verbal interaction tag based on voice feedback information, a tag set and a voice timestamp provided in an embodiment of the present invention; based on any of the above embodiments, refer to the attached Figure 4 As shown, for the trigger time point corresponding to the non-language interaction label, it can not only be determined by a pre-trained deep learning model, but also by analyzing and processing the time axis corresponding to the voice feedback information. At this time, based on the voice feedback information, the label set and the voice timestamp, determining the trigger time point corresponding to the non-language interaction label may include:

[0118] Step S401: Acquire the time axis corresponding to the voice feedback information.

[0119] Step S402: Based on the correspondence between the tag set and the voice feedback information, and the time axis, the tag set is associated with the voice timestamp to obtain the trigger time point corresponding to the non-language interaction tag.

[0120] After obtaining the voice feedback information, the voice feedback information can be analyzed and processed to obtain the time axis corresponding to the voice feedback information. In some instances, the time axis can be determined by the voice broadcast duration of the voice feedback information. At this time, obtaining the time axis corresponding to the voice feedback information can include: performing voice recognition on the voice feedback information to obtain the voice broadcast duration of the voice feedback information; based on the voice broadcast duration, constructing the time axis corresponding to the voice feedback information, thereby completing the time axis acquisition operation.

[0121] In other instances, the timeline can be determined not only by the voice broadcast duration of the voice feedback information, but also by the number of characters, voice type, and speaking speed information of the voice feedback information. In this case, obtaining the timeline corresponding to the voice feedback information may include: obtaining the number of characters, language type, and speaking speed information corresponding to the voice feedback information; and generating a timeline corresponding to the voice feedback information based on the number of characters, language type, and speaking speed information.

[0122] For voice feedback information, the voice broadcast duration of the voice feedback information is often related to the number of characters, voice type and speech speed information in the voice feedback information. In some instances, the voice broadcast duration of the voice feedback information is often positively correlated with the number of characters, that is, the more characters there are in the voice feedback information, the longer the voice broadcast duration of the voice feedback information; the fewer characters there are in the voice feedback information, the shorter the voice broadcast duration of the voice feedback information. Similarly, the voice broadcast duration of the voice feedback information is often negatively correlated with the speech speed information, that is, the faster the speech speed information in the voice feedback information, the shorter the voice broadcast duration of the voice feedback information; the slower the speech speed information in the voice feedback information, the longer the voice broadcast duration of the voice feedback information.

[0123] In order to ensure the accuracy and reliability of the time axis corresponding to the voice feedback information to a certain extent, after obtaining the voice feedback information, the voice feedback information can be analyzed and processed to obtain the number of characters, language type and speech speed information in the voice feedback information, wherein the number of characters and speech speed information corresponding to the voice feedback information can be determined by analyzing and processing the voice feedback information using the automatic speech recognition (Automatic Speech Recognition, referred to as ASR) technology. Specifically, the audio in the voice feedback information can be converted into text, and then the text can be processed by character recognition to obtain the number of characters and speech speed information corresponding to the voice feedback information. At the same time, for the text corresponding to the voice feedback information, the language type of the text can be identified by the language identification technology (Language Identification, referred to as LID), so as to obtain the language type corresponding to the voice feedback information.

[0124] For the speech speed information corresponding to the voice feedback information, it can not only be determined by directly analyzing and processing the voice feedback information through ASR technology, but also the speech speed information of the voice feedback information can be determined in combination with the language type corresponding to the voice feedback information. At this time, obtaining the speech speed information of the voice feedback information may include: obtaining preset speech speed information of the voice feedback information; based on the language type corresponding to the voice feedback information, determining the speech speed offset for adjusting the preset speech speed information; based on the speech speed offset and the preset speech speed information, determining the speech speed information of the voice feedback information.

[0125] Specifically, for the voice feedback information, preset speech speed information is configured for performing voice broadcast operations on the voice feedback information. The preset speech speed information can be a pre-configured default speech speed information. When the speech speed information of the voice feedback information needs to be determined, the preset speech speed information of the voice feedback information can be obtained by accessing a preset area or a preset device. Since the voice broadcast speed of the voice feedback information is related to the language type of the language feedback information, in order to accurately determine the speech speed information of the voice feedback information to a certain extent, the speech speed offset for adjusting the preset speech speed information can be determined based on the language type corresponding to the voice feedback information, wherein the speech speed offset can be determined by the language type and the preset mapping relationship.

[0126] After determining the speech rate offset for adjusting the preset speech rate information, the speech rate feedback information and the preset speech rate information can be analyzed and processed to determine the speech rate information corresponding to the speech rate feedback information. In some examples, the speech rate offset can be a value greater than 0, less than 0, or equal to 0, and the speech rate information corresponding to the speech rate feedback information can be determined by the sum of the preset speech rate information and the speech rate offset, that is, the speech rate information = preset speech rate information + speech rate offset. Alternatively, the speech rate offset can be a value greater than 0 and less than 1, and the speech rate information corresponding to the voice feedback information can be determined by the product value between the preset speech rate information and the speech rate offset, that is, the speech rate information = preset speech rate information * speech rate offset, which ensures the accuracy and reliability of obtaining the speech rate information corresponding to the voice feedback information to a certain extent.

[0127] After obtaining the speech rate information corresponding to the voice feedback information, the number of characters, language type and speech rate information corresponding to the voice feedback information can be analyzed and processed to generate a time axis corresponding to the voice feedback information, which improves the accuracy and reliability of the time axis generation to a certain extent. After obtaining the time axis corresponding to the voice feedback information, the label set can be associated with the voice timestamp based on the correspondence between the label set and the voice feedback information and the time axis, so that the trigger time point corresponding to the non-language interaction label can be obtained, which effectively ensures the flexibility and reliability of the generation of the trigger time point.

[0128] In this embodiment, by acquiring the time axis corresponding to the voice feedback information, and then associating the tag set with the voice timestamp based on the correspondence between the tag set and the voice feedback information, and the time axis, the trigger time point corresponding to the non-verbal interaction tag is obtained, thereby completing the acquisition operation of the trigger time point and being able to ensure the flexibility and reliability of acquiring the trigger time point to a certain extent.

[0129] In specific applications, take 3D virtual characters as examples, refer to the attached Figure 5 As shown, this application embodiment provides a method for interactive control of a 3D virtual character, which can achieve high-precision synchronization of the voice, expression and action of the 3D virtual character in the time dimension, and can fundamentally improve the interactive experience between the user and the 3D virtual character; specifically, the method may include the following steps:

[0130] Step 1: Receive interaction information input by the user.

[0131] When a user has an interactive need for a 3D virtual character, the interactive information input by the user can be obtained. The interactive information can be implemented as any one of the following: text interactive information or voice interactive information. The intention information and emotional information of the interactive information can then be parsed. The above-mentioned intention information and emotional information can provide a basis for subsequent content generation and synchronous operation of multimodal data.

[0132] Specifically, when the interactive information is text interactive information, semantic analysis technology can be used to extract the intention information of the text interactive information, and combined with the sentiment analysis model to generate the user's emotional information (for example: "excited", "sad", "sad", "calm", etc.).

[0133] When the interactive information is voice interaction information, automatic speech recognition ASR technology can be used to transcribe the voice interaction information to obtain intention information, and then the sentiment analysis model can be used to extract sentiment information from the voice content, thereby obtaining sentiment information. The sentiment information corresponding to the voice interaction information can be related to factors such as the tone and speech speed corresponding to the voice interaction information.

[0134] After acquiring the intention information and emotion information of the interaction information by analyzing and processing the interaction information, structured intention information and emotion information can be acquired to facilitate interactive control operations on 3D virtual characters based on the intention information and emotion information. The above-mentioned structured intention information and emotion information can be realized through a json file, that is, the json file can include fields corresponding to the intention information and field information corresponding to the emotion information.

[0135] Step 2: Generate labeled voice feedback content based on the intent information and emotion information output in step 1.

[0136] After obtaining the interactive information input by the user, a large language model (for example, LLM) can be called to label the interactive information to obtain a multimodal label corresponding to the interactive information. The multimodal label can include at least one of the following: an action label, an expression label. Specifically, the large language model can generate labeled voice feedback content based on the intention information, the emotion information, and in combination with the character material library (including at least one of the following: basic data such as expression, action, logic, music, etc.) and the scene material library (including at least one of the following: regional data, regional coordinate points, coordinate point attributes (lighting, environmental data), etc.), wherein the voice feedback content can correspond to a series of emotion labels and action labels (such as ) for describing the behavior of the virtual character.<emotion_happy> ,<action_wave> Etc.), wherein the emotion tag may include the voice emotion information, expression information, etc. of the 3D virtual character.

[0137] Step 3: Synchronization of speech elements and generation of timestamps.

[0138] After obtaining the voice feedback content, the voice feedback content can be synthesized through the Text-to-Speech (TTS) module, and then the phoneme synchronization algorithm can be used to generate accurate start timestamp and end timestamp for each phoneme character in the voice feedback content. The above-mentioned start timestamp is used to identify the start time of the pronunciation of the character, and the end timestamp is used to identify the end time of the pronunciation of the character.

[0139] Specifically, the phoneme synchronization algorithm generates a start timestamp and an end timestamp for each character in the voice content, which may include: segmenting the voice feedback content and the corresponding multimodal label through the phoneme synchronization algorithm to obtain processed voice content; and then processing the processed voice content to generate an accurate start timestamp and an end timestamp for each character in the voice content.

[0140] After obtaining the start timestamp and end timestamp corresponding to each character, a timeline corresponding to the voice feedback content can be constructed based on the start timestamp and end timestamp of each character. Through the constructed timeline, the generated start timestamp corresponding to each character can be associated and bound with the multimodal tag corresponding to the voice feedback content, so that the timeline corresponding to the multimodal tag can be constructed. The timeline corresponding to the multimodal tag can include a voice timestamp (with emotional information integrated), an expression trigger time point, and an action trigger time point, so as to coordinate and control the triggering sequence of the voice, expression, and action of the 3D virtual character through the timeline.

[0141] It should be noted that when the expressions of a 3D virtual character include multiple different expressions, a transition expression between any two different expressions can be automatically generated among the different expressions, and the timestamp of the transition expression can be between the start timestamps corresponding to the two different expressions; similarly, when the actions of the 3D virtual character include multiple different actions, a transition action between any two different actions can be automatically generated between the different actions, and the timestamp of the transition action can be between the start timestamps corresponding to the two different actions.

[0142] Step 4: Perform character rendering operations based on the timeline corresponding to the multimodal tag.

[0143] After obtaining the timeline corresponding to the multimodal tag, the character rendering data of the 3D character can be generated based on the timeline corresponding to the multimodal tag, and then the real-time rendering operation of the 3D virtual character can be completed through the rendering engine. Specifically, in order to accurately realize the real-time rendering operation of the 3D virtual character, the expression information and action information of the 3D virtual character can be generated based on the character rendering data, wherein the generation operation of the expression information can be performed by<emotion_happy> By performing parsing operations, facial expressions that match emotional characteristics (such as smiling, frowning) can be dynamically generated. Similarly, the generation of action information can be done by analyzing action tags.<action_wave> By performing parsing operations, body movement information (such as waving and nodding) that is semantically related to the voice feedback content can be generated, thereby realizing the collaborative output operation of multimodal label information.

[0144] After acquiring the expression information and action information, the generated voice feedback content, expression information and action information can be passed to the real-time rendering engine to generate a smooth 3D character dynamic display effect.

[0145] Step 5: Multimodal result output of 3D virtual character.

[0146] Among them, after obtaining the voice feedback content, expression information and action information of the 3D virtual character, the final interactive effect visible to the user can be output for the 3D virtual character and presented, which can specifically include the following aspects: (1) Voice output: voice content generated by the character playback, with voice synchronization with high-precision timestamps; (2) Expression output: dynamic facial expressions consistent with emotions; (3) Action output: body movements related to semantics; (4) Command output: at least one command information used to implement the action of the 3D virtual character.

[0147] The technical solution provided by this application embodiment can achieve the following technical effects:

[0148] (1) Based on the phoneme synchronization algorithm, accurate synchronization operations between multimodal data are achieved;

[0149] The phoneme synchronization algorithm generates accurate start and end timestamps for each phoneme in the voice feedback content, and binds them to a unified timeline with emotion and action tags, thereby ensuring a high degree of consistency in the time dimension between the voice feedback content, expression information, and action information. This solves the asynchrony problem between different modal data in related technologies (the problem of asynchrony between voice, expression, and action in the time dimension), significantly improving the naturalness of the interaction of 3D virtual characters, and effectively achieving a fundamental improvement in the user's interactive experience.

[0150] (2) The consistency of the 3D virtual character’s expression, movements, and voice emotions;

[0151] The LLM-based labeling content generation framework generates corresponding multimodal data labels. Specifically, it can perform parsing operations based on the semantic information and emotional information input by the user, and can generate emotional labels (such as<emotion_happy> ) and action tags (such as<action_wave> ), the framework can dynamically adjust the expression and action parameters of the 3D virtual character to make them highly consistent with the speech emotion and semantics; this effectively realizes the coordinated generation of speech, expression and action through the label-driven mechanism, greatly improving the naturalness and fluency of the 3D virtual character interaction, and because of the combination of emotional generation technology, it not only improves the emotional expressiveness and naturalness of multimodal content, making the output of multimodal data labels of 3D virtual characters more humanized, but also significantly reduces the time cost of manually producing multimodal content, effectively improving the user's immersive interactive experience.

[0152] (3) Adapt to multi-language scenarios;

[0153] In multilingual scenarios, it supports multiple language scenarios such as Chinese, English, Japanese, and Korean. By optimizing the adaptation logic of voice speed and intonation, it automatically adapts to the voice rhythm and modal coordination requirements of different languages, thereby achieving cross-language modal synchronization. In addition, the triggering timing of voice, expression, and action is coordinated through the timeline management mechanism to ensure natural interaction in a multilingual environment, which effectively optimizes the cross-language interactive experience and broadens the scope of international application.

[0154] In addition, this technical solution can also combine the multi-language adaptation module to generate the actions and expressions of 3D virtual characters. Specifically, the trigger logic of speech generation and action expression generation can be dynamically adjusted according to the speech speed and tone characteristics of different languages. For example, in the Japanese scene, the trigger time of expressions and actions can be optimized by appropriately increasing the speech speed offset. In addition, the timeline management mechanism supports cross-language time mapping to ensure modal consistency in a multi-language environment.

[0155] (4) Real-time multimodal rendering technology;

[0156] A tag-driven parallel rendering mechanism was introduced to decouple the speech, expression, and action generation operations, and to coordinate their triggering time points through a timeline. This allows for parallel processing of speech, expression, and action generation tasks. This is then combined with an efficient real-time rendering engine (Unity) to achieve low-latency multimodal information output, further optimizing rendering performance and reducing latency.

[0157] (5) The integrated multimodal generation process framework integrates the complete process from user command parsing, labeled content generation, phoneme synchronization to multimodal rendering, and realizes the automated generation of voice, expression, and action. This method can be widely used in virtual humans, 3D animations, and metaverse scenes, significantly reducing the time cost of manually producing multimodal content and further improving the scope of application and practicality of this method.

[0158] Figure 6 A schematic diagram of a virtual character interactive control device provided by an embodiment of the present invention; Figure 6 As shown, this embodiment provides an interactive control device for a virtual character, and the interactive control device is used to perform the above Figure 2 The interactive control method of the virtual character shown, specifically, the interactive control device may include:

[0159] A first acquisition module 11 is used to generate voice feedback information corresponding to the interaction information in response to the interaction information for the virtual character, wherein the voice feedback information corresponds to a tag set, the tag set includes at least one non-verbal interaction tag for defining the virtual character, and timing information corresponding to the non-verbal interaction tag;

[0160] A first determining module 12, used to determine the speech timestamp corresponding to the speech element in the speech feedback information;

[0161] The first determination module 12 is further used to determine the trigger time point corresponding to the non-language interaction tag based on the voice feedback information, the tag set and the voice timestamp;

[0162] The first processing module 13 is used to control the virtual character to perform interactive operations based on the voice feedback information, the voice timestamp and the trigger time point corresponding to the non-language interaction tag.

[0163] In some instances, when the first acquisition module 11 generates voice feedback information corresponding to the interaction information, the first acquisition module 11 is used to execute: obtaining the interaction intention and interaction emotion information corresponding to the interaction information; and generating voice feedback information corresponding to a label set based on the interaction intention and interaction emotion information.

[0164] In some instances, when the first acquisition module 11 acquires the interaction intention corresponding to the interaction information, the first acquisition module 11 is used to execute: acquiring the triggering user information and / or context information of the interaction information; and determining the interaction intention corresponding to the interaction information based on the interaction information, the triggering user information and / or the context information.

[0165] In some instances, when the first acquisition module 11 acquires interactive emotion information corresponding to the interactive information, the first acquisition module 11 is used to execute: when the interactive information is text interactive information, acquiring at least one reference emotional information; determining a first vector representation of the interactive intention and a second vector representation of the reference emotional information; and determining the interactive emotion information corresponding to the interactive information based on the first vector representation and the second vector representation.

[0166] In some instances, when the first acquisition module 11 determines the interactive emotion information corresponding to the interactive information based on the first vector representation and the second vector representation, the first acquisition module 11 is used to perform: obtaining the similarity between the first vector representation and the second vector representation; determining the target vector representation corresponding to the maximum similarity among multiple second vector representations; and determining the reference emotion information corresponding to the target vector representation as the interactive emotion information corresponding to the interactive information.

[0167] In some instances, when the first acquisition module 11 acquires interactive emotion information corresponding to the interactive information, the first acquisition module 11 is used to execute: when the interactive information is voice interactive information, acquire at least one reference emotional information and a voice feature corresponding to the voice interactive information; determine a first vector representation of the interactive intention, a second vector representation of the reference emotional information, and a third vector representation corresponding to the voice feature; and determine the interactive emotion information corresponding to the interactive information based on the first vector representation, the second vector representation, and the third vector representation.

[0168] In some instances, when the first acquisition module 11 determines the interactive emotion information corresponding to the interactive information based on the first vector representation, the second vector representation, and the third vector representation, the first acquisition module 11 is used to perform: obtaining a first similarity between the first vector representation and the second vector representation, and a second similarity between the third vector representation and the second vector representation; determining a fusion similarity between the speech interaction information and the reference emotion information based on the first similarity and the second similarity; and determining the reference emotion information corresponding to the maximum fusion similarity as the interactive emotion information corresponding to the interactive information.

[0169] In some instances, when the first acquisition module 11 generates voice feedback information corresponding to a label set based on interaction intention and interaction emotion information, the first acquisition module 11 is used to execute: obtaining a character material library and a scene material library corresponding to the virtual character, the character material library including expression data and action data that the virtual character can achieve, and the scene material library including scene feedback data that the virtual character can generate; based on the character material library, the scene material library, interaction intention and interaction emotion information, generating voice feedback information corresponding to the label set.

[0170] In some instances, when the first acquisition module 11 acquires the character material library and the scene material library corresponding to the virtual character, the first acquisition module 11 is used to execute: acquiring the character identification of the virtual character and the scene identification corresponding to the virtual character; based on the character identification, determining the character material library corresponding to the virtual character; based on the scene identification, determining the scene material library corresponding to the virtual character.

[0171] In some instances, when the first determination module 12 determines the voice timestamp corresponding to the voice element in the voice feedback information, the first determination module 12 is used to execute: segmenting the label set and the voice feedback information to obtain processed feedback information; determining the voice elements in the processed feedback information, wherein the voice elements include at least one of the following: character elements, phrase elements; determining the start timestamp and end timestamp corresponding to the voice elements based on the processed feedback information; and determining the start timestamp and end timestamp corresponding to the voice elements as the voice timestamp corresponding to the voice elements.

[0172] In some instances, when the first determination module 12 determines the trigger time point corresponding to the non-verbal interaction tag based on the voice feedback information, the tag set and the voice timestamp, the first determination module 12 is used to execute: obtaining the time axis corresponding to the voice feedback information; based on the correspondence between the tag set and the voice feedback information, and the time axis, associating the tag set with the voice timestamp to obtain the trigger time point corresponding to the non-verbal interaction tag.

[0173] In some instances, when the first determination module 12 obtains the timeline corresponding to the voice feedback information, the first determination module 12 is used to execute: obtaining the number of characters, language type, and speaking speed information corresponding to the voice feedback information; and generating a timeline corresponding to the voice feedback information based on the number of characters, language type, and speaking speed information.

[0174] In some instances, when the first determination module 12 obtains the speech speed information of the voice feedback information, the first determination module 12 is used to execute: obtaining the preset speech speed information of the voice feedback information; determining the speech speed offset for adjusting the preset speech speed information based on the language type corresponding to the voice feedback information; and determining the speech speed information of the voice feedback information based on the speech speed offset and the preset speech speed information.

[0175] In some instances, when the first processing module 13 controls the virtual character to perform interactive operations based on the voice feedback information, the voice timestamp, and the trigger time point corresponding to the non-verbal interaction tag, the first processing module 13 is used to execute: based on the voice feedback information, the voice timestamp, and the trigger time point corresponding to the non-verbal interaction tag, generate non-verbal interaction data corresponding to the virtual character, the non-verbal interaction data including at least one of the following: expression data, action data; send the voice feedback information and the non-verbal interaction data to the rendering engine, so that the rendering engine renders the virtual character based on the voice feedback information and the non-verbal interaction data, and controls the virtual character to perform interactive operations.

[0176] Figure 6 The device shown can perform Figure 1-Figure 5 For the method of the embodiment shown in the figure, the part not described in detail in this embodiment can be referred to Figure 1-Figure 5 The implementation process and technical effects of this technical solution refer to Figure 1-Figure 5 The description in the illustrated embodiment will not be repeated here.

[0177] In one possible design, Figure 6 The structure of the interactive control device for the virtual character shown can be implemented as an electronic device, which can be a controller, a personal computer, a server, or other devices. Figure 7 As shown, the electronic device may include: a first processor 21 and a first memory 22. The first memory 22 is used to store the corresponding electronic device to execute the above Figure 1-Figure 5 The program of the interactive control method of the virtual character provided in the illustrated embodiment, the first processor 21 is configured to execute the program stored in the first memory 22 .

[0178] The program includes one or more computer instructions, wherein the one or more computer instructions can implement the following steps when executed by the first processor 21: in response to interaction information for a virtual character, generate voice feedback information corresponding to the interaction information, wherein the voice feedback information corresponds to a tag set, the tag set includes at least one non-verbal interaction tag for limiting the virtual character, and timing information corresponding to the non-verbal interaction tag; determine the voice timestamp corresponding to the voice element in the voice feedback information; determine the trigger time point corresponding to the non-verbal interaction tag based on the voice feedback information, the tag set and the voice timestamp; and control the virtual character to perform an interactive operation based on the voice feedback information, the voice timestamp and the trigger time point corresponding to the non-verbal interaction tag.

[0179] Furthermore, the first processor 21 is also used to execute the aforementioned Figure 1-Figure 5 All or part of the steps in the illustrated embodiments.

[0180] The structure of the electronic device may further include a first communication interface 23 for the electronic device to communicate with other devices or a communication network.

[0181] In addition, an embodiment of the present invention provides a computer storage medium for storing computer software instructions used by electronic devices, which includes instructions for executing the above Figure 1-Figure 5 The program involved in the interactive control method of the virtual character in the illustrated embodiment.

[0182] In addition, an embodiment of the present invention provides a computer program product, including: a computer-readable storage medium storing computer instructions, when the computer instructions are executed by one or more processors, causing the one or more processors to execute the above Figure 1-Figure 5 The steps in the interactive control method of the virtual character in the method embodiment are shown.

[0183] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in the embodiments of the present application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0184] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, i.e., they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Those of ordinary skill in the art may understand and implement it without creative effort.

[0185] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by adding a necessary general hardware platform, and of course can also be implemented by combining hardware and software. Based on such an understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a computer product, and the present invention can be in the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0186] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable device to produce a machine, so that the instructions executed by the processor of the computer or other programmable device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0187] These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable device to work in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 These computer program instructions can also be loaded onto a computer or other programmable device so that a series of operation steps are executed on the computer or other programmable device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable device provide the functions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps of the functions specified in the blocks or multiple blocks. In a typical configuration, the computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory. The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0188] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0189] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for interactive control of a virtual character, characterized in that: include: In response to interaction information for a virtual character, voice feedback information corresponding to the interaction information is generated, wherein the voice feedback information corresponds to a tag set, the tag set including at least one non-verbal interaction tag for defining the virtual character, and timing information corresponding to the non-verbal interaction tag; Determining a speech timestamp corresponding to a speech element in the speech feedback information; Determining a trigger time point corresponding to the non-verbal interaction tag based on the voice feedback information, the tag set and the voice timestamp; Based on the voice feedback information, the voice timestamp and the triggering time point corresponding to the non-verbal interaction tag, the virtual character is controlled to perform an interactive operation.

2. The method according to claim 1, characterized in that Generating voice feedback information corresponding to the interaction information, including: Acquire interaction intention and interaction emotion information corresponding to the interaction information; Based on the interaction intention and the interaction emotion information, voice feedback information corresponding to the label set is generated.

3. The method according to claim 2, characterized in that Acquiring an interaction intention corresponding to the interaction information includes: Acquire triggering user information and / or context information of the interaction information; Based on the interaction information, triggering user information and / or context information, an interaction intention corresponding to the interaction information is determined.

4. The method according to claim 2, characterized in that: Acquiring interactive emotion information corresponding to the interactive information includes: When the interaction information is text interaction information, obtaining at least one reference emotion information; Determine a first vector representation of the interaction intention and a second vector representation of the reference emotion information; Based on the first vector representation and the second vector representation, interactive emotion information corresponding to the interactive information is determined.

5. The method according to claim 4, characterized in that Determining interaction emotion information corresponding to the interaction information based on the first vector representation and the second vector representation includes: Obtaining a similarity between the first vector representation and the second vector representation; Determine a target vector representation corresponding to a maximum similarity among the plurality of second vector representations; The reference emotion information corresponding to the target vector representation is determined as the interactive emotion information corresponding to the interactive information.

6. The method according to claim 2, characterized in that Acquiring interactive emotion information corresponding to the interactive information includes: In the case where the interaction information is voice interaction information, obtaining at least one reference emotion information and a voice feature corresponding to the voice interaction information; Determine a first vector representation of the interaction intention, a second vector representation of the reference emotion information, and a third vector representation corresponding to the voice feature; Based on the first vector representation, the second vector representation, and the third vector representation, interactive emotion information corresponding to the interactive information is determined.

7. The method according to claim 6, characterized in that Determining interactive emotion information corresponding to the interactive information based on the first vector representation, the second vector representation, and the third vector representation includes: Obtaining a first similarity between the first vector representation and the second vector representation, and a second similarity between the third vector representation and the second vector representation; Determining a fusion similarity between the voice interaction information and the reference emotion information based on the first similarity and the second similarity; The reference emotion information corresponding to the maximum fusion similarity is determined as the interactive emotion information corresponding to the interactive information.

8. The method according to claim 2, characterized in that: Based on the interaction intention and the interaction emotion information, voice feedback information corresponding to the label set is generated, including: Acquire a character material library and a scene material library corresponding to the virtual character, wherein the character material library includes expression data and action data that the virtual character can realize, and the scene material library includes scene feedback data that the virtual character can generate; Based on the role material library, the scene material library, the interaction intention and the interaction emotion information, voice feedback information corresponding to the label set is generated.

9. The method according to claim 8, characterized in that Acquiring a character material library and a scene material library corresponding to the virtual character, including: Obtaining a role identifier of the virtual role and a scene identifier corresponding to the virtual role; Based on the character identifier, determining a character material library corresponding to the virtual character; Based on the scene identifier, a scene material library corresponding to the virtual character is determined.

10. The method according to any one of claims 1 to 9, characterized in that: Determining a speech timestamp corresponding to the speech element in the speech feedback information includes: Segmenting the tag set and the voice feedback information to obtain processed feedback information; Determining a speech element in the processed feedback information, wherein the speech element includes at least one of the following: a character element and a phrase element; Based on the processed feedback information, determining a start timestamp and an end timestamp corresponding to the speech element; The start timestamp and the end timestamp corresponding to the voice element are determined as the voice timestamp corresponding to the voice element.

11. The method according to any one of claims 1 to 9, characterized in that: Determining a trigger time point corresponding to the non-language interaction tag based on the voice feedback information, the tag set, and the voice timestamp includes: Obtaining the time axis corresponding to the voice feedback information; Based on the correspondence between the tag set and the voice feedback information, and the time axis, the tag set is associated with the voice timestamp to obtain the triggering time point corresponding to the non-language interaction tag.

12. The method according to claim 11, characterized in that Obtaining the time axis corresponding to the voice feedback information includes: Obtaining the number of characters, language type, and speech speed information corresponding to the voice feedback information; Based on the number of characters, the language type, and the speech rate information, a time axis corresponding to the voice feedback information is generated.

13. The method according to claim 12, characterized in that Acquiring speech speed information of the voice feedback information includes: Acquire preset speech speed information of the voice feedback information; Determining a speech rate offset for adjusting the preset speech rate information based on the language type corresponding to the voice feedback information; Based on the speech rate offset and the preset speech rate information, the speech rate information of the voice feedback information is determined.

14. The method according to any one of claims 1 to 9, characterized in that: Based on the voice feedback information, the voice timestamp, and the trigger time point corresponding to the non-language interaction tag, controlling the virtual character to perform an interactive operation includes: Based on the voice feedback information, the voice timestamp and the trigger time point corresponding to the non-language interaction tag, non-language interaction data corresponding to the virtual character is generated, and the non-language interaction data includes at least one of the following: expression data and action data; The voice feedback information and the non-language interaction data are sent to a rendering engine, so that the rendering engine renders the virtual character based on the voice feedback information and the non-language interaction data, and controls the virtual character to perform interactive operations.

15. An interactive control device for a virtual character, characterized in that: include: A first acquisition module, configured to generate, in response to interaction information for a virtual character, voice feedback information corresponding to the interaction information, wherein the voice feedback information corresponds to a tag set, the tag set including at least one non-language interaction tag for defining the virtual character, and timing information corresponding to the non-language interaction tag; A first determining module, used to determine the speech timestamp corresponding to the speech element in the speech feedback information; The first determination module is further configured to determine a trigger time point corresponding to the non-language interaction tag based on the voice feedback information, the tag set and the voice timestamp; The first processing module is used to control the virtual character to perform interactive operations based on the voice feedback information, the voice timestamp and the trigger time point corresponding to the non-language interaction tag.

16. An electronic device, characterized in that: include: A memory, a processor; wherein the memory is used to store one or more computer instructions, wherein the one or more computer instructions, when executed by the processor, implement the method as described in any one of claims 1-14.

17. A computer storage medium, characterized in that: Used to store a computer program, which enables a computer to implement the method of any one of claims 1 to 14 when executed.

18. A computer program product, characterized in that include: A computer program, when executed by a processor of an electronic device, causes the processor to execute the steps of any one of the methods of claims 1 to 14.

Images