Method, device and equipment for content interaction, storage medium and program product

Abstract

The invention discloses a content interaction method and device, equipment, a storage medium and a program product. The method comprises the steps that in a 3D modeling engine of the electronic equipment, a data callback monitoring function for an operating system of the electronic equipment is set, the operating system is at least configured to execute voice synthesis, and the 3D modeling engine is configured to generate audio and video interaction content of a digital person based on voice; in response to the received to-be-synthesized text, generating voice corresponding to the text by using a voice synthesis interface included in the operating system; sending the generated voice to a 3D modeling engine via a data callback monitoring function; and generating, via a 3D modeling engine, audio-video interaction content of the digital person based on the speech. In this way, the audio and video interaction content of the digital human can be generated by the 3D modeling engine according to the voice generated by the operating system, so that the waiting time of a user is shortened, and the user experience is improved.

Description

TECHNICAL FIELD

[0001] Example embodiments of the present disclosure generally relate to the field of computers, and in particular, to a method, an apparatus, a device, a computer-readable storage medium, and a computer program product for content interaction. BACKGROUND

[0002] With the development of technologies such as Virtual Reality (VR), Augmented Reality (AR), and Mix Reality (MR), more and more three-dimensional virtual objects (e.g., three-dimensional digital humans) are applied in different scenarios. Accordingly, with the rise of three-dimensional (3D) modeling engines, it has become possible to synchronize a digital human form based on speech. SUMMARY

[0003] In a first aspect of the present disclosure, a method for content interaction is provided. The method includes: in a 3D modeling engine of an electronic device, setting a data callback listening function for an operating system of the electronic device, the operating system being configured to at least perform speech synthesis, and the 3D modeling engine being configured to generate audio and video interaction content of a digital human based on speech; in response to receiving text to be synthesized, generating speech corresponding to the text by using a speech synthesis interface included in the operating system; sending the generated speech to the 3D modeling engine via the data callback listening function; and generating the audio and video interaction content of the digital human based on the speech via the 3D modeling engine.

[0004] In a second aspect of the present disclosure, an apparatus for content interaction is provided. The apparatus includes: a data callback listening function setting module configured to, in a 3D modeling engine of an electronic device, set a data callback listening function for an operating system of the electronic device, the operating system being configured to at least perform speech synthesis, and the 3D modeling engine being configured to generate audio and video interaction content of a digital human based on speech; a speech generation module configured to, in response to receiving text to be synthesized, generate speech corresponding to the text by using a speech synthesis interface included in the operating system; a speech sending module configured to send the generated speech to the 3D modeling engine via the data callback listening function; and an audio and video interaction content generation module configured to generate the audio and video interaction content of the digital human based on the speech via the 3D modeling engine.

[0005] In a third aspect of the present disclosure, an electronic device is provided. The device includes at least one processing unit; and at least one memory coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit. The instructions, when executed by the at least one processing unit, cause the device to perform the method of the first aspect.

[0006] In a fourth aspect of this disclosure, a computer-readable storage medium is provided. The computer-readable storage medium stores a computer program that can be executed by a processor to implement the method of the first aspect.

[0007] In a fifth aspect of this disclosure, a computer program product is provided. The computer program product is tangibly stored in a computer storage medium and includes computer-executable instructions that, when executed by a device, cause the device to perform the method of the first aspect.

[0008] It should be understood that the content described in this section is not intended to limit the key or essential features of the embodiments of this disclosure, nor is it intended to restrict the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0009] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. In the drawings, the same or similar reference numerals denote the same or similar elements, wherein:

[0010] Figure 1 A schematic diagram of an example environment in which embodiments of the present disclosure can be implemented is shown;

[0011] Figure 2 A schematic diagram of an example architecture for content interaction according to some embodiments of the present disclosure is shown;

[0012] Figure 3 A flowchart illustrating a content interaction process according to some embodiments of this disclosure is shown;

[0013] Figure 4 Block diagrams of apparatus for content interaction according to some embodiments of the present disclosure are shown; and

[0014] Figure 5 A block diagram of an electronic device capable of implementing one or more embodiments of the present disclosure is shown. Detailed Implementation

[0015] It is understood that before using the technical solutions disclosed in the various embodiments of this disclosure, users should be informed of the types, scope of use, and usage scenarios of the personal information involved in this disclosure in an appropriate manner in accordance with relevant laws and regulations, and user authorization should be obtained.

[0016] For example, upon receiving a user's active request, a prompt message is sent to the user to explicitly inform them that the requested operation will require the acquisition and use of the user's personal information. This allows the user to independently choose whether to provide personal information to the software or hardware, such as the electronic device, application, server, or storage medium performing the operations of this disclosed technical solution, based on the prompt message.

[0017] As an optional but non-limiting implementation, in response to a user's active request, sending a prompt message to the user can be done via a pop-up window, where the prompt message can be presented in text format. Furthermore, the pop-up window can also include a selection control allowing the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0018] It is understood that the above notification and user authorization process are merely illustrative and do not constitute a limitation on the implementation of this disclosure. Other methods that comply with relevant laws and regulations may also be applied to the implementation of this disclosure.

[0019] It is understood that the data involved in this technical solution (including but not limited to the data itself, the acquisition or use of the data) shall comply with the requirements of relevant laws, regulations and related provisions.

[0020] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0021] It should be noted that the headings of any section / subsection provided herein are not limiting. Various embodiments are described throughout this document, and embodiments of any type may be included under any section / subsection. Furthermore, embodiments described in any section / subsection may be combined in any way with any other embodiments described in the same section / subsection and / or different sections / subsections.

[0022] In the description of embodiments of this disclosure, the term "comprising" and similar terms should be understood as open-ended inclusion, i.e., "including but not limited to". The term "based on" should be understood as "at least partially based on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The term "some embodiments" should be understood as "at least some embodiments". Other explicit and implicit definitions may also be included below. The terms "first", "second", etc., may refer to different or the same objects. Other explicit and implicit definitions may also be included below.

[0023] As briefly mentioned earlier, with the rise of 3D modeling engines, it has become possible to synchronize digital lip movements based on speech. Currently, some operating systems allow the generation of speech synthesis files using third-party speech synthesis application programming interfaces (APIs). These speech files are then passed to the 3D engine, where lip-syncing algorithms are used to achieve digital lip movement synchronization.

[0024] However, generating speech-synthesized files through network interface requests using third-party speech synthesis services takes a considerable amount of time. Furthermore, the large file size generated in this way increases download time, thus degrading the user experience.

[0025] In view of this, embodiments of the present disclosure propose an improved content interaction scheme. According to various embodiments of the present disclosure, a data callback monitoring function for the operating system of an electronic device is set in the 3D modeling engine of the electronic device. The operating system is at least configured to perform speech synthesis, and the 3D modeling engine is configured to generate audio-visual interactive content for a digital human based on the speech. If text to be synthesized is received, the speech corresponding to the text is generated using the speech synthesis interface included in the operating system. Accordingly, the generated speech is sent to the 3D modeling engine via the data callback monitoring function. Then, the audio-visual interactive content for the digital human is generated based on the speech via the 3D modeling engine.

[0026] Therefore, this method enables the 3D modeling engine to generate audio and video interactive content for digital humans based on the voice generated by the operating system, thereby reducing user waiting time and improving user experience.

[0027] Figure 1 A schematic diagram of an example environment 100 in which embodiments of the present disclosure can be implemented is shown. For example... Figure 1 As shown, example environment 100 may include electronic device 110. In this example environment 100, electronic device 110 may run an application 120 that supports content interaction. Application 120 may be any suitable type of application for content interaction, examples of which may include, but are not limited to: 3D modeling applications, game engines, graphics applications, AR (augmented reality) applications, or other suitable applications. User 140 may interact with application 120 via electronic device 110 and / or its attached devices.

[0028] In some embodiments, application 120 may utilize machine learning model 160 (which may include one or more machine learning models, such as machine learning model 160-1, machine learning model 160-2, ..., machine learning model 160-N, etc., where N is a positive integer. For ease of description, the one or more machine learning models are collectively referred to herein as machine learning model 160) to support interaction with user 140. For example, application 120 or its digital assistant may utilize one or more machine learning models 160 to provide question-and-answer services to user 140. In some embodiments, machine learning model 160 may be of different types. In some embodiments, the machine learning model used is a content-generating model, capable of generating corresponding outputs based on model inputs.

[0029] exist Figure 1 In environment 100, if application 120 is active, electronic device 110 can present interface 150 for interactive control via application 120. Interface 150 can present audio and video interactive content of the digital human, such as spoken video from the digital human. In some embodiments, the digital human may sometimes be referred to as a virtual avatar. Furthermore, in addition to 3D models of humans, other digital humans can be designed.

[0030] In some embodiments, electronic device 110 communicates with server 130 to provide services to application 120. For example... Figure 1 As shown, server 130 can invoke machine learning model 160 to support human-computer dialogue between application 120 and user 140 based on the output of machine learning model 160. In some embodiments, electronic device 110 can also directly invoke machine learning model 160 to support human-computer dialogue between application 120 and user 140 based on the output of machine learning model 160. Electronic device 110 can be any type of mobile terminal, fixed terminal, or portable terminal, including mobile phones, desktop computers, laptop computers, notebook computers, netbook computers, tablet computers, media computers, multimedia tablets, handheld computers, portable gaming terminals, VR / AR devices, personal communication system (PCS) devices, personal navigation devices, personal digital assistants (PDAs), audio / video players, digital cameras / camcorders, positioning devices, television receivers, radio receivers, e-book devices, gaming devices, or any combination of the foregoing, including accessories and peripherals of these devices or any combination thereof. In some embodiments, electronic device 110 can also support any type of user-facing interface (such as "wearable" circuitry).

[0031] Server 130 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks, and big data and artificial intelligence platforms. Server 130 may include, for example, computing systems / servers such as mainframes, edge computing nodes, computing devices in a cloud environment, etc. Server 130 can provide backend services for applications 120 that support interactive control in electronic devices 110.

[0032] A communication connection can be established between server 130 and electronic device 110. This communication connection can be established via wired or wireless means. The communication connection may include, but is not limited to, Bluetooth, mobile network, Universal Serial Bus (USB), and Wireless Fidelity (WiFi) connections; the embodiments of this disclosure are not limited in this respect. In the embodiments of this disclosure, server 130 and electronic device 110 can achieve signaling interaction through the communication connection between them.

[0033] It should be understood that the structure and function of the various elements in environment 100 are described for illustrative purposes only and do not imply any limitation on the scope of this disclosure.

[0034] The following is for reference Figure 2 Example embodiments of this disclosure are described. Figure 2 A schematic diagram of an example architecture 200 for content interaction according to some embodiments of the present disclosure is shown. Example architecture 200 can be implemented at electronic device 110.

[0035] In some embodiments, the electronic device 110 includes a data callback listening function for its operating system within its included three-dimensional (3D) modeling engine. The operating system is configured at least to perform speech synthesis. The 3D modeling engine in the electronic device 110 is configured to generate audio-visual interactive content for a digital human based on speech.

[0036] In some examples, electronic device 110 establishes a data interaction channel between the operating system and the 3D modeling engine by setting up a data callback listener function in the 3D modeling engine that targets the operating system (e.g., the distributed operating system Harmony, but not limited to this). The following first refers to... Figure 2 The description of electronic device 110 establishes a data interaction channel between the operating system and the 3D modeling engine.

[0037] In some embodiments, the electronic device 110 creates a script file for audio within a 3D modeling engine. The electronic device 110 registers an audio management object, included in the operating system, into the script file; this audio management object is configured to transmit speech to the 3D modeling engine. In some embodiments, the audio management object is a singleton object. Then, after the audio management object is registered to the script file, the electronic device 110 registers a data callback listener function for the audio management object within the 3D modeling engine.

[0038] like Figure 2 In the example architecture 200 shown, the electronic device 110 creates a script file for audio within the 3D modeling engine 220 (e.g., a script file that enables the 3D modeling engine to call audio objects in the operating system, which may be represented as U3dAudioPlugin.tslib, but is not limited to this). In some examples, the electronic device 110 may also add methods to the script file for export to the operating system (e.g., the RegisterU3dAudioPlugin method). In some examples, the RegisterU3dAudioPlugin method may instruct the 3D modeling engine 220 to export a method for use by the operating system 210. Accordingly, the electronic device 110 registers (217) the audio management objects included in the operating system 210 that need to be exported to the 3D modeling engine 220 to the script file. In this way, the 3D modeling engine 220 can call the audio management objects included in the operating system 210.

[0039] In some embodiments, the audio management object included in the operating system 210 may be an audio management singleton object. That is, the electronic device 110 adds an audio management singleton object 216 to the operating system 210. The audio management singleton object 216 is used to transmit audio stream data for speech synthesis. Then, the electronic device 110 customizes the digital human audio processing script in the 3D modeling engine 220, and after obtaining the audio management singleton object 216 included in the operating system 210, it registers (218) a data callback listening function (e.g., an object for passing callbacks, which can be represented as OpenHarmonyJSCallback) 219 on the audio management singleton object 216. Thus, the operating system 210 can transmit data to the 3D modeling engine 220 through this callback listening function 219.

[0040] In some embodiments, the electronic device 110 adds an event subscription to the operating system, the event subscription indicating cross-thread transmission of voice. Then, the electronic device 110 transmits the generated voice from the thread corresponding to speech synthesis in the operating system to the callback thread corresponding to the 3D modeling engine via the event subscription, the callback thread instructing the 3D modeling engine to retrieve the voice generated by the operating system from the callback thread.

[0041] Understandably, the 3D modeling engine 220 performs callbacks in a separate thread, and the operating system 210 synthesizes speech in a separate thread. Since the operating system (e.g., Harmony, but not limited to) does not support cross-thread communication, the audio management singleton object 216 becomes invalid during cross-thread communication, thus preventing direct writing of data to the callback. Therefore, the electronic device 110 adds an audio stream data event subscription in its constructor. If other threads publish audio stream data corresponding to the speech to this event subscription, the audio stream data corresponding to the speech included in the operating system can be transmitted to the callback thread.

[0042] The following is a reference. Figure 2 This describes the process of content interaction after the operating system 210 and the 3D modeling engine 220 are integrated.

[0043] In some embodiments, if the electronic device 110 receives text to be synthesized, it uses the speech synthesis interface included in the operating system to generate corresponding speech. In some embodiments, the text to be synthesized received by the electronic device 110 may be text generated from user messages via a large language model.

[0044] In some embodiments, electronic device 110 receives user messages from a user. Subsequently, electronic device 110 obtains text based on the user messages using a large language model. For example... Figure 2 As shown, a machine learning model (e.g., a large language model) 160 generates text 213 for user message 212 (e.g., a question-and-answer message). For example, if electronic device 110 receives user message 212 from a user to a health large model: "How to treat dry eye?", it requests the large model interface service to receive text 213 from the large model in response: "Dry eye sufferers should not look at electronic products too much...". In some embodiments, the text may also be text provided by the user.

[0045] In some embodiments, the electronic device 110 may utilize a speech synthesis interface included in the operating system to generate at least one speech segment based on text and predetermined playback parameters. In some examples, the predetermined playback parameters (playType: 0) indicate that only the speech is synthesized and not played back, and return the audio stream corresponding to the speech to transmit the synthesized speech to the 3D modeling engine.

[0046] like Figure 2As shown, the electronic device 110 utilizes the speech synthesis interface included in the operating system 210 to generate at least one speech segment 214 (e.g., it may include speech segment 214-1, speech segment 214-2, ..., speech segment 214-N) based on text 213 and predetermined playback parameters. In some embodiments, the electronic device 110 stores at least one speech segment in a cache module included in the operating system. Accordingly, the electronic device 110 sorts the at least one speech segment data in the cache module based on a sequence function.

[0047] Then, the electronic device 110 merges at least one sorted speech segment into speech. For example... Figure 2 As shown, electronic device 110 can cache at least one asynchronously returned voice segment 214 in the cache module of operating system 210. In some examples, the cache module of operating system 210 can refer to an array buffer.

[0048] If the electronic device 110 detects that at least one speech segment 214 corresponding to the text has been synthesized, it can reorder the cached at least one speech segment 214 according to the sequence function. Then, the electronic device 110 merges the data streams corresponding to all speech segments to obtain speech 215 in a first data format (e.g., bytes).

[0049] In some embodiments, the electronic device 110 sends the generated voice to the 3D modeling engine via a data callback listening function. For example... Figure 2 As shown, after generating the voice 215 in the first data format, the electronic device 110 can send the voice 215 in the first data format to the 3D modeling engine 220 through the data callback listening function set in the audio management singleton object 216.

[0050] In some embodiments, the electronic device 110 generates audio-visual interactive content for the digital human based on speech using a 3D modeling engine. In some embodiments, the electronic device 110 utilizes an audio playback component within the 3D modeling engine to play the corresponding speech. For example... Figure 2 As shown, after receiving the callback, the 3D modeling engine 220 can parse the speech by opening the operating system array buffer (OpenHarmonyJSArrayBuffer). Then, the electronic device 110 uses the audio playback component (AudioSource) 222 in the 3D modeling engine 220 to play the speech corresponding to the speech. In some examples, the audio playback component 222 and the lip-sync tool (uLipSync) 223 can be deployed in the digital human model.

[0051] In some embodiments, the electronic device 110 can convert speech in a first data format into speech in a second data format that matches the audio playback component. The electronic device 110 then uses the audio playback component in the 3D modeling engine to play the speech in the second data format.

[0052] like Figure 2 As shown, electronic device 110 can parse the audio stream byte[] data corresponding to the speech by opening the operating system array buffer (OpenHarmonyJSArrayBuffer). Since the audio playback component (AudioSource) 222 requires the audio segment data type to be float[], electronic device 110 converts the speech 215 of byte[] data type to speech 221 of float[] data type. That is, electronic device 110 iterates through byte[], processes 4 bytes at a time, and uses the BitConverter class to convert these 4 bytes into floating-point numbers to obtain the new float[]. After converting the speech 215 of the first data format to the speech 221 of the second data format, electronic device 110 can create an audio clip (AudioClip) and use the audio playback component 222 to play the speech 221 of the second data format.

[0053] Accordingly, during the playback of voice, the electronic device 110 uses a lip-sync tool to drive the digital human's lip movements to synchronize with the audio, thereby generating audio-visual interactive content for the digital human. For example... Figure 3 As shown, during the playback of the second data format audio 221 by the electronic device 110, the lip-sync tool (uLipSync) 223 works synchronously to drive the digital human's lip movements to synchronize with the audio. This generates audio-visual interactive content 150 for the digital human, such as audio-visual videos broadcast by the digital human based on speech, or audio corresponding to text read by the digital human.

[0054] The embodiments disclosed herein facilitate data interaction between the operating system and the 3D modeling engine. Speech synthesis is performed using the operating system's API, and the synthesized speech segments are sorted, merged, and sent to the 3D modeling engine. The 3D modeling engine then converts the speech into the field types required by the speech playback component for playback, and simultaneously achieves lip-sync through a lip-sync tool on the digital human model.

[0055] In summary, the embodiments of this disclosure enable the 3D modeling engine to generate audio and video interactive content for digital humans based on the voice generated by the operating system, thereby reducing user waiting time and improving user experience.

[0056] Figure 1A flowchart of a process 300 for content interaction according to some embodiments of the present disclosure is shown. Process 300 can be implemented at electronic device 110. Reference is made below. Figure 4 Describe the process 300.

[0057] In box 310, electronic device 110 sets up a data callback listening function for the operating system of the electronic device in the 3D modeling engine of the electronic device. The operating system is configured to perform speech synthesis at least, and the 3D modeling engine is configured to generate audio and video interactive content of digital humans based on speech.

[0058] In box 320, electronic device 110, in response to receiving text to be synthesized, uses the speech synthesis interface included in the operating system to generate speech corresponding to the text.

[0059] In frame 330, electronic device 110 sends the generated voice to the 3D modeling engine via a data callback listening function.

[0060] In frame 340, electronic device 110 generates audio and video interactive content for digital humans based on voice via a 3D modeling engine.

[0061] In some embodiments, setting up a data callback listening function for the operating system of an electronic device includes: creating a script file for audio in a 3D modeling engine; registering an audio management object included in the operating system to the script file, the audio management object being configured to transmit speech to the 3D modeling engine; and registering a data callback listening function for the audio management object in the 3D modeling engine after the audio management object is registered to the script file.

[0062] In some embodiments, the audio management object is a singleton object.

[0063] In some embodiments, a data callback listening function for the operating system of the electronic device is set up: an event subscription is added to the operating system, the event subscription instructs the cross-thread transmission of voice; and via the event subscription, the generated voice is transmitted from the thread corresponding to the voice synthesis in the operating system to the callback thread corresponding to the 3D modeling engine, the callback thread instructs the 3D modeling engine to call back the voice generated by the operating system from the callback thread.

[0064] In some embodiments, the text includes text generated from user messages for a user via a large language model.

[0065] In some embodiments, generating text for a user's user message via a large language model includes: receiving the user's user message; and obtaining text based on the user message using a large language model.

[0066] In some embodiments, generating speech corresponding to text using a speech synthesis interface included in the operating system includes: generating at least one speech segment based on text and predetermined playback parameters using the speech synthesis interface included in the operating system; storing the at least one speech segment in a cache module included in the operating system; sorting the at least one speech segment data in the cache module based on a sequence function; and merging the sorted at least one speech segment into speech.

[0067] In some embodiments, generating audio-visual interactive content for a digital human based on speech via a 3D modeling engine includes: playing speech corresponding to the speech using an audio playback component in the 3D modeling engine; and during the playback of the speech, using a lip-sync tool to drive the digital human's lip movements to synchronize with the speech, so as to generate audio-visual interactive content for the digital human.

[0068] In some embodiments, the voice data format is a first data format, and playing the voice corresponding to the voice using the audio playback component in the 3D modeling engine includes: converting the voice in the first data format into a voice in a second data format that matches the audio playback component; and playing the voice in the second data format using the audio playback component in the 3D modeling engine.

[0069] Figure 5 A schematic structural block diagram of a device 400 for content interaction according to certain embodiments of the present disclosure is shown. Device 400 may be implemented as or included in electronic device 110. Various modules / components in device 400 may be implemented by hardware, software, firmware, or any combination thereof.

[0070] As shown in the figure, device 400 includes a data callback monitoring function setting module 410, configured to set a data callback monitoring function for the operating system of the electronic device in the 3D modeling engine of the electronic device. The operating system is configured to perform speech synthesis, and the 3D modeling engine is configured to generate audio-visual interactive content for the digital human based on the speech. A speech generation module 420 is configured to generate speech corresponding to the received text using the speech synthesis interface included in the operating system. A speech sending module 430 is configured to send the generated speech to the 3D modeling engine via the data callback monitoring function. An audio-visual interactive content generation module 440 is configured to generate audio-visual interactive content for the digital human based on the speech via the 3D modeling engine.

[0071] In some embodiments, the data callback listening function setting module 410 is further configured to create a script file for audio in the 3D modeling engine; register an audio management object included in the operating system to the script file, the audio management object being configured to transmit voice to the 3D modeling engine; and after the audio management object is registered to the script file, register a data callback listening function for the audio management object in the 3D modeling engine.

[0072] In some embodiments, the audio management object is a singleton object.

[0073] In some embodiments, the data callback listening function setting module 410 is further configured to add an event subscription in the operating system, the event subscription indicating cross-thread transmission of voice; and via the event subscription, transmit the generated voice from the thread corresponding to the voice synthesis in the operating system to the callback thread corresponding to the 3D modeling engine, the callback thread instructing the 3D modeling engine to call back the voice generated by the operating system from the callback thread.

[0074] In some embodiments, the text includes text generated from user messages for a user via a large language model.

[0075] In some embodiments, the speech generation module 420 is further configured to receive user messages from a user; and to obtain text based on the user messages using a large language model.

[0076] In some embodiments, the speech generation module 420 is further configured to generate at least one speech segment based on text and predetermined playback parameters using a speech synthesis interface included in the operating system; store the at least one speech segment in a cache module included in the operating system; sort the at least one speech segment data in the cache module based on a sequence function; and merge the sorted at least one speech segment into speech.

[0077] In some embodiments, the audio-visual interactive content generation module 440 is further configured to play the speech corresponding to the speech using the audio playback component in the 3D modeling engine; and during the playback of the speech, to drive the lip movements of the digital human to synchronize with the speech using a lip-sync tool, so as to generate the audio-visual interactive content of the digital human.

[0078] In some embodiments, the voice data format is a first data format, and the audio-visual interactive content generation module 440 is further configured to convert the voice in the first data format into voice in a second data format that matches the audio playback component; and to play the voice in the second data format using the audio playback component in the 3D modeling engine.

[0079] The units and / or modules included in device 400 can be implemented in various ways, including software, hardware, firmware, or any combination thereof. In some embodiments, one or more units and / or modules can be implemented using software and / or firmware, such as machine-executable instructions stored on a storage medium. In addition to or as an alternative to machine-executable instructions, some or all of the units and / or modules in device 300 can be implemented at least partially by one or more hardware logic components. By way of example and not limitation, exemplary types of hardware logic components that can be used include field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), complex programmable logic devices (CPLDs), and so on.

[0080] Figure 5 A block diagram of an electronic device 500 in which one or more embodiments of the present disclosure may be implemented is shown. It should be understood that... Figure 5 The electronic device 500 shown is merely exemplary and should not be construed as limiting the functionality and scope of the embodiments described herein. Figure 1 The electronic device 500 shown can be used to achieve Figure 4 Electronic devices 110 or Figure 5 Device 400.

[0081] like Figure 5 As shown, electronic device 500 is in the form of a general-purpose computing device. Components of electronic device 500 may include, but are not limited to, one or more processors or processing units 510, memory 520, storage devices 530, one or more communication units 540, one or more input devices 550, and one or more output devices 550. Processing unit 510 may be a physical or virtual processor and is capable of performing various processes according to programs stored in memory 520. In a multiprocessor system, multiple processing units execute computer-executable instructions in parallel to improve the parallel processing capability of electronic device 500.

[0082] Electronic device 500 typically includes multiple computer storage media. Such media can be any accessible media that is accessible to electronic device 500, including but not limited to volatile and non-volatile media, removable and non-removable media. Memory 520 can be volatile memory (e.g., registers, cache, random access memory (RAM)), non-volatile memory (e.g., read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory), or some combination thereof. Storage device 530 can be removable or non-removable media and can include machine-readable media, such as flash drives, disks, or any other media that can be used to store information and / or data and can be accessed within electronic device 500.

[0083] Electronic device 500 may further include additional removable / non-removable, volatile / non-volatile storage media. Although not explicitly stated... ​ As shown, disk drives for reading from or writing to removable, non-volatile disks (e.g., "floppy disks") and optical disk drives for reading from or writing to removable, non-volatile optical disks can be provided. In these cases, each drive can be connected to a bus (not shown) via one or more data media interfaces. Memory 520 may include computer program product 525 having one or more program modules configured to perform various methods or actions of various embodiments of this disclosure.

[0084] The communication unit 540 enables communication with other computing devices via a communication medium. Additionally, the functionality of the components of the electronic device 500 can be implemented as a single computing cluster or multiple computing machines capable of communicating via communication connections. Therefore, the electronic device 500 can operate in a networked environment using logical connections to one or more other servers, network personal computers (PCs), or another network node.

[0085] Input device 550 can be one or more input devices, such as a mouse, keyboard, trackball, etc. Output device 550 can be one or more output devices, such as a monitor, speaker, printer, etc. Electronic device 500 can also communicate with one or more external devices (not shown) via communication unit 540 as needed. These external devices include storage devices, display devices, etc., and can communicate with one or more devices that enable user interaction with electronic device 500, or with any device that enables electronic device 500 to communicate with one or more other computing devices (e.g., network card, modem, etc.). Such communication can be performed via input / output (I / O) interface (not shown).

[0086] According to an exemplary implementation of this disclosure, a computer-readable storage medium is provided that stores computer-executable instructions thereon, wherein the computer-executable instructions are executed by a processor to implement the methods described above. According to an exemplary implementation of this disclosure, a computer program product is also provided, which is tangibly stored on a non-transitory computer-readable medium and includes computer-executable instructions, which are executed by a processor to implement the methods described above.

[0087] Various aspects of this disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatuses, devices, and computer program products implemented according to this disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0088] These computer-readable program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processing unit of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner. Thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0089] Computer-readable program instructions can be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions that execute on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0090] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction, which contains one or more executable instructions for implementing the specified logical function. In some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0091] Various implementations of this disclosure have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed implementations. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described implementations. The terminology used herein is chosen to best explain the principles, practical applications, or improvements to technology in the market, or to enable others skilled in the art to understand the various implementations disclosed herein.

Claims

1. A method of content interaction, comprising: setting, in a 3D modeling engine of an electronic device, a data callback listening function for an operating system of the electronic device, the operating system being configured to at least perform speech synthesis, the 3D modeling engine being configured to generate audio-visual interactive content of a digital person based on the speech; generating, in response to receiving a text to be synthesized, speech corresponding to the text using a speech synthesis interface included in the operating system; sending, via the data callback listening function, the generated speech to the 3D modeling engine; and generating, via the 3D modeling engine, the audio-visual interactive content of the digital person based on the speech.

2. The method of claim 1, wherein setting the data callback listening function for the operating system of the electronic device comprises: creating, in the 3D modeling engine, a script file for audio; registering an audio management object included in the operating system to the script file, the audio management object being configured to transmit speech to the 3D modeling engine; and registering, in the 3D modeling engine, a data callback listening function for the audio management object after the audio management object is registered to the script file.

3. The method of claim 2, wherein the audio management object is a singleton object.

4. The method of claim 2, wherein setting the data callback listening function for the operating system of the electronic device comprises: adding an event subscription in the operating system, the event subscription indicating cross-thread transmission of speech; and transmitting, via the event subscription, the generated speech from a thread corresponding to the speech synthesis in the operating system to a callback thread corresponding to the 3D modeling engine, the callback thread indicating that the 3D modeling engine calls back speech generated by the operating system from the callback thread.

5. The method of claim 1, wherein the text comprises text generated via a large language model for a user message of a user.

6. The method of claim 5, wherein generating text via a large language model for a user message of a user comprises: receiving a user message of the user; and obtaining the text based on the user message using the large language model.

7. The method of claim 1, wherein generating speech corresponding to the text using a speech synthesis interface included in the operating system comprises: generating, using the speech synthesis interface included in the operating system, at least one speech segment based on the text and predetermined playback parameters; storing the at least one speech segment to a cache module included in the operating system; ordering at least one speech segment data in the cache module based on a sequence function; and merging the ordered at least one speech segment into the speech.

8. The method of claim 1, wherein generating, via the 3D modeling engine, the audio-visual interactive content of the digital person based on the speech comprises: playing, using an audio playback component in the 3D modeling engine, speech corresponding to the speech; and ​ ​ ​ ​ ​ ​ In a process of playing the voice, a lip synchronization tool is utilized to drive the digital person's mouth shape to be synchronized with the voice, so as to generate the audio-video interactive content of the digital person. 9.The method of claim 8, wherein a data format of the voice is a first data format, and playing the voice corresponding to the voice by using an audio playing component in the 3D modeling engine comprises: converting the voice in the first data format into a voice in a second data format which matches the audio playing component; and playing the voice in the second data format by using the audio playing component in the 3D modeling engine. 10.An apparatus for content interaction, comprising: a data callback listening function setting module configured to set, in a 3D modeling engine of an electronic device, a data callback listening function for an operating system of the electronic device, the operating system being configured to at least perform voice synthesis, the 3D modeling engine being configured to generate audio-video interactive content of a digital person based on a voice; a voice generation module configured to, in response to receiving a text to be synthesized, generate a voice corresponding to the text by using a voice synthesis interface included in the operating system; a voice sending module configured to send the generated voice to the 3D modeling engine via the data callback listening function; and an audio-video interactive content generation module configured to generate the audio-video interactive content of the digital person based on the voice via the 3D modeling engine. 11.An electronic device, comprising: at least one processing unit; and at least one memory coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit, the instructions, when executed by the at least one processing unit, causing the electronic device to perform the method according to any one of claims 1 to 9. 12.A computer-readable storage medium having stored thereon a computer program, the computer program being executable by a processor to implement the method according to any one of claims 1 to 9. 13.A computer program product tangibly stored in a computer storage medium and comprising computer executable instructions that, when executed by a device, cause the device to perform the method according to any one of claims 1 to 9. ​ ​

Images