Digital human application system, application method, computer device and storage medium

Abstract

The present application relates to a digital human application system, application method, computer equipment and storage medium. By obtaining a multimodal interaction request, and transparently transmitting the multimodal interaction request to the corresponding digital human server through the WebSocket protocol, the rendered playback content returned by the digital human server is received, and then the rendered playback content is transparently transmitted to the corresponding H5 page. Since data collection, processing and transmission in the multimodal interaction process are all implemented natively, the processing of audio and video signals in the multimodality can be better completed, providing necessary support for multimodal audio and video interaction; and since the fusion and processing of multimodal signals are completed through back-end services, the performance of multimodal audio and video interaction using the Hybrid architecture is guaranteed, breaking away from the limitations of specific hardware resources and platforms, and being able to provide users with a refreshing multimodal audio and video interaction experience.

Description

Technical Field

[0001] The present application relates to the field of software technology, and in particular to a digital human application system, application method, computer equipment, and storage medium. Background Art

[0002] An SDK (Software Development Kit) is a specific software package, framework, or other set of tools designed to assist in the development of a specific type of software. SDKs typically include documentation, examples, and tools. SDKs can be categorized as system SDKs and application SDKs. A system SDK is a collection of development tools used to implement a specific software package, software framework, hardware platform, or operating system. Application SDKs, on the other hand, are developed based on the system SDK and are independent of specific services and have specific functionality.

[0003] Traditionally, to integrate a digital human application system, one can load the digital human application through HTML5 to provide external digital human services. Alternatively, a digital human application SDK can be provided to facilitate integration in a wider range of scenarios and terminals. There are several common approaches to building a digital human application SDK: one is a purely native implementation, which is customized for different scenarios, hardware devices, and systems. Another approach involves cross-device development using current cross-device technology frameworks such as Flutter (an open-source toolkit for building user interfaces) and RN (React Native, a mobile development component). The third approach is to build on a hybrid architecture.

[0004] However, using H5 to load digital human applications places specific browser requirements, and many browsers are incompatible. Secondly, using H5 significantly impacts the interactive experience of digital humans, and offers limited customization and modifiability unless the underlying browser is modified. Finally, using H5 to load digital human applications places significant restrictions on digital human application scenarios and terminals, hindering the rapid advancement of digital human application systems across business, scenarios, and terminals.

[0005] From the perspective of SDK construction methods, native SDKs have the following major problems. First, the development cost is high, especially when multiple mobile devices are required for testing; second, the development and maintenance costs are high under different development languages; third, the online review process and release time are uncertain; fourth, when obtaining a new version, the application needs to be downloaded and updated again; fifth, in terms of front-end page display, since each terminal has its own native view (graphic display) component, it is impossible to achieve a consistent experience for each terminal. SDKs built based on some cross-end technical frameworks have limited performance for digital human SDKs that focus on interaction. Moreover, some cross-end framework communities are not yet complete, and you need to reinvent the wheel. Although cross-end can be achieved, the actual development, testing, and maintenance costs are higher.

[0006] With the rapid development of financial technology, the application scenarios of digital humans are becoming increasingly diverse and complex. For various terminal devices, such as mobile phones, VTMs (Video Teller Machines), cockpits, large screens, PADs (tablets), and iPads (iOS tablets), the system must accommodate both Android and iOS. Therefore, a digital human application system that can better adapt to business scenarios and terminal requirements is urgently needed. Summary of the Invention

[0007] Based on this, it is necessary to provide a digital human application system, application method, device, computer equipment, computer-readable storage medium and computer program product that can better adapt to business scenarios and terminal requirements in response to the above technical problems.

[0008] A digital human application system, based on a hybrid mode architecture, comprising:

[0009] Digihuman SDK, which is used to encapsulate the underlying Digihuman application capabilities into an SDK to provide the general capabilities involved in the underlying applications;

[0010] The H5 layer is responsible for business logic control and corresponding resource display;

[0011] The business layer SDK is used for communication and data transmission between the Digital Human SDK, H5 layer and client application. The client application is an application with integration requirements.

[0012] In one embodiment, the H5 layer includes resource display pages customized based on vue.js for different business scenarios.

[0013] In one embodiment, the business layer SDK is called based on Restful services.

[0014] In one embodiment, the Digital Human SDK uses the WebSocket protocol to establish a long connection with the corresponding Digital Human server.

[0015] A digital human application method, the method comprising:

[0016] Get the multimodal interaction request initiated by the user based on the initialized H5 page;

[0017] Send the multimodal interaction request to the corresponding Digital Human server via the WebSocket protocol;

[0018] Receiving the rendered playback content returned by the digital human server, wherein the rendered playback content is obtained after the digital human server performs corresponding processing according to the multimodal interaction request;

[0019] The rendered playback content is transparently transmitted to the corresponding H5 page, so that the H5 page performs page rendering and display based on the rendered playback content.

[0020] In one embodiment, before obtaining a multimodal interaction request initiated by a user based on the initialized H5 page, the method further includes: executing an initialization process in response to an initialization instruction of the client application.

[0021] In one embodiment, the execution initialization process includes: initializing the H5 page according to the initialization instruction, and establishing a long connection with the digital human server according to the initialization instruction, so that the digital human server starts the digital human service; obtaining the corresponding initialization result, and returning the initialization result to the client application.

[0022] In one embodiment, the method further includes: when a page jump instruction for the client application is obtained, sending the page jump instruction to the client application so that the client application performs a corresponding page jump; and pausing the digital human service according to the page jump instruction.

[0023] In one embodiment, the method further includes: when a control instruction for the digital human service is obtained, controlling the state of the digital human service according to the control instruction.

[0024] In one embodiment, the control instruction includes any one of a pause instruction, a mute instruction, and an exit instruction.

[0025] In one embodiment, the method further includes: when an exception is detected, sending a corresponding exception prompt message to the client application, so that the client application performs corresponding processing according to the exception prompt message.

[0026] In one embodiment, after establishing a long connection with the digital human server according to the initialization instruction, the method further includes: establishing a heartbeat mechanism with the client application.

[0027] An application device of a digital human application system, comprising:

[0028] A multimodal interaction request acquisition module is used to acquire a multimodal interaction request, wherein the multimodal interaction request is initiated by the user based on an initialized H5 page or RTC;

[0029] A sending module, used to transparently transmit the multimodal interaction request to the corresponding digital human server through the WebSocket protocol;

[0030] A rendering and playing content receiving module, configured to receive the rendering and playing content returned by the Digital Human server, wherein the rendering and playing content is obtained after the Digital Human server performs corresponding processing according to the multimodal interaction request;

[0031] The sending module is further used to transparently transmit the rendered playback content to a corresponding H5 page, so that the H5 page performs page rendering and display based on the rendered playback content.

[0032] A computer device includes a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the above method when executing the computer program.

[0033] A computer-readable storage medium stores a computer program, which implements the steps of the method described above when executed by a processor.

[0034] A computer program product comprises a computer program, which implements the steps of the method described above when the computer program is executed by a processor.

[0035] The aforementioned digital human application system, application method, apparatus, computer device, computer-readable storage medium, and computer program product obtain a multimodal interaction request and transparently transmit the multimodal interaction request to the corresponding digital human server via the WebSocket protocol. The system then receives the rendered playback content returned by the digital human server and transparently transmits the rendered playback content to the corresponding H5 page. Because data acquisition, processing, and transmission during the multimodal interaction process are all implemented natively, the processing of multimodal audio and video signals can be better completed, providing the necessary support for multimodal audio and video interaction. Furthermore, because the fusion and processing of multimodal signals are completed through back-end services, the performance of multimodal audio and video interaction achieved using a hybrid architecture is guaranteed, eliminating the limitations of specific hardware resources and platforms and providing users with a refreshing multimodal audio and video interaction experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 This is an architecture diagram of a digital human application system in one embodiment;

[0037] Figure 2 A flowchart of an application method based on a digital human application system in one embodiment;

[0038] Figure 3 A schematic flow chart of an initialization step in one embodiment;

[0039] Figure 4 A schematic flow chart of an application method based on a digital human application system in another embodiment;

[0040] Figure 5 Schematic diagram of a flow chart of an application method based on a digital human application system in another embodiment;

[0041] Figure 6 This is a structural block diagram of an application device based on the digital human application system in one embodiment;

[0042] Figure 7 FIG. 1 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0043] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0044] In one embodiment, Figure 1 As shown, a digital human application system is provided. The system is based on the Hybrid mode architecture and specifically includes a digital human SDK, a business layer SDK, and an H5 layer, wherein:

[0045] The Digital Human SDK encapsulates the underlying Digital Human application capabilities into an SDK, providing common capabilities associated with the underlying applications, such as basic audio and video signal processing, rendering, and communication. The H5 layer is responsible for business logic control and corresponding resource presentation, such as customized WebViews (web views), business process control, native capability invocation, and semantic understanding and data parsing. The business layer SDK facilitates communication and data transmission between the Digital Human SDK, the H5 layer, and the client application (APP), which is the application with integration requirements. This fully decouples the business logic (i.e., H5) from the underlying capabilities (i.e., the Digital Human SDK), enabling parallel front-end and back-end development, which not only improves development efficiency. Furthermore, in each project, only the business logic WebView portion needs to be developed. All WebView components are distributed from the WebView entry point, while the native layer (i.e., the Digital Human SDK) can be reused. By passing the WebSocket address and WebView address from the APP (client application), switching between different businesses and environments can be achieved.

[0046] The above-mentioned digital human application system, by dividing the native SDK into a digital human SDK and a business layer SDK, and being independent of WebView (i.e., the H5 layer), is architected based on the Hybrid mode, making the overall system architecture replicable and customizable. In addition, the H5 page supports both ends, eliminating cross-end development work. Compared with the pure native SDK construction method, it has the advantages of being replicable, customizable, and easy to publish, thus being able to better adapt to different business scenarios and business terminals.

[0047] In one embodiment, the H5 layer includes resource display pages (i.e., customized WebViews) customized based on Vue.js for different business scenarios. Therefore, by implementing corresponding encapsulation based on different business scenarios, a digital human application system with corresponding functions can be implemented, thereby facilitating rapid integration of various scenarios and making the development and daily maintenance process centralized, shorter, more economical and efficient. By using HTML as the display, this embodiment can more flexibly and conveniently utilize open source resources within the large front-end ecosystem of the current Internet while ensuring a gorgeous display, and is more inclusive.

[0048] In one embodiment, the business layer SDK is based on a standard interface and uses RESTful (a design style and development method for network applications) services for calling, thereby making the call more convenient.

[0049] In one embodiment, the above-mentioned Digital Human SDK uses the WebSocket protocol to establish a long connection with the corresponding Digital Human server, so that it can better connect to the server under container deployment, and the way different channels connect to the server is completely consistent, making service development and calling faster.

[0050] In one embodiment, Figure 2 As shown, an application method based on the digital human application system is also provided, which specifically includes the following steps:

[0051] Step 210: Obtain a multimodal interaction request initiated by the user based on the initialized H5 page.

[0052] Multimodality is the integration of multiple senses, while multimodal interaction involves human-computer interaction through various means, including text, voice, vision, movement, and the environment, fully simulating human-to-human interaction. A multimodal interaction request is a user-initiated instruction or command for multimodal human-computer interaction, including but not limited to interaction requests initiated through voice, video images, keyboard text, clicks, and touch. The initialized H5 page is the H5 page initialized by the client application integrated with the Digital Human application system based on the Digital Human application system. In this embodiment, users can initiate multimodal interaction requests through the initialized H5 page and RTC, and the Digital Human application system can then receive the multimodal interaction request.

[0053] Step 220: Transparently transmit the multimodal interaction request to the corresponding Digital Human server via the WebSocket protocol.

[0054] The Digihuman server is a server that provides backend services for the Digihuman application system. For example, the Digihuman server can allocate resources such as multimedia conversation rooms and Digihuman RTC based on multimodal interaction requests. In this embodiment, when the Digihuman application system receives a multimodal interaction request from a user, it transparently transmits the request to the corresponding Digihuman server via the WebSocket protocol, allowing the server to perform appropriate processing and resource allocation.

[0055] Step 230: Receive the rendered content returned by the Digital Human server.

[0056] The rendered playback content is obtained by the Digihuman server after performing corresponding processing based on the multimodal interaction request. Specifically, after receiving the relevant request, the Digihuman server performs ASR (Automatic Speech Recognition) processing on the corresponding audio signal. In addition, it combines the video signal in the multimodal interaction request to perform specific face detection, tracking, and recognition. After multimodal signal processing, it is sent to the corresponding cognitive platform and the corresponding business logic is invoked to return the corresponding rendered playback content. The Digihuman application system can then receive the rendered playback content returned by the Digihuman server.

[0057] Step 240: Transmit the rendered content to the corresponding H5 page.

[0058] After receiving the rendered playback content returned by the Digital Human server, the Digital Human application system transparently transmits the rendered playback content to the corresponding H5 page, so that the H5 page can render and display the page based on the rendered playback content.

[0059] In the aforementioned application method based on the Digital Human application system, a multimodal interaction request initiated by the user based on the initialized H5 page is obtained, and the multimodal interaction request is transparently transmitted to the corresponding Digital Human server via the WebSocket protocol. The rendered playback content returned by the Digital Human server is received, and the rendered playback content is then transparently transmitted to the corresponding H5 page. Because data collection, processing, and transmission in the multimodal interaction process are all implemented natively, the processing of audio and video signals in the multimodal environment can be better completed, providing the necessary support for multimodal audio and video interaction. Moreover, because the fusion and processing of multimodal signals are completed through back-end services, the performance of multimodal audio and video interaction using the Hybrid architecture is guaranteed, breaking away from the limitations of specific hardware resources and platforms, and providing users with a refreshing multimodal audio and video interaction experience.

[0060] In one embodiment, before obtaining a multimodal interaction request initiated by a user based on an initialized H5 page, the method further includes: executing an initialization process in response to an initialization instruction from a client application. The client application is an application that integrates the digital human application system based on business needs. For example, a mobile banking app might need to provide external smart shopping guide services. However, due to its limitations in financial scenarios, the frequency of app packaging and release is limited, while the business requires frequent changes and commissioning. Therefore, a hybrid architecture can be adopted to construct a multimodal digital human SDK (i.e., a digital human application system), and then the digital human application system can be integrated with the native app and initialized through the initialization interface exposed by the digital human application system. In this embodiment, the client application can initiate an initialization instruction, and the digital human application system responds to the initialization instruction and executes the initialization process to initialize.

[0061] In one embodiment, Figure 3 As shown, the initialization process may specifically include the following steps:

[0062] Step 310: Initialize the H5 page according to the initialization instruction, and establish a long connection with the Digital Human server according to the initialization instruction.

[0063] Specifically, after the Digital Human application system obtains the initialization instruction of the client application, it initializes the H5 page according to the initialization instruction on the one hand, and establishes a long connection with the Digital Human server on the other hand according to the initialization instruction. Specifically, it can establish a long connection with the Digital Human server based on the WebSocket protocol, so that the Digital Human server starts the Digital Human service, that is, allocates corresponding resources to the client application, such as the corresponding multimedia conversation room, Digital Human streaming media resources, etc.

[0064] Step 320: Obtain the corresponding initialization result and return the initialization result to the client application.

[0065] The initialization result can be a status indicating whether the initialization was successful or not. Specifically, the Digihuman application system initializes the H5 page according to the initialization instructions and can obtain the corresponding initialization status, for example, whether the H5 page initialization was successful or unsuccessful. The Digihuman application system establishes a persistent connection with the Digihuman server according to the initialization instructions and can also obtain the corresponding connection status, for example, whether the persistent connection was successfully established or unsuccessful. It will be understood that in this embodiment, the initialization result can include the initialization status indicating whether the H5 page was successfully initialized and the connection status indicating whether the persistent connection was successfully established with the Digihuman server. If the initialization result is successful, it indicates that the client application has successfully integrated the Digihuman application system, allowing it to perform corresponding business processing. If the initialization result is unsuccessful, it indicates that the client application failed to integrate the Digihuman application system. Therefore, after the Digihuman application system obtains the corresponding initialization result, it can return the initialization result to the client application so that the user can know whether the corresponding business processing can be performed.

[0066] In one embodiment, Figure 4 As shown, the above method may further include:

[0067] Step 410: When a page jump instruction for the client application is obtained, the page jump instruction is sent to the client application.

[0068] The page jump instruction to the client application refers to an instruction to jump to the native page of the client application. In this embodiment, when the digital human application system obtains a page jump instruction to the client application initiated by the user based on the initialized H5 page, it sends the page jump instruction to the client application, thereby causing the client application to execute the corresponding page jump.

[0069] Step 420: suspend the Digital Human service according to the page jump instruction.

[0070] Furthermore, since the aforementioned jump instruction is an instruction to jump to the native page of the client application, it means that only the original functions of the client application are currently used, and the functions of the integrated digital human application system are not required. Therefore, the digital human service can be controlled to be suspended based on the page jump instruction, allowing the client application to quickly respond to user instructions and provide users with efficient services.

[0071] In one embodiment, the method may further include: upon receiving a control instruction for the digital human service, controlling the state of the digital human service according to the control instruction. The control instruction may include any one of a pause instruction, a mute instruction, and an exit instruction. Specifically, when the control instruction received by the digital human application system is a pause instruction, the digital human service is controlled to be paused. When the control instruction received by the digital human application system is a mute instruction, the digital human service is controlled to be mute. When the control instruction received by the digital human application system is an exit instruction, the digital human service is controlled to exit. This enables flexible control of the digital human service.

[0072] In one embodiment, the method further includes: when an anomaly is detected, sending a corresponding anomaly prompt message to the client application, so that the client application can perform corresponding processing based on the anomaly prompt message. Specifically, because the Digital Human Application System establishes a persistent connection with the Digital Human Server, the Digital Human Application System can detect anomalies on the Digital Human Server as well as its own anomalies. For example, when the persistent connection is disconnected, the Digital Human Application System can detect the connection anomaly and then send a corresponding anomaly prompt message to the client application, so that the client application can perform corresponding processing based on the anomaly prompt message.

[0073] In one embodiment, after establishing a persistent connection with the digital human server according to the initialization instruction, the method further includes: establishing a heartbeat mechanism with the client application, thereby enabling multimodal interaction.

[0074] In one embodiment, Figure 5 As shown, the above method is further described below, which may specifically include the following steps:

[0075] Step 501: The application APP (i.e., client application program) initiates an initialization instruction to the SDK (i.e., the digital human application system).

[0076] Step 502: SDK initializes the H5 page according to the initialization instruction.

[0077] Step 503: SDK establishes a long connection with the Digital Human server according to the initialization instruction.

[0078] Step 504: The Digital Human server starts the Digital Human service.

[0079] Step 505: SDK completes initialization and obtains the corresponding initialization result.

[0080] Step 506: The SDK returns the initialization result to the client application.

[0081] In step 507, the SDK obtains a multimodal interaction request initiated by the user based on the initialized H5 page.

[0082] In step 508, the SDK transparently transmits the multimodal interaction request to the corresponding digital human server.

[0083] Step 509: SDK receives the rendered content returned by the Digital Human server.

[0084] In step 510, the SDK transparently transmits the rendered playback content to the corresponding H5 page.

[0085] Step 511: The H5 page is rendered and displayed.

[0086] Step 512: The SDK obtains a page jump instruction for the application APP.

[0087] Step 513: SDK sends a page jump instruction to the application APP.

[0088] Step 514: SDK suspends the Digital Human service according to the page jump instruction.

[0089] Step 515: SDK receives control instructions for the Digital Human service.

[0090] Step 516: SDK controls the status of the Digital Human service according to the control instruction.

[0091] Step 517: SDK detects an exception.

[0092] Step 518: The SDK sends a corresponding abnormal prompt message to the application APP.

[0093] Specifically, this embodiment uses a virtual digital human SDK for integration with a mobile banking app and the provision of an intelligent shopping guide service as an example. Because mobile banking apps are limited by financial scenarios, the frequency of app packaging and release is limited, and the business requires frequent changes and commissioning, a hybrid architecture is considered for building a multimodal digital human application system. The native SDK is split into a digital human SDK, providing underlying general capabilities such as basic audio and video signal processing, rendering, and communication capabilities. The business layer SDK primarily serves as a communication bridge, completing data transmission between the digital human SDK, the app, and the business layer. Business logic and front-end resources are primarily implemented through the H5 layer.

[0094] First, the native app integrates the Digital Human application system, initializes it through the initialization interface exposed by the Digital Human application system, and transmits it to the Digital Human server through the system's WS communication capabilities. The server then allocates the corresponding multimedia conversation room and Digital Human SDK. Furthermore, because the Digital Human application system supports a variety of interaction methods, including text keyboard, touch, audio and video communication, and voice input, the overall implementation integrates the RTC (real-time audio and video communication) package within the system, giving the system basic audio and video communication capabilities, including the acquisition, preprocessing, transmission, and rendering of audio and video signals. Secondly, the traditional touch and text keyboard input functions are integrated into the H5 side, allowing the corresponding user text input and touch operations to be transparently transmitted to the SDK through the specified communication protocol, and then sent to the Digital Human server through the SDK communication layer. After receiving the relevant request, the digital human server performs ASR (speech-to-text) processing on the corresponding audio signal. In addition, it combines the video signal sent by the customer to perform specific face detection, tracking, and recognition. At the same time, after multimodal signal processing, it is sent to the corresponding cognitive platform, calls the corresponding business logic, and returns the corresponding rendering and playback content. Through TTS (Text To Speech) and A2A (area to area), the corresponding BS (background template) sequence is generated, and the rendering of the 3D virtual digital human is completed through UE4. It is output to the end through webrtc (real-time audio and video), and the corresponding real-time audio and video playback is completed through the RTC package on the end, thus providing users with a refreshing multimodal audio and video interactive experience.

[0095] In the above-mentioned embodiment, the digital human service is provided externally through an SDK, facilitating the development and integration of business scenarios. Furthermore, the digital human application capabilities are customizable and extensible, thus better breaking free from browser limitations. Furthermore, because the digital human application system is implemented based on a hybrid architecture, it is not only replicable, but its native capability layer can be reused across clients, helping new clients quickly access the application and reducing customized R&D costs. It is also customizable, supporting clients to customize personalized business scenarios and facilitating integration. It also facilitates rapid location. Because its native capabilities are decoupled from business logic, when problems arise, there is no need to track which layer the problem occurred in order to quickly locate it. Furthermore, because the H5 page supports both ends, it can be quickly launched without the need for app store review.

[0096] It should be understood that although Figure 1-Figure 5 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. In addition, Figure 1-Figure 5At least part of the steps may include multiple steps or multiple stages. These steps or stages are not necessarily performed at the same time, but can be performed at different times. The order of execution of these steps or stages is not necessarily one by one, but can be performed in turn or alternately with other steps or at least part of the steps or stages in other steps.

[0097] In one embodiment, Figure 6 As shown, an application device of a digital human application system is provided, comprising: a multimodal interaction request acquisition module 602, a sending module 604 and a rendering and playback content receiving module 606, wherein:

[0098] The multimodal interaction request acquisition module 602 is used to acquire the multimodal interaction request initiated by the user based on the initialized H5 page;

[0099] The sending module 604 is used to send the multimodal interaction request to the corresponding digital human server through the WebSocket protocol;

[0100] The rendering and playing content receiving module 606 is used to receive the rendering and playing content returned by the digital human server. The rendering and playing content is obtained by the digital human server after performing corresponding processing according to the multimodal interaction request;

[0101] The sending module 604 is further configured to transparently transmit the rendered playback content to a corresponding H5 page, so that the H5 page performs page rendering and display based on the rendered playback content.

[0102] In one embodiment, the apparatus further comprises an initialization module configured to execute an initialization process in response to an initialization instruction of a client application.

[0103] In one embodiment, the initialization module is specifically used to: initialize the H5 page according to the initialization instruction, and establish a long connection with the digital human server according to the initialization instruction, so that the digital human server can start the digital human service; obtain the corresponding initialization result, and return the initialization result to the client application.

[0104] In one embodiment, the sending module is further used to: when a page jump instruction for the client application is obtained, send the page jump instruction to the client application so that the client application executes the corresponding page jump; and suspend the digital human service according to the page jump instruction.

[0105] In one embodiment, the above-mentioned device further includes a control module, which is used to control the state of the digital human service according to the control instruction when a control instruction for the digital human service is obtained.

[0106] In one embodiment, the control instruction includes any one of a pause instruction, a mute instruction, and an exit instruction.

[0107] In one embodiment, the sending module is further configured to: when an exception is detected, send a corresponding exception prompt message to the client application, so that the client application performs corresponding processing according to the exception prompt message.

[0108] In one embodiment, the above device further comprises a heartbeat establishment module, and after the user establishes a long connection with the digital human server according to the initialization instruction, a heartbeat mechanism is established with the client application.

[0109] The specific definition of the application device of the Digital Human Application System can be found in the definition of the application method of the Digital Human Application System above and will not be repeated here. Each module in the application device of the Digital Human Application System described above can be implemented in whole or in part through software, hardware, or a combination thereof. Each of these modules can be embedded in or independent of the processor of the computer device in hardware form, or stored in the memory of the computer device in software form, so that the processor can call and execute the corresponding operations of each of these modules.

[0110] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 7 As shown. The computer device includes a processor, a memory, and a network interface connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for the operation of the operating system and computer program in the non-volatile storage medium. The database of the computer device is used to store digital human application system data. The network interface of the computer device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, it implements an application method of a digital human application system.

[0111] Those skilled in the art will understand that Figure 7 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0112] In one embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the following steps are implemented:

[0113] Get the multimodal interaction request initiated by the user based on the initialized H5 page;

[0114] Send the multimodal interaction request to the corresponding Digital Human server via the WebSocket protocol;

[0115] Receiving the rendered playback content returned by the digital human server, wherein the rendered playback content is obtained after the digital human server performs corresponding processing according to the multimodal interaction request;

[0116] The rendered playback content is transparently transmitted to the corresponding H5 page, so that the H5 page performs page rendering and display based on the rendered playback content.

[0117] In one embodiment, when executing the computer program, the processor further implements the following steps: executing an initialization process in response to an initialization instruction of the client application.

[0118] In one embodiment, when the processor executes the computer program, it also implements the following steps: initializing the H5 page according to the initialization instruction, and establishing a long connection with the digital human server according to the initialization instruction, so that the digital human server starts the digital human service; obtaining the corresponding initialization result, and returning the initialization result to the client application.

[0119] In one embodiment, when executing the computer program, the processor further implements the following steps: when obtaining a page jump instruction for the client application, sending the page jump instruction to the client application so that the client application executes the corresponding page jump; pausing the digital human service according to the page jump instruction.

[0120] In one embodiment, when executing the computer program, the processor further implements the following steps: when a control instruction for the digital human service is obtained, controlling the state of the digital human service according to the control instruction.

[0121] In one embodiment, when executing the computer program, the processor further implements the following steps: when an exception is detected, sending a corresponding exception prompt message to the client application, so that the client application performs corresponding processing according to the exception prompt message.

[0122] In one embodiment, when the processor executes the computer program, the processor further implements the following steps: establishing a heartbeat mechanism with the client application.

[0123] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the following steps are implemented:

[0124] Get the multimodal interaction request initiated by the user based on the initialized H5 page;

[0125] Send the multimodal interaction request to the corresponding Digital Human server via the WebSocket protocol;

[0126] Receiving the rendered playback content returned by the digital human server, wherein the rendered playback content is obtained after the digital human server performs corresponding processing according to the multimodal interaction request;

[0127] The rendered playback content is transparently transmitted to the corresponding H5 page, so that the H5 page performs page rendering and display based on the rendered playback content.

[0128] In one embodiment, when the computer program is executed by the processor, the following steps are further implemented: in response to the initialization instruction of the client application, executing the initialization process.

[0129] In one embodiment, when the computer program is executed by the processor, the following steps are also implemented: initializing the H5 page according to the initialization instruction, and establishing a long connection with the digital human server according to the initialization instruction, so that the digital human server starts the digital human service; obtaining the corresponding initialization result, and returning the initialization result to the client application.

[0130] In one embodiment, when the computer program is executed by the processor, the following steps are also implemented: when a page jump instruction for the client application is obtained, the page jump instruction is sent to the client application so that the client application performs a corresponding page jump; and the digital human service is suspended according to the page jump instruction.

[0131] In one embodiment, when the computer program is executed by the processor, the following steps are further implemented: when a control instruction for the digital human is obtained, the state of the digital human is controlled according to the control instruction.

[0132] In one embodiment, when the computer program is executed by the processor, the following steps are further implemented: when an exception is detected, a corresponding exception prompt message is sent to the client application, so that the client application performs corresponding processing according to the exception prompt message.

[0133] In one embodiment, when the computer program is executed by a processor, the following steps are further implemented: establishing a heartbeat mechanism with the client application.

[0134] In one embodiment, a computer program product is provided, comprising a computer program, which, when executed by a processor, implements the following steps:

[0135] Get the multimodal interaction request initiated by the user based on the initialized H5 page;

[0136] Send the multimodal interaction request to the corresponding Digital Human server via the WebSocket protocol;

[0137] Receiving the rendered playback content returned by the digital human server, wherein the rendered playback content is obtained after the digital human server performs corresponding processing according to the multimodal interaction request;

[0138] The rendered playback content is transparently transmitted to the corresponding H5 page, so that the H5 page performs page rendering and display based on the rendered playback content.

[0139] In one embodiment, when the computer program is executed by the processor, the following steps are further implemented: in response to the initialization instruction of the client application, executing the initialization process.

[0140] In one embodiment, when the computer program is executed by the processor, the following steps are also implemented: initializing the H5 page according to the initialization instruction, and establishing a long connection with the digital human server according to the initialization instruction, so that the digital human server turns on the digital human; obtaining the corresponding initialization result, and returning the initialization result to the client application.

[0141] In one embodiment, when the computer program is executed by the processor, the following steps are also implemented: when a page jump instruction for the client application is obtained, the page jump instruction is sent to the client application so that the client application executes the corresponding page jump; and the digital human is paused according to the page jump instruction.

[0142] In one embodiment, when the computer program is executed by the processor, the following steps are further implemented: when a control instruction for the digital human is obtained, the state of the digital human is controlled according to the control instruction.

[0143] In one embodiment, when the computer program is executed by the processor, the following steps are further implemented: when an exception is detected, a corresponding exception prompt message is sent to the client application, so that the client application performs corresponding processing according to the exception prompt message.

[0144] In one embodiment, when the computer program is executed by a processor, the following steps are further implemented: establishing a heartbeat mechanism with the client application.

[0145] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties.

[0146] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application may include at least one of non-volatile and volatile memory. Non-volatile memory may include read-only memory (ROM), magnetic tape, floppy disk, flash memory or optical memory, etc. Volatile memory may include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).

[0147] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0148] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that a person skilled in the art could make various modifications and improvements without departing from the spirit of the present application, all of which fall within the scope of protection of the present application. Therefore, the scope of protection of the present patent application shall be determined by the appended claims.

Claims

1. A digital human application system, characterized in that: The system is based on a Hybrid mode architecture and includes: The Digital Human SDK is used to encapsulate the underlying Digital Human application capabilities into an SDK to provide the general capabilities involved in the underlying applications. The Digital Human SDK uses the WebSocket protocol to establish a persistent connection with the corresponding Digital Human server. The Digital Human SDK is reusable. The Digital Human application system corresponds to an initialization interface for initialization. The Digital Human server allocates the corresponding multimedia session room and Digital Human SDK according to the initialization instructions of the client application. The H5 layer is responsible for business logic control and corresponding resource display. The H5 layer includes resource display pages customized based on Vue.js for different business scenarios; The business layer SDK is used for communication and data transmission between the Digital Human SDK, H5 layer and client application. The client application is an application with integration requirements, and the business layer SDK is called based on Restful services.

2. An application method based on the digital human application system according to claim 1, characterized in that: The method comprises: Get the multimodal interaction request initiated by the user based on the initialized H5 page; Send the multimodal interaction request to the corresponding Digital Human server via the WebSocket protocol; Receiving the rendered playback content returned by the digital human server, wherein the rendered playback content is obtained after the digital human server performs corresponding processing according to the multimodal interaction request; The rendered playback content is transparently transmitted to the corresponding H5 page, so that the H5 page performs page rendering and display based on the rendered playback content.

3. The method according to claim 2, characterized in that Before obtaining the multimodal interaction request initiated by the user based on the initialized H5 page, the method further includes: In response to the initialization instruction of the client application, the initialization process is executed.

4. The method according to claim 2, characterized in that The execution initialization process includes: Initialize the H5 page according to the initialization instruction, and establish a long connection with the Digital Human server according to the initialization instruction, so that the Digital Human server starts the Digital Human service; Obtain a corresponding initialization result, and return the initialization result to the client application.

5. The method according to claim 4, characterized in that The method further comprises: When a page jump instruction for the client application is obtained, the page jump instruction is sent to the client application so that the client application executes the corresponding page jump; The digital human service is paused according to the page jump instruction.

6. The method according to claim 4, characterized in that The method further comprises: When a control instruction for the digital human service is obtained, the state of the digital human service is controlled according to the control instruction.

7. The method according to claim 6, characterized in that The control instruction includes any one of a pause instruction, a mute instruction and an exit instruction.

8. The method according to claim 4, characterized in that The method further comprises: When an exception is detected, a corresponding exception prompt message is sent to the client application, so that the client application performs corresponding processing according to the exception prompt message.

9. The method according to claim 4, characterized in that After establishing a long connection with the digital human server according to the initialization instruction, the method further includes: Establish a heartbeat mechanism with the client application.

10. An application device of a digital human application system, characterized in that: The device is applied to the digital human application system as claimed in claim 1, comprising: A multimodal interaction request acquisition module is used to acquire a multimodal interaction request, wherein the multimodal interaction request is initiated by the user based on an initialized H5 page or RTC; A sending module, used to transparently transmit the multimodal interaction request to the corresponding digital human server through the WebSocket protocol; A rendering and playing content receiving module, configured to receive the rendering and playing content returned by the Digital Human server, wherein the rendering and playing content is obtained after the Digital Human server performs corresponding processing according to the multimodal interaction request; The sending module is further used to transparently transmit the rendered playback content to a corresponding H5 page, so that the H5 page performs page rendering and display based on the rendered playback content.

11. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 2 to 9 are implemented.

12. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 2 to 9 are implemented.

13. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 2 to 9 are implemented.

Images