Interactive control methods, devices, equipment, and storage media

Abstract

The present disclosure provides an interactive control method, device, equipment and storage medium, which relate to the field of artificial intelligence technology, specifically computer vision, virtual reality, deep learning and other technical fields. The specific implementation scheme is: by monitoring the schedulable resources of the server, when the schedulable resources of the server are less than the first threshold, at least one target client is selected from the multiple clients served by the server, the first interaction mode corresponding to the target client is switched to the second interaction mode, and the multimedia data generated under the second interaction mode is sent to the target client. When the schedulable resources of the server are greater than the second threshold, the second interaction mode is switched back to the first interaction mode, thereby improving the efficiency of resource utilization.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, specifically computer vision, virtual reality, deep learning and other technical fields, and can be applied to scenes such as artificial intelligence-based content generation and metaverse, and in particular to interactive control methods, devices, equipment and storage media. Background Art

[0002] Digital human technology is an interactive technology that uses artificial intelligence and is currently widely used. For example, a personalized digital doctor can interact with users through video, voice or text to provide medical consultation, diagnosis, treatment and other services. It is used in products or projects such as online medical platforms, health management applications, and smart home devices.

[0003] The user establishes a connection with the server through a client device, and then interacts with the model running on the server in the form of language or text through the connection, and displays the digital human generated by the model based on the interactive content provided by the client on the client. Summary of the invention

[0004] The present disclosure provides an interactive control method, apparatus, device and storage medium.

[0005] According to one aspect of the present disclosure, there is provided an interactive control method, comprising:

[0006] Monitor the server's schedulable resources;

[0007] When the schedulable resources of the server are less than a first threshold, selecting at least one target client from a plurality of clients served by the server;

[0008] Switching the first interaction mode corresponding to the target client to a second interaction mode, and sending multimedia data generated in the second interaction mode to the target client, wherein the amount of resources required to generate the multimedia data in the first interaction mode is greater than the amount of resources required in the second interaction mode;

[0009] When the schedulable resources of the server are greater than a second threshold, the second interaction mode is switched back to the first interaction mode, wherein the second threshold is greater than or equal to the first threshold.

[0010] According to another aspect of the present disclosure, there is provided an interactive control device, comprising:

[0011] A monitoring module, used to monitor the schedulable resources of the server;

[0012] A selection module, configured to select at least one target client from a plurality of clients served by the server when the schedulable resources of the server are less than a first threshold;

[0013] A first switching module is used to switch the first interaction mode corresponding to the target client to a second interaction mode, and send the multimedia data generated in the second interaction mode to the target client, wherein the amount of resources required to generate the multimedia data in the first interaction mode is greater than the amount of resources required in the second interaction mode;

[0014] The second switching module is used to switch the second interaction mode back to the first interaction mode when the schedulable resources of the server are greater than a second threshold, wherein the second threshold is greater than or equal to the first threshold.

[0015] According to another aspect of the present disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method described in the embodiment of the first aspect of the present disclosure.

[0016] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable a computer to execute the method described in the embodiment of the first aspect of the present disclosure.

[0017] According to another aspect of the present disclosure, a computer program product is provided, including a computer program, wherein when the computer program is executed by a processor, the method described in the embodiment of the first aspect of the present disclosure is implemented.

[0018] The interactive control method, device, equipment and storage medium provided by the present disclosure monitor the schedulable resources of the server, and when the schedulable resources of the server are less than a first threshold, select at least one target client from multiple clients served by the server, switch the first interactive mode corresponding to the target client to a second interactive mode, and send the multimedia data generated in the second interactive mode to the target client. When the schedulable resources of the server are greater than the second threshold, switch the second interactive mode back to the first interactive mode, thereby improving resource utilization efficiency.

[0019] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0021] Figure 1 A flowchart of an interactive control method provided by an embodiment of the present disclosure;

[0022] Figure 2 A schematic diagram of the architecture provided by the embodiment of the present disclosure;

[0023] Figure 3 A schematic diagram of the interaction process;

[0024] Figure 4 A flowchart of an interactive control method provided by an embodiment of the present disclosure;

[0025] Figure 5 A flow chart showing the interaction between the target client and the server;

[0026] Figure 6 A schematic diagram of the structure of an interactive control device 600 provided in an embodiment of the present disclosure;

[0027] Figure 7 A schematic block diagram of an example electronic device 700 that may be used to implement embodiments of the present disclosure is shown. DETAILED DESCRIPTION

[0028] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0029] In the related art, when digital human technology is used to interact with a client, a connection between each client and a server is usually established. After the connection is established, a GPU resource needs to be occupied in the server so as to run the model required by the digital human through the GPU.

[0030] However, this method will not be able to serve more clients if the server resources are insufficient, and the service quality of connected clients may also be affected due to insufficient transmission resources.

[0031] In the present disclosure, during the interaction between the server and the client, it is possible to determine whether to use a pre-recorded video stream or other interactive forms that occupy less resources with the client based on the information of the schedulable resources, so as to release various resources including GPU resources or reduce resource occupation. In the subsequent interactive process, according to the resource recovery situation, the interactive mode of using digital humans for interaction is restored. This solves the problem of reduced service quality when resources are insufficient in the related art.

[0032] Figure 1 A flow chart of an interactive control method provided by an embodiment of the present disclosure is shown as follows: Figure 1 As shown, including:

[0033] Step 101, monitoring the schedulable resources of the server.

[0034] The schedulable resources may be image processing resources, or network resources for multimedia data transmission, which is not limited in this embodiment.

[0035] In some possible implementations, the image processing resource may be a GPU resource.

[0036] Optionally, the server executing this method can estimate the resources occupied by the server based on the access volume, thereby determining the remaining resources based on the total resources of the server. In some possible cases, it is also necessary to deduct the resources reserved for high-priority services from the remaining resources to obtain the schedulable resources of the server.

[0037] Alternatively, the server may directly monitor the occupancy of the GPU and the network to determine the remaining resources of the server. In some possible cases, the resources reserved for high-priority services may also be deducted from the remaining resources to obtain the schedulable resources of the server.

[0038] Step 102: When the schedulable resources of the server are less than a first threshold, at least one target client is selected from a plurality of clients served by the server.

[0039] The first threshold is set based on manual experience, and may be set in a server resource ratio manner, for example, a set ratio of total server resources is used as the first threshold.

[0040] Those skilled in the art will appreciate that the first threshold may also be set in other ways, which are not limited in this embodiment.

[0041] Step 103, switching the first interaction mode corresponding to the target client to a second interaction mode, and sending the multimedia data generated under the second interaction mode to the target client, wherein the amount of resources required to generate the multimedia data under the first interaction mode is greater than the amount of resources required for the second interaction mode.

[0042] Optionally, the order of multiple interaction modes is queried, wherein the amount of resources required by the interaction modes ranked later is less than the amount of resources required by the interaction modes ranked earlier; according to the order, the interaction mode ranked after the first interaction mode is used as the second interaction mode; and the multimedia data generated in the second interaction mode is sent to the target client. In the case of a large number of interaction modes, this ordering method can facilitate the rapid determination of the switching order.

[0043] Or, optionally, a second interaction mode corresponding to the first interaction mode is pre-configured, and the amount of resources required by the second interaction mode is less than the amount of resources required by the first interaction mode.

[0044] In some possible scenarios, the multiple interaction modes include at least the following modes:

[0045] In a first manner, a matching synthetic digital human video is generated according to the synthetic speech, wherein the action of the digital human in the synthetic digital human video matches the synthetic speech, and multimedia data is generated based on the synthetic digital human video and the synthetic speech.

[0046] The second method is to generate multimedia data based on pre-recorded digital human video and synthesized speech, wherein the movements of the digital human in the pre-recorded digital human video and the synthesized speech may not match.

[0047] The third method is to use synthesized speech as multimedia data.

[0048] The fourth method is to treat text as multimedia data.

[0049] The resources required by the above four methods decrease in sequence. When the first method is switched to the second method, if the schedulable resources are still insufficient, the method is switched to the third method, and so on, until the method is switched to the fourth method.

[0050] Step 104: When the schedulable resources of the server are greater than a second threshold, switch the second interaction mode back to the first interaction mode, wherein the second threshold is greater than or equal to the first threshold.

[0051] In the disclosed embodiment, by monitoring the schedulable resources of the server, when the schedulable resources of the server are less than a first threshold, at least one target client is selected from the multiple clients served by the server, the first interaction mode corresponding to the target client is switched to the second interaction mode, and the multimedia data generated under the second interaction mode is sent to the target client. When the schedulable resources of the server are greater than the second threshold, the second interaction mode is switched back to the first interaction mode. Since the amount of resources required to generate multimedia data under the first interaction mode is greater than the amount of resources required for the second interaction mode, after switching to the second interaction mode, the schedulable resources of the server will be increased, which is conducive to serving more clients and services with higher quality requirements, and improving the efficiency of resource utilization.

[0052] As a possible architecture, Figure 2 The architecture diagram provided in the embodiment of the present disclosure includes a server 21 and a client 22, wherein Figure 2 As shown, there are multiple clients 22 , and a communication connection is established between each client 22 and the server 21 .

[0053] In the personalized digital doctor scenario, Figure 2 The architecture shown is a system that can interact with users through video, voice or text to provide medical consultation, diagnosis, treatment and other services. It is used in products or projects such as online medical platforms, health management applications, and smart home devices.

[0054] Those skilled in the art will appreciate that the present invention may also be used in other digital character scenarios, such as digital pets and game scenarios, which is not limited in this embodiment.

[0055] In the personalized digital doctor scenario, Figure 2 Each client 22 and server 21 in Figure 3 The interaction process shown, Figure 3 is a schematic diagram of the interaction process, such as Figure 3 As shown, including:

[0056] Step 301: The client sends a connection request, wherein the connection request carries user information.

[0057] Optionally, the user information may be the digital doctor image information selected by the client, or may be the user's personal attribute information, based on which the user's preferred digital doctor image is predicted.

[0058] Attribute information may be personal information such as gender and age that is authorized by the user.

[0059] Step 302: The server receives a connection request sent by the client. The server selects a suitable digital doctor model based on the user's information and loads it onto the GPU for real-time rendering.

[0060] Optionally, based on the digital doctor image corresponding to the user information obtained in the previous step, a suitable digital doctor model is determined to improve user satisfaction. The server allocates a GPU to each client, and the digital doctor model is loaded on the GPU allocated by the server to the client.

[0061] The digital doctor model is a model used to visually display the digital doctor. The voice stream content of the digital doctor interacting with the client is generated based on the large model, or, optionally, is real-time input under the control of a real person, which is not limited in this embodiment.

[0062] Step 303: The server sends the rendered digital doctor video stream and voice stream as multimedia data streams to the client device for interaction.

[0063] Step 304: The client performs presentation based on the multimedia data stream and sends user feedback to the server.

[0064] Among them, user feedback can be responses to questions raised by the digital doctor or questions raised by the user.

[0065] Step 305: The server updates the state of the digital doctor model according to the user feedback and the logic of the digital doctor, so as to resend the multimedia data used for the interaction between the digital doctor and the client to the client.

[0066] It should be noted that step 304 and step 305 need to be executed repeatedly for multiple rounds until the client exits the interaction.

[0067] Step 306: The client exits the interaction and sends an end request to the server.

[0068] Step 307: In response to receiving the end request, the server releases GPU resources and saves the interaction record between the client and the digital doctor.

[0069] It can be seen that in the above process, the server allocates one GPU resource to the client. Due to the limitation of server hardware conditions, GPU resources are limited. In order to solve the problem that the server may have insufficient resources to serve multiple clients, Figure 4 A flow chart of an interactive control method provided by an embodiment of the present disclosure is shown as follows: Figure 4 As shown, including:

[0070] Step 401: The server monitors schedulable resources while serving multiple clients.

[0071] Step 402: When the schedulable resources of the server are less than a first threshold, the server queries a GPU in an idle state in the server.

[0072] Among them, the idle GPU may be caused by two situations:

[0073] In one case, the user is in an interactive state with the digital doctor, but the client is currently displaying multimedia data sent by the server. The server is waiting for user feedback provided by the client after the display is completed, causing it to be in an idle state.

[0074] In another case, the user and the digital doctor are in a suspended state of interaction, the client has completed the display of multimedia data sent by the server, and the server has not received any user feedback, causing it to be in an idle state.

[0075] In either of the above two situations, the GPU of the server can be regarded as being in an idle state.

[0076] Step 403: Push the pre-recorded idle-state video stream to the client served by the idle-state GPU, and release the idle-state GPU.

[0077] The idle video stream may be used to display random actions or to set a digital doctor for meaningless actions.

[0078] Optionally, after continuously pushing pre-recorded idle video streams to the client served by the idle GPU, the idle GPU can be released and reallocated to a newly connected client. For the client that pushes the idle video stream, the GPU is reallocated after providing user feedback. In this way, the utilization rate of the GPU is improved.

[0079] Step 404: when the idle GPU has been released, if the schedulable resources of the server are still less than the first threshold, determine the quality priority of each of the clients according to the business scenarios to which the multiple clients belong.

[0080] Among them, the business scenario to which the client belongs can be divided according to whether the synthesized speech part in the multimedia data is manually taken over or generated by a large model.

[0081] Optionally, in the case of manual takeover, the priority of business scenarios is higher than that of large model generation, so as to avoid switching the interaction mode of these manually taken-over clients and avoid degradation of service quality. In addition, the access intention of the client can be further identified based on its behavior, and if the interaction intention is not strong, it can be identified as a business scenario with lower quality priority. In this way, the impact of switching the interaction mode on users can be reduced.

[0082] Step 405: selecting at least one target client from the plurality of clients served by the server in order of quality priority from low to high.

[0083] Step 406: Switch the first interaction mode adopted by the target client to a second interaction mode, and send the multimedia data generated in the second interaction mode to the target client.

[0084] Among them, in the first interaction mode, a matching synthetic digital human video is generated according to the synthetic voice, the movements of the digital human in the synthetic digital human video are matched with the synthetic voice, and multimedia data is generated based on the synthetic digital human video and the synthetic voice.

[0085] In the second interaction mode, multimedia data is generated based on the pre-recorded digital human video and synthesized speech.

[0086] Step 407: The server continues to monitor the schedulable resources, and switches the second interaction mode back to the first interaction mode when it is detected that the schedulable resources are greater than a second threshold, wherein the second threshold is greater than or equal to the first threshold.

[0087] In the disclosed embodiment, by monitoring the schedulable resources of the server, when the schedulable resources of the server are less than a first threshold, at least one target client is selected from the multiple clients served by the server, the first interaction mode corresponding to the target client is switched to the second interaction mode, and the multimedia data generated under the second interaction mode is sent to the target client. When the schedulable resources of the server are greater than the second threshold, the second interaction mode is switched back to the first interaction mode. Since the amount of resources required to generate multimedia data under the first interaction mode is greater than the amount of resources required for the second interaction mode, after switching to the second interaction mode, the schedulable resources of the server will be increased, which is conducive to serving more clients and services with higher quality requirements, and improving the efficiency of resource utilization.

[0088] The interaction process is described for the aforementioned target client. Figure 5 A flow chart showing the interaction between the target client and the server, such as Figure 5 As shown, including:

[0089] Step 501: The target client sends a connection request to the server, wherein the connection request carries user information.

[0090] Please refer to the relevant description in the aforementioned implementation for details, which will not be described in detail in this embodiment.

[0091] Step 502: The server selects a digital doctor model based on user information and loads it into the GPU to obtain multimedia data.

[0092] The multimedia data in this step is generated by the digital doctor video stream and synthesized speech obtained by real-time rendering by GPU.

[0093] Step 503: Send multimedia data to the target client.

[0094] Step 504: The server monitors the schedulable resources and determines whether resource optimization or system degradation is required.

[0095] Step 505: When it is determined that resource optimization is required based on the schedulable resources, the idle video stream is pushed to the target client to release the resources.

[0096] Step 506: When it is determined based on the schedulable resources that system degradation is required, the service form switching interaction mode is adjusted to reduce resource consumption.

[0097] Optionally, the first interaction mode used by the server to generate multimedia data is switched to a second interaction mode, and in the second interaction mode, multimedia data is generated based on the pre-recorded digital human video and synthesized speech; and the multimedia data generated in the second interaction mode is sent to the target client. By pre-recording the digital human video, the resources required by the GPU to generate the digital human video are reduced, thereby improving the interaction efficiency.

[0098] Among them, generating multimedia data based on pre-recorded digital human videos and synthesized speech includes: reading multiple pre-recorded digital human videos; generating corresponding interaction contexts according to the interaction contexts sent by the target client; performing speech synthesis according to the interaction contexts to obtain the synthesized speech; synthesizing the multiple pre-recorded digital human videos according to the length of the synthesized speech to obtain videos with matching lengths; and using the videos with matching lengths and the synthesized speech as the multimedia data, so that the audio and video can be aligned.

[0099] Optionally, the first interaction mode used by the server to generate multimedia data is switched to a second interaction mode, and in the second interaction mode, the synthesized speech is used as the multimedia data; and the multimedia data generated in the second interaction mode is sent to the target client. In the case of insufficient resources, the synthesized speech can be used as multimedia data to further reduce resource usage.

[0100] The corresponding interaction context can be generated according to the interaction context sent by the target client, and speech synthesis can be performed according to the interaction context to obtain synthesized speech as the multimedia data. The speech synthesis is performed based on the generation of the corresponding interaction context based on the interaction context without occupying GPU resources.

[0101] Optionally, the first interaction mode used by the server to generate multimedia data is switched to a second interaction mode, and in the second interaction mode, text is used as multimedia data; and the multimedia data generated in the second interaction mode is sent to the target client. In the case of pure text interaction, the server resources occupied can be ignored.

[0102] Wherein, according to the interaction context sent by the target client, a corresponding interaction context is generated, and the text of the interaction context is used as the multimedia data, without occupying GPU resources.

[0103] In this step, the client and the server interact with each other in the switched interaction mode.

[0104] Optionally, the client sends user feedback to the server, for example, the digital doctor may reply in text or voice, or may ask questions to the digital doctor in text or voice.

[0105] Step 507: The server monitors the status of schedulable resources and determines whether to restore the original interaction mode.

[0106] Step 508: If it is determined that the original interaction mode needs to be restored, restore the original interaction mode.

[0107] Optionally, the original interaction mode, i.e., the first interaction mode, can generate a matching synthetic digital human video according to the synthesized voice under the first interaction mode, wherein the movements of the digital human in the synthetic digital human video match the synthesized voice; generate multimedia data based on the synthetic digital human video and the synthesized voice; and send the multimedia data generated under the first interaction mode to the target client, thereby improving the interaction experience.

[0108] In order to match the synthesized speech with the mouth shape of the digital human in the synthesized digital human video, a matching synthesized digital human video can be generated based on the synthesized speech, including generating a corresponding interaction context based on the interaction context sent by the target client; performing speech synthesis based on the interaction context to obtain the synthesized speech; and generating the synthesized digital human video based on the pronunciation of each frame in the synthesized speech.

[0109] In order to match the synthesized speech with the mouth shape of the digital human in the synthesized digital human video, the pronunciation can be used as an input to drive the model. Optionally, the pronunciation of each frame in the synthesized speech can be used as an input to the digital character model to generate the synthesized digital human video based on the sequence of actions corresponding to the pronunciation of each frame in the synthesized speech.

[0110] In the disclosed embodiment, by monitoring the schedulable resources of the server, when the schedulable resources of the server are less than a first threshold, at least one target client is selected from the multiple clients served by the server, the first interaction mode corresponding to the target client is switched to the second interaction mode, and the multimedia data generated under the second interaction mode is sent to the target client. When the schedulable resources of the server are greater than the second threshold, the second interaction mode is switched back to the first interaction mode. Since the amount of resources required to generate multimedia data under the first interaction mode is greater than the amount of resources required for the second interaction mode, after switching to the second interaction mode, the schedulable resources of the server will be increased, which is conducive to serving more clients and services with higher quality requirements, and improving the efficiency of resource utilization.

[0111] According to an embodiment of the present disclosure, the present disclosure also provides an interactive control device, Figure 6 A schematic diagram of the structure of an interactive control device 600 provided in an embodiment of the present disclosure is shown in FIG. Figure 6 As shown, it includes: a monitoring module 601, a selection module 602, a first switching module 603 and a second switching module 604.

[0112] The monitoring module 601 is used to monitor the schedulable resources of the server.

[0113] The selection module 602 is configured to select at least one target client from a plurality of clients served by the server when the schedulable resources of the server are less than a first threshold.

[0114] The first switching module 603 is used to switch the first interaction mode corresponding to the target client to the second interaction mode, and send the multimedia data generated under the second interaction mode to the target client, wherein the amount of resources required to generate the multimedia data under the first interaction mode is greater than the amount of resources required for the second interaction mode.

[0115] The second switching module 604 is configured to switch the second interaction mode back to the first interaction mode when the schedulable resources of the server are greater than a second threshold, wherein the second threshold is greater than or equal to the first threshold.

[0116] In some possible implementations, the first switching module 603 includes:

[0117] A first switching unit, configured to switch the first interaction mode used by the server to generate multimedia data to a second interaction mode, and generate multimedia data based on the pre-recorded digital human video and synthesized speech in the second interaction mode;

[0118] A sending unit is used to send the multimedia data generated in the second interaction mode to the target client.

[0119] Optionally, the first switching unit is used to: read multiple pre-recorded digital human videos; generate a corresponding interaction context based on the interaction context sent by the target client; perform speech synthesis based on the interaction context to obtain the synthesized speech; synthesize the multiple pre-recorded digital human videos based on the length of the synthesized speech to obtain a video with matching length; and use the video with matching length and the synthesized speech as the multimedia data.

[0120] In some possible implementations, the first switching module 603 includes:

[0121] A second switching unit, configured to switch the first interaction mode used by the server to generate multimedia data to a second interaction mode, and in the second interaction mode, use the synthesized speech as the multimedia data;

[0122] A sending unit is used to send the multimedia data generated in the second interaction mode to the target client.

[0123] Optionally, the second switching unit is used to: generate a corresponding interaction context according to the interaction context sent by the target client; and perform speech synthesis according to the interaction context to obtain synthesized speech as the multimedia data.

[0124] In some possible implementations, the first switching module 603 includes:

[0125] A third switching unit, configured to switch the first interaction mode used by the server to generate multimedia data to a second interaction mode, and in the second interaction mode, use the text as the multimedia data;

[0126] A sending unit is used to send the multimedia data generated in the second interaction mode to the target client.

[0127] Optionally, the third switching unit is used to: generate a corresponding interaction context according to the interaction context sent by the target client, and use the text of the interaction context as the multimedia data.

[0128] In some possible implementations, the second switching module 604 is further configured to:

[0129] In the first interaction mode, a matching synthetic digital human video is generated according to the synthetic voice, wherein the action of the digital human in the synthetic digital human video matches the synthetic voice;

[0130] Generating multimedia data based on the synthesized digital human video and synthesized speech;

[0131] The multimedia data generated in the first interaction mode is sent to the target client.

[0132] Among them, the second switching module is also used to: generate a corresponding interaction context according to the interaction context sent by the target client; perform speech synthesis according to the interaction context to obtain the synthesized speech; and generate the synthesized digital human video according to the pronunciation of each frame in the synthesized speech.

[0133] Based on this, the second switching module is also used to: input the pronunciation of each frame in the synthesized speech as input to the digital character model, so as to generate the synthesized digital human video based on the sequence of actions corresponding to the pronunciation of each frame in the synthesized speech.

[0134] The interactive control device 600 further includes:

[0135] The loading module is used to receive the user information sent by the target client; and select the digital character model according to the user information.

[0136] In some possible implementations, the first switching module 603 further includes:

[0137] A sorting unit is used to query the sorting of multiple interaction modes, wherein the amount of resources required by the interaction mode ranked later is less than the amount of resources required by the interaction mode ranked earlier; according to the sorting, the interaction mode ranked after the first interaction mode is used as the second interaction mode; and the multimedia data generated under the second interaction mode is sent to the target client.

[0138] In some possible implementations, the selection module 602 is configured to:

[0139] When the schedulable resources of the server are less than a first threshold, querying a GPU in the server that is in an idle state;

[0140] Pushing a pre-recorded idle-state video stream to a client served by the idle-state GPU, and releasing the idle-state GPU;

[0141] In a case where the idle GPU has been released, if the schedulable resources of the server are still less than the first threshold, at least one target client is selected from a plurality of clients served by the server.

[0142] In some possible implementations, the selection module 602 is configured to:

[0143] Determining the quality priority of each of the clients according to the business scenarios to which the multiple clients belong;

[0144] At least one target client is selected from the plurality of clients served by the server in an ascending order of the quality priorities.

[0145] In the disclosed embodiment, by monitoring the schedulable resources of the server, when the schedulable resources of the server are less than a first threshold, at least one target client is selected from the multiple clients served by the server, the first interaction mode corresponding to the target client is switched to the second interaction mode, and the multimedia data generated under the second interaction mode is sent to the target client. When the schedulable resources of the server are greater than the second threshold, the second interaction mode is switched back to the first interaction mode. Since the amount of resources required to generate multimedia data under the first interaction mode is greater than the amount of resources required for the second interaction mode, after switching to the second interaction mode, the schedulable resources of the server will be increased, which is conducive to serving more clients and services with higher quality requirements, and improving the efficiency of resource utilization.

[0146] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0147] Figure 7 A schematic block diagram of an example electronic device 700 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0148] like Figure 7 As shown, the device 700 includes a computing unit 701, which can perform various appropriate actions and processes according to a computer program stored in a ROM (Read-Only Memory) 702 or a computer program loaded from a storage unit 708 to a RAM (Random Access Memory) 703. In the RAM 703, various programs and data required for the operation of the device 700 can also be stored. The computing unit 701, the ROM 702, and the RAM 703 are connected to each other via a bus 704. An I / O (Input / Output) interface 705 is also connected to the bus 704.

[0149] A number of components in the device 700 are connected to the I / O interface 705, including: an input unit 706, such as a keyboard, a mouse, etc.; an output unit 707, such as various types of displays, speakers, etc.; a storage unit 708, such as a disk, an optical disk, etc.; and a communication unit 709, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 709 allows the device 700 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0150] The computing unit 701 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 701 include, but are not limited to, a CPU (Central Processing Unit), a GPU (Graphic Processing Units), various dedicated AI (Artificial Intelligence) computing chips, various computing units running machine learning model algorithms, a DSP (Digital Signal Processor), and any appropriate processor, controller, microcontroller, etc. The computing unit 701 performs the various methods and processes described above, such as the interactive control method. For example, in some embodiments, the interactive control method may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 708. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 700 via the ROM 702 and / or the communication unit 709. When the computer program is loaded into the RAM 703 and executed by the computing unit 701, one or more steps of the interactive control method described above may be performed. Alternatively, in other embodiments, the computing unit 701 may be configured to execute the method in any other suitable manner (eg, by means of firmware).

[0151] Various embodiments of the systems and techniques described above herein may be implemented in digital electronic circuit systems, integrated circuit systems, FPGAs (Field Programmable Gate Arrays), ASICs (Application-Specific Integrated Circuits), ASSPs (Application Specific Standard Products), SOCs (System On Chips), CPLDs (Complex Programmable Logic Devices), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include: being implemented in one or more computer programs that may be executed and / or interpreted on a programmable system including at least one programmable processor that may be a dedicated or general-purpose programmable processor that may receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0152] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0153] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or device, or any suitable combination of the foregoing. More specific examples of machine-readable storage media may include electrical connections based on one or more lines, portable computer disks, hard disks, RAM, ROM, EPROM (Electrically Programmable Read-Only-Memory) or flash memory, optical fiber, CD-ROM (Compact Disc Read-Only Memory), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0154] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (Cathode-Ray Tube) or LCD (Liquid Crystal Display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0155] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: LAN (Local Area Network), WAN (Wide Area Network), the Internet, and blockchain networks.

[0156] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship between the client and the server is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services ("Virtual Private Server", or "VPS" for short). The server may also be a server of a distributed system, or a server combined with a blockchain.

[0157] It should be noted that artificial intelligence is a discipline that studies how computers can simulate certain human thought processes and intelligent behaviors (such as learning, reasoning, thinking, planning, etc.), and includes both hardware-level and software-level technologies. Artificial intelligence hardware technologies generally include technologies such as sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, and big data processing; artificial intelligence software technologies mainly include computer vision technology, speech recognition technology, natural language processing technology, as well as machine learning / deep learning, big data processing technology, knowledge graph technology, and other major directions.

[0158] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0159] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. An interactive control method, comprising: Monitor the server's schedulable resources; When the schedulable resources of the server are less than a first threshold, selecting at least one target client from a plurality of clients served by the server; Switching the first interaction mode corresponding to the target client to the second interaction mode, and sending the multimedia data generated in the second interaction mode to the target client, wherein the amount of resources required to generate the multimedia data in the first interaction mode is greater than the amount of resources required in the second interaction mode; wherein, in the first interaction mode, the multimedia data is generated based on a synthesized digital human video and a synthesized voice, and the movements of the digital human in the synthesized digital human video match the synthesized voice; and in the second interaction mode, the multimedia data is generated based on a pre-recorded digital human video and the synthesized voice, or is generated based on the synthesized voice, or is generated based on text; When the schedulable resources of the server are greater than a second threshold, the second interaction mode is switched back to the first interaction mode, wherein the second threshold is greater than or equal to the first threshold.

2. The method according to claim 1, wherein: The step of switching the first interaction mode corresponding to the target client to the second interaction mode, and sending the multimedia data generated in the second interaction mode to the target client includes: Switching the first interaction mode used by the server to generate multimedia data to a second interaction mode, and generating multimedia data based on the pre-recorded digital human video and synthesized speech in the second interaction mode; The multimedia data generated in the second interaction mode is sent to the target client.

3. The method according to claim 2, wherein: The method of generating multimedia data based on pre-recorded digital human video and synthesized speech includes: Read multiple pre-recorded digital human videos; Generate a corresponding interaction context according to the interaction context sent by the target client; Performing speech synthesis according to the interactive context to obtain the synthesized speech; Synthesizing the multiple pre-recorded digital human videos according to the length of the synthesized speech to obtain videos with matching lengths; The video with matching length and the synthesized speech are used as the multimedia data.

4. The method according to claim 1, wherein: The step of switching the first interaction mode corresponding to the target client to the second interaction mode, and sending the multimedia data generated in the second interaction mode to the target client includes: Switching a first interaction mode used by the server to generate multimedia data to a second interaction mode, and using synthesized speech as multimedia data in the second interaction mode; The multimedia data generated in the second interaction mode is sent to the target client.

5. The method according to claim 4, wherein: The synthesized speech is used as multimedia data, including: Generate a corresponding interaction context according to the interaction context sent by the target client; Speech synthesis is performed according to the interaction context to obtain synthesized speech as the multimedia data.

6. The method according to claim 1, wherein: The step of switching the first interaction mode corresponding to the target client to the second interaction mode, and sending the multimedia data generated in the second interaction mode to the target client includes: Switching a first interaction mode used by the server to generate multimedia data to a second interaction mode, and using text as multimedia data in the second interaction mode; The multimedia data generated in the second interaction mode is sent to the target client.

7. The method according to claim 6, wherein: The method of using the text as multimedia data includes: A corresponding interaction context is generated according to the interaction context sent by the target client, and the text of the interaction context is used as the multimedia data.

8. The method according to claim 1, wherein: After the second interaction mode is switched back to the first interaction mode when the schedulable resources of the server are greater than the second threshold, the method further includes: In the first interaction mode, a matching synthetic digital human video is generated according to the synthetic voice, wherein the action of the digital human in the synthetic digital human video matches the synthetic voice; Generating multimedia data based on the synthesized digital human video and synthesized speech; The multimedia data generated in the first interaction mode is sent to the target client.

9. The method according to claim 8, wherein: The method of generating a matching synthetic digital human video according to the synthetic speech comprises: Generate a corresponding interaction context according to the interaction context sent by the target client; Performing speech synthesis according to the interactive context to obtain the synthesized speech; The synthesized digital human video is generated according to the pronunciation of each frame in the synthesized speech.

10. The method according to claim 9, wherein: The step of generating the synthesized digital human video according to a sequence of actions corresponding to pronunciations of each frame in the synthesized speech comprises: The pronunciation of each frame in the synthesized speech is used as input and input into the digital character model to generate the synthesized digital human video based on the sequence of actions corresponding to the pronunciation of each frame in the synthesized speech.

11. The method according to claim 10, wherein: The method further comprises: Receiving user information sent by the target client; The digital character model is selected according to the user information.

12. The method according to any one of claims 1 to 11, wherein: The step of switching the first interaction mode corresponding to the target client to the second interaction mode, and sending the multimedia data generated in the second interaction mode to the target client includes: Query the ranking of multiple interaction modes, wherein the amount of resources required by the interaction modes ranked later is less than the amount of resources required by the interaction modes ranked earlier; According to the ranking, taking the interaction mode ranked after the first interaction mode as the second interaction mode; The multimedia data generated in the second interaction mode is sent to the target client.

13. The method according to any one of claims 1 to 11, wherein: The selecting at least one target client from a plurality of clients served by the server when the schedulable resources of the server are less than a first threshold comprises: When the schedulable resources of the server are less than a first threshold, querying a GPU in the server that is in an idle state; Pushing a pre-recorded idle-state video stream to a client served by the idle-state GPU, and releasing the idle-state GPU; In a case where the idle GPU has been released, if the schedulable resources of the server are still less than the first threshold, at least one target client is selected from a plurality of clients served by the server.

14. The method according to claim 13, wherein: The digital person displayed by the multimedia data is a digital doctor; the client served by the idle GPU has received the multimedia data sent by the server and is displaying the interactive content of the digital doctor based on the multimedia data.

15. The method according to claim 13, wherein: The digital person displayed by the multimedia data is a digital doctor; the client served by the idle GPU is in a state of suspended interaction with the digital doctor, wherein in the state, the client has completed displaying the interactive content of the digital doctor based on the multimedia data and has not collected any user feedback.

16. The method according to any one of claims 1 to 11, wherein: The selecting at least one target client from the plurality of clients served by the server comprises: Determining the quality priority of each of the clients according to the business scenarios to which the multiple clients belong; At least one target client is selected from the plurality of clients served by the server in an ascending order of the quality priorities.

17. An interactive control device, comprising: A monitoring module, used to monitor the schedulable resources of the server; A selection module, configured to select at least one target client from a plurality of clients served by the server when the schedulable resources of the server are less than a first threshold; A first switching module is used to switch the first interaction mode corresponding to the target client to a second interaction mode, and send the multimedia data generated in the second interaction mode to the target client, wherein the amount of resources required to generate the multimedia data in the first interaction mode is greater than the amount of resources required in the second interaction mode; wherein, in the first interaction mode, the multimedia data is generated based on a synthesized digital human video and a synthesized voice, and the action of the digital human in the synthesized digital human video matches the synthesized voice; and in the second interaction mode, the multimedia data is generated based on a pre-recorded digital human video and the synthesized voice, or is generated based on the synthesized voice, or is generated based on text; The second switching module is used to switch the second interaction mode back to the first interaction mode when the schedulable resources of the server are greater than a second threshold, wherein the second threshold is greater than or equal to the first threshold.

18. The device according to claim 17, wherein: The first switching module includes: A first switching unit, configured to switch the first interaction mode used by the server to generate multimedia data to a second interaction mode, and generate multimedia data based on the pre-recorded digital human video and synthesized speech in the second interaction mode; A sending unit is used to send the multimedia data generated in the second interaction mode to the target client.

19. The device according to claim 18, wherein: The first switching unit is used to: Read multiple pre-recorded digital human videos; Generate a corresponding interaction context according to the interaction context sent by the target client; Performing speech synthesis according to the interactive context to obtain the synthesized speech; Synthesizing the multiple pre-recorded digital human videos according to the length of the synthesized speech to obtain videos with matching lengths; The video with matching length and the synthesized speech are used as the multimedia data.

20. The device according to claim 17, wherein: The first switching module includes: A second switching unit, configured to switch the first interaction mode used by the server to generate multimedia data to a second interaction mode, and in the second interaction mode, use the synthesized speech as the multimedia data; A sending unit is used to send the multimedia data generated in the second interaction mode to the target client.

21. The device according to claim 20, wherein: The second switching unit is used to: Generate a corresponding interaction context according to the interaction context sent by the target client; Speech synthesis is performed according to the interaction context to obtain synthesized speech as the multimedia data.

22. The device according to claim 17, wherein: The first switching module includes: A third switching unit, configured to switch the first interaction mode used by the server to generate multimedia data to a second interaction mode, and in the second interaction mode, use the text as the multimedia data; A sending unit is used to send the multimedia data generated in the second interaction mode to the target client.

23. The device according to claim 22, wherein: The third switching unit is used to: A corresponding interaction context is generated according to the interaction context sent by the target client, and the text of the interaction context is used as the multimedia data.

24. The device according to claim 17, wherein: The second switching module is further used for: In the first interaction mode, a matching synthetic digital human video is generated according to the synthetic voice, wherein the action of the digital human in the synthetic digital human video matches the synthetic voice; Generating multimedia data based on the synthesized digital human video and synthesized speech; The multimedia data generated in the first interaction mode is sent to the target client.

25. The device according to claim 24, wherein: The second switching module is further used for: Generate a corresponding interaction context according to the interaction context sent by the target client; Performing speech synthesis according to the interactive context to obtain the synthesized speech; The synthesized digital human video is generated according to the pronunciation of each frame in the synthesized speech.

26. The device according to claim 25, wherein The second switching module is further used for: The pronunciation of each frame in the synthesized speech is used as input and input into the digital character model to generate the synthesized digital human video based on the sequence of actions corresponding to the pronunciation of each frame in the synthesized speech.

27. The device according to claim 26, wherein: The device also includes: The loading module is used to receive the user information sent by the target client; and select the digital character model according to the user information.

28. The device according to any one of claims 17 to 27, wherein: The first switching module includes: A sorting unit is used to query the sorting of multiple interaction modes, wherein the amount of resources required by the interaction mode ranked later is less than the amount of resources required by the interaction mode ranked earlier; according to the sorting, the interaction mode ranked after the first interaction mode is used as the second interaction mode; and the multimedia data generated under the second interaction mode is sent to the target client.

29. The device according to any one of claims 17 to 27, wherein: The selection module is used to: When the schedulable resources of the server are less than a first threshold, querying a GPU in the server that is in an idle state; Pushing a pre-recorded idle-state video stream to a client served by the idle-state GPU, and releasing the idle-state GPU; In a case where the idle GPU has been released, if the schedulable resources of the server are still less than the first threshold, at least one target client is selected from a plurality of clients served by the server.

30. The device according to claim 29, wherein: The digital person displayed by the multimedia data is a digital doctor; the client served by the idle GPU has received the multimedia data sent by the server and is displaying the interactive content of the digital doctor based on the multimedia data.

31. The device according to claim 29, wherein: The digital person displayed by the multimedia data is a digital doctor; the client served by the idle GPU is in a state of suspended interaction with the digital doctor, wherein in the state, the client has completed displaying the interactive content of the digital doctor based on the multimedia data and has not collected any user feedback.

32. The device according to any one of claims 17 to 27, wherein: The selection module is used to: Determining the quality priority of each of the clients according to the business scenarios to which the multiple clients belong; At least one target client is selected from the plurality of clients served by the server in an ascending order of the quality priorities.

33. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 16.

34. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-16.

35. A computer program product comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 16.

Images