Method, device and equipment for synthesizing digital human at client and medium
By synthesizing digital humans on the client side and rendering them using client-side hardware resources and adapted models, the dependence on server resources in traditional methods is solved, achieving efficient and personalized digital human synthesis, and improving user experience and system stability.
Patent Information
- Application Number
- CN202411098432.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-12
- Publication Date
- 2024-11-26
AI Technical Summary
Traditional digital human synthesis methods are highly dependent on server-side computing resources, resulting in high implementation costs and limited user experience, especially when network conditions are poor.
The digital human is synthesized on the client side by acquiring the trained digital human synthesis model and converting it into a format adapted to the client. The client's hardware computing resources are used for rendering, and natural language processing and computer vision technologies are combined for sentiment analysis to dynamically adjust the digital human's performance.
It significantly improves the efficiency and user experience of digital human synthesis, reduces reliance on server resources, reduces development and deployment costs, and enhances the scalability and personalization of the system.
Smart Images

Figure CN119027299A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a method, device, equipment and medium for synthesizing a digital human on a client. Background Art
[0002] In recent years, with the rapid progress of artificial intelligence (AI), digital human synthesis technology has been increasingly used in many fields, including entertainment, education, medical care, and customer service. This technology can not only create highly realistic virtual characters, but also provide users with a more vivid and interactive experience. However, traditional digital human synthesis methods are often highly dependent on powerful computing resources on the server side, which not only increases the complexity of the system, but also limits the user's instant interactive experience to a certain extent.
[0003] Specifically, server-side computing methods usually require high-bandwidth network connections and powerful computing capabilities as support, which undoubtedly increases the implementation cost. More importantly, in an environment with poor network conditions, this reliance may cause a significant impact on the user experience, thereby reducing overall satisfaction.
[0004] Therefore, there is an urgent need to provide a more efficient and convenient digital human synthesis technology solution to further enhance the user experience and promote the sustainable development of related fields. Summary of the invention
[0005] The purpose of the present invention is to provide a method, device, medium and equipment for synthesizing a digital human on a client side, so as to improve the problem of excessive reliance on computing resources on a server side in traditional solutions.
[0006] In the first aspect, an embodiment of the present invention provides a method for synthesizing a digital human on a client, comprising: obtaining a trained first digital human synthesis model from a server; converting the first digital human synthesis model into a second digital human synthesis model corresponding to a target format, wherein the target format is compatible with an operating system of the client; integrating the second digital human synthesis model into a target application, and calling the hardware computing power resources of the client to run the target application; after acquiring user data through an acquisition device, using the second digital human synthesis model to process the user data, and calling the hardware computing power resources to render the processed data on the client to synthesize a digital human.
[0007] In a possible implementation, the user data is processed using the second digital human synthesis model, and the hardware computing resources are called to render the processed data on the client to synthesize a digital human, including: using the second digital human synthesis model and the hardware computing resources of the client to perform speech recognition, and converting the speech part in the user data into text in real time; inputting the text into the question and answer module of the second digital human synthesis model to obtain a text answer; converting the text answer into audio data; and calling the hardware computing resources to synthesize image data corresponding to the image part in the user data; rendering the audio data and image data on the client to synthesize a digital human.
[0008] In a possible implementation, the audio data and image data are rendered on the client to synthesize a digital human, including: combining natural language processing and computer vision technology to perform sentiment analysis on the user data, and passing the analysis results to a rendering engine; the audio data and image data are rendered on the client to synthesize a digital human, and facial muscle movements, eye expression changes, and voice intonation of the digital human are adjusted based on the analysis results.
[0009] In a possible implementation, when the client renders the audio data and image data, it also includes: using augmented reality technology to capture the user's environmental information through a camera, and using a deep learning algorithm to analyze the environmental information to obtain environmental features; and dynamically adjusting the rendering effect of the digital human based on the environmental features.
[0010] In a possible implementation, the user data is acquired through a collection device, including: collecting the user's body posture and facial expression through a motion capture device and a camera integrated with various sensors, and collecting the user's voice through a microphone to obtain the user data.
[0011] In a possible implementation, after synthesizing the digital human, the process further includes: receiving a configuration operation performed by a user on the digital human configuration interface; and adjusting the parameters of the digital human in response to the configuration operation to obtain a customized digital human.
[0012] In a possible implementation, the target format is adapted to the client system, including: when the client system is an iOS system, the target format is a Core ML format; when the client system is an Android system, the target format is a TensorFlow Lite format.
[0013] In a second aspect, an embodiment of the present invention further provides a device for synthesizing a digital human on a client, comprising:
[0014] A data acquisition unit, used for acquiring the trained first digital human synthesis model from the server;
[0015] A data processing unit, configured to convert the first digital human synthesis model into a second digital human synthesis model corresponding to a target format, wherein the target format is compatible with an operating system of a client; integrate the second digital human synthesis model into a target application, and call the hardware computing resources of the client to run the target application;
[0016] The digital human synthesis unit is used to obtain user data through the acquisition device, process the user data using the second digital human synthesis model, and call the hardware computing resources to render the processed data on the client to synthesize the digital human.
[0017] In a third aspect, an embodiment of the present invention further provides an electronic device, comprising a memory and a processor, wherein the memory stores a program that can be run on the processor, and when the program is executed by the processor, the electronic device implements a method in any possible embodiment of the first aspect.
[0018] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium includes a program, and when the program runs on an electronic device, the electronic device executes a method in any possible implementation of the first aspect.
[0019] In a fifth aspect, an embodiment of the present invention further provides a computer program product, which, when executed on an electronic device, enables the electronic device to execute a method of any possible implementation of the first aspect.
[0020] The present invention provides a method, device, medium and equipment for synthesizing a digital human on a client. The beneficial effects of the technical solution are mainly reflected in the following aspects:
[0021] 1. Significantly improve synthesis efficiency: This invention utilizes the client's hardware computing resources (such as GPU, neural network chip, etc.) to directly perform digital human synthesis locally, avoiding the delay of data transmission and server processing in traditional methods, thereby significantly improving the real-time and efficiency of digital human synthesis. The adaptability design for different operating systems (such as iOS and Android) ensures that no matter which platform, the hardware performance can be fully utilized to achieve the best synthesis effect.
[0022] 2. Optimize resource usage: Transferring computing tasks from the server to the client reduces the processing pressure on the server, reduces the dependence on and occupation of server resources, and improves the overall stability and scalability of the system. The client's hardware resources are fully utilized, reducing resource waste and improving overall resource utilization efficiency.
[0023] 3. High scalability and flexibility: The digital human synthesis method proposed in the present invention is highly modular and scalable. New modules and functions can be easily added according to different scenarios and requirements to meet the ever-changing user needs. The client configuration interface allows users to customize the parameters of the digital human, such as facial features, voice intonation, etc., which increases the personalization and flexibility of digital human synthesis.
[0024] 4. Enhanced user experience: By combining natural language processing and computer vision technology, we conduct sentiment analysis on user data and adjust the digital human's facial muscle movements, eye expression changes, and voice intonation accordingly, making the digital human's performance more natural and vivid, enhancing the user's sense of immersion and interactive experience. By using augmented reality technology, we dynamically adjust the rendering effect of the digital human according to the user's environment, so that the digital human can better integrate into the real scene and enhance the user's visual experience.
[0025] 5. Reduce development and deployment costs: The client-side method of implementing digital human synthesis reduces the reliance on server resources and reduces the procurement and maintenance costs of server resources. The ability to quickly deploy and expand enables developers to quickly respond to market demand and reduce the development and deployment cycle and cost.
[0026] In summary, the method for synthesizing digital humans on the client side proposed in the present invention not only improves the efficiency of digital human synthesis and the rationality of resource utilization, but also enhances user experience and system scalability, reduces development and deployment costs, and has significant beneficial effects. BRIEF DESCRIPTION OF THE DRAWINGS
[0027] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0028] Figure 1 A schematic flow chart of a method for synthesizing a digital human on a client provided by an embodiment of the present invention;
[0029] Figure 2 A schematic diagram of another digital human synthesis system provided by an embodiment of the present invention;
[0030] Figure 3 A schematic diagram of the structure of a device for synthesizing a digital human on a client provided by an embodiment of the present invention;
[0031] Figure 4 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION
[0032] The embodiments of the present invention relate to artificial intelligence (AI) and machine learning (ML) technologies, and implement solutions based on deep learning networks and machine learning in artificial intelligence.
[0033] With the research and advancement of artificial intelligence technology, artificial intelligence has been studied and applied in many fields, such as common smart homes, smart customer service, virtual assistants, smart speakers, smart marketing, unmanned driving, autonomous driving, robots, smart medical care, etc. I believe that with the development of technology, artificial intelligence will be applied in more fields and play an increasingly important role.
[0034] Machine learning is a multi-disciplinary subject that involves probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory and other disciplines. It specializes in studying how computers simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications are spread across all areas of artificial intelligence. Machine learning and deep learning usually include artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning and other technologies.
[0035] In the description of the embodiments of the present invention, the terms used in the following embodiments are only for the purpose of describing specific embodiments, and are not intended to be used as limitations of the present invention. As used in the specification of the present invention and the appended claims, the singular expressions "a", "said", "above", "the" and "this" are intended to also include expressions such as "one or more", unless there is a clear contrary indication in the context. It should also be understood that in the following embodiments of the present invention, "at least one", "one or more" refer to one or more (including two). The term "and / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist; for example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship.
[0036] References to "one embodiment" or "some embodiments" etc. described in this specification mean that one or more embodiments of the present invention include specific features, structures or characteristics described in conjunction with the embodiment. Therefore, the statements "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. that appear in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways. The term "connection" includes direct connection and indirect connection, unless otherwise specified. "First" and "second" are used for descriptive purposes only and cannot be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated.
[0037] In the embodiments of the present invention, the words "exemplarily" or "for example" are used to indicate examples, illustrations or explanations. Any embodiment or implementation scheme described as "exemplarily" or "for example" in the embodiments of the present invention should not be interpreted as being more preferred or more advantageous than other embodiments or implementation schemes. Specifically, the use of words such as "exemplarily" or "for example" is intended to present related concepts in a specific way.
[0038] The embodiment of the present invention provides a method for synthesizing a digital human on a client. Figure 1 As shown, the process of the method can be executed by an electronic device, and the method includes the following steps:
[0039] S101, obtaining a trained first digital human synthesis model from a server;
[0040] S102, converting the first digital human synthesis model into a second digital human synthesis model corresponding to a target format, wherein the target format is compatible with an operating system of a client;
[0041] S103, integrating the second digital human synthetic model into a target application, and calling the hardware computing resources of the client to run the target application;
[0042] S104, after acquiring the user data through the acquisition device, the user data is processed using the second digital human synthesis model, and the hardware computing resources are called to render the processed data on the client to synthesize the digital human.
[0043] In this embodiment, the target format is adapted to the client system, including: when the client operating system is iOS system, the target format is Core ML format, and generally, the AI chip used by the terminal device integrated with iOS system is a neural network chip (Apple Neural Engine, ANE). When the client system is Android system, the target format is TensorFlow Lite format, and generally, the hardware computing resources and corresponding model frameworks of the terminal device integrated with Android system are hardware acceleration technology and deep learning framework (such as TensorFlow Lite and Android Neural Networks API, NNAPI).
[0044] In the above S104, the user data is obtained through the acquisition device, including: collecting the user's body posture and facial expression through a motion capture device and a camera that integrates various sensors, and collecting the user's voice through a microphone to obtain the user data. That is to say, first of all, through the advanced motion capture device that integrates various sensors, it is possible to accurately capture every movement and posture of the user's body, thereby obtaining relevant data on the user's body movement. At the same time, the camera can collect the user's facial expressions, which include smiles, frowns, etc., which can reflect the user's emotional changes and psychological state. In addition, the user's voice data will also be collected through a microphone. These data include the user's speech content, voice volume, speech speed, etc., so as to deeply understand and analyze the user's language expression. All of the above user data will be fully recorded as user data and used for subsequent analysis and processing, in order to achieve better user experience and service quality.
[0045] In addition, in the above S104, the user data is processed by using the second digital human synthesis model, and the hardware computing power resources are called to render the processed data on the client to synthesize a digital human, including: using the second digital human synthesis model and the hardware computing power resources of the client to perform speech recognition, and converting the speech part in the user data into text in real time; inputting the text into the question and answer module of the second digital human synthesis model to obtain a text answer; converting the text answer into audio data; and calling the hardware computing power resources to synthesize image data corresponding to the image part in the user data; rendering the audio data and image data on the client to synthesize a digital human.
[0046] Among them, a rendering and synthesis of digital human can be: combining natural language processing and computer vision technology, performing sentiment analysis on the user data, and transmitting the analysis results to the rendering engine; rendering the audio data and image data on the client to synthesize the digital human, and adjusting the digital human's facial muscle movements, eye changes, and voice intonation based on the analysis results. In this embodiment, during the digital human rendering process, the user's voice, facial expression, or text input can be analyzed with the help of a real-time sentiment analysis module, and the digital human's expression and tone can be adjusted accordingly to make it closer to human's natural reaction.
[0047] Another possible way to render a synthetic digital human may be: using augmented reality technology, capturing the user's environment information through a camera, and using a deep learning algorithm to analyze the environment information to obtain environmental features; dynamically adjusting the rendering effect of the digital human based on the environmental features. In this embodiment, the rendering effect of the digital human is automatically adjusted according to the actual environment of the user (such as light, background, etc.) to better integrate it into the surrounding environment.
[0048] It should be understood that the above two possible rendering modes may be used alone or simultaneously, and this embodiment does not limit this.
[0049] In another possible embodiment, after synthesizing the digital human, the process further includes: receiving a configuration operation performed by the user on the digital human configuration interface; and adjusting the parameters of the digital human in response to the configuration operation to obtain a customized digital human. In this embodiment, the digital human configuration interface allows the user to easily customize the various functions of the digital human by dragging, selecting, adjusting, etc., and can customize the digital human's appearance, personality, voice style and other features according to their own preferences to meet the personalized needs of different users; at the same time, the user's customized data can also be saved to the cloud for synchronous use across different devices.
[0050] Exemplary, combined Figure 2For example, for the integrated iOS operating system, the digital human synthesis method provided in this embodiment mainly utilizes the efficient computing power of ANE and the convenient model deployment function of Core ML to achieve real-time digital human synthesis. The specific steps are as follows: Server-side model training: Use a large data set on the server to train the digital human synthesis model to ensure that the model has high precision and high fidelity; Model conversion: Convert the trained model to Core ML format so that it can run efficiently on iOS devices; Model deployment: Deploy the model to the iOS target application through Core ML, and use ANE for accelerated calculation to achieve real-time digital human rendering and interaction. The target application obtains the user's real-time data through the camera or other sensors, and the speech recognition module uses Core ML and ANE for speech recognition to convert the user's voice input into text in real time; Combined with the speech recognition framework that comes with iOS, ensure efficient and accurate speech-to-text processing; Convert the large-scale question-and-answer model trained on the server to Core ML format and deploy it to the client; Use ANE to accelerate model reasoning and generate real-time responses based on user input. Use Core ML and iOS’s Text-to-Speech (TTS) framework to convert generated text responses into natural speech output; leverage ANE for accelerated processing to ensure low latency and high quality of speech output.
[0051] Core ML and ANE are used to synthesize digital human videos, combining the user's real-time data with the synthesis model to generate realistic digital human images; efficient digital human image rendering is achieved through iOS graphics processing interfaces (such as Metal). In addition, the iOS audio and video playback framework is integrated to control the playback, pause, fast forward, and playback functions of digital human synthesized videos and audio, ensuring the synchronization and smooth playback of audio and video.
[0052] As another example, for client devices with integrated Android operating systems, the digital human synthesis method provided in this embodiment mainly uses the hardware acceleration capabilities and deep learning framework of the device to achieve similar functions. The specific steps are as follows: Use a large data set to train the digital human synthesis model on the server side; Convert the trained model to TensorFlow Lite format so that it can run efficiently on Android devices; Deploy the model to the Android application through TensorFlow Lite and NNAPI, and use the hardware acceleration capabilities of the device (such as GPU or dedicated AI acceleration chip) for accelerated calculation to achieve real-time digital human rendering and interaction; The application obtains the user's real-time data through the camera or other sensors, processes it using the TensorFlow Lite model, and renders it on the client side to generate a realistic digital human image.
[0053] In general, the present invention provides a method, device, medium and equipment for synthesizing digital human on the client. The beneficial effects of the technical solution are mainly reflected in the following aspects:
[0054] 1. Significantly improve synthesis efficiency: This invention utilizes the client's hardware computing resources (such as GPU, neural network chip, etc.) to directly perform digital human synthesis locally, avoiding the delay of data transmission and server processing in traditional methods, thereby significantly improving the real-time and efficiency of digital human synthesis. The adaptability design for different operating systems (such as iOS and Android) ensures that no matter which platform, the hardware performance can be fully utilized to achieve the best synthesis effect.
[0055] 2. Optimize resource usage: Transferring computing tasks from the server to the client reduces the processing pressure on the server, reduces the dependence on and occupation of server resources, and improves the overall stability and scalability of the system. The client's hardware resources are fully utilized, reducing resource waste and improving overall resource utilization efficiency.
[0056] 3. High scalability and flexibility: The digital human synthesis method proposed in the present invention is highly modular and scalable. New modules and functions can be easily added according to different scenarios and requirements to meet the ever-changing user needs. The client configuration interface allows users to customize the parameters of the digital human, such as facial features, voice intonation, etc., which increases the personalization and flexibility of digital human synthesis.
[0057] 4. Enhanced user experience: By combining natural language processing and computer vision technology, we conduct sentiment analysis on user data and adjust the digital human's facial muscle movements, eye expression changes, and voice intonation accordingly, making the digital human's performance more natural and vivid, enhancing the user's sense of immersion and interactive experience. By using augmented reality technology, we dynamically adjust the rendering effect of the digital human according to the user's environment, so that the digital human can better integrate into the real scene and enhance the user's visual experience.
[0058] 5. Reduce development and deployment costs: The client-side method of implementing digital human synthesis reduces the reliance on server resources and reduces the procurement and maintenance costs of server resources. The ability to quickly deploy and expand enables developers to quickly respond to market demand and reduce the development and deployment cycle and cost.
[0059] In summary, the method for synthesizing digital humans on the client side proposed in the present invention not only improves the efficiency of digital human synthesis and the rationality of resource utilization, but also enhances user experience and system scalability, reduces development and deployment costs, and has significant beneficial effects.
[0060] In some embodiments of the present invention, the present invention also discloses a device for synthesizing a digital human on a client, such as Figure 3 As shown, the device is used to implement the methods described in the above method embodiments, and includes: a data acquisition unit 301, a data processing unit 302 and a digital human synthesis unit 303, wherein:
[0061] A data acquisition unit 301 is used to acquire the trained first digital human synthesis model from a server;
[0062] The data processing unit 302 is used to convert the first digital human synthesis model into a second digital human synthesis model corresponding to a target format, wherein the target format is compatible with the operating system of the client; integrate the second digital human synthesis model into a target application, and call the hardware computing resources of the client to run the target application;
[0063] The digital human synthesis unit 303 is used to obtain user data through the collection device, process the user data using the second digital human synthesis model, and call the hardware computing resources to render the processed data on the client to synthesize the digital human.
[0064] All relevant contents of each step involved in the above method embodiment can be referred to the functional description of the corresponding functional module, and will not be repeated here.
[0065] The above device may be executed by a chip or a chip module. The various modules / units included in the various devices and products described in the above embodiments may be software modules / units or hardware modules / units, or may be partially software modules / units and partially hardware modules / units.
[0066] Next, an electronic device provided by an embodiment of the present application is introduced. Figure 4 , Figure 4 This is a schematic diagram of a structure of an electronic device provided in an embodiment of the present application. The electronic device 400 can be specifically a mobile phone, a tablet, a laptop computer, a smart wearable device, etc., which is not limited here. Figure 1 Functions of the corresponding method embodiments. Specifically, the electronic device 400 includes: a receiver 401, a transmitter 402, a processor 403 and a memory 404 (wherein the number of processors 403 in the electronic device 400 can be one or more), wherein the processor 403 may include an application processor 4031 and a communication processor 4032. In some embodiments of the present application, the receiver 401, the transmitter 402, the processor 403 and the memory 404 may be connected via a bus or other means.
[0067] The memory 404 may include a read-only memory and a random access memory, and provides instructions and data to the processor 403. A portion of the memory 404 may also include a non-volatile random access memory (NVRAM). The memory 404 stores processor and operation instructions, executable modules or data structures, or subsets thereof, or extended sets thereof, wherein the operation instructions may include various operation instructions for implementing various operations.
[0068] The processor 403 controls the operation of the electronic device. In a specific application, the various components of the electronic device are coupled together through a bus system, wherein the bus system includes not only a data bus but also a power bus, a control bus, and a status signal bus, etc. However, for the sake of clarity, various buses are referred to as bus systems in the figure.
[0069] The method disclosed in the above embodiment of the present application can be applied to the processor 403, or implemented by the processor 403. The processor 403 can be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by an integrated logic circuit of hardware or software instructions in the processor 403. The above processor 403 can be a general-purpose processor, a digital signal processor (digital signal processing, DSP), a microprocessor or a microcontroller, and a vision processor (vision processing unit, VPU), a tensor processor (tensorprocessing unit, TPU) and other processors suitable for AI computing, and can further include an application specific integrated circuit (application specific integrated circuit, ASIC), a field programmable gate array (field-programmable gate array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The processor 403 can implement or execute the disclosed methods, steps and logic block diagrams in the embodiments of the present application. The general processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in the embodiment of the present application can be directly embodied as being executed by a hardware decoding processor, or being executed by a combination of hardware and software modules in a decoding processor. The software module can be located in a mature storage medium in the art such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 404, and the processor 403 reads the information in the memory 404, and completes the steps in the above method embodiment in combination with its hardware.
[0070] The receiver 401 can be used to receive input digital or character information and generate signal input related to the relevant settings and function control of the electronic device. The transmitter 402 can be used to output digital or character information through the first interface; the transmitter 402 can also be used to send instructions to the disk group through the first interface to modify the data in the disk group; the transmitter 402 can also include a display device such as a display screen.
[0071] A computer program product is also provided in an embodiment of the present application. When the computer program product is run on a computer, the computer executes the steps executed by the aforementioned device, or the computer executes the steps executed by the aforementioned device.
[0072] A computer-readable storage medium is also provided in an embodiment of the present application. The computer-readable storage medium stores a program for performing signal processing. When the program is run on a computer, the computer executes the steps executed by the aforementioned device, or the computer executes the steps executed by the aforementioned device.
[0073] The execution device, training device or electronic device provided in the embodiments of the present application may specifically be a chip, and the chip includes: a processing unit and a communication unit, wherein the processing unit may be, for example, a processor, and the communication unit may be, for example, an input / output interface, a pin or a circuit, etc. The processing unit may execute the computer execution instructions stored in the storage unit so that the chip in the execution device executes the data processing method described in the above embodiment, or so that the chip in the training device executes the data processing method described in the above embodiment. Optionally, the storage unit is a storage unit in the chip, such as a register, a cache, etc. The storage unit may also be a storage unit located outside the chip in the wireless access device, such as a read-only memory (ROM) or other types of static storage devices that can store static information and instructions, a random access memory (RAM), etc.
[0074] It should also be noted that the device embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed over multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. In addition, in the drawings of the device embodiments provided by the present application, the connection relationship between the modules indicates that there is a communication connection between them, which may be specifically implemented as one or more communication buses or signal lines.
[0075] Through the description of the above implementation mode, the technicians in the field can clearly understand that the present application can be implemented by means of software plus necessary general hardware, and of course, it can also be implemented by special hardware including special integrated circuits, special CPUs, special memories, special components, etc. In general, all functions completed by computer programs can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be various, such as analog circuits, digital circuits or special circuits. However, for the present application, software program implementation is a better implementation mode in more cases. Based on such an understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a readable storage medium, such as a computer floppy disk, a U disk, a mobile hard disk, a ROM, a RAM, a disk or an optical disk, etc., including a number of instructions to enable a computer device (which can be a personal computer, a training device, or a network device, etc.) to execute the methods described in each embodiment of the present application.
[0076] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware or any combination thereof. When implemented by software, all or part of the embodiments may be implemented in the form of a computer program product.
[0077] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from a website site, a computer, a training device, or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, training device, or data center. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a training device, a data center, etc. that includes one or more available media integrations. The available medium may be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)), etc.
Claims
1. A method for synthesizing a digital human on a client, characterized in that: include: Obtain the trained first digital human synthesis model from the server; Converting the first digital human synthesis model into a second digital human synthesis model corresponding to a target format, wherein the target format is compatible with an operating system of a client; Integrate the second digital human synthetic model into a target application, and call the hardware computing resources of the client to run the target application; After the user data is acquired through the acquisition device, the user data is processed using the second digital human synthesis model, and the hardware computing resources are called to render the processed data on the client to synthesize the digital human.
2. The method according to claim 1, characterized in that The user data is processed using the second digital human synthesis model, and the hardware computing resources are called on the client to render the processed data to synthesize the digital human, including: Using the second digital human synthesis model and the hardware computing resources of the client to perform speech recognition, converting the speech part in the user data into text in real time; Inputting the text into the second digital human synthetic model question-answering module to obtain a text answer; Convert the text answer into audio data; and call the hardware computing resources to synthesize image data corresponding to the image part in the user data; The audio data and the image data are rendered on the client to synthesize a digital human.
3. The method according to claim 2, characterized in that Rendering the audio data and the image data on the client to synthesize a digital human includes: Combining natural language processing and computer vision technology to perform sentiment analysis on the user data, and passing the analysis results to the rendering engine; The audio data and the image data are rendered on the client to synthesize a digital human, and the facial muscle movements, eye expression changes, and voice intonation of the digital human are adjusted based on the analysis results.
4. The method according to claim 2, characterized in that: When the client renders the audio data and the image data, it also includes: Using augmented reality technology, the user's environment information is captured through a camera, and the environment information is analyzed using a deep learning algorithm to obtain environmental features; The rendering effect of the digital human is dynamically adjusted based on the environmental characteristics.
5. The method according to claim 1, characterized in that The user data obtained through the collection device includes: The user data is obtained by collecting the user's body posture and facial expression through a motion capture device and a camera that integrates various sensors, and collecting the user's voice through a microphone.
6. The method according to claim 1, characterized in that After synthesizing the digital human, it also includes: Receiving configuration operations performed by the user on the digital human configuration interface; In response to the configuration operation, the parameters of the digital human are adjusted to obtain a customized digital human.
7. The method according to claim 1, characterized in that The target format is adapted to the client system, including: when the client system is an iOS system, the target format is a Core ML format; when the client system is an Android system, the target format is a TensorFlow Lite format.
8. A device for synthesizing digital human on a client, characterized in that: include: A data acquisition unit, used for acquiring the trained first digital human synthesis model from the server; A data processing unit, configured to convert the first digital human synthesis model into a second digital human synthesis model corresponding to a target format, wherein the target format is compatible with an operating system of a client; integrate the second digital human synthesis model into a target application, and call the hardware computing resources of the client to run the target application; The digital human synthesis unit is used to obtain user data through the acquisition device, process the user data using the second digital human synthesis model, and call the hardware computing resources to render the processed data on the client to synthesize the digital human.
9. An electronic device, characterized in that: include: At least one memory, at least one processor, the at least one processor is used to execute instructions stored in the at least one memory, so that the electronic device performs the method as described in any one of claims 1 to 7.
10. A computer-readable storage medium, wherein a program is stored in the computer-readable storage medium, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.
