Display device and training method of motion-driven model

Abstract

The present application relates to a display device comprising: a display configured to display content from a broadcast system or a network and / or a user interface; and at least one processor connected with the display and configured to execute instructions to enable the display device to: in response to a voice query instruction, obtain a corresponding reply text, and generate audio data, text data and action control data based on the reply text; processing the audio data, the text data and the action control data through an action driving model to obtain action driving data; and playing the audio data and a target video, and controlling the display to display the reply text, the target video being a video generated by controlling a virtual object based on the action driving data. And the limb movement is more flexible.

Description

Technical Field

[0001] The present application relates to the technical field of display devices, and in particular to a display device and a training method for an action-driven model. Background Art

[0002] With the rapid development of the concepts of metaverse and digital humans, the application of 3D digital humans is becoming more and more extensive, including 3D film and television, 3D games, AR / VR, virtual office, virtual social networking, live streaming and other scenarios. At present, the application of 3D interactive digital humans is mainly driven by facial voice and preset actions, and the actions are relatively monotonous. Summary of the invention

[0003] Based on this, it is necessary to provide a display device and a training method for a motion-driven model that can improve the agility of 3D digital humans in response to the above technical problems.

[0004] In a first aspect, the present application provides a display device, including:

[0005] a display configured to display content from a broadcast system or network and / or a user interface;

[0006] and at least one processor connected to the display and configured to execute instructions to cause the display device to:

[0007] In response to the voice query instruction, obtaining a corresponding reply text, and generating audio data, text data and action control data based on the reply text;

[0008] Processing the audio data, the text data and the action control data through an action drive model to obtain action drive data;

[0009] The audio data and the target video are played, and the display is controlled to display the reply text, wherein the target video is a video generated by controlling a virtual object based on the action driving data.

[0010] This multi-dimensional control method makes the virtual object's body movements more flexible.

[0011] In a second aspect, the present application provides a method for training an action-driven model, comprising:

[0012] Training the pre-built action encoder and decoder to obtain a trained action encoder and decoder;

[0013] The pre-built action alignment module is trained by using the trained action encoder to obtain a trained action alignment module;

[0014] Based on the trained action alignment module and the trained decoder, an action-driven model is constructed.

[0015] This enables the motion-driven model to support multi-dimensional control, making the body movements of virtual objects more flexible. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the drawings required for use in the embodiments of the present application or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0017] FIG1 is a schematic diagram of an operation scenario between a display device and a control device provided in some embodiments of the present application;

[0018] Figure 2 A schematic diagram of the hardware configuration of a display device provided in some embodiments of the present application;

[0019] Figure 3 A schematic diagram of the hardware configuration of a control device provided in some embodiments of the present application;

[0020] Figure 4 A schematic diagram of software configuration of a display device provided in some embodiments of the present application;

[0021] Figure 5 A schematic diagram of a flow chart of a virtual object driving method provided in some embodiments of the present application;

[0022] Figure 6 Interface diagram 1 provided for some embodiments of the present application;

[0023] Figure 7 Interface diagrams provided for some embodiments of the present application Figure 2 ;

[0024] Figure 8 A schematic diagram of module interaction provided for some embodiments of the present application;

[0025] Fig. 9 A flowchart of a method for training an action-driven model provided in some embodiments of the present application;

[0026] Fig.10 A schematic diagram of the design of a motion encoder in the training phase provided in some embodiments of the present application;

[0027] Fig.11 A framework diagram of a training rhythm alignment module provided in some embodiments of the present application;

[0028] Fig.12 Figure 1 is a framework diagram of a training semantic alignment module provided in some embodiments of the present application;

[0029] Fig.13 Framework for training semantic alignment module provided in some embodiments of the present application Figure 2 ;

[0030] Fig.14 A framework diagram of a left-right hand alignment training module provided in some embodiments of the present application;

[0031] Fig.15 A framework diagram of a training header height alignment module provided in some embodiments of the present application;

[0032] Fig.16 A framework diagram of a training step activity alignment module provided in some embodiments of the present application;

[0033] Fig.17 A schematic diagram of interaction with a server of a display device provided in some embodiments of the present application. DETAILED DESCRIPTION

[0034] The following embodiments are described in detail, and examples thereof are shown in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The implementations described in the following embodiments do not represent all implementations consistent with the present application. They are only examples of systems and methods consistent with some aspects of the present application as detailed in the claims.

[0035] It should be noted that the brief description of terms in this application is only for the convenience of understanding the embodiments described below, and is not intended to limit the embodiments of this application. Unless otherwise specified, these terms should be understood according to their common and usual meanings.

[0036] The terms "first", "second", "third", etc. in the specification and claims of this application and the above drawings are used to distinguish similar or similar objects or entities, and do not necessarily mean to limit a specific order or sequence, unless otherwise noted. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances.

[0037] The terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover but not exclude inclusion, for example, a product or device comprising a list of components is not necessarily limited to all the components expressly listed but may include other components not expressly listed or inherent to such product or device.

[0038] The term "module" refers to any known or later developed hardware, software, firmware, artificial intelligence, fuzzy logic, or combination of hardware and / or software code that is capable of performing the functions associated with that element.

[0039] In the embodiment of the present application, the display device 200 generally refers to a device with image display and data processing capabilities. For example, the display device 200 includes but is not limited to a smart TV, a mobile terminal, a computer, a monitor, an advertising screen, a wearable device, a virtual reality device, an augmented reality device, etc.

[0040] Figure 1 is a schematic diagram of an operation scenario between a display device and a control device provided in some embodiments of the present application. As shown in Figure 1, a user can operate a display device 200 through touch operation, a mobile terminal 300, and a control device 100. For example, the control device 100 can be a remote controller, a stylus, a handle, etc.

[0041] The mobile terminal 300 can be used as a control device for performing human-computer interaction between the user and the display device 200. The mobile terminal 300 can also be used as a communication device for establishing a communication connection with the display device 200 and performing data interaction. In some embodiments, the mobile terminal 300 can install software applications with the display device 200, and achieve connection and communication through a network communication protocol to achieve the purpose of one-to-one control operation and data communication. The audio and video content displayed on the mobile terminal 300 can also be transmitted to the display device 200 to achieve a synchronous display function.

[0042] 1 , the display device 200 also communicates data with the server 400 through various communication methods. The display device 200 may be allowed to communicate and connect through a local area network (LAN), a wireless local area network (WLAN) and other networks.

[0043] The display device 200 may provide a broadcast receiving television function, and may also additionally provide an intelligent network television function of a computer support function, including but not limited to network television, smart television, Internet Protocol television (IPTV), etc.

[0044] Figure 2 FIG. 1 is a block diagram of a hardware configuration of a display device 200 provided for some embodiments of the present application.

[0045] In some embodiments, the display device 200 may include at least one of a tuner 210, a communication device 220, a detector 230, a device interface 240, a controller 250, a display 260, an audio output device 270, a memory, a power supply, and a user input interface.

[0046] In some embodiments, the detector 230 is used to collect signals of the external environment or external interaction. For example, the detector 230 includes a light receiver, a sensor for collecting the intensity of ambient light; or, the detector 230 includes an image collector, such as a camera, which can be used to collect external environment scenes, user attributes or user interaction gestures; or, the detector 230 includes a sound collector, such as a microphone, etc., for receiving external sounds.

[0047] In some embodiments, the display 260 includes a display function component for presenting a picture, and a driving component for driving an image display. The display 260 is used to receive an image signal output from the controller 250 for display. For example, the display 260 can be used to display video content, image content, and components of a menu control interface and a user control UI interface.

[0048] In some embodiments, the communication device 220 is a component for communicating with an external device or server 400 according to various communication protocol types. The display device 200 may be provided with a plurality of communication devices 220 according to different supported communication modes. For example, when the display device 200 supports wireless network communication, the display device 200 may be provided with a communication device 220 including a WiFi function. When the display device 200 supports Bluetooth connection communication, the display device 200 needs to be provided with a communication device 220 including a Bluetooth function.

[0049] The communication device 220 can enable the display device 200 to communicate with the external device or server 400 by wireless or wired connection. Among them, the wired connection can connect the display device 200 with the external device through components such as data cables and interfaces. The wireless connection can connect the display device 200 with the external device through wireless signals or wireless networks. The display device 200 can establish a connection relationship with the external device directly, or indirectly establish a connection relationship through a gateway, a router, a connection device, etc.

[0050] In some embodiments, the controller 250 may include at least one of a central processing unit, a video processor, an audio processor, a graphics processor, and a power processor, and a first interface to an nth interface for input / output. The controller 250 controls the operation of the display device and responds to the user's operation through various software control programs stored in the memory. The controller 250 controls the overall operation of the display device 200.

[0051] In some embodiments, the controller 250 and the tuner-demodulator 210 may be located in different separate devices, that is, the tuner-demodulator 210 may also be located in an external device of the main device where the controller 250 is located, such as an external set-top box.

[0052] In some embodiments, the user may input a user command through a graphical user interface (GUI) displayed on the display 260 , and the user input interface receives the user input command through the graphical user interface (GUI).

[0053] In some embodiments, the audio output device 270 may be a local speaker of the display device 200, or may be an external audio output device of the display device 200. In particular, for the external audio output device of the display device 200, the display device 200 may also be provided with an external audio output terminal, and the audio output device may be connected to the display device 200 through the external audio output terminal to output the sound of the display device 200.

[0054] In some embodiments, the user input interface 280 may be used to receive instructions from a user.

[0055] Figure 3 The hardware configuration block diagram of the control device in FIG1 is provided for some embodiments of the present application. Figure 3 As shown, the control device 100 may include: a controller 110, a communication interface 130, a user input / output interface, a memory, and a power supply.

[0056] The control device 100 is configured to control the display device 200 , and can receive user input operation instructions, and convert the operation instructions into instructions that the display device 200 can recognize and respond to, playing the role of an interactive intermediary between the user and the display device 200 .

[0057] In some embodiments, the control device 100 may be a smart device, for example, the control device 100 may be installed with various applications for controlling the display device 200 according to user needs.

[0058] In some embodiments, as shown in FIG. 1 , the mobile terminal 300 or other intelligent electronic device can perform similar functions as the control device 100 after installing an application for operating the display device 200 .

[0059] The controller 110 includes a processor 112, a RAM 113, a ROM 114, a communication interface 130, and a communication bus. The controller 110 is used to control the operation and operation of the control device 100, as well as the communication and cooperation between the internal components and the external and internal data processing functions.

[0060] The communication interface 130 implements communication of control signals and data signals with the display device 200 under the control of the controller 110. The communication interface 130 may include at least one of other near field communication modules such as a WiFi chip 131, a Bluetooth module 132, and an NFC module 133.

[0061] The user input / output interface 140 , wherein the input interface includes at least one of other input interfaces such as a microphone 141 , a touch panel 142 , a sensor 143 , and a button 144 .

[0062] In some embodiments, the control device 100 includes at least one of a communication interface 130 and an input / output interface 140. The control device 100 is configured with a communication interface 130, such as a WiFi, Bluetooth, NFC or other module, and can encode the user input command through the WiFi protocol, Bluetooth protocol, or NFC protocol and send it to the display device 200.

[0063] The memory 190 is used to store various operating programs, data and applications for driving and controlling the control device 100 under the control of the controller. The memory 190 can store various control signal instructions input by the user.

[0064] The power supply 180 is used to provide operating power support for each component of the control device 100 under the control of the controller.

[0065] In order to perform user interaction, in some embodiments, the display device 200 may run an operating system. The operating system is a computer program for managing and controlling hardware resources and software resources in the display device 200. The operating system may provide a user interface (control the display device), allow the user to interact with the display device 200, and support the running of various application programs.

[0066] It should be noted that the operating system may be a native operating system based on a specific operating platform, or a third-party operating system deeply customized based on a specific operating platform, or an independent operating system specially developed for the display device.

[0067] The operating system can be divided into different modules or layers according to the functions implemented, such as Figure 4 As shown, in some embodiments, the system is divided into four layers, from top to bottom, namely, the application layer (Applications layer) (referred to as "application layer"), the application framework layer (Application Framework layer) (referred to as "framework layer"), the system library layer and the kernel layer.

[0068] In some embodiments, the application layer is used to provide services and interfaces for applications so that the display device 200 can run applications and interact with users based on applications. At least one application can be run in the application layer, and these applications can be window programs, system settings programs, clock programs, etc. that come with the operating system; they can also be applications developed by third-party developers. In specific implementations, the application packages in the application layer are not limited to the above examples.

[0069] The framework layer provides application programming interfaces (APIs) and programming frameworks for applications. The application framework layer includes some predefined functions. The application framework layer is equivalent to a processing center that determines the actions of applications in the application layer. Applications can access system resources and obtain system services during execution through the API interface.

[0070] like Figure 4 As shown, in the embodiment of the present application, the application framework layer includes a view system, managers, content providers, etc., wherein the view system can design and implement the interface and interaction of the application, and the view system includes lists, grids, text boxes, buttons, etc. The manager includes at least one of the following modules: an activity manager is used to interact with all activities running in the system; a location manager is used to provide system services or applications with access to system location services; a package manager is used to retrieve various information related to the application package currently installed on the device; a notification manager is used to control the display and clearing of notification messages; and a window manager is used to manage icons, windows, toolbars, wallpapers, and desktop components on the user interface.

[0071] In some embodiments, the activity manager is used to manage the life cycle of each application and the usual navigation back function, such as controlling the exit, opening, and back of the application. The window manager is used to manage all window programs, such as obtaining the display screen size, determining whether there is a status bar, locking the screen, capturing the screen, and controlling the display window changes, for example, reducing the display window, shaking the display, distorting the display, etc.

[0072] In some embodiments, the system runtime layer can provide support for the framework layer. When the framework layer is used, the operating system will run the instruction library contained in the system runtime layer, such as the C / C++ instruction library, to implement the functions to be implemented by the framework layer.

[0073] In some embodiments, the kernel layer is a functional layer between the hardware and software of the display device 200. The kernel layer can implement functions such as hardware abstraction, multitasking, and memory management. Figure 4As shown, the kernel layer may be configured with hardware drivers, and the drivers included in the kernel layer may be at least one of the following drivers: audio driver, display driver, Bluetooth driver, camera driver, WIFI driver, USB driver, HDMI driver, sensor driver (such as fingerprint sensor, temperature sensor, pressure sensor, etc.), and power driver, etc.

[0074] It should be noted that the above example is only a simple division of the operating system functions and does not constitute a limitation on the specific operating system form of the display device 200 in the embodiment of the present application. Depending on factors such as the function of the display device and the type of operating system, the number of levels and specific level types contained in the operating system may be expressed in other forms.

[0075] In some embodiments, a display device is provided, comprising: a display configured to display content from a broadcast system or network and / or a user interface; one or more external device interfaces for communicating with one or more external devices according to a communication protocol, the communication protocol including at least a Bluetooth protocol; and at least one processor connected to the display and the one or more external device interfaces and configured to execute instructions to enable the display device to implement the following scheme, see Figure 5 As shown:

[0076] Step 501, in response to a voice query instruction, obtain a corresponding reply text, and generate audio data, text data and action control data based on the reply text.

[0077] Optionally, the voice query instruction is generated based on interactive voice input by the user to the display device.

[0078] Optionally, the user can speak a voice with specific text to wake up the voice interaction function of the display device. For example, the user can speak a voice, and the voice interaction function in the standby state converts the voice into text after receiving the voice. If the text is specific text, it switches to the awake state and plays the audio of the specific text. When the user hears the display device playing the audio of the specific text, it can be determined that the voice interaction function of the display device is awakened.

[0079] For example, the user can speak a voice, and after the voice interaction function in standby state receives the voice, it converts the voice into text. If the text is "Hello", it switches to the awakening state and broadcasts the audio of "Hello". When the user hears the display device broadcasting the audio, it can be determined that the voice interaction function of the display device is awakened. It should be noted that the above-mentioned specific text can be pre-defined, and the "Hello" here is only an example and does not constitute a limitation on the embodiments of the present application.

[0080] Optionally, the voice interaction function of the display device is an application.

[0081] Optionally, after the voice interaction function of the display device receives the voice of specific text spoken by the user, it switches to a wake-up state and uses the next voice spoken by the user as the interactive voice input by the user.

[0082] Optionally, the voice interaction function of the display device can convert the received interactive voice into text in real time and display the converted text on the display.

[0083] For example, assuming that the interactive voice input by the user is "Please help me check today's weather", the interface of the display may be as follows: Figure 6 As shown; it should be noted that: no matter which interface is displayed on the display, the user can speak the interactive voice. Figure 6 This is only an example of the home page and does not constitute a limitation on the embodiments of the present application.

[0084] Optionally, after converting the interactive voice into text, the voice interaction function of the display device can perform intent recognition on the text to obtain the user's query intent, generate a corresponding voice query instruction based on the query intent, and in response to the voice query instruction, the processor can call the corresponding microservice to obtain the corresponding reply text.

[0085] Exemplarily, the text corresponding to the interactive voice is "Please help me check today's weather". After intent recognition of the text, it can be determined that the user's query intention is to query today's weather, and a weather query instruction is generated. The weather query instruction carries today's date. In response to the weather query instruction, the weather query microservice is called to obtain the corresponding query result. Exemplarily, the query result can be "It is light rain today, 19~26℃, breeze is less than level 3, air quality: good, please bring rain gear when traveling to prevent rain from wetting clothes or affecting the itinerary". The query result can be used as a reply text.

[0086] Optionally, after obtaining the reply text, corresponding audio may be synthesized based on the reply text as audio data.

[0087] Optionally, a text-to-speech (TTS) system may be used to synthesize the reply text into audio data.

[0088] Optionally, after obtaining the reply text, the reply text can be directly used as text data; or, based on the reply text, a large model can be used to generate a corresponding action description text, and the action description text can be used as text data. The specific method depends on the source of the text samples used in the model training phase, which will be described in detail later.

[0089] Optionally, after obtaining the reply text, the action control data may be obtained by randomly generating the data.

[0090] Optionally, the motion control data includes left and right hand control data, hand height control data and footstep movement control data.

[0091] Optionally, the left and right hand control data can be an array of left and right hand control items, which includes two elements, one element is used to indicate whether the left hand performs an action, and for the convenience of explanation, this element is called the left hand identifier; the other element is used to indicate whether the right hand performs an action, and for the convenience of explanation, this element is called the right hand identifier; the positions of the left hand identifier and the right hand identifier in the left and right hand control item arrays are not limited, and the left hand identifier can be in front and the right hand identifier can be in the back, or the right hand identifier can be in front and the left hand identifier can be in the back; it needs to be consistent with the model training process.

[0092] Optionally, when the left hand identifier is the first preset value, it means that the left hand is to perform an action, and when the left hand identifier is the second preset value, it means that the left hand does not need to perform an action; similarly, when the right hand identifier is the first preset value, it means that the right hand is to perform an action, and when the right hand identifier is the second preset value, it means that the right hand does not need to perform an action.

[0093] Optionally, the first preset value may be 1, and the second preset value may be 0.

[0094] Exemplarily, in the left-hand control item array, the left-hand identifier is in front and the right-hand identifier is in the back. The left-hand control item array is, for example, [1, 0], indicating that the left hand needs to perform an action, but the right hand does not need to perform an action.

[0095] Optionally, the hand height control data can be a hand height control item array, which includes two elements, one element is the left hand relative height value, and the other element is the right hand relative height value. The relative height value refers to the relative value of the target height of the hand relative to the height of the hand when the arm is naturally hanging down. The positions of the left hand relative height value and the right hand relative height value in the hand height control item array are not limited. The left hand relative height value can be in front and the right hand relative height value can be in the back, or the right hand relative height value can be in front and the left hand relative height value can be in the back; it needs to be consistent with the model training process.

[0096] Optionally, the audio data is synthesized based on the reply text, and the action-driven model can divide the audio into audio frames, and process them one by one. In this way, for each audio frame, the action-driven model will output a frame of action-driven data, so that the virtual object displayed on the display can be connected with the audio frame, so that the action performed by the virtual object echoes the audio, and the body movements are more flexible. It should be noted that the text data and action control data corresponding to all audio frames are the same.

[0097] Optionally, the footstep activity control data may be a footstep activity control item array, which includes four elements, one element being the displacement difference of the left heel on the ground plane relative to the previous frame in the horizontal direction, one element being the displacement difference of the left heel on the ground plane relative to the previous frame in the vertical direction, one element being the displacement difference of the right heel on the ground plane relative to the previous frame in the horizontal direction, and one element being the displacement difference of the right heel on the ground plane relative to the previous frame in the vertical direction. The previous frame refers to the action drive data of the previous frame.

[0098] Optionally, the horizontal axis direction may be the direction facing the virtual object, and the vertical axis direction may be the right-hand side direction of the virtual object; or, the vertical axis direction may be the direction facing the virtual object, and the horizontal axis direction may be the right-hand side direction of the virtual object.

[0099] Step 502: Process the audio data, text data and action control data through the action drive model to obtain action drive data.

[0100] Among them, the action-driven model can be integrated on the display device side or deployed on the server side. When deployed on the server side, the display device can send audio data, text data and action control data to the server. The server processes the audio data, text data and action control data through the action-driven model to obtain action-driven data, and then returns the action-driven data to the display device.

[0101] Step 503, play the audio data and the target video, and control the display to display the reply text, the target video is a video generated by controlling the virtual object based on the action drive data.

[0102] Optionally, audio data, text data, and motion control data can be input into the motion drive model, and the motion drive model will output the motion drive data. After obtaining the motion drive data, the display device can, on the one hand, play the audio data and the target video, and on the other hand, control the display to display the above reply text. The target video is a video generated by controlling the virtual object based on the motion drive data. As described above, the motion drive model can process each audio frame in sequence to obtain multiple frames of motion drive data, each frame of motion drive data corresponds to an action, so that the action performed by the virtual object echoes the audio, and the body movements are more flexible.

[0103] For example, see Figure 7As shown, assuming that the interactive voice input by the user is "Please help me check today's weather", the reply text obtained by the method of the above embodiment is "It is light raining today, 19~26℃, breeze is less than level 3, air quality: good, please bring rain gear when traveling to prevent rain from wetting clothes or affecting the trip", the target video can be played in area 1 of the current display interface, and the reply text can be displayed in area 2. This display method is only an example, and other methods can also be used to display the target video and reply text. Figure 7 It does not constitute a limitation to the embodiments of the present application.

[0104] In some embodiments, the action-driven method provided in the embodiments of the present application can also be applied in other scenarios, such as live broadcast scenarios. Users can record the product explanation audio in advance, and the live broadcast device can use the product explanation audio as audio data; the live broadcast device can convert the product explanation audio into text, and use the text as text data; users can also set action control data, or the live broadcast device can randomly generate action control data. The live broadcast device can process the above audio data, text data and action control data through the action-driven model to obtain action-driven data; during the live broadcast process, the live broadcast device can play the above product explanation audio, and inject the action-driven data into the pre-set virtual anchor. The virtual anchor will make corresponding actions according to the action-driven data without the need for real people to appear on the screen. Moreover, the actions performed by the virtual anchor echo the above product explanation audio, and the body movements are flexible, which can improve the live broadcast effect.

[0105] It should be noted that in the above live broadcast scenario, using the text corresponding to the product explanation audio as text data is only one implementation method. You can also obtain action descriptive text based on the text corresponding to the product explanation audio and use the action descriptive text as text data. The specific method is related to the source of the text samples used in the model training stage, which will be described in detail later.

[0106] In the above embodiment, a display device is provided, including: a display configured to display content from a broadcast system or network and / or a user interface; at least one processor connected to the display and configured to execute instructions to enable the display device to: obtain a corresponding reply text in response to a voice query instruction, and generate audio data, text data, and action control data based on the reply text; process the audio data, text data, and action control data through an action-driven model to obtain action-driven data; play audio data and a target video, and control the display to display the reply text, wherein the target video is a video generated by controlling a virtual object based on the action-driven data. This multi-dimensional control method of voice, text, and action makes the body movements of virtual objects more flexible.

[0107] In some embodiments, the action drive model includes a rhythm alignment module, a semantic alignment module, a limb movement alignment module and a decoder; at least one processor executes processing of audio data, text data and action control data through the action drive model to obtain action drive data, and is further configured to: extract speech features from the audio data, input the extracted speech features into the rhythm alignment module, and align the speech features by the rhythm alignment module to obtain a rhythm action vector; extract text features from the text data, input both the extracted text features and speech features into the semantic alignment module, and fuse the text features and speech features by the semantic alignment module to obtain a speech-text fusion feature, and align the speech-text fusion feature to obtain a semantic action vector; input the action control data into the limb movement alignment module, and align the action control data by the limb movement alignment module to obtain a limb movement vector; superimpose the rhythm action vector, the semantic action vector and the limb movement vector and input them into the decoder, and decode them by the decoder to obtain the action drive data.

[0108] Optionally, the action-driven model includes a rhythm alignment module, a semantic alignment module, a body movement alignment module and a decoder. In addition, the action-driven model may also include a feature superposition module, the rhythm alignment module, the semantic alignment module, the body movement alignment module are all connected to the feature superposition module, and the feature superposition module is connected to the decoder.

[0109] Optionally, the action-driven model may further include a speech feature extraction module and a text feature extraction module. The speech feature extraction module is connected to the rhythm alignment module and the semantic alignment module respectively, and the text feature extraction module is connected to the semantic alignment module.

[0110] In a specific implementation, the audio data can be input into a speech feature extraction module, which extracts features from the audio data to obtain speech features. The speech feature extraction module further inputs the speech features into a rhythm alignment module, which aligns the speech features into rhythm action vectors and outputs the rhythm action vectors to a feature superposition module.

[0111] The text data can be input into the text feature extraction module, which extracts the characteristics of the text data to obtain text features, and the text feature extraction module further inputs the text features into the semantic alignment module. In addition, the speech feature extraction module inputs the speech features into the rhythm alignment module while also inputting the speech features into the semantic alignment module. The semantic alignment module fuses the received text features and speech features to obtain speech-text fusion features, and aligns the speech-text fusion features to obtain a semantic action vector; the semantic alignment module outputs the semantic action vector to the feature superposition module.

[0112] The motion control data may be input to a limb movement alignment module, which aligns the motion control data into a limb movement vector and outputs the limb movement vector to a feature superposition module.

[0113] Among them, the feature superposition module superimposes the received rhythm action vector, semantic action vector and limb activity vector, inputs the superimposed features into the decoder, and the decoder outputs the action driving data.

[0114] In the above embodiment, the action drive model includes an alignment module for processing data of different dimensions, specifically a rhythm alignment module, a semantic alignment module and a body movement alignment module, the rhythm alignment module is used to align speech features into rhythm action vectors, the semantic alignment module is used to align speech text fusion features into semantic action vectors, and the body movement alignment module is used to align action control data into body movement vectors. The action drive model also includes a decoder, which can decode the superimposed features of the rhythm action vector, the semantic action vector and the body movement vector into action drive data. The body movements of the virtual object are controlled in combination with multiple dimensional data, making the body movements of the virtual object more flexible.

[0115] In some embodiments, the motion control data includes left and right hand control data; the limb movement alignment module includes: a left and right hand alignment module; at least one processor executes inputting the motion control data into the limb movement alignment module, and the limb movement alignment module performs alignment processing on the motion control data to obtain a limb movement vector, and is further configured to: inputting the left and right hand control data into the left and right hand alignment module, and the left and right hand alignment module performs alignment processing on the left and right hand control data to obtain left and right hand vectors, and the limb movement vector includes the left and right hand vectors.

[0116] Among them, the action control data includes left and right hand control data; the limb activity alignment module includes: left and right hand alignment module; the left and right hand control data can be input into the left and right hand alignment module, the left and right hand alignment module aligns the control data of the left and right hand control dimensions into left and right hand vectors, and the left and right hand alignment module outputs the left and right hand vectors to the feature superposition module. The feature superposition module superimposes the received rhythm action vector, semantic action vector and left and right hand vectors, and inputs the superimposed features into the decoder, and the decoder outputs the action drive data.

[0117] In the above embodiment, the action control data includes left and right hand control data, and the user can input control data of whether the left and right hands are to perform actions, as left and right hand control data, the left and right hand alignment module can align the left and right hand control data into left and right hand vectors, and the decoder decodes the left and right hand vectors to obtain action drive data. The action drive data is generated in combination with the left and right hand control data, which allows the user to control whether the left and right hands of the virtual object perform actions, and the body control is more refined.

[0118] In some embodiments, the motion control data includes hand height control data; the limb movement alignment module includes: a hand height alignment module; at least one processor executes inputting the motion control data into the limb movement alignment module, and the limb movement alignment module performs alignment processing on the motion control data to obtain a limb movement vector, and is further configured to: inputting the hand height control data into the hand height alignment module, and the hand height alignment module performs alignment processing on the hand height control data to obtain a hand height vector, and the limb movement vector includes the hand height vector.

[0119] Among them, the action control data includes hand height control data; the limb activity alignment module includes: a hand height alignment module; the hand height control data can be input into the hand height alignment module, the hand height alignment module can align the hand height control data into a hand height vector, and the hand height alignment module outputs the hand height vector to the feature superposition module. The feature superposition module superimposes the received rhythm action vector, semantic action vector, left and right hand vectors, and hand height vector, and inputs the superimposed features into the decoder, and the decoder outputs the action drive data.

[0120] In the above embodiment, the action control data includes hand height control data, and the user can input the relative height value of the left hand and the relative height value of the right hand as the hand height control data. The hand height alignment module can align the hand height control data into a hand height vector, and the decoder decodes the hand height vector to obtain the action driving data. The action driving data is generated in combination with the hand height control data. This method allows the user to control the height of the left and right hands of the virtual object, and the limb control is more refined.

[0121] In some embodiments, the motion control data includes step activity control data; the limb activity alignment module includes: a step activity alignment module; at least one processor executes inputting the motion control data into the limb activity alignment module, and the limb activity alignment module performs alignment processing on the motion control data to obtain a limb activity vector, and is further configured to: inputting the step activity control data into the step activity alignment module, and the step activity alignment module performs alignment processing on the step activity control data to obtain a step activity vector, and the limb activity vector includes the step activity vector.

[0122] Among them, the action control data includes footstep activity control data; the limb activity alignment module includes: a footstep activity alignment module; the footstep activity control data can be input into the footstep activity alignment module, the footstep activity alignment module can align the footstep activity control data into a footstep activity vector, and the footstep activity alignment module outputs the footstep activity vector to the feature superposition module. The feature superposition module superimposes the received rhythm action vector, semantic action vector, left and right hand vectors, hand height vector and footstep activity vector, and inputs the superimposed features into the decoder, and the decoder outputs the action driving data.

[0123] In the above embodiment, the motion control data includes footstep activity control data, and the user can input the displacement difference of the left heel on the ground plane relative to the previous frame in the horizontal direction, the displacement difference of the left heel on the ground plane relative to the previous frame in the vertical direction, the displacement difference of the right heel on the ground plane relative to the previous frame in the horizontal direction, and the displacement difference of the right heel on the ground plane relative to the previous frame in the vertical direction as footstep activity control data. The footstep activity alignment module can align the footstep activity control data into a footstep activity vector, and the decoder decodes the footstep activity vector to obtain the motion driving data. The motion driving data is generated in combination with the footstep activity control data. This method allows the user to control the footstep activity of the virtual object, and the body control is more refined.

[0124] In some embodiments, when the semantic alignment module is trained based on the converted text of voice samples, the text data is obtained by converting audio data into text; when the semantic alignment module is trained based on action descriptive text, the text data is the action descriptive text corresponding to the audio data.

[0125] Optionally, when the semantic alignment module is trained based on the converted text of voice samples, the audio data can be converted into text to obtain text data. The detailed process of processing the text data by the semantic alignment module can be found in the aforementioned embodiment and will not be repeated here.

[0126] Optionally, when the semantic alignment module is trained based on the action descriptive text, the audio data is converted into text, and based on the converted text, the corresponding action descriptive text is generated using the large model, and the action descriptive text is used as text data. The detailed process of processing text data by the semantic alignment module is shown in the above embodiment, which will not be repeated here.

[0127] That is to say, the embodiment of the present application provides two action-driven models, and the difference between the two action-driven models lies in the different sources of samples used in training the semantic alignment module.

[0128] In the above embodiment, when the semantic alignment module is trained based on the converted text of the voice sample, the audio data is converted to text to obtain text data; when the semantic alignment module is trained based on the action descriptive text, the audio data is converted to text, and based on the converted text, the corresponding action descriptive text is generated using the large model, and the action descriptive text is used as text data. Different training methods can obtain different action-driven models. Before the display device leaves the factory, a suitable model can be configured for the display device based on actual conditions.

[0129] See also Figure 8 As shown in Figure 1, the action-driven model includes a speech feature extraction module, a text feature extraction module, a rhythm alignment module, a semantic alignment module, a left-right hand alignment module, a hand height alignment module, a footstep activity alignment module, a feature superposition module and a decoder. The connection relationship between each module can be seen in Figure 1. Figure 8 , the working process of each module includes: S1, audio data can be input into the speech feature extraction module; S2, text data can be input into the text feature extraction module; S3, left and right hand control item arrays can be input into the left and right hand alignment module; S4, hand height control item arrays can be input into the hand height alignment module; S5, footstep activity control item arrays can be input into the footstep activity alignment module; S6, the speech feature extraction module extracts speech features from the audio data to obtain speech features, and inputs the speech features into the rhythm alignment module and the semantic alignment module; S7, the text feature extraction module extracts text features from the text data to obtain text features, and inputs the text features into the semantic alignment module; S8, the rhythm alignment module aligns the speech features into rhythm action vectors, and outputs the rhythm action vectors to the feature superposition module; S9 , the semantic alignment module fuses the text features and the speech features to obtain the speech-text fusion features, aligns the speech-text fusion features into semantic action vectors, and outputs the semantic action vectors to the feature superposition module; S10, the left and right hand alignment module aligns the left and right hand control item arrays into left and right hand vectors, and outputs the left and right hand vectors to the feature superposition module; S11, the hand height alignment module aligns the hand height control item array into the hand height vector, and outputs the hand height vector to the feature superposition module; S12, the footstep activity alignment module aligns the footstep activity control item array into the footstep activity vector, and outputs the footstep activity vector to the feature superposition module; S13, the feature superposition module superimposes the received features, and outputs the superimposed features to the decoder, S14, the decoder decodes the superimposed features to obtain action driving data.

[0130] In some embodiments, at least one processor is configured to execute instructions to enable the display device to: train a pre-built action encoder and decoder to obtain a trained action encoder and decoder; train a pre-built action alignment module through the trained action encoder to obtain a trained action alignment module; and construct an action-driven model based on the trained action alignment module and the trained decoder.

[0131] Optionally, the pre-built action encoder may include a rhythm encoder, a semantic encoder, and a body movement encoder. The body movement encoder may include a left and right hand encoder, a hand height encoder, and a foot movement encoder.

[0132] Optionally, the pre-built action alignment module may include a rhythm alignment module, a semantic alignment module and a limb activity alignment module, and the limb activity alignment module may include a left and right hand alignment module, a hand height alignment module and a footstep activity alignment module.

[0133] Optionally, first train the rhythm encoder, semantic encoder, left and right hand encoder, hand height encoder, foot activity encoder and decoder; then train the rhythm alignment module, semantic alignment module, left and right hand alignment module, hand height alignment module and footstep activity alignment module through the trained rhythm encoder, semantic encoder, left and right hand encoder, hand height encoder and footstep activity encoder to obtain the trained rhythm alignment module, semantic alignment module, left and right hand alignment module, hand height alignment module and footstep activity alignment module. Based on the trained rhythm alignment module, semantic alignment module, left and right hand alignment module, hand height alignment module and footstep activity alignment module and the trained decoder, construct the action drive model.

[0134] In the above embodiment, a method for constructing an action-driven model is provided, wherein the pre-constructed action encoder and decoder are first trained to obtain the trained action encoder and decoder; the pre-constructed action alignment module is trained through the trained action encoder to obtain the trained action alignment module; and the action-driven model is constructed based on the trained action alignment module and the trained decoder. The action-driven model supports the user to input control data of multiple control dimensions, such as voice control dimension, text control dimension, and local limb control dimension, thereby realizing refined control of the limb movements of the virtual object, and the limb movements of the virtual object are also more flexible.

[0135] In some embodiments, see Fig. 9 As shown, the embodiment of the present application also provides a method for training an action-driven model, the method comprising the following steps:

[0136] Step 901: train the pre-built motion encoder and decoder to obtain trained motion encoder and decoder.

[0137] Optional, see Fig.10 As shown, the pre-built action encoder may include a rhythm encoder, a semantic encoder, and a body activity encoder. The body activity encoder may include a left and right hand encoder, a hand height encoder, and a foot activity encoder.

[0138] Optionally, the rhythm encoder is connected to the feature superposition module, and the feature superposition module is connected to the decoder, and the rhythm encoder and the decoder are trained using action sample data that is compatible with the rhythm encoder; after the training is completed, the semantic encoder is also connected to the feature superposition module on the basis of the above connection relationship, and the semantic encoder and the decoder are trained using action sample data that is compatible with the semantic encoder while freezing the parameters of the rhythm encoder; after the training is completed, the left and right hand encoders are also connected to the feature superposition module on the basis of the above connection relationship, and the left and right hand encoders and the decoder are trained using action sample data that is compatible with the left and right hand encoders while freezing the parameters of the rhythm encoder and the semantic encoder; after the training is completed, On the basis of the connection relationship, the hand height encoder is also connected to the feature superposition module, and the parameters of the rhythm encoder, semantic encoder, left and right hand encoders are frozen, and the action sample data adapted to the hand height encoder is used to train the hand height encoder and decoder; after the training is completed, on the basis of the above connection relationship, the foot activity encoder is also connected to the feature superposition module, and the parameters of the rhythm encoder, semantic encoder, left and right hand encoders, and hand height encoder are frozen, and the action sample data adapted to the foot activity encoder is used to train the foot activity encoder and decoder; thereby obtaining the trained rhythm encoder, semantic encoder, left and right hand encoder, hand height encoder, foot activity encoder and decoder.

[0139] Step 902: Train the pre-built action alignment module through the trained action encoder to obtain a trained action alignment module.

[0140] Optionally, the trained action encoder includes a rhythm encoder, a semantic encoder, a left and right hand encoder, a hand height encoder, and a foot activity encoder.

[0141] Optionally, the pre-built action alignment module includes a rhythm alignment module, a semantic alignment module and a limb activity alignment module, and the limb activity alignment module may include a left and right hand alignment module, a hand height alignment module and a footstep activity alignment module.

[0142] Optionally, the rhythm alignment module can be trained using a trained rhythm encoder to obtain a trained rhythm alignment module; the semantic alignment module can be trained using a trained semantic encoder to obtain a trained semantic alignment module; the left and right hand encoders can be trained using a trained left and right hand alignment module to obtain a trained left and right hand alignment module; the hand height alignment module can be trained using a trained hand height encoder to obtain a trained hand height alignment module; the foot activity encoder can be trained using a trained foot activity alignment module to obtain a trained foot activity alignment module.

[0143] Step 903: construct an action-driven model based on the trained action alignment module and the trained decoder.

[0144] Optionally, the trained rhythm alignment module, semantic alignment module, left and right hand alignment module, hand height alignment module and footstep activity alignment module can be connected to the feature superposition module, and the feature superposition module can be connected to the trained decoder to obtain the action-driven model.

[0145] In the above embodiment, the pre-built action encoder and decoder are trained to obtain the trained action encoder and decoder; the pre-built action alignment module is trained through the trained action encoder to obtain the trained action alignment module; the action drive model is constructed based on the trained action alignment module and the trained decoder. The action drive model supports the user to input control data of multiple control dimensions, such as voice control dimension, text control dimension and local limb control dimension, etc., to achieve refined control of the virtual object's limb movements, and the virtual object's limb movements are also more flexible.

[0146] In some embodiments, a pre-constructed action encoder includes: a rhythm encoder, a semantic encoder and a limb movement encoder; training the pre-constructed action encoder and decoder to obtain a trained action encoder and decoder includes: using random action data containing rhythm information to train the rhythm encoder and decoder to obtain a trained rhythm encoder and a rhythm-optimized decoder; based on the trained rhythm encoder and the rhythm-optimized decoder, using random action data and semantic action data containing semantic information to train the semantic encoder and the rhythm-optimized decoder to obtain a trained semantic encoder and a semantic-optimized decoder; based on the trained rhythm encoder, the trained semantic encoder and the semantic-optimized decoder, using random action data, semantic action data and local action data to train the limb movement encoder and the semantic-optimized decoder to obtain a trained limb movement encoder and a local action-optimized decoder.

[0147] Optionally, the rhythm encoder is connected to the feature superposition module, and the feature superposition module is connected to the decoder. Random action data containing rhythm information is obtained as action sample data adapted to the rhythm encoder. Since the random actions during speech are rich in a large amount of rhythm information, mainly including action timing, action speed, action frequency, etc., these rhythm information are the most basic information of action data and are consistent with the speech rhythm during speech. Therefore, a large amount of random action data during speech can be used as random action data containing rhythm information.

[0148] Optionally, there are many ways to obtain random motion data, which can be obtained by motion capture methods or extracted from videos containing human movements using open source models.

[0149] Optionally, when a person is speaking, each movement will generate a set of random motion data, and a set of random motion data contains information such as the postures of key points of the body. The information contained in a set of random motion data conforms to a certain format, which is not limited in this embodiment of the present application.

[0150] The following uses a set of action sample data as an example to introduce the training process of the rhythm encoder and decoder;

[0151] The action sample data is input into the rhythm encoder, the rhythm encoder encodes the action sample data to obtain rhythm encoding information, the rhythm encoding information is input into the feature superposition module, the feature superposition module outputs the rhythm encoding information to the decoder, the decoder decodes the encoding information to obtain decoded information, the loss is calculated based on the decoded information and the action sample data, and the parameters of the rhythm encoder and decoder are adjusted based on the loss until the loss meets the preset conditions, thereby obtaining a trained rhythm encoder and a rhythm-optimized decoder.

[0152] Optionally, after obtaining the trained rhythm encoder and the rhythm-optimized decoder, the semantic encoder is also connected to the feature superposition module based on the connection relationship during the training of the rhythm encoder, and the parameters of the rhythm encoder are frozen. On the basis of the random action data, semantic action data containing semantic information is added as action sample data adapted to the semantic encoder. Semantic action data containing semantic information, for example, include action data corresponding to greetings, thumbs-up, OK gestures, and cheering gestures. In addition, changes in hand shape in random actions also fall into the category of semantic actions, such as single index finger emphasis during key semantics, finger index actions of "first, second, third", etc.; and changes in actions caused by different emotions also fall into the category of semantic actions, such as actions such as opening the palm or clenching a fist under different emotions. The action data corresponding to these behaviors can all be used as semantic action data.

[0153] Similarly, there are many ways to obtain semantic action data. It can be obtained by motion capture methods, or it can be extracted from videos containing human actions using open source models.

[0154] Similarly, a set of semantic action data contains information such as the posture of each key point of the body. The information contained in a set of semantic action data conforms to a certain format, which is not limited in the embodiments of the present application.

[0155] The following uses a set of action sample data as an example to introduce the training process of the semantic encoder and the decoder after rhythm optimization;

[0156] The action sample data is input into the semantic encoder and the rhythm encoder, the rhythm encoder encodes the action sample data to obtain rhythm encoding information, and the rhythm encoding information is input into the feature superposition module; the semantic encoder encodes the action sample data to obtain semantic encoding information, and the semantic encoding information is input into the feature superposition module; the feature superposition module superimposes the rhythm encoding information and the semantic encoding information, and inputs the superimposed information into the rhythm-optimized decoder, the rhythm-optimized decoder decodes the received information to obtain decoding information, calculates the loss based on the decoding information and the action sample data, and adjusts the parameters of the semantic encoder and the rhythm-optimized decoder based on the loss until the loss meets the preset conditions, thereby obtaining the trained semantic encoder and the semantic-optimized decoder.

[0157] Optionally, after obtaining the trained semantic encoder and the semantically optimized decoder, the limb activity encoder is also connected to the feature superposition module based on the connection relationship when training the semantic encoder, and the parameters of the rhythm encoder and the semantic encoder are frozen. On the basis of the random action data and semantic action data, local action data is added as action sample data adapted to the limb activity encoder.

[0158] Similarly, there are many ways to obtain local motion data. It can be obtained by motion capture method, or it can be extracted from videos containing human movements using open source models.

[0159] Similarly, a set of local motion data contains information such as the posture of each key point of the body. The information contained in a set of local motion data conforms to a certain format, which is not limited in this embodiment of the present application.

[0160] The following uses a set of action sample data as an example to introduce the training process of the body movement encoder and the semantically optimized decoder;

[0161] The action sample data is input into the semantic encoder, the rhythm encoder and the limb activity encoder. The rhythm encoder encodes the action sample data to obtain rhythm encoding information, and the rhythm encoding information is input into the feature superposition module; the semantic encoder encodes the action sample data to obtain semantic encoding information, and the semantic encoding information is input into the feature superposition module; the limb activity encoder encodes the action sample data to obtain limb activity encoding information, and the limb activity encoding information is input into the feature superposition module; the feature superposition module superimposes the rhythm encoding information, the semantic encoding information and the limb activity encoding information, and inputs the superimposed information into the semantically optimized decoder. The semantically optimized decoder decodes the received information to obtain decoding information, calculates the loss based on the decoding information and the action sample data, and adjusts the parameters of the limb activity encoder and the semantically optimized decoder based on the loss until the loss meets the preset conditions, thereby obtaining a trained limb activity encoder and a local action optimized decoder.

[0162] In the above embodiment, the action codec is designed to include a rhythm encoder, a semantic encoder, and a body movement encoder, which decouples the feature information of different dimensions of the action and lays a foundation for more refined action driving.

[0163] In some embodiments, the limb activity encoder includes: left and right hand encoders, hand height encoder and foot activity encoder; based on the trained rhythm encoder, the trained semantic encoder and the semantically optimized decoder, the limb activity encoder and the semantically optimized decoder are trained using random motion data, semantic motion data and local motion data to obtain the trained limb activity encoder and the local motion optimized decoder, including: based on the trained rhythm encoder, the trained semantic encoder and the semantically optimized decoder, the left and right hand encoders and the semantically optimized decoder are trained using random motion data, semantic motion data and left and right hand motion data to obtain the trained left and right hand encoders and the left and right hand motion optimized decoder; based on the trained rhythm encoder, the trained semantic encoder and the semantically optimized decoder, the left and right hand encoders and the semantically optimized decoder are trained using random motion data, semantic motion data and left and right hand motion data to obtain the trained left and right hand encoders and the left and right hand motion optimized decoder. The semantic encoder, the trained left and right hand encoders, and the decoder after left and right hand movement optimization, use random action data, semantic action data, left and right hand action data, and different hand height action data to train the hand height encoder and the decoder after left and right hand action optimization, so as to obtain the trained hand height encoder and the decoder after hand height optimization; based on the trained rhythm encoder, the trained semantic encoder, the trained left and right hand encoder, the trained hand height encoder, and the decoder after hand height optimization, use random action data, semantic action data, left and right hand action data, different hand height action data, and foot activity action data to train the foot activity encoder and the decoder after hand height optimization, so as to obtain the trained foot activity encoder and the decoder after foot activity optimization.

[0164] Optionally, after obtaining the trained semantic encoder and the semantically optimized decoder, the left and right hand encoders are also connected to the feature superposition module based on the connection relationship during the training of the semantic encoder, and the parameters of the rhythm encoder and the semantic encoder are frozen. On the basis of the random action data and the semantic action data, the left and right hand action data are added as action sample data adapted to the left and right hand encoders. The left and right hand action data include action data of only the left hand, only the right hand, and both hands.

[0165] Similarly, there are many ways to obtain left and right hand motion data. It can be obtained by motion capture method, or it can be extracted from human motion videos using open source models.

[0166] Similarly, a set of left and right hand motion data contains information such as the postures of key points of the body. The information contained in a set of left and right hand motion data conforms to a certain format, which is not limited in the embodiments of the present application.

[0167] The following uses a set of action sample data as an example to introduce the training process of the left and right hand encoders and the semantically optimized decoder;

[0168] The action sample data is input into the semantic encoder, the rhythm encoder and the left-right hand encoder. The rhythm encoder encodes the action sample data to obtain rhythm encoding information, and inputs the rhythm encoding information into the feature superposition module; the semantic encoder encodes the action sample data to obtain semantic encoding information, and inputs the semantic encoding information into the feature superposition module; the left-right hand encoder encodes the action sample data to obtain left-right hand encoding information, and inputs the left-right hand encoding information into the feature superposition module; the feature superposition module superimposes the rhythm encoding information, the semantic encoding information and the left-right hand encoding information, and inputs the superimposed information into the semantically optimized decoder. The semantically optimized decoder decodes the received information to obtain decoding information, calculates the loss based on the decoding information and the action sample data, and adjusts the parameters of the left-right hand encoder and the semantically optimized decoder based on the loss until the loss meets the preset conditions, thereby obtaining the trained left-right hand encoder and the left-right hand action optimized decoder.

[0169] Optionally, after obtaining the trained left and right hand encoders and the decoders after optimizing the left and right hand movements, the hand height encoder is also connected to the feature superposition module based on the connection relationship when training the left and right hand encoders, and the parameters of the rhythm encoder, semantic encoder, and left and right hand encoders are frozen. On the basis of random movement data, semantic movement data, and left and right hand movement data, different hand height movement data are added as movement sample data adapted to the hand height encoder. The different hand height movement data include the movement data corresponding to the hand at the hip, waist, shoulder, head, and above the head.

[0170] Similarly, there are many ways to obtain high-level hand motion data. It can be obtained using motion capture methods, or it can be extracted from videos containing human movements using open source models.

[0171] Similarly, a set of hand height motion data contains information such as the posture of each key point of the body. The information contained in a set of hand height motion data conforms to a certain format, which is not limited in the embodiments of the present application.

[0172] The following uses a set of action sample data as an example to introduce the training process of the hand height encoder and the decoder optimized for left and right hand actions;

[0173] The action sample data is input into the semantic encoder, the rhythm encoder, the left and right hand encoders and the hand height encoder. The rhythm encoder encodes the action sample data to obtain the rhythm encoding information, and the rhythm encoding information is input into the feature superposition module; the semantic encoder encodes the action sample data to obtain the semantic encoding information, and the semantic encoding information is input into the feature superposition module; the left and right hand encoders encode the action sample data to obtain the left and right hand encoding information, and the left and right hand encoding information is input into the feature superposition module; the hand height encoder encodes the action sample data to obtain the hand height encoding information, and the hand height encoding information is input into the feature superposition module; the feature superposition module superimposes the rhythm encoding information, the semantic encoding information, the left and right hand encoding information and the hand height encoding information, and inputs the superimposed information into the semantically optimized decoder. The semantically optimized decoder decodes the received information to obtain the decoded information, calculates the loss based on the decoded information and the action sample data, and adjusts the parameters of the hand height encoder and the decoder after the left and right hand action optimization based on the loss until the loss meets the preset conditions, thereby obtaining the trained hand height encoder and the hand height optimized decoder.

[0174] Optionally, after obtaining the trained hand height encoder and the decoder after hand height optimization, the foot activity encoder is also connected to the feature superposition module based on the connection relationship when training the hand height encoder, and the parameters of the rhythm encoder, semantic encoder, left and right hand encoder, and hand height encoder are frozen. On the basis of random action data, semantic action data, left and right hand action data, and different hand height action data, foot activity action data is added as action sample data adapted to the foot activity encoder. The foot activity action data includes action data corresponding to the dynamic left and right separation of the foot, dynamic backward step of the foot, dynamic forward step of the foot, and other behaviors.

[0175] Similarly, there are many ways to obtain footstep activity motion data. It can be obtained by motion capture method, or it can be extracted from videos containing human movements using open source models.

[0176] Similarly, a set of footstep activity data contains information such as the postures of key points of the body. The information contained in a set of footstep activity data conforms to a certain format, which is not limited in this embodiment of the present application.

[0177] The following uses a set of action sample data as an example to introduce the training process of the foot activity encoder and the hand highly optimized decoder;

[0178] The action sample data is input into the semantic encoder, rhythm encoder, left and right hand encoder, hand height encoder and foot activity encoder. The rhythm encoder encodes the action sample data to obtain rhythm encoding information, and inputs the rhythm encoding information into the feature superposition module; the semantic encoder encodes the action sample data to obtain semantic encoding information, and inputs the semantic encoding information into the feature superposition module; the left and right hand encoders encode the action sample data to obtain left and right hand encoding information, and inputs the left and right hand encoding information into the feature superposition module; the hand height encoder encodes the action sample data to obtain hand height encoding information, and inputs the hand height encoding information into the feature superposition module; the foot activity encoder The action sample data is encoded to obtain footstep activity encoding information, and the footstep activity encoding information is input into the feature superposition module; the feature superposition module superimposes the rhythm encoding information, semantic encoding information, left and right hand encoding information, hand height encoding information and footstep activity encoding information, and inputs the superimposed information into the hand height optimized decoder, the hand height optimized decoder decodes the received information to obtain decoded information, calculates the loss based on the decoded information and the action sample data, and adjusts the parameters of the foot activity encoder and the hand height optimized decoder based on the loss until the loss meets the preset conditions, thereby obtaining the trained foot activity encoder and the foot activity optimized decoder.

[0179] In the above embodiment, the limb activity encoder is designed to include: left and right hand encoders, hand height encoder and foot activity encoder; the feature information of different dimensions of the action is decoupled, laying the foundation for more refined action driving.

[0180] In some embodiments, the trained action encoder includes a rhythm encoder; the pre-built action alignment module includes: a rhythm alignment module; the pre-built action alignment module is trained through the trained action encoder to obtain a trained action alignment module, including: obtaining rhythm alignment data samples, the rhythm alignment data samples include action data samples and speech samples corresponding to the action data samples; performing speech feature extraction on the speech samples, and inputting the extracted speech feature samples into the rhythm alignment module; inputting the action data samples into the rhythm encoder to obtain rhythm vector samples; performing alignment processing on the speech feature samples through the rhythm alignment module to obtain speech alignment vector samples, calculating the model loss based on the speech alignment vector samples and the rhythm vector samples, and adjusting the parameters of the rhythm alignment module based on the model loss.

[0181] The rhythm alignment data samples include motion data samples and speech samples corresponding to the motion data samples. The motion data samples here can be any set of motion sample data used when training the motion encoder.

[0182] Optional, see Fig.11 As shown, the speech feature extraction module can be used to extract speech features from the speech sample, and the extracted speech feature samples are input into the rhythm alignment module; the action data samples are input into the trained rhythm encoder, the rhythm encoder encodes the action data samples to obtain rhythm vector samples, the rhythm alignment module performs alignment processing on the speech feature samples to obtain speech alignment vector samples, the model loss is calculated based on the speech alignment vector samples and the rhythm vector samples, and the parameters of the rhythm alignment module are adjusted based on the model loss, so that the speech alignment vector samples obtained by the alignment processing and the rhythm vector samples output by the rhythm encoder become more and more similar, until the model loss meets the preset conditions.

[0183] Optionally, the speech feature extraction module can be integrated into the action-driven model, or can be used as a preprocessing module of the action-driven model. The speech feature extraction module can use a known pre-trained model, such as wenent, wav2vect, etc.

[0184] In the above embodiment, the training process of the rhythm alignment module is introduced. The trained rhythm alignment module can be used to process audio data, laying a foundation for multimodal control.

[0185] In some embodiments, the trained action encoder includes a semantic encoder; the pre-built action alignment module includes: a semantic alignment module; the semantic alignment module can be obtained based on the converted text training of the speech sample, and the pre-built action alignment module is trained through the trained action encoder to obtain the trained action alignment module, including: obtaining semantic alignment data samples, the semantic alignment data samples include action data samples and speech samples and text samples corresponding to the action data samples, and the text samples are the converted texts of the speech samples; performing speech feature extraction on the speech samples, and inputting the extracted speech feature samples into the semantic alignment module; performing text feature extraction on the text samples, and inputting the extracted text feature samples into the semantic alignment module; inputting the action data samples into the semantic encoder to obtain semantic vector samples; fusing the speech feature samples and the text feature samples through the semantic alignment module to obtain fused feature samples; aligning the fused feature samples to obtain fused vector samples, calculating the model loss based on the fused vector samples and the semantic vector samples, and adjusting the parameters of the semantic alignment module based on the model loss.

[0186] The semantic alignment data samples include action data samples and speech samples and text samples corresponding to the action data samples. The action data samples here can be any set of action sample data used when training the action encoder. The text samples are the converted texts of the speech samples, and the text samples here can be obtained through the TTS system.

[0187] Optional, see Fig.12As shown, after the speech feature extraction module extracts speech features from the speech sample, the extracted speech feature samples are input into the rhythm alignment module and the semantic alignment module at the same time; the text feature extraction module extracts text features from the text sample and the extracted text feature samples are input into the semantic alignment module; the action data samples are input into the trained semantic encoder, the semantic encoder encodes the action data samples to obtain semantic vector samples, the semantic alignment module fuses the speech feature samples and the text feature samples to obtain fused feature samples; the fused feature samples are aligned to obtain fused vector samples, the model loss is calculated based on the fused vector samples and the semantic vector samples, and the parameters of the semantic alignment module are adjusted based on the model loss, so that the fused vector samples obtained by the alignment processing and the semantic vector samples output by the semantic encoder become more and more similar until the model loss meets the preset conditions.

[0188] Optionally, the text feature extraction module can be integrated into the action-driven model, or can be used as a preprocessing module of the action-driven model. The text feature extraction module can use a known pre-trained model, such as wav2vect or other models.

[0189] In the above embodiment, a training process of using speech-to-text to train a semantic alignment module is introduced. The trained semantic alignment module can be used to process audio data and text data, laying a foundation for multimodal control.

[0190] In some embodiments, the trained action encoder includes a semantic encoder; the pre-built action alignment module includes: a semantic alignment module; the semantic alignment module can be obtained based on action descriptive text training, and the pre-built action alignment module is trained through the trained action encoder to obtain a trained action alignment module, including: obtaining semantic alignment data samples, the semantic alignment data samples include action data samples and speech samples and text samples corresponding to the action data samples, and the text samples are action descriptive texts; performing speech feature extraction on the speech samples, and inputting the extracted speech feature samples into the semantic alignment module; performing text feature extraction on the text samples, and inputting the extracted text feature samples into the semantic alignment module; inputting the action data samples into the semantic encoder to obtain semantic vector samples; fusing the speech feature samples and the text feature samples through the semantic alignment module to obtain fused feature samples; aligning the fused feature samples to obtain fused vector samples, calculating the model loss based on the fused vector samples and the semantic vector samples, and adjusting the parameters of the semantic alignment module based on the model loss.

[0191] The semantic alignment data samples include action data samples and speech samples and text samples corresponding to the action data samples. The action data samples here can be any set of action sample data used when training the action encoder. The text samples are action descriptive texts, which can be obtained through a multimodal large model or manually annotated in advance.

[0192] For example, examples of action descriptive text may be “OK gesture”, “hands raised high”, “both hands raised in front of the body”, etc.

[0193] Similarly, see Fig.13 As shown, after the speech feature extraction module extracts speech features from the speech sample, the extracted speech feature samples are input into the rhythm alignment module and the semantic alignment module at the same time; the text feature extraction module extracts text features from the text sample (action descriptive text), and the extracted text feature samples are input into the semantic alignment module; the action data sample is input into the trained semantic encoder, the semantic encoder encodes the action data sample to obtain a semantic vector sample, and the semantic alignment module fuses the speech feature sample and the text feature sample to obtain a fused feature sample; the fused feature sample is aligned to obtain a fused vector sample, the model loss is calculated based on the fused vector sample and the semantic vector sample, and the parameters of the semantic alignment module are adjusted based on the model loss, so that the fused vector sample obtained by the alignment process and the semantic vector sample output by the semantic encoder become more and more similar until the model loss meets the preset conditions.

[0194] In the above embodiment, a training process of using action descriptive text to train a semantic alignment module is introduced. The trained semantic alignment module can be used to process audio data and text data, laying a foundation for multimodal control.

[0195] In some embodiments, the trained motion encoder includes a limb movement encoder; the pre-built motion alignment module includes: a limb movement alignment module; the pre-built motion alignment module is trained through the trained motion encoder to obtain a trained motion alignment module, including: obtaining limb movement data samples, the limb movement data samples include motion data samples and limb movement identifiers corresponding to the motion data samples, the limb movement identifiers are used to characterize the limb movement in the motion data samples; the motion data samples are input into the limb movement encoder to obtain limb movement vector samples; the limb movement identifiers are input into the limb movement alignment module; the limb movement identifiers are aligned through the limb movement alignment module to obtain limb alignment vector samples, the model loss is calculated based on the limb alignment vector samples and the limb movement vector samples, and the parameters of the limb movement alignment module are adjusted based on the model loss.

[0196] The limb activity data samples include motion data samples and limb activity identifiers corresponding to the motion data samples. The motion data samples here can be any set of motion sample data used when training the motion encoder.

[0197] Optionally, the motion data samples are input into a limb activity encoder, which encodes the motion data samples to obtain limb activity vector samples; the limb activity identifiers are input into a limb activity alignment module; the limb activity alignment module performs alignment processing on the limb activity identifiers to obtain limb alignment vector samples, and the model loss is calculated based on the limb alignment vector samples and the limb activity vector samples, and the parameters of the limb activity alignment module are adjusted based on the model loss. This makes the limb alignment vector samples obtained by the alignment processing and the limb activity vector samples output by the semantic encoder more and more similar until the model loss meets the preset conditions.

[0198] In the above embodiment, the training process of the limb movement alignment module is introduced. The trained limb movement alignment module can be used to process motion control data, laying a foundation for multi-modal control.

[0199] In some embodiments, the limb movement encoder includes left and right hand encoders; the limb movement alignment module includes a left and right hand alignment module; the limb movement identifier includes a left and right hand identifier, and the left and right hand identifiers are indication information of the hands that have movements in the action data sample; the limb movement vector sample includes a left and right hand vector sample; and the limb alignment vector sample includes a left and right hand alignment vector.

[0200] Optionally, left and right hand data samples may be obtained, where the left and right hand data samples include action data samples and left and right hand identifiers corresponding to the action data samples.

[0201] The left and right hand labels corresponding to the action data samples can be manually identified or automatically identified, and the result of the identification is in the form of an array [left_label, right_label]. The automatic identification method is as follows: record the left and right arms in the natural hanging state, including the data of the left and right hands, as Arm_left_natural and Arm_right_natural, respectively, and then compare the current left and right arm and hand data Arm_left_data and Arm_right_data with the difference in the natural vertical state frame by frame, Arm_left_diff = abs(Arm_left_data - Arm_left_natural), Arm_right_diff = abs(Arm_right_data - Arm_right_natural), where abs() refers to the absolute value operation. If only the difference Arm_left_diff is greater than the threshold T_arm_diff, it is considered that only the left hand is currently moving, and the left and right hands are marked as the array [1, 0]; if only the difference Arm_right_diff is greater than the threshold T_arm_diff, it is considered that only the right hand is currently moving, and the left and right hands are marked as the array [0, 1]; if both the difference Arm_left_diff and Arm_right_diff are greater than the threshold T_arm_diff, it is considered that the current movement is a two-hand movement, and the left and right hands are marked as the array [1, 1]; if both the difference Arm_left_diff and Arm_right_diff are less than or equal to the threshold T_arm_diff, it is considered that there is no movement at present, and the left and right hands are marked as the array [0, 0].

[0202] Optional, see Fig.14 As shown, the motion data samples can be input into the left and right hand encoders, which encode the motion data samples to obtain left and right hand vector samples; the left and right hand identifiers are input into the left and right hand alignment module, which aligns the left and right hand identifiers to obtain left and right hand alignment vectors; the model loss is calculated based on the left and right hand alignment vectors and the left and right hand vector samples, and the parameters of the left and right hand alignment module are adjusted based on the model loss. The left and right hand alignment vectors obtained by the alignment process and the left and right hand vector samples output by the left and right hand encoders are more and more similar until the model loss meets the preset conditions.

[0203] In the above embodiment, the training process of the left-hand and right-hand alignment modules is introduced. The trained left-hand and right-hand alignment modules can be used to process control data of left-hand and right-hand control dimensions, laying a foundation for multi-modal control.

[0204] In some embodiments, the limb movement encoder includes a hand height encoder; the limb movement alignment module includes a hand height alignment module; the limb movement identifier includes a hand height identifier, and the hand height identifier represents the relative height value of the left and right hands in the motion data sample; the limb movement vector sample includes a hand height vector sample; and the limb alignment vector sample includes a hand height alignment vector.

[0205] Optionally, a hand height data sample may be obtained, where the hand height data sample includes a motion data sample and a hand height identifier corresponding to the motion data sample.

[0206] The hand height identifier corresponding to the action data sample can be obtained in the following way:

[0207] First, mark the hand height H_low when the arm is naturally hanging down and the hand height H_high when the arm is stretched upward in advance, and then calculate the relative height value of the hand in the action data sample. The calculation formula is H_relative = (H – H_low) / (H_high – H_low), where H is the hand height value in the action data sample. H_relative needs to be calculated for the left and right hands respectively, and the array [H_relative_left, H_relative_right] is used as the hand height identifier. The positions of the left and right hand relative height values ​​can be interchanged, but once the order is determined, the order should be kept unchanged in training and subsequent applications.

[0208] Optional, see Fig.15 As shown, the motion data sample can be input into the hand height encoder, which encodes the motion data sample to obtain the hand height vector sample; the hand height identifier is input into the hand height alignment module, which aligns the hand height identifier to obtain the hand height alignment vector; the model loss is calculated based on the hand height alignment vector and the hand height vector sample, and the parameters of the hand height alignment module are adjusted based on the model loss. The hand height alignment vector obtained by the alignment process and the hand height vector sample output by the hand height encoder are more and more similar until the model loss meets the preset conditions.

[0209] In the above embodiment, the training process of the hand height alignment module is introduced. The trained hand height alignment module can be used to process control data of the hand height control dimension, laying a foundation for multimodal control.

[0210] In some embodiments, the limb movement encoder includes a footstep movement encoder; the limb movement alignment module includes a footstep movement alignment module; the limb movement identifier includes a footstep movement identifier, the footstep movement identifier represents the displacement difference of the left and right feet on the ground plane of the action data sample relative to the previous frame; the limb movement vector sample includes a footstep movement vector sample; the limb alignment vector sample includes a footstep movement alignment vector.

[0211] Optionally, a footstep activity data sample may be obtained, where the footstep activity data sample includes a motion data sample and a footstep activity identifier corresponding to the motion data sample.

[0212] The footstep activity identifier corresponding to the motion data sample can be obtained in the following way:

[0213] Calculate the displacement difference between the left heel position and the right heel position on the ground plane between the current action data sample and the previous frame action data sample, marked as array FOOT_DIFF = [left_foot_diff_x, left_foot_diff_y, right_foot_diff_x, right_foot_diff_y]. The FOOT_DIFF of the first frame data sample is [0.0, 0.0, 0.0,0.0]. Use this array as the footstep activity identifier.

[0214] The positions of the four elements in the array can be interchanged, but once the order is determined, it should remain unchanged during training and subsequent applications.

[0215] Optional, see Fig.16 As shown, the motion data sample can be input into the footstep activity encoder, which encodes the motion data sample to obtain the footstep activity vector sample; the footstep activity identifier is input into the footstep activity alignment module, which aligns the footstep activity identifier to obtain the footstep activity alignment vector; the model loss is calculated based on the footstep activity alignment vector and the footstep activity vector sample, and the parameters of the footstep activity alignment module are adjusted based on the model loss. The footstep activity alignment vector obtained by the alignment process and the footstep activity vector sample output by the footstep activity encoder are more and more similar until the model loss meets the preset conditions.

[0216] In the above embodiment, the training process of the step activity alignment module is introduced. The trained step activity alignment module can be used to process the control data of the step activity control dimension, thus laying a foundation for multi-modal control.

[0217] In some embodiments, a display device is provided, and the display device and a server cooperate to implement an action-driven method, wherein the display device comprises: a display configured to display content from a broadcast system or a network and / or a user interface; and at least one processor connected to the display and configured to execute instructions to enable the display device to implement the steps on the display device side in the above-mentioned action-driven method. Fig.17 As shown, the above action driving method includes:

[0218] Step 1701: The display device obtains the corresponding reply text in response to the voice query instruction, and generates audio data, text data, and action control data based on the reply text.

[0219] Step 1702: The display device sends the audio data, text data, and motion control data to the server.

[0220] Step 1703: The server processes the audio data, text data, and action control data through an action drive model to obtain action drive data.

[0221] Step 1704: The server returns the action-driven data to the display device.

[0222] Step 1705: the display device plays the audio data and the target video, and controls the display to display the reply text, where the target video is a video generated by controlling the virtual object based on the action drive data.

[0223] The specific implementation of each step can refer to the relevant content in the aforementioned embodiments, and this application will not go into details here.

[0224] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0225] In one embodiment, a computer device is further provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above method embodiments when executing the computer program.

[0226] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0227] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0228] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., but are not limited to this.

[0229] The technical features of the above embodiments can be combined arbitrarily. In order to make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of the present application. The above-mentioned embodiments only express several implementation methods of the present application. The description is relatively specific and detailed, but it cannot be understood as a limitation on the scope of the patent of the present application. It should be pointed out that for ordinary technicians in this field, several variations and improvements can be made without departing from the concept of the present application, which all belong to the scope of protection of the present application. Therefore, the scope of protection of the present application shall be based on the attached claims.

[0230] For the convenience of explanation, the above description has been made in conjunction with specific embodiments. However, the above exemplary discussion is not intended to be exhaustive or limit the embodiments to the specific forms disclosed above. Based on the above teachings, various modifications and variations can be obtained. The selection and description of the above embodiments are to better explain the principles and practical applications, so that those skilled in the art can better use the embodiments and various different variations of the embodiments suitable for specific use considerations.

Claims

1. A display device, characterized in that: include: a display configured to display content from a broadcast system or network and / or a user interface; and at least one processor connected to the display and configured to execute instructions to cause the display device to: In response to the voice query instruction, obtaining a corresponding reply text, and generating audio data, text data and action control data based on the reply text; Processing the audio data, the text data and the action control data through an action drive model to obtain action drive data; The audio data and the target video are played, and the display is controlled to display the reply text, wherein the target video is a video generated by controlling a virtual object based on the action driving data.

2. The display device according to claim 1, characterized in that The action-driven model includes a rhythm alignment module, a semantic alignment module, a body movement alignment module and a decoder; The at least one processor processes the audio data, the text data, and the action control data through an action drive model to obtain action drive data, and is further configured to: Extracting speech features from the audio data, inputting the extracted speech features into the rhythm alignment module, and allowing the rhythm alignment module to align the speech features to obtain a rhythm motion vector; Extracting text features from the text data, inputting the extracted text features and the speech features into the semantic alignment module, fusing the text features and the speech features by the semantic alignment module to obtain speech-text fusion features, and aligning the speech-text fusion features to obtain semantic action vectors; Inputting the motion control data into the limb activity alignment module, and the limb activity alignment module performs alignment processing on the motion control data to obtain a limb activity vector; The rhythm action vector, the semantic action vector and the limb movement vector are superimposed and input into the decoder, and the action driving data is obtained by decoding by the decoder.

3. The display device according to claim 2, characterized in that The motion control data includes left and right hand control data; the limb activity alignment module includes: a left and right hand alignment module; The at least one processor executes inputting the motion control data into the limb activity alignment module, and the limb activity alignment module performs alignment processing on the motion control data to obtain a limb activity vector, and is further configured to: The left and right hand control data are input into the left and right hand alignment module, and the left and right hand alignment module performs alignment processing on the left and right hand control data to obtain left and right hand vectors, and the limb activity vector includes the left and right hand vectors.

4. The display device according to claim 2, characterized in that The motion control data includes hand height control data; the limb activity alignment module includes: a hand height alignment module; The at least one processor executes inputting the motion control data into the limb activity alignment module, and the limb activity alignment module performs alignment processing on the motion control data to obtain a limb activity vector, and is further configured to: The hand height control data is input into the hand height alignment module, and the hand height alignment module performs alignment processing on the hand height control data to obtain a hand height vector, and the limb movement vector includes the hand height vector.

5. The display device according to claim 2, characterized in that The motion control data includes footstep activity control data; The limb activity alignment module includes: a footstep activity alignment module; The at least one processor executes inputting the motion control data into the limb activity alignment module, and the limb activity alignment module performs alignment processing on the motion control data to obtain a limb activity vector, and is further configured to: The step activity control data is input into the step activity alignment module, and the step activity alignment module performs alignment processing on the step activity control data to obtain a step activity vector, and the limb activity vector includes the step activity vector.

6. The display device according to claim 2, characterized in that In the case where the semantic alignment module is trained based on the converted text of the speech sample, the text data is obtained by converting the audio data into text; In the case where the semantic alignment module is trained based on action descriptive text, the text data is the action descriptive text corresponding to the audio data.

7. The display device according to claim 1, characterized in that The at least one processor is configured to execute instructions to cause the display device to: Training the pre-built action encoder and decoder to obtain a trained action encoder and decoder; The pre-built action alignment module is trained by using the trained action encoder to obtain a trained action alignment module; Based on the trained action alignment module and the trained decoder, an action-driven model is constructed.

8. A method for training an action-driven model, characterized in that: include: Training the pre-built action encoder and decoder to obtain a trained action encoder and decoder; The pre-built action alignment module is trained by using the trained action encoder to obtain a trained action alignment module; Based on the trained action alignment module and the trained decoder, an action-driven model is constructed.

9. The method according to claim 7, characterized in that: The pre-built action encoder includes: a rhythm encoder, a semantic encoder and a body movement encoder; The step of training the pre-built motion encoder and decoder to obtain the trained motion encoder and decoder includes: Using random motion data containing rhythm information to train the rhythm encoder and the decoder to obtain a trained rhythm encoder and a rhythm-optimized decoder; Based on the trained rhythm encoder and the rhythm-optimized decoder, the random action data and the semantic action data containing semantic information are used to train the semantic encoder and the rhythm-optimized decoder to obtain the trained semantic encoder and the semantically optimized decoder; Based on the trained rhythm encoder, the trained semantic encoder and the semantically optimized decoder, the random action data, the semantic action data and the local action data are used to train the limb movement encoder and the semantically optimized decoder to obtain a trained limb movement encoder and a local action optimized decoder.

10. The method according to claim 8, characterized in that The limb activity encoder includes: left and right hand encoders, a hand height encoder and a foot activity encoder; The method comprises: training the body movement encoder and the semantically optimized decoder based on the trained rhythm encoder, the trained semantic encoder and the semantically optimized decoder by using the random action data, the semantic action data and the local action data to obtain the trained body movement encoder and the semantically optimized decoder, including: Based on the trained rhythm encoder, the trained semantic encoder and the semantically optimized decoder, the random action data, the semantic action data and the left and right hand action data are used to train the left and right hand encoders and the semantically optimized decoder to obtain trained left and right hand encoders and left and right hand action optimized decoders; Based on the trained rhythm encoder, the trained semantic encoder, the trained left and right hand encoders, and the decoder after the left and right hand movements are optimized, the hand height encoder and the decoder after the left and right hand movements are trained using the random movement data, the semantic movement data, the left and right hand movement data, and the different hand height movement data, to obtain the trained hand height encoder and the decoder after the hand height optimization; Based on the trained rhythm encoder, the trained semantic encoder, the trained left and right hand encoders, the trained hand height encoder and the hand height optimized decoder, the foot activity encoder and the hand height optimized decoder are trained using the random motion data, the semantic motion data, the left and right hand motion data, different hand height motion data and foot activity motion data to obtain a trained foot activity encoder and a foot activity optimized decoder.

11. The method according to claim 7, characterized in that: The trained motion encoder includes a rhythm encoder; the pre-built motion alignment module includes: a rhythm alignment module; The pre-built action alignment module is trained by the trained action encoder to obtain the trained action alignment module, including: Acquire a rhythm-aligned data sample, wherein the rhythm-aligned data sample includes a motion data sample and a speech sample corresponding to the motion data sample; Extracting speech features from the speech sample, and inputting the extracted speech feature sample into the rhythm alignment module; Inputting the motion data sample into the rhythm encoder to obtain a rhythm vector sample; Performing alignment processing on the speech feature samples by the rhythm alignment module to obtain speech alignment vector samples; A model loss is calculated based on the speech alignment vector samples and the rhythm vector samples, and parameters of the rhythm alignment module are adjusted based on the model loss.

12. The method according to claim 7, characterized in that: The trained action encoder includes a semantic encoder; the pre-built action alignment module includes: a semantic alignment module; The pre-built action alignment module is trained by the trained action encoder to obtain the trained action alignment module, including: Acquire a semantically aligned data sample, wherein the semantically aligned data sample includes an action data sample and a voice sample and a text sample corresponding to the action data sample, wherein the text sample is a converted text of the voice sample; Extracting speech features from the speech sample, and inputting the extracted speech feature sample into the semantic alignment module; Extracting text features from the text sample, and inputting the extracted text feature sample into the semantic alignment module; Inputting the action data sample into the semantic encoder to obtain a semantic vector sample; The speech feature sample and the text feature sample are fused by the semantic alignment module to obtain a fused feature sample; the fused feature sample is aligned to obtain a fused vector sample; A model loss is calculated based on the fusion vector samples and the semantic vector samples, and parameters of the semantic alignment module are adjusted based on the model loss.

13. The method according to claim 7, characterized in that: The trained action encoder includes a semantic encoder; the pre-built action alignment module includes: a semantic alignment module; The pre-built action alignment module is trained by the trained action encoder to obtain the trained action alignment module, including: Acquire a semantically aligned data sample, wherein the semantically aligned data sample includes an action data sample and a voice sample and a text sample corresponding to the action data sample, wherein the text sample is an action descriptive text; Extracting speech features from the speech sample, and inputting the extracted speech feature sample into the semantic alignment module; Extracting text features from the text sample, and inputting the extracted text feature sample into the semantic alignment module; Inputting the action data sample into the semantic encoder to obtain a semantic vector sample; The speech feature sample and the text feature sample are fused by the semantic alignment module to obtain a fused feature sample; the fused feature sample is aligned to obtain a fused vector sample; A model loss is calculated based on the fusion vector samples and the semantic vector samples, and parameters of the semantic alignment module are adjusted based on the model loss.

14. The method according to claim 7, characterized in that: The trained motion encoder includes a limb activity encoder; the pre-built motion alignment module includes: a limb activity alignment module; The pre-built action alignment module is trained by the trained action encoder to obtain the trained action alignment module, including: Acquire a limb activity data sample, wherein the limb activity data sample includes a motion data sample and a limb activity identifier corresponding to the motion data sample, wherein the limb activity identifier is used to characterize a limb activity situation in the motion data sample; Inputting the motion data sample into the limb movement encoder to obtain a limb movement vector sample; inputting the limb movement identifier into the limb movement alignment module; Performing alignment processing on the limb movement identifiers by the limb movement alignment module to obtain limb alignment vector samples; A model loss is calculated based on the limb alignment vector samples and the limb movement vector samples, and parameters of the limb movement alignment module are adjusted based on the model loss.

15. The method according to claim 14, characterized in that The limb movement encoder includes left and right hand encoders; the limb movement alignment module includes a left and right hand alignment module; the limb movement identifier includes a left and right hand identifier, and the left and right hand identifiers are indication information of the hands that have actions in the action data samples; the limb movement vector samples include left and right hand vector samples; the limb alignment vector samples include left and right hand alignment vectors.

16. The method according to claim 14, characterized in that The limb movement encoder includes a hand height encoder; the limb movement alignment module includes a hand height alignment module; the limb movement identifier includes a hand height identifier, and the hand height identifier represents the relative height value of the left and right hands in the motion data sample; the limb movement vector sample includes a hand height vector sample; and the limb alignment vector sample includes a hand height alignment vector.

17. The method according to claim 14, characterized in that The limb movement encoder includes a footstep movement encoder; the limb movement alignment module includes a footstep movement alignment module; the limb movement identifier includes a footstep movement identifier, and the footstep movement identifier represents the displacement difference of the left and right feet on the ground plane of the action data sample relative to the previous frame; the limb movement vector sample includes a footstep movement vector sample; the limb alignment vector sample includes a footstep movement alignment vector.

18. A display device, characterized in that: include: a display configured to display content from a broadcast system or network and / or a user interface; and at least one processor connected to the display and configured to execute instructions to cause the display device to: In response to the voice query instruction, obtaining a corresponding reply text, and generating audio data, text data and action control data based on the reply text; Sending the audio data, the text data, and the action control data to a server to instruct the server to process the audio data, the text data, and the action control data through an action-driven model to obtain action-driven data; Receiving the action driving data returned by the server; The audio data and the target video are played, and the display is controlled to display the reply text, wherein the target video is a video generated by controlling a virtual object based on the action driving data.

19. An action driving method, characterized in that: Applied to a server, the method comprises: Receiving audio data, text data, and action control data sent by a display device; the audio data, the text data, and the action control data are generated by the display device based on the reply text after obtaining the corresponding reply text in response to the voice query instruction; Processing the audio data, the text data and the action control data through an action drive model to obtain action drive data; The action driving data is returned to the display device to instruct the display device to play the audio data and the target video, and control the display to display the reply text, wherein the target video is a video generated by controlling the virtual object based on the action driving data.

20. A display device, comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 8 to 17 are implemented.

21. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 8 to 17 are implemented.

22. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 8 to 17 are implemented.

Images