A data processing method and apparatus

Abstract

A data processing method applied to action generation based on voice or text, the method comprising: obtaining voice data; determining a plurality of segmentation point positions from the voice data according to audio features of the voice data, the segmentation point positions corresponding to predicted rhythm points of body movements of a character object when the voice data is uttered; obtaining a feature representation through a feature extraction network according to the voice data and information indicating the plurality of segmentation point positions; and generating action data through an action generation network according to the feature representation. The application can make the generated action have accurate rhythm, and thus make the action more consistent with the action of the character object when the voice is actually uttered.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of artificial intelligence, and in particular to a data processing method and device thereof. BACKGROUND

[0002] In public speeches, body language can increase the rhythm of the speech, making the speech more vivid and more persuasive. Studies have shown that body language plays an important role in communication. Body movements can more accurately express intentions and convey emotions, complementing the information conveyed by speech; secondly, body movements can help users focus more on the content of communication with digital humans; body movements can improve the persuasiveness, credibility and realism of digital humans; body movements can reflect the intentions and personalities of the speaker. Lack of body language or rigid body movements in communication can lead to the uncanny valley effect.

[0003] People expect virtual digital humans (or simply digital humans) to behave more like humans. Virtual digital humans can make rhythmic and emphatic movements in conjunction with speech, just like humans. For some special speech semantics, there can be specific body movements, for example, when people say "good" or "ok", they tend to make an OK gesture, and when they say "first" or "firstly", they tend to pause slightly and make a 1 hand shape.

[0004] Therefore, how to generate corresponding gestures and other body movements according to speech is a problem that needs to be solved urgently. SUMMARY

[0005] The present application provides a data processing method, which can make the generated movements have accurate rhythm, and thus make the movements more suitable for the movements of a character object when actually speaking.

[0006] In a first aspect, the present application provides a data processing method, the method comprising: obtaining speech data; determining, from the speech data, a plurality of segmentation point positions corresponding to predicted rhythm points of body movements made by a character object when speaking the speech data, according to audio features of the speech data; obtaining a feature representation by a feature extraction network, according to the speech data and information indicating the plurality of segmentation point positions; and generating movement data by a movement generation network, according to the feature representation.

[0007] In a possible implementation, the movement data can be joint rotation angles of body movements or 3D point cloud information of movements, which is not limited here.

[0008] In a possible implementation, the action data of a character object uttering the voice data can be predicted according to the voice data. In this case, the user makes certain actions when uttering (for example, when making a speech), and the actions have certain rhythm points, for example, the rhythm points can be the critical points from stillness to movement or from movement to stillness when the body actions are made. If the rhythm information can be explicitly identified before feature extraction and used as the input of the model, the subsequent generated actions can have accurate rhythm, and thus the actions are more consistent with the actions of the character object when actually uttering the voice.

[0009] In a possible implementation, the audio features are related to volume and / or pitch.

[0010] In a possible implementation, the plurality of split point positions include a first position and a second position adjacent to each other, and a time interval between the first position and the second position is within a preset range. The first position and the second position are determined in the following manner: the first position and the second position are onset points in the voice data; the first position is an onset point in the voice data, the second position is not an onset point in the voice data, and the second position is a time point at which the volume of the voice data is greater than a threshold.

[0011] In a possible implementation, the action rhythm points usually appear on the onsets (audio features related to volume) of the voice, but not all onsets are rhythm points of the user actions. The interval between adjacent rhythm points of the voice is within a range, and the range is related to the speaker style. Assuming that the range is [t1, t2], the time interval less than t1 is usually caused by noise, fillers, or unsmooth pronunciation. The time interval greater than t2 is usually corresponding to a pause or a quiet phase of not speaking. Based on these observations, the onsets with a time interval between adjacent onsets within the preset range can be determined as rhythm points, and thus the rhythm points of the body actions made by the user when uttering the voice data can be identified, and the voice data can be segmented based on the rhythm points.

[0012] In a possible implementation, a heuristic strategy can be used to identify the voice rhythm. The voice is segmented, and the time interval less than t1 is not identified as a rhythm point, and a pseudo rhythm point can be inserted for the time interval greater than t2. The pseudo rhythm point can be inserted in the following manner: the time interval from the previous rhythm point is greater than t1, and the volume of the voice is greater than a threshold I a The threshold can be set as the average volume of the environmental noise, so that the time interval between adjacent split point positions is within the preset range. If all the volumes in the interval are less than I aThe pseudo-rhythm points are evenly inserted to make time intervals of positions of adjacent segmentation points within a preset range.

[0013] In a possible implementation, the voice data and the information indicating the plurality of segmentation point positions include: a plurality of voice segments into which the voice data is divided according to the plurality of segmentation point positions; or characters of the voice data and the plurality of segmentation point positions.

[0014] In a possible implementation, the feature extraction network includes a plurality of serial network layers, and the feature representation includes feature representations output by the plurality of network layers. The method further includes: determining an action category of the body action according to feature representations output by one or more network layers close to an output layer in the plurality of network layers; and generating, by the action generation network, the action data according to the feature representation, including: generating, by the action generation network, the action data according to part or all of the feature representations output by the plurality of network layers and the action category.

[0015] In a possible implementation, in addition to inputting the feature representation obtained by the feature extraction network to the action generation network, a type of the action can also be identified by a neural network according to the feature representation (for example, high-level features in the feature representation), and the action type and the feature representation are input to the action generation network.

[0016] Since the action category itself carries semantics, the feature input to the action generation network and the subsequent generated body action have semantic relevance.

[0017] In a possible implementation, the feature extraction network includes a plurality of serial network layers, and the feature representation includes feature representations output by the plurality of network layers. The method further includes: obtaining, by an encoder, a style code of the body action according to feature representations output by one or more network layers away from an output layer in the plurality of network layers; and generating, by the action generation network, the action data according to the feature representation, including: generating, by the action generation network, the action data according to part or all of the feature representations output by the plurality of network layers and the style code.

[0018] In a possible implementation, the style code is uniformly distributed.

[0019] In a second aspect, the present application provides a data processing method, the method comprising:

[0020] obtaining voice data of a character object and first action data, the first action data being real action data of a body action made by the character object when the voice data is emitted;

[0021] determine, from the audio features of the speech data, a plurality of split point positions corresponding to predicted rhythm points of body movements made by a character object when uttering the speech data;

[0022] obtain, from the speech data and the information indicative of the plurality of split point positions, a feature representation by a feature extraction network;

[0023] generate, from the feature representation, second movement data by a movement generation network; and a difference between the first movement data and the second movement data is used to update the feature extraction network and the movement generation network.

[0024] In a possible implementation, the feature extraction network comprises a plurality of serially connected network layers, and the feature representation comprises feature representations output by the plurality of network layers; and the generating, from the feature representation, second movement data by a movement generation network comprises:

[0025] generate, from the feature representation and an initialized style code, second movement data by a movement generation network;

[0026] obtain, from feature representations output by one or more network layers of the plurality of network layers that are away from an output layer, a first style code of the body movements by an encoder;

[0027] update the initialized style code by a preset loss function to obtain an updated style code;

[0028] a difference between the first style code and the updated style code is used to update the encoder.

[0029] In a possible implementation, the audio features are related to volume and / or pitch.

[0030] In a possible implementation, the plurality of split point positions comprises a first position and a second position adjacent to each other, and a time interval between the second position and the first position is within a preset range; and the first position and the second position are determined in a manner that:

[0031] the first position and the second position are onset points in the speech data;

[0032] the first position is an onset point in the speech data, the second position is not an onset point in the speech data, and the second position is a time point in the speech data at which volume is greater than a threshold.

[0033] In a possible implementation, the speech data and the information indicative of the plurality of split point positions comprise:

[0034] divide the speech data into a plurality of speech segments according to the plurality of split point positions; or

[0035] the speech data and characters indicative of the plurality of split point positions.

[0036] In a possible implementation, the feature extraction network comprises a plurality of serially connected network layers, the feature representation comprises feature representations output by the plurality of network layers, and the method further comprises:

[0037] determining, according to feature representations output by one or more network layers close to an output layer in the plurality of network layers, a motion category of the body motion;

[0038] generating, according to the feature representations, second motion data by a motion generation network, comprises:

[0039] generating, according to part or all of the feature representations output by the plurality of network layers and the motion category, second motion data by a motion generation network.

[0040] In a possible implementation, the feature extraction network comprises a plurality of serially connected network layers, the feature representation comprises feature representations output by the plurality of network layers, and the method further comprises:

[0041] obtaining, according to feature representations output by one or more network layers away from an output layer in the plurality of network layers, a style code of the body motion by an encoder;

[0042] generating, according to the feature representations, second motion data by a motion generation network, comprises:

[0043] generating, according to part or all of the feature representations output by the plurality of network layers and the style code, second motion data by a motion generation network.

[0044] In a third aspect, the present application provides a data processing apparatus, comprising:

[0045] an acquisition module configured to acquire speech data;

[0046] a processing module configured to determine, according to an audio feature of the speech data, a plurality of split point positions from the speech data, the split point positions corresponding to predicted rhythm points of a body motion made by a character object when the speech data is uttered;

[0047] obtaining, according to the speech data and information indicative of the plurality of split point positions, a feature representation by a feature extraction network;

[0048] generating, according to the feature representation, motion data by a motion generation network.

[0049] In a possible implementation, the audio feature is related to volume and / or pitch.

[0050] In a possible implementation, the plurality of split point positions comprises a first position and a second position adjacent to the first position, and a time interval between the second position and the first position is within a preset range; and the first position and the second position are determined in the following manner:

[0051] The first position and the second position are onset points in the speech data.

[0052] The first position is an onset point in the speech data, the second position is not an onset point in the speech data, and the second position is a time point at which volume in the speech data is greater than a threshold.

[0053] In a possible implementation, the speech data and information indicating the plurality of split point positions comprise:

[0054] a plurality of speech segments into which the speech data is divided according to the plurality of split point positions; or

[0055] characters in the speech data and information indicating the plurality of split point positions.

[0056] In a possible implementation, the feature extraction network comprises a plurality of serially connected network layers, and the feature representation comprises feature representations output by the plurality of network layers; and the processing module is further configured to:

[0057] determine, according to feature representations output by one or more network layers close to an output layer in the plurality of network layers, a motion category of the body motion;

[0058] The processing module is specifically configured to:

[0059] generate, according to part or all of the feature representations output by the plurality of network layers and the motion category, motion data by using a motion generation network.

[0060] In a possible implementation, the feature extraction network comprises a plurality of serially connected network layers, and the feature representation comprises feature representations output by the plurality of network layers; and the processing module is further configured to:

[0061] obtain, according to feature representations output by one or more network layers far away from an output layer in the plurality of network layers, a style code of the body motion by using an encoder.

[0062] The processing module is specifically configured to:

[0063] The action data is generated by an action generation network according to part or all of feature representations output by the plurality of network layers and the style code.

[0064] In a possible implementation, the distribution of the style code is a uniform distribution.

[0065] In a fourth aspect, the present application provides a data processing apparatus, the apparatus comprising:

[0066] An obtaining module is configured to obtain voice data of a character object and first action data, the first action data being real action data of a body action made by the character object when the voice data is uttered;

[0067] A processing module is configured to determine a plurality of split point positions from the voice data according to audio features of the voice data, the split point positions corresponding to predicted rhythm points of the body action made by the character object when the voice data is uttered.

[0068] According to the voice data and information indicating the plurality of split point positions, a feature extraction network is used to obtain a feature representation.

[0069] According to the feature representation, an action generation network is used to generate second action data, and a difference between the first action data and the second action data is used to update the feature extraction network and the action generation network.

[0070] In a possible implementation, the feature extraction network comprises a plurality of serially connected network layers, and the feature representation comprises feature representations output by the plurality of network layers.

[0071] According to the feature representation and an initialized style code, an action generation network is used to generate second action data.

[0072] According to feature representations output by one or more network layers of the plurality of network layers that are far away from an output layer, an encoder is used to obtain a first style code of the body action.

[0073] The initialized style code is updated by a preset loss function to obtain an updated style code.

[0074] A difference between the first style code and the updated style code is used to update the encoder.

[0075] In a possible implementation, the audio features are related to volume and / or pitch.

[0076] In a possible implementation, the plurality of split point positions includes a first position and a second position adjacent to the first position, and a time interval between the second position and the first position is within a preset range; and the first position and the second position are determined in the following manner:

[0077] The first position and the second position are onset points in the speech data.

[0078] The first position is an onset point in the speech data, the second position is not an onset point in the speech data, and the second position is a time point at which a volume of the speech data is greater than a threshold.

[0079] In a possible implementation, the speech data and the information indicating the plurality of split point positions include:

[0080] A plurality of speech segments into which the speech data is divided according to the plurality of split point positions; or

[0081] The speech data and characters indicating the plurality of split point positions.

[0082] In a possible implementation, the feature extraction network includes a plurality of serially connected network layers, and the feature representation includes feature representations output by the plurality of network layers; and the processing module is further configured to:

[0083] determine, according to feature representations output by one or more network layers close to an output layer in the plurality of network layers, a motion category of the body motion;

[0084] The processing module is specifically configured to:

[0085] generate, according to part or all of the feature representations output by the plurality of network layers and the motion category, second motion data by using a motion generation network.

[0086] In a possible implementation, the feature extraction network includes a plurality of serially connected network layers, and the feature representation includes feature representations output by the plurality of network layers; and the processing module is further configured to:

[0087] obtain, according to feature representations output by one or more network layers far from an output layer in the plurality of network layers, a style code of the body motion by using an encoder;

[0088] The processing module is specifically configured to:

[0089] generate, according to part or all of the feature representations output by the plurality of network layers and the style code, second motion data by using a motion generation network.

[0090] In a fifth aspect, an embodiment of the present application provides a data processing apparatus, which can include a memory, a processor and a bus system, wherein the memory is configured to store a program, and the processor is configured to execute the program in the memory to perform the method of the first aspect and any possible implementation thereof, or the method of the second aspect and any possible implementation thereof.

[0091] In a sixth aspect, an embodiment of the present application provides a computer readable storage medium, which stores a computer program, and when the computer program is run on a computer, the computer is enabled to perform the method of the first aspect and any possible implementation thereof, or the method of the second aspect and any possible implementation thereof.

[0092] In a seventh aspect, an embodiment of the present application provides a computer program, which, when run on a computer, enables the computer to perform the method of the first aspect and any possible implementation thereof, or the method of the second aspect and any possible implementation thereof.

[0093] In an eighth aspect, the present application provides a chip system, which includes a processor configured to support a function of performing a device or a training device, for example, transmitting or processing data or information involved in the above method. In a possible design, the chip system further includes a memory configured to store necessary program instructions and data for the performing device or the training device. The chip system can be composed of a chip, or can include the chip and other discrete devices. BRIEF DESCRIPTION OF DRAWINGS

[0094] Figure 1A A structural schematic diagram of an artificial intelligence subject framework;

[0095] Figure 1B And Figure 2 A schematic diagram of an application system framework of the present application;

[0096] Figure 3 An optional hardware structural schematic diagram of a terminal;

[0097] Figure 4 A structural schematic diagram of a server;

[0098] Figure 5 A schematic diagram of a system architecture of the present application;

[0099] Figure 6 A flow of a cloud service;

[0100] Figure 7 A flow of a cloud service;

[0101] Figure 8 A flow schematic of a data processing method provided by an embodiment of the present application;

[0102] Figure 9 A flowchart of a data processing method provided by an embodiment of the application is shown in FIG. 1;

[0103] Figure 10 A flowchart of a data processing method provided by an embodiment of the application is shown in FIG. 1;

[0104] Figure 11A A flowchart of a data processing method provided by an embodiment of the application is shown in FIG. 1;

[0105] Figure 11B A flowchart of a data processing method provided by an embodiment of the application is shown in FIG. 1;

[0106] Figure 12A A flowchart of a data processing method provided by an embodiment of the application is shown in FIG. 1;

[0107] Figure 12B A flowchart of a data processing method provided by an embodiment of the application is shown in FIG. 1;

[0108] Figure 13 A structural diagram of a data processing device provided by an embodiment of the application is shown in FIG. 1;

[0109] Figure 14 A structural diagram of a data processing device provided by an embodiment of the application is shown in FIG. 1;

[0110] Figure 15 A structural diagram of an execution device provided by an embodiment of the application is shown in FIG. 1;

[0111] Figure 16 A structural diagram of a training device provided by an embodiment of the application is shown in FIG. 1;

[0112] Figure 17 A structural diagram of a chip provided by an embodiment of the application is shown in FIG. 1. DETAILED DESCRIPTION

[0113] The embodiments of the application will be described in conjunction with the drawings in the embodiments of the application. The terms used in the embodiments of the application are only used to explain the specific embodiments of the application, and are not intended to limit the application.

[0114] The embodiments of the application will be described in conjunction with the drawings in the embodiments of the application. The terms used in the embodiments of the application are only used to explain the specific embodiments of the application, and are not intended to limit the application.

[0115] The terms "first", "second", and the like, as used in the specification and in the claims of the application, and in the above Description of Embodiments, unless otherwise specified, are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. It is to be understood that the terms so used are interchangeable under appropriate circumstances such that the embodiments of the application described herein are, for example, capable of use in either order. It is therefore intended that the following claims not be limited to the described order or sequence, except where the description below claims otherwise.

[0116] The terms "substantially", "about", and like terms as used herein are used in a similarly relative sense and are intended to account for variations in measurement values or calculated values that would be recognized by those of ordinary skill in the art. Further, the use of "may" when describing embodiments of the application means, for example, one or more embodiments. The terms "use", "using", and "used" as used herein can be taken in their broadest possible context as being synonymous with the terms "utilize", "utilizing", and "utilized", respectively. Additionally, the term "exemplary" is intended to refer to an instance or illustration.

[0117] First, the overall workflow of the artificial intelligence system is described, please see Figure 1A , Figure 1A The structure of the artificial intelligence subject framework is shown, and the above artificial intelligence subject framework is described from two dimensions of "intelligent information chain" (horizontal axis) and "IT value chain" (vertical axis). Among them, the "intelligent information chain" reflects a series of processes from data acquisition to processing. For example, it can be the general process of intelligent information perception, intelligent information representation and formation, intelligent reasoning, intelligent decision-making, intelligent execution and output. In this process, data undergoes the condensation process of "data-information-knowledge-wisdom". The "IT value chain" reflects the value brought by artificial intelligence to the information technology industry from the bottom infrastructure of human intelligence, information (provision and processing technology implementation) to the industrial ecological process of the system.

[0118] (1) Infrastructure

[0119] Infrastructure provides computing power support for artificial intelligence systems, realizes communication with the external world, and realizes support through the underlying platform. Communication with the outside world through sensors; computing power is provided by intelligent chips (CPU, NPU, GPU, ASIC, FPGA, etc. Hardware acceleration chips); the underlying platform includes distributed computing framework and network-related platform support and support, which can include cloud storage and computing, interconnection network, etc. For example, sensors and external communication obtain data, which are provided to intelligent chips in the distributed computing system provided by the underlying platform for calculation.

[0120] (2) Data

[0121] The data of the upper layer of the infrastructure is used to represent the data source in the field of artificial intelligence. The data relates to graphics, images, voice, text, and also relates to the Internet of Things data of traditional devices, including the business data of existing systems and the sensing data of force, displacement, liquid level, temperature, humidity, etc.

[0122] (3) Data processing

[0123] Data processing usually includes data training, machine learning, deep learning, search, reasoning, decision, etc.

[0124] Among them, machine learning and deep learning can model, extract, preprocess, train, etc. Symbolic and formalized intelligent information.

[0125] Reasoning refers to the process of simulating human intelligent reasoning methods in computers or intelligent systems, using formalized information to perform machine thinking and solve problems according to reasoning control strategies, and the typical function is search and matching.

[0126] Decision-making refers to the process of decision-making after intelligent information is reasoned, which usually provides functions such as classification, sorting, prediction, etc.

[0127] (4) General capabilities

[0128] After the data is processed as mentioned above, some general capabilities can be formed based on the results of data processing, such as algorithms or a general system, for example, translation, text analysis, computer vision processing, speech recognition, image recognition, etc.

[0129] (5) Intelligent products and industry applications

[0130] Intelligent products and industry applications refer to products and applications of artificial intelligence systems in various fields, which are the packaging of the overall solution of artificial intelligence, and realize the application of intelligent information decision-making products. Its application fields mainly include: intelligent terminals, intelligent transportation, intelligent medical treatment, automatic driving, smart city, etc.

[0131] Firstly, the application scenario of the present application is introduced. The present application can be applied in, but is not limited to, an application program (hereinafter can be referred to as a virtual human generation application program) with the function of generating a video containing a virtual human (or other objects in addition to a virtual human) based on text or voice or a cloud service provided by a cloud-side server, and the like. Next, the present application is introduced respectively:

[0132] I. Virtual human generation application program

[0133] The product form of the embodiment of the present application can be a virtual human generation application program. The virtual human generation application program can run on a terminal device or a server on the cloud side.

[0134] In a possible implementation, the virtual human generation application program can implement the task of generating a video containing a virtual human (or other objects in addition to a virtual human) based on text or voice. In the scenario of intelligent answering, the text or voice can be a voice or text of a request, inquiry or other request interaction input by a user. The video of the virtual human generated by the application program can contain a response voice corresponding to the voice or text input by the user and a virtual human corresponding to the issuance of the response voice. The virtual human can make human-like actions, that is, can make body movements matched with the issuance of the response voice by a human. Exemplarily, the body movements can be gestures.

[0135] In the scenario of action generation, the text or voice can be a voice or text input by a user. The video of the virtual human generated by the application program can contain a virtual human corresponding to the issuance of a voice corresponding to the input voice or text by the user. The virtual human can make human-like actions, that is, can make body movements matched with the issuance of the voice corresponding to the input voice or text by a human.

[0136] In a possible implementation, a user can open a virtual human generation application program installed on a terminal device and input a voice or text (which can be active input or passive collection, for example, collected through an audio sensor on the terminal device). The virtual human generation application program can generate a virtual human or action data of the virtual human based on the voice or text by using the method provided in the present application, and present the virtual human or the action data of the virtual human to the user (the presentation mode can be, but is not limited to, display, saving, uploading to the cloud side, and the like).

[0137] In a possible implementation, a user can open a virtual human generation application installed on a terminal device, and input voice or text (which can be active input or passive collection, such as collection through an audio sensor on the terminal device). The virtual human generation application can send the voice or text to a server on the cloud side. The server on the cloud side generates a virtual human or action data of the virtual human based on the voice or text by using the method provided in the embodiments of the present application, and returns the virtual human or action data of the virtual human to the terminal device. The terminal device can present the virtual human or action data of the virtual human to the user (the presentation manner can be, but is not limited to, display, saving, uploading to the cloud side, etc.).

[0138] In a possible implementation, the virtual human generation implemented by the virtual human generation application can be specifically used for virtual character driving in application scenarios such as augmented reality (AR), virtual reality (VR), mixed reality (MR) remote conference, sports health, metaverse, etc.

[0139] Next, the virtual human generation application in the embodiments of the present application is introduced from the functional architecture and the product architecture implementing the functions respectively.

[0140] Referring to Figure 1B , Figure 1B The functional architecture of the virtual human generation application in the embodiments of the present application is shown in the following figure:

[0141] In a possible implementation, as Figure 1B shown, the virtual human generation application 102 can receive an input parameter 101 (for example, voice or text containing a human body) and generate action data 103 of a virtual human (or information of a virtual character restored based on the action data of the virtual human). The virtual human generation application 102 can be executed on, for example, at least one computer system, and includes computer code that, when executed by one or more computers, causes the computers to perform the data processing method described herein.

[0142] Referring to Figure 2 , Figure 2 The entity architecture of the virtual human generation application in the embodiments of the present application is shown in the following figure:

[0143] Referring to Figure 2 , Figure 2 A system architecture diagram is shown. The system can include a terminal 100 and a server 200. The server 200 can include one or more servers Figure 2The server 200 can provide virtual human generation services for one or more terminals (taking a server including one server as an example).

[0144] The terminal 100 can install a virtual human generation application or open a virtual human generation related webpage, the application and the webpage can provide an interface, the terminal 100 can receive parameters input by a user on the virtual human generation interface, and send the parameters to the server 200, the server 200 can obtain a processing result based on the received parameters, and return the processing result to the terminal 100.

[0145] It should be understood that in some optional implementations, the terminal 100 can also complete the action of obtaining a data processing result based on received parameters by itself without the cooperation of the server, and the embodiments of the present application are not limited.

[0146] Next, the product form of the terminal 100 is described. Figure 2

[0147] The terminal 100 in the embodiments of the present application can be a mobile phone, a tablet computer, a wearable device, a vehicle-mounted device, an augmented reality (AR) / virtual reality (VR) device, a notebook computer, an ultra-mobile personal computer (UMPC), a netbook, a personal digital assistant (PDA), etc., and the embodiments of the present application do not make any limitation thereto.

[0148] Figure 3 An optional hardware structure schematic diagram of the terminal 100 is shown.

[0149] Referring to Figure 3 As shown in the figure, the terminal 100 can include a radio frequency unit 110, a memory 120, an input unit 130, a display unit 140, a camera 150 (optional), an audio circuit 160 (optional), a speaker 161 (optional), a microphone 162 (optional), a processor 170, an external interface 180, a power supply 190, etc. Those skilled in the art can understand that Figure 3 The above-mentioned terminal or multi-functional device is only an example and does not constitute a limitation on the terminal or multi-functional device, and can include more or fewer components than the illustrated components, or combine certain components, or different components.

[0150] ​The input unit 130 can be used to receive inputted digital or character information, and to generate key signal inputs related to user settings of the portable multifunctional device and control of functions. Specifically, the input unit 130 can include a touch screen 131 (optional) and / or other input devices 132. The touch screen 131 can collect touch operations of a user thereon or thereabout (such as operations of the user using a finger, a knuckle, a stylus, or any suitable object on or near the touch screen), and drive corresponding connected devices according to pre-set programs. The touch screen can detect touch actions of the user on the touch screen, convert the touch actions into touch signals and send the touch signals to the processor 170, and can receive commands from the processor 170 and execute the commands; the touch signals at least include touch point coordinate information. The touch screen 131 can provide an input interface and an output interface between the terminal 100 and the user. In addition, the touch screen can be implemented in various types such as resistive, capacitive, infrared, and surface acoustic wave. In addition to the touch screen 131, the input unit 130 can also include other input devices. Specifically, the other input devices 132 can include, but are not limited to, one or more of a physical keyboard, function keys (such as volume control keys, on-off keys 133, etc.), trackballs, mice, joysticks, etc.

[0151] Among them, the other input devices 132 can receive inputted voice or text, etc.

[0152] The display unit 140 can be used to display information inputted by the user or provided to the user, various menus of the terminal 100, interactive interfaces, file display, and / or playing of any kind of multimedia files. In the embodiments of the present application, the display unit 140 can be used to display interfaces of virtual human generation application programs, virtual humans obtained based on voice or text, etc.

[0153] The storage 120 can be used to store instructions and data. The storage 120 can mainly include a storage instruction area and a storage data area. The storage data area can store various data such as multimedia files, texts, etc.; the storage instruction area can store software units such as operating systems, applications, instructions required by at least one function, etc., or their subsets, expanded sets. It can also include a non-volatile random access memory; provide the processor 170 with software and applications that include management of hardware, software, and data resources in computing processing devices, support control. It is also used for storage of multimedia files, and storage of running programs and applications.

[0154] The processor 170 is the control center of the terminal 100, connects each part of the whole terminal 100 by various interfaces and lines, executes various functions of the terminal 100 and processes data by running or executing the instructions stored in the memory 120 and calling the data stored in the memory 120, thereby overall controlling the terminal device. Optionally, the processor 170 can include one or more processing units; preferably, the processor 170 can integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface and application program, etc., and the modem processor mainly processes wireless communication. It can be understood that the above-mentioned modem processor can also not be integrated into the processor 170. In some embodiments, the processor, the memory, can be implemented on a single chip, and in some embodiments, they can also be implemented on separate chips respectively. The processor 170 can also be used to generate corresponding operation control signals to the corresponding components of the computing processing device, read and process the data in the software, especially read and process the data and programs in the memory 120, so that each functional module therein executes corresponding functions, thereby controlling the corresponding components to act according to the requirements of the instructions.

[0155] The memory 120 can be used to store software codes related to the data processing method, and the processor 170 can execute the steps of the data processing method of the chip, or can also schedule other units (such as the above-mentioned input unit 130 and display unit 140) to realize corresponding functions.

[0156] The RF unit 110 (optional) can be used to receive and send signals during a call or information transmission, for example, receiving downlink information from a base station and sending uplink data to the base station. Generally, the RF circuit includes, but is not limited to, an antenna, at least one amplifier, a transceiver, a coupler, a low noise amplifier (LNA), a duplexer, etc. In addition, the RF unit 110 can also communicate with network devices and other devices through wireless communication. The wireless communication can use any communication standard or protocol, including but not limited to Global System for Mobile Communication (GSM), General Packet Radio Service (GPRS), Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), Long Term Evolution (LTE), email, Short Messaging Service (SMS), etc.

[0157] In the embodiments of the present application, the RF unit 110 can send voice or text to the server 200 and receive motion data of a virtual person sent by the server 200 or information of a virtual person restored based on the motion data of the virtual person.

[0158] It should be understood that the RF unit 110 is optional and can be replaced by other communication interfaces, for example, a network interface.

[0159] The terminal 100 also includes a power supply 190 (such as a battery) for supplying power to each component. Preferably, the power supply can be logically connected to the processor 170 through a power management system, so that the power management system can realize functions such as management of charging, discharging, and power consumption management.

[0160] The terminal 100 also includes an external interface 180, which can be a standard Micro USB interface or a multi-pin connector. The external interface 180 can be used to connect the terminal 100 to other devices for communication, and can also be used to connect a charger to charge the terminal 100.

[0161] Although not shown, the terminal 100 can also include a flash, a wireless fidelity (WiFi) module, a Bluetooth module, sensors with different functions, etc., which will not be described here. Some or all of the methods described below can be applied to the terminal 100 as described above. Figure 3The terminal 100 shown.

[0162] Next, the product form of the server 200 is described. Figure 2 The product form of the server 200 is described.

[0163] Figure 4 A structural diagram of the server 200 is provided, as shown in the figure. Figure 4 The server 200 includes a bus 201, a processor 202, a communication interface 203, and a memory 204, as shown in the figure. The processor 202, the memory 204, and the communication interface 203 communicate through the bus 201.

[0164] The bus 201 can be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 4 In the figure, only one thick line is used, but it does not mean that there is only one bus or one type of bus.

[0165] The processor 202 can be any one or more of a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP), etc.

[0166] The memory 204 can include a volatile memory, such as a random access memory (RAM). The memory 204 can also include a non-volatile memory, such as a read-only memory (ROM), a flash memory, a mechanical hard drive (HDD), or a solid state drive (SSD).

[0167] The memory 204 can be used to store software code related to the data processing method, and the processor 202 can execute the steps of the chip data processing method or schedule other units to realize the corresponding functions.

[0168] It should be understood that the terminal 100 and the server 200 described above can be centralized or distributed devices, and the processors (for example, the processor 170 and the processor 202) in the terminal 100 and the server 200 can be hardware circuits (for example, an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA), a general-purpose processor, a digital signal processing (DSP), a microprocessor, a microcontroller, or the like) or a combination of the hardware circuits. For example, the processor can be a hardware system with an instruction execution function, such as a CPU, a DSP, or the like, or a hardware system without an instruction execution function, such as an ASIC, an FPGA, or the like, or a combination of the hardware system without an instruction execution function and the hardware system with an instruction execution function.

[0169] It should be understood that the steps related to the model inference process in the embodiments of the present application involve AI-related operations. When performing AI operations, the instruction execution architecture of the terminal device and the server is not limited to the architecture of the processor combined with the memory described above. The following will be described in combination with the instruction execution architecture of the processor combined with the memory. Figure 5 The system architecture provided by the embodiments of the present application will be described in detail.

[0170] Figure 5 The system architecture provided by the embodiments of the present application will be described in detail. Figure 5 As shown in the figure, the system architecture 500 includes an execution device 510, a training device 520, a database 530, a client device 540, a data storage system 550, and a data collection device 560.

[0171] The execution device 510 includes a computing module 511, an I / O interface 512, a preprocessing module 513, and a preprocessing module 514. The target model / rule 501 can be included in the computing module 511, and the preprocessing module 513 and the preprocessing module 514 are optional.

[0172] The execution device 510 can be a terminal device or a server running a virtual human generation application.

[0173] The data collection device 560 is used to collect training samples. The training samples can be voice or text, and annotations of characters in the voice or text (for example, real action data of characters), and the like. After collecting the training samples, the data collection device 560 stores the training samples in the database 530.

[0174] The training device 520 can train the neural network to be trained (for example, the feature extraction network, the action generation network, the encoder, etc. in the embodiments of the present application) based on the training samples maintained in the database 530 to obtain the target model / rule 501.

[0175] It should be noted that in actual applications, the training samples maintained in the database 530 do not necessarily come from the collection of the data collection device 560, but can also be received from other devices. In addition, it should be noted that the training device 520 does not necessarily train the target model / rule 501 based on the training samples maintained in the database 530, but can also obtain training samples from the cloud or other places for model training. The above description should not be regarded as a limitation of the embodiments of the present application.

[0176] The target model / rule 501 trained by the training device 520 can be applied to different systems or devices, such as the execution device 510 shown in the figure. The execution device 510 can be a terminal such as a mobile phone terminal, a tablet computer, a notebook computer, an augmented reality (AR) / virtual reality (VR) device, a vehicle-mounted terminal, etc., and can also be a server, etc. Figure 5

[0177] Specifically, the training device 520 can deliver the trained model to the execution device 510.

[0178] In the Figure 5 , the execution device 510 is configured with an input / output (I / O) interface 512 for data interaction with external devices. The user can input data (for example, voice or text, etc. in the embodiments of the present application) to the I / O interface 512 through the client device 540.

[0179] The pre-processing module 513 and the pre-processing module 514 are used for pre-processing the input data received by the I / O interface 512. It should be understood that there can be no pre-processing module 513 and pre-processing module 514 or only one pre-processing module. When there is no pre-processing module 513 and pre-processing module 514, the input data can be directly processed by the calculation module 511.

[0180] During the pre-processing of the input data by the execution device 510 or during the calculation and other related processing of the calculation module 511 of the execution device 510, the execution device 510 can call the data, code, etc. in the data storage system 550 for corresponding processing, or store the data, instructions, etc. obtained by the corresponding processing in the data storage system 550.

[0181] ​Finally, the I / O interface 512 provides the processing result (e.g. the action data of the virtual person or the information of the virtual person restored based on the action data of the virtual person, etc.) to the client device 540, thereby providing the user.

[0182] In Figure 5 In the case shown, the user can manually give the input data, which can be operated through the interface provided by the I / O interface 512. In another case, the client device 540 can automatically send the input data to the I / O interface 512, and if the authorization of the user is required for the client device 540 to automatically send the input data, the user can set the corresponding permission in the client device 540. The user can view the result output by the execution device 510 in the client device 540, and the specific presentation form can be a specific way such as display, sound, action, etc. The client device 540 can also be used as a data collection end to collect the input data input into the I / O interface 512 and the output result output from the I / O interface 512 as new sample data and store into the database 530. Of course, the collection can also be performed without the client device 540, and the I / O interface 512 directly stores the input data input into the I / O interface 512 and the output result output from the I / O interface 512 as new sample data into the database 530.

[0183] It is worth noting that, Figure 5 The system architecture shown in the figure is only a schematic diagram of a system architecture provided by the embodiments of the present application, and the positional relationship between the devices, components, modules, etc. shown in the figure does not constitute any limitation. For example, in Figure 5 In the case shown, the data storage system 550 is an external memory relative to the execution device 510, and in other cases, the data storage system 550 can also be placed in the execution device 510. It should be understood that the execution device 510 described above can be deployed in the client device 540.

[0184] From the inference side of the model:

[0185] In the embodiments of the present application, the computing module 511 of the execution device 510 can obtain the code stored in the data storage system 550 to implement the steps related to the model inference process in the embodiments of the present application.

[0186] In this embodiment of the application, the computing module 511 of the execution device 510 may include hardware circuits (such as application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), general-purpose processors, digital signal processors (DSPs), microprocessors or microcontrollers, etc.) or combinations of these hardware circuits. For example, the training device 520 may be a hardware system with instruction execution capabilities, such as a CPU or DSP, or a hardware system without instruction execution capabilities, such as an ASIC or FPGA, or a combination of the aforementioned hardware systems without instruction execution capabilities and hardware systems with instruction execution capabilities.

[0187] Specifically, the computing module 511 of the execution device 510 can be a hardware system with the function of executing instructions. The steps related to the model inference process provided in this application embodiment can be software code stored in the memory. The computing module 511 of the execution device 510 can obtain the software code from the memory and execute the obtained software code to implement the steps related to the model inference process provided in this application embodiment.

[0188] It should be understood that the computing module 511 of the execution device 510 can be a combination of a hardware system without the function of executing instructions and a hardware system with the function of executing instructions. Some steps related to the model reasoning process provided in the embodiments of this application can also be implemented by the hardware system in the computing module 511 of the execution device 510 without the function of executing instructions, which is not limited here.

[0189] From the training side of the model:

[0190] In this embodiment of the application, the training device 520 can access the memory ( Figure 5 (Not shown in the diagram, but can be integrated into the training device 520 or deployed separately from the training device 520) The code stored in the diagram can be used to implement the steps related to model training in the embodiments of this application.

[0191] In the embodiments of the present application, the training device 520 can include a hardware circuit (such as an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA), a general processor, a digital signal processor (DSP), a microprocessor or a microcontroller, etc.), or a combination of these hardware circuits. For example, the training device 520 can be a hardware system with an instruction execution function, such as a CPU, a DSP, etc., or a hardware system without an instruction execution function, such as an ASIC, an FPGA, etc., or a combination of the hardware system without an instruction execution function and the hardware system with an instruction execution function.

[0192] It should be understood that the training device 520 can be a combination of a hardware system without an instruction execution function and a hardware system with an instruction execution function, and the part of the steps related to the training of the neutralization model provided in the embodiments of the present application can also be implemented by the hardware system without an instruction execution function in the training device 520, which is not limited here.

[0193] II. Virtual human generation cloud service provided by the server:

[0194] In a possible implementation, the server can provide the virtual human generation service for the terminal side through an application programming interface (API).

[0195] In this case, the terminal device can send relevant parameters (such as voice or text) to the server through the API provided by the cloud, the server can obtain a processing result (such as action data of the virtual human or information of the virtual human restored based on the action data of the virtual human, etc.) based on the received parameters, and return the processing result to the terminal.

[0196] The description of the terminal and the server can be described in the above embodiments, which will not be repeated here.

[0197] As Figure 6 A flow of using a virtual human generation cloud service provided by a cloud platform is shown.

[0198] 1. Open and purchase the content review service.

[0199] 2. The user can download the software development kit (SDK) corresponding to the content review service. The cloud platform usually provides multiple development versions of the SDK for the user to select according to the needs of the development environment, such as a JAVA version of the SDK, a python version of the SDK, a PHP version of the SDK, an Android version of the SDK, and the like.

[0200] 3. The user downloads the corresponding version of the SDK according to the needs to the local, imports the SDK project to the local development environment, configures and debugs in the local development environment, and can also develop other functions in the local development environment, so as to form an application that integrates the virtual human generation class capability.

[0201] 4. In the process of using the virtual human generation class application, when virtual human generation is needed, the API call of virtual human generation can be triggered. When the application triggers the virtual human generation function, an API request is initiated to the running instance of the virtual human generation class service in the cloud environment, wherein the API request carries voice or text, and the running instance in the cloud environment processes the voice or text to obtain a processing result (for example, action data of a virtual human or information of a virtual character restored based on the action data of the virtual human, etc.).

[0202] 5. The cloud environment returns the processing result to the application, thereby completing a virtual human generation service call.

[0203] III. Model training class cloud service provided by the server:

[0204] In a possible implementation, the server can provide a virtual human generation model adapted to a character object (or a more generalized character) in the voice or text provided by the client.

[0205] In a possible implementation, the server can provide a character information restoration service for the terminal side through an application programming interface (API).

[0206] The terminal device can send relevant parameters (such as voice or text) to the server through the API provided by the cloud, the server can obtain a processing result based on the received parameters, and return the processing result (such as a virtual human generation model adapted to a character object (or a more generalized character) in the voice or text) to the terminal.

[0207] As Figure 7 A flow of using a model training class cloud service provided by a cloud platform is shown.

[0208] The description of the terminal and the server can refer to the description of the above embodiments, which will not be repeated here.

[0209] Since the embodiments of the present application involve the application of a large number of neural networks, in order to facilitate understanding, the related terms and concepts related to neural networks involved in the embodiments of the present application will be introduced first.

[0210] (1) Neural network

[0211] The neural network can be composed of neural units, and the neural unit can be an operation unit with xs and intercept 1 as inputs. The output of the operation unit can be:

[0212]

[0213] where s = 1, 2, … n, n is a natural number greater than 1, Ws is the weight of xs, b is the bias of the neural unit. f is the activation function of the neural unit, which is used to introduce non-linear characteristics into the neural network to convert the input signal in the neural unit into an output signal. The output signal of the activation function can be used as the input of the next convolutional layer. The activation function can be a sigmoid function. The neural network is a network formed by connecting many single neural units described above, that is, the output of one neural unit can be the input of another neural unit. The input of each neural unit can be connected to the local receptive field of the previous layer to extract the features of the local receptive field, and the local receptive field can be a region composed of several neural units.

[0214] (2) Deep neural network

[0215] Deep neural network (DNN) can be understood as a neural network with many hidden layers. There is no special measurement standard for "many" here. The multi-layer neural network and deep neural network commonly referred to are essentially the same thing. From the division of DNN according to the position of different layers, the neural network inside DNN can be divided into three categories: input layer, hidden layer, and output layer. Generally, the first layer is the input layer, the last layer is the output layer, and the number of layers in between is the hidden layer. The layers are fully connected, that is, any neuron in the i-th layer is connected to any neuron in the i+1-th layer. Although DNN looks very complex, in terms of the work of each layer, it is actually not complex. In simple terms, it is expressed as the following linear relationship expression: wherein, is the input vector, is the output vector, is the offset vector, W is the weight matrix (also called coefficient), and α() is the activation function. Each layer only processes the input vector After such a simple operation, we get the output vector Since DNN has many layers, the number of coefficients W and offset vectors is also very large. So, how are the specific parameters defined in DNN? First, let's look at the definition of the coefficient W. Take a three-layer DNN as an example: the linear coefficient of the fourth neuron of the second layer to the second neuron of the third layer is defined as The superscript 3 represents the layer number of the coefficient W, and the subscript corresponds to the third layer index 2 of the output and the second layer index 4 of the input. In summary, the coefficient of the kth neuron of the L-1th layer to the jth neuron of the Lth layer is defined as Note that the input layer has no W parameters. In deep neural networks, more hidden layers allow the network to better describe complex situations in the real world. In theory, the more parameters a model has, the higher its complexity and "capacity", which means it can complete more complex learning tasks.

[0216] (3) Convolutional Neural Network (CNN) is a deep neural network with convolutional structure. Convolutional Neural Network contains a feature extractor composed of convolutional layers and subsampling layers. The feature extractor can be regarded as a filter, and the convolution process can be regarded as using a trainable filter to convolve with an input image or convolution feature plane. The convolutional layer refers to the neural layer that performs convolution processing on the input signal in the convolutional neural network. In the convolutional layer of the convolutional neural network, a neuron can only be connected to part of the adjacent layer neurons. A convolutional layer usually contains several feature planes, and each feature plane can be composed of some rectangularly arranged neural units. The neural units of the same feature plane share weights, and the shared weights are the convolution kernel. Shared weights can be understood as the way of extracting image information regardless of the location. The implicit principle is that the statistical information of a part of the image is the same as that of other parts. That is, the image information learned in one part can also be used in another part. Therefore, for all positions on the image, we can use the same learned image information. In the same convolutional layer, multiple convolution kernels can be used to extract different image information. Generally, the more convolution kernels, the more image information reflected by the convolution operation.

[0217] The convolution kernel can be initialized in the form of a random size matrix, and the convolution kernel can obtain reasonable weights through learning in the training process of the convolutional neural network. In addition, the direct benefit of shared weights is to reduce the connections between layers of the convolutional neural network, while also reducing the risk of overfitting.

[0218] (4) Backpropagation algorithm

[0219] The convolutional neural network can adopt a backpropagation (BP) algorithm to correct the size of the parameters in the initial super-resolution model during the training process, so that the reconstruction error loss of the super-resolution model becomes smaller and smaller. Specifically, the forward transmission of the input signal until the output generates an error loss, and the error loss information is propagated backward to update the parameters in the initial super-resolution model, so as to make the error loss converge. The backpropagation algorithm is a backpropagation movement dominated by error loss, aiming to obtain the optimal parameters of the super-resolution model, such as the weight matrix.

[0220] (5) Loss function

[0221] In the process of training a deep neural network, because it is desired that the output of the deep neural network is as close as possible to the value that is truly intended to be predicted, the weight vector of each layer of the neural network can be updated according to the difference between the predicted value of the current network and the truly intended target value by comparing the predicted value of the current network with the truly intended target value (of course, there is usually an initialization process before the first update, that is, the parameters of each layer of the deep neural network are pre-configured), for example, if the predicted value of the network is too high, the weight vector is adjusted to make it predict lower, and the adjustment is continuously made until the deep neural network can predict the truly intended target value or a value very close to the truly intended target value. Therefore, it is necessary to define in advance "how to compare the difference between the predicted value and the target value", which is the loss function or the objective function, which is an important equation for measuring the difference between the predicted value and the target value. Among them, taking the loss function as an example, the higher the output value (loss) of the loss function, the greater the difference, and then the training of the deep neural network becomes a process of trying to minimize this loss.

[0222] (6) Backpropagation algorithm

[0223] The neural network can adopt a backpropagation (BP) algorithm to correct the size of the parameters in the initial neural network model during the training process, so that the reconstruction error loss of the neural network model becomes smaller and smaller. Specifically, the forward transmission of the input signal until the output generates an error loss, and the error loss information is propagated backward to update the parameters in the initial neural network model, so as to make the error loss converge. The backpropagation algorithm is a backpropagation movement dominated by error loss, aiming to obtain the optimal parameters of the neural network model, such as the weight matrix.

[0224] In public speeches, body language can increase the rhythm of the speech, making the speech more vivid and more persuasive. Studies have shown that body language plays an important role in communication. First, body movements more accurately express intentions and convey emotions, complementing the information conveyed by speech; second, body movements can help users focus more on the content of the communication with the digital person; third, it can improve the persuasiveness, credibility and authenticity of the digital person; and finally, it can reflect the speaker's intentions and personality. Lack of body language or rigid body movements in communication can lead to the uncanny valley effect.

[0225] People expect virtual digital people (or simply digital people) to behave more like humans. Virtual digital people can make rhythmic and emphatic movements in speeches, just like humans. For some special speech semantics, there can be specific movements, for example, when people say "good / ok", they tend to make an OK gesture, and when they say "first / first", they tend to pause slightly and make a 1 hand shape.

[0226] Therefore, how to generate corresponding body movements such as gestures based on speech is a problem to be solved.

[0227] To solve the above problems, the embodiments of the present application provide a data processing method. The data processing method of the embodiments of the present application will be described in detail below with reference to the accompanying drawings.

[0228] Reference Figure 8 , Figure 8 The flowchart of the data processing method provided by the embodiments of the present application is shown in Figure 8 The data processing method provided by the embodiments of the present application can include steps 801 to 804, which will be described in detail below.

[0229] 801, obtain the speech data.

[0230] In the action generation application, the speech data of the user can be obtained, and the corresponding action data can be generated based on the speech data. The action data can be used to generate a corresponding virtual person.

[0231] In the action generation application, the text data of the user can be obtained, and the corresponding speech data can be generated based on the text data, and then the corresponding action data can be generated based on the speech data (optionally, it can also include text data). The action data can be used to generate a corresponding virtual person.

[0232] In an interactive application, speech data of a user can be obtained, a reply text can be determined based on text in the speech data, and corresponding speech data can be generated according to the reply text, and then corresponding action data can be generated based on the speech data (optionally, the action data can also include the reply text). The action data can be used to generate a corresponding virtual human.

[0233] In a possible implementation, the action data can be a gesture action, etc.

[0234] 802. Determine, from the speech data, a plurality of split point positions corresponding to predicted rhythm points of a body action made by a character object when the speech data is uttered, according to audio features of the speech data.

[0235] In a possible implementation, action data of a character object uttering speech data can be predicted according to the speech data. When a user utters speech (for example, when giving a speech), the user will make a certain action, and the action has certain rhythm points, which are critical points from stillness to movement or from movement to stillness when the body action is made. If the rhythm information can be explicitly identified before feature extraction and used as an input of a model, the action generated subsequently can have accurate rhythm, and thus the action is more consistent with the action of the character object when the speech is actually uttered.

[0236] Next, how to identify the above rhythm information is introduced.

[0237] In a possible implementation, a plurality of split point positions can be determined from the speech data according to audio features of the speech data, wherein the audio features relate to at least one of volume and pitch.

[0238] In a possible implementation, rhythm points usually appear on onsets (audio features related to volume) of speech, but not all onsets are rhythm points of user actions. The interval between adjacent rhythm points of speech has a range, which is related to the style of the speaker. Assuming that the range is [t1, t2], the time interval less than t1 is usually caused by noise, fillers, or unsmooth pronunciation. The time interval greater than t2 is usually corresponding to a pause or a quiet phase of not speaking. Based on these observations, onsets with a time interval between adjacent onsets within a preset range can be determined as rhythm points, and then rhythm points of a body action made by a user when the speech data is uttered can be identified, and the speech data can be segmented based on the rhythm points.

[0239] Taking a first position and a second position as an example, the first position and the second position are adjacent split point positions in the plurality of split point positions.

[0240] In a possible implementation, the first position and the second position are determined in the following manner: the first position and the second position are onset points in the voice data, and a time interval between the onset points is within a preset range. The preset range can be [t1, t2] as described above.

[0241] In a possible implementation, a heuristic strategy can be used to identify the rhythm of the voice. The voice is segmented, and points less than t1 are not identified as rhythm points, and points greater than t2 can be inserted as pseudo-rhythm points. The pseudo-rhythm points can be inserted in the following manner: the time interval from the last rhythm point is greater than t1, and the volume of the voice is greater than a threshold I a The threshold can be set as the average volume of the ambient noise, so that the time interval between adjacent segment positions is within a preset range. If all the volumes in the interval are less than I a The least pseudo-rhythm points are inserted uniformly to make the time interval between adjacent segment positions within a preset range.

[0242] In a possible implementation, the plurality of segment positions includes adjacent first and second positions; the first position and the second position can be determined in the following manner: the first position is an onset point in the voice data, the second position is not an onset point in the voice data, and the second position is a time point in the voice data with a volume greater than a threshold.

[0243] 803. According to the voice data and the information indicating the plurality of segment positions, a feature representation is obtained through a feature extraction network.

[0244] 804. According to the feature representation, action data is generated through an action generation network.

[0245] In a possible implementation, the voice data can be segmented according to the segment positions to obtain a plurality of voice segments, and the plurality of voice segments can be input into the feature network.

[0246] In a possible implementation, the voice data can be segmented according to the segment positions to obtain a plurality of voice segments, and the plurality of voice segments can be input into the feature network.

[0247] In a possible implementation, the input voice can be segmented according to the identified rhythm points, and the segmented voice segments can be normalized to a length of t2. The action features and the text features corresponding to the voice can also be resampled to a length of t2.

[0248] In a possible implementation, the speech data and the character indicating the plurality of split point positions can be input to a feature extraction network to obtain a feature representation, and action data can be generated by an action generation network according to the feature representation.

[0249] In a possible implementation, in addition to inputting the feature representation obtained by the feature extraction network to the action generation network, a type of action can be identified by a neural network according to the feature representation (for example, high-level features in the feature representation), and the action type and the feature representation can be input to the action generation network.

[0250] According to the above rhythm-based segmentation method, all actions in the data set can be segmented into action segments of equal length. The categories corresponding to these actions can be obtained by a neural network, and the action categories can be considered as high-level features of the actions.

[0251] Clustering the high-level speech features in the data set can find that the corresponding texts in each category are similar, for example, “many”, “quite a few”, “lots of”, “much”, “more”, and the like; inputting the actions corresponding to the high-level speech features to the trained action encoder, and visualizing the action encoding, it can be found that these actions are concentrated in a few categories.

[0252] In a specific implementation, the vq-vae can be used to cluster the actions in the hidden space of the actions, or the kmeans can be directly used to cluster the actions. According to the segmentation result, a series of action segments {M1, M2, …, M i ,…M n} are obtained, and a gesture dictionary library

[0253] In a possible implementation, the feature extraction network includes a plurality of serial network layers, the feature representation includes feature representations output by the plurality of network layers, and the action category of the body action can be determined according to the feature representations output by one or more network layers close to an output layer in the plurality of network layers (that is, high-level features); the action data is generated by the action generation network according to the feature representation, including generating the action data by the action generation network according to part or all of the feature representations output by the plurality of network layers and the action category.

[0254] Since the action category itself carries semantics, the features input to the action generation network and the subsequent generated body action have semantic relevance.

[0255] In addition, for any action segment M iThe encoder of the VQ-VAE can know the corresponding high-level feature gesture vocabulary (action category) S i The style code Z corresponding to the action segment can be decomposed i The method is shown in Figure 10 The action M of the previous segment i-1 The corresponding action feature is obtained through the action encoder The low-level feature of the current period of speech The low-level speech features of the two adjacent periods And The audio feature of the current period is obtained through the audio encoder S S i And the learnable gesture style code Z i are fused together and input into the action generation network to generate the final action.

[0256] In a possible implementation, the feature extraction network includes a plurality of serial network layers, the feature representation includes feature representations output by the plurality of network layers, and the style code of the body action can be obtained through an encoder according to the feature representations output by one or more network layers far from the output layer in the plurality of network layers; and the action data is generated through the action generation network according to the feature representations, specifically including: the action data is generated through the action generation network according to part or all of the feature representations output by the plurality of network layers and the style code.

[0257] In a possible implementation, the distribution of the style code is a uniform distribution.

[0258] In a possible implementation, the decoupling of the high-level features and the low-level features described above can refer to Figure 9 : Given audio and its corresponding text, a language pre-training model is used as a supervision signal for contrastive learning, so that the high-level features of the pre-training model of the speech are close to the high-level features of the text, and the low-level features of the pre-training model of the speech are close to the low-level features of the text.

[0259] In a possible implementation, the action data can be the joint rotation angle of the body action or the 3D point cloud information of the action, which is not limited here.

[0260] For example, given a speech input, the method provided by the embodiment of the application can segment the speech segment into normalized feature blocks Then the corresponding action is generated in sequence The overall process is shown in Figure 11A , wherein

[0261]

[0262] wherein, ε M , ε A are the action generator, the action encoder and the audio encoder learned from the previous.

[0263] In the above formula, only the gesture vocabulary and the gesture style code cannot be directly obtained from the input speech, and they are inferred from the action class prediction module and the action style prediction module.

[0264] In a possible implementation, text data can also be taken as input. In addition, the text scheme can be simplified, and in the case of only audio input without its corresponding text and speaker ID, the corresponding action can also be inferred. The flowchart is shown in Figure 10 , in which case only audio is input.

[0265] Next, the data processing method in the embodiments of the present application is introduced from the training side.

[0266] Referring to Figure 11B , Figure 11B a flowchart of a data processing method provided by the embodiments of the present application, the method comprises:

[0267] 1101, obtaining speech data of a character object and first action data, the first action data being real action data of a body action made by the character object when the speech data is output;

[0268] The first action data can be the true value of the action data corresponding to the speech data.

[0269] 1102, determining a plurality of segmentation point positions from the speech data according to the audio features of the speech data, the segmentation point positions corresponding to predicted rhythm points of the body action made by the character object when the speech data is output;

[0270] For specific description of step 1102, refer to the introduction of step 802 in the above embodiments, which will not be repeated here.

[0271] 1103, obtaining a feature representation through a feature extraction network according to the speech data and information indicating the plurality of segmentation point positions;

[0272] For specific description of step 1103, refer to the introduction of step 803 in the above embodiments, which will not be repeated here.

[0273] 1104、According to the feature representation, a second action data is generated by an action generation network; and a difference between the first action data and the second action data is used to update the feature extraction network and the action generation network.

[0274] The specific description of step 1104 can refer to the description of step 804 in the above embodiment, which will not be repeated here.

[0275] In addition to the difference between the first action data and the second action data, other losses can also be included, which can be specifically:

[0276] L gen =w rec L rec +w perc L perc +w lexeme L lexeme +w z L z …(1)

[0277] Where the reconstruction loss is:

[0278]

[0279] is the MSE error between the generated action segment and the GT action segment M i .

[0280] The perception error loss is:

[0281]

[0282] Where ε is the pre-trained action encoder.

[0283] Suppose the category of gesture action is determined by the vocabulary, and the information of other channels can only affect the variability of the action. In order to strengthen this constraint, a series of actions are generated using the current gesture vocabulary and some random other features

[0284]

[0285] Where P is the position encoding, and the sign language word loss is defined as:

[0286]

[0287] J is a random subset of all motion segments in the training set, and N J is the size of J. N J can be adjusted according to the size of the data set.

[0288] Finally, the KL divergence loss is used to normalize the style code:

[0289]

[0290] where μ z and are the mean and variance of the gesture style code in the training mini-batch, respectively.

[0291] In a possible implementation, the feature extraction network comprises a plurality of serially connected network layers, the feature representation comprises feature representations output by the plurality of network layers, and the generating, by the action generation network, of the second action data according to the feature representation comprises: generating, by the action generation network, the second action data according to the feature representation and an initialized style code; obtaining, by an encoder, a first style code of the body action according to feature representations output by one or more network layers of the plurality of network layers that are away from an output layer; and updating, by a preset loss function, the initialized style code to obtain an updated style code.

[0292] A difference between the first style code and the updated style code is used to update the encoder.

[0293] In a possible implementation, the audio features are related to volume and / or pitch.

[0294] In a possible implementation, the plurality of segmentation point positions comprises a first position and a second position adjacent to each other, and a time interval between the second position and the first position is within a preset range; and the first position and the second position are determined in one of the following manners:

[0295] The first position and the second position are onset points in the speech data.

[0296] The first position is an onset point in the speech data, the second position is not an onset point in the speech data, and the second position is a time point at which volume in the speech data is greater than a threshold.

[0297] In a possible implementation, the speech data and the information indicating the plurality of segmentation point positions comprise:

[0298] a plurality of speech segments into which the speech data is divided according to the plurality of segmentation point positions; or

[0299] characters in the speech data and indicating the plurality of segmentation point positions.

[0300] In one possible implementation, the feature extraction network includes multiple cascaded network layers, the feature representation includes feature representations output by the multiple network layers, and the method further includes: determining the action category of the body movement based on the feature representations output by one or more network layers that are closest to the output layer among the multiple network layers;

[0301] The step of generating second action data through an action generation network based on the feature representation includes:

[0302] Based on some or all of the feature representations output by the multiple network layers and the action category, a second action data is generated through an action generation network.

[0303] In one possible implementation, the feature extraction network includes multiple cascaded network layers, and the feature representation includes feature representations output by the multiple network layers. The method further includes: obtaining a style code for the body movement by an encoder based on the feature representations output by one or more network layers that are far from the output layer; and generating second motion data by an action generation network based on the feature representations, which includes: generating second motion data by an action generation network based on some or all of the feature representations output by the multiple network layers and the style code.

[0304] Reference Figure 12A , Figure 12A This is a flowchart illustrating the process of generating motion data using text data, see [link / reference]. Figure 12B , Figure 12A This is a flowchart illustrating how to generate motion data without using text data.

[0305] Reference Figure 13 , Figure 13 This is a schematic diagram of the structure of a data processing apparatus provided in an embodiment of this application, such as... Figure 13 As shown in the figure, an embodiment of this application provides a data processing apparatus 1300, comprising:

[0306] Module 1301 is used to acquire voice data;

[0307] The specific description of the acquisition module 1301 can be found in the description of step 801 in the above embodiment, and will not be repeated here.

[0308] Processing module 1302 is used to determine multiple segmentation point positions from the speech data based on the audio features of the speech data, wherein the segmentation point positions correspond to the predicted rhythm points of the body movements made by the person when uttering the speech data;

[0309] Based on the speech data and the information indicating the positions of the multiple segmentation points, a feature representation is obtained through a feature extraction network;

[0310] According to the feature representation, action data is generated by an action generation network.

[0311] Specific description about the processing module 1302 can refer to the description of steps 802 to 804 in the above embodiment, and will not be repeated here.

[0312] In a possible implementation, the audio features are related to volume and / or pitch.

[0313] In a possible implementation, the plurality of split point positions include a first position and a second position adjacent to the first position, and a time interval between the second position and the first position is within a preset range; and the first position and the second position are determined in the following manner:

[0314] The first position and the second position are onset points in the speech data.

[0315] The first position is an onset point in the speech data, the second position is not an onset point in the speech data, and the second position is a time point at which volume in the speech data is greater than a threshold.

[0316] In a possible implementation, the speech data and the information indicating the plurality of split point positions include:

[0317] According to the plurality of split point positions, the speech data is divided into a plurality of speech segments; or

[0318] The speech data and the information indicating the plurality of split point positions include characters.

[0319] In a possible implementation, the feature extraction network includes a plurality of serially connected network layers, the feature representation includes feature representations output by the plurality of network layers, and the processing module is further configured to:

[0320] According to feature representations output by one or more network layers close to an output layer in the plurality of network layers, an action category of the body action is determined.

[0321] The processing module is specifically configured to:

[0322] According to part or all of the feature representations output by the plurality of network layers and the action category, action data is generated by an action generation network.

[0323] In a possible implementation, the feature extraction network includes a plurality of serially connected network layers, the feature representation includes feature representations output by the plurality of network layers, and the processing module is further configured to:

[0324] According to the feature representation output by one or more network layers far from the output layer in the plurality of network layers, a style code of the body action is obtained through an encoder;

[0325] The processing module is specifically configured to:

[0326] According to part or all of the feature representation output by the plurality of network layers and the style code, action data is generated through an action generation network.

[0327] In a possible implementation, the distribution of the style code is a uniform distribution.

[0328] Referring to Figure 14 , Figure 14 The structure of a data processing apparatus provided by an embodiment of the present application is shown in Figure 14 The data processing apparatus 1400 provided by an embodiment of the present application includes:

[0329] The acquisition module 1401 is configured to acquire voice data of a character object and first action data, the first action data being real action data of a body action made by the character object when the voice data is emitted.

[0330] Specific descriptions about the acquisition module 1401 can be referred to the introduction of step 1101 in the above embodiments, which will not be repeated here.

[0331] The processing module 1402 is configured to determine a plurality of segmentation point positions from the voice data according to audio features of the voice data, the segmentation point positions corresponding to predicted rhythm points of a body action made by the character object when the voice data is emitted.

[0332] According to the voice data and information indicating the plurality of segmentation point positions, a feature representation is obtained through a feature extraction network.

[0333] According to the feature representation, second action data is generated through an action generation network; and a difference between the first action data and the second action data is used to update the feature extraction network and the action generation network.

[0334] Specific descriptions about the processing module 1402 can be referred to the introduction of steps 1102 to 1104 in the above embodiments, which will not be repeated here.

[0335] In a possible implementation, the feature extraction network includes a plurality of serially connected network layers, the feature representation includes feature representations output by the plurality of network layers, and the processing module is specifically configured to:

[0336] According to the feature representation and the initialized style code, second action data is generated by an action generation network.

[0337] According to the feature representation output by one or more network layers far from the output layer in the plurality of network layers, a first style code of the body action is obtained by an encoder;

[0338] The initialized style code is updated by a preset loss function to obtain an updated style code.

[0339] A difference between the first style code and the updated style code is used to update the encoder.

[0340] In a possible implementation, the audio features are related to volume and / or pitch.

[0341] In a possible implementation, the plurality of split point positions include a first position and a second position adjacent to each other, and a time interval between the second position and the first position is within a preset range; and the first position and the second position are determined in a manner as follows:

[0342] The first position and the second position are onset points in the speech data.

[0343] The first position is an onset point in the speech data, the second position is not an onset point in the speech data, and the second position is a time point at which volume in the speech data is greater than a threshold.

[0344] In a possible implementation, the speech data and the information indicating the plurality of split point positions include:

[0345] A plurality of speech segments into which the speech data is divided according to the plurality of split point positions; or

[0346] The speech data and characters indicating the plurality of split point positions.

[0347] In a possible implementation, the feature extraction network includes a plurality of serially connected network layers, and the feature representation includes feature representations output by the plurality of network layers; and the processing module is further configured to:

[0348] According to the feature representation output by one or more network layers close to the output layer in the plurality of network layers, a motion category of the body action is determined.

[0349] The processing module is specifically configured to:

[0350] According to part or all of the feature representations output by the plurality of network layers and the motion category, second action data is generated by an action generation network.

[0351] In a possible implementation, the feature extraction network comprises a plurality of serially connected network layers, the feature representation comprises feature representations output by the plurality of network layers, and the processing module is further configured to:

[0352] obtain, by an encoder, a style code of the body action according to the feature representations output by one or more network layers of the plurality of network layers that are away from an output layer;

[0353] The processing module is specifically configured to:

[0354] generate, by an action generation network, second action data according to part or all of the feature representations output by the plurality of network layers and the style code.

[0355] Next, an execution device provided by an embodiment of the present application is introduced. Please refer to Figure 15 , Figure 15 A structural schematic diagram of the execution device provided by the embodiment of the present application is shown in FIG. 15. The execution device 1500 can be a virtual reality (VR) device, a mobile phone, a tablet computer, a notebook computer, a smart wearable device, a monitoring data processing device, a server, or the like, which is not limited herein. Specifically, the execution device 1500 comprises a receiver 1501, a transmitter 1502, a processor 1503, and a memory 1504 (wherein the number of processors 1503 in the execution device 1500 can be one or more, and one processor is taken as an example in the embodiment), wherein the processor 1503 can comprise an application processor 15031 and a communication processor 15032. In some embodiments of the present application, the receiver 1501, the transmitter 1502, the processor 1503, and the memory 1504 can be connected through a bus or other means. Figure 15

[0356] The memory 1504 can comprise a read-only memory and a random access memory, and provide the processor 1503 with instructions and data. Part of the memory 1504 can also comprise a non-volatile random access memory (NVRAM). The memory 1504 stores processor and operation instructions, executable modules or data structures, or subsets thereof, or expanded sets thereof, wherein the operation instructions can comprise various operation instructions for implementing various operations.

[0357] The processor 1503 controls the operation of the execution device. In a specific application, various components of the execution device are coupled together through a bus system, wherein the bus system can comprise a data bus, a power supply bus, a control bus, and a state signal bus, etc. in addition to the data bus. However, for the sake of clarity, all kinds of buses are referred to as a bus system in the figure. ​

[0358] The method disclosed in the embodiments of the present application can be applied to the processor 1503 or implemented by the processor 1503. The processor 1503 can be an integrated circuit chip having a signal processing capability. In the implementation process, each step of the above method can be completed by an integrated logic circuit or an instruction in the form of software in the processor 1503. The processor 1503 described above can be a general processor, a digital signal processor (DSP), a microprocessor or a microcontroller, and can further include an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component. The processor 1503 can implement or execute the methods, steps and logic block diagrams disclosed in the embodiments of the present application. The general processor can be a microprocessor or the processor can also be any conventional processor or the like. The steps of the method disclosed in combination with the embodiments of the present application can be directly embodied as a hardware code processor for execution, or a combination of hardware and software modules in the code processor for execution. The software module can be located in a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register or other mature storage medium in the art. The storage medium is located in the storage 1504, and the processor 1503 reads the information in the storage 1504 and combines the hardware to complete the steps related to the model inference process in the above method.

[0359] The receiver 1501 can be used to receive input digital or character information, and generate signal input related to the relevant settings and function control of the execution device. The transmitter 1502 can be used to output digital or character information through the first interface; the transmitter 1502 can also be used to send instructions to the disk group through the first interface to modify the data in the disk group; the transmitter 1502 can also include a display device such as a display screen.

[0360] The embodiments of the present application also provide a training device, please refer to Figure 16 , Figure 16is a structural schematic diagram of a training device provided by an embodiment of the present application. Specifically, the training device 1600 is implemented by one or more servers. The training device 1600 can have a large difference due to different configurations or performances, and can include one or more central processing units (CPUs) 1616 (for example, one or more processors) and a memory 1632, one or more storage media 1630 (for example, one or more mass storage devices) storing an application program 1642 or data 1644. The memory 1632 and the storage media 1630 can be temporary storage or persistent storage. The program stored in the storage media 1630 can include one or more modules (not shown in the figure), and each module can include a series of instruction operations in the training device. Further, the central processing unit 1616 can be configured to communicate with the storage media 1630 and execute the series of instruction operations in the storage media 1630 on the training device 1600.

[0361] The training device 1600 can further include one or more power supplies 1626, one or more wired or wireless network interfaces 1650, one or more input / output interfaces 1658, or one or more operating systems 1641, such as Windows ServerTM, Mac OS XTM, UnixTM, LinuxTM, FreeBSDTM, and the like.

[0362] In an embodiment of the present application, the central processing unit 1616 is configured to perform the actions related to model training in the above embodiments.

[0363] An embodiment of the present application further provides a computer program product, which, when running on a computer, causes the computer to perform the steps performed by the above execution device, or causes the computer to perform the steps performed by the above training device.

[0364] An embodiment of the present application further provides a computer readable storage medium, which stores a program for signal processing, and when running on a computer, causes the computer to perform the steps performed by the above execution device, or causes the computer to perform the steps performed by the above training device.

[0365] The execution device, the training device or the terminal device provided by the embodiments of the present application can be a chip, which includes a processing unit, for example, a processor, and a communication unit, for example, an input / output interface, a pin or a circuit, etc. The processing unit can execute computer execution instructions stored in a storage unit, so that the chip in the execution device executes the data processing method described in the above embodiments, or so that the chip in the training device executes the data processing method described in the above embodiments. Optionally, the storage unit is a storage unit in the chip, such as a register, a cache, etc., and the storage unit can also be a storage unit outside the chip in the wireless access device, such as a read-only memory (ROM) or other types of static storage devices that can store static information and instructions, a random access memory (RAM), etc.

[0366] Specifically, refer to Figure 17 , Figure 17 A structural schematic diagram of the chip provided by the embodiments of the present application is shown in FIG. 17. The chip can be a neural network processor NPU 1700, which is mounted on a host CPU (Host CPU) as a coprocessor and is assigned tasks by the Host CPU. The core part of the NPU is an operation circuit 1703, which extracts matrix data in a memory and performs multiplication operation under the control of a controller 1704.

[0367] In some implementations, the operation circuit 1703 internally includes a plurality of processing units (PEs). In some implementations, the operation circuit 1703 is a two-dimensional systolic array. The operation circuit 1703 can also be a one-dimensional systolic array or other electronic circuits capable of performing mathematical operations such as multiplication and addition. In some implementations, the operation circuit 1703 is a general-purpose matrix processor.

[0368] For example, it is assumed that there are an input matrix A, a weight matrix B and an output matrix C. The operation circuit takes corresponding data of the matrix B from the weight memory 1702 and buffers it on each PE in the operation circuit. The operation circuit takes the matrix A data from the input memory 1701 and performs matrix operation with the matrix B, and the partial result or final result of the obtained matrix is saved in an accumulator 1708.

[0369] The unified memory 1706 is used to store input data and output data. The weight data is transferred to the weight memory 1702 through the Direct Memory Access Controller (DMAC) 1705. The input data is also transferred to the unified memory 1706 through the DMAC.

[0370] The BIU is the Bus Interface Unit 1710, which is used for the interaction between the AXI bus and the DMAC and the instruction fetch buffer (IFB) 1709.

[0371] The BIU 1710 is used for the instruction fetch buffer 1709 to fetch instructions from the external memory, and is also used for the Direct Memory Access Controller 1705 to fetch the original data of the input matrix A or the weight matrix B from the external memory.

[0372] The DMAC is mainly used to transfer the input data in the external memory DDR to the unified memory 1706, or to transfer the weight data to the weight memory 1702, or to transfer the input data to the input memory 1701.

[0373] The vector calculation unit 1707 includes a plurality of operation processing units, which further process the output of the operation circuit 1703 as needed, such as vector multiplication, vector addition, exponential operation, logarithmic operation, size comparison, etc. It is mainly used for non-convolutional / full connection layer network calculation in neural networks, such as Batch Normalization, pixel-level summation, upsampling of feature planes, etc.

[0374] In some implementations, the vector calculation unit 1707 can store the processed output vector to the unified memory 1706. For example, the vector calculation unit 1707 can apply a linear function; or, a nonlinear function to the output of the operation circuit 1703, such as linear interpolation on the feature planes extracted by the convolutional layer, and again, for example, a vector of accumulated values to generate activation values. In some implementations, the vector calculation unit 1707 generates normalized values, pixel-level summation values, or both. In some implementations, the processed output vector can be used as an activation input to the operation circuit 1703, for example, for use in subsequent layers in the neural network.

[0375] The controller 1704 is connected to the instruction fetch buffer 1709, which is used to store instructions used by the controller 1704;

[0376] The unified memory 1706, the input memory 1701, the weight memory 1702, and the instruction memory 1709 are on-chip memories. The external memory is private to the NPU hardware architecture.

[0377] Any processor mentioned in the above can be a general central processing unit, a microprocessor, an ASIC, or one or more integrated circuits for controlling execution of the above programs.

[0378] In addition, it should be noted that the apparatus embodiments described above are merely illustrative, and the units described as separate components can or can not be physically separated, and the components shown as units can or can not be physical units, i.e., they can be located in one place or distributed on multiple network units. Part or all of the modules can be selected to achieve the purpose of the embodiment according to actual needs. In addition, the connection relationship between the modules in the apparatus embodiment provided in the present application indicates that there is a communication connection between them, which can be implemented as one or more communication buses or signal lines.

[0379] Through the description of the above embodiments, those skilled in the art can clearly understand that the present application can be implemented by means of software and the necessary general hardware, and of course it can also be implemented by special hardware including special integrated circuits, special CPUs, special memories, special components, etc. Generally, functions completed by computer programs can be easily implemented by corresponding hardware, and the specific hardware structure for implementing the same function can also be various, such as analog circuits, digital circuits or special circuits. However, for the present application, software program implementation is a better embodiment. Based on this understanding, the technical solutions of the present application can be embodied in the form of a software product, which is stored in a readable storage medium, such as a computer's floppy disk, U disk, mobile hard disk, ROM, RAM, magnetic disk or optical disk, etc., including a plurality of instructions for making a computer device (which can be a personal computer, a training device, or a network device, etc.) execute the methods described in various embodiments of the present application.

[0380] In the above embodiments, all or part can be implemented by software, hardware, firmware, or any combination thereof. When implemented by software, it can be implemented in the form of a computer program product in whole or in part.

[0381] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the processes or functions described in the embodiments of the present application are generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions can be transmitted from one website, computer, training device or data center to another website, computer, training device or data center through wired (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (such as infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that can be stored by the computer or a data storage device such as a training device, a data center, etc. integrated with one or more available media sets. The available media can be a magnetic medium (for example, a floppy disk, a hard disk, a magnetic tape), an optical medium (for example, a DVD), or a semiconductor medium (for example, a solid state disk (SSD)), etc.

Claims

1. A data processing method, characterized by, The method comprises: obtaining voice data; determining, from the voice data, a plurality of split point positions corresponding to predicted rhythm points of a body movement made by a character object when uttering the voice data, the rhythm points being critical points from stillness to movement or from movement to stillness when making the body movement, according to audio features of the voice data; obtaining a feature representation by a feature extraction network, according to the voice data and information indicating the plurality of split point positions; generating, by a motion generation network, motion data according to the feature representation; after the step of obtaining a feature representation by a feature extraction network, according to the voice data and information indicating the plurality of split point positions, the method further comprises: determining a motion category of the body movement according to the feature representation; the step of generating, by a motion generation network, motion data according to the feature representation comprises: generating, by a motion generation network, the motion data according to the feature representation and the motion category.

2. The method of claim 1, wherein, The audio features are related to volume and / or pitch.

3. The method according to claim 1 or 2, characterized in that, The plurality of split point positions comprises a first position and a second position adjacent to the first position, and a time interval between the second position and the first position is within a preset range; the first position and the second position are determined in one of the following ways: The first position and the second position are onset points in the voice data. The first position is an onset point in the voice data, and the second position is not an onset point in the voice data, and the second position is a time point in the voice data at which the volume is greater than a threshold.

4. The method according to claim 1 or 2, characterized in that, The voice data and the information indicating the plurality of split point positions comprise: a plurality of voice segments into which the voice data is divided according to the plurality of split point positions; or characters in the voice data and indicating the plurality of split point positions.

5. The method according to claim 1 or 2, characterized in that, The feature extraction network comprises a plurality of serial network layers, and the feature representation comprises feature representations output by the plurality of network layers; the method further comprises: determining a motion category of the body movement according to feature representations output by one or more network layers close to an output layer in the plurality of network layers; the step of generating, by a motion generation network, motion data according to the feature representation comprises: generating, by a motion generation network, the motion data according to part or all of the feature representations output by the plurality of network layers and the motion category.

6. The method of claim 1 or 2, wherein, The feature extraction network comprises a plurality of serial network layers, and the feature representation comprises feature representations output by the plurality of network layers; the method further comprises: obtaining a style code of the body movement by an encoder according to feature representations output by one or more network layers away from an output layer in the plurality of network layers; the step of generating, by a motion generation network, motion data according to the feature representation comprises: generating, by a motion generation network, the motion data according to part or all of the feature representations output by the plurality of network layers and the style code.

7. The method of claim 6, wherein, The distribution of the style code is uniform distribution.

8. A data processing method, characterized by, The method comprises: acquiring voice data of a character object and first action data, the first action data being real action data of a body action made by the character object when the voice data is uttered; determining, according to an audio feature of the voice data, a plurality of split point positions from the voice data, the split point positions corresponding to predicted rhythm points of a body action made by a character object when the voice data is uttered, a rhythm point being a critical point from stillness to movement or from movement to stillness when the body action is made; obtaining, according to the voice data and information indicating the plurality of split point positions, a feature representation through a feature extraction network; generating, according to the feature representation, second action data through an action generation network; a difference between the first action data and the second action data being used to update the feature extraction network and the action generation network; after the obtaining, according to the voice data and the information indicating the plurality of split point positions, the feature representation through the feature extraction network, the method further comprises: determining, according to the feature representation, an action category of the body action; the generating, according to the feature representation, the second action data through the action generation network, comprises: generating, according to the feature representation and the action category, the second action data through the action generation network.

9. The method of claim 8, wherein, the feature extraction network comprises a plurality of serial network layers, the feature representation comprises feature representations output by the plurality of network layers, and the generating, according to the feature representation, the second action data through the action generation network, comprises: generating, according to the feature representation and an initialized style code, the second action data through the action generation network; obtaining, according to feature representations output by one or more network layers of the plurality of network layers away from an output layer, a first style code of the body action through an encoder; updating the initialized style code through a preset loss function to obtain an updated style code; a difference between the first style code and the updated style code is used to update the encoder.

10. The method according to claim 8 or 9, characterized in that, the audio feature is related to volume and / or pitch.

11. The method according to claim 8 or 9, characterized in that, the plurality of split point positions comprises a first position and a second position adjacent to each other, a time interval between the second position and the first position is within a preset range, and the first position and the second position are determined in the following manner: the first position and the second position are onset points in the voice data; the first position is an onset point in the voice data, the second position is not an onset point in the voice data, and the second position is a time point at which volume in the voice data is greater than a threshold.

12. The method of claim 8 or 9, wherein, the voice data and the information indicating the plurality of split point positions comprise: a plurality of voice segments into which the voice data is divided according to the plurality of split point positions; or characters in the voice data and indicating the plurality of split point positions.

13. The method of claim 8 or 9, wherein, the feature extraction network comprises a plurality of serial network layers, the feature representation comprises feature representations output by the plurality of network layers, and the method further comprises: determine, according to the feature representation output by one or more network layers close to an output layer in the plurality of network layers, a motion category of the body motion; generate, according to the feature representation, second motion data by a motion generation network, including: generate, according to part or all of the feature representation output by the plurality of network layers and the motion category, second motion data by a motion generation network.

14. The method of claim 8 or 9, wherein, The feature extraction network includes a plurality of serial network layers, the feature representation includes feature representations output by the plurality of network layers, and the method further includes: obtain, according to the feature representation output by one or more network layers far from an output layer in the plurality of network layers, a style code of the body motion by an encoder; generate, according to the feature representation, second motion data by a motion generation network, including: generate, according to part or all of the feature representation output by the plurality of network layers and the style code, second motion data by a motion generation network.

15. A data processing apparatus, characterized by The apparatus includes: an acquisition module configured to acquire voice data; a processing module configured to determine, according to an audio feature of the voice data, a plurality of segmentation point positions from the voice data, the segmentation point positions corresponding to predicted rhythm points of a body motion made when the voice data is played, a rhythm point being a critical point from stillness to motion or from motion to stillness when the body motion is made; obtain, according to the voice data and information indicating the plurality of segmentation point positions, a feature representation by a feature extraction network; generate, according to the feature representation, motion data by a motion generation network; The processing module is further configured to: determine, according to the feature representation, a motion category of the body motion; When generating, according to the feature representation, motion data by a motion generation network, the processing module is specifically configured to: generate, according to the feature representation and the motion category, the motion data by a motion generation network.

16. The apparatus of claim 15, wherein, The audio feature is related to volume and / or pitch.

17. The apparatus of claim 15 or 16, wherein, The plurality of segmentation point positions includes adjacent first and second positions, a time interval between the second position and the first position is within a preset range; and the first and second positions are determined in one of the following ways: The first and second positions are onset points in the voice data; The first position is an onset point in the voice data, the second position is not an onset point in the voice data, and the second position is a time point in the voice data at which the volume is greater than a threshold.

18. The apparatus of claim 15 or 16, wherein, The voice data and the information indicating the plurality of segmentation point positions include: a plurality of voice segments into which the voice data is divided according to the plurality of segmentation point positions; or characters of the voice data and indicating the plurality of segmentation point positions.

19. The apparatus of claim 15 or 16, wherein, The feature extraction network includes a plurality of serial network layers, the feature representation includes feature representations output by the plurality of network layers, and the processing module is further configured to: determine, according to the feature representation output by one or more network layers close to an output layer in the plurality of network layers, a motion category of the body motion; The processing module is specifically configured to: The action data is generated by an action generation network according to part or all of the feature representations output by the plurality of network layers and the action category.

20. The apparatus of claim 15 or 16, wherein, The feature extraction network comprises a plurality of serial network layers, and the feature representations comprise feature representations output by the plurality of network layers. The style code of the body action is obtained by an encoder according to feature representations output by one or more network layers of the plurality of network layers that are away from an output layer. The processing module is specifically configured to: The action data is generated by an action generation network according to part or all of the feature representations output by the plurality of network layers and the style code.

21. The apparatus of claim 20, wherein, The distribution of the style code is a uniform distribution.

22. A data processing apparatus, characterized in that, The apparatus comprises: An acquisition module is configured to acquire speech data of a character object and first action data, the first action data being real action data of a body action made by the character object when the speech data is uttered. A processing module is configured to determine a plurality of segmentation point positions from the speech data according to audio features of the speech data, the segmentation point positions corresponding to predicted rhythm points of a body action made by the character object when the speech data is uttered, a rhythm point being a critical point from stillness to movement or from movement to stillness when the body action is made. A feature extraction network is used to obtain feature representations according to the speech data and information indicating the plurality of segmentation point positions. A second action data is generated by an action generation network according to the feature representations, and a difference between the first action data and the second action data is used to update the feature extraction network and the action generation network. The processing module is further configured to: An action category of the body action is determined according to the feature representations. The processing module is specifically configured to: The second action data is generated by an action generation network according to the feature representations and the action category.

23. The apparatus of claim 22, wherein, The feature extraction network comprises a plurality of serial network layers, and the feature representations comprise feature representations output by the plurality of network layers. The processing module is specifically configured to: A second action data is generated by an action generation network according to the feature representations and an initialized style code. A first style code of the body action is obtained by an encoder according to feature representations output by one or more network layers of the plurality of network layers that are away from an output layer. The initialized style code is updated by a preset loss function to obtain an updated style code.

24. The apparatus of claim 22 or 23, wherein, A difference between the first style code and the updated style code is used to update the encoder.

25. The apparatus of claim 22 or 23, wherein, The audio features are related to volume and / or pitch. The plurality of segmentation point positions comprise a first position and a second position adjacent to each other, a time interval between the second position and the first position being within a preset range. The first position and the second position are onset points in the speech data. The first position is an onset point in the voice data, the second position is not an onset point in the voice data, and the second position is a time point in the voice data at which a volume is greater than a threshold.

26. The apparatus of claim 22 or 23, wherein, The voice data and information indicating the plurality of split point positions include: A plurality of voice segments into which the voice data is divided according to the plurality of split point positions; or The voice data and characters indicating the plurality of split point positions.

27. The apparatus of claim 22 or 23, wherein, The feature extraction network includes a plurality of serially connected network layers, and the feature representation includes feature representations output by the plurality of network layers. The processing module is further configured to: Determine, according to feature representations output by one or more network layers close to an output layer in the plurality of network layers, a motion category of the body motion; The processing module is specifically configured to: Generate, according to part or all of the feature representations output by the plurality of network layers and the motion category, second motion data by a motion generation network.

28. The apparatus of claim 22 or 23, wherein, The feature extraction network includes a plurality of serially connected network layers, and the feature representation includes feature representations output by the plurality of network layers. The processing module is further configured to: Obtain, according to feature representations output by one or more network layers away from an output layer in the plurality of network layers, a style code of the body motion by an encoder; The processing module is specifically configured to: Generate, according to part or all of the feature representations output by the plurality of network layers and the style code, second motion data by a motion generation network.

29. A computer storage medium, comprising, The computer storage medium stores one or more instructions, which, when executed by one or more computers, cause the one or more computers to perform the operations of the method of any one of claims 1-14.

30. A computer program product, characterised in that, The computer storage medium stores one or more instructions, which, when executed by one or more computers, cause the one or more computers to perform the operations of the method of any one of claims 1-14.

31. A communication system comprising at least one processor, at least one memory; the processor, the memory are connected through a communication bus and complete mutual communication; The at least one memory is configured to store code; The at least one processor is configured to execute the code to perform the method of any one of claims 1-14.

31. A communication system comprising at least one processor, at least one memory; the processor, the memory are connected through a communication bus and complete mutual communication; The at least one memory is configured to store code; The at least one processor is configured to execute the code to perform the method of any one of claims 1-14.

Images