Animation generation method and device, electronic equipment and storage medium

Abstract

The invention discloses an animation generation method and device, electronic equipment and a storage medium, first information and second information input for a virtual digital object are acquired, and the virtual digital object comprises a plurality of skeleton key points; determining position information of each skeleton key point according to the first information and the second information; according to the position information of the skeleton key points, the skeleton posture of the virtual digital object is updated; and generating an animation of the virtual digital object according to the updated skeleton attitude. The animation design requirements of different users are met, the animation design quality is guaranteed, and meanwhile the flexibility of animation design is enhanced.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to an animation generation method, device, electronic device and storage medium (computer-readable storage medium). Background Art

[0002] With the development of information technology, intelligent interaction technology is no longer limited to dialogue. The application of animation generated by intelligent interaction has become increasingly widespread. In some scenarios, some fixed dynamic effects are set for users to choose from, so that users can choose different types of animations. Summary of the invention

[0003] In response to the above technical problems, embodiments of the present application provide an animation generation method, device, electronic device and storage medium.

[0004] In a first aspect, an embodiment of the present application provides an animation generation method, the method comprising:

[0005] Acquire first information and second information input for a virtual digital object, wherein the virtual digital object includes a plurality of skeleton key points;

[0006] Determine the position information of each of the skeleton key points according to the first information and the second information;

[0007] Update the skeleton posture of the virtual digital object according to the position information of the skeleton key points;

[0008] Animation of the virtual digital object is generated according to the updated skeleton posture.

[0009] In a second aspect, the present application provides an animation generation device, comprising:

[0010] An acquisition module, used to acquire first information and second information input for a virtual digital object, wherein the virtual digital object includes a plurality of skeleton key points;

[0011] A determination module, used to determine the position information of each of the skeleton key points according to the first information and the second information;

[0012] An updating module, used for updating the skeleton posture of the virtual digital object according to the position information of the skeleton key points;

[0013] The processing module is used to generate the animation of the virtual digital object according to the updated skeleton posture.

[0014] In a third aspect, the present application provides an electronic device, the electronic device comprising:

[0015] one or more processors;

[0016] Memory; and

[0017] One or more applications, wherein the one or more applications are stored in the memory and configured to be executed by the processor to implement the steps in any one of the animation generation methods.

[0018] In a fourth aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program is loaded by a processor to execute the steps in any one of the animation generation methods described above.

[0019] In a fifth aspect, the present application provides a computer product, which includes a computer program or instructions. When the computer program or instructions are executed by a processor, the steps of any animation generation method provided in the embodiments of the present application are implemented.

[0020] The animation generation method, device, electronic device and storage medium provided in the present application determine the position information of each skeletal key point of a virtual digital object by inputting first information and second information of the virtual digital object, and generate corresponding animation according to the position information of the skeletal key points, so as to meet the animation design requirements of different user inputs, ensure the quality of animation design and enhance the flexibility of animation design. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0022] Figure 1 A schematic diagram of an application environment of the animation generation method provided by an embodiment of the present application is shown;

[0023] Figure 2 A schematic diagram of one implementation flow of the animation generation method provided in the implementation scheme of the present application;

[0024] Figure 3 A schematic diagram of one application scenario of the animation generation method of this application;

[0025] Figure 4 A schematic diagram of a flow chart of one implementation scheme for determining position information in an animation generation method provided in an implementation scheme of the present application;

[0026] Figure 5 A schematic diagram of a flow chart of one implementation scheme for determining first information in the animation generation method provided in an embodiment of the present application;

[0027] Figure 6A schematic diagram of a flow chart of one implementation scheme for determining the second information in the animation generation method provided in an embodiment of the present application;

[0028] Figure 7 This is a schematic diagram of one of the implementation schemes of the information configuration page in the implementation scheme of this application;

[0029] Figure 8 A schematic diagram of one implementation process of the animation generation method provided in the implementation scheme of the present application;

[0030] Fig. 9 is a schematic diagram of the structure of an animation generating device provided in an embodiment of the present application;

[0031] Fig.10 It is a schematic diagram of the internal structure of the electronic device provided in the embodiment of the present application. DETAILED DESCRIPTION

[0032] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without making creative work are within the scope of protection of the present application. At the same time, in the description of the embodiments of the present application, the terms "first", "second", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance. Thus, the features defined as "first" and "second" may explicitly or implicitly include one or more features. In the description of the embodiments of the present application, the meaning of "multiple" is two or more, unless otherwise clearly and specifically defined.

[0033] The animation generation method in one embodiment of the present application can be run on a local electronic device or a server. When the game interaction method is run on a server, the method can be implemented and executed based on a cloud interaction system, wherein the cloud interaction system includes a server and a client device.

[0034] In order to better understand the animation generation method, device, electronic device and storage medium provided in the embodiments of the present application, the application environment applicable to the embodiments of the present application is first described below.

[0035] See also Figure 1 , Figure 1 A schematic diagram of an application environment of the animation generation method provided in an embodiment of the present application is shown. As an implementation mode, the animation generation method and the object control model training method provided in the embodiment of the present application can be applied to the same electronic device. The electronic device can be as follows: Figure 1The server 110 shown in the figure can be connected to the electronic device 120 via a network. The network is used to provide a medium for a communication link between the server 110 and the electronic device 120. The network can include various connection types, such as a wired communication link, a wireless communication link, etc., which are not limited in the embodiments of the present application. Optionally, in other embodiments, the electronic device can also be a smart phone, a laptop computer, etc.

[0036] It should be understood that Figure 1 The server 110, network and electronic device 120 are merely illustrative. Any number of servers, networks and electronic devices may be provided as required. For example, the server 110 may be a physical server or a server cluster composed of multiple servers, and the electronic device 120 may be a mobile phone, a tablet, a desktop computer, a laptop computer and the like. It is understood that the embodiments of the present application may also allow multiple electronic devices 120 to access the server 110 at the same time.

[0037] In some embodiments, the electronic device 120 can record the user's voice to obtain the user's audio data. Further, the electronic device 120 sends the user's audio data to the server 110 through the network. After the server 110 receives the user's audio data, the audio data can be processed by the animation generation method described in the embodiment of the present application.

[0038] As another implementation, the animation generation method and the object control model training method provided in the embodiment of the present application can be applied to different electronic devices. For example, the animation generation method is applied to electronic device A, and the object control model training method is applied to electronic device B. In addition, the animation generation method, the object control model training method and the animation generation method provided in the embodiment of the present application can be applied to the same electronic device. For example, the animation generation method and the object control model training method are applied to electronic device A, etc. The embodiment of the present application does not limit the electronic devices to which the above three methods are applied.

[0039] The following is a detailed description in conjunction with the accompanying drawings. In this embodiment, the execution subject is an electronic device as an example. It should be noted that the description order of the following embodiments is not intended to limit the preferred order of the embodiments. Although the logical order is shown in the flow chart, in some cases, the steps shown or described may be performed in an order different from that shown in the accompanying drawings.

[0040] The animation generation method of this embodiment can be applied to a scenario in which animation is generated based on multimedia data, for example, an object animation scene matching the multimedia data is generated based on multimedia data such as voice or text input by a user.

[0041] Among them, the object animation can be an animation corresponding to objects such as people, birds, and dogs. It can be understood that the character animation can include an animation of the whole body posture of the character or an animation of part of the character posture. It can be understood that the whole body posture animation is, for example, a whole body movement change effect of the character, and the posture of part of the character can include a hand posture change effect of the character, or a facial expression change effect of the character. The facial expressions of the character include laughing, crying, smiling, lip shape changes, etc.

[0042] In some scenarios, users input some information such as text or audio in order to generate animations corresponding to the information. The animations include the action effects of the virtual characters corresponding to the information, such as turning around, raising hands, etc. However, in the related art, a preset animation database is generally used, and each preset animation in the preset animation database is displayed during the interaction process for the user to select and determine, and the corresponding animation is formed based on the user's selection and the corresponding input information. In this way, the animation effect is single and fixed, and the correlation between the animation and the user input information is low, and the animation generation effect is poor.

[0043] Therefore, the present application provides an animation generation method, which determines the position information of each skeleton key point of the virtual digital object by inputting first information and second information of the virtual digital object, and generates corresponding dynamic effects (dynamic effects can be referred to as animations) according to the position information of the skeleton key points, wherein the virtual digital object corresponds to a basic model of the animation to be designed, which can be a character model. In this scheme, a basic posture model corresponding to the virtual digital object is preset, and the position information of multiple three-dimensional key points corresponding to the virtual digital object is determined according to the multimedia data and style information, and then the virtual digital object is driven to generate a coherent object animation corresponding to the style information corresponding to the multimedia data, so that the obtained target animation has a higher matching degree, and at the same time meets the animation design needs of different users, ensuring the quality of animation design while enhancing the flexibility of animation design.

[0044] For details, please refer to Figure 2 , Figure 2 The following is a flow chart of one implementation of the animation generation method provided in the implementation scheme of the present application. The specific process may be as follows: Step S201 to Step S204, wherein:

[0045] S201, obtaining first information and second information input for a virtual digital object, wherein the virtual digital object includes a plurality of skeleton key points.

[0046] The first information is extracted based on multimedia data input for the virtual digital object, and is used to indicate the posture of the virtual digital object; the second information is style information, and is used to indicate the action style of the virtual digital object corresponding to the multimedia data.

[0047] Specifically, the multimedia data may include at least one of audio data and text data. Furthermore, the audio data may be music, voice corresponding to a user's speech, etc., and the multimedia data may be data collected in real time or data collected and uploaded from a local database. This application does not make any specific limitations.

[0048] Specifically, the type of movement amplitude or movement frequency corresponding to the action is determined according to the action style. That is, it can be understood that the action style can be classified into different numbers of action styles corresponding to different movement amplitudes or movement frequencies. Exemplarily, the action style corresponds to the gradual increase of the movement amplitude, including calm, happy, excited, crazy and other styles in turn.

[0049] Among them, the virtual digital object is the basic model of the animation to be designed, which can be a character model, or a model of other objects other than characters, such as a kitten, a puppy, and a grass. It can be understood that the virtual digital object can be represented by a data point cloud composed of different three-dimensional position coordinates and stored in the cloud or server. In some implementation schemes, a visual model of the virtual digital object can also be rendered based on the data point cloud of the virtual digital object for user selection.

[0050] It can be understood that the virtual digital object includes a plurality of skeleton key points, and the posture corresponding to the virtual digital object can be controlled to change by adjusting the position information of each skeleton key point.

[0051] Specifically, after receiving a user's animation generation trigger operation based on a virtual digital object, this solution can obtain the first information and the second information, that is, obtain the multimedia data and style information to be processed, and obtain the first information corresponding to each time unit by extracting the multimedia data corresponding to each time unit, and determine the corresponding relationship between the style information and each time unit.

[0052] It can be understood that the time unit can be 1 minute, 1 second or 3 seconds, etc. In the animation generation scenario, the user can also set the animation duration for the input multimedia data, that is, the duration of the animation generated by the multimedia data. If the multimedia data is audio data, the audio duration can also be directly used as the animation duration.

[0053] It can be understood that all first information can correspond to one or more second information, that is, one multimedia data can correspond to one style information or different style information. For example, in a scenario where all first information can correspond to two second information, the first half of a multimedia data corresponds to calm style information, and the second half of the multimedia data corresponds to pleasant style information. This application does not make any specific limitations.

[0054] For example, see Figure 3 , Figure 3 This is a schematic diagram of one application scenario of the animation generation method of this application. In the animation generation application scenario of this application, see Figure 3 3-1, based on the electronic device displaying a selectable virtual digital object visualization model, exemplarily, the visualization model of the virtual digital object may include a puppy visualization model 301, a human visualization model 302, and a Bull Demon King cartoon character visualization model 303, the user triggers the animation generation for the virtual digital object by single-clicking or double-clicking, thereby triggering the acquisition of the first information and the second information input for the virtual digital object, the virtual digital object includes a plurality of skeletal key points, that is, see Figure 3 3-2, the user clicks on the character visualization model 302, that is, triggers the information collection window 304 to pop up (see Figure 3 3-3), the user can collect real-time voice data or input text data as multimedia data through the information collection window, or access the local database or cloud database of the electronic device through the information collection window to obtain multimedia data, which is the multimedia data corresponding to the character visualization model. For example, the user can trigger Figure 3 Any one of the voice data control 305, text data collection control 306, local database access control 307 and cloud database access control 308 in 3-3 can realize the collection of real-time voice data, input text data, local data or cloud data. It can be understood that local data and cloud data can include audio data and text data, respectively, and this application does not make specific limitations.

[0055] It is understandable that after obtaining the multimedia data, style information can be further configured according to the multimedia data, and the configured style information can be used as the second information; illustratively, the style information can be configured on a configuration page of the multimedia data generated corresponding to the acquired multimedia data, or, after collecting the multimedia data, a style information configuration window can be popped up by long pressing the character visualization model, so that the user can configure the style information of the virtual digital object, and the style information configuration window is based on the multimedia data and the style information input corresponding to the virtual digital object.

[0056] For example, in one embodiment of the present application, after the user triggers the acquisition of multimedia data of the character visualization model (virtual digital object) through the above display page, by clicking Figure 3The text collection mark in 3-3 in , inputs text data as multimedia data, and further configures the style information corresponding to the multimedia data, and then can further pop up an animation duration collection window in the form of a pop-up window to collect the animation duration.

[0057] Optionally, in some embodiments of the present application, the target animation duration can be determined by analyzing the data duration of the multimedia data, and then the time interval between each action feature in the action feature sequence can be configured according to the target animation duration to control the action change rate between each action of the virtual digital object. Exemplarily, in one embodiment of the present application, the multimedia data is audio data, and the data duration of the audio data can be set as the target animation duration.

[0058] It is understandable that the above-mentioned operation information such as long press, click, double click, etc. is only for example, and does not limit the operation method of triggering the acquisition of the first information and the second information corresponding to the input of the virtual digital object.

[0059] S202: Determine the position information of each of the skeleton key points according to the first information and the second information.

[0060] The position information is the coordinates of each key point of each skeleton key point in the point cloud coordinate system corresponding to the point cloud data corresponding to the virtual digital object.

[0061] Specifically, in the implementation scheme of the present application, posture information of a virtual digital object is obtained for each time unit, and the posture is represented by the position information of each skeleton key point in the virtual digital object.

[0062] Specifically, in one implementation scheme of the present application, after acquiring the first information and the second information, the first information and the second information are fused, and the position information of each of the skeleton key points corresponding to the time unit is determined based on the fused first information and the second information, that is, the posture information of the virtual digital object of the time unit is obtained.

[0063] S203: Update the skeleton posture of the virtual digital object according to the position information of the skeleton key points.

[0064] Specifically, in the implementation scheme of the present application, after obtaining the position information of the skeletal key points corresponding to each time unit, for each time unit, the skeletal key points corresponding to the digital virtual object are controlled to move to the position corresponding to the position information to realize the skeletal posture update of the virtual digital object.

[0065] S204: Generate an animation of the virtual digital object according to the updated skeleton posture.

[0066] Specifically, in the implementation scheme of the present application, after obtaining the skeletal posture corresponding to each time unit, the skeletal posture is sorted according to the size of the time unit to generate a skeletal posture sequence, and the skeletal posture sequence is dynamically processed to generate an animation corresponding to the animation duration, and the animation and multimedia information are aligned in time units and then fused to obtain the animation corresponding to the multimedia information matching.

[0067] That is, it can be understood that, according to the time sequence relationship between each action feature in the action feature sequence, each key in representing the virtual digital object can be controlled to be adjusted to the position corresponding to the action feature in turn, thereby realizing the dynamic processing of the virtual digital object and obtaining an animation.

[0068] Optionally, in some of the embodiments of the present application, each unit time in the action feature sequence may be configured to control the action change rate between the actions corresponding to the virtual digital object.

[0069] Exemplarily, in some other embodiments of the present application, the target animation duration can also be corrected according to the style information of the multimedia data. Exemplarily, if the style information corresponds to a large movement amplitude of the virtual digital object, the animation duration can be appropriately extended to ensure the smoothness of the dynamic effect of the virtual digital object.

[0070] The scheme of this embodiment obtains multimedia data corresponding to the virtual digital object to be controlled, and style information of the virtual digital object, determines first information and second information corresponding to each time unit according to the multimedia data and style information, determines position information of each of the skeleton key points according to the first information and second information; updates the skeleton posture of the virtual digital object according to the position information of the skeleton key points; generates animation of the virtual digital object according to the updated skeleton posture. In this scheme, a basic posture model corresponding to the virtual digital object is preset, and the position information of the skeleton key points corresponding to the virtual digital object is determined according to the multimedia data and style information, and then a coherent object animation of the virtual digital object corresponding to the multimedia data and style information is generated, so that the obtained animation has a higher matching degree with the multimedia data and style information, and at the same time meets the animation design needs of different users, ensures the quality of animation design, and enhances the flexibility of animation design.

[0071] Furthermore, the present application also provides a specific implementation scheme for obtaining location information, see Figure 4 , Figure 4 A schematic flow chart of one implementation scheme for determining position information in the animation generation method provided in the implementation scheme of the present application, specifically including steps S401-S402:

[0072] S401: Perform information fusion processing on the first information and the second information to obtain fused information.

[0073] Specifically, in the implementation scheme of the present application, by performing feature extraction and splicing on the first information and the second information respectively, a splicing feature is obtained as the fusion information.

[0074] Optionally, the style features can be extracted by a feature extraction model.

[0075] Optionally, the style feature can also be obtained by searching through index mapping. For example, in one embodiment of the present application, each input speech corresponds to a style index, and the style index is used to select a corresponding style vector from a style code as a style feature. The style code is a set of learnable parameter vectors that define style features corresponding to different style information.

[0076] In some implementation schemes of the present application, information fusion processing is performed on the first information and the second information to obtain fused information, which specifically includes the steps of:

[0077] (1) performing feature extraction on the first information and feature extraction on the second information to obtain first information features of the first information and second information features of the second information;

[0078] (2) performing information feature splicing on the first information feature and the second information feature to obtain fused information.

[0079] Optionally, in one implementation scheme of the present application, one multimedia data corresponds to one style information, that is, all first information features correspond to the same style feature, then each of the first information features is respectively spliced ​​with the style feature to obtain a fusion feature corresponding to each multimedia feature, as the fused information corresponding to the first information and the second information.

[0080] Optionally, in some other embodiments of the present application, one multimedia data corresponds to multiple style information. Then, based on the correspondence between the style information and the multimedia data, the second information features corresponding to each multimedia feature are determined, and the first information features and the second information features with the corresponding relationship are further spliced ​​to obtain a fused feature.

[0081] S402: Predict the position information of the skeleton key points according to the fused information to obtain the position information of each skeleton key point.

[0082] Specifically, the position information of each skeleton key point is obtained by inputting the fused information into a preset model to predict the position of each skeleton key point, that is, the position of each skeleton key point is predicted through the fused information of the preset model to obtain the action features corresponding to each fused feature.

[0083] It can be understood that the preset model is trained by a first information sample, a second information sample and a position information sample, the first information sample is obtained by extracting the audio information of the sample object in the video sample, the position information sample is obtained by extracting the position information of each bone key point of the sample object in the video sample, and the second information sample characterizes the position change amplitude between each position information in the position information sample and the initial position of each bone key point.

[0084] The initial positions of the skeleton key points are positions corresponding to the skeleton key points when the virtual digital object is in an initial posture, and the initial posture may be a preset posture.

[0085] This solution predicts the fusion feature sequence corresponding to the multimedia data and the style information through a preset model, reduces the amount of model data and processing, and enhances the reliability and accuracy of data processing.

[0086] In one embodiment of the present application, there is also provided an implementation scheme for obtaining the first information and the second information, which specifically includes the steps of:

[0087] (1) obtaining an animation time of the virtual digital object animation to be generated, wherein the animation time includes a plurality of time units;

[0088] (2) Acquire first information corresponding to each of the time units input for the virtual digital object, and acquire second information corresponding to each of the time units.

[0089] Specifically, the first information is obtained by extracting multimedia data inputted into the corresponding virtual digital object, and details are not described in detail above. The second information is determined based on the correspondence between the first information and each time unit.

[0090] Further, based on the above implementation scheme, the present application also provides a specific implementation scheme for determining the first information. In the implementation scheme of the present application, the multimedia data audio information, the first information is the extracted audio frame; specifically, see Figure 5 , Figure 5 A schematic flow chart of one implementation scheme for determining the first information in the animation generation method provided in the embodiment of the present application, specifically including steps S501-S503:

[0091] S501: Acquire audio information input for the virtual digital object.

[0092] The specific process is shown in the above implementation scheme.

[0093] S502: Perform signal conversion on the audio information to obtain a feature spectrum corresponding to the audio information.

[0094] The characteristic spectrum is used to characterize the frequency characteristics of the speech signal and is a digital representation of the speech signal.

[0095] Specifically, in the implementation scheme of the present application, after the electronic device collects multimedia data, if the multimedia data is audio data, such as input voice (audio), the audio data is converted into a series of continuous feature spectra through an audio feature extractor (AFE), that is, these feature spectra are the result of analyzing the frequency characteristics of the voice signal, which can be understood as a digital representation of the voice signal.

[0096] It can be understood that the audio feature extractor can be stored in the cloud or in the local memory of the electronic device, and by inputting the multimedia data into the audio feature extractor, the feature map corresponding to the multimedia data is output.

[0097] S503: Perform audio frame sampling on the feature map according to each of the time units to obtain an audio frame corresponding to each of the time units.

[0098] It is understandable that since the density of multimedia features is too high, the similarity of action features corresponding to adjacent multimedia features may be relatively large, that is, there are some unnecessary multimedia features. Therefore, this embodiment samples the vital sign map to reduce the feature map information while ensuring the completeness of the information, thereby improving data processing efficiency.

[0099] Exemplarily, in one embodiment of the present application, the feature spectrum can be sampled according to the sampling frequency determined by the time unit to obtain at least two spectrum frames as multimedia features. Exemplarily, if the time unit is three seconds, the sampling frequency is to collect one audio frame every 3 seconds, and the corresponding input speech audio features are collected from the spectrum features as the first information corresponding to each time unit.

[0100] It can be understood that the time unit can be set according to the animation generation accuracy of the virtual digital object, that is, the greater the animation generation accuracy, the more action features are required, the smaller the time unit is, the greater the sampling frequency is, and vice versa.

[0101] Optionally, in some implementation schemes of the present application, in order to reduce the amount of data for features, each audio frame is further compressed and encoded, and the timing information between the compressed and encoded multimedia features is analyzed, and the compressed first information features are arranged to obtain a feature sequence with a time order.

[0102] Optionally, compression encoding of audio frames can be achieved through an encoder, which is usually constructed by MLP or Conv1D. The encoder's task is to further encode and compress the feature map to generate a series of more compact encoded features. This step aims to reduce the dimensionality and complexity of the data while retaining important information.

[0103] Optionally, analyzing the timing information between audio frames can be achieved through a recurrent neural network (RNN), which processes the above multimedia features in sequence to capture the timing relationship between multimedia features and further determine the audio frame corresponding to the time unit. The output of the RNN is a series of sequential encoded features, which now contain the time series information of the input speech.

[0104] Further, based on any of the above implementation schemes, this application also provides an implementation scheme determined by the second information, for details, see Figure 6 , Figure 6 A schematic flow chart of one implementation scheme for determining the second information in the animation generation method provided in an embodiment of the present application, specifically including steps S601-S605:

[0105] S601: Acquire first initial information input for the virtual digital object, where the first initial information includes first information corresponding to each of the time units.

[0106] Among them, the first initial information is the multimedia data described in the above implementation scheme, and the specific acquisition process is not repeated here.

[0107] S602: Display an information configuration page including the first initial information.

[0108] Wherein, an information configuration page is generated and displayed according to the multimedia data, the information configuration page includes the multimedia data, and the information configuration page is displayed for the user to configure the information style for the multimedia data. Optionally, the information configuration page also includes an information configuration control.

[0109] S603: Determine second information corresponding to each information segment in the first initial information according to an operation of performing information configuration on the first information in the information configuration page.

[0110] Optionally, in some embodiments of the present application, the information configuration control includes a style index control, that is, the information configuration page includes multimedia data and a style index control, and the user triggers a drop-down list corresponding to the index control by clicking on the style index control, and the drop-down list includes a plurality of optional style information. The user further operates the drop-down list to select the style information corresponding to the multimedia information as the second information.

[0111] Further optionally, in some other embodiments of the present application, the information configuration control includes a style index control, multimedia data, and a separation mark point adding control corresponding to the multimedia data; see Figure 7 , Figure 7 This is a schematic diagram of an implementation interface of an information configuration page in the implementation scheme of the present application, wherein the information configuration page includes multimedia data 701 and also includes a separation mark point adding control 702 corresponding to the multimedia data, and by clicking the add control, a slidable mark point 703 is created in the multimedia data, and according to the corresponding position information of the mark point in the multimedia data, the multimedia data is divided into multiple information segments, and corresponding style information is configured for each information segment as the second information corresponding to the information segment.

[0112] S604: Determine the second information corresponding to each first information according to the information segment to which each first information belongs and the correspondence between each information segment and the second information.

[0113] Specifically, after obtaining the second information corresponding to each information segment, the second information corresponding to each first information is determined according to the information segment to which each first information belongs.

[0114] S605: Determine the second information corresponding to each of the time units according to the corresponding relationship between each of the first information and each of the time units, and the corresponding relationship between each of the first information and each of the second information.

[0115] This solution provides an information configuration page to facilitate users to configure multiple styles for media data, thereby improving the flexibility and diversity of style information configuration.

[0116] Further, based on the above implementation scheme, the present application also provides an implementation method for determining the second information corresponding to each information fragment in the first initial information according to the operation of configuring the first information in the information configuration page, which specifically includes the steps of:

[0117] (1) in response to a split operation on the first initial information in the information configuration page, updating the information configuration page, wherein the updated information configuration page includes the information segments corresponding to the split operation and an information configuration control for each of the information segments;

[0118] (2) Determine the second information corresponding to each of the information segments according to the operation information of the information configuration control.

[0119] For details, see Figure 7 In some other implementation schemes of the present application, the separator mark point adding control 702 is used to segment the first initial information, and a slidable mark point 703 is created in the multimedia data by clicking the adding control, and the multimedia data is divided into multiple information segments according to the corresponding position information of the mark point in the multimedia data. Specifically, after the user clicks the separator mark point adding control 702, a slidable mark point 703 is added to the multimedia data in the information configuration page, and a style feature configuration bar 704 corresponding to the newly added slidable mark point 703 is created, and the configuration bar includes a style index control 705. The user further clicks the style index control 705 to trigger a drop-down list corresponding to the index control 705 to pop up, and the drop-down list includes multiple optional style information. The user further operates the drop-down list to select the style information corresponding to the information segment corresponding to the newly added slidable mark point 703.

[0120] Specifically, referring to the above embodiment, the user configures the style information of the multimedia data through the information configuration page, see Figure 7 After the user clicks on the separator mark point to add a control 702, a new slidable mark point 703 is added to the multimedia data in the information configuration page, and a style feature configuration bar 704 corresponding to the newly added slidable mark point 703 is created, wherein the configuration bar includes a style index control 705. Based on the style index control, the user triggers a drop-down list corresponding to the index control 705 to pop up, wherein the drop-down list includes a plurality of optional style information. The user further operates the drop-down list, and then selects the style information corresponding to the information segment corresponding to the newly added slidable mark point 703 as the second information corresponding to the information segment.

[0121] Specifically, in the implementation scheme of the present application, each information fragment corresponds to a style index, which is used to select a corresponding style vector from a style code. The style code is a set of learnable parameter vectors that define different styles.

[0122] Further, based on any of the above implementation schemes, the present application also provides an implementation scheme for determining a preset model, which specifically includes the steps of:

[0123] (1) Obtain video samples.

[0124] The video sample includes position information of skeleton key points corresponding to the skeleton posture of the virtual digital object, and text information or audio data corresponding to the virtual digital object.

[0125] Specifically, the video sample may be an animation video, or a video captured by a camera, or other types of video such as a CG video, which is not specifically limited in this application.

[0126] It is understandable that the video sample can be stored in the electronic device or in the cloud corresponding to the electronic device, and this application does not make any specific limitation.

[0127] (2) sampling the video samples to obtain multimedia data frames and time-aligned video frames;

[0128] (3) extracting the position information (3D landmark) of the skeleton key points of the virtual digital object in each of the video frames as the position information sample, and determining the second information sample according to the position change between the corresponding position information of the same skeleton key points in adjacent video frames.

[0129] Exemplarily, in one implementation scheme of the present application, video frame sampling and audio frame sampling are performed on multimedia samples according to the same sampling frequency, and the audio frame is obtained as the multimedia data frame as the first information sample, and the position information (3D landmark) of the skeletal key points of the virtual digital object in the sampled video frame is identified as the position information sample, and according to the position change amplitude between the position information corresponding to the same skeletal key points between adjacent video frames, the style information corresponding to the position information is determined as the second information sample.

[0130] (4) Using the location information sample as a label, and the first information sample and the second information sample as input to train a preset training model to obtain a preset model.

[0131] Specifically, this solution performs training through corresponding information of video samples to ensure the accuracy of training.

[0132] Optionally, in some embodiments of the present application, updating the skeletal posture of the virtual digital object according to the position information of the skeletal key points includes: determining the motion trajectory of the skeletal key points in the virtual digital object based on the position information of the skeletal key points; controlling the virtual digital object to move according to the motion trajectory of the skeletal key points, and updating the skeletal posture of the virtual digital object. Specifically, the update can be achieved by controlling the virtual digital object to move according to the motion trajectory of the skeletal key points through a rotation estimator.

[0133] This scheme connects the position information of the skeleton key points corresponding to two adjacent skeleton postures through the rotation estimator, ensuring the accuracy and coordination of the motion generation of the virtual digital object and the accuracy of the animation generation.

[0134] Optional, see Figure 8 , Figure 8 A schematic diagram of one implementation process of the animation generation method provided for the implementation scheme of the present application. In the implementation scheme of the present application, for a segment of input speech audio, the speech is first converted into a continuous feature map through AFE. The feature map is further encoded and compressed by the encoder, and then input into an RNN unit, and the encoded features are output sequentially. Each audio corresponds to a style index styleindex, which is used to select the corresponding style vector style vector from the style code style code. Style code is a set of learnable parameter vectors, which also participates in the parameter update of the model during training. After the style code corresponding to each audio is selected, it will be spliced ​​with all the encoded features, and then input into the decoder decoder to obtain the final 3D landmark. The 3D landmark output by the decoder represents the movement of the 3D skeleton. By calculating the rotation information of the skeleton, the 3D landmark action generated by the audio and style code control can be directly used to drive the 3D human body model.

[0135] The animation generation method provided by the embodiment of the present scheme obtains first information and second information input for a virtual digital object, wherein the virtual digital object includes multiple skeletal key points; determines the position information of each of the skeletal key points according to the first information and the second information; updates the skeletal posture of the virtual digital object according to the position information of the skeletal key points; and generates the animation of the virtual digital object according to the updated skeletal posture. In the present scheme, the position information of each skeletal key point of the virtual digital object is determined by inputting the first information and the second information of the virtual digital object, and the corresponding animation is generated according to the position information of the skeletal key points, so as to meet the animation design requirements input by different users, ensure the quality of the animation design, and enhance the flexibility of the animation design.

[0136] It should be understood that, although each step in the flowcharts involved in the above-mentioned embodiments is displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0137] Based on the same inventive concept, the embodiment of the present application also provides an animation generation device for implementing the animation generation method involved above, and an animation generation device for implementing the animation generation method. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in the embodiments of one or more animation generation devices and animation generation devices provided below can refer to the above limitations on the animation generation method and animation generation method, and the specific limitations will not be repeated here.

[0138] This embodiment also provides an animation generation device, which can be integrated into an electronic device. Fig. 9 As shown, the animation generating device may include modules 1001-1004:

[0139] An acquisition module 1001 is used to acquire first information and second information inputted for a virtual digital object, wherein the virtual digital object includes a plurality of skeleton key points;

[0140] A determination module 1002, configured to determine the position information of each of the skeleton key points according to the first information and the second information;

[0141] An updating module 1003 is used to update the skeleton posture of the virtual digital object according to the position information of the skeleton key points;

[0142] The processing module 1004 is used to generate the animation of the virtual digital object according to the updated skeleton posture.

[0143] The above-mentioned animation generation device determines the position information of each skeleton key point of the virtual digital object based on the input first information and second information of the virtual digital object, and generates corresponding animation according to the position information of the skeleton key points, so as to meet the animation design requirements of different users, ensure the quality of animation design and enhance the flexibility of animation design.

[0144] In some embodiments, the determination module 1002 is used to determine the position information of each of the skeleton key points according to the first information and the second information, including:

[0145] Performing information fusion processing on the first information and the second information to obtain fused information;

[0146] The position information of the skeleton key points is predicted based on the fused information to obtain the position information of each skeleton key point.

[0147] In some embodiments, the determination module 1002 is used to perform information fusion processing on the first information and the second information to obtain fused information, including:

[0148] Performing feature extraction on the first information and feature extraction on the second information to obtain a first information feature of the first information and a second information feature of the second information;

[0149] The first information feature and the second information feature are concatenated to obtain fused information.

[0150] In some embodiments, the determination module 1002 is used to predict the position information of the skeleton key points according to the fused information to obtain the position information of each of the skeleton key points, including:

[0151] The fused information is input into a preset model to predict the position of each skeleton key point, so as to obtain the position information of each skeleton key point;

[0152] Among them, the preset model is trained by a first information sample, a second information sample and a position information sample, the first information sample is obtained by extracting the audio information of the sample object in the video sample, the position information sample is obtained by extracting the position information of each bone key point of the sample object in the video sample, and the second information sample represents the position change amplitude between each position information in the position information sample and the initial position of each bone key point.

[0153] In some embodiments, the acquisition module 1001 in the animation generation device is used to acquire the first information and the second information input for the virtual digital object, and further includes:

[0154] Acquire the animation time of the virtual digital object animation to be generated, wherein the animation time includes multiple time units;

[0155] First information corresponding to each of the time units input for the virtual digital object is obtained, and second information corresponding to each of the time units is obtained.

[0156] In some embodiments, the first information includes an audio frame, and the acquisition module 1001 is used to acquire the first information corresponding to each time unit input for the virtual digital object, including:

[0157] Acquiring audio information input for the virtual digital object;

[0158] Performing signal conversion on the audio information to obtain a feature spectrum corresponding to the audio information;

[0159] Audio frames are sampled for the feature map according to each of the time units to obtain audio frames corresponding to each of the time units.

[0160] In some embodiments, the acquisition module 1001 is used to acquire the second information corresponding to each time unit input for the virtual digital object, including:

[0161] Acquire the first initial information input for the virtual digital object, wherein the first initial information includes first information corresponding to each of the time units;

[0162] Displaying an information configuration page including the first initial information;

[0163] Determine, according to an operation of performing information configuration on the first information in the information configuration page, second information corresponding to each information segment in the first initial information;

[0164] Determine the second information corresponding to each first information according to the information segment to which each first information belongs and the correspondence between each information segment and the second information;

[0165] The second information corresponding to each of the time units is determined according to the corresponding relationship between each of the first information and each of the time units, and the corresponding relationship between each of the first information and each of the second information.

[0166] In some implementation schemes, the acquisition module 1001 is used to determine the second information corresponding to each information segment in the first initial information according to the information configuration operation of the first information in the information configuration page, including:

[0167] In response to a split operation on the first initial information in the information configuration page, updating the information configuration page, wherein the updated information configuration page includes information segments corresponding to the split operation and information configuration controls for each of the information segments;

[0168] The second information corresponding to each of the information segments is determined according to the operation information of the information configuration control.

[0169] In some embodiments, the updating module 1003 is used to update the skeleton posture of the virtual digital object according to the position information of the skeleton key points, including:

[0170] Based on the position information of the skeleton key points, determining the motion trajectory of the skeleton key points in the virtual digital object;

[0171] The virtual digital object is controlled to move according to the motion trajectory of the skeleton key points, and the skeleton posture of the virtual digital object is updated.

[0172] Based on the same inventive concept, the embodiment of the present application also provides a computer device, which may be a server or an electronic device, and includes a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps of the above animation generation method when executing the computer program. Thereby, various functions are realized, for example:

[0173] Acquire first information and second information input for a virtual digital object, wherein the virtual digital object includes a plurality of skeleton key points;

[0174] Determine the position information of each of the skeleton key points according to the first information and the second information;

[0175] Update the skeleton posture of the virtual digital object according to the position information of the skeleton key points;

[0176] Animation of the virtual digital object is generated according to the updated skeleton posture.

[0177] The specific implementation of the above operations can be found in the previous embodiments, which will not be described in detail here.

[0178] In one embodiment, the computer device is an electronic device, for example, its internal structure diagram can be as follows: Fig.10 As shown. The computer device includes a processor, a memory, an input / output interface, a communication interface, a display unit and an input device. The processor, the memory and the input / output interface are connected via a system bus, and the communication interface, the display unit and the input device are connected to the system bus via the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be implemented through WIFI, a mobile cellular network, NFC (near field communication) or other technologies. When the computer program is executed by the processor, an animation generation method is implemented. The display unit of the computer device is used to form a visually visible image, and can be a display screen, a projection device or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen. The input device of the computer device can be a touch layer covered on the display screen, or a button, trackball or touchpad set on the computer device casing, or an external keyboard, touchpad or mouse, etc.

[0179] Those skilled in the art will understand that Fig.10 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0180] Based on the same inventive concept, an embodiment of the present application also provides a computer-readable storage medium, which may include: a read-only memory (ROM), a random access memory (RAM), a disk or an optical disk, etc.

[0181] Since the computer program stored in the computer-readable storage medium can execute any one of the animation generation methods provided in the embodiments of the present application, the beneficial effects that can be achieved by any one of the animation generation methods provided in the embodiments of the present application can be achieved. Please refer to the previous embodiments for details and will not be repeated here.

[0182] Based on the same inventive concept, the embodiment of the present application further provides a computer program product or a computer program, which includes a computer instruction, and the computer instruction is stored in a computer-readable storage medium. The processor of the computer device reads the computer instruction from the computer-readable storage medium, and the processor executes the computer instruction, so that the computer device executes the method provided in various optional implementations in the above embodiments.

[0183] It should be noted that the object data (including but not limited to user device information, user personal information, etc.) and conversation data involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data need to comply with relevant laws, regulations and standards of relevant countries and regions. A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods.

[0184] Any reference to the memory, database or other media used in the embodiments provided in the present application may include at least one of non-volatile and volatile memory. Non-volatile memory may include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (Re RAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory may include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM may be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc.

[0185] The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. The non-relational database may include a distributed database based on blockchain, etc., but is not limited thereto. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but is not limited thereto.

[0186] In the above-mentioned animation generation device, computer-readable storage medium, computer equipment, and computer program product embodiments, the description of each embodiment has its own emphasis. For parts not described in detail in a certain embodiment, reference can be made to the relevant description of other embodiments. Those skilled in the art can clearly understand that for the convenience and simplicity of description, the specific working process and beneficial effects of the above-mentioned animation generation device, computer-readable storage medium, computer program product, computer equipment and its corresponding units can refer to the description of the animation generation method in the above embodiment, and will not be repeated here.

[0187] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0188] The above is a detailed introduction to an animation generation method, device, electronic device and storage medium provided in the embodiments of the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for technical personnel in this field, according to the idea of ​​the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. An animation generation method, It is characterized in that The method comprises: Acquire first information and second information input for a virtual digital object, wherein the virtual digital object includes a plurality of skeleton key points; Determine the position information of each of the skeleton key points according to the first information and the second information; Update the skeleton posture of the virtual digital object according to the position information of the skeleton key points; Animation of the virtual digital object is generated according to the updated skeleton posture.

2. The animation generation method according to claim 1, It is characterized in that Determining the position information of each of the skeleton key points according to the first information and the second information includes: Performing information fusion processing on the first information and the second information to obtain fused information; The position information of the skeleton key points is predicted based on the fused information to obtain the position information of each skeleton key point.

3. The animation generation method according to claim 2, It is characterized in that The performing information fusion processing on the first information and the second information to obtain fused information includes: Performing feature extraction on the first information and feature extraction on the second information to obtain a first information feature of the first information and a second information feature of the second information; The first information feature and the second information feature are concatenated to obtain fused information.

4. The animation generation method according to claim 2, It is characterized in that The step of predicting the position information of the skeleton key points according to the fused information to obtain the position information of each skeleton key point includes: The fused information is input into a preset model to predict the position of each skeleton key point, so as to obtain the position information of each skeleton key point; Among them, the preset model is trained by a first information sample, a second information sample and a position information sample, the first information sample is obtained by extracting the audio information of the sample object in the video sample, the position information sample is obtained by extracting the position information of each bone key point of the sample object in the video sample, and the second information sample represents the position change amplitude between each position information in the position information sample and the initial position of each bone key point.

5. The animation generation method according to claim 1, It is characterized in that The step of obtaining the first information and the second information inputted for the virtual digital object comprises: Acquire the animation time of the virtual digital object animation to be generated, wherein the animation time includes multiple time units; First information corresponding to each of the time units input for the virtual digital object is obtained, and second information corresponding to each of the time units is obtained.

6. The animation generation method according to claim 5, It is characterized in that The first information includes an audio frame; and the obtaining of the first information corresponding to each time unit input for the virtual digital object includes: Acquiring audio information input for the virtual digital object; Performing signal conversion on the audio information to obtain a feature spectrum corresponding to the audio information; Audio frames are sampled for the feature map according to each of the time units to obtain audio frames corresponding to each of the time units.

7. The animation generation method according to claim 5, It is characterized in that The acquiring the second information corresponding to each time unit input for the virtual digital object includes: Acquire the first initial information input for the virtual digital object, wherein the first initial information includes first information corresponding to each of the time units; Displaying an information configuration page including the first initial information; Determine, according to an operation of performing information configuration on the first information in the information configuration page, second information corresponding to each information segment in the first initial information; Determine the second information corresponding to each first information according to the information segment to which each first information belongs and the correspondence between each information segment and the second information; The second information corresponding to each of the time units is determined according to the corresponding relationship between each of the first information and each of the time units, and the corresponding relationship between each of the first information and each of the second information.

8. The animation generation method according to claim 7, It is characterized in that The determining, according to the operation of performing information configuration on the first information in the information configuration page, the second information corresponding to each information segment in the first initial information includes: In response to a split operation on the first initial information in the information configuration page, updating the information configuration page, wherein the updated information configuration page includes information segments corresponding to the split operation and information configuration controls for each of the information segments; The second information corresponding to each of the information segments is determined according to the operation information of the information configuration control.

9. The animation generation method according to claim 1, It is characterized in that The updating of the skeleton posture of the virtual digital object according to the position information of the skeleton key points includes: Based on the position information of the skeleton key points, determining the motion trajectory of the skeleton key points in the virtual digital object; The virtual digital object is controlled to move according to the motion trajectory of the skeleton key points, and the skeleton posture of the virtual digital object is updated.

10. An animation generating device, It is characterized in that The device comprises: An acquisition module, used to acquire first information and second information input for a virtual digital object, wherein the virtual digital object includes a plurality of skeleton key points; A determination module, used to determine the position information of each of the skeleton key points according to the first information and the second information; An updating module, used for updating the skeleton posture of the virtual digital object according to the position information of the skeleton key points; The processing module is used to generate the animation of the virtual digital object according to the updated skeleton posture.

11. An electronic device, It is characterized in that The electronic device comprises: one or more processors; Memory; and One or more applications, wherein the one or more applications are stored in the memory and are configured to be executed by the processor to implement the steps in the animation generation method according to any one of claims 1 to 9.

12. A computer-readable storage medium, It is characterized in that A computer program is stored thereon, and the computer program is loaded by a processor to execute the steps in the animation generation method according to any one of claims 1 to 9.

Images