Smooth model training method and device, and electronic equipment

Abstract

The application provides a smooth model training method and device and electronic equipment. In the embodiment, the prediction action type corresponding to each training sample is determined through the sample action data sequence composed of multiple key point data frames included in the training sample, the smooth training parameter corresponding to the training sample is determined according to the prediction action type, then the smooth model training is performed on the preset smooth model by using the smooth training parameter and the sample action data sequence, and the smooth model used for smooth processing of the action data sequence is obtained. In this way, the trained smooth model can automatically perform smooth processing of different degrees on the action data sequence of different action types, so as to balance the smooth effect and processing timeliness.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, in particular to a smoothing model training method and device and electronic equipment. BACKGROUND

[0002] With the rapid development of image processing technology, technologies such as data analysis, 3D modeling or interactive logic processing based on action key point data are applied in various fields. In some schemes, the action key point data is obtained by identifying and analyzing images or videos based on a neural network image recognition model. In such schemes, the image recognition model may have recognition bias or recognition errors when performing image recognition, resulting in data noise or data jitter in the obtained action key point data, which in turn affects subsequent processing actions. Therefore, it is usually necessary to use a smoothing model to smooth the action key point data obtained by the image recognition model.

[0003] In some smoothing processing schemes, a uniform smoothing coefficient is used to smooth the action key point data, but the uniform smoothing coefficient is usually difficult to take into account key point data of different action frequencies. A too large smoothing parameter may cause high frequency actions to lose action details due to smoothing processing, and a too small smoothing parameter may result in insufficient smoothing of low frequency actions. In other smoothing processing schemes, the smoothing weight can be dynamically adjusted according to the action key point data in a period of time, but the calculation amount is large and the response speed is not good. SUMMARY

[0004] In order to overcome the above-mentioned deficiencies in the prior art, one of the purposes of the present application is to provide a smoothing model training method, which comprises:

[0005] obtaining a plurality of training samples, wherein each training sample comprises a sample action data sequence composed of a plurality of action key point data frames;

[0006] determining a predicted action type corresponding to each training sample according to the sample action data sequence of the training sample, and determining a smoothing training parameter corresponding to the training sample according to the predicted action type;

[0007] using the smoothing training parameter and the sample action data sequence to perform model training on a preset smoothing model; the smoothing model is used for smoothing action data sequences.

[0008] In some possible implementation manners, the smoothing model comprises an encoder, a decoder and an action classification sub-model; and the step of determining a predicted action type corresponding to each training sample according to the sample action data sequence of the training sample, and determining a smoothing training parameter corresponding to the training sample according to the predicted action type, comprises:

[0009] For each of the training samples, processing a sample action data sequence of the training sample by the encoder to obtain an action feature corresponding to the training sample;

[0010] inputting the action feature into the action classification sub-model to obtain a predicted action type corresponding to the training sample, and determining a smoothing training parameter corresponding to the training sample according to the predicted action type.

[0011] In some possible implementation manners, the step of using the smoothing training parameter and the sample action data sequence to perform model training on a preset smoothing model comprises:

[0012] inputting the action feature of each of the training samples into a decoder of the smoothing model to obtain a predicted action data sequence corresponding to each of the training samples;

[0013] determining a first loss function value according to the sample action data sequence, the predicted action data sequence and the smoothing training parameter of each of the training samples, and adjusting model parameters of the encoder and the decoder of the smoothing model according to the first loss function value.

[0014] In some possible implementation manners, the first loss function value is:

[0015]

[0016] wherein, T is a number of action key point data frames of the training sample, is a t-th action key point data frame in the sample action data sequence, is a t-th action key point data frame in the predicted action data sequence;

[0017] B is a number of the training samples, is a smoothing training parameter corresponding to a b-th training sample.

[0018] In some possible implementation manners, the step of determining the smoothing training parameter corresponding to the training sample according to the predicted action type comprises:

[0019] querying a smoothing training parameter corresponding to the predicted action type according to a preset smoothing training parameter correspondence table;

[0020] wherein, the higher the action frequency corresponding to the predicted action type is, the larger the smoothing training parameter is.

[0021] In some possible implementation manners, the training sample further comprises an action type label; and the method further comprises:

[0022] According to the action type label and the predicted action type of the training sample, a second loss function value is determined for each training sample;

[0023] The model parameters of the action classification sub-model are adjusted according to the second loss function value.

[0024] In some possible implementation manners, the second loss function value is:

[0025]

[0026] wherein N is the number of the predicted action types, is a predicted probability value corresponding to the i-th predicted action type of the training sample, is a one-hot code vector corresponding to the action type label.

[0027] In some possible implementation manners, the method further includes:

[0028] obtaining a to-be-processed action data sequence composed of a plurality of action key point data frames;

[0029] inputting the to-be-processed action data sequence into the adjusted smoothing model;

[0030] processing the to-be-processed action data sequence through the encoder and the decoder of the smoothing model to obtain a smoothed action data sequence.

[0031] In some possible implementation manners, the trained smoothing model is configured in a server of a live broadcast platform, the server further includes an image recognition model and a virtual image generation model, and the method further includes:

[0032] obtaining live video data, and identifying action key point data of a host from the live video data through the image recognition model;

[0033] performing smoothing processing on the action key point data through the trained smoothing model;

[0034] generating a virtual image corresponding to a host action of the host according to the smoothed action key point data through the virtual image generation model;

[0035] generating stream data according to the virtual image and pushing the stream data to a viewer terminal.

[0036] Another purpose of the present application is to provide a smoothing model training device, the smoothing model training device comprising:

[0037] a data acquisition module, configured to acquire a plurality of training samples, wherein each of the training samples comprises a sample action data sequence composed of a plurality of action key point data frames;

[0038] an action prediction module, configured to determine a predicted action type corresponding to each of the training samples according to the sample action data sequence of the training sample, and determine a smoothing training parameter corresponding to the training sample according to the predicted action type;

[0039] a model training module, configured to perform model training on a preset smoothing model using the smoothing training parameter and the sample action data sequence; the smoothing model is configured to perform smoothing processing on an action data sequence.

[0040] Another purpose of the present application is to provide an electronic device comprising a machine readable storage medium and one or more processors, the machine readable storage medium stores machine executable instructions, when the machine executable instructions are executed by the one or more processors, the smoothing model training method provided by the present application is implemented.

[0041] Compared with the prior art, the present application has the following beneficial effects:

[0042] The smoothing model training method, device and electronic device provided by the present application, by predicting the action type of the training sample and determining the smoothing training parameter corresponding to the action type in the process of training the smoothing model, and then combining the smoothing training parameter and the sample action data sequence in the training sample to train the smoothing model. In this way, the trained smoothing model can automatically perform smoothing processing of different degrees on the to-be-processed action data sequence of different action types, so as to balance the smoothing effect and processing timeliness. BRIEF DESCRIPTION OF DRAWINGS

[0043] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the embodiments. It should be understood that the following drawings only show some embodiments of the present application, and therefore should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can also be obtained without creative labor on the basis of these drawings.

[0044] Figure 1a One of the use scenarios of the smoothing model provided by the embodiments of the present application;

[0045] Figure 1b Another use scenario of the smoothing model provided by the embodiments of the present application;

[0046] Figure 2 One of the use scenarios of the smoothing model provided by the embodiments of the present application;

[0047] Figure 3 A structural schematic diagram of the smoothing model provided for the embodiment of the present application is shown in FIG. 1.

[0048] Figure 4 A sub-step flowchart of step S120 provided for the embodiment of the present application is shown in FIG. 2.

[0049] Figure 5 A sub-step flowchart of step S130 provided for the embodiment of the present application is shown in FIG. 3.

[0050] Figure 6 A second schematic diagram of the smoothing model training method provided for the embodiment of the present application is shown in FIG. 4.

[0051] Figure 7 A block schematic diagram of the server provided for the embodiment of the present application is shown in FIG. 5.

[0052] Figure 8 A functional module schematic diagram of the smoothing model training apparatus provided for the embodiment of the present application is shown in FIG. 6. DETAILED DESCRIPTION

[0053] In order to make the objectives, technical solutions and advantages of the embodiments of the present application clearer, the following will be combined with the accompanying drawings for the embodiments of the present application to make a clear and complete description of the technical solutions of the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, but not all the embodiments. The components of the embodiments of the present application described and shown in the accompanying drawings can be arranged and designed in various different configurations.

[0054] Therefore, the following detailed description of the embodiments of the present application provided in the accompanying drawings is not intended to limit the scope of the claimed present application, but only represents selected embodiments of the present application. All other embodiments obtained by those of ordinary skill in the art based on the embodiments in the present application without making creative efforts are within the scope of protection of the present application.

[0055] It should be noted that: similar reference numbers and letters represent similar items in the following drawings, therefore, once an item is defined in one drawing, it does not need to be further defined and explained in the subsequent drawings.

[0056] In the description of the present application, the terms “first”, “second”, “third” and the like are only used to distinguish the description, and cannot be understood as indicating or implying relative importance.

[0057] In the description of the present application, it also needs to be explained that, unless otherwise explicitly specified and limited, the terms "set", "install", "connect", "connect" should be understood broadly, for example, it can be fixedly connected, or it can be detachably connected, or integrally connected; it can be mechanically connected, or it can be electrically connected; it can be directly connected, or it can be indirectly connected through an intermediate medium, or it can be the communication inside two elements. For those skilled in the art, the specific meaning of the above terms in the present application can be understood according to the specific circumstances.

[0058] The inventor found that in some existing smoothing schemes, a uniform smoothing parameter can be used to smooth the action key point data output by the image recognition model. For example, in a sliding window smoothing scheme, for time t, the result output by the image recognition model is Pt, and the final output result of the smoothing model at time t-1 (historical time) is Ft-1, then the final output result Ft of the smoothing model at the current time is:

[0059]

[0060] wherein, is a smoothing parameter, The greater the value, the closer the final result to the direct output result of the image recognition model, which is reflected in the faster response of the final result, but there is a certain jitter; The smaller the value, the closer the result to the historical output result of the smoothing model, which is reflected in the slower response of the result, but it is very stable.

[0061] Since the smoothing parameter is a fixed value, it is difficult to determine a suitable smoothing parameter that can have good adaptability to action key point data of different action frequencies. A too large smoothing parameter will cause high-frequency action to lose action details due to smoothing processing, for example, for a high-frequency waving action, a too large smoothing parameter will cause the hand key point data after smoothing processing to only remain a slight shaking at the middle position of the entire waving range; and a too small smoothing parameter will cause it to be unable to eliminate obvious data noise or data jitter in low-frequency action, thereby resulting in poor smoothing effect.

[0062] In another common smoothing processing scheme, the smoothing weight can be dynamically adjusted according to the action key point data in a period of time. For example, a polynomial smoothing algorithm (Savitzky-Golay, SG algorithm) is adopted, a window size is first determined, a polynomial function is fitted according to the sequence values in the window, then the parameters of the polynomial are solved by matrix, and finally the polynomial equation is obtained for data processing to obtain the smoothed result. The essence of this scheme is to linearly combine each point in the window, and use different weights to obtain the final smoothing result. However, in this smoothing scheme, each frame of action key point data of any action needs to go through a large amount of data processing process, and it is impossible to dynamically adjust the weight according to different actions to obtain the smoothed result, resulting in poor overall timeliness, especially in the smoothing processing of high-frequency actions with high processing timeliness requirements, the processing result has obvious lag.

[0063] Therefore, the embodiment provides a smoothing model training method, device and electronic equipment capable of training a smoothing model that can identify action types to automatically adjust the smoothing degree. The scheme provided by the embodiment is explained in detail below.

[0064] In some possible implementation manners, the smoothing model training method provided by the embodiment can be used to train a preset smoothing model 100. Please refer to Figure 1a The smoothing model 100 can be used to perform smoothing processing on the action key point data obtained by an image recognition model 200 performing image recognition on image data (such as live video data). For example, the image recognition model 200 performs key point recognition on image or video data to obtain action key point data, and then the smoothing model 100 performs smoothing processing on the action key point data to remove data noise or data jitter in the action key point data, and obtain smoothed action key point data. The smoothed action key point data can be used for subsequent data processing actions such as data analysis, 3D modeling or interactive interaction.

[0065] Taking some live scenes as an example, the smoothing model 100 can be applied to a virtual image synchronization scheme. Please refer to Figure 1b In this scenario, the image recognition model 200, the smoothing model 100 and the virtual image generation model 300 can be configured in the server 10 of the live platform. The server 10 can obtain live video data from the host terminal 20, then recognize the action key point data of the host from the live video data through the image recognition model 200, perform smoothing processing on the action key point data through the smoothing model 100, then generate a virtual image corresponding to the action of the host through the virtual image generation model 300 according to the smoothed action key point data, and then obtain live streaming data according to the virtual image and push it to the audience terminal 30.

[0066] In one example, the image recognition model 200 can recognize 2D action key point data from live video images, the virtual image generation model 300 can perform 3D action modeling according to the smoothed 2D action key point data, and then control the 3D virtual image to perform corresponding actions according to the obtained 3D action data; in another example, the image recognition model 200 can also directly recognize 3D action key point data from live video data, and then the smoothing model 100 performs smoothing processing on the 3D action key point data.

[0067] It can be understood that in the live scene using virtual images, the timeliness of following the actions of virtual images has a high requirement, and therefore the smoothing model 100 needs to consider both the smoothing effect and the timeliness of smoothing processing.

[0068] Based on the above scenario, please refer to Figure 2 The embodiment provides a smoothing model training method for training a smoothing model, and each step of the method is described in detail below.

[0069] In step S110, a plurality of training samples are obtained, wherein each training sample includes a sample action data sequence composed of a plurality of action key point data frames.

[0070] In the embodiment, the sample action data sequence can be composed of a plurality of action key point data frames, and each action key point data frame can include position data of a plurality of limb key points. The position data of the limb key points can be position data in a 2D image or position data in a 3D space.

[0071] Optionally, in the embodiment, the sample action data sequence can be obtained by image recognition of a video image by an image recognition model, and the sample action data sequence can be an action data sequence with a required smoothing degree after screening.

[0072] In step S120, a predicted action type corresponding to each training sample is determined according to the sample action data sequence of the training sample, and a smoothing training parameter corresponding to the training sample is determined according to the predicted action type.

[0073] In step S130, a preset smoothing model is trained using the smoothing training parameter and the sample action data sequence.

[0074] In the embodiment, the smoothing model can perform action type recognition according to the input sample action data sequence, so as to obtain the predicted action type corresponding to the training sample and determine the smoothing training parameter corresponding to the predicted action type. Then, the loss function value of the smoothing model is determined in combination with the smoothing training parameter and the sample action data sequence, and the model parameter of the smoothing model is adjusted according to the loss function value.

[0075] In this way, the trained smoothing model can automatically perform smoothing processing of different degrees for the to-be-processed action data sequence of different action types, so as to further take into account the processing timeliness while ensuring the smoothing processing effect.

[0076] In some possible implementation manners, referring to Figure 3 , the smoothing processing model can include an encoder, a decoder and an action classification sub-model. In the embodiment, the action classification sub-model can be regarded as a branch sub-model added in the conventional encoder-decoder model. The input of the action classification sub-model is the output of the encoder. The action classification sub-model can classify the training sample according to the abstract feature data extracted by the encoder, so as to obtain the predicted action type of the training sample. The input of the decoder is the output of the encoder. Therefore, in the embodiment, the loss function value can be determined according to the sample action data sequence input into the encoder, the predicted action data sequence output by the decoder and the smoothing training parameter determined by the action classification sub-model, and the smoothing model is adjusted according to the loss function value until the trained smoothing model meets the convergence condition. For example, the convergence condition can be that the loss function value no longer changes or the training times reach a preset training times, and the embodiment is not specifically limited.

[0077] Optionally, referring to Figure 4 , the step S120 can include the following sub-steps.

[0078] In the sub-step S121, for each training sample, the sample action data sequence of the training sample is processed by the encoder to obtain the action feature corresponding to the training sample.

[0079] In the embodiment, the encoder can include a first number of fully connected networks connected in sequence, and the encoder can perform encoding processing on the sample action data sequence to obtain the action feature of the training sample.

[0080] In the step S122, the action feature is input into the action classification sub-model to obtain the predicted action type corresponding to the training sample, and the smoothing training parameter corresponding to the training sample is determined according to the predicted action type.

[0081] In the embodiment, the action classification sub-model can include a convolutional layer and a classification network, and the action classification sub-model is trained to classify the action features output by the encoder to obtain the smooth training parameter corresponding to the training sample.

[0082] Optionally, referring to Figure 5 , the step S130 can include the following sub-steps.

[0083] In step S131, the action features of each training sample are input into the decoder of the smoothing model respectively to obtain a predicted action data sequence corresponding to each training sample.

[0084] In the embodiment, the decoder can include a second number of sequentially connected fully connected networks, and the decoder can perform decoding processing on the action features output by the encoder to obtain a predicted action data sequence corresponding to the training sample, the predicted action data sequence being a smoothed action data sequence. The second number can be equal to the first number.

[0085] In step S132, a first loss function value is determined according to the sample action data sequence, the predicted action data sequence and the smooth training parameter of each training sample, and the model parameters of the encoder and the decoder of the smoothing model are adjusted according to the first loss function value.

[0086] In the embodiment, the predicted action data sequence is the result of smoothing processing on the sample action data sequence, the smooth training parameter is related to the predicted action type determined according to the sample action data sequence, and the smooth training parameter can represent the smoothness of the predicted action type corresponding to the training sample. In this way, by pre-setting the correspondence between the smooth training parameter and the predicted action type, the smoothing model can learn to determine the action type from the action data sequence in the training process, and then use the appropriate smoothness for smoothing processing.

[0087] Specifically, in one possible implementation, the first loss function value is:

[0088]

[0089] wherein T is the number of action key point data frames of the training sample, is the tth action key point data frame in the sample action data sequence, is the tth action key point data frame in the predicted action data sequence;

[0090] B is the number of training samples, The smoothing training parameter corresponding to the bth training sample. In this embodiment, multiple batches of training samples can be used for training, and each batch uses multiple training samples to obtain a first loss function value to adjust the parameters of the model. Through multiple rounds of adjustment of the model parameters in multiple training batches, the final smoothing model is obtained.

[0091] Further, in this embodiment, a smoothing training parameter correspondence table can be preconfigured, which records different smoothing training parameters corresponding to different predicted action types. In step S120, the smoothing training parameter corresponding to the predicted action type can be queried according to the pre-set smoothing training parameter correspondence table. The higher the action frequency corresponding to the predicted action type, the larger the smoothing training parameter.

[0092] For example, taking the ordinary action, waving, clapping, slow motion gesture, and large hand raising in the predicted action type as an example, the corresponding smoothing training parameter can be set according to the following table according to the action frequency.

[0093]

[0094] Among them, the "slow motion gesture", "ordinary action", "wave", "large hand raising" and "clap" in the predicted action type correspond to different action frequencies respectively, and the action frequencies increase in turn. As can be seen, the higher the action frequency, the larger the corresponding smoothing training coefficient a, so that in the training process, the smoothing model can learn that under the condition of meeting a certain loss function value, when processing this type of input data, the difference between the predicted action data sequence and the sample action data sequence can be reduced, thereby reducing the smoothing degree and improving the processing efficiency. The lower the action frequency, the smaller the corresponding smoothing training coefficient a, so that in the training process, the smoothing model can learn that under the condition of meeting a certain loss function value, when processing this type of input data, the difference between the predicted action data sequence and the sample action data sequence can be increased, thereby increasing the smoothing degree.

[0095] In a possible implementation, the smoothing model provided by the embodiment includes three fully connected networks connected in sequence. According to the connection relationship, the input and output relationship of the three fully connected networks is as follows: inputting N-dimensional data and outputting 512-dimensional data, inputting 512-dimensional data and outputting 512-dimensional data, and inputting 512-dimensional data and outputting 256-dimensional data. The decoder can also include three fully connected networks connected in sequence. According to the connection relationship, the input and output relationship of the three fully connected networks is as follows: inputting 256-dimensional data and outputting 512-dimensional data, inputting 512-dimensional data and outputting 512-dimensional data, and inputting 512-dimensional data and outputting N-dimensional data. N is the dimension of the input and output data required in the actual situation. For example, a sample action data sequence of a training sample includes 8 action key point data frames, and each action key point data frame includes X-Y axis coordinate data of 7 limb key points. Therefore, N is 112.

[0096] In some possible implementations, the training sample further includes an action type label. In the training process, the action classification submodel of the smoothing model can also be trained. Specifically, referring to Figure 6 The method provided by the embodiment can further include the following steps.

[0097] In step S210, for each training sample, a second loss function value is determined according to the action type label of the training sample and the predicted action type.

[0098] In the embodiment, the training sample processing can include a sample action data sequence and an action type label. The action type label can be in the form of a one-hot code vector.

[0099] In step S220, the model parameters of the action classification submodel are adjusted according to the second loss function value.

[0100] In the training process, the action classification submodel outputs a predicted action type. The second loss function value is determined according to the predicted action type and the action type label, and the model parameters of the action classification submodel are adjusted. In this way, the predicted action type output by the action classification submodel gradually approaches the action type label of the training sample.

[0101] Specifically, in a possible implementation, the second loss function value is as follows:

[0102]

[0103] N is the number of the predicted action types, is a predicted probability value corresponding to the i th predicted action type of the training sample, is a one-hot code vector corresponding to the action type label.

[0104] Thus, the smaller the similarity between the predicted action type and the action type label, the greater the second loss function value; the greater the similarity between the predicted action type and the action type label, the smaller the second loss function value. Based on this, the action classification sub-model can be improved through multiple training to determine the similarity between the predicted action type of the training sample and the action type label, thereby improving the accuracy of the predicted action type judgment.

[0105] Based on the above design, the smooth model training method provided in this embodiment can predict the predicted action type corresponding to the training sample by training the smooth model, and then adjust the model parameters according to the smooth training parameters corresponding to the predicted action type, so that the trained smooth model can extract the semantics of the action data sequence itself to determine the corresponding action type, and then perform smooth processing of the corresponding smooth degree.

[0106] The smooth model trained by the smooth model training method provided in this embodiment can include the following steps in actual use.

[0107] Step S310: Obtain a to-be-processed action data sequence composed of multiple action key point data frames;

[0108] Step S320: Input the to-be-processed action data sequence into the adjusted smooth model;

[0109] Step S330: Process the to-be-processed action data sequence through the encoder and the decoder of the smooth model in sequence to obtain a smooth-processed action data sequence.

[0110] The model parameters retained in the adjusted smooth model are the model parameters obtained by training in steps S110 to S130, and therefore the smooth model can automatically perform smooth processing of different degrees for the to-be-processed action data sequence of different action types.

[0111] In this embodiment, Figure 2 The smooth model training method shown in the figure can be executed by an electronic device with data processing capability, which can be Figure 1b The server 10 shown in the figure. Please refer to Figure 7 , Figure 7 for the hardware structure diagram of the server 10. The server 10 can include one or more processors 130 and machine readable storage medium 120. The processor 130 and the machine readable storage medium 120 can communicate via the system bus 140. And the machine readable storage medium 120 stores machine executable instructions, by reading and executing the machine executable instructions corresponding to the smooth model training logic in the machine readable storage medium 120, the processor 130 can execute the smooth model training method described above.

[0112] The machine readable storage medium 120 can be, but is not limited to, a Random Access Memory (RAM), a Read Only Memory (ROM), a Programmable Read-Only memory (PROM), an Erasable Programmable Read-Only memory (EPROM), an Electric Erasable Programmable Read-Only memory (EEPROM), etc. The machine readable storage medium 120 is configured to store a program, and the processor 130 is configured to execute the program after receiving an execution instruction.

[0113] The processor 130 can be an integrated circuit chip having a processing capability of signals. The processor can be a general processor, including a Central Processing Unit (CPU), a Network Processor (NP), etc. The processor can also be a Digital Signal Processor (DSP), an Application-Specific Integrated Circuit (ASIC), a Field-Programmable Gate Array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. The processor can implement or execute the disclosed methods, steps and logic block diagrams in the embodiments of the present application. The general processor can be a microprocessor or the processor can also be any conventional processor.

[0114] Please refer to Figure 6 The embodiment further provides a smoothing model training apparatus 110, which includes at least one functional module stored in the machine readable storage medium 120 in the form of software. Functionally, the smoothing model training apparatus 110 can include a data acquisition module 111, an action prediction module 112 and a model training module 113.

[0115] The data acquisition module 111 is configured to acquire a plurality of training samples, wherein each of the training samples includes a sample action data sequence composed of a plurality of action key point data frames.

[0116] In the embodiment, the data acquisition module 111 can be configured to perform the step S110, and the specific description of the data acquisition module 111 can refer to the description of the step S110. Figure 2 The specific description of the data acquisition module 111 can refer to the description of the step S110.

[0117] The action prediction module 112 is configured to determine a predicted action type corresponding to each training sample according to the sample action data sequence of the training sample, and determine a smoothing training parameter corresponding to the training sample according to the predicted action type.

[0118] In this embodiment, the action prediction module 112 can be configured to perform Figure 2 The specific description of the action prediction module 112 can refer to the description of step S120.

[0119] The model training module 113 is configured to perform model training on a preset smoothing model using the smoothing training parameter and the sample action data sequence; and the smoothing model is configured to perform smoothing processing on an action data sequence.

[0120] In this embodiment, the model training module 113 can be configured to perform Figure 2 The specific description of the model training module 113 can refer to the description of step S130.

[0121] Further, in this embodiment, the smoothing model training apparatus 110 can further include a data processing module 114 configured to perform data smoothing processing by using the trained smoothing model. In one possible implementation, the data processing module 114 can be configured to:

[0122] Obtain a to-be-processed action data sequence composed of a plurality of action key point data frames;

[0123] Input the to-be-processed action data sequence into the trained smoothing model;

[0124] Process the to-be-processed action data sequence by using the encoder and the decoder of the smoothing model in sequence, to obtain a smoothed action data sequence.

[0125] Further, in another possible implementation, the data processing module 114 can be further configured to:

[0126] Obtain live video data, and identify action key point data of a host from the live video data by using an image recognition model;

[0127] Perform smoothing processing on the action key point data by using the trained smoothing model;

[0128] Generate a virtual image corresponding to a host action of the host according to the smoothed action key point data by using a virtual image generation model;

[0129] Generate live streaming data according to the virtual image, and push the live streaming data to a viewer terminal.

[0130] To sum up, the smoothing model training method, device and electronic equipment provided in the present application, by predicting the action type of the training sample and determining the smoothing training parameter corresponding to the action type in the process of training the smoothing model, then training the smoothing model in combination with the smoothing training parameter and the sample action data sequence in the training sample. In this way, the trained smoothing model can automatically perform smoothing processing of different degrees for the to-be-processed action data sequence of different action types, thereby balancing the smoothing effect and processing timeliness.

[0131] In the embodiments provided in the present application, it should be understood that the disclosed apparatus and method can also be implemented in other manners. The embodiment of the apparatus described above is only illustrative, for example, the flowcharts and block diagrams in the accompanying drawings show the possible implementation architectures, functions and operation of the apparatus, method and computer program product according to the embodiments of the present application. In this regard, each block in the flowcharts or block diagrams can represent a module, a program segment or a part of code, which contains one or more executable instructions for implementing the specified logic function. It should also be noted that, in some alternative implementations, the functions noted in the blocks can occur in different orders from those noted in the accompanying drawings. For example, two consecutive blocks can actually be executed substantially in parallel, and sometimes they can be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and the combination of blocks in the block diagrams and / or flowcharts, can be implemented by a dedicated hardware-based system for performing the specified functions or actions, or can be implemented by a combination of dedicated hardware and computer instructions.

[0132] In addition, each functional module in the various embodiments of the present application can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0133] If the functions are implemented in the form of software function modules and sold or used as independent products, they can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application can be embodied in the form of a software product, and the computer software product is stored in a storage medium, and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various program code storage media.

[0134] It should be noted that, in this paper, relational terms such as first and second are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply that there is any such actual relationship or order between these entities or operations. Moreover, the terms "include", "contain" or any other variants thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements not only includes those elements, but also includes other elements not explicitly listed or inherent to such process, method, article or device. Without more limitations, the element defined by the statement "including a" does not exclude the presence of other identical elements in the process, method, article or device including the element.

[0135] The above is only various embodiments of the present application, but the protection scope of the present application is not limited thereto, and any person skilled in the art can easily think of changes or replacements within the technical scope disclosed in the present application, which should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A method for training a smoothing model, characterized in that, The smoothing model comprises an encoder and a decoder; the smoothing model is used for smoothing action data sequences; the method comprises: obtaining a plurality of training samples, wherein each training sample comprises a sample action data sequence composed of a plurality of action key point data frames; performing action type recognition on the sample action data sequence of each training sample, determining the predicted action type corresponding to the training sample, and determining the smoothing training parameter corresponding to the training sample according to the predicted action type; for each training sample, processing the sample action data sequence of the training sample through the encoder to obtain the action feature corresponding to the training sample; inputting the action feature of each training sample into the decoder of the smoothing model respectively to obtain the predicted action data sequence corresponding to each training sample; determining a first loss function value according to the sample action data sequence, the predicted action data sequence and the smoothing training parameter of each training sample, and adjusting the model parameters of the encoder and the decoder of the smoothing model according to the first loss function value.

2. The method of claim 1, wherein, The smoothing model comprises an action classification sub-model; the step of determining the predicted action type corresponding to each training sample according to the sample action data sequence of the training sample, and determining the smoothing training parameter corresponding to the training sample according to the predicted action type, comprises: inputting the action feature into the action classification sub-model to obtain the predicted action type corresponding to the training sample, and determining the smoothing training parameter corresponding to the training sample according to the predicted action type.

3. The method of claim 1, wherein, The first loss function value is: wherein T is a number of action key point data frames of the training sample, is the t-th action key point data frame in the sample action data sequence, is the t-th action key point data frame in the predicted action data sequence. B is the number of the training samples, is the smoothed training parameter corresponding to the b-th training sample.

4. The method of claim 3, wherein, The step of determining the smoothing training parameter corresponding to the training sample according to the predicted action type comprises: querying the smoothing training parameter corresponding to the predicted action type according to a preset smoothing training parameter corresponding relationship table; wherein the higher the action frequency corresponding to the predicted action type is, the larger the smoothing training parameter is.

5. The method of claim 2, wherein, The training sample further comprises an action type label; the method further comprises: for each training sample, determining a second loss function value according to the action type label and the predicted action type of the training sample; adjusting the model parameters of the action classification sub-model according to the second loss function value; wherein the second loss function value is: wherein N is the number of the predicted action types, is a predicted probability value corresponding to the i-th predicted action type of the training sample, is a one-hot code vector corresponding to the action type label.

6. The method according to any one of claims 1 to 5, characterized in that, The method further comprises: obtaining a to-be-processed action data sequence composed of a plurality of action key point data frames; inputting the to-be-processed action data sequence into the trained smoothing model; processing the to-be-processed action data sequence through the encoder and the decoder of the smoothing model in sequence to obtain a smoothed action data sequence.

7. The method according to any one of claims 1 to 5, characterized in that, The trained smoothing model is configured in a server of a live broadcast platform, the server further comprises an image recognition model and a virtual image generation model, and the method further comprises: obtaining live video data, and recognizing the action key point data of a host through the image recognition model from the live video data; performing smoothing processing on the action key point data through the trained smoothing model; The virtual image generation model generates a virtual image corresponding to the anchor action of the anchor according to the smoothed action key point data; The virtual image generation pushes stream data to the audience terminal.

8. A smoothing model training apparatus characterized by comprising: The smoothing model comprises an encoder and a decoder; and the smoothing model is configured to smooth the action data sequence. The smoothing model training apparatus comprises: a data acquisition module configured to acquire a plurality of training samples, wherein each training sample comprises a sample action data sequence composed of a plurality of action key point data frames; an action prediction module configured to identify an action type according to the sample action data sequence of each training sample, determine a predicted action type corresponding to the training sample, and determine a smoothing training parameter corresponding to the training sample according to the predicted action type; for each training sample, the encoder is configured to process the sample action data sequence of the training sample to obtain an action feature corresponding to the training sample; a model training module configured to input the action feature of each training sample into the decoder of the smoothing model to obtain a predicted action data sequence corresponding to each training sample; determine a first loss function value according to the sample action data sequence, the predicted action data sequence and the smoothing training parameter of each training sample, and adjust the model parameters of the encoder and the decoder of the smoothing model according to the first loss function value.

9. An electronic device, comprising: The electronic device comprises a machine readable storage medium and one or more processors, the machine readable storage medium stores machine executable instructions, and the machine executable instructions are executed by the one or more processors to implement the method of any one of claims 1-7.

Images