Text processing method and apparatus, and computer device, storage medium and computer program product

Abstract

Provided in the present disclosure are a text processing method and apparatus, and a computer device and a storage medium. The method comprises: acquiring target text and style prompt information corresponding to the target text; on the basis of the style prompt information, generating a stylized text image corresponding to the target text; on the basis of the stylized text image and a dynamic stylization model, generating a plurality of video frames corresponding to the stylized text image; and on the basis of the video frames, generating a stylized text video corresponding to the target text.

Description

Word processing method, device, computer equipment, storage medium and computer program product

[0001] This application claims priority to Chinese Patent Application No. 202311280743.4 filed on September 28, 2023. The contents of the above-mentioned Chinese patent application disclosure are hereby incorporated by reference in their entirety as a part of this application. Technical Field

[0002] The present disclosure relates to a word processing method, apparatus, computer equipment, storage medium and computer program product. Background Art

[0003] In many scenarios, such as video editing and dynamic poster creation, dynamic text is required to enhance the visual effect. Generally, this is done in the form of artistic text, which is typically designed by professional designers. This results in a limited number of available artistic text types. Furthermore, designing dynamic artistic text takes a significant amount of time, resulting in low efficiency.

[0004] Summary of the Invention

[0005] The embodiments of the present disclosure at least provide a word processing method, apparatus, computer equipment, storage medium, and computer program product.

[0006] In a first aspect, an embodiment of the present disclosure provides a text processing method, including:

[0007] Obtaining target text and style prompt information corresponding to the target text;

[0008] generating a stylized text image corresponding to the target text according to the style prompt information;

[0009] generating a plurality of video frames corresponding to the stylized text image based on the stylized text image and the dynamic stylization model;

[0010] Based on the video frames, a stylized text video corresponding to the target text is generated.

[0011] In a possible implementation, generating a stylized text image corresponding to the target text according to the style prompt information includes:

[0012] Determining font information corresponding to the target text;

[0013] generating a text image corresponding to the target text according to the font information;

[0014] A mask image corresponding to a target character in the character image is determined, and the mask image and style prompt information corresponding to the target character are input into a stylized model to determine a stylized character image corresponding to the target character.

[0015] In one possible implementation, generating a plurality of video frames corresponding to the stylized text image based on the stylized text image and the dynamic stylization model includes:

[0016] stacking the stylized text images to obtain a target number of initial images;

[0017] Performing feature extraction on the target number of initial images to obtain first image features, and adding noise to the first image features;

[0018] Based on the first image feature after adding noise and the dynamic stylization model, a plurality of video frames corresponding to the stylized text image are generated.

[0019] In a possible implementation, the first image feature includes sub-image features corresponding to each initial image;

[0020] The adding noise to the first image feature comprises:

[0021] Noise is added to each sub-image feature in the first image feature respectively; wherein the noise added to different sub-image features is different.

[0022] In a possible implementation, the adding noise to each sub-image feature in the first image feature includes:

[0023] For the sub-image feature, determining a noise feature corresponding to the sub-image feature;

[0024] The sub-image feature is fused with the noise feature to obtain a sub-image feature after adding noise.

[0025] In a possible implementation, the first image feature includes sub-image features corresponding to each initial image;

[0026] The step of generating a plurality of video frames corresponding to the stylized text image based on the first image feature after adding noise and the dynamic stylization model includes:

[0027] Determine the mask constraint features corresponding to each sub-image feature in the first image feature after adding noise;

[0028] The noise-added sub-image features and the corresponding mask constraint features are fused to obtain the second image features;

[0029] The second image feature is input into the dynamic stylization model to generate a plurality of video frames corresponding to the stylized text image.

[0030] In one possible implementation, inputting the second image feature into the dynamic stylization model to generate a plurality of video frames corresponding to the stylized text image includes:

[0031] Inputting the second image feature as an input feature into the dynamic stylization model to obtain an output feature;

[0032] Using the output features as input features again, and returning to the step of inputting the dynamic stylized model until the preset number of cycles are repeated;

[0033] The output features obtained from the last cycle are decoded to obtain multiple video frames corresponding to the stylized text images.

[0034] In one possible implementation, the dynamic stylization model is adjusted based on the following method:

[0035] Get reference videos;

[0036] extracting reference image features of the video frames of the reference video;

[0037] adding supervisory noise to the reference image features, inputting the reference image features after adding supervisory noise into the dynamic stylization model to be adjusted, and determining predicted noise contained in the image features input into the dynamic stylization model as predicted by the dynamic stylization model to be adjusted;

[0038] The dynamic stylization model to be adjusted is adjusted based on the predicted noise and the supervised noise.

[0039] In a second aspect, an embodiment of the present disclosure further provides a word processing device, comprising:

[0040] An acquisition module is configured to acquire target text and style prompt information corresponding to the target text;

[0041] A first generating module is configured to generate a stylized text image corresponding to the target text according to the style prompt information;

[0042] a second generating module configured to generate a plurality of video frames corresponding to the stylized text image based on the stylized text image and the dynamic stylization model;

[0043] The determination module is configured to generate a stylized text video corresponding to the target text based on the video frame.

[0044] In a third aspect, an embodiment of the present disclosure further provides a computer device comprising: a processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the computer device is running, the processor and the memory communicate via the bus, and when the machine-readable instructions are executed by the processor, the steps of the above-mentioned first aspect or any possible implementation of the first aspect are performed.

[0045] In a fourth aspect, the present disclosure further provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the steps of the first aspect or any possible implementation of the first aspect are executed.

[0046] In a fifth aspect, an embodiment of the present disclosure further provides a computer program product, comprising a computer program, which implements the above method when executed by a processor.

[0047] In order to make the above-mentioned objectives, features and advantages of the present disclosure more obvious and easy to understand, preferred embodiments are given below and described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the following briefly introduces the drawings required for use in the embodiments. The drawings herein are incorporated into and constitute a part of the specification. These drawings illustrate embodiments consistent with the present disclosure and, together with the specification, are used to illustrate the technical solutions of the present disclosure. It should be understood that the following drawings only illustrate certain embodiments of the present disclosure and should not be regarded as limiting the scope. For those of ordinary skill in the art, other relevant drawings can be obtained based on these drawings without inventive effort.

[0049] FIG1 shows a flowchart of a text processing method provided by an embodiment of the present disclosure;

[0050] FIG2 shows a schematic diagram of a stylized text image in the text processing method provided by an embodiment of the present disclosure;

[0051] FIG3 shows a flow chart of a method for training a dynamic stylization model provided by an embodiment of the present disclosure;

[0052] FIG4 shows an overall flow chart of a training process of a dynamic stylization model provided by an embodiment of the present disclosure;

[0053] FIG5 shows an overall flow chart of the reasoning process of a dynamic stylization model provided by an embodiment of the present disclosure;

[0054] FIG6 shows a schematic diagram of the architecture of a word processing device provided by an embodiment of the present disclosure;

[0055] FIG7 shows a schematic structural diagram of a computer device provided by an embodiment of the present disclosure. DETAILED DESCRIPTION

[0056] In order to make the purpose, technical solutions and advantages of the embodiments of the present disclosure clearer, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are only part of the embodiments of the present disclosure, not all of the embodiments. The components of the embodiments of the present disclosure generally described and shown in the drawings herein can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present disclosure provided in the drawings is not intended to limit the scope of the disclosure for which protection is sought, but merely represents selected embodiments of the present disclosure. Based on the embodiments of the present disclosure, all other embodiments obtained by those skilled in the art without making creative work are within the scope of protection of the present disclosure.

[0057] Research has found that in many scenarios, such as video editing and dynamic poster creation, dynamic text is required to enhance the visual effect. Typically, this is done in the form of artistic text, which is typically designed by professional designers. This results in a limited selection of artistic text available to users. Furthermore, designing dynamic artistic text takes a significant amount of time, resulting in low efficiency.

[0058] Based on the above research, the present disclosure provides a text processing method, apparatus, computer device, and storage medium. After acquiring target text and corresponding style prompt information, the method can generate a stylized text image corresponding to the target text according to the style prompt information. Multiple video frames are then generated based on the stylized text image and a three-dimensional stylized pattern. Based on the video frames, a stylized text video corresponding to the target text is generated. In this way, personalized stylized text videos can be generated in real time, improving the processing speed and quality of stylized text videos.

[0059] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, it does not need to be further defined or explained in subsequent drawings.

[0060] The term "and / or" herein simply describes an association relationship, indicating that three relationships can exist. For example, A and / or B can represent the existence of A alone, the simultaneous existence of A and B, and the existence of B alone. In addition, the term "at least one" herein refers to any combination of at least two of any one or more of a plurality of items. For example, "at least one of A, B, and C" can represent any one or more elements selected from the set consisting of A, B, and C.

[0061] It is understandable that before using the technical solutions disclosed in the various embodiments of this disclosure, the type, scope of use, usage scenarios, etc. of the personal information involved in this disclosure should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations.

[0062] For example, in response to a user's active request, a prompt message is sent to the user to clearly inform the user that the operation requested will require the acquisition and use of the user's personal information. This allows the user to independently choose whether to provide personal information to the electronic device, application, server, storage medium, or other software or hardware that performs the operations of the disclosed technical solution based on the prompt message.

[0063] As an optional but non-limiting implementation, in response to receiving a user's active request, the prompt information may be sent to the user in the form of a pop-up window, in which the prompt information may be presented in text form. Furthermore, the pop-up window may also contain a selection control for the user to select "agree" or "disagree" to provide personal information to the electronic device.

[0064] It is understandable that the above notification and user authorization process are merely illustrative and do not limit the implementation of the present disclosure. Other methods that comply with relevant laws and regulations may also be applied to the implementation of the present disclosure.

[0065] To facilitate understanding of this embodiment, a text processing method disclosed in an embodiment of the present disclosure is first introduced in detail. The execution subject of the text processing method provided in the embodiment of the present disclosure is generally a computer device with certain computing capabilities. The computer device includes, for example, a terminal device or a server or other processing device. The terminal device can be a user equipment (UE), a mobile device, a user terminal, a terminal, a cellular phone, a cordless phone, a personal digital assistant (PDA), a handheld device, a computing device, an in-vehicle device, a wearable device, etc. In some possible implementations, the text processing method can be implemented by a processor calling computer-readable instructions stored in a memory.

[0066] 1 is a flowchart of a text processing method according to an embodiment of the present disclosure. The method includes steps 101 to 104, wherein:

[0067] Step 101: Obtain target text and style prompt information corresponding to the target text;

[0068] Step 102: Generate a stylized text image corresponding to the target text according to the style prompt information;

[0069] Step 103: Generate multiple video frames corresponding to the stylized text image based on the stylized text image and the dynamic stylization model;

[0070] Step 104: Generate a stylized text video corresponding to the target text based on the video frame.

[0071] The following is a detailed description of the above steps.

[0072] For step 101 and step 102,

[0073] The target text may be a problem to be stylized. In one possible implementation, when obtaining the target text, the text may be received as text input by a user, or optical character recognition (OCR) may be performed on an image, and the recognized text is then used as the target text.

[0074] In a possible application scenario, the target text may be text to be added to the video to be processed. For example, in a video editing scenario, the target text may be text that needs to be added to the video to be processed. Therefore, obtaining the target text may be obtaining the target text to be added to the video to be processed.

[0075] In a possible implementation, when generating a stylized text image corresponding to the target text according to the style prompt information, the following steps may be exemplified:

[0076] Step a1, determining the font information corresponding to the target text;

[0077] Step a2: generating a text image corresponding to the target text according to the font information;

[0078] Step a3: Determine a mask image corresponding to the target text in the text image, and input the mask image and style prompt information corresponding to the target text into a stylized model to determine a stylized text image corresponding to the target text.

[0079] Among them, the font information corresponding to the target text may include, for example, font types such as Songti, Kaiti, and Heiti; when determining the font information corresponding to the target text, it may be possible to respond to the user's setting instructions and determine the font information carried in the setting instructions as the font information corresponding to the target text.

[0080] Optionally, the font information may further include font size, font weight, etc.

[0081] In another possible implementation, when obtaining the target text and determining the font information corresponding to the target text, it is possible to perform content recognition on the video to be processed after obtaining the video to be processed, and determine the target text that matches the video to be processed and the font information corresponding to the target text based on the content recognition result.

[0082] Specifically, in a video editing scenario, text may be added to the video by automatically adding text, and the target text may be the text automatically generated when automatically adding text to the video.

[0083] Exemplarily, when automatically matching text, a text matching model can be used. After the video to be processed is input into the text matching model, the text matching model can automatically output target text that matches the video to be processed, as well as font information corresponding to the target text.

[0084] For example, if the video to be processed is a cartoon, the target text that matches the video to be processed may be the lyrics of a children's song.

[0085] In step a3, the stylization model may be a generative artificial intelligence (AIGC) model; the style prompt corresponding to the target text is used to prompt the stylization model to process the text image into a certain style. Optionally, when determining the style prompt corresponding to the target text, any of the following methods may be used:

[0086] Method 1: Receive style prompt information corresponding to the target text input by the user.

[0087] Specifically, when inputting the target text, the user can input the style prompt information corresponding to the target text at the same time.

[0088] Method 2: Performing style detection on the video to be processed, and determining style prompt information corresponding to the target text based on the style detection result.

[0089] Here, the video to be processed may refer to the video in which the target text is located when the target text is obtained; since the target text needs to be added to the video to be processed, the final stylized text needs to be consistent with the style of the video to be processed.

[0090] In a specific implementation, when performing style detection on the video to be processed, the video to be processed can be input into the style detection model through a pre-trained style detection model, and the style detection model can output the style detection result, which includes the style of the video to be processed.

[0091] After determining the style detection result, the text style corresponding to the video style of the video to be processed may be determined based on a pre-established correspondence between video style and text style, and the determined text style may be used as the style prompt information.

[0092] Method 3: Perform content detection on the target text, and determine style prompt information corresponding to the text based on the content detection result.

[0093] Specifically, the content detection of the target text may include performing word segmentation on the target text to determine keywords contained in the target text, then detecting whether the keywords contained in the target text include the target keyword, and using the target keyword as the style hint information. The word segmentation may be performed, for example, using an N-gram model.

[0094] The target keyword may be a word used to represent style information, such as comic, ancient style, cool, etc. The target keyword may be a pre-set keyword.

[0095] Optionally, when the style prompt information includes the name of the main object, the stylized text image may be text composed based on the main object. The combination mentioned here may be understood as patchwork.

[0096] For example, if the style prompt information is "melting ice cream" and the target text is "WORD", the text in the generated stylized text image is "WORD" composed of a combination of melted ice cream, as shown in Figure 2.

[0097] In a case where the style prompt information does not include the name of the main object, the text in the stylized text image corresponding to the target text is text composed based on the associated objects associated with the style prompt information.

[0098] Here, the associated object associated with the style prompt information can be understood as an object that can reflect the style indicated by the style prompt information. For example, if the style prompt information is fruit, the associated object can be apples, grapes, bananas, etc.; if the style prompt information is flowers, the associated object can be lilies, sunflowers, roses, etc.

[0099] It should be noted that the main object / associated object of the above-mentioned text composition can be one or more types of objects.

[0100] For step 103,

[0101] In one possible implementation, when generating multiple video frames corresponding to the stylized text image based on the stylized text image and the dynamic stylization model, the following steps may be referred to:

[0102] Step b1: stacking the stylized text images to obtain a target number of initial images;

[0103] Step b2: performing feature extraction on the target number of initial images to obtain first image features, and adding noise to the first image features;

[0104] Step b3: generating a plurality of video frames corresponding to the stylized text image based on the first image features after adding noise and the dynamic stylization model.

[0105] In step b1, the stacking process of the stylized text images can be understood as copying N stylized text images and arranging the N stylized text images in sequence, where N is the target number.

[0106] Here, the purpose of stacking the stylized text images is to modify each stylized text image based on subsequent steps, and then arrange the modified stylized text images to achieve the effect of dynamic stylized text images, that is, stylized text videos.

[0107] The stylized text images are stacked to obtain a 3×N×H×W data block, where 3 is the number of channels of the image (generally RGB channels), N is the target number, and H×W is the size of the stylized text image.

[0108] In step b2, when performing feature extraction on the target number of initial images, feature extraction can be performed on the 3×N×H×W data block through an encoder. When performing feature extraction, feature extraction can be performed layer by layer in the "N" direction, that is, feature extraction can be performed image by image.

[0109] In this way, after feature extraction, the first image feature obtained should include sub-image features corresponding to each initial image. Correspondingly, in step b3, when adding noise to the first image feature, noise can be added to each sub-image feature in the first image feature respectively; wherein the noise added to different sub-image features is different.

[0110] In a specific implementation, when adding noise to each sub-image feature in the first image feature, a noise image corresponding to each sub-image feature can be determined for each sub-image feature, and then the sub-image feature and the noise feature are fused to obtain the sub-image feature after adding noise.

[0111] The noise corresponding to each sub-image feature may be randomly generated. For example, for each initial image, a noise image corresponding to the image may be randomly generated, and then feature extraction is performed on the noise image to obtain the noise feature.

[0112] The feature sizes of the sub-image feature and the noise feature can be the same. When the sub-image feature and the noise feature are fused, for example, the values ​​of the sub-image feature and the noise image feature at the corresponding feature positions can be weighted summed to obtain the value of the sub-image feature at the feature position after the noise is added.

[0113] Exemplarily, when the sub-image feature and the noise feature are fused, they can be expressed as follows:

[0114] L mid =L im *α+L noise *(1-α)

[0115] Among them, L mid represents the sub-image feature after adding noise, L im represents the sub-image feature, L noise represents the noise feature, α represents the weight corresponding to the sub-image feature, and 1-α represents the weight corresponding to the noise feature.

[0116] Optionally, after the sub-image features are fused with the noise features, the fused sub-image features are spliced ​​in the stacking order during the stacking process to obtain the first image features after adding noise, and then the first image features after adding noise are input into the dynamic stylization model to generate multiple video frames corresponding to the stylized text image.

[0117] In a possible implementation, to ensure the stability of fonts in the generated video frames, font constraints may be added during the process of generating the video frames to ensure the stability of the fonts.

[0118] Specifically, when generating multiple video frames corresponding to the stylized text image based on the first image feature after adding noise and the dynamic stylization model, the mask constraint features corresponding to each sub-image feature in the first image feature after adding noise can be determined first; then, each sub-image feature after adding noise and the corresponding mask constraint feature are fused to obtain a second image feature; and then, the second image feature is input into the dynamic stylization model to generate multiple video frames corresponding to the stylized text image.

[0119] Here, the mask constraint features corresponding to different sub-image features can be different. The reason is that if the same font constraint is applied to all images, the resulting stylized text video may have poor dynamic effects. Generally, the constraints are stronger for the first few frames of the stylized text video, and weaker for the last few frames.

[0120] For example, when determining the mask constraint features corresponding to each sub-image feature, the following formula may be used for calculation:

[0121] mask_t=(Nt)*mask / 10

[0122] Among them, mask_t represents the mask constraint feature corresponding to the sub-image feature of the initial image of the tth frame, N represents the number of initial images, that is, the number of targets, and mask represents the mask feature, which can be the feature after feature extraction of the mask image corresponding to the target text.

[0123] From the above formula, we can see that the larger t is, the smaller Nt is, and accordingly, the smaller the value of mask_t is. Therefore, the sub-image features of the initial image with a smaller frame number have stronger corresponding constraints, and the sub-image features of the initial image with a larger frame number have weaker corresponding constraints.

[0124] When fusing the features of each sub-image after adding noise with the corresponding mask constraint features, the following formula can be used as an example:

[0125] L mix =L mid_0 *mask_t+L mid_t *(1-mask_t)

[0126] Among them, L mix It represents the features after fusion of each sub-image feature after adding noise and the corresponding mask constraint feature, L mid_0 Indicates the sub-image features after adding noise to the initial image of the first frame, L mid_t Represents the sub-image features after adding noise to the initial image of frame t.

[0127] In this way, when fusing the features of each sub-image and the corresponding mask constraint features, the sub-image features of the initial image of the first frame are combined and used as constraints to avoid excessive gaps between frames, which may affect the effect of the generated stylized text video.

[0128] After fusing the noise-added sub-image features and the corresponding mask constraint features, the fused image features can be spliced ​​in the stacking order of the corresponding initial images to obtain the second image features.

[0129] When the second image feature is input into the dynamic stylization model to generate multiple video frames corresponding to the stylized text image, the second image feature can be used as an input feature and input into the dynamic stylization model to obtain an output feature; then the output feature is used again as the input feature, and the step of inputting the dynamic stylization model is returned to be executed until a preset number of cycles are completed; and then the output feature obtained from the last cycle is decoded to obtain multiple video frames corresponding to the stylized text image.

[0130] Here, the purpose of looping multiple times is to predict the noise features in the second image features multiple times through the dynamic stylization model to improve the correlation and dynamics of the generated video frames and eliminate static interference between video frames. The specific preset number of loops can be adjusted according to actual conditions.

[0131] For step 104,

[0132] After determining the plurality of video frames, the plurality of video frames may be arranged in a stacking order of the initial image, and the stylized text video may be obtained after the arrangement.

[0133] It should be noted that after decoding the output features, the difference between two adjacent video frames is not large, but there are differences between each video frame.

[0134] The present disclosure also provides a method for training a dynamic stylized model. FIG3 is a flowchart of a method for training a dynamic stylized model provided by the present disclosure, including the following steps:

[0135] Step 301: Obtain a reference video.

[0136] Step 302: Extract reference image features of the video frames of the reference video.

[0137] Step 303: Add supervisory noise to the reference image features, and input the reference image features after adding supervisory noise into the dynamic stylization model to be adjusted, and determine the predicted noise contained in the image features input into the dynamic stylization model as predicted by the dynamic stylization model to be adjusted.

[0138] Step 304: Adjust the dynamic stylization model to be adjusted based on the predicted noise and the supervisory noise.

[0139] Here, adding supervisory noise to the reference image features may refer to randomly generating image noise and adding the generated image noise to the reference image features. Since the type of noise added to the reference image features can be determined, the image noise added to the reference image features may be regarded as supervisory data.

[0140] After the dynamic stylization model inputs image features, the dynamic stylization model can predict the type of image noise added to the input image features. Then, based on the supervised data, i.e., the supervised noise, and the predicted noise predicted by the dynamic stylization model, the model loss value in this training process can be determined, and based on the loss value, the model parameter values ​​of the dynamic stylization model can be adjusted.

[0141] A dynamic stylization model trained in this way can learn the motion of different objects from a large number of reference videos, such as the visual effects corresponding to "drifting flames." When the trained dynamic stylization model performs model inference, since the input image features of the dynamic stylization model are the image features of multiple static images, the dynamic stylization model can interpret these static images as having lost their motion due to the addition of noise. Therefore, by predicting the noise contained in the input image features, the dynamic stylization model can "restore" the motion of objects in these static images.

[0142] It should be noted that the image noise contained in the predicted image features during the inference of the 3D stylization described here is not the same as the image noise added in the above step b2.

[0143] The following will introduce the training process and inference process of the above dynamic stylization model respectively with the help of the overall flowchart.

[0144] 4 shows an overall flow chart of a training process for a dynamic stylization model provided by an embodiment of the present disclosure, which includes the following steps:

[0145] First, feature extraction is performed on the acquired reference video to obtain reference image features. Then, noise is randomly generated and added to the reference image features. The reference image features with added noise are then input into the dynamic stylization model to be trained. The dynamic stylization model can output predicted noise, and then the loss is calculated based on the predicted noise and the noise added to the reference image features.

[0146] The detailed description of the above steps refers to the introduction of the embodiment of FIG3 , which will not be repeated here.

[0147] 5 , which is a flowchart of the overall inference process of a dynamic stylization model provided in an embodiment of the present disclosure, includes the following steps:

[0148] First, a stylized text image of the target text is generated based on the stylization model. Then, the image features of the stylized text image are extracted to obtain a first image feature. Then, randomly generated image noise is added to the first image feature, and the first image feature after adding image noise is input as the input feature to the dynamic stylization model. The output features of the dynamic stylization model are also input as the input features to the dynamic stylization model. This cycle is repeated multiple times, and the output features of the last cycle are decoded to obtain multiple video frames corresponding to the stylized text image.

[0149] Here, the stylized model is different from the dynamic stylized model. The dynamic stylized model adds a motion modeling module on the basis of the stylized model. The motion modeling module can learn the motion form of the object in the reference video.

[0150] Those skilled in the art will understand that in the above-mentioned method of the specific implementation method, the writing order of each step does not mean a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of each step should be determined by its function and possible internal logic.

[0151] Based on the same inventive concept, a word processing device corresponding to the word processing method is also provided in the embodiment of the present disclosure. Since the principle of solving the problem by the device in the embodiment of the present disclosure is similar to the above-mentioned word processing method in the embodiment of the present disclosure, the implementation of the device can refer to the implementation of the method, and the repeated parts will not be repeated.

[0152] 6 , which is a schematic diagram of the architecture of a text processing device provided by an embodiment of the present disclosure, the device includes: an acquisition module 601, a first generation module 602, a second generation module 603, a determination module 604, and an adjustment module 605; wherein,

[0153] The acquisition module 601 is configured to acquire target text and style prompt information corresponding to the target text;

[0154] A first generating module 602 is configured to generate a stylized text image corresponding to the target text according to the style prompt information;

[0155] A second generating module 603 is configured to generate a plurality of video frames corresponding to the stylized text image based on the stylized text image and the dynamic stylization model;

[0156] The determination module 604 is configured to generate a stylized text video corresponding to the target text based on the video frame.

[0157] In a possible implementation, when generating a stylized text image corresponding to the target text according to the style prompt information, the first generating module 602 is configured to:

[0158] Determining font information corresponding to the target text;

[0159] generating a text image corresponding to the target text according to the font information;

[0160] A mask image corresponding to a target character in the character image is determined, and the mask image and style prompt information corresponding to the target character are input into a stylized model to determine a stylized character image corresponding to the target character.

[0161] In one possible implementation, when the second generation module 603 generates a plurality of video frames corresponding to the stylized text image based on the stylized text image and the dynamic stylization model, it is configured to:

[0162] stacking the stylized text images to obtain a target number of initial images;

[0163] Performing feature extraction on the target number of initial images to obtain first image features, and adding noise to the first image features;

[0164] Based on the first image feature after adding noise and the dynamic stylization model, a plurality of video frames corresponding to the stylized text image are generated.

[0165] In a possible implementation, the first image feature includes sub-image features corresponding to each initial image;

[0166] The second generating module 603, when adding noise to the first image feature, is configured to:

[0167] Noise is added to each sub-image feature in the first image feature respectively; wherein the noise added to different sub-image features is different.

[0168] In a possible implementation, when adding noise to each sub-image feature in the first image feature, the second generating module 603 is configured to:

[0169] For the sub-image feature, determining a noise feature corresponding to the sub-image feature;

[0170] The sub-image feature is fused with the noise feature to obtain a sub-image feature after adding noise.

[0171] In a possible implementation, the first image feature includes sub-image features corresponding to each initial image;

[0172] The second generating module 603 is configured to: generate a plurality of video frames corresponding to the stylized text image based on the first image feature after adding noise and the dynamic stylization model;

[0173] Determine the mask constraint features corresponding to each sub-image feature in the first image feature after adding noise;

[0174] The noise-added sub-image features and the corresponding mask constraint features are fused to obtain the second image features;

[0175] The second image feature is input into the dynamic stylization model to generate a plurality of video frames corresponding to the stylized text image.

[0176] In one possible implementation, when the second generation module 603 inputs the second image feature into the dynamic stylization model to generate a plurality of video frames corresponding to the stylized text image, it is configured to:

[0177] Inputting the second image feature as an input feature into the dynamic stylization model to obtain an output feature;

[0178] Using the output features as input features again, and returning to the step of inputting the dynamic stylized model until the preset number of cycles are repeated;

[0179] The output features obtained from the last cycle are decoded to obtain multiple video frames corresponding to the stylized text images.

[0180] In a possible implementation, the apparatus further includes an adjustment module 605 configured to adjust the dynamic stylized model based on the following method:

[0181] Get reference videos;

[0182] extracting reference image features of the video frames of the reference video;

[0183] adding supervisory noise to the reference image features, inputting the reference image features after adding supervisory noise into the dynamic stylization model to be adjusted, and determining predicted noise contained in the image features input into the dynamic stylization model as predicted by the dynamic stylization model to be adjusted;

[0184] The dynamic stylization model to be adjusted is adjusted based on the predicted noise and the supervised noise.

[0185] For descriptions of the processing flow of each module in the device and the interaction flow between each module, reference can be made to the relevant descriptions in the above method embodiment, which will not be described in detail here.

[0186] Based on the same technical concept, the embodiment of the present disclosure also provides a computer device. Referring to Figure 7, it is a structural diagram of a computer device 700 provided in the embodiment of the present disclosure. The computer device in the embodiment of the present disclosure may include but is not limited to mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc., or various forms of servers, such as independent servers or server clusters. The computer device shown in Figure 7 is only an example and should not bring any limitations to the functions and scope of use of the embodiment of the present disclosure.

[0187] As shown in FIG7 , a computer device 700 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 701, which can perform various appropriate actions and processes according to a program stored in a read-only memory device (ROM) 702 or a program loaded from a storage device 705 into a random access memory device (RAM) 703. Various programs and data required for the operation of the computer device 700 are also stored in the RAM 703. The processing device 701, the ROM 702, and the RAM 703 are connected to each other via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0188] Typically, the following devices may be connected to I / O interface 705: input device 706 including, for example, a touch screen, touchpad, keyboard, mouse, camera, microphone, accelerometer, gyroscope, etc.; output device 707 including, for example, a liquid crystal display (LCD), speaker, vibrator, etc.; storage device 708 including, for example, a magnetic tape, hard disk, etc.; and communication device 709. Communication device 709 may allow computer device 700 to communicate with other devices wirelessly or by wire to exchange data. Although FIG7 illustrates computer device 700 with various devices, it should be understood that not all of the illustrated devices are required to be implemented or present. More or fewer devices may alternatively be implemented or present.

[0189] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes program code for executing the word processing method. In such an embodiment, the computer program can be downloaded and installed from the network via the communication device 709, or installed from the storage device 708, or installed from the ROM 702. When the computer program is executed by the processing device 701, the above-mentioned functions defined in the method of the embodiment of the present disclosure are performed.

[0190] The present disclosure also provides a computer-readable storage medium having a computer program stored thereon. When executed by a processor, the computer program executes the steps of the text processing method described in the above method embodiment. The storage medium may be a volatile or non-volatile computer-readable storage medium.

[0191] The embodiments of the present disclosure also provide a computer program product, which carries program code. The instructions included in the program code can be used to execute the steps of the text processing method described in the above method embodiment. For details, please refer to the above method embodiment and will not be repeated here.

[0192] The computer program product may be implemented in hardware, software, or a combination thereof. In one embodiment, the computer program product is implemented as a computer storage medium. In another embodiment, the computer program product is implemented as a software product, such as a software development kit (SDK).

[0193] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems and devices described above can refer to the corresponding processes in the aforementioned method embodiments, and will not be repeated here. In the several embodiments provided in the present disclosure, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0194] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0195] In addition, each functional unit in each embodiment of the present disclosure may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0196] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a non-volatile computer-readable storage medium that is executable by a processor. Based on this understanding, the technical solution of the present disclosure, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present disclosure. The aforementioned storage medium includes various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0197] Finally, it should be noted that the above-described embodiments are only specific implementation methods of the present disclosure, which are used to illustrate the technical solutions of the present disclosure, rather than to limit them. The scope of protection of the present disclosure is not limited thereto. Although the present disclosure has been described in detail with reference to the above-described embodiments, those skilled in the art should understand that any person skilled in the art can modify or easily conceive of changes to the technical solutions described in the above-described embodiments within the technical scope disclosed in the present disclosure, or replace some of the technical features therein with equivalents. Such modifications, changes, or replacements do not deviate from the spirit and scope of the technical solutions of the embodiments of the present disclosure, and should be included in the scope of protection of the present disclosure. Therefore, the scope of protection of the present disclosure shall be subject to the scope of protection of the claims.

Claims

1. A word processing method, comprising: Obtaining target text and style prompt information corresponding to the target text; Generating a stylized text image corresponding to the target text according to the style prompt information; Based on the stylized text image and the dynamic stylization model, generating a plurality of video frames corresponding to the stylized text image; Based on the video frames, a stylized text video corresponding to the target text is generated.

2. The method according to claim 1, wherein: Generating a stylized text image corresponding to the target text according to the style prompt information includes: Determine the font information corresponding to the target text; Generating a text image corresponding to the target text according to the font information; A mask image corresponding to a target text in the text image is determined, and the mask image and style prompt information corresponding to the target text are input into a stylized model to determine a stylized text image corresponding to the target text.

3. The method according to claim 1, wherein: The step of generating a plurality of video frames corresponding to the stylized text image based on the stylized text image and the dynamic stylized model includes: Performing stacking processing on the stylized text images to obtain a target number of initial images; Performing feature extraction on the target number of initial images to obtain first image features, and adding noise to the first image features; Based on the first image feature after adding noise and the dynamic stylization model, a plurality of video frames corresponding to the stylized text image are generated.

4. The method according to claim 3, wherein: The first image features include sub-image features corresponding to each initial image; The adding noise to the first image feature comprises: Noise is added to each sub-image feature in the first image feature respectively; wherein different noises are added to different sub-image features.

5. The method according to claim 4, wherein: The adding noise to each sub-image feature in the first image feature respectively comprises: For the sub-image feature, determining a noise feature corresponding to the sub-image feature; The sub-image feature is fused with the noise feature to obtain the sub-image feature after adding noise.

6. The method according to claim 3, wherein: The first image features include sub-image features corresponding to each initial image; The step of generating a plurality of video frames corresponding to the stylized text image based on the first image feature after adding noise and the dynamic stylization model includes: Determine the mask constraint features respectively corresponding to each sub-image feature in the first image feature after adding noise; The features of each sub-image after adding noise are fused with the corresponding mask constraint features to obtain the second image features; The second image feature is input into the dynamic stylization model to generate a plurality of video frames corresponding to the stylized text image.

7. The method according to claim 6, wherein: The step of inputting the second image feature into the dynamic stylization model to generate a plurality of video frames corresponding to the stylized text image includes: Inputting the second image feature as an input feature into the dynamic stylization model to obtain an output feature; The output feature is used as the input feature again, and the step of inputting the dynamic stylization model is returned to be executed until the preset number of cycles is repeated; The output features obtained from the last cycle are decoded to obtain multiple video frames corresponding to the stylized text images.

8. The method according to any one of claims 1 to 7, wherein: The dynamic stylization model is adjusted based on the following method: Get reference videos; Extracting reference image features of video frames of the reference video; adding supervisory noise to the reference image features, and inputting the reference image features after the supervisory noise is added into the dynamic stylization model to be adjusted, and determining the predicted noise contained in the image features input into the dynamic stylization model predicted by the dynamic stylization model to be adjusted; The dynamic stylization model to be adjusted is adjusted based on the prediction noise and the supervision noise.

9. A word processing device comprising: An acquisition module is configured to acquire a target text and style prompt information corresponding to the target text; A first generating module is configured to generate a stylized text image corresponding to the target text according to the style prompt information; A second generating module is configured to generate a plurality of video frames corresponding to the stylized text image based on the stylized text image and the dynamic stylization model; The determination module is configured to generate a stylized text video corresponding to the target text based on the video frame.

10. A computer device comprising: A processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the computer device is running, the processor and the memory communicate via the bus, and when the machine-readable instructions are executed by the processor, the steps of the word processing method as described in any one of claims 1 to 8 are performed.

11. A computer-readable storage medium, wherein: The computer-readable storage medium stores a computer program, which, when executed by a processor, executes the steps of the word processing method as claimed in any one of claims 1 to 8.

12. A computer program product comprising a computer program, wherein: When the computer program is executed by a processor, the method according to any one of claims 1 to 8 is implemented.

Images