Text information processing method, device, server, storage medium and program product

Abstract

The present disclosure provides a text information processing method, device, server, storage medium and program product, which can be applied to various scenarios such as cloud technology, artificial intelligence, smart transportation, and assisted driving. The method includes: extracting sample text content features and sample style features; fusing the sample text content features and sample style features multiple times to obtain sample cascade fusion features, where the sample cascade fusion features are features that enhance both text content and style; training a font generation model based on sample text information, sample style images, sample text content features and sample cascade fusion features. The present disclosure fuses the extracted sample text content features and sample style features multiple times, and the text content and style are enhanced in the sample cascade fusion features obtained by fusion. The font generation model trained based on the sample cascade fusion features has higher accuracy, and the style transfer images generated based on the model have better effects.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, and in particular to a text information processing method, device, server, storage medium, and program product. Background Art

[0002] In modern society, information is exploding. Faced with a vast array of information and complex content, people's attention spans on promotional images, cover photos, and other text-based images are shrinking. Text, as the key element in text images, serves as the primary means of expression. Fonts that are aesthetically pleasing, personalized, and trendy are generally preferred by users. To enhance the possibilities and freshness of artistic creation and attract more attention, text fonts need to be processed.

[0003] In the related art, a sample style picture and a sample text picture are obtained, wherein the text contents of the sample style picture and the sample text picture are the same but the styles are different; features of the sample style picture and the sample text picture are extracted to obtain sample style features and sample text content features; the sample style features and the sample text content features are fused to obtain fused features; based on the fused features, a font generation model is trained to obtain a trained font generation model, and the trained font generation model is used to generate a style-transferred text picture with the font style of any text picture.

[0004] However, the sample fusion features used by related technologies to train font generation models contain less information and are unable to learn more information from sample style images and sample text images, resulting in low accuracy of the trained font generation model and poor effect of style transfer text images generated based on the font generation model. Summary of the Invention

[0005] The present disclosure provides a text information processing method, apparatus, server, storage medium, and program product. By extracting more informative sample cascade fusion features, the accuracy of a trained font generation model is improved, resulting in better style transfer text images generated based on the font generation model. The technical solution is as follows:

[0006] In one aspect, a text information processing method is provided, the method comprising:

[0007] Extracting features of sample text information and sample style images to obtain sample text content features and sample style features, wherein the sample style images are sample special effects images or sample text images, and the sample text images contain text from other sample text information in addition to the sample text information;

[0008] Fusing the sample text content feature and the sample style feature to obtain a first sample fusion feature;

[0009] Fusing the first sample fusion feature with the sample text content feature to obtain a second sample fusion feature;

[0010] Performing feature fusion on the sample style feature, the first sample fusion feature, and the second sample fusion feature to obtain a sample cascade fusion feature, wherein the sample cascade fusion feature is a feature that enhances both text content and style;

[0011] Based on the sample text information, the sample style picture, the sample text content features and the sample cascade fusion features, a font generation model is trained to obtain a trained font generation model. The trained font generation model is used to generate a style-transferred text picture with the style of any style picture.

[0012] In another aspect, a text information processing method is provided, the method being applied to the font generation model trained in the above aspect, the method comprising:

[0013] Obtain the style image with the style to be transferred and the text information to be processed;

[0014] The font generation model is called to process the style image and the text information to be processed to obtain a style-transferred text image.

[0015] In another aspect, a text information processing device is provided, the device comprising:

[0016] An extraction module is configured to extract features of sample text information and sample style images to obtain sample text content features and sample style features, wherein the sample style images are sample special effects images or sample text images, and the sample text images contain characters in the sample text information other than the sample text information;

[0017] A fusion module, configured to fuse the sample text content feature and the sample style feature to obtain a first sample fusion feature;

[0018] The fusion module is further configured to fuse the first sample fusion feature with the sample text content feature to obtain a second sample fusion feature;

[0019] The fusion module is further configured to fuse the sample style feature, the first sample fusion feature, and the second sample fusion feature to obtain a sample cascade fusion feature, wherein the sample cascade fusion feature is a feature that enhances both text content and style.

[0020] A training module is used to train a font generation model based on the sample text information, the sample style picture, the sample text content features and the sample cascade fusion features to obtain a trained font generation model, and the trained font generation model is used to generate a style-transferred text picture with the style of any style picture.

[0021] In another aspect, a text information processing device is provided, the device comprising:

[0022] An acquisition module is used to obtain style images with the style to be transferred and text information to be processed;

[0023] The processing module is used to call the font generation model, process the style image and the text information to be processed, and obtain a style-transferred text image.

[0024] On the other hand, a server is provided, comprising a processor and a memory, wherein the memory stores at least one program code, and the at least one program code is loaded and executed by the processor to implement the text information processing method as described in the first aspect.

[0025] On the other hand, a computer-readable storage medium is provided, wherein at least one program code is stored in the storage medium, and the at least one program code is loaded and executed by a processor to implement the text information processing method as described in the first aspect.

[0026] On the other hand, a computer program product is provided, which includes a computer program code, the computer program code being stored in a computer-readable storage medium, a processor of a server reading the computer program code from the computer-readable storage medium, and the processor executing the computer program code, so that the server performs the text information processing method as described in the first aspect.

[0027] The technical solutions provided by the embodiments of the present disclosure have the following beneficial effects:

[0028] In the disclosed embodiment, the sample style picture can be a sample special effect picture, or a sample text picture containing characters from other sample text information in addition to the sample text information. The sample style picture is different from the text content of the sample text information to be transferred to the font style, so that the sample style features and sample text content features to be fused are richer in quantity and information, so that the model can learn more knowledge about font generation. In the process of feature fusion, the first sample fusion feature obtained by the first feature fusion learns the style style of the sample style picture and the text content of the sample text information. The second sample fusion feature obtained by the second feature fusion further enhances the sample text content feature on the basis of the first sample fusion feature. The sample cascade fusion feature obtained by the third feature fusion further enhances the sample style feature on the basis of the first sample fusion feature and the second sample fusion feature. In short, the relevant features of the style sample and the text content in the sample cascade fusion features obtained by the above-mentioned several feature fusions are enhanced, and contain more information in the sample style picture and the sample text information. Therefore, the font generation model trained based on the sample cascade fusion feature is more accurate, and the effect of the style transfer text picture generated based on the font generation model is better. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] In order to more clearly illustrate the technical solutions in the embodiments of the present disclosure, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0030] Figure 1 is a schematic diagram of an implementation environment involved in a text information processing method provided by an embodiment of the present disclosure;

[0031] Figure 2 This is a network architecture diagram of a generator in a font generation model provided by an embodiment of the present disclosure;

[0032] Figure 3 This is a network architecture diagram of a discriminator in a font generation model provided by an embodiment of the present disclosure;

[0033] Figure 4 is a flowchart of a text information processing method provided by an embodiment of the present disclosure;

[0034] Figure 5 is a flowchart of another text information processing method provided by an embodiment of the present disclosure;

[0035] Figure 6 is a schematic diagram of a smooth mask image acquisition process provided by an embodiment of the present disclosure;

[0036] Figure 7 is a schematic diagram of a generator processing process in a font generation model provided by an embodiment of the present disclosure;

[0037] Figure 8 is a schematic diagram of the processing process of a generator and a discriminator in a font generation model provided by an embodiment of the present disclosure;

[0038] Figure 9 is a flowchart of another text information processing method provided by an embodiment of the present disclosure;

[0039] Figure 10 is a structural diagram of a text information processing device provided by an embodiment of the present disclosure;

[0040] Figure 11 is a structural diagram of a text information processing device provided by an embodiment of the present disclosure;

[0041] Figure 12 The present invention is a server for text information processing according to an exemplary embodiment. DETAILED DESCRIPTION

[0042] In order to make the objectives, technical solutions and advantages of the present disclosure more clear, the embodiments of the present disclosure will be further described in detail below with reference to the accompanying drawings.

[0043] It should be understood that the terms "each," "plurality," and "any" used in the embodiments of the present disclosure include two or more, each refers to each of the corresponding plurality, and any refers to any one of the corresponding plurality. For example, if a plurality of words includes 10 words, each refers to each of the 10 words, and any refers to any one of the 10 words.

[0044] The information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, stored data, displayed data, etc.) and signals involved in this disclosure are all authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions.

[0045] Before further describing the solutions in the embodiments of the present disclosure, the nouns and terms involved in the embodiments of the present disclosure are first explained.

[0046] GAN (Generative Adversarial Network): A key generative model in deep learning, GAN consists of a generator and a discriminator. The generator converts received data into images, while the discriminator determines the authenticity of the images generated by the generator.

[0047] Auto-Encoders (AEs) are a type of artificial neural networks (ANNs) used in semi-supervised and unsupervised learning. Their function is to learn representations of input information by using that information as a learning target. Autoencoders have the general functionality of representation learning algorithms and are used in dimensionality reduction and anomaly detection. Autoencoders constructed with convolutional layers can be applied to computer vision problems such as image denoising and neural style transfer.

[0048] OCR (Optical Character Recognition): The process of extracting text from an image using image algorithms.

[0049] The technologies involved in the embodiments of the present disclosure are introduced.

[0050] Artificial Intelligence (AI) refers to the theories, methods, techniques, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, to perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive technology within computer science that seeks to understand the essence of intelligence and produce new intelligent machines that can respond in a manner similar to human intelligence. AI also studies the design principles and implementation methods of various intelligent machines, enabling them to possess the capabilities of perception, reasoning, and decision-making.

[0051] Artificial intelligence (AI) technology is a comprehensive discipline encompassing a wide range of fields, encompassing both hardware and software technologies. Foundational AI technologies generally include sensors, specialized AI chips, cloud computing, distributed storage, big data processing, operating / interaction systems, and mechatronics. AI software technologies primarily encompass computer vision, natural language processing, and machine learning / deep learning.

[0052] Computer vision (CV) is the science of making machines "see." Specifically, it refers to using cameras and computers to replace the human eye in identifying and measuring objects, and then further processing them to create images more suitable for human observation or transmission to instruments. As a scientific discipline, computer vision studies related theories and technologies, aiming to build artificial intelligence systems that can extract information from images or multidimensional data. Computer vision technologies generally include image processing, image recognition, image semantic understanding, image retrieval, optical character recognition (OCR), video processing, video semantic understanding, video content / action recognition, 3D object reconstruction, 3D (three-dimensional) technology, virtual reality, augmented reality, simultaneous localization and mapping, and common biometric recognition technologies such as facial recognition and fingerprint recognition.

[0053] Natural language processing (NLP) is a key area of ​​research in computer science and artificial intelligence. It studies the theories and methods that enable effective communication between humans and computers using natural language. Natural language processing (NLP) integrates linguistics, computer science, and mathematics. Therefore, research in this field involves natural language—the language we use in everyday life—and is closely linked to the study of linguistics. Natural language processing technologies typically include text processing, semantic understanding, machine translation, robotic question answering, and knowledge graphs.

[0054] Machine learning (ML) is a multidisciplinary field that encompasses probability theory, statistics, approximation theory, convex analysis, and algorithmic complexity theory. It specifically studies how computers can simulate or implement human learning behaviors to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is at the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications span all areas of AI. Machine learning and deep learning typically include techniques such as artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and self-learning.

[0055] Cloud technology refers to a hosting technology that unifies hardware, software, and network resources within a wide or local area network (WAN) to enable data computing, storage, processing, and sharing. Cloud technology encompasses network, information, integration, management platform, and application technologies, all based on the cloud computing business model. It can be used to create a flexible and convenient resource pool for on-demand use.

[0056] Please refer to Figure 1 , which shows the implementation environment involved in the text information processing method provided by the embodiment of the present disclosure, see Figure 1 The implementation environment includes: a server 101. The server 101 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers. It can also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms. The server 101 is used to train a font generation model. The font generation model can generate a style-transferred text image with the same text content as the text information to be processed and the style of the style image based on any given style image with the style to be transferred and the text information to be processed.

[0057] In practical applications, the implementation environment may further include a terminal 102. The terminal 102 may be, but is not limited to, a smartphone, a computer, an intelligent voice interaction device, a smart home appliance, or an in-vehicle terminal. The terminal 102 communicates with the server 101 via a network 103, which may be a wired network or a wireless network.

[0058] In some embodiments, the terminal 102 can load the font generation model trained by the server 101, so that when it is necessary to perform style transfer on the font of the characters in the text information to be processed, based on the text information to be processed and the style image, by calling the font generation model, a style transfer text image with the same text content as the text information to be processed and the style of the style image is generated. The font style of the characters in the style transfer text image is more beautiful and personalized, and attracts more attention than the original text information to be processed.

[0059] In some embodiments, the terminal 102 may not load the font generation model trained by the server 101. When it is necessary to perform style transfer on the font of the characters in the text information to be processed, the terminal 102 sends a font generation request carrying the text information to be processed and the style image to the server 101. In response to the font generation request, the server 101 calls the font generation model to generate a style-transferred text image with the same text content as the text information to be processed and the style of the style image, and then sends the generated style-transferred text image to the terminal 102. The font style of the characters in the style-transferred text image is more beautiful and personalized, and is more eye-catching than the original text information to be processed.

[0060] The text information processing method described in the embodiments of the present disclosure can be applied to various scenarios that require font style migration, including but not limited to the following scenarios:

[0061] The first scenario: The style picture is a text picture

[0062] For this scenario, for any given text picture, the method provided by the embodiments of the present disclosure can migrate the font style of the text in the text picture to the text of the text information to be processed, so as to generate a style migration text picture with a font style close to that of the text picture and the same text content as the text content of the text information to be processed. For example, given a text picture with the regular script "Plum", and another text picture with the Song typeface "Red", using the method provided by the embodiments of the present disclosure, a text picture with the "Red" in a similar regular script style can be generated.

[0063] The second scenario: The style picture is a special effect picture

[0064] For this scenario, the method provided by the embodiments of the present disclosure uses OCR technology to recognize the text in the special effect picture. If the special effect picture contains text, the text information of the text is extracted, and the text information includes text content, font style, etc., and from the various style styles provided by the special effect picture, the style style to be migrated this time is selected, and then the text with this style style is updated at an appropriate position (such as a blank position, the original position of the text, etc.) in the special effect picture. For example, given a promotional picture of a game scene, the promotional picture is drawn with a flame scene and the text "Knife strikes explode artifacts", when the selected style style is flame, using the method provided by the embodiments of the present disclosure, the text "Knife strikes explode artifacts" with flame special effects can be displayed at the original position of the text in the promotional picture. Of course, even if the special effect picture does not contain text, but if the text matching the special effect picture can be obtained, special effect text can also be added to the special effect picture.

[0065] The neural network structure of the font generation model trained in the embodiments of the present disclosure mainly adopts the GAN network. Among them, the generator part can adopt an autoencoder structure, and the discriminator part can adopt a convolutional neural network structure. For the network structure of the generator part and the network architecture of the discriminator part, the following will be combined with Figure 2 and Figure 3 for description.

[0066] See Figure 2The generator mainly includes a style encoder, a content encoder, a feature mixer, and a feature mixer decoder. The style encoder can be composed of a residual convolutional neural network (ResNet), and its main function is to extract the style features of the input image. The content encoder can be composed of a deformable convolutional neural network (Deformable Convolution), and its main function is to extract the content features (including text features) of the input information. The feature mixer can be composed of a residual convolutional neural network, and its main function is to fuse the style features extracted by the style encoder with the content features extracted by the content encoder. The feature fusion decoder is composed of a convolutional neural network and a deformable convolutional neural network. Its main function is to perform global feature fusion using a symmetric number of ordinary convolutional layers and deformable convolutional layers with the content encoder and style encoder, obtaining a cascaded fusion feature. The cascaded fusion feature is then decoded to generate a style-transferred text image. Because the cascaded fusion feature contains richer text content and style information, the feature mixer decoder can obtain more information about style and content, compensating for the information lost during the neural network extraction process. For the two application scenarios mentioned above, the images input to the style encoder are different. For the first scenario, the style encoder inputs text images, the content encoder inputs text information, and the feature hybrid decoder generates text images; for the second scenario, the style encoder inputs special effects images, the content encoder inputs text information, and the feature hybrid decoder generates text images.

[0067] See also Figure 3 The network structure of the discriminator includes a convolution layer and a classification judgment layer. The convolution layer is used to perform a convolution operation on the input image to extract the features of the image, which includes real images and style transfer text images generated by the generator. The classification judgment layer is used to output the judgment label (true / false) of the image based on the extracted features, thereby judging whether the input image is a real image.

[0068] The embodiment of the present disclosure provides an image processing method, which can be applied to various scenarios, including but not limited to cloud technology, artificial intelligence, smart transportation, assisted driving, etc. Take the server 101 executing the embodiment of the present disclosure as an example, see Figure 4 , the method process provided by the embodiment of the present disclosure includes:

[0069] 401. The server extracts features of the sample text information and the sample style image to obtain sample text content features and sample style features.

[0070] The sample text information is used to provide text content, which can be in the form of an image or text, etc. The text in the sample text information can be any type of text, such as Chinese characters, English, Japanese, etc. Based on the text type of the characters in the sample text information, the trained font generation model can migrate the font styles of characters of different text types. For example, if the type of the characters in the sample text information is Chinese characters, the font generation model can migrate the font style of Chinese characters; if the type of the characters in the sample text information is English, the font generation model can migrate the font style of English; if the type of the characters in the sample text information is Japanese, the font generation model can migrate the font style of Japanese.

[0071] The sample style information is used to provide a style. The sample style image can be a sample special effects image or a sample text image. The sample text image contains text from other sample text information in addition to the sample text information. The sample special effects image is typically an image with clear texture and high contrast, for example, an image that provides special effects such as wind, fire, thunder, lightning, water, rain, and snow.

[0072] The server extracts the text content features of the sample text information and obtains the sample text content features corresponding to the sample text information. The sample text content features are features that can characterize the text content in the sample text information, including text type, text color, text size, text content, etc. The server extracts the style features of the sample style image and obtains the sample style features corresponding to the sample style image. The sample style features are features that can reflect the style of the sample style image. When the sample style image is a special effect style image, the sample style features include special effect style, etc.; when the sample style image is a sample text image, the sample style features include font style (such as Kaiti, Songti, Heiti, etc.), etc.

[0073] 402. The server performs feature fusion on the sample text content feature and the sample style feature to obtain a first sample fusion feature.

[0074] When fusing the sample text content features and the sample style features, the server may perform a weighted multiplication of the sample text content features and the sample style features. The resulting first sample fused feature initially learns the style of the sample style image and the text content of the sample text information. While this first sample fused feature can generate a style-transferred text image, it contains relatively little information about the style and text content, resulting in a low accuracy model.

[0075] 403. The server performs feature fusion on the first sample fusion feature and the sample text content feature to obtain a second sample fusion feature.

[0076] When fusing the first sample fusion feature with the sample text content feature, the server may perform a weighted multiplication of the first sample fusion feature and the sample text content feature. Because the sample text content feature is fused twice in the resulting second sample fusion feature, the second sample fusion feature contains richer information about the text content and can better reflect the text content in the sample text information than the first sample fusion feature. The model trained based on the second sample fusion feature can more accurately identify the text content in the text information to be processed, thereby reducing the content error rate of the generated text image.

[0077] 404. The server performs feature fusion on the sample style feature, the first sample fusion feature, and the second sample fusion feature to obtain a sample cascade fusion feature.

[0078] When fusing the sample style features, the first sample fusion features, and the second sample fusion features, the server may perform a weighted multiplication of the sample style features, the first sample fusion features, and the second sample fusion features. Because the resulting sample cascade fusion features undergo multi-level feature fusion, the features used to characterize text content and style in the sample cascade fusion features are enhanced. The font generation model trained based on the sample cascade fusion features learns more font generation knowledge, resulting in higher model accuracy.

[0079] 405. Based on the sample text information, sample style pictures, sample text content features and sample cascade fusion features, a font generation model is trained to obtain a trained font generation model.

[0080] The trained font generation model is used to generate style-transferred text images with the style of any style image.

[0081] In the method provided by the embodiments of the present disclosure, the sample style image can be a sample special effect image, or a sample text image containing characters from other sample text information in addition to the sample text information. The sample style image is different from the text content of the sample text information to be transferred to the font style, so that the sample style features and sample text content features to be fused are richer in both quantity and information content, thereby enabling the model to learn more knowledge about font generation. During the feature fusion process, the first sample fusion feature obtained by the first feature fusion learns the style of the sample style image and the text content of the sample text information. The second sample fusion feature obtained by the second feature fusion further enhances the sample text content feature on the basis of the first sample fusion feature. The sample cascade fusion feature obtained by the third feature fusion further enhances the sample style feature on the basis of the first sample fusion feature and the second sample fusion feature. In short, the relevant features of the style sample and text content in the sample cascade fusion features obtained by the above-mentioned several feature fusions are enhanced, and contain more information in the sample style image and the sample text information. Therefore, the font generation model trained based on the sample cascade fusion feature has higher accuracy, and the effect of the style transfer text image generated based on the font generation model is better.

[0082] The embodiment of the present disclosure provides an image processing method, which can be applied to various scenarios, including but not limited to cloud technology, artificial intelligence, smart transportation, assisted driving, etc. Figure 1 For example, the server in the embodiment of the present disclosure executes the embodiment of the present disclosure. Figure 5 , the method process provided by the embodiment of the present disclosure includes:

[0083] 501. The server obtains sample text information.

[0084] During the preparatory phase of model training, the server needs to obtain a large amount of sample text information. Each sample text information must contain at least one character with the same font style. To improve the generalization ability of the trained model, the text content and font styles of these sample text information must be as diverse as possible. Therefore, when obtaining sample information, it is important to avoid selecting text information with the same content or font style.

[0085] To increase the diversity of text content and font styles in the sample text information, the server in the embodiment of the present disclosure can select multiple fonts based on existing fonts, and select multiple commonly used characters, and then process each selected character based on each selected font to obtain sample text information for each character in each font. Taking the text type as Chinese characters and the sample text information in the form of images as an example, the server can obtain 300 existing fonts and select 1,000 commonly used Chinese characters. Then, using each of the 300 fonts, it processes each of the 1,000 Chinese characters to generate a text image for each Chinese character in each font, and ultimately obtain a total of 300,000 text images.

[0086] In order to facilitate the management of the multiple sample text information obtained and ensure the model accuracy of the trained font generation model, the embodiment of the present disclosure also divides the multiple sample text information obtained into a text information training set and a text information verification set, wherein the sample text information in the text information training set is used for model training, and the sample text information in the text information verification set is used for model verification. For example, for the 1,000 selected Chinese characters, 810 Chinese characters can be randomly selected, and then the text images corresponding to the selected 810 Chinese characters are composed of the text information training set. Correspondingly, the text images corresponding to the remaining 190 Chinese characters are composed of the text information verification set. After the text images corresponding to the 1,000 Chinese characters are divided by the above method, a text information training set containing 243,000 text images and a text information verification set containing 5,700 text images are obtained.

[0087] 502. The server obtains the sample style image.

[0088] The sample style image in the embodiment of the present disclosure can be a sample special effect image. The sample special effect image is an image containing a special effect object. The special effect object refers to an object with clear texture and strong contrast, such as wind, fire, thunder, lightning, etc. The texture of the special effect object is clear and the contrast is strong, so the style features can be extracted by the neural network. The sample style image can also be a sample text image containing Chinese characters in the sample text information. For the two types of sample style images, the server needs to adopt different methods to obtain the sample style images:

[0089] For the sample text images included in the sample style images, the server may use the images containing the characters in the sample text information obtained in step 501 as the sample style images based on the sample text information obtained in step 501 .

[0090] For the sample special effects style images included in the sample style images, the server can obtain images with clear texture and strong contrast (such as posters, promotional images, etc.) from the Internet as sample special effects images. In order to enable the trained model to learn more style knowledge of more sample special effects images, after obtaining the sample special effects images, the embodiment of the present disclosure will also process the sample special effects images to obtain a smooth mask image that can highlight the style of the sample special effects grid images. The specific processing process includes the following steps:

[0091] In the first step, the server segments the target area on the sample special effect image used to highlight the style of the sample special effect image to obtain a special effect mask image.

[0092] The server selects an area with clear texture and high contrast from the sample special effects image as the target area. It then uses an image segmentation algorithm to segment the target area from the sample special effects image, generating a special effects mask image containing the target area. Image segmentation algorithms include threshold segmentation, region segmentation, edge segmentation, and histogram segmentation.

[0093] In the second step, the server smoothes the special effect mask image to obtain a smoothed mask image.

[0094] Based on the special effect mask image obtained by segmentation, the server uses a smoothing algorithm to smooth the special effect mask image to remove noise in the special effect mask image and obtain a smoothed mask image. The smoothing algorithm includes interpolation method, linear smoothing method, convolution method, etc.

[0095] Figure 6 The process of generating a smooth mask image based on a sample special effect image is shown in Figure 6 For a sample special effects style picture containing a maple leaf image, the maple leaf image area is first segmented from the sample special effects style picture to obtain a special effects style mask containing the maple leaf image, and then the special effects style mask is smoothed to obtain a smooth mask picture with smooth edges.

[0096] Furthermore, after obtaining multiple sample style images, the server also divides the multiple sample style images into a style image training set for model training and a style image verification set for verifying the accuracy of the trained model. Considering that the font generation model is trained based on the sample text information in the text information training set and the sample style images in the style image training set, and the sample style images include sample text images, in order to make the functions of different data sets clearer and avoid data overlap in different data sets, when dividing the multiple sample style images into different data sets, the server may divide the sample text images containing the sample text information in the text information training set into the style image training set, and divide the sample text images containing the sample text information in the text information test set into the style image test set.

[0097] 503. The server extracts features of the sample text information and the sample style image to obtain sample text content features and sample style features.

[0098] Generally, the more samples used for model training, the higher the model accuracy. To improve the generalization ability and model accuracy of the trained font generation model, the server can combine each sample text message in the text information training set with each sample special effect image in the style image training set and a sample text image containing characters from other sample text messages other than the sample text message, thereby obtaining multiple image groups, and then perform model training based on the multiple image groups. Each image group includes a sample text message and a sample style image, where the sample style image is any sample special effect image or a sample text image containing characters from any sample text message other than the sample text message.

[0099] For each of the multiple image groups, the server invokes the font generation model to extract text content features of the sample text information and sample style features of the sample style image in each image group. Specifically, the server inputs the sample text information into the content encoder of the font generation model. The content encoder extracts the text content features of the sample text information by performing a convolution operation to obtain sample text content features. The server inputs the sample style image into the style encoder of the font generation model. The style encoder extracts the style features of the sample style image by performing a convolution operation to obtain sample style features.

[0100] 504. The server performs feature fusion on the sample text content feature and the sample style feature to obtain a first sample fusion feature.

[0101] Since the sample style image can be a sample special effect image or a sample text image, the server adopts different methods to fuse the sample text content features and the sample style features according to different situations of the sample style images.

[0102] In the first case, the sample style image is a sample text image

[0103] For the first case, the server inputs the sample style features of the sample style image and the sample text content features of the sample text image into the feature mixer of the font generation model. The feature mixer performs a convolution operation on the sample text content features and the sample style features to achieve feature fusion of the sample text content features and the sample style features, and obtains the first sample fusion feature.

[0104] In the second case, the sample style picture is a sample special effect picture

[0105] For the second case, the server obtains the smooth mask image corresponding to the sample special effect image, and then inputs the smooth mask image into the content encoder of the font generation model. The content encoder extracts the mask content features of the smooth mask image by performing a convolution operation on the smooth mask image. Next, the server inputs the mask content features and the sample style features into the feature mixer of the font generation model. The feature mixer performs a convolution operation on the mask content features and the sample style features to achieve feature fusion of the mask content features and the sample style features, thereby obtaining a mask fusion feature. Then, the server inputs the mask fusion feature, the sample text content feature, and the sample style feature into the feature mixer of the font generation model. The feature mixer performs a convolution operation on the mask fusion feature, the sample text content feature, and the sample style feature to achieve feature fusion of the mask fusion feature, the sample text content feature, and the sample style feature, thereby obtaining a first sample fusion feature.

[0106] In the disclosed embodiment, the server extracts the mask content features of the smoothed mask image and fuses the mask content features with other features, so that the model can learn the smoothing knowledge of the image content from the smoothed mask image, and thus can smooth the sample text content features extracted from the sample text image based on the learned knowledge, so that the font in the generated style transfer text image is smoother and more natural.

[0107] 505. The server performs feature fusion on the first sample fusion feature and the sample text content feature to obtain a second sample fusion feature.

[0108] The server inputs the first sample fusion feature and the sample text content feature into the feature mixing decoder of the font generation model. The feature mixing decoder performs a convolution operation on the first sample fusion feature and the sample text content feature to achieve feature fusion of the first sample fusion feature and the sample text content feature, thereby obtaining a second sample fusion feature.

[0109] 506. The server performs feature fusion on the sample style feature, the first sample fusion feature, and the second sample fusion feature to obtain a sample cascade fusion feature.

[0110] The server inputs the sample style features, the first sample fusion features and the second sample fusion features into the feature mixing decoder of the font generation model. The feature mixing decoder performs convolution operations on the sample style features, the first sample fusion features and the second sample fusion features to achieve feature fusion of the sample style features, the first sample fusion features and the second sample fusion features, and obtains the sample cascade fusion features.

[0111] In the embodiment of the present disclosure, the network structure used for feature fusion in the font generation model (for example, a feature mixer and a feature mixing decoder) includes a deformable convolution layer. When performing feature fusion, the deformable convolution layer can use the weight value of the pre-trained convolution kernel to perform convolution operations. However, this type of convolution operation does not achieve a good fusion effect between the features, and the fused features cannot inherit the relevant content of the features before fusion very well. In order to better integrate the sample style features and the sample text content features, so that the model can learn more relevant features of the style in the sample style pictures, the embodiment of the present disclosure uses the sample style features as the weight value of the convolution kernel in the deformable convolution layer, and then performs convolution operations on different features based on the weight value of the convolution kernel to obtain corresponding fusion features. For the fusion process involved in the embodiment of the present disclosure, the fusion features obtained include the following:

[0112] Based on the weight value of the convolution kernel, the server performs a convolution operation on the sample text content feature and the sample style feature to obtain a first sample fusion feature.

[0113] Based on the weight value of the convolution kernel, the server performs a convolution operation on the first sample fusion feature and the sample text content feature to obtain a second sample fusion feature.

[0114] Based on the weight value of the convolution kernel, the server performs a convolution operation on the sample style feature, the first sample fusion feature, and the second sample fusion feature to obtain a sample cascade fusion feature.

[0115] Figure 7 The feature extraction and feature fusion process from step 503 to step 506 is shown in FIG. Figure 7, the server inputs the sample style picture with the text content of "red" and the font style of official script into the style encoder, and outputs the sample style feature. The server inputs the sample text picture with the text content of "red" and the font style of regular script into the content encoder, and outputs the sample text content feature. Then, the server inputs the sample style feature and the sample text content feature into the feature mixer, and outputs the first sample fusion feature. Then, the server inputs the first sample fusion feature and the sample text content feature into the feature mixing decoder, and outputs the second sample fusion feature. Then, the server inputs the first sample fusion feature, the second sample fusion feature and the sample style feature into the feature mixing decoder, and outputs the sample cascade fusion feature. Then, the feature mixing decoder decodes the sample cascade fusion feature to obtain Figure 7 The style transferred text image shown.

[0116] 507. The server trains a font generation model based on the sample text information, the sample style image, the sample text content features and the sample cascade fusion features to obtain a trained font generation model.

[0117] Based on the sample text information, sample style images, sample text content features, and sample cascade fusion features, the server trains the font generation model. When obtaining the trained font generation model, the following method can be used:

[0118] 5071. The server decodes the sample cascade fusion features to obtain the first style transfer text image.

[0119] The server calls the feature hybrid decoder of the font generator to decode the sample cascade fusion features to obtain the first style transfer text image. This decoding process is actually a dimensionality increase process for the sample cascade fusion features.

[0120] 5072. The server calls the font generation model to perform authenticity recognition on the sample style image and the first style transfer text image to obtain a first recognition result and a second recognition result.

[0121] Since the first transferred text image is not a real image that originally existed, but a new image generated by the generator of the font generation model through feature extraction, feature fusion and feature decoding of sample text information and sample style images, when the accuracy of the generator of the font generation model is high, the generated first style transfer text image is closer to the real image. When the accuracy of the generator of the font generation model is low, the generated first style transfer text image does not look real enough. The main purpose of the model training in the embodiment of the present disclosure is to train style transfer text images that can look more real. The real is not a true or false image in the true sense, but is measured based on the recognition result of the discriminator of the font generation model. The discrimination result of the discriminator directly determines the training progress of the model. If the discriminator recognizes that the first style transfer text image is a real image, the model training process can be stopped. If the discriminator recognizes that the first style transfer text image is a fake image, the model needs to continue training.

[0122] In order to verify the authenticity of the first style-transferred text image generated by the generator, the server calls the font generation model, inputs the first style-transferred text image and the sample style image into the discriminator, and outputs a first recognition result of the authenticity of the sample style image and a second recognition result of the authenticity of the first style-transferred text image.

[0123] 5073. The server extracts the text content features of the first style-transferred text image to obtain the transferred text content features.

[0124] The server inputs the first style-transferred text image into the content encoder of the generator, extracts text content features of the first style-transferred text image, and obtains transferred text content features.

[0125] 5074. The server uses the sample text image containing Chinese characters in the sample text information as the style image, calls the font generation model, processes the sample text information and the sample text image containing Chinese characters in the sample text information, and obtains a second style transfer text image.

[0126] In order to better train the generator in the font generation model, the server also uses a sample text image containing characters in the sample text information as a style image, and then inputs the sample text information into the content encoder of the generator, outputs the sample text content features of the sample text information, and inputs the sample text image containing characters in the sample text information into the style encoder of the generator, outputs the sample style features of the sample text image, and then inputs the sample text content features and the sample style features into the feature mixer for feature fusion to obtain a first sample fusion feature, and then inputs the first sample fusion feature and the sample text content feature into the feature mixing decoder for feature fusion to obtain a second sample fusion feature, and then inputs the first sample fusion feature, the second sample fusion feature and the sample style feature into the feature mixing decoder, outputs the sample cascade fusion feature, and then decodes the sample fusion feature to generate a second style transfer text image.

[0127] 5075. The server adjusts the model parameters of the font generation model based on the sample text image containing characters in the sample text information, the second style transferred text image, the first recognition result, the second recognition result, the sample text content features, the transferred text content features and the target loss function to obtain a trained font generation model.

[0128] In order to complete the training task of the font generation model, the embodiment of the present disclosure designs a target loss function, which includes an adversarial loss function, a content loss function and a reconstruction loss function, and can be composed of the weighted addition of the adversarial loss function, the content loss function and the reconstruction loss function.

[0129] Among them, the adversarial loss function is a function that ensures that the first style transfer text image is a real image. To achieve this goal, the image generated by the generator is required to be as consistent as possible with the real image, so that it can try to deceive the discriminator. At the same time, the discriminator is required to have the largest possible value for the recognition results of the generated image and the real image, so that the generated image can achieve the effect of being indistinguishable from the real image. At this time, solving the problem of generating adversarial networks can be abstracted into a minimum-maximum optimization problem. If the discriminator makes an incorrect judgment during the solution process, it will be punished by the adversarial loss function. Adversarial loss function L adv The form can be:

[0130]

[0131] Among them, G represents the generator, D s I represents the discriminator; s Represents the sample style picture, I c Indicates sample text information, p s Represents the sample distribution of the style picture training set, p crepresents the sample distribution of the text information training set, E[*] represents the expected value of the distribution function, and D s (I s ) represents the probability that the discriminator identifies the sample style picture as a real picture, D s (G(I s ,I c ) represents the probability that the discriminator identifies the first style-transferred text image as a real image.

[0132] The above-mentioned adversarial loss function mainly focuses on the authenticity of the generated image. In order to prevent the model from collapsing and ensure that the feature codes extracted from the same content are consistent, the embodiment of the present disclosure designs a content loss function, which is a function that ensures the consistency of the content of the first style transfer text image and the sample text information. The content loss function L cnt The form can be:

[0133]

[0134] Among them, Z c represents the sample text content features extracted from the sample text information, f c (G(I s , I c ) represents the transferred text content features extracted from the first style transferred text image.

[0135] The reconstruction loss function is a function that ensures that the sample text image containing the characters in the sample text information and the second style transfer text image are the same image when the sample text image containing the characters in the sample text information is used as the style image. img The form can be:

[0136]

[0137] Among them, G(I c , I c ) represents the second style transfer text image.

[0138] Based on the three loss functions set above, as well as the sample text information, the second style transferred text image, the first recognition result, the second recognition result, the sample text content features, and the transferred text content features, the server adjusts the model parameters of the font generation model to obtain a trained font generation model. The specific training process is as follows:

[0139] In the first step, the server inputs the first recognition result and the second recognition result into the adversarial loss function, inputs the transferred text content features and the sample text content features into the content loss function, inputs the sample text image containing the text in the sample text information and the second style transferred text image into the reconstruction loss function, and outputs the adversarial loss function value, the content loss function value and the reconstruction loss function value.

[0140] In the second step, the server adjusts the model parameters of the font generation model based on the adversarial loss function value, the content loss function value, and the reconstruction loss function value to obtain the trained font generation model.

[0141] The training goal of the font generation model in the disclosed embodiment is to make the images generated by the generator as realistic as possible, and the discriminator's ability to discriminate images as strong as possible. The closer the images generated by the generator are to the real images, and the smaller the difference between the images generated by the generator and the real images, the larger the value of the discriminator's recognition result, and the correspondingly smaller the values ​​of the adversarial loss function, content loss function, and reconstruction loss function. It can be seen that the authenticity of the images generated by the generator and the discriminator's ability to discriminate are reflected in the function values ​​of the three loss functions. By calculating the function values ​​of the three loss functions, the font generation model can be trained.

[0142] To achieve the model training goal, the server continuously adjusts the model parameters of the font generation model and calculates the function values ​​of the adversarial loss function, the content loss function, and the reconstruction loss function based on the data obtained from each adjustment of the model parameters, thereby obtaining the adversarial loss function value, the content loss function value, and the reconstruction loss function value. After multiple parameter adjustment processes, the server obtains the parameter values ​​of the model parameters that minimize the adversarial loss function value, the content loss function value, and the reconstruction loss function value, and then uses the font generation model corresponding to the parameter values ​​of the model parameters as the trained font generation model.

[0143] Figure 8 For the model training process based on the target loss function, see Figure 8 , a generator is used to generate font style transfer text images, the generated style transfer text images and real data (including sample style images) are input into the discriminator, and the recognition results are output. Then, according to the loss function related to the generator and the loss function related to the discriminator, the model parameters of the generator and the discriminator are adjusted to obtain the trained generator and the trained discriminator, and then the trained generator and the trained discriminator constitute the trained font generation model.

[0144] After training the font generation model using the above method, the server will further train the trained font generation model based on a text information test set and a style image test set. When the trained font generation model meets the expected training results, it can be published, allowing devices equipped with the font generation model to complete the style transfer task and generate a style-transferred text image. The font of the text in this style-transferred text image not only meets the creator's personalized needs but also attracts more attention.

[0145] In the method provided by the embodiments of the present disclosure, the sample style image can be a sample special effect image, or a sample text image containing characters from other sample text information in addition to the sample text information. The sample style image is different from the text content of the sample text information to be transferred to the font style, so that the sample style features and sample text content features to be fused are richer in both quantity and information content, thereby enabling the model to learn more knowledge about font generation. During the feature fusion process, the first sample fusion feature obtained by the first feature fusion learns the style of the sample style image and the text content of the sample text information. The second sample fusion feature obtained by the second feature fusion further enhances the sample text content feature on the basis of the first sample fusion feature. The sample cascade fusion feature obtained by the third feature fusion further enhances the sample style feature on the basis of the first sample fusion feature and the second sample fusion feature. In short, the relevant features of the style sample and text content in the sample cascade fusion features obtained by the above-mentioned several feature fusions are enhanced, and contain more information in the sample style image and the sample text information. Therefore, the font generation model trained based on the sample cascade fusion feature has higher accuracy, and the effect of the style transfer text image generated based on the font generation model is better.

[0146] The embodiment of the present disclosure provides a text information processing method, taking an electronic device as an example to execute the method, the electronic device is equipped with Figure 8 The font generation model trained in the embodiment shown. The electronic device can be Figure 1 The server 101 in Figure 1 Terminal 102 in Figure 9 , the method process provided by the embodiment of the present disclosure includes:

[0147] 901. The electronic device obtains a style picture with a style to be transferred and text information to be processed.

[0148] The style image with the style to be transferred can provide at least one style. The style image with the style to be transferred can be a style special effects image that does not contain any text but contains special effects objects, or a style special effects image that contains text in at least one font style and contains special effects objects, or a plain text image that contains text in at least one font style. The text information to be processed can come from the style image with the style to be transferred. For example, if the style image is a picture with a flame scene and the words "Da Dao Explodes the Artifact" written on it, the text information to be processed is the words "Da Dao Explodes the Artifact" in the picture. The text information to be processed can also come from the input operation of the operation object, that is, the text information to be processed is the text information input by the operation object. The text information to be processed can also come from other text images.

[0149] 902. The electronic device calls the font generation model to process the style image and the text information to be processed to obtain a style-transferred text image.

[0150] The electronic device inputs the style image and the text information to be processed into the font generation model, and outputs a style-transferred text image. Depending on the number of style patterns provided by the style image, it can be divided into the following situations:

[0151] In the first case, when a style image provides a style pattern, the electronic device calls the font generation model to directly transfer the style pattern of the style image to the text of the text information to be processed, and generates a style-transferred text image with the same text content as the text information to be processed and the style pattern of the style image.

[0152] In the second case, when the style image can provide at least two styles, the electronic device can return a style selection request of at least two styles to the operation object, and the operation object selects one style from the at least two styles and returns it to the electronic device. Based on the style selected by the operation object, the electronic device calls the font generation model to migrate the style selected by the operation object to the text of the text information to be processed, and generates a style-migrated text image with the same text content as the text content of the text information to be processed and the style selected by the operation object.

[0153] The above is explained using the example of migrating the same style to each character in the text information to be processed. Of course, when the style image can provide at least two styles and the text information to be processed contains at least two characters, the style to be migrated can be selected for different characters, thereby generating style-migrated text images with different style styles for different characters.

[0154] See also Figure 10 , an embodiment of the present disclosure provides a text information processing device, the device comprising:

[0155] Extraction module 1001, configured to extract features of sample text information and sample style images to obtain sample text content features and sample style features. The sample style images are sample special effects images or sample text images, and the sample text images contain other sample text information in addition to the sample text information.

[0156] A fusion module 1002 is configured to fuse the sample text content feature and the sample style feature to obtain a first sample fusion feature;

[0157] The fusion module 1002 is further configured to fuse the first sample fusion feature with the sample text content feature to obtain a second sample fusion feature;

[0158] The fusion module 1002 is further configured to fuse the sample style feature, the first sample fusion feature, and the second sample fusion feature to obtain a sample cascade fusion feature, wherein the sample cascade fusion feature is a feature that enhances both text content and style.

[0159] The training module 1003 is used to train the font generation model based on sample text information, sample style pictures, sample text content features and sample cascade fusion features to obtain a trained font generation model. The trained font generation model is used to generate a style-transferred text picture with the style of any style picture.

[0160] In another embodiment of the present disclosure, when the sample style image is a sample special effect image, the extraction module 1001 is further configured to extract features of a smooth mask image corresponding to the sample special effect image to obtain mask content features, where the smooth mask image is an image that can highlight the style of the sample special effect image.

[0161] The fusion module 1002 is further used to fuse the sample style features with the mask content features to obtain the mask fusion features;

[0162] The fusion module 1002 is further configured to fuse the mask fusion feature, the sample text content feature, and the sample style feature to obtain a first sample fusion feature.

[0163] In another embodiment of the present disclosure, the apparatus further comprises:

[0164] A segmentation module is used to segment a target area on the sample special effect image for highlighting the style of the sample special effect image to obtain a special effect mask image;

[0165] The processing module is used to smooth the special effect mask image to obtain a smoothed mask image.

[0166] In another embodiment of the present disclosure, the network structure for feature fusion in the font generation model includes a deformable convolution layer, and the apparatus further includes:

[0167] A determination module is used to use the sample style features as the weight value of the convolution kernel in the deformable convolution layer;

[0168] The convolution module is used to perform convolution operations on different features based on the weight values ​​of the convolution kernel to obtain corresponding fusion features.

[0169] In another embodiment of the present disclosure, a convolution module is used to perform a convolution operation on a sample text content feature and a sample style feature based on a weight value of a convolution kernel to obtain a first sample fusion feature; to perform a convolution operation on the first sample fusion feature and the sample text content feature based on a weight value of the convolution kernel to obtain a second sample fusion feature; and to perform a convolution operation on the sample style feature, the first sample fusion feature and the second sample fusion feature based on the weight value of the convolution kernel to obtain a sample cascade fusion feature.

[0170] In another embodiment of the present disclosure, the training module 1003 is used to decode the sample cascade fusion features to obtain a first style transfer text image; call the font generation model to perform authenticity identification on the sample style image and the first style transfer text image to obtain a first identification result and a second identification result; extract the text content features of the first style transfer text image to obtain the transfer text content features; use the sample text image containing sample text information as the style image, call the font generation model, process the sample text information and the sample text image containing sample text information to obtain a second style transfer text image; adjust the model parameters of the font generation model based on the sample text image containing sample text information, the second style transfer text image, the first recognition result, the second recognition result, the sample text content features, the transfer text content features and the target loss function to obtain a trained font generation model.

[0171] In another embodiment of the present disclosure, the target loss function includes an adversarial loss function, a content loss function, and a reconstruction loss function. The training module 1003 is configured to input the first recognition result and the second recognition result into the adversarial loss function, input the transferred text content features and the sample text content features into the content loss function, input the sample text image containing the sample text information and the second style transferred text image into the reconstruction loss function, and output the adversarial loss function value, the content loss function value, and the reconstruction loss function value; based on the adversarial loss function value, the content loss function value, and the reconstruction loss function value, adjust the model parameters of the font generation model to obtain a trained font generation model;

[0172] Among them, the adversarial loss function is a function to ensure that the first style-transferred text image is a real image, the content loss function is a function to ensure that the content of the first style-transferred text image and the sample text information is consistent, and the reconstruction loss function is a function to ensure that the sample text image containing the sample text information and the second style-transferred text image are the same image.

[0173] In summary, in the device provided by the embodiment of the present disclosure, the sample style picture can be a sample special effect picture, or a sample text picture of other sample text information other than the sample text information. The sample style picture is different from the text content of the sample text information to be subjected to font style transfer, so that the sample style features and sample text content features to be fused are richer in both quantity and information volume, thereby enabling the model to learn more knowledge about font generation. During the feature fusion process, the first sample fusion feature obtained by the first feature fusion learns the style of the sample style picture and the text content of the sample text information. The second sample fusion feature obtained by the second feature fusion further enhances the sample text content feature on the basis of the first sample fusion feature. The sample cascade fusion feature obtained by the third feature fusion further enhances the sample style feature on the basis of the first sample fusion feature and the second sample fusion feature. In short, the relevant features of the style sample and text content in the sample cascade fusion features obtained by the above-mentioned several feature fusions are enhanced, and contain more information in the sample style picture and the sample text information. Therefore, the font generation model trained based on the sample cascade fusion feature has higher accuracy, and the effect of the style transfer text picture generated based on the font generation model is better.

[0174] See also Figure 11 , an embodiment of the present disclosure provides a text information processing device, the device comprising:

[0175] An acquisition module 1101 is configured to acquire a style image having a style to be transferred and text information to be processed;

[0176] The processing module 1102 is used to call the font generation model to process the style image and the text information to be processed to obtain a style-transferred text image.

[0177] Figure 12 A server for text information processing is shown according to an exemplary embodiment. Figure 12Server 1200 includes a processing component 1222, which further includes one or more processors, and memory resources represented by memory 1232 for storing instructions executable by processing component 1222, such as applications. The applications stored in memory 1232 may include one or more modules, each corresponding to a set of instructions. Furthermore, processing component 1222 is configured to execute the instructions to perform the functions performed by the server in the above-described text message processing method.

[0178] The server 1200 may also include a power supply component 1226 configured to perform power management of the server 1200, a wired or wireless network interface 1250 configured to connect the server 1200 to the network, and an input / output (I / O) interface 1258. The server 1200 may operate based on an operating system stored in the memory 1232, such as Windows Server 200. TM , Mac OS X TM , Unix TM ,Linux TM , FreeBSD TM or similar.

[0179] An embodiment of the present disclosure provides a computer-readable storage medium, wherein at least one program code is stored in the storage medium, and the at least one program code is loaded and executed by a processor to implement a text information processing method. The computer-readable storage medium can be non-transitory. For example, the computer-readable storage medium can be a read-only memory (ROM), a random access memory (RAM), a CD-ROM (Compact Disc Read-Only Memory), a magnetic tape, a floppy disk, an optical data storage device, etc.

[0180] An embodiment of the present disclosure provides a computer program product, which includes computer program code. The computer program code is stored in a computer-readable storage medium. A processor of a server reads the computer program code from the computer-readable storage medium, and the processor executes the computer program code, so that the server performs a text information processing method.

[0181] Those skilled in the art will understand that all or part of the steps to implement the above embodiments may be accomplished by hardware, or by a program to instruct the relevant hardware, and the program may be stored in a computer-readable storage medium, which may be a read-only memory, a disk, or an optical disk, etc.

[0182] The above description is merely an optional embodiment of the present disclosure and is not intended to limit the present disclosure. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present disclosure shall be included in the scope of protection of the present disclosure.

Claims

1. A text information processing method, characterized in that: The method comprises: Extracting features of sample text information and sample style images to obtain sample text content features and sample style features, wherein the sample style images are sample special effects images or sample text images, and the sample text images contain text in other sample text information except the sample text information; Performing a convolution operation on the sample text content feature and the sample style feature through a feature mixer of a font generation model to obtain a first sample fusion feature; Inputting the first sample fusion feature and the sample text content feature into a feature mixing decoder of the font generation model and performing a convolution operation to obtain a second sample fusion feature, wherein the second sample fusion feature reflects the text content in the sample text information better than the first sample fusion feature reflects the text content in the sample text information; Inputting the sample style feature, the first sample fusion feature, and the second sample fusion feature into the feature hybrid decoder, performing a convolution operation, and obtaining a sample cascade fusion feature, wherein the sample cascade fusion feature is a feature that enhances both text content and style; The feature mixer and the feature mixing decoder include a deformable convolution layer, and the convolution operations performed in the feature mixer and the feature mixing decoder use the sample style features as weight values ​​of the convolution kernels in the corresponding deformable convolution layers; Based on the sample text information, the sample style picture, the sample text content features and the sample cascade fusion features, the font generation model is trained to obtain a trained font generation model, and the trained font generation model is used to generate a style-transferred text picture with the style of any style picture.

2. The method according to claim 1, characterized in that In a case where the sample style picture is a sample special effect picture, before performing a convolution operation on the sample text content feature and the sample style feature to obtain a first sample fusion feature, the method further includes: Extracting features of a smooth mask image corresponding to the sample special effect image to obtain mask content features, wherein the smooth mask image is an image that can highlight the style of the sample special effect image; Performing feature fusion on the sample style feature and the mask content feature to obtain a mask fusion feature; The convolution operation of the sample text content feature and the sample style feature to obtain a first sample fusion feature includes: The mask fusion feature, the sample text content feature, and the sample style feature are fused to obtain the first sample fusion feature.

3. The method according to claim 2, characterized in that Before extracting the features of the smooth mask image corresponding to the sample special effect image and obtaining the mask content features, the method further includes: Segmenting a target area on the sample special effect picture for highlighting the style of the sample special effect picture to obtain a special effect mask picture; The special effect mask image is smoothed to obtain the smoothed mask image.

4. The method according to claim 1, wherein The method further comprises: Based on the weight value of the convolution kernel, performing a convolution operation on the sample text content feature and the sample style feature to obtain the first sample fusion feature; Based on the weight value of the convolution kernel, convolution operation is performed on the first sample fusion feature and the sample text content feature to obtain a second sample fusion feature; Based on the weight value of the convolution kernel, a convolution operation is performed on the sample style feature, the first sample fusion feature, and the second sample fusion feature to obtain the sample cascade fusion feature.

5. The method according to claim 1, wherein The step of training the font generation model based on the sample text information, the sample style image, the sample text content features, and the sample cascade fusion features to obtain the trained font generation model includes: Decoding the sample cascade fusion features to obtain a first style transfer text image; Calling the font generation model to perform authenticity recognition on the sample style image and the first style-transferred text image to obtain a first recognition result and a second recognition result; Extracting text content features of the first style-transferred text image to obtain transferred text content features; Using a sample text image containing characters in the sample text information as a style image, calling the font generation model, processing the sample text information and the sample text image containing characters in the sample text information, and obtaining a second style-transferred text image; Based on a sample text image containing characters in the sample text information, the second style transferred text image, the first recognition result, the second recognition result, the sample text content features, the transferred text content features and the target loss function, the model parameters of the font generation model are adjusted to obtain the trained font generation model.

6. The method according to claim 5, characterized in that The target loss function includes an adversarial loss function, a content loss function, and a reconstruction loss function. The method adjusts the model parameters of the font generation model based on the sample text image containing characters in the sample text information, the second style transferred text image, the first recognition result, the second recognition result, the sample text content feature, the transferred text content feature, and the target loss function to obtain the trained font generation model, including: Inputting the first recognition result and the second recognition result into the adversarial loss function, inputting the transferred text content features and the sample text content features into the content loss function, inputting the sample text image containing the characters in the sample text information and the second style transferred text image into the reconstruction loss function, and outputting the adversarial loss function value, the content loss function value, and the reconstruction loss function value; Adjusting the model parameters of the font generation model based on the adversarial loss function value, the content loss function value, and the reconstruction loss function value to obtain the trained font generation model; Among them, the adversarial loss function is a function that ensures that the first style-transferred text image is a real image, the content loss function is a function that ensures that the content of the first style-transferred text image is consistent with that of the sample text information, and the reconstruction loss function is a function that ensures that the sample text image containing the characters in the sample text information and the second style-transferred text image are the same image.

7. A text information processing method, characterized in that: The method is applied to the font generation model trained by any one of claims 1 to 6, and the method comprises: Obtain the style image with the style to be transferred and the text information to be processed; The font generation model is called to process the style image and the text information to be processed to obtain a style-transferred text image.

8. A text information processing device, characterized in that: The device comprises: an extraction module, configured to extract features of sample text information and sample style images to obtain sample text content features and sample style features, wherein the sample style images are sample special effects images or sample text images, and the sample text images contain text from other sample text information in addition to the sample text information; a fusion module, configured to perform a convolution operation on the sample text content feature and the sample style feature through a feature mixer of a font generation model to obtain a first sample fusion feature; The fusion module is further configured to input the first sample fusion feature and the sample text content feature into a feature mixing decoder of the font generation model to perform a convolution operation to obtain a second sample fusion feature, wherein the second sample fusion feature reflects the text content in the sample text information better than the first sample fusion feature does. The fusion module is further configured to input the sample style feature, the first sample fusion feature, and the second sample fusion feature into the feature hybrid decoder, perform a convolution operation, and obtain a sample cascade fusion feature, wherein the sample cascade fusion feature is a feature that enhances both text content and style; The feature mixer and the feature mixing decoder include a deformable convolution layer, and the convolution operations performed in the feature mixer and the feature mixing decoder use the sample style features as weight values ​​of the convolution kernels in the corresponding deformable convolution layers; A training module is used to train the font generation model based on the sample text information, the sample style picture, the sample text content features and the sample cascade fusion features to obtain a trained font generation model, and the trained font generation model is used to generate a style-transferred text picture with the style of any style picture.

9. The device according to claim 8, characterized in that In the case where the sample style picture is a sample special effect picture, the extraction module is used to extract features of a smooth mask picture corresponding to the sample special effect picture to obtain mask content features, wherein the smooth mask picture is a picture that can highlight the style of the sample special effect picture; The fusion module is further configured to fuse the sample style feature with the mask content feature to obtain a mask fusion feature; The fusion module is further configured to fuse the mask fusion feature, the sample text content feature, and the sample style feature to obtain the first sample fusion feature.

10. The device according to claim 9, characterized in that The device further comprises: a segmentation module, configured to segment a target area on the sample special effect picture for highlighting the style of the sample special effect picture to obtain a special effect mask picture; The processing module is used to perform smoothing processing on the special effect mask image to obtain the smoothed mask image.

11. The device according to claim 8, characterized in that The apparatus further comprises a convolution module, configured to: Based on the weight value of the convolution kernel, performing a convolution operation on the sample text content feature and the sample style feature to obtain the first sample fusion feature; Based on the weight value of the convolution kernel, convolution operation is performed on the first sample fusion feature and the sample text content feature to obtain a second sample fusion feature; Based on the weight value of the convolution kernel, a convolution operation is performed on the sample style feature, the first sample fusion feature, and the second sample fusion feature to obtain the sample cascade fusion feature.

12. The device according to claim 8, characterized in that The training module is used to: Decoding the sample cascade fusion features to obtain a first style transfer text image; Calling the font generation model to perform authenticity recognition on the sample style image and the first style-transferred text image to obtain a first recognition result and a second recognition result; Extracting text content features of the first style-transferred text image to obtain transferred text content features; Using a sample text image containing characters in the sample text information as a style image, calling the font generation model, processing the sample text information and the sample text image containing characters in the sample text information, and obtaining a second style-transferred text image; Based on a sample text image containing characters in the sample text information, the second style transferred text image, the first recognition result, the second recognition result, the sample text content features, the transferred text content features and the target loss function, the model parameters of the font generation model are adjusted to obtain the trained font generation model.

13. The device according to claim 12, characterized in that The target loss function includes an adversarial loss function, a content loss function, and a reconstruction loss function. The training module is used to: Inputting the first recognition result and the second recognition result into the adversarial loss function, inputting the transferred text content features and the sample text content features into the content loss function, inputting the sample text image containing the characters in the sample text information and the second style transferred text image into the reconstruction loss function, and outputting the adversarial loss function value, the content loss function value, and the reconstruction loss function value; Adjusting the model parameters of the font generation model based on the adversarial loss function value, the content loss function value, and the reconstruction loss function value to obtain the trained font generation model; Among them, the adversarial loss function is a function that ensures that the first style-transferred text image is a real image, the content loss function is a function that ensures that the content of the first style-transferred text image is consistent with that of the sample text information, and the reconstruction loss function is a function that ensures that the sample text image containing the characters in the sample text information and the second style-transferred text image are the same image.

14. A text information processing device, characterized in that: The device is applied to the font generation model trained by any one of claims 1 to 6, and comprises: An acquisition module is used to obtain style images with the style to be transferred and text information to be processed; The processing module is used to call the font generation model, process the style image and the text information to be processed, and obtain a style-transferred text image.

15. A server, characterized in that: The server includes a processor and a memory, wherein the memory stores at least one program code, and the at least one program code is loaded and executed by the processor to implement the text information processing method according to any one of claims 1 to 6.

16. A computer-readable storage medium, characterized in that The storage medium stores at least one program code, and the at least one program code is loaded and executed by the processor to implement the text information processing method according to any one of claims 1 to 6.

17. A computer program product, characterized in that The computer program product includes a computer program code, which is stored in a computer-readable storage medium. A processor of a server reads the computer program code from the computer-readable storage medium, and the processor executes the computer program code, so that the server executes the text information processing method according to any one of claims 1 to 6.

Images