An unsupervised general artistic character generation method

Abstract

The application discloses a kind of unsupervised general artistic character generation methods.The method comprises: constructing general artistic character generation model, including encoder, first decoder and second decoder;Optimize general artistic character generation model, optimization process includes texturing stage, font migration stage and texturing stage, in texturing stage, input image is font reference image, after passing through encoder and second decoder, the texture information that this font reference image carries is removed, and the corresponding mask image is obtained;In font migration stage, input image is content image and the mask image obtained in texturing stage, after passing through encoder and first decoder, the image after doing font migration to content image is obtained;In texturing stage, input image is the image obtained in font migration stage and texture reference image, after passing through encoder and second decoder, texture result image is obtained.The application can be realized in the fine adjustment of artistic character generation process and has strong universality.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and more specifically, to an unsupervised, general method for generating artistic characters. Background Technology

[0002] Computer vision is shifting from statistical methods to deep learning neural network methods. Deep learning, as a powerful tool enabling machines to learn autonomously, has been continuously developing in recent years. The development of artificial neural networks allows computers to train themselves based on input information within supervised or unsupervised learning frameworks, resulting in significantly improved autonomous learning performance. Deep learning can be applied to various aspects of computer vision, such as image recognition, object tracking, image classification, and object detection.

[0003] Image generation is a crucial problem in computer vision, closely related to animation production. Broadly speaking, any problem involving the generation of new image content can be categorized as image generation. Image stylization refers to using algorithms to map the style of an artistic image onto other natural images, allowing the original natural image to retain its original semantic content while possessing the artistic style of the original artistic image. The concept of image stylization arose from people's attraction to the works of certain master artists and their desire to possess images in the same artistic style. However, redrawing images in a specific style requires significant investment of technical personnel and resources, leading researchers to develop algorithms for image stylization. The excellent performance of deep learning in computer vision has drawn the attention of stylization researchers. With the advent of convolutional neural networks, high-level image features can be effectively utilized, eliminating the limitations of traditional stylization algorithms. The intervention of deep learning has allowed traditional image processing methods to be better utilized, and image feature information is fully leveraged. It can be said that neural networks have significantly advanced research in the field of image stylization. Although some progress has been made in the field of image stylization, enabling some simple applications, continuous development and improvement are still needed before stylization technology can be fully mature.

[0004] Artistic font generation is an emerging research topic in the field of image generation, attracting increasing attention due to its wide application. Artistic fonts can be considered to have three basic components: content, font, and texture. For controllable and general-purpose artistic font generation, existing artistic font generation models usually focus on adjusting one of these three components, which is a suboptimal solution.

[0005] Artistic font generation has wide applications in the field of art and design and is gradually becoming an important research area in computer vision. The goal of artistic font style transfer is to render arbitrary font images based on given artistic font effect images. The challenges of artistic font style transfer mainly stem from the diversity of artistic fonts and the difficulty in collecting paired training data.

[0006] Specifically, artistic fonts possess inherent diversity, which can be categorized into three dimensions: content, font, and texture. Content diversity refers to the diversity of characters, such as Chinese characters, English characters, and characters from other languages. For a given character content, the style of artistic fonts can be divided into font style and texture style. Existing artistic font style transfer algorithms can only adjust one of these two styles. For example, deformable convolution is used to handle font transfer tasks. Another example is focusing on texture transfer, which involves rendering a font image into a target texture. Existing technologies typically only consider a subtask of the artistic font generation task, namely the font transfer task or the texture transfer task, leading to limitations in real-world scenarios.

[0007] Furthermore, many recent models require supervised training data. Due to the data-driven nature of deep neural networks, a large amount of training data is necessary for the network to learn effective and robust representations of artistic characters. This supervised training method suffers from a lack of paired character image data; the abundant artistic character images already existing in real-world scenarios cannot be used to train deep neural network models. Therefore, it is necessary to introduce unsupervised training methods into the field of artistic character generation to provide a key solution to the aforementioned problems. Summary of the Invention

[0008] The purpose of this invention is to overcome the shortcomings of the prior art and provide an unsupervised, universal method for generating artistic fonts, which includes the following steps:

[0009] Construct a general artistic font generation model, including an encoder, a first decoder, and a second decoder;

[0010] The optimization of the general artistic font generation model includes a detexturization stage, a font transfer stage, and a texturization stage, wherein:

[0011] In the detexturization stage, the input image is a font reference image. After passing through the encoder and the second decoder, the texture information of the font reference image is removed to obtain the corresponding mask image.

[0012] In the font migration stage, the input image is the content image and the mask image obtained in the detexturization stage. After passing through the encoder and the first decoder, the image after font migration of the content image is obtained.

[0013] In the texturing stage, the input image is the image obtained in the font migration stage and the texture reference image. After passing through the encoder and the second decoder, the texturing result image is obtained.

[0014] Compared with existing technologies, the advantages of this invention are that it proposes an unsupervised general artistic word generation model that can achieve fine-tuning in the artistic word generation process; and in order to solve the difficulty of collecting paired data, the model of this invention can be optimized under unsupervised settings, so that any artistic word image can be used effectively.

[0015] Other features and advantages of the invention will become clear from the following detailed description of exemplary embodiments of the invention with reference to the accompanying drawings. Attached Figure Description

[0016] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments of the invention and, together with their description, serve to explain the principles of the invention.

[0017] Figure 1 This is a schematic diagram of the process of an unsupervised general artistic font generation method according to an embodiment of the present invention;

[0018] Figure 2 This is a flowchart of an unsupervised general artistic font generation method according to an embodiment of the present invention;

[0019] Figure 3 This is a network structure diagram of an unsupervised general artistic word generation model according to an embodiment of the present invention;

[0020] Figure 4 This is a structural diagram of an encoder according to an embodiment of the present invention;

[0021] Figure 5 This is a structural diagram of a decoder according to an embodiment of the present invention;

[0022] Figure 6 This is a structural diagram of a domain discriminator according to an embodiment of the present invention;

[0023] Figure 7 This is a structural diagram of an image block discriminator according to an embodiment of the present invention;

[0024] Figure 8 This is a schematic diagram illustrating the effects of the contrastive learning loss function and the image patch discriminator loss function according to an embodiment of the present invention;

[0025] Figure 9 This is a schematic diagram illustrating the application process of an unsupervised general artistic font generation model according to an embodiment of the present invention. Detailed Implementation

[0026] Various exemplary embodiments of the present invention will now be described in detail with reference to the accompanying drawings. It should be noted that, unless otherwise specifically stated, the relative arrangement, numerical expressions, and values ​​of the components and steps set forth in these embodiments do not limit the scope of the invention.

[0027] The following description of at least one exemplary embodiment is merely illustrative and is in no way intended to limit the invention or its application or use.

[0028] Techniques, methods, and equipment known to those skilled in the art may not be discussed in detail, but where appropriate, such techniques, methods, and equipment should be considered part of the specification.

[0029] In all the examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not as limitations. Therefore, other examples of exemplary embodiments may have different values.

[0030] It should be noted that similar labels and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be discussed further in subsequent figures.

[0031] This invention proposes a novel general artistic font generation method (or GenText). This method decouples font transfer and texture transfer through unsupervised learning, thereby achieving general artistic font style transfer. By analyzing the essential characteristics of artistic fonts, it decomposes artistic fonts into three basic components: content, font, and texture. An unsupervised general artistic font generation model is established, and after training and optimizing the model with data, general artistic font generation can be achieved.

[0032] See Figure 1 As shown, this invention integrates three distinct stages: texturing, detexturing, and font transfer, all accomplished using a single network structure. In the detexturing stage, font information is extracted from the font reference image; then, in the font transfer stage, a font image consistent with the font reference image in terms of font type is generated; finally, in the texturing stage, this font image is rendered into a final image consistent with the texture reference image in terms of texture. In terms of network structure, the provided unsupervised general artistic font generation model includes a powerful encoder and two independent decoders. One encoder encodes glyph structure information, font style information, and texture style information; one decoder is used for font transfer; and the other decoder is used for texturing and detexturing.

[0033] Specifically, see Figure 2 As shown, the provided unsupervised general artistic font generation method includes the following steps:

[0034] Step S210: Construct a general artistic font generation model that integrates three tasks: texturing, detexturing, and font migration.

[0035] An unsupervised general-purpose artistic font generation model is used to achieve fine-tuning of the artistic font generation process. "General-purpose" is mainly reflected in two aspects. First, the model can be trained under unsupervised conditions, allowing a large number of artistic font images from real-world scenarios to serve as training data, thus optimizing a robust model as much as possible. Second, by considering content, font style, and texture style as the three basic components of artistic fonts, the model can be applied to general artistic font generation tasks.

[0036] Specifically, an encoder is used to encode glyph structure information into a spatial representation map, and font style and texture style information are encoded into global representation vectors respectively. Two decoders are used to achieve the artistic font generation task: one decoder for font transfer, and the other for texturing and detexturing. Depending on the objective, the model can be divided into three distinct stages: detexturing, font transfer, and texturing. First, in the detexturing stage, font information is extracted from the font reference image. Then, in the font transfer stage, a font image consistent with the font reference image in terms of font type is generated. Finally, in the texturing stage, the font image is rendered into a final image consistent with the texture reference image in terms of texture. Therefore, the model of this invention can not only achieve the classic font transfer and texture transfer tasks, but also perform fine-tuning of the font and texture information in artistic fonts.

[0037] See Figure 3 As shown, in terms of network structure, the general artistic font generation model contains a powerful encoder E and two independent decoders {G}. F G T}, where decoder G F Decoder G is used for font migration. T Used for texturing and detexturing. For any image img, encoder E yields two representations z. sp ,z gl =E(img), where z sp It is a feature map with spatial dimensions, containing spatial structural information of the input image (img), z gl It is a feature vector without spatial dimension, containing global style information (font style or texture style) of the input image (img). Additionally, it includes three discriminators: a domain discriminator D... x and D y and image patch discriminator D patch .

[0038] Encoder E maps the input image to a spatial feature code (a feature map with spatial dimensions) and a global feature code (a feature vector without spatial dimensions).

[0039] See Figure 4 As shown, in one embodiment, the backbone of the encoder network consists of four downsampled residual blocks. The input image is transformed into an intermediate representation map after passing through the encoder backbone, and then passed through two parallel branches to obtain spatial feature encoding and global feature encoding, respectively. Specifically, spatial features are obtained by passing the intermediate representation map through two convolutional layers, while global features are obtained by passing the intermediate representation map through two convolutional layers, average pooling, and one fully connected layer. The asymmetric design of spatial and global features strengthens the inductive bias of the encoder, promoting orthogonal encoding. For example, for an input image of size 256×256, the size of the spatial features obtained after encoder processing is 16×16×8, and the size of the global features is 1×1×2048. The biggest difference between spatial and global features lies in whether they encode spatial information. Due to the fully convolutional structure and the limited receptive field of the convolution, spatial features have a strong inductive bias to encode spatial neighborhood information in the original input image. Due to the presence of the fully connected layer in the global feature branch, global features do not carry information about the spatial scale of the original input image.

[0040] Two decoders G T and G F The network structures are completely identical, but they do not share network parameters. The network structure receives a spatial feature and a global feature, fuses them, and decodes them into an image. See also Figure 5 As shown, the decoder's backbone network comprises four resolution-preserving residual modules and four upsampled residual modules. The decoder uses spatial features as input to the backbone network and simultaneously injects global representations into the backbone network through a modulation-demodulation layer. Finally, a convolutional layer adjusts the number of channels to generate the final image. Unlike the decoder in StyleGAN2, which uses only one skip connection, this decoder employs a skip connection approach more akin to the discriminator in StyleGAN2. To achieve independent region editing, the decoder does not use any normalization methods, such as instance normalization or batch normalization.

[0041] Two domain discriminators D x and D y The network structure is consistent, but weights are not shared. See [link / reference]. Figure 6 As shown, the overall structure of the domain discriminator is a reverse mirror image of the decoder, that is, the upsampling module in the decoder is replaced with a downsampling module, the input and output positions are reversed, and the modulation-demodulation layer is removed.

[0042] Image Patch Discriminator D patchIts function is to determine whether an image patch belongs to a set of reference image patches, which is a set of image patches randomly cropped from a reference image. See also Figure 7 As shown, each image patch extracts its own features through 5 downsampled residual blocks, 1 resolution-preserving residual block, and 1 convolutional layer. The feature set extracted from the reference image patch set is averaged to obtain the average reference image patch feature, which is then concatenated with the input image patch feature along the channel dimension. Finally, the output result is obtained through 3 fully connected layers (linear layers).

[0043] Step S220: Set the loss function and train the general artistic font generation model in an unsupervised manner by independently optimizing the texturing, detexturing, and font transfer tasks or by joint optimization.

[0044] Still combined Figure 3 As shown, in the forward propagation process, there are three input images: a font image, a content image, and a texture image. The final output image integrates these three aspects: font, content, and texture. The entire forward propagation process is divided into three stages: detexturization, font transfer, and texturization. First, in the detexturization stage, the input image is the font image, which is processed by the encoder E and decoder G. T The texture information of the font image is then removed to obtain the corresponding mask image, which facilitates subsequent font information extraction. Next comes the font transfer stage, where the input images are the content image and the mask image obtained in the detexturization stage (the previous stage), which are then processed by encoder E and decoder G. F The resulting image, after font transfer of the content image, retains both the font information of the font image and the content information of the content image. Finally, the texturing stage takes place, with the input images being the image obtained in the font transfer stage (the previous stage) and the texture image, processed by encoder E and decoder G. T After completing a rendering and coloring process, the final output image of the entire forward propagation process is obtained.

[0045] In reverse optimization, the optimization is performed in a loop in the order of texturing, detexturing and font transfer, and in each optimization sub-stage, the optimization method of generative adversarial network is followed, that is, the generator and discriminator are trained alternately.

[0046] I) Loss function in the texturing stage

[0047] The goal of texturing is to, given a font image x F and a texture image y T The resulting textured image O is obtained. T , making O T With font image x FMaintain font consistency with texture image y T Maintain texture consistency. Specifically, x F and y T After passing through encoder E, their respective representations are obtained. and Decoder G T take over and As input, the output O is obtained. T ,Right now Unlike existing artistic font style transfer algorithms, this method employs unsupervised optimization to effectively utilize all artistic font images. The key to optimization lies in effectively maintaining consistency in font structure and texture style. The loss function in the texturing stage is divided into a reconstruction loss function and a consistency loss function.

[0048] 1) Reconstruct the loss function

[0049] To optimize the combination of spatial and style representations, a reconstruction loss function is used to optimize the model. Specifically, for the font image x... F and texture image y T Reconstruct them separately to obtain and Due to decoder G T It was designed for texture transfer, and to avoid confusion with font styles, it was used in the reconstruction of x. F The time used is Instead in Is with Constants of the same size but all with values ​​of 1. The l1 function is used as the reconstruction loss function. Since the l1 function tends to reconstruct low-frequency information in the image, this can lead to image blurring. Therefore, a domain discriminator D is further introduced. x and D y , respectively used for recovery and High-frequency information in.

[0050] In summary, in one embodiment, the reconstruction loss function is set as follows:

[0051]

[0052] 2) Consistency Loss Function

[0053] The reconstruction loss function jointly optimizes spatial and global representations, but it doesn't constrain the information encoded by each. Therefore, a consistency loss function is further introduced to ensure that the spatial representation focuses on extracting spatial information from the image, while the global representation focuses on extracting global style information. Specifically, the NCE (Noise-Constrastive Estimation Loss) loss function is used to constrain O. T With x F Consistency in glyph structure between characters is achieved using an image block discriminator D. patch To constrain O T With y T Texture style consistency between them, and domain discriminator D y To constrain O T It belongs to the texture image domain.

[0054] In summary, in one embodiment, the consistency loss function is set as follows:

[0055]

[0056] Therefore, the total loss function in the texturing stage is:

[0057]

[0058] II) Loss function in the detexturization stage

[0059] The goal of detexturization is to detexturize the font image y F After removing texture effects, we obtain its corresponding x. F To integrate texturing and detexturing into the same model without conflict, we define x. F The texture is a font-type texture, and it uses the aforementioned representation vector with all values ​​equal to 1. This is used to encode the texture. The final detextured image is... From this perspective, detexturization can be seen as a special type of texturization, that is, detexturing the font image y F Textured texture is then converted to a font-type texture. Therefore, detexturization and textured texture optimization share many similarities. However, a key difference is that the target texture for textured texture can be diverse, determined by the texture reference image, while the target texture for detexturization is only one type: a font-type texture. The training data used in the detexturization stage is the same as that used in the textured stage, i.e., the font image x. F and texture image x T Loss functions are also divided into reconstruction loss functions and consistency loss functions.

[0060] 1) Reconstruct the loss function

[0061] The reconstruction loss function is exactly the same as that used during the texturing stage, and is expressed as:

[0062]

[0063] 2) Consistency Loss Function

[0064] Similar to the texturing stage, the NCE loss function is used to maintain glyph structure consistency; specifically, O0... DeT With y F The consistency of the glyph structure between characters. Furthermore, since the target texture in the detexturization stage is only a font-type texture, there is no need to use an image patch discriminator; only a domain discriminator D is required. x That's all.

[0065] In summary, the consistency loss function is:

[0066]

[0067] Therefore, the total loss function for the detexturization stage is:

[0068]

[0069] The role of the contrastive learning loss function is to preserve glyph structure information.

[0070] In summary, the provided general-purpose artistic font generation model is trained in an unsupervised manner. See also Figure 8 As shown, artistic font texture information is determined in an unsupervised manner, achieving unsupervised learning of texture information through an image patch discriminator loss function; artistic font structure information is determined in an unsupervised manner, achieving unsupervised learning of texture information through a contrastive learning loss function. The set image patch discriminator loss function can preserve texture information.

[0071] III) Loss function in the font migration stage

[0072] Font transfer is considered a special type of style transfer, where style refers to font style. Font style representations are extracted using encoder E (the same encoder used in texturing and detexturing). Although the font style representations and the texture style representations extracted during the texturing stage are in the same representation space, they are subsequently fed into different generators to ensure they perform different functions.

[0073] In the font transfer stage, the training image used is the font image x. F and content image x C (For example, the font is Microsoft YaHei Bold). Font image x F and content image x C After inputting into encoder E, the following results are obtained: and Based on these representations, decoder G T Generate two reconstructed images and and font migration result images

[0074] 1) Reconstruct the loss function

[0075] Similar to texturing and detexturing, the reconstruction loss function is constructed using the L1 loss function and the corresponding domain discriminator:

[0076]

[0077] 2) Consistency Loss Function

[0078] Font transfer and texturing tasks differ in two key ways. Firstly, from the perspective of glyph structure information, texturing aims to preserve the glyph structure of the input image in the final image, while font transfer aims to geometrically deform the output image's glyph structure to conform to the characteristics of a specific font type. Secondly, font information is carried on the font skeleton, not local image patches; therefore, image patch discriminators should not be able to distinguish font information. Based on these two points, it can be concluded that in the font transfer result image O... F Applying the NCE loss function is harmful, and applying the image patch discriminator loss function is unnecessary. In the experimental section, several ablation experiments were designed to demonstrate these two points. Therefore, only for O... F The domain discriminator function constitutes the consistency loss function, expressed as:

[0079]

[0080] Therefore, the total loss function in the font migration stage is:

[0081]

[0082] IV) Overall Loss Function of Joint Optimization

[0083] The provided model offers a flexible solution for the artistic font generation task. On one hand, independent models can be trained for the three tasks (texturization, detexturization, and font transfer). On the other hand, a unified model can be obtained by jointly optimizing the three tasks; for example, the overall loss function for joint optimization can be set as follows:

[0084]

[0085] The model obtained after joint optimization of the three tasks can achieve fine adjustment of the artistic font generation process.

[0086] In summary, this invention removes the texture information of the font reference image during the detexturing stage, then extracts its font information during the font transfer stage and applies it to the content image to obtain the font-transferred result. Finally, in the texturing stage, the font-transferred result image is rendered with a specified texture. Furthermore, considering the difficulty of collecting paired datasets, the provided model can be trained under unsupervised conditions, allowing for the effective utilization of unpaired data. In each training stage of texturing, font transfer, and texturing, the loss function consists of a reconstruction loss function and a consistency loss function. Qualitative and quantitative experiments show that this model can achieve general generation of artistic fonts and can be used to develop a user-interactive interface for convenient visualization.

[0087] Step S230: Use the trained general artistic word generation model to achieve fine-tuning in the artistic word generation task.

[0088] This fine-tuning process is also known as the forward propagation process of the model (see...). Figure 3 (As shown). First, the font reference image y F After the detexturization stage, its corresponding glyph image O is obtained. DeT Then, O DeT Reference image x with content C Together, they serve as inputs to the font transfer stage to obtain the font transfer result image O. F Finally, the texturing stage uses the texture reference image x T Rendering textures onto O F The final textured image O is obtained. T The above steps enable fine-tuning in the artistic font generation task. For application details, please refer to [link / reference]. Figure 9 As shown.

[0089] The model training process involved in this invention can be performed offline on a server or in the cloud. The trained model can then be embedded into an electronic device to achieve real-time artistic word generation. This electronic device can be a terminal device or a server. Terminal devices include any device such as mobile phones, tablets, personal digital assistants (PDAs), point-of-sale (POS) terminals, in-vehicle computers, and smart wearable devices (smartwatches, virtual reality glasses, virtual reality headsets, etc.). Servers include, but are not limited to, application servers or web servers, and can be independent servers, cluster servers, or cloud servers.

[0090] The general technical character generation method provided by this invention has a wide range of applications, including but not limited to the following aspects:

[0091] 1) General generation of artistic fonts: In practical applications, people often want to adjust only one aspect of the artistic font while keeping other aspects unchanged. This allows users to precisely adjust the artistic font image according to their own ideas, thereby generating the artistic font image they desire.

[0092] 2) It is a crucial guarantee for scene-based artistic text rewriting tasks: Some users, after discovering their favorite artistic text in images from open-collection scenes such as posters, advertisements, and road signs, wish to replace the text while preserving its original texture. The model of this invention can effectively address this user need. Experiments show that it can achieve relatively good texture transfer for artistic text collected and cropped in open-collection scenes. Obtaining the texture-transferred individual artistic text is an important prerequisite technology for scene-based artistic text rewriting.

[0093] 3) Artistic Font Texture Blending: For multiple artistic font texture styles, blending them can generate highly artistic new textures, which can greatly enrich the artistic font texture library. Manually designing an artistic font texture is a time-consuming and labor-intensive task. The artistic font texture blending technology of this invention can generate a large number of other textures from a limited number of artistic font textures.

[0094] To further verify the effectiveness of the proposed model, a series of ablation experiments were conducted on the largest currently available artistic text dataset (Shuai Yang, Wenjing Wang, and Jiaying Liu. Te141k: artistic text benchmark for text effect transfer. IEEE Transactions on Pattern Analysis and Machine Intelligence, 43(10):3709–3723, 2020). For the classic artistic text texture transfer task, the model of this invention achieved the best performance under unsupervised settings. In terms of fine adjustment of artistic text, it can produce high-quality artistic text images. In addition, it performs well in artistic text generation tasks in open scenes, indicating that the model has a certain generalization ability. Experiments show that on the classic artistic text texture transfer task, the model of this invention achieved the best performance under unsupervised settings. It also has good texture transfer effects for artistic text images collected in open scenes.

[0095] In summary, compared with the prior art, the technical effects of the present invention are mainly reflected in the following aspects:

[0096] 1) These three tasks are integrated into a single model in a non-conflicting manner, and this model includes many details. For example, to integrate texturing and detexturing, a vector with all values ​​of 1 is designed as a special vector to represent the texture of the glyph image. The network structures of the encoder, decoder, and discriminator are changed. Hyperparameters related to training optimization are also changed.

[0097] 2) Although the training dataset is an existing open-source dataset, there is no precedent for existing models to integrate texturing, detexturing, and font transfer in a decoupled and adjustable manner. Therefore, existing models cannot achieve the effect achieved by the model of this invention.

[0098] 3) Existing methods generally require real images as supervised signals and the design of reconstruction loss functions for optimization training. This invention distinguishes artistic letter texture information in an unsupervised manner, achieving unsupervised learning of texture information through an image patch discriminator loss function.

[0099] 4) Existing methods generally require supervised training using real images and the design of reconstruction loss functions for optimization. This invention identifies the structural information of artistic characters in an unsupervised manner. Unsupervised learning of character structure information is achieved through a contrastive learning loss function.

[0100] 5) Existing methods generally cannot achieve universality, meaning that texture transfer and font transfer cannot be achieved simultaneously under decoupled conditions. The model of this invention takes into account the three most important influencing factors of artistic fonts, namely content, font, and texture, and achieves decoupled adjustment of these three factors, thus having strong universality.

[0101] 6) This invention can be applied to texture transfer of artistic text in open-set scenarios. Experiments have shown that the model of this invention has strong generalization ability and can achieve good texture transfer for artistic text in open-set scenarios.

[0102] It should be noted that, without departing from the spirit and scope of this invention, those skilled in the art can make appropriate changes or modifications to the above embodiments. For example, the input image size of the encoder, the number of convolutional layers in each network structure, and the number of upsampled and downsampled residual blocks can be set according to actual needs.

[0103] This invention can be a system, method, and / or computer program product. A computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for causing a processor to implement various aspects of the invention.

[0104] Computer-readable storage media can be tangible devices capable of holding and storing instructions for use by an instruction execution device. Computer-readable storage media can be, for example, but not limited to, electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or recessed protrusions storing instructions thereon, and any suitable combination thereof. The computer-readable storage media used herein are not to be construed as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses through fiber optic cables), or electrical signals transmitted through wires.

[0105] The computer-readable program instructions described herein can be downloaded from computer-readable storage media to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to the computer-readable storage media in the respective computing / processing device.

[0106] The computer program instructions used to perform the operations of this invention may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, Python, etc., and conventional procedural programming languages ​​such as "C" or similar languages. The computer-readable program instructions may be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuitry, such as programmable logic circuitry, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), is personalized by utilizing state information from the computer-readable program instructions. This electronic circuitry can execute the computer-readable program instructions to implement various aspects of the invention.

[0107] Various aspects of the present invention are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0108] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processor of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner; thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0109] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0110] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions. It will be known to those skilled in the art that implementation in hardware, implementation in software, and implementation using a combination of software and hardware are equivalent.

[0111] The various embodiments of the present invention have been described above. These descriptions are exemplary and not exhaustive, and are not limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or technical improvements to the embodiments in the market, or to enable others skilled in the art to understand the embodiments disclosed herein. The scope of the invention is defined by the appended claims.

Claims

1. An unsupervised, general method for generating artistic fonts, comprising the following steps: Construct a general artistic font generation model, including an encoder, a first decoder, and a second decoder; The optimization of the general artistic font generation model includes a detexturization stage, a font transfer stage, and a texturization stage, wherein: In the detexturization stage, the input image is a font reference image. After passing through the encoder and the second decoder, the texture information of the font reference image is removed to obtain the corresponding mask image. In the font migration stage, the input image is the content image and the mask image obtained in the detexturization stage. After passing through the encoder and the first decoder, the image after font migration of the content image is obtained. In the texturing stage, the input image is the image obtained in the font migration stage and the texture reference image. After passing through the encoder and the second decoder, the textured result image is obtained. The total loss function for training the general artistic character generation model is set as follows: in, It is the reconstruction loss function in the texturing stage. It is the consistency loss function in the texturing stage. The reconstruction loss function in the detexturization stage is: It is the consistency loss function in the detexturization stage. It is the reconstruction loss function in the font migration stage. It is the consistency loss function during the font migration stage; Among them, the reconstruction loss function in the texturing stage for: Consistency loss function in the texturing stage for: Reconstruction loss function in the detexturization stage for: Consistency loss function in the detexturization stage for: Reconstruction loss function in the font migration stage for: Consistency loss function during font migration for: in, The encoder is represented by G, and the first decoder is represented by G. F The second decoder is represented as G. T D patch It is an image patch discriminator, D x and D y It is a domain discriminator; x F It is a font reference image, y T It is a texture reference image, O T This is the textured result image, where NCE represents the noise contrast estimation loss. and For the font reference image x F and texture reference image y T The reconstructed image, x F and y T After encoder Their respective representations were then obtained. and Second decoder G T take over and As input, the output O is obtained. T , x F The texture is a font-type texture, and it uses a representation vector with all values ​​of 1. To encode the texture, the final detextured image is Content Image x C Input to encoder Two representations were subsequently obtained. Second decoder G T Generate reconstructed image The font migration result image is O F .

2. The method according to claim 1, characterized in that, In optimizing the general artistic font generation model, an image block discriminator loss function is used to learn texture information in an unsupervised manner, and a contrastive learning loss function is used to learn glyph structure information in an unsupervised manner. The image block discriminator is used to determine whether an image block belongs to the set of reference image blocks. The contrastive learning loss function is used to reflect the loss between the font reference image and the textured result image. The image block discriminator loss function is used to reflect the loss between the texture reference image and the textured result image.

3. The method according to claim 1, characterized in that, The encoder includes a backbone network, a spatial coding branch, and a global coding branch. The backbone network is used to extract a representation map from the input image. The spatial coding branch is used to obtain spatial features based on the representation map. The global coding branch obtains global features based on the representation map. The spatial features are feature maps that include spatial dimensions, and the global features are feature maps that do not include spatial dimensions.

4. The method according to claim 3, characterized in that, The first decoder and the second decoder have the same structure, each including a backbone network, a modulation and demodulation layer and a convolutional layer. The backbone network takes the spatial features from the decoder as input, and the corresponding global features output by the decoder are injected into the backbone network through the modulation and demodulation layer, and then the number of channels is adjusted through a convolutional layer.

5. The method according to claim 3, characterized in that, The encoder's backbone network contains four downsampled residual blocks, the spatial coding branch contains two convolutional layers, and the global coding branch contains two convolutional layers, an average pooling layer, and a fully connected layer.

6. The method according to claim 2, characterized in that, The image block discriminator takes image blocks as input and sequentially includes a first convolutional layer, multiple downsampled residual blocks, a resolution-preserving residual block, a second convolutional layer, a channel connection layer, and multiple fully connected layers.

7. A computer-readable storage medium having a computer program stored thereon, wherein, When executed by a processor, the computer program implements the steps of the method according to any one of claims 1 to 6.

8. A computer device comprising a memory and a processor, wherein a computer program capable of running on the processor is stored in the memory, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 6.

Images