Double-branch wordart typesetting method based on main body and environment

Abstract

The invention discloses a double-branch wordart typesetting method based on a main body and an environment, and relates to the field of image generation in three-level vision. The problem that an existing wordart typesetting method is poor in flexibility and controllability is solved. The input character to be edited is divided into a main body part and an environment part, and the balance between artistry (namely, flexibility) and readability (controllability) in the typesetting of the wordart is realized through controllable geometric transformation of fonts by a double-branch method of dividing the main body part and the environment part. The method is mainly used for generating the wordart image.

Description

Technical Field

[0001] The present invention relates to the field of image generation in three-level vision. Background Art

[0002] The task of the art word generation work is defined as: given an input character or word, visually express this character in a certain artistic form. The artistic form is mainly divided into two types. One is texture stylization, which is characterized by rendering the texture and style of the character to be edited. For example, attaching a flame texture to the character "fire". The other is artistic typesetting, which is characterized by deforming the structure of the character to a certain extent to present a specific semantics. For example, changing the character "person" into a walking person. In real life, in order to consider both readability and artistry, it is a very time-consuming task to manually design an art word.

[0003] In recent years, with the further development of image generation technology, the field of art word generation has also made breakthrough progress. Scholars use vector graph transformation or "depth of field to image" technology to deform the font structure and use the font structure as a control condition to guide the generator to generate relevant art words. However, there are two problems with the art words generated using existing algorithms. On the one hand, directly deforming the overall structure of the character will result in a rigid and uncontrollable change, which greatly affects the readability of the art word. On the other hand, existing art word generation models cannot introduce multiple independent concepts, which also cannot flexibly and completely express the semantics that the user wants to express to a certain extent. That is, there are defects of poor flexibility and controllability. The above problems need to be solved urgently. Summary of the Invention

[0004] The purpose of the present invention is to solve the problem of poor flexibility and controllability existing in the existing art word typesetting method; the present invention provides a dual-branch art word typesetting method based on the subject and the environment.

[0005] The dual-branch art word typesetting method based on the subject and the environment includes the following steps:

[0006] S1. Obtain the subject description P sub and the environment description P surr ;

[0007] According to the character to be edited and the given font file, use the font generation technology to obtain the character image I of the character to be edited;

[0008] S2. According to the subject description P sub perform region decomposition on the character image to obtain the subject image I sub the environment image I surr and the mask image M;

[0009] S3, using semantic typesetting technology to describe P sub For the main image I sub Edit and get a new subject image

[0010] New main image In the main description P sub Under the guidance of style rendering, and environmental image I surr In the environmental description P surr After achieving style rendering under the guidance of and environment image I surr The fusion of the word art image of the word to be edited is obtained to realize the word art typesetting.

[0011] Preferably, according to the subject description P sub Perform regional decomposition on the character image to obtain the main image I sub 、Environmental image I surr The implementation of the mask image M includes:

[0012] S21, detect the character image through the open domain based object detection technology, and describe the subject P sub For the matched parts, multiple bounding boxes including confidence scores are obtained;

[0013] S22, calculating the ratio of the area of ​​each bounding box to the overall area of ​​the character image, screening out bounding boxes whose ratios are within a preset threshold range, and taking the bounding box with the highest confidence score from the screened out bounding boxes as the optimal bounding box;

[0014] S23, extracting the characters and character images in the area where the optimal bounding box is located, and the overall background, to obtain the main image I sub At the same time, after extracting the characters and character images outside the optimal bounding box, the overall background is obtained to obtain the environment image I surr At the same time, the background color of the area where the optimal bounding box is located is extracted and the color is inverted, and then combined with the background outside the optimal bounding box to obtain the mask image M.

[0015] Preferably, the preset threshold ranges from 0.3 to 0.6.

[0016] Preferably, in S3, the new subject image In the main description P sub Under the guidance of style rendering, and environmental image I surr In the environmental description P surr After achieving style rendering under the guidance of and environment image I surrThe implementation method of obtaining the artistic word image of the word to be edited includes:

[0017] S31. Subject Description P sub and environment description P surr Encoding is performed to obtain the subject prompt tensor and the environment prompt tensor, and the tensors are sent to the subject control network and the environment control network respectively;

[0018] The noise tensor ∈ is randomly sampled from a Gaussian distribution T , and the noise tensor ∈ T Send to the main control network and the environmental control network at the same time;

[0019] The subject control network is also used to receive new subject images and environment image I surr ;

[0020] Quantize the mask image M to obtain a mask tensor;

[0021] S32, setting the total number of denoising times to T;

[0022] The subject control network is used to transform the subject cue tensor to the new subject image The guided process is used as a control condition for the noise tensor ∈ T Perform a denoising to obtain the subject prediction noise tensor

[0023] The environment control network is used to transform the environment prompt tensor to the environment image I surr The guided process is used as a control condition for the noise tensor ∈ T Perform a denoising to obtain the environment prediction noise tensor

[0024] S33, multiply the mask tensor by the set hyperparameter γ as the weight matrix to guide the subject to predict the noise tensor and the environmental prediction noise tensor Fusion is performed to obtain the total predicted noise tensor ∈ T-1 ;

[0025] It is determined whether T is equal to 1. If the result is no, T=T-1, and step S32 is executed. If the result is yes, the total predicted noise tensor ∈ is decoded by the decoder to obtain the artistic word image of the word to be edited.

[0026] Preferably,

[0027] Preferably, both the main control network and the environment control network are implemented using a diffusion model.

[0028] Preferably, the main description P of the word to be edited is obtained according to the semantics of the word to be edited.sub and environment description P surr , implemented using a large language model.

[0029] Preferably, the font generation technology is implemented using a font generation module.

[0030] A computer-readable storage device stores a computer program, which, when executed, implements a dual-branch artistic word typesetting method based on subject and environment.

[0031] A dual-branch artistic word typesetting device based on subject and environment comprises a storage device, a processor and a computer program stored in the storage device and executable on the processor. The processor executes the computer program to implement a dual-branch artistic word typesetting method based on subject and environment.

[0032] The beneficial effects brought by the present invention are:

[0033] The present invention proposes the concepts of "subject" and "environment". The "subject" expresses the core meaning through deformation, and the "environment" sets off the subject and enriches the content.

[0034] The present invention only performs structural deformation on a certain part (main body) of the input character that is most similar to the core semantics, which greatly reduces the damage to the overall frame of the character during deformation, thereby improving the overall readability of the character without losing the artistry;

[0035] The present invention decouples the generation process of the subject and the environment, can expand a single subject into multiple subjects, and simultaneously customize different concepts for different subjects and environments, thereby achieving a more user-friendly customized art word design.

[0036] Secondly, this dual-branch approach based on the concepts of "subject" and "environment" has the potential to expand a single subject into multiple subjects, thereby achieving a more user-friendly user experience and richer artistic font design. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 It is a schematic diagram of the principle of the dual-branch artistic word typesetting method based on subject and environment described in the present invention;

[0038] Figure 2 It is a schematic diagram of the principle of the process of regional decomposition and subject and environment combination;

[0039] Figure 3 The figure is a schematic diagram of the principle of generating an artistic word image of a word to be edited through combined control. DETAILED DESCRIPTION

[0040] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts fall within the scope of protection of the present invention.

[0041] It should be noted that, without conflict, the embodiments in the present invention and the features in the embodiments can be combined with each other.

[0042] The present invention introduces the concepts of "subject" and "environment", and adopts controllable structural deformation to ensure artistic expression while ensuring good readability. Compared with the limitation of the prior art that can only generate artistic words containing a single concept, the present invention provides a dual-branch artistic word layout method based on the subject and the environment, which can realize the combined generation of multiple subject concepts, thereby supporting more complex artistic word designs, such as signatures, slogans, advertising words, etc.

[0043] Specific implementation manner 1. Refer to Figure 1 and Figure 2 To illustrate this implementation manner, for the dual-branch artistic word layout method based on the subject and the environment described in this implementation manner, in the preliminary preparation stage, prepare the word to be edited, the core semantics of the word to be edited, and the given font file;

[0044] The layout method includes the following steps:

[0045] S1. Obtain the subject description P sub and the environment description P surr of the word to be edited according to the semantics of the word to be edited; specifically, the large language model imagines the concepts of the subject and the environment according to the semantics of the word to be edited, and then designs the subject description P sub and the environment description P surr , and the large language model can be Chat Generative Pretrained Transformer (ChatGPT). For example, for the character "蔷", the subject description can be "a blooming red rose", and the environment description can be "a slender brown flower branch, green leaves, full of a romantic atmosphere", refer to Figure 2 ;

[0046] According to the word to be edited and the given font file, the character image I of the word to be edited is obtained by using the font generation technology; specifically, the font generation technology is implemented by using the font generation module, and the character image of the word to be edited is obtained by using the font generation module. The specific process is as follows: first, the stroke outline of the word to be edited in the font is extracted by using the FreeType font library, and converted into the form of a cubic Bezier curve. Subsequently, the cubic Bezier curve is converted into a differentiable vector graphics by using the Differentiable Vector Graphics (DiffVG) technology, and saved as the character image I;

[0047] S2. According to the subject description P sub Perform regional decomposition on the character image to obtain the main image I sub 、Environmental image I surr and mask image M;

[0048] S3, using semantic typesetting technology to describe P sub For the main image I sub Edit and get a new subject image Specifically, the subject image I sub and the subject description P sub Input to the "Depth-to-image" model in the Latent Diffusion Model (LDM) to adjust the subject image I sub The geometric shape of the new subject image is obtained

[0049] New main image In the main description P sub Under the guidance of style rendering, and environmental image I surr In the environmental description P surr After achieving style rendering under the guidance of and environment image I surr The fusion of the word art image of the word to be edited is obtained to realize the word art typesetting.

[0050] Principle analysis: The input character to be edited is divided into two parts: "main body" and "environment". The "main body" part vividly conveys the core concept through flexible deformation, and the "environment" part is used to enrich the background information, highlight the "main body", and ensure the stability of the character shape. This dual-branch method of dividing the "main body" and "environment" will achieve a balance between artistry (i.e. flexibility) and readability (controllability) in artistic typesetting through controllable transformation of glyph geometry. Secondly, this dual-branch method based on the concepts of "main body" and "environment" has the potential to expand a single subject into multiple subjects, thereby achieving a more friendly user experience and richer artistic character design.

[0051] For further information, see Figure 2 , Regional decomposition: According to the subject description P sub Perform regional decomposition on the character image to obtain the main image I sub 、Environmental image I surr The implementation of the mask image M includes:

[0052] S21, detect the character image through the open domain based object detection technology, and describe the subject P sub For the matched parts, multiple bounding boxes including confidence scores are obtained;

[0053] S22, calculating the ratio of the area of ​​each bounding box to the overall area of ​​the character image, filtering out bounding boxes whose ratios are within a preset threshold range, and taking the bounding box with the highest confidence score from the filtered bounding boxes as the optimal bounding box; in specific applications, the preset threshold range is from 0.3 to 0.6, filtering out bounding boxes whose area ratios are less than 0.3 or greater than 0.6;

[0054] S23, extracting the characters and character images in the area where the optimal bounding box is located, and the overall background, to obtain the main image I sub At the same time, after extracting the characters and character images outside the optimal bounding box, the overall background is obtained to obtain the environment image I surr At the same time, the background color of the area where the optimal bounding box is located is extracted and the color is inverted, and then combined with the background outside the optimal bounding box to obtain the mask image M, see Figure 2 .

[0055] For further information, see Figure 3 , S3, new subject image In the main description P sub Under the guidance of style rendering, and environmental image I surr In the environmental description P surr After achieving style rendering under the guidance of and environment image I surrThe implementation method of obtaining the artistic word image of the word to be edited includes:

[0056] S31. Subject Description P sub and environment description P surr Encoding is performed to obtain the subject cue tensor and the environment cue tensor, and they are sent to the subject control network and the environment control network respectively. Specifically, for the style rendering of the subject image, the semantic segmentation version of the control network is selected as the subject control network (ControlNet sub ), for the style rendering of the environment image, the graffiti version of the control network is selected as the environment control network (ControlNet surr ). Furthermore, both the main control network and the environment control network are implemented using diffusion models.

[0057] The noise tensor ∈ is randomly sampled from a Gaussian distribution T , and the noise tensor ∈ T Send to the main control network and the environmental control network at the same time;

[0058] The subject control network is also used to receive new subject images and environment image I surr , and the new subject image and environment image I surr Use them as the main prompt image and the environment prompt image respectively, and cut and scale them to the same size;

[0059] Quantize the mask image M to obtain a mask tensor;

[0060] S32, setting the total number of denoising times to T;

[0061] The subject control network is used to transform the subject cue tensor to the new subject image The guided process is used as a control condition for the noise tensor ∈ T Perform a denoising to obtain the subject prediction noise tensor

[0062] The environment control network is used to transform the environment prompt tensor to the environment image I surr The guided process is used as a control condition for the noise tensor ∈ T Perform a denoising to obtain the environment prediction noise tensor

[0063] S33, multiply the mask tensor by the set hyperparameter γ as the weight matrix to guide the subject to predict the noise tensor and the environmental prediction noise tensor Fusion is performed to obtain the total predicted noise tensor ∈ T-1 ;

[0064] Determine whether T is equal to 1. If the result is no, T = T-1, and execute step S32. If the result is yes, then the total predicted noise tensor ∈ 0 After being decoded by the decoder, the artistic word image of the word to be edited is obtained.

[0065] In the above preferred implementation, open domain target detection can be used to reasonably obtain the subject image so that the subsequent deformation is more natural; the semantic version technology based on "depth of field to image" can optimize the subject shape to make its expression semantics more appropriate; the combined control generation technology can be used to render the subject and environment images while achieving a good fusion of the two.

[0066] Furthermore, γ is a hyperparameter used to characterize the weight matrix and is used to adjust the prominence of the subject in the combined graph.

[0067] In this embodiment, the designed T-1 The expression has the function of learning the information of both the subject and the environment at the same time to eliminate the sense of separation in the combination generation process.

[0068] Specific implementation method 2: A computer-readable storage device described in this implementation method stores a computer program, and is characterized in that when the computer program is executed, the dual-branch artistic word typesetting method based on the subject and the environment is implemented.

[0069] Specific implementation method three: A dual-branch artistic word typesetting device based on a subject and an environment described in this implementation method includes a storage device, a processor, and a computer program stored in the storage device and executable on the processor. The processor executes the computer program to implement the dual-branch artistic word typesetting method based on a subject and an environment as described.

[0070] Although the present invention is described herein with reference to specific embodiments, it should be understood that these embodiments are merely examples of the principles and applications of the present invention. It should therefore be understood that many modifications may be made to the exemplary embodiments and that other arrangements may be devised without departing from the spirit and scope of the present invention as defined by the appended claims. It should be understood that the various dependent claims and features described herein may be combined in a manner different from that described in the original claims. It should also be understood that features described in conjunction with individual embodiments may be used in other described embodiments.

Claims

1. A dual-branch artistic word typesetting method based on subject and environment, characterized in that: The method comprises the following steps: S1. Obtain the main description P of the word to be edited according to the semantics of the word to be edited sub and environment description P surr ; According to the word to be edited and a given font file, a character image I of the word to be edited is obtained by using a font generation technology; S2. According to the subject description P sub Perform regional decomposition on the character image I to obtain the main image I sub 、Environmental image I surr and mask image M; S3, using semantic typesetting technology to describe P sub For the main image I sub Edit and get a new subject image New main image In the main description P sub Under the guidance of style rendering, and environmental image I surr In the environmental description P surr After achieving style rendering under the guidance of and environment image I surr The fusion of the word art image of the word to be edited is obtained to realize the word art typesetting.

2. The dual-branch artistic word typesetting method based on subject and environment according to claim 1 is characterized in that: According to the subject description P sub Perform regional decomposition on the character image I to obtain the main image I sub 、Environmental image I surr The implementation of the mask image M includes: S21, detect the character image I and the subject description P by using the open domain based target detection technology sub For the matched parts, multiple bounding boxes including confidence scores are obtained; S22, calculating the ratio of the area of ​​each bounding box to the overall area of ​​the character image I, screening out bounding boxes whose ratios are within a preset threshold range, and taking the bounding box with the highest confidence score from the screened out bounding boxes as the optimal bounding box; S23, extracting the characters in the area where the optimal bounding box is located and the overall background of the character image I to obtain the main image I sub At the same time, after extracting the characters outside the optimal bounding box and the overall background of the character image I, the environment image I is obtained surr At the same time, the background color of the area where the optimal bounding box is located is extracted and the color is inverted, and then combined with the background outside the optimal bounding box to obtain the mask image M.

3. The dual-branch artistic word typesetting method based on subject and environment according to claim 2 is characterized in that: The preset threshold ranges from 0.3 to 0.

6.

4. The dual-branch artistic word typesetting method based on subject and environment according to claim 1, characterized in that: S3, new subject image In the main description p sub Under the guidance of style rendering, and environmental image I surr In the environmental description P surr After achieving style rendering under the guidance of and environment image I surr The implementation method of obtaining the artistic word image of the word to be edited includes: S31. Subject Description P sub and environment description P surr Encoding is performed to obtain the subject prompt tensor and the environment prompt tensor, and the tensors are sent to the subject control network and the environment control network respectively; The noise tensor ∈ is randomly sampled from a Gaussian distribution T , and the noise tensor ∈ T Send to the main control network and the environmental control network at the same time; The subject control network is also used to receive new subject images and environment image I surr ; Quantize the mask image M to obtain a mask tensor; S32, setting the total number of denoising times to T; The subject control network is used to transform the subject cue tensor to the new subject image The guided process is used as a control condition for the noise tensor ∈ T Perform a denoising to obtain the subject prediction noise tensor The environment control network is used to transform the environment prompt tensor to the environment image I surr The guided process is used as a control condition for the noise tensor ∈ T Perform a denoising to obtain the environment prediction noise tensor S33, multiply the mask tensor by the set hyperparameter γ as the weight matrix to guide the subject to predict the noise tensor and the environmental prediction noise tensor Fusion is performed to obtain the total predicted noise tensor ∈ T-1 ; It is determined whether T is equal to 1. If the result is no, T=T-1, and step S32 is executed. If the result is yes, the total predicted noise tensor ∈ is decoded by the decoder to obtain the artistic word image of the word to be edited.

5. The dual-branch artistic word typesetting method based on subject and environment according to claim 4 is characterized in that:

6. The dual-branch artistic word typesetting method based on subject and environment according to claim 1, characterized in that: Both the main control network and the environmental control network are implemented using the diffusion model.

7. The dual-branch artistic word typesetting method based on subject and environment according to claim 1, characterized in that: Obtain the main description P of the word to be edited according to the semantics of the word to be edited sub and environment description P surr , implemented using a large language model.

8. The dual-branch artistic word typesetting method based on subject and environment according to claim 1, characterized in that: The font generation technology is implemented using a font generation module.

9. A computer-readable storage device storing a computer program, characterized in that: When the computer program is executed, the method according to any one of claims 1 to 8 is implemented.

10. A dual-branch artistic word typesetting device based on subject and environment, comprising a storage device, a processor, and a computer program stored in the storage device and executable on the processor, characterized in that: The processor executes the computer program to implement the method according to any one of claims 1 to 8.

Images