Image editing and model training method and device, storage medium and program product

Abstract

The embodiment of the invention provides an image editing and model training method and device, a storage medium and a program product. In the embodiment of the invention, an image-text editing model is provided, image editing is carried out on a target text area of an original image from a font dimension and a style dimension, guarantee is provided for the accuracy of image editing through text coding features and character morphological features in the font dimension, and the consistency with a background is kept; in the style dimension, by injecting text style features, the original image style and additional styles are kept, and the availability rate of image editing is improved. Image editing is carried out from the two dimensions of font and style, and the image quality of the obtained image can be improved. Furthermore, in the layout dimension and model training process, local and transverse and longitudinal modification of various alignment modes is supported through a layout loss function and adaptive learning layout information, and the image quality is further improved.

Description

Technical Field

[0001] The present application relates to the field of Internet technology, and in particular to an image editing and model training method, device, storage medium and program product. Background Art

[0002] In many application scenarios, there is often a need to edit text on images. For example, in the e-commerce field, when participating in different marketing activities, merchants need to adaptively modify the text content in product images to adapt to different marketing activities.

[0003] In the prior art, professional image editing software can be used to edit the text of a product image. For example, a merchant erases "Celebrate May Day" according to the outline of the text in the product image, repairs the image background after the text is erased, and adds "Happy Children's Day" to the repaired image background to obtain a modified product image. However, the image quality obtained by this text editing method is not high, for example, the background style does not fit the text. Summary of the invention

[0004] Multiple aspects of the present application provide an image editing and model training method, device, storage medium and program product to improve the quality of images obtained by image editing.

[0005] An embodiment of the present application provides an image editing method, comprising: obtaining an original image and target text information submitted by a user, the original image comprising a target text area to be edited, the target text area being used to carry the target text information, and the target text information comprising at least one character; generating a first glyph image corresponding to the target text information based on a preset glyph library, the first glyph image reflecting the character morphological features in the target text information; extracting text style features adapted to the target text area from the original image or an associated image of the original image; calling a target graphic editing model according to the text style features, the character morphological features and the text encoding features of the target text information, and performing text editing processing on the target text area in the original image to obtain a target image; wherein the target image comprises target text information having text style features and character morphological features.

[0006] The embodiment of the present application also provides a model training method, including: obtaining multiple sample information, the sample information including: sample image, sample text information, sample text encoding features, sample character morphological features and sample text style features; calling an initial image-text editing model according to the sample text style features, sample character morphological features and sample text encoding features, performing text editing processing on the sample text area in the sample image to obtain a target image, the target image including sample text information with text style features and character morphological features; constructing a loss function of the initial image-text editing model according to the original text information included in the sample image and the sample text information included in the target image; and iteratively training the initial image-text editing model according to the loss function and the multiple sample information to obtain a target image-text editing model.

[0007] An embodiment of the present application also provides an electronic device, comprising: a memory and a processor; the memory is used to store a computer program; the processor is coupled to the memory and is used to execute the computer program to implement each step in the image editing method and model training method provided in the embodiment of the present application.

[0008] An embodiment of the present application also provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, the processor implements each step in the image editing method and model training method provided in the embodiment of the present application.

[0009] An embodiment of the present application also provides a computer program product, including a computer program / instructions. When the computer program / instructions are executed by a processor, the processor implements the steps in the image editing method and model training method provided in the embodiment of the present application.

[0010] In an embodiment of the present application, a graphic and text editing model is provided, which performs image editing on a target text area of ​​an original image from a glyph dimension and a style dimension. In the glyph dimension, text encoding features and character morphological features are used to ensure the accuracy of image editing and maintain consistency with the background; in the style dimension, text style features are injected to support the retention of the original image style and additional styles, thereby improving the availability of image editing. Performing image editing from both the glyph and style dimensions can improve the image quality of the resulting image. Furthermore, in the layout dimension, during the model training process, the layout loss function is used to adaptively learn the layout information, and various alignment methods are supported for local and horizontal and vertical modifications, thereby further improving the image quality. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0012] Figure 1 A flowchart of an image editing method provided for an exemplary embodiment of the present application;

[0013] Figure 2a A schematic diagram of an original image provided for an exemplary embodiment of the present application;

[0014] Figure 2b A schematic diagram of selecting a target text area for an original image provided by an exemplary embodiment of the present application;

[0015] Figure 2c A schematic diagram of a target image provided for an exemplary embodiment of the present application;

[0016] Figure 3 A flowchart of another image editing method provided for an exemplary embodiment of the present application;

[0017] Figure 4 A schematic diagram of a model training provided for an exemplary embodiment of the present application;

[0018] Figure 5 A schematic diagram of an attention map provided for an exemplary embodiment of the present application;

[0019] Figure 6 A schematic diagram of the structure of an image editing device provided by an exemplary embodiment of the present application;

[0020] Figure 7 A schematic structural diagram of an electronic device provided for an exemplary embodiment of the present application. DETAILED DESCRIPTION

[0021] In order to make the purpose, technical solution and advantages of the present application clearer, the technical solution of the present application will be clearly and completely described below in combination with the specific embodiments of the present application and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present application.

[0022] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0023] The various models involved in this application (including but not limited to language models or large models) are in compliance with relevant laws and standards.

[0024] In response to the above technical problems, in an embodiment of the present application, a graphic and text editing model is provided to perform image editing on the target text area of ​​the original image from the glyph dimension and the style dimension. In the glyph dimension, the accuracy of image editing is guaranteed by text encoding features and character morphological features, and consistency with the background is maintained; in the style dimension, by injecting text style features, it supports retaining the original image style and additional style, thereby improving the availability of image editing. Image editing from the two dimensions of glyph and style can improve the image quality of the resulting image. Furthermore, in the layout dimension, during the model training process, the layout loss function is used to adaptively learn the layout information, support various alignment methods, local and horizontal and vertical modifications, and further improve the image quality.

[0025] Furthermore, users need to regenerate the text information of the image in marketing activities, which may bring high labor and time costs. In the embodiment of the present application, the user can directly use this function to select the text area in the modified image and enter the target text information to be modified. The image editing method provided by the embodiment of the present application can help users quickly and accurately replace the target text information (such as slogans, prices, promotional activities) with the original text information in the text area, reducing manpower and time consumption. In addition, the generated target image has no editing traces, consistent background, accurate font and content, consistent style, and unified layout, especially in Chinese editing to achieve good results.

[0026] A solution provided by an embodiment of the present application is described in detail below in conjunction with the accompanying drawings.

[0027] Figure 1 The following is a flow chart of an image editing method provided by an exemplary embodiment of the present application. Figure 1 As shown, the method includes:

[0028] 101. Acquire an original image and target text information submitted by a user, wherein the original image includes a target text area to be edited, the target text area is used to carry the target text information, and the target text information includes at least one character;

[0029] 102. Generate a first glyph image corresponding to the target text information based on a preset glyph library, where the first glyph image reflects character morphological features in the target text information;

[0030] 103. Extracting text style features adapted to the target text area from the original image or an image associated with the original image;

[0031] 104. Based on the text style features, character morphology features and text encoding features of the target text information, call the target image-text editing model to perform text editing processing on the target text area in the original image to obtain a target image; wherein the target image includes the target text information having text style features and character morphology features.

[0032] In this embodiment, the original image is an image with image editing requirements, and the original image includes a target text area to be edited, and the target text area is used to carry target text information. For example, before the original image is edited, the target text area may carry the original text information, and by executing the image editing method on the original image, the original text information in the target text area can be replaced with the target text information. Alternatively, before the original image is edited, the target text area can be a blank area without carrying text information, and by executing the image editing method on the original image, the target text information can be added to the target text area.

[0033] For example, the original text information in the original image is replaced with the target text information. Figure 2a The phrase "heart to heart" is changed to Figure 2c In the "Tmall Good Price Festival". Figure 2b In the example, the text area where “heart to heart is connected” is the target text area.

[0034] The target text information may include at least one character, and the characters may include but are not limited to: words, numbers, letters, and symbols. It should be noted that the data of the characters included in the original text information included in the target text area may be the same as or different from the number of characters included in the target text information. For example, the original text information includes 5 characters, and the target text information may include 3 characters, 5 characters, or 7 characters, without limitation.

[0035] In this embodiment, the implementation method of obtaining the original image and target text information submitted by the user is not limited. Depending on the execution subject of the image editing method, the implementation method of obtaining the original image and target text information submitted by the user is also different. The following is an exemplary description.

[0036] Example A1, when the image editing method is implemented by the server device, the terminal device obtains the original image and target text information submitted by the user and provides them to the server device, and the server device receives the original image and target text information sent by the terminal device. The method for the terminal device to obtain the original image and target text information can be seen in Example A2, which is not described in detail here.

[0037] Example A2, when the image editing method is implemented by a terminal device, the terminal device obtains the original image and target text information submitted by the user. The terminal device provides an image editing page to the user, and the image editing page includes an image upload control and a text input control. The terminal device can obtain the original image in response to the user's trigger operation on the image upload control; and obtain the target text information in response to the input operation on the text input control.

[0038] Optionally, when the terminal device obtains the original image, character recognition is performed on the original image to identify the characters in the original image and the position information of each character in the original image. The user can select characters on the original image, and the terminal device responds to the user's selection operation on the original image and uses the area where the selected characters are located as the target text area. The selected characters can be one or more, and there is no limitation on this. Figure 3 As shown, character recognition is performed on the original image, and the target text area is determined for exemplary display.

[0039] There is no limitation on the way of character recognition for the original image. For example, optical character recognition (OCR) technology can be used to perform field recognition and character recognition on the original image to determine the global bounding box (bbox) of each field and the layout bounding box bbox of the characters included in each field. For example, "heart is connected to heart" can be considered as a field, and each of "heart", "and", "heart", "phase" and "connected" is a character. Furthermore, the area where the selected character is located is the area defined by the local bounding box of the selected character. Figure 2b As shown, the local bounding box corresponding to “hearts connected” is exemplarily displayed.

[0040] Furthermore, the local bounding boxes of the characters included in each field are corrected according to the global bounding box of each field. For example, the local bounding box is corrected according to the height, width, aspect ratio and other information of the global bounding box of the field; the local bounding box of the characters without local bounding boxes is estimated, and the maximum effective padding area of ​​each character is calculated, and the maximum effective padding area is used as the local bounding box of the character.

[0041] In this embodiment, in the above method, the area where the local boundary box corresponding to the character selected by the user is located is used as the target text area. In addition, if the user hopes to add target text information to the area in the original image that is not filled with original text information, the user can select the target text area on the original image. For example, the user selects a rectangular area through a rectangular box on the original image, and the terminal device responds to the user's selection operation and uses the rectangular area selected by the user through the rectangular box as the target text area.

[0042] In this embodiment, in terms of glyphs, considering that the characters included in the target text information may be words, numbers or letters, and words contain a variety of complex glyph strokes, the accuracy of glyph generation can be guaranteed by glyph images. Specifically, a first glyph image corresponding to the target text information is generated based on a preset glyph library, and the first glyph image reflects the character morphological features in the target text information. Among them, the glyph library includes glyphs of various characters, and the glyph of each character in the target text information can be obtained from the glyph library, and the glyph of each character is displayed on a solid color background. For example, the first glyph image can be a white background with black characters image of the target text information including a set font. Among them, the set font can include: Songti, Heiti or Kaiti, etc. As Figure 4 As shown, an exemplary display is made for the first glyph image, and the glyph image includes "GALAXY" in black characters on a white background, but is not limited thereto.

[0043] In this embodiment, in terms of font style, in order to maintain the consistency between the target text information and the text information included in the original image, text style features that are adapted to the target text area can be extracted from the original image. Alternatively, the user can extract text style features that are adapted to the target text area from the associated image of the original image according to his or her own wishes, so that the target text information is the same or similar in style to that in the associated image. Among them, the associated image of the original image is an image that assists in extracting text style features, and the associated image can be provided by the user, or the user can select an image that meets his or her wishes from multiple images provided by the terminal device as the associated image of the original image. Among them, text style features refer to the attribute information that the target text information has when it is visually presented, such as font, color, weight (thickness), underline, shadow, background color, and border.

[0044] This embodiment provides a target text-image editing model, which can perform text editing on a target text area of ​​an original image, replace the original text information in the target text area with the target text information, or add the target text information to the target text area. The text-image editing model can achieve controllable editing of the text area through an inpainting algorithm.

[0045] The implementation method of the image and text editing model is not limited. For example, the image and text editing model can be implemented as a Stable Diffusion model or a Stable Diffusion eXtended Large (SDXL) model. For example, image restoration is implemented through SDXL. Character morphological features and text encoding features of target text information can be injected into the target image and text editing model as text condition information to guide the model to learn character morphology and understand the generated text content; text style features can be injected into the target image and text editing model as style condition information to help the model understand the style information of the text.

[0046] Optionally, the image and text editing model includes: an image feature extraction network, a multi-scale feature extraction network, and an image feature conversion network.

[0047] The image feature extraction network can be any network that can extract image features, and the image feature conversion network can restore image features to images. The image feature extraction network and the image feature conversion network can be implemented as encoders and decoders, respectively. For example, encoders and decoders involved in models such as Convolutional Neural Networks (CNN), Residual Networks (ResNet), and Variational Autoencoder (VAE).

[0048] The multi-scale feature extraction network may be any network capable of realizing multi-scale feature extraction. For example, the multi-scale feature extraction network may be a downsampling-upsampling architecture based on a skip connections mechanism, such as a Unet network.

[0049] exist Figure 4 In the figure, the internal structure of the initial image and text editing model during the model training process is taken as an example for illustration, wherein the image feature extraction network is a VAE encoder, the image feature conversion network is a VAE decoder, and the multi-scale feature extraction network is a Unet model as an example, but it is not limited to this.

[0050] besides, Figure 4 The text encoding model for extracting text encoding features and the style feature extraction model for extracting text style features are included. The detailed introduction of extracting text encoding features and text style features can be found in the subsequent embodiments, which will not be repeated here.

[0051] According to the text encoding features of text style features, character form features, and target text information, call the target graphic and text editing model to perform text editing on the target text area in the original image to obtain the target image. Among them, the target image includes target text information with text style features and character form features.

[0052] Among them, the text encoding features can reflect the character content features and character order features of the target text information. For example, determine which characters are included in the target text information and the order of the characters in the target text information. For example, when the target text information is implemented as: "Mid-Autumn Festival", the text encoding features can reflect that the target text information includes the three characters "中", "秋", and "节", and the order of the three characters is: 中 -> 秋 -> 节, rather than: 秋 -> 中 -> 节.

[0053] Among them, by controlling the glyph from two aspects of character form features and text encoding features, the accuracy of glyph control can be improved, which not only supports the editing of numbers and English letters, but also has good generation effects in Chinese editing and the editing of other languages.

[0054] In the embodiment of the present application, a graphic and text editing model is provided, which performs image editing on the target text area of the original image from the glyph dimension and the style dimension. In the glyph dimension, the text encoding features and character form features ensure the accuracy of image editing and maintain consistency with the background; in the style dimension, by injecting text style features, it supports retaining the original image style and additional styles, improving the availability of image editing. Performing image editing from the two dimensions of glyph and style can improve the image quality of the obtained image. Further, in the layout dimension, during the model training process, through the layout loss function, adaptively learn layout information, support local and horizontal and vertical modifications of various alignment methods, and further improve the image quality.

[0055] In an optional embodiment, the implementation manner of extracting the text style features adapted to the target text area from the original image or the associated image of the original image is not limited. The following is an exemplary description.

[0056] Example B1, when the target text area includes the original text information, extract the text style features adapted to the target text area from the original image, and maximize the model's retention of the font style in the original image.

[0057] When the target text area includes original text information, the text style features of the original text information are extracted, and the text style features of the original text information are used as text style features adapted to the target text area. This style injection method can make the text style of the generated target text information consistent with the style of the original text information, and can still maintain the original style even without additional reference text, so that the final generated target image is harmonious and unified on the background.

[0058] Based on this, an implementation method for extracting text style features that are adapted to a target text area from an original image when the target text area includes original text information includes: extracting a local image where the original text information is located from the original image, and generating a second glyph image corresponding to the original text information based on a preset glyph library; inputting the local image and the second glyph image into a style feature extraction model for feature extraction, respectively, to obtain first feature information and second feature information; the first feature information includes text style features and text content features of the original text information, and the second feature information includes text content features of the original text information; according to the second feature information, obtaining text style features of the original text information from the first feature information as text style features that are adapted to the target text area.

[0059] Among them, the implementation method of extracting the local image where the original text information is located from the original image includes: taking the area where the entire original text information belongs to the original image as the local image, or, when the character style included in the original text information is consistent, taking the area where some characters of the original text information belong to as the local image.

[0060] The implementation method of generating the second glyph image corresponding to the original text information based on the preset glyph library can refer to the aforementioned implementation method of generating the first glyph image for the target text information, which will not be repeated here.

[0061] The style feature extraction model can be any model that extracts features from images. For example, the style feature extraction model can include but is not limited to: self-supervised visual representation learning (DINOv2). The DINOv2 model is a pre-trained feature model that can extract multi-level features of an image. These features are usually extracted in the middle layer (rather than the last layer) because the features of the middle layer retain certain spatial information and have high-level abstract capabilities.

[0062] Among them, it can be considered that the original text information in the second glyph image belongs to the standard glyph, and the second feature information extracted for the second glyph image does not carry text style features, but only includes text content features. Feature extraction is performed on the original text information, and the first feature information obtained may include text content features and text style features. According to the second feature information, the text style features of the original text information are obtained from the first feature information. As an implementation method of the text style features adapted to the target text area, a subtraction operation can be performed on the first feature information and the second feature information, and the text style features of the original text information are obtained by subtracting the second feature information from the first feature information.

[0063] Example B2, when the target text region does not include original text information, text style features adapted to the target text region are extracted from the original image.

[0064] When the target text area does not include the original text information, and other areas of the original image include the original text information, a partial image where the original text information is located is extracted from the other areas of the original image, and based on the partial image, the text style features of the original text information are extracted as the text style features adapted to the target text area with reference to Example B1. The other areas of the original image are areas outside the target text area in the original image.

[0065] Example B3, extracting text style features adapted to the target text area from the associated image of the original image.

[0066] For a detailed introduction to the associated images of the original image, please refer to the aforementioned embodiments. For ease of distinction and description, the text information included in the associated images of the original image is referred to as associated text information. An implementation method for extracting text style features that are adapted to a target text area from an associated image of an original image includes: extracting a local image where the associated text information is located from an associated image of the original image, and generating a third glyph image corresponding to the associated text information based on a preset glyph library; inputting the local image and the third glyph image into a style feature extraction model for feature extraction, respectively, to obtain third feature information and fourth feature information; the third feature information includes text style features and text content features of the associated text information, and the fourth feature information includes text content features of the associated text information; according to the fourth feature information, obtaining text style features of the associated text information from the third feature information as text style features that are adapted to the target text area.

[0067] In an optional embodiment, how to convert the target text information into a form that the target graphic editing model can understand is crucial. For example, in some feature encoding models, the text is segmented from a semantic level, such as, some words with complete meanings are processed as a whole, such as, "I" and "we" will be treated as a processing unit (token), which helps to capture the meaning of the vocabulary level. However, in an embodiment of the present application, it is hoped that the text encoding model has recognition ability for each character (such as Chinese characters). According to the above method of segmenting the text from a semantic level, semantic adhesion problems may be caused. Semantic adhesion (Semantic Conflation) generally refers to semantic ambiguity or confusion caused by improper use of vocabulary. Therefore, the text encoding model provided in the embodiment of the present application performs text encoding from the character dimension, and the text encoding model has the ability to distinguish for each character. The text encoding model can also be called a coding model for word separation, and word separation refers to text encoding from the character dimension, rather than text encoding from the semantic dimension.

[0068] Based on this, the image editing method provided in the embodiment of the present application further includes: generating text encoding features of the target text information. Specifically, the target text information is input into a text encoding model, characters and character positions in the target text information are jointly encoded to obtain at least one character encoding; and at least one character encoding is spliced ​​to obtain the text encoding features of the target text information.

[0069] The target text information is input into the text encoding model, specifically, at least one character included in the target text information is input into the text encoding model respectively, and the text encoding model outputs a character code for each character.

[0070] Optionally, an implementation method of jointly encoding characters and character positions in target text information to obtain at least one character code includes: searching for character code features of at least one character included in the target text information from a character database; performing position coding on at least one character according to the order of at least one character in the target text information to obtain the position coding features of at least one character; generating a character code for at least one character according to the character coding features and position coding features of at least one character. The character database includes character coding features of various characters (such as Chinese characters, numbers or letters, etc.), and the character database can be implemented as a code book. The character coding features can be character embedding vectors. The position coding mainly assists the target graphic editing model in understanding the relative or absolute position relationship between the characters in the target text information. An implementation method of generating a character code for at least one character according to the character coding features and position coding features of at least one character can be to concatenate each character coding feature and its position coding feature to obtain an intermediate feature, and input the intermediate feature into a text encoder for text encoding to obtain a character code, such as Figure 4 Alternatively, the intermediate feature is used as a character code.

[0071] In an optional embodiment, a method for calling a target image-text editing model based on text style features, character morphology features, and text encoding features of target text information to perform text editing processing on a target text area in an original image to obtain a target image includes: inputting text style features, text encoding features, a first glyph image and an original image, a first mask image, and a second mask image into the target image-text editing model, wherein the first mask image is obtained by masking the target text area in the original image, and the second mask image is a mask image for extracting the image area where the target text information is located; generating initial image features based on the first glyph image, the original image, the first mask image, and the second mask image; based on a cross-attention mechanism, repairing the initial image features using text style features and text encoding features to obtain target image features, wherein the target image features integrate text style features, character morphology features, and text layout features; generating a target image based on the target image features. Figure 3 The illustrations are provided from the dimensions of font shape, layout and style, but are not limited to these.

[0072] The first mask image is obtained by masking the target text area in the original image. For example, in the first mask image, the target text area is masked by black pixels, and other areas remain unchanged. The layout feature can reflect information such as font size, line spacing, character spacing, and alignment of the target text information.

[0073] The region of interest of the second mask image corresponds to the target text region, the second mask image may be a binary image, and the region of interest may be represented by 0 or 1. When the region of interest is represented by 1, the region of interest is a white region.

[0074] Optionally, an implementation method for generating initial image features based on a first glyph image, an original image, a first mask image and a second mask image includes: using an image feature extraction network to extract features from the first glyph image, the original image and the first mask image, respectively, to obtain character morphological features, basic image features and background image features, respectively; downsampling the second mask image to obtain mask features; and splicing the mask features, character morphological features, basic image features and background image features to obtain initial image features.

[0075] Optionally, an implementation method based on a cross-attention mechanism, using the text style features and the text encoding features to repair the initial image features to obtain target image features, includes: based on the cross-attention layer in the graphic editing model, performing cross-attention calculations on the initial image features, the text style features and the text encoding features, respectively, to obtain a first cross-attention result and a second cross-attention result; generating the target image features based on the first cross-attention result and the second cross-attention result.

[0076] For example, the target image feature is generated according to the weighted sum of the first cross attention result and the second cross attention result. For another example, the first cross attention result and the second cross attention result are subjected to self-attention calculation, spatial attention calculation, convolution or residual calculation, etc. to obtain the target output feature.

[0077] Among them, the cross-attention layer in the image and text editing model includes multiple, for example, the first cross-attention layer performs cross-attention calculation on the initial image features and text style features to obtain a first cross-attention result; the second cross-attention layer performs cross-attention calculation on the initial image features and text encoding features to obtain a second cross-attention result. Optionally, the image and text editing model includes: a multi-scale feature extraction network, the multi-scale feature extraction network includes a first cross-attention layer and a second cross-attention layer of multiple scales, and the cross-attention calculation can be performed at each scale by the first cross-attention layer and the second cross-attention layer. For the introduction of the multi-scale feature extraction network, please refer to the aforementioned embodiment, which will not be repeated here.

[0078] Further optionally, the text style features and the text encoding features have an initial scale, and the initial image features also have the same initial scale. In this case, an implementation method based on a cross-attention layer, in which the initial image features are cross-attended with the text style features and the text encoding features, respectively, to obtain a first cross-attention result and a second cross-attention result, includes: based on the text style features and text encoding features with the initial scale, combined with multiple scale information supported by the multi-scale feature extraction network in the target image and text editing model, generate text style features of multiple scales and text encoding features of multiple scales; using the cross-attention layer in the multi-scale feature extraction model, the input features of any scale in the multi-scale feature extraction network are cross-attended with the text style features and text encoding features of any scale, respectively, to obtain a first cross-attention result and a second cross-attention result; wherein the input features on the first scale are the initial image features, and the output features on any scale are the input features on the next scale; based on the first cross-attention result and the second cross-attention result, generate output features on any scale; and use the output features on the last scale as the target image features.

[0079] Among them, the implementation method of generating text style features of multiple scales and text encoding features of multiple scales according to text style features and text encoding features with initial scales, combined with multiple scale information of the multi-scale feature extraction network in the target image and text editing model is not limited. For example, a conditional injection model can be used to generate text style features of multiple scales and text encoding features of multiple scales according to text style features and text encoding features with initial scales, combined with multiple scale information of the target image and text editing model, and the text style features on multiple scales and the text encoding features on multiple scales are input into the target image and text editing model. Among them, the conditional injection model can be an implicit potential alignment (IPA) model or a control model ControlNet).

[0080] Among them, the implicit latent alignment model is a method for introducing conditional information into the target image-text editing model. By aligning image features at different scales in the latent space, IPA can effectively inject text style features or text encoding features into the target image-text editing model. ControlNet is a method for introducing conditional information into the target image-text editing model. This method introduces a control module into the target image-text editing model. The module receives text style features or text encoding features, converts them into different scales, and performs cross-attention calculations on the input features to guide the generation of the target image.

[0081] Among them, for any scale, the text style features and input features at the scale are cross-attention calculated to obtain a first cross-attention result; the text encoding features and input features at the scale are cross-attention calculated to obtain a second cross-attention result; based on the first cross-attention result and the second cross-attention result, the output features at any scale are generated.

[0082] Optionally, an implementation method of generating a target image based on target image features includes: decoding the target image features to obtain a predicted image, the predicted image including target text information with text style features, and directly using the predicted image as the target image.

[0083] Optionally, another implementation method of generating a target image based on target image features includes: decoding the target image features to obtain a predicted image, the predicted image includes target text information having text style features, character morphology features, and text layout features; extracting the local image area where the target text information is located from the predicted image based on the second mask image, the target text area being the region of interest in the second mask image; replacing the target text area in the original image with the local image area to obtain the target image. The second mask image can be a binary image, and the region of interest can be represented by 0 or 1. When the region of interest is represented by 1, the region of interest is a white area.

[0084] In the above two implementations, the target image features can be decoded by an image feature conversion network to obtain a predicted image. The introduction of the image feature conversion network can be found in the above embodiments, which will not be repeated here.

[0085] Further optionally, in order to make the local image area more closely fit the original image, the transparency is set for the edge of the region of interest. When the edge of the region of interest has a certain degree of transparency, it can display part of the background information of the original image when synthesized with the original image, rather than suddenly switching from the background of the original image to the local image area. This smooth transition reduces the hard boundary at the edge of the region of interest, making the local image area look more like a part of the original image rather than a rigidly superimposed area. Based on this, an implementation method for extracting the local image area where the target text information is located from the predicted image according to the mask image of the target text area includes: adjusting the transparency of the associated area of ​​the region of interest in the second mask image to obtain a second mask image with optimized edges; extracting the local image area where the target text information is located from the predicted image according to the second mask image with optimized edges. Figure 3 The method of edge optimization and generating the target image is demonstrated in FIG.

[0086] The associated area of ​​the edge position may be an area with a distance threshold set from the edge position, and the distance threshold may be 1px, 2px, or 5px, etc., where px is a pixel. The transparency adjustment may be to set the transparency of the pixels in the associated area to a uniform value, for example, 80% or 50%. 100% is fully transparent and 0% is opaque. Alternatively, the transparency adjustment may be to reduce the transparency in order from near to far from the position of the pixels in the associated area to the edge position.

[0087] In addition to providing the above-mentioned image editing method, the embodiment of the present application also provides a model training method, which includes:

[0088] Acquire multiple sample information, the sample information including: sample image, original text information, sample text encoding features, sample character morphology features and sample text style features;

[0089] According to the sample text style features, the sample character morphology features and the sample text encoding features, the initial image-text editing model is called to perform text editing processing on the sample text area in the sample image to obtain a target image; the target image includes original text information with text style features and character morphology features;

[0090] Constructing a loss function of an initial image-text editing model according to the original text information included in the sample image and the sample text information included in the target image;

[0091] According to the loss function and multiple sample information, the initial image and text editing model is iteratively trained to obtain the target image and text editing model.

[0092] In this embodiment, the sample image includes a sample text area for carrying original text information. For a detailed introduction to sample text encoding features, sample character morphological features, and sample text style features, please refer to the aforementioned introduction to text encoding features, character morphological features, and text style features, which will not be repeated here. In particular, regarding the input of the model and the specific implementation of the model in the model training process, please refer to the aforementioned embodiment, which will not be repeated here.

[0093] In this embodiment, during the model training process, the target image generated by the initial image-text editing model includes sample text information, and the sample image includes original text information. In order to facilitate the calculation of the loss of the initial image-text editing model, the sample text information and the original text information have the same text content. Subsequently, the model loss can be calculated based on the target image including the sample text information and the sample image including the original text information.

[0094] Optionally, an implementation method of constructing a loss function of an initial image-text editing model based on original text information included in a sample image and sample text information included in a target image includes: constructing a first glyph loss function based on pixel features of the sample image and the target image; inputting the sample image and the target image into a character recognition model to extract glyph features, respectively, to obtain a first glyph feature and a second glyph feature; the first glyph feature includes glyph features of the sample text information, and the second glyph feature includes glyph features of the original text information in the sample image; constructing a second glyph loss function based on the first glyph feature and the second glyph feature; constructing a glyph loss function as the loss function of the initial image generation model based on the first glyph loss function and the second glyph loss function. Figure 4 As shown, the training process of the initial image and text editing model is exemplarily demonstrated, but it is not limited to this.

[0095] The first glyph loss function may include but is not limited to: mean squared error (MSE), mean absolute error (MAE), and cross-entropy loss (Cross-Entropy Loss).

[0096] Among them, the optical character recognition (OCR) model is mainly used to extract character features from images to ensure the accuracy of image text editing. Character recognition models can include but are not limited to: 1) Convolutional neural network (CNN). In the process of character recognition by the CNN model, features are extracted through convolutional layers and pooling layers, and then classified through fully connected layers; 2) Convolutional recurrent neural network (CRNN). CRNN combines the advantages of CNN and RNN (recurrent neural network) and is suitable for the recognition of sequence data, such as handwritten text recognition or long text line recognition; 3) Connectionist temporal classification (CTC) model. CTC is a technology for sequence labeling, which is often used for handwritten text recognition and can also be used in combination with CNN for character recognition. 4) Transformer-based model. Transformer is used for image recognition tasks, including character recognition. For example, Transformer is combined with CNN for text recognition. 5) RNN-based model. RNN can capture the sequential information between characters and is suitable for the recognition of long text lines. 6) Based on deep CNN models such as ResNet, use deeper network structures such as ResNet to improve recognition accuracy.

[0097] Among them, an implementation method of constructing a glyph loss function as the loss function of an initial image generation model based on a first glyph loss function and a second glyph loss function includes: weighted summing the first glyph loss function and the second glyph loss function to obtain the glyph loss function as the loss function of the initial image generation model.

[0098] Among them, the first glyph loss function in the pixel dimension can make the glyph and background color harmonious and unified. For example, if you want the characters to be clearly displayed in the background area, the character color and the background color need to have a certain difference. The second glyph loss function perceives the loss from the character dimension and constrains the consistency of the glyph. The two glyph loss functions can realize the glyph control of the model, improve the ability to understand complex text, and enable the initial graphic editing model to correctly generate the strokes, radicals, glyph structure, etc. of the characters.

[0099] Optionally, characters can have many different layout information, such as font size, orientation (horizontal or vertical) and spacing, etc., and there is a natural ordered correlation between characters, that is, the order of characters in the text information is fixed. Therefore, it is important to perform layout control on the target text information while ensuring the correct order of characters. Among them, when performing layout control on the characters in the target text information, if there is an inconsistency in the number of characters or the type of characters between the original text information and the target text information, it may cause the generated target text information to be inconsistent with the layout of other text information in the target image. For example, the original text information includes 7 characters, and accordingly, the target text area can carry 7 characters. However, the target text information includes 5 characters. In the process of generating the target text information, the 5 characters may fill the area where the 7 characters are located, resulting in the font size of the 5 characters may become larger, or the spacing between the 5 characters may become larger.

[0100] Based on this, during the training process, a layout loss function is constructed. Through the layout loss function, the initial image and text editing model learns the layout control ability of characters, so that the initial image and text editing model can accurately predict the position of each character in the target text information under different layouts.

[0101] The model training method provided by the embodiment of the present application further includes: The initial graphic and text editing model includes cross-attention layers at multiple scales, and the cross-attention layer is used to perform cross-attention calculation on the text style features and text encoding features at its corresponding scale; according to the input features, text style features, and text encoding features of the cross-attention layers at multiple scales, generate target attention maps at multiple scales, and the target attention map reflects the sample character position information of the characters in the sample text information in the target image; according to the sample character position information and the original character position information corresponding to the original text information included in the sample image, construct a layout loss function; according to the glyph loss function and the layout loss function, construct the loss function of the initial graphic and text editing model.

[0102] Among them, the cross-attention mechanism allows the initial graphic and text editing model to be able to "focus" on different parts of the target image when calculating cross-attention, so as to better understand and generate the output. For example, various attention maps are generated through the query (Query, Q) vector, key (Key, K) vector, and value (Value, V) vector involved in the cross-attention calculation process. The attention map is a tool used to visualize the working mode of the attention mechanism. The attention map can reflect the information of the target image in each dimension. For example, some attention maps can reflect the position information of the characters in the sample text information in the target image, some attention maps reflect the shape of the characters, and some attention maps reflect the background information in the target image, etc. In Figure 5 it, taking the attention maps corresponding to the four characters "In stock and shipped quickly" included in the target image as an example for illustration, but not limited thereto.

[0103] In this embodiment, the attention map that reflects the position information of the characters in the sample text information in the target image is called the target attention map. Among them, the position information of the characters in the sample text information in the target attention map can be called the sample character position information. According to the sample character position information and the original character position information corresponding to the original text information included in the sample image, construct a layout loss function to promote the initial graphic and text editing model to learn the layout ability.

[0104] Among them, an implementation manner of constructing the loss function of the initial graphic and text editing model according to the glyph loss function and the layout loss function includes: adding the glyph loss function and the layout loss function with weights to obtain the loss function of the initial graphic and text editing model, where the weights of the weights can be determined according to application requirements. For example, if the focus is on glyph generation, the weight of the glyph loss function is set to be greater than the weight of the layout loss function; if the focus is on layout, the weight of the glyph loss function is set to be less than the weight of the layout loss function. Of course, the weighted weights of the glyph loss function and the layout loss function can also be the same.

[0105] Among them, by constructing a layout loss function, the initial graphic editing model can realize the position alignment of characters in the target text information. When the user enters the target text information, the layout control can be realized through spaces. For example, the space can be used as a placeholder. When the target text area can carry 7 characters and the target text information includes 5 characters, if the target text information is desired to be right-aligned in the target text area, the user can enter "space", "space" and "5 characters" when entering the target text information to realize the right alignment of 5 characters in the target text area. If the left alignment is to be realized, the same is true. It will not be repeated here. In addition, when the user enters the target text information, the modification of multiple lines of text can also be realized by "line break", that is, the target text information is displayed in the target image in the form of multiple lines. In addition, if the user selects the original text information to determine the target text area, if the user wants to generate vertical target text information, the user can select the vertical original text information to obtain the vertical target text area, thereby realizing the horizontal and vertical control of the text.

[0106] In an optional embodiment, considering that during the model training process, one of the inputs of the model is a first mask image, the introduction of the first mask image can be found in the aforementioned embodiment and will not be repeated here. The first mask image includes a mask area, which corresponds to the target text area. Considering that the model may determine the position and font size of the characters in the target text information according to the size information and position information of the mask area, that is, the mask area leaks the position information and font size information of the target text information, resulting in the initial graphic editing model overfitting instead of learning more general text editing capabilities.

[0107] Therefore, in order to improve the robustness of the initial image and text editing model, the mask area in the first mask image is randomly expanded to increase the diversity of training data, so that the initial image and text editing model is more robust to different mask positions and shapes. This strategy effectively improves the performance of text editing. Among them, the random expansion method can be to use morphological operations (such as expansion) for random expansion, and to achieve random expansion by randomly defining the shape and size of the expansion. Or, random expansion can be achieved by randomly moving the boundary points of the mask area.

[0108] Regarding the embodiments of the present application Figure 1 The detailed implementation and beneficial effects of each step in the method have been described in detail in the aforementioned embodiments and will not be elaborated here.

[0109] It should be noted that the execution subject of each step of the method provided in the above embodiment can be the same device, or the method can be executed by different devices. For example, the execution subject of steps 101 to 103 can be device A; for another example, the execution subject of steps 101 and 102 can be device A, and the execution subject of step 103 can be device B; and so on.

[0110] In addition, in some of the processes described in the above embodiments and the accompanying drawings, multiple operations that appear in a specific order are included, but it should be clearly understood that these operations may not be executed in the order in which they appear in this article or executed in parallel, and the sequence numbers of the operations, such as 101, 102, etc., are only used to distinguish between different operations, and the sequence numbers themselves do not represent any execution order. In addition, these processes may include more or fewer operations, and these operations may be executed in sequence or in parallel. It should be noted that the descriptions of "first", "second", etc. in this article are used to distinguish different messages, devices, modules, etc., do not represent the order of precedence, and do not limit the "first" and "second" to be different types.

[0111] Figure 6 A schematic diagram of the structure of an image editing device provided by an exemplary embodiment of the present application is shown in FIG. Figure 6 As shown, the device includes: an acquisition module 61, a generation module 62, an extraction module 63 and an editing module 64.

[0112] An acquisition module 61 is used to acquire an original image and target text information submitted by a user, wherein the original image includes a target text area to be edited, the target text area is used to carry the target text information, and the target text information includes at least one character;

[0113] A generating module 62, configured to generate a first glyph image corresponding to the target text information based on a preset glyph library, wherein the first glyph image reflects the character morphological features in the target text information;

[0114] An extraction module 63, used to extract text style features adapted to the target text area from the original image or an image associated with the original image;

[0115] The editing module 64 is used to call the target image-text editing model according to the text style features, character morphology features and text encoding features of the target text information, and perform text editing processing on the target text area in the original image to obtain a target image; wherein the target image includes target text information with text style features and character morphology features.

[0116] In an optional embodiment, when the target text area includes original text information, the extraction module is specifically used to: extract a local image where the original text information is located from the original image, and generate a second glyph image corresponding to the original text information based on a preset glyph library; input the local image and the second glyph image into a style feature extraction model for feature extraction, respectively, to obtain first feature information and second feature information; the first feature information includes text style features and text content features of the original text information, and the second feature information includes text content features of the original text information; based on the second feature information, obtain the text style features of the original text information from the first feature information as text style features adapted to the target text area.

[0117] In an optional embodiment, the device further includes: an encoding module and a splicing module. The encoding module is used to input the target text information into the text encoding model, and jointly encode the characters and character positions in the target text information to obtain at least one character encoding; the splicing module is used to splice at least one character encoding to obtain the text encoding feature of the target text information.

[0118] Optionally, the encoding module is specifically used to: search for character encoding features of at least one character included in the target text information from a character database; perform position encoding on at least one character according to the order of the at least one character in the target text information to obtain a position encoding feature of the at least one character; and generate a character encoding for at least one character according to the character encoding feature and position encoding feature of the at least one character.

[0119] In an optional embodiment, the editing module is specifically used to: input text style features, text encoding features, a first glyph image, an original image, a first mask image and a second mask image into a target graphic editing model, wherein the first mask image is obtained by masking a target text area in the original image, and the second mask image is a mask image used to extract an image area where target text information is located; generate initial image features based on the first glyph image, the original image, the first mask image and the second mask image; based on a cross-attention mechanism, repair the initial image features using text style features and text encoding features to obtain target image features, which integrate text style features, character morphology features and text layout features; generate a target image based on the target image features, and the target image includes target text information having text style features, character morphology features and text layout features.

[0120] Optionally, the editing module is specifically used to: use an image feature extraction network to perform feature extraction on the first glyph image, the original image and the first mask image, respectively, to obtain character morphological features, basic image features and background image features, respectively; downsample the second mask image to obtain mask features; and splice the mask features, character morphological features, basic image features with added noise and background image features to obtain initial image features.

[0121] Optionally, the editing module is specifically used to: based on the cross-attention layer in the image and text editing model, perform cross-attention calculations on the initial image features, the text style features, and the text encoding features, respectively, to obtain a first cross-attention result and a second cross-attention result; generate target image features based on the first cross-attention result and the second cross-attention result.

[0122] Optionally, the editing module is specifically used to: perform feature decoding on the target image features to obtain a predicted image, the predicted image including target text information having text style features, character morphology features and text layout features; extract a local image area where the target text information is located from the predicted image according to a second mask image; replace the target text area in the original image with the local image area to obtain the target image.

[0123] Further optionally, the editing module is specifically used to: adjust the transparency of the associated area of ​​the area of ​​interest in the second mask image to obtain the edge-optimized second mask image; and extract the local image area where the target text information is located from the predicted image based on the edge-optimized second mask image.

[0124] In an optional embodiment, the device further includes: a training module. The training module is used to obtain multiple sample information, the sample information includes: sample images, sample text information, sample text encoding features, sample character morphological features and sample text style features; according to the sample text style features, sample character morphological features and sample text encoding features, call the initial image-text editing model to perform text editing processing on the sample text area in the sample image to obtain a target image, the target image includes sample text information with text style features and character morphological features; according to the original text information included in the sample image and the sample text information included in the target image, construct a loss function of the initial image-text editing model; according to the loss function and the multiple sample information, iteratively train the initial image-text editing model to obtain a target image-text editing model.

[0125] Optionally, the training module is specifically used to: construct a first glyph loss function based on pixel features of a sample image and a target image; input the sample image and the target image into a character recognition model respectively to extract glyph features to obtain a first glyph feature and a second glyph feature; the first glyph feature includes glyph features of sample text information, and the second glyph feature includes glyph features of original text information in the sample image; construct a second glyph loss function based on the first glyph feature and the second glyph feature; construct a glyph loss function based on the first glyph loss function and the second glyph loss function as the loss function of the initial image generation model.

[0126] Further optionally, the initial image and text editing model includes cross-attention layers at multiple scales, and the cross-attention layers are used to perform cross-attention calculations on text style features and text encoding features at their corresponding scales; the training module is also used to: generate target attention maps at multiple scales based on the input features, text style features and text encoding features of the cross-attention layers at multiple scales, and the target attention maps reflect the sample character position information of the characters in the sample text information in the target image; construct a layout loss function based on the sample character position information and the original character position information corresponding to the original text information included in the sample image; and construct the loss function of the initial image and text editing model based on the glyph loss function and the layout loss function.

[0127] An embodiment of the present application also provides a model training device, which includes: an acquisition module, an editing module, a construction module and an iteration module.

[0128] An acquisition module is used to acquire multiple sample information, the sample information includes: sample image, sample text information, sample text encoding features, sample character morphology features and sample text style features;

[0129] The editing module is used to call the initial image-text editing model according to the sample text style features, the sample character morphology features and the sample text encoding features, and perform text editing processing on the sample text area in the sample image to obtain a target image, wherein the target image includes sample text information with text style features and character morphology features;

[0130] A construction module, used to construct a loss function of an initial image-text editing model according to original text information included in the sample image and sample text information included in the target image;

[0131] The iteration module is used to iteratively train the initial image and text editing model according to the loss function and multiple sample information to obtain the target image and text editing model.

[0132] Regarding the embodiments of the present application Figure 6The detailed implementation and beneficial effects of each step in the illustrated device have been described in detail in the aforementioned embodiments and will not be elaborated on here.

[0133] Figure 7 A schematic diagram of the structure of an electronic device provided by an exemplary embodiment of the present application is shown in FIG. Figure 7 As shown, the device includes: a memory 74 and a processor 75.

[0134] The memory 74 is used to store computer programs and can be configured to store various other data to support operations on the electronic device. Examples of such data include instructions for any application or method used to operate on the electronic device.

[0135] The processor 75 is coupled to the memory 74 and is used to execute the computer program in the memory 74, so as to: obtain an original image and target text information submitted by a user, wherein the original image includes a target text area to be edited, and the target text area is used to carry the target text information, and the target text information includes at least one character; generate a first glyph image corresponding to the target text information based on a preset glyph library, and the first glyph image reflects the character morphological features in the target text information; extract text style features adapted to the target text area from the original image or an associated image of the original image; call a target graphic editing model according to the text style features, the character morphological features and the text encoding features of the target text information, and perform text editing processing on the target text area in the original image to obtain a target image; wherein the target image includes the target text information having text style features and character morphological features.

[0136] In an optional embodiment, when the processor 75 extracts text style features that are adapted to the target text area from the original image when the target text area includes original text information, the processor is specifically used to: extract a local image where the original text information is located from the original image, and generate a second glyph image corresponding to the original text information based on a preset glyph library; input the local image and the second glyph image into a style feature extraction model for feature extraction, respectively, to obtain first feature information and second feature information; the first feature information includes text style features and text content features of the original text information, and the second feature information includes text content features of the original text information; according to the second feature information, obtain the text style features of the original text information from the first feature information as the text style features that are adapted to the target text area.

[0137] In an optional embodiment, the processor 75 is also used to: input the target text information into a text encoding model, jointly encode the characters and character positions in the target text information to obtain at least one character encoding; and concatenate at least one character encoding to obtain text encoding features of the target text information.

[0138] Optionally, when the processor 75 jointly encodes the characters and character positions in the target text information to obtain at least one character code, it is specifically used to: search for the character encoding feature of at least one character included in the target text information from the character database; perform position encoding on at least one character according to the order of the at least one character in the target text information to obtain the position encoding feature of the at least one character; and generate the character code of at least one character according to the character encoding feature and position encoding feature of the at least one character.

[0139] In an optional embodiment, when the processor 75 calls the target image-text editing model based on the text style features, character morphology features and text encoding features of the target text information, and performs text editing processing on the target text area in the original image to obtain the target image, it is specifically used to: input the text style features, text encoding features, the first glyph image, the original image, the first mask image and the second mask image into the target image-text editing model, the first mask image is obtained by masking the target text area in the original image, and the second mask image is a mask image for extracting the image area where the target text information is located; generate initial image features based on the first glyph image, the original image, the first mask image and the second mask image; based on the cross-attention mechanism, repair the initial image features using the text style features and the text encoding features to obtain target image features, the target image features integrating the text style features, the character morphology features and the text layout features; generate a target image based on the target image features, the target image including the target text information having the text style features, the character morphology features and the text layout features.

[0140] Optionally, when the processor 75 generates initial image features based on the first glyph image, the original image, the first mask image and the second mask image, it is specifically used to: use an image feature extraction network to extract features from the first glyph image, the original image and the first mask image, respectively, to obtain character morphological features, basic image features and background image features, respectively; downsample the second mask image to obtain mask features; and splice the mask features, character morphological features, basic image features with added noise and background image features to obtain initial image features.

[0141] Optionally, when the processor 75 repairs the initial image features using text style features and text encoding features based on the cross-attention mechanism to obtain the target image features, it is specifically used to: based on the cross-attention layer in the graphic and text editing model, perform cross-attention calculations on the initial image features, the text style features and the text encoding features, respectively, to obtain a first cross-attention result and a second cross-attention result; generate the target image features according to the first cross-attention result and the second cross-attention result.

[0142] Optionally, when the processor 75 generates a target image based on the target image features, it is specifically used to: perform feature decoding on the target image features to obtain a predicted image, the predicted image including target text information having text style features, character morphology features and text layout features; extract a local image area where the target text information is located from the predicted image based on a second mask image; and replace the target text area in the original image with the local image area to obtain the target image.

[0143] Further optionally, when the processor 75 extracts the local image area where the target text information is located from the predicted image based on the second mask image, it is specifically used to: adjust the transparency of the associated area of ​​the area of ​​interest in the second mask image to obtain the second mask image with optimized edges; and extract the local image area where the target text information is located from the predicted image based on the second mask image with optimized edges.

[0144] In an optional embodiment, the processor 75 is also used to: obtain multiple sample information, the sample information including: sample images, sample text information, sample text encoding features, sample character morphological features and sample text style features; call the initial image and text editing model based on the sample text style features, sample character morphological features and sample text encoding features, and perform text editing processing on the sample text area in the sample image to obtain a target image, wherein the target image includes sample text information with text style features and character morphological features; construct a loss function of the initial image and text editing model based on the original text information included in the sample image and the sample text information included in the target image; and iteratively train the initial image and text editing model based on the loss function and multiple sample information to obtain a target image and text editing model.

[0145] Optionally, when the processor 75 constructs the loss function of the initial image-text editing model based on the original text information included in the sample image and the sample text information included in the target image, it is specifically used to: construct a first glyph loss function based on pixel features of the sample image and the target image; input the sample image and the target image into the character recognition model respectively to extract glyph features to obtain a first glyph feature and a second glyph feature; the first glyph feature includes the glyph feature of the sample text information, and the second glyph feature includes the glyph feature of the original text information in the sample image; construct a second glyph loss function based on the first glyph feature and the second glyph feature; construct a glyph loss function as the loss function of the initial image generation model based on the first glyph loss function and the second glyph loss function.

[0146] Optionally, the initial image and text editing model includes cross-attention layers at multiple scales, and the cross-attention layers are used to perform cross-attention calculations on text style features and text encoding features at their corresponding scales; the processor 75 is also used to: generate target attention maps at multiple scales based on the input features, text style features and text encoding features of the cross-attention layers at multiple scales, and the target attention maps reflect the sample character position information of the characters in the sample text information in the target image; construct a layout loss function based on the sample character position information and the original character position information corresponding to the original text information included in the sample image; and construct the loss function of the initial image and text editing model based on the glyph loss function and the layout loss function.

[0147] Regarding the embodiments of the present application Figure 7 The detailed implementation and beneficial effects of each step in the illustrated device have been described in detail in the aforementioned embodiments and will not be elaborated on here.

[0148] Further, if Figure 7 As shown, the electronic device also includes: a communication component 76, a display 77, a power component 78, an audio component 79 and other components. Figure 7 Only some components are shown schematically, which does not mean that the electronic device only includes Figure 7 In addition, Figure 7 The components in the dashed box are optional components, not mandatory components, and the specific components depend on the product form of the electronic device. The electronic device of this embodiment can be implemented as a terminal device such as a desktop computer, a laptop computer, a smart phone, or an IOT (Internet of Things) device and a smart wearable device (such as a smart watch, a smart bracelet), or a conventional server, a cloud server, or a server array. If the electronic device of this embodiment is implemented as a terminal device such as a desktop computer, a laptop computer, a smart phone, etc., it can include Figure 7If the electronic device of this embodiment is implemented as a server device such as a conventional server, a cloud server or a server array, it may not include Figure 7 Components within the dashed box.

[0149] The embodiment of the present application also provides an electronic device, the implementation structure of the electronic device is Figure 7 The implementation structure of the electronic device shown is the same or similar, and can refer to Figure 7 The electronic device provided in this embodiment is implemented as follows: Figure 7 The difference between the electronic devices in the illustrated embodiments mainly lies in that the functions implemented by the processor executing the computer program stored in the memory are different. For the electronic device provided in this embodiment, its processor executes the computer program stored in the memory, which can be used to: obtain multiple sample information, the sample information includes: sample image, sample text information, sample text encoding features, sample character morphological features and sample text style features; according to the sample text style features, sample character morphological features and sample text encoding features, call the initial image and text editing model, perform text editing processing on the sample text area in the sample image to obtain a target image, the target image includes sample text information with text style features and character morphological features; according to the original text information included in the sample image and the sample text information included in the target image, construct the loss function of the initial image and text editing model; according to the loss function and multiple sample information, iteratively train the initial image and text editing model to obtain the target image and text editing model.

[0150] The detailed implementation and beneficial effects of the electronic device provided in the embodiments of the present application have been described in detail in the aforementioned embodiments and will not be elaborated here.

[0151] Accordingly, an embodiment of the present application also provides a computer-readable storage medium storing a computer program, which, when executed, can implement the steps that can be performed by an electronic device in the above-mentioned image editing method and model training method embodiments.

[0152] Accordingly, an embodiment of the present application also provides a computer program product, including a computer program / instructions. When the computer program / instructions are executed by a processor, the processor implements the steps that can be executed by an electronic device in the above-mentioned image editing method and model training method embodiments.

[0153] The above-mentioned memory can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read only memory (EEPROM), erasable programmable read only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk or optical disk.

[0154] The above-mentioned communication component is configured to facilitate wired or wireless communication between the device where the communication component is located and other devices. The device where the communication component is located can access a wireless network based on a communication standard, such as WiFi, 2G, 3G, 4G / LTE, 5G and other mobile communication networks, or a combination thereof. In an exemplary embodiment, the communication component receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be based on Radio Frequency Identification (RFID) technology, Infrared Data Association (IrDA) technology, Ultra Wide Band (UWB) technology, Bluetooth (BT) technology and other technologies.

[0155] The above-mentioned display includes a screen, and the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from a user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundary of a touch or slide action, but also detect the duration and pressure associated with the touch or slide operation.

[0156] The power supply assembly provides power to various components of the device where the power supply assembly is located. The power supply assembly may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the device where the power supply assembly is located.

[0157] The above-mentioned audio component can be configured to output and / or input audio signals. For example, the audio component includes a microphone (Microphone, MIC), and when the device where the audio component is located is in an operating mode, such as a call mode, a recording mode, and a speech recognition mode, the microphone is configured to receive an external audio signal. The received audio signal can be further stored in a memory or sent via a communication component. In some embodiments, the audio component also includes a speaker for outputting an audio signal.

[0158] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-readable storage media (including but not limited to disk storage, compact disc read-only memory (Compact Disc Read-Only Memory, CD-ROM), optical storage, etc.) containing computer-usable program code.

[0159] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0160] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0161] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0162] In a typical configuration, a computing device includes one or more processors (Central Processing Unit, CPU), input / output interface, network interface and memory.

[0163] Memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. Memory is an example of a computer-readable medium.

[0164] Computer readable media include permanent and non-permanent, removable and non-removable media that can be used to store information by any method or technology. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase-change random access memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, read-only compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic tape disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0165] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0166] The above are only embodiments of the present application and are not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included within the scope of the claims of the present application.

Claims

1. An image editing method, characterized in that: include: Acquire an original image and target text information submitted by a user, wherein the original image includes a target text area to be edited, the target text area is used to carry the target text information, and the target text information includes at least one character; Generating a first glyph image corresponding to the target text information based on a preset glyph library, wherein the first glyph image reflects the character morphological features in the target text information; Extracting text style features adapted to the target text area from the original image or an image associated with the original image; According to the text style feature, the character morphology feature and the text encoding feature of the target text information, a target image-text editing model is called to perform text editing processing on the target text area in the original image to obtain a target image; The target image includes target text information having the text style features and the character morphology features.

2. The method according to claim 1, characterized in that In the case where the target text region includes original text information, extracting text style features adapted to the target text region from the original image includes: Extracting a partial image where the original text information is located from the original image, and generating a second glyph image corresponding to the original text information based on a preset glyph library; Inputting the partial image and the second glyph image into a style feature extraction model for feature extraction, respectively, to obtain first feature information and second feature information; the first feature information includes text style features and text content features of the original text information, and the second feature information includes text content features of the original text information; According to the second feature information, a text style feature of the original text information is acquired from the first feature information as a text style feature adapted to the target text region.

3. The method according to claim 1, characterized in that Also includes: Inputting the target text information into a text encoding model, jointly encoding the characters and character positions in the target text information to obtain at least one character encoding; The at least one character code is concatenated to obtain a text code feature of the target text information.

4. The method according to claim 3, characterized in that The characters and character positions in the target text information are jointly encoded to obtain at least one character code, including: Searching a character encoding feature of at least one character included in the target text information from a character database; According to the order of the at least one character in the target text information, position coding is performed on the at least one character to obtain a position coding feature of the at least one character; A character code of each of the at least one character is generated according to the character code feature and the position code feature of each of the at least one character.

5. The method according to any one of claims 1 to 4, characterized in that: According to the text style feature, the character morphology feature and the text encoding feature of the target text information, calling a target image-text editing model, and performing text editing processing on the target text area in the original image to obtain a target image, including: Inputting the text style feature, the text encoding feature, the first glyph image, the original image, a first mask image and a second mask image into a target image-text editing model, wherein the first mask image is obtained by masking the target text area in the original image, and the second mask image is a mask image used to extract the image area where the target text information is located; generating initial image features according to the first glyph image, the original image, the first mask image and the second mask image; Based on the cross attention mechanism, the initial image features are repaired using the text style features and the text encoding features to obtain target image features, where the target image features integrate the text style features, the character morphology features and the text layout features; The target image is generated according to the target image features, wherein the target image includes target text information having the text style features, the character morphology features, and the text layout features.

6. The method according to claim 5, characterized in that Generating initial image features according to the first glyph image, the original image, the first mask image and the second mask image includes: Extracting features from the first glyph image, the original image, and the first mask image to obtain character morphological features, basic image features, and background image features; Downsampling the second mask image to obtain a mask feature; The mask features, character morphological features, basic image features with added noise, and background image features are concatenated to obtain initial image features.

7. The method according to claim 5, characterized in that Based on the cross attention mechanism, the initial image features are repaired using the text style features and the text encoding features to obtain target image features, including: Based on the cross attention layer in the image-text editing model, cross attention calculation is performed on the initial image feature, the text style feature and the text encoding feature, respectively, to obtain a first cross attention result and a second cross attention result; The target image feature is generated according to the first cross-attention result and the second cross-attention result.

8. The method according to claim 5, characterized in that Generating the target image according to the target image feature includes: Decoding the target image features to obtain a predicted image, wherein the predicted image includes target text information having the text style features, the character morphology features, and the text layout features; According to the second mask image, extracting the local image area where the target text information is located from the predicted image; The target text area in the original image is replaced by the local image area to obtain a target image.

9. The method according to claim 8, characterized in that Extracting the local image region where the target text information is located from the predicted image according to the second mask image includes: Adjusting the transparency of the region associated with the region of interest in the second mask image to obtain a second mask image with optimized edges; According to the edge-optimized second mask image, a local image region where the target text information is located is extracted from the predicted image.

10. The method according to any one of claims 1 to 4 and 6 to 9, characterized in that: The method further comprises: Acquire multiple sample information, the sample information including: sample image, sample text information, sample text encoding features, sample character morphology features and sample text style features; According to the sample text style feature, the sample character morphology feature and the sample text encoding feature, an initial image-text editing model is called to perform text editing processing on the sample text area in the sample image to obtain a target image, wherein the target image includes sample text information having the text style feature and the character morphology feature; Constructing a loss function of the initial image-text editing model according to the original text information included in the sample image and the sample text information included in the target image; The initial image-text editing model is iteratively trained according to the loss function and the plurality of sample information to obtain a target image-text editing model.

11. The method according to claim 10, characterized in that According to the original text information included in the sample image and the sample text information included in the target image, a loss function of the initial image-text editing model is constructed, including: Constructing a first glyph loss function according to pixel features of the sample image and the target image; The sample image and the target image are respectively input into a character recognition model to extract glyph features, thereby obtaining a first glyph feature and a second glyph feature; the first glyph feature includes glyph features of the sample text information, and the second glyph feature includes glyph features of the original text information in the sample image; Constructing a second glyph loss function according to the first glyph feature and the second glyph feature; According to the first glyph loss function and the second glyph loss function, a glyph loss function is constructed as the loss function of the initial image generation model.

12. The method according to claim 11, characterized in that The initial image and text editing model includes a cross-attention layer at multiple scales, and the cross-attention layer is used to perform cross-attention calculations on text style features and text encoding features at corresponding scales; the method also includes: Generate target attention maps at the multiple scales according to input features of the cross attention layer at the multiple scales, text style features, and text encoding features, wherein the target attention maps reflect sample character position information of characters in the sample text information in the target image; Constructing a layout loss function according to the sample character position information and the original character position information corresponding to the original text information included in the sample image; A loss function of the initial image-text editing model is constructed according to the glyph loss function and the layout loss function.

13. A model training method, characterized in that: include: Acquire multiple sample information, the sample information including: sample image, sample text information, sample text encoding features, sample character morphology features and sample text style features; According to the sample text style feature, the sample character morphology feature and the sample text encoding feature, an initial image-text editing model is called to perform text editing processing on the sample text area in the sample image to obtain a target image, wherein the target image includes sample text information having the text style feature and the character morphology feature; Constructing a loss function of the initial image-text editing model according to the original text information included in the sample image and the sample text information included in the target image; The initial image-text editing model is iteratively trained according to the loss function and the plurality of sample information to obtain a target image-text editing model.

14. An electronic device, characterized in that: include: Memory and processor; The memory is used to store a computer program; the processor is coupled to the memory and is used to execute the computer program to implement the steps in the method of any one of claims 1 to 12 and claim 13.

15. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the processor is caused to implement the steps of the method according to any one of claims 1 to 12 and claim 13.

16. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the processor is caused to implement the steps in the method of any one of claims 1 to 12 and claim 13.

Images