Text editing method and device based on scene image, readable storage medium and program product

Abstract

The application relates to a text editing method and device based on a scene image, computer equipment, a computer readable storage medium and a computer program product. The method comprises the following steps: acquiring a first image to be edited, determining a first text to be replaced in the first image and a second text used for replacing the first text; acquiring image content where the first text is located in the first image to obtain a first text image; rendering the second text according to the stroke geometry structure of the second text to obtain a second text image; extracting visual appearance features of the first text from the first text image and extracting glyph structure features of the second text from the second text image; generating a replacement text image by constraining visual appearance based on the visual appearance features and constraining glyph structure based on the glyph structure features; and replacing the image content where the first text is located in the first image with the replacement text image to obtain a second image, thereby improving the consistency of visual appearance before and after text replacement.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to a text editing method, apparatus, computer device, computer-readable storage medium, and computer program product based on scene images. Background Technology

[0002] With the development of image processing technology, techniques have emerged for editing text content within scene images. Scene images refer to images that contain text, such as road signs, shop signs, product packaging, or promotional posters. In situations involving cross-language content localization, image material reuse, and information anonymization, it is often necessary to replace one piece of text in a scene image with another while keeping the rest of the scene image unchanged.

[0003] In traditional techniques, the above text replacement process is usually treated as an image inpainting task. First, the area where the text to be replaced is located in the scene image is occluded using a mask, so that the image content in that area is removed. Then, the image generation model regenerates the text content in the occluded area based on the text description and merges it with the surrounding image.

[0004] However, the above method discards the original pixels of the area where the text to be replaced is located during the occlusion step. When regenerating the text content, the only reference objects are the surrounding images of that area. Therefore, the color, texture, font and other visual appearance of the newly generated text are inferred from the surrounding images, and there are often differences between them and the visual appearance of the text before replacement, resulting in poor image processing effect. Summary of the Invention

[0005] Based on this, this application provides a text editing method, apparatus, computer device, computer-readable storage medium, and computer program product based on scene images, which can improve the consistency of visual appearance before and after text replacement and optimize image processing effects.

[0006] The application provides a text editing method based on scene images, including:

[0007] Obtain the first image to be edited, and determine the first text to be replaced in the first image and the second text to replace the first text;

[0008] The image content containing the first text is obtained from the first image to obtain the first text image; and the second text is rendered according to the stroke geometry of the second text to obtain the second text image.

[0009] Extract the visual appearance features of the first text from the first text image, and extract the glyph structure features of the second text from the second text image;

[0010] The visual appearance is constrained by visual appearance features, and the glyph structure is constrained by glyph structure features to generate replacement text images.

[0011] The second image is obtained by replacing the image content containing the first text in the first image with the image containing the replacement text.

[0012] In one embodiment, the steps of obtaining the image content containing the first text in the first image to obtain the first text image, and rendering the second text according to the stroke geometry of the second text to obtain the second text image include:

[0013] Determine the bounding shape of the region where the first text is located in the first image; obtain the image content within the bounding shape of the first image to obtain the first text image; wherein the first text image retains the presentation attributes of the first text in the first image, and the presentation attributes include at least one of color, texture, font and lighting; determine the stroke geometry of the second text; render the stroke geometry according to a preset arrangement to obtain the second text image.

[0014] In one embodiment, after obtaining the image content containing the first text in the first image to obtain the first text image; and after rendering the second text according to the stroke geometry of the second text to obtain the second text image, the method further includes:

[0015] Add first explanatory information to a first text image; wherein the first explanatory information is used to indicate that the first text image provides a visual appearance; add second explanatory information to a second text image; wherein the second explanatory information is used to indicate that the second text image provides a glyph structure.

[0016] In one embodiment, the steps of extracting visual appearance features of the first text from the first text image and extracting glyph structure features of the second text from the second text image include:

[0017] The process involves: determining the region where the first text is located in the first image; removing the image content located within that region from the first image to obtain a background image; wherein the image content outside the region in the background image remains consistent with the first image; stitching together the first text image, the second text image, and the background image to obtain a composite visual input; extracting the visual appearance features of the first text from the composite visual input according to the instructions of the first explanatory information; and extracting the glyph structure features of the second text from the composite visual input according to the instructions of the second explanatory information.

[0018] In one embodiment, the steps of extracting visual appearance features of a first text from the composite visual input according to the instructions of the first explanatory information, and extracting glyph structure features of a second text from the composite visual input according to the instructions of the second explanatory information, include:

[0019] The composite visual input is mapped to a preset representation space to obtain a visual representation; the text information containing the first explanatory information and the second explanatory information is encoded to obtain text features; the visual representation is processed across modally using the text features to obtain the visual appearance features of the first text and the glyph structure features of the second text.

[0020] In one embodiment, the step of performing cross-modal processing on the visual representation using text features to obtain the visual appearance features of the first text and the glyph structure features of the second text includes:

[0021] While maintaining the independence of visual representation and text features, cross-attention processing is performed on visual representation and text features to obtain aligned visual representation and aligned text features; the aligned visual representation and aligned text features are concatenated to obtain feature sequence; self-attention processing is performed on feature sequence to obtain visual appearance features of the first text and glyph structure features of the second text.

[0022] In one embodiment, the step of generating a replacement text image by constraining the visual appearance with visual appearance features and constraining the glyph structure with glyph structure features includes:

[0023] The part of the visual representation corresponding to the region in the background image is determined to obtain the representation to be processed; the visual appearance of the representation to be processed is constrained by visual appearance features, and the glyph structure of the representation to be processed is constrained by glyph structure features. The representation to be processed is iteratively processed to obtain the target representation; the target representation is decoded to obtain the replacement text image; wherein, the replacement text image presents the second text.

[0024] On the one hand, this application also provides a text editing device based on scene images, including:

[0025] The acquisition module is used to acquire the first image to be edited, and to determine the first text to be replaced in the first image and the second text to replace the first text;

[0026] The image construction module is used to obtain the image content where the first text is located in the first image to obtain the first text image; and to render the second text according to the stroke geometry of the second text to obtain the second text image.

[0027] The feature extraction module is used to extract the visual appearance features of the first text from the first text image and the glyph structure features of the second text from the second text image.

[0028] The generation module is used to constrain the visual appearance with visual appearance features and constrain the glyph structure with glyph structure features to generate replacement text images.

[0029] The replacement module is used to replace the image content containing the first text in the first image with a replacement text image to obtain the second image.

[0030] On the one hand, this application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to perform the following steps:

[0031] A first image to be edited is obtained, and the first text to be replaced and the second text to replace the first text in the first image are determined; the image content where the first text is located in the first image is obtained to obtain the first text image; and the second text is rendered according to the stroke geometry of the second text to obtain the second text image; the visual appearance features of the first text are extracted from the first text image, and the glyph structure features of the second text are extracted from the second text image; the visual appearance is constrained by the visual appearance features, and the glyph structure is constrained by the glyph structure features to generate the replacement text image; the image content where the first text is located in the first image is replaced by the replacement text image to obtain the second image.

[0032] On the one hand, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, performs the following steps:

[0033] A first image to be edited is obtained, and the first text to be replaced and the second text to replace the first text in the first image are determined; the image content where the first text is located in the first image is obtained to obtain the first text image; and the second text is rendered according to the stroke geometry of the second text to obtain the second text image; the visual appearance features of the first text are extracted from the first text image, and the glyph structure features of the second text are extracted from the second text image; the visual appearance is constrained by the visual appearance features, and the glyph structure is constrained by the glyph structure features to generate the replacement text image; the image content where the first text is located in the first image is replaced by the replacement text image to obtain the second image.

[0034] On the one hand, this application also provides a computer program product, including a computer program that, when executed by a processor, performs the following steps:

[0035] A first image to be edited is obtained, and the first text to be replaced and the second text to replace the first text in the first image are determined; the image content where the first text is located in the first image is obtained to obtain the first text image; and the second text is rendered according to the stroke geometry of the second text to obtain the second text image; the visual appearance features of the first text are extracted from the first text image, and the glyph structure features of the second text are extracted from the second text image; the visual appearance is constrained by the visual appearance features, and the glyph structure is constrained by the glyph structure features to generate the replacement text image; the image content where the first text is located in the first image is replaced by the replacement text image to obtain the second image.

[0036] The aforementioned text editing method, apparatus, computer device, computer-readable storage medium, and computer program product based on scene images obtain a first text image by acquiring the image content containing the first text from a first image. This image content retains the pixels of the first text itself, including its color, texture, font, and lighting conditions. Therefore, the visual appearance features extracted from the first text image directly originate from the first text itself, rather than from speculation about the surrounding scene. Furthermore, constraining the visual appearance of the generated content with these visual appearance features ensures that the visual appearance of the replacement text image matches the visual appearance of the first text. Simultaneously, constraining the glyph structure of the generated content with the glyph structure features extracted from the second text image ensures that the text content presented in the replacement text image is the second text. These two constraints work together to ensure that in the replaced second image, the text content at the location of the first text has been replaced with the second text, while its visual appearance remains consistent with the first text before replacement, thereby improving the consistency of the visual appearance before and after text replacement and optimizing image processing effects. Attached Figure Description

[0037] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments of this application or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0038] Figure 1 This is an application environment diagram of a text editing method based on scene images in one embodiment;

[0039] Figure 2 This is a flowchart illustrating a text editing method based on scene images in one embodiment;

[0040] Figure 3This is a flowchart illustrating the steps of obtaining the first text image and the second text image in one embodiment;

[0041] Figure 4 This is a flowchart illustrating the steps of adding explanatory information in one embodiment;

[0042] Figure 5 This is a flowchart illustrating the steps for extracting visual appearance features and character structure features in one embodiment;

[0043] Figure 6 This is a flowchart illustrating the steps for extracting visual appearance features and character structure features in another embodiment;

[0044] Figure 7 This is a flowchart illustrating the cross-modal processing steps in one embodiment;

[0045] Figure 8 This is a flowchart illustrating the steps of generating a replacement text image in one embodiment;

[0046] Figure 9 This is a structural block diagram of a text editing device based on a scene image in one embodiment;

[0047] Figure 10 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0048] To make the objectives, technical solutions, and beneficial effects of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0049] Text in a scene image has two identities: semantic symbols, representing the words or phrases they convey; and visual pixels, representing the text's color, texture, font, stroke thickness, and the lighting and shadows it casts. Replacing text in a scene image aims to change the semantic symbol identity while preserving the visual pixel identity.

[0050] The relevant technology transforms this process into an image inpainting task, which includes an occlusion step. This involves removing the entire image content within the area containing the text to be replaced using a mask. This step removes semantic symbols along with the data carrying the visual pixels. Afterward, the only reference objects for regenerating the text content are the surrounding imagery, such as the background color of a sign or the texture of a wall. Therefore, the color, texture, and font of the newly generated text are inferred from the surrounding imagery: if the original text has unique outlines, gradients, or artistic fonts not present in the surrounding imagery, these appearances cannot be reproduced in the inference; even if the colors are similar, the texture of the strokes and the direction of lighting can easily differ from the original. Thus, although the text content has changed in the replaced image, its visual appearance is inconsistent with the original text.

[0051] Based on this, embodiments of this application provide a text editing method based on scene images. This method no longer discards the entire image content of the area containing the text to be replaced, but instead extracts and retains the image content separately as a reference object for providing visual appearance; simultaneously, it renders the text to be replaced as an image reflecting its stroke geometry structure, serving as a reference object for providing glyph structure; then, it extracts visual appearance features and glyph structure features from both, and uses these two features to jointly constrain the generated content. Since the visual appearance features are taken from the text itself before replacement, the visual appearance of the generated content matches the text before replacement, thereby improving the consistency of the visual appearance before and after text replacement.

[0052] The text editing method based on scene images provided in this application can be applied to, for example... Figure 1In the application environment shown, terminal 102 communicates with server 104 via a network. A data storage system can store the data that server 104 needs to process. The data storage system can be integrated onto server 104 or placed on a cloud or other network server. Terminal 102 can acquire a first image to be edited, receive the user's instruction on the first text to be replaced in the first image, and the input second text, then send the first image, the instruction information for the first text, and the second text to server 104. Server 104 acquires the image content containing the first text in the first image, obtains a first text image, and renders the second text according to the stroke geometry of the second text, obtaining a second text image. Server 104 extracts the visual appearance features of the first text from the first text image and the glyph structure features of the second text from the second text image, constrains the visual appearance with the visual appearance features, and constrains the glyph structure with the glyph structure features, generating a replacement text image. Server 104 replaces the image content containing the first text in the first image with the replacement text image, obtaining a second image, and returns the second image to terminal 102 for display. In other embodiments, all of the above steps may be performed by terminal 102 or by server 104, and no limitation is made here.

[0053] The above methods can be applied to various scenarios. For example, in cross-language content localization, text in one language on promotional posters or product packaging can be replaced with text in another language, while the background texture, font style, and lighting effects of the poster remain unchanged. In augmented reality text translation, text on road signs in real-world scenes can be replaced with text in the user's language, while the reflections and wear marks on the road signs remain unchanged. In image sample amplification, text content can be replaced in batches on existing scene images to obtain samples with complex backgrounds and diverse lighting, which can be used to train text detection and character recognition models. In information desensitization, sensitive text such as ID numbers and license plate numbers in an image can be replaced with characters that do not point to real objects, while the original visual distribution characteristics of the image are not destroyed.

[0054] The terminal 102 can be, but is not limited to, various personal computers, laptops, smartphones, tablets, IoT devices, and portable wearable devices. IoT devices can include smart speakers, smart TVs, smart air conditioners, smart in-vehicle systems, and projection devices. Portable wearable devices can include smartwatches, smart bracelets, and head-mounted displays. Head-mounted displays can be virtual reality (VR) devices, augmented reality (AR) devices, and smart glasses. The server 104 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services.

[0055] In one exemplary embodiment, such as Figure 2 As shown, a text editing method based on scene images is provided, which can be applied to... Figure 1 Taking the server in the example, the explanation includes the following steps 201 to 205. Wherein:

[0056] Step 201: Obtain the first image to be edited, and determine the first text to be replaced in the first image and the second text to replace the first text.

[0057] The first image refers to the image to be edited that contains text, also known as a scene image. For example, an image containing road signs, signs, or product packaging obtained by a camera device, or a poster image containing promotional slogans obtained by design software.

[0058] The first text refers to the section of text in the first image that needs to be replaced, such as a word that was originally on a road sign.

[0059] The second text refers to the text used to replace the first text, which is given in the form of a string, such as a word entered by the user.

[0060] Optionally, the first image can be received by the server from the terminal, or it can be read by the server from a local image library or data storage system. The first text can be indicated by the user on the terminal through methods such as selecting by box or clicking, and the server determines the text to be replaced in the first image accordingly; alternatively, the server can first perform text detection on the first image to obtain the location of each segment of text in the first image, and then determine the first text based on the user's selection or preset filtering conditions.

[0061] The second text can be entered by the user on the terminal, or it can be generated by the server after processing the first text according to preset rules such as translation and desensitization.

[0062] It should be noted that the number of characters in the second text may be the same as or different from the number of characters in the first text; the language of the second text may be the same as or different from the language of the first text.

[0063] Step 202: Obtain the image content containing the first text in the first image to obtain the first text image; and render the second text according to the stroke geometry of the second text to obtain the second text image.

[0064] Here, the first text image refers to the portion of the image content that contains the first text, taken from the first image. Since this portion of the image content is taken from the first image itself, it retains the original pixels of the first text, and the color, texture, font, and lighting conditions of the first text at its location are all contained within these pixels.

[0065] The second text image refers to the image obtained after rendering the second text, which presents the stroke geometry of the second text, that is, the stroke direction of each character in the second text and the positional relationship between the strokes.

[0066] The visual appearance of the first text does not exist independently in the form of words or numbers, but is distributed among the pixels in the area where the first text is located. By extracting the image content of this area separately to obtain the first text image, these pixels are preserved, thus allowing the visual appearance features of the first text to be extracted.

[0067] The second text itself is a string containing the encoding information of each character, but not the shape of each character when written. Rendering the second text yields a second text image, where the stroke shapes are presented in pixel form, allowing for the extraction of glyph structural features.

[0068] Optionally, when obtaining the image content containing the first text in the first image, the area occupied by the first text in the first image can be determined first, and then the image content within that area can be extracted. This area can be a shape that fits the outline of the first text, or it can be a regular shape that contains the first text. When rendering the second text, an image processing tool can be called to draw each character of the second text onto an image according to a preset font size and spacing; the preset font size and spacing can be fixed values, or they can be determined based on the number of characters in the second text.

[0069] Step 203: Extract the visual appearance features of the first text from the first text image, and extract the glyph structure features of the second text from the second text image.

[0070] Among them, visual appearance features refer to the features used to characterize the appearance of the first text in the first image. They can characterize the appearance information of the first text, such as color, texture, font, and lighting.

[0071] Character structure features refer to the features used to characterize the stroke geometry of a second text. They can characterize structural information such as the stroke direction of each character in the second text and the positional relationship between strokes.

[0072] It should be noted that the first text image and the second text image are both images, and they are similar in form. In order to extract appearance information from the first text image and structural information from the second text image, information indicating their purpose can be configured for each image. This processing method will be further explained in the embodiments below.

[0073] Optionally, the first and second text images can be input into a pre-trained feature extraction network, which will output visual appearance features and character structure features respectively; alternatively, the two images can be mapped to the same representation space first, and then two features can be obtained in the representation space respectively.

[0074] Step 204: Constrain the visual appearance with visual appearance features and constrain the glyph structure with glyph structure features to generate a replacement text image.

[0075] The replacement text image is the generated image used to replace the content of the image containing the first text in the first image.

[0076] During the generation process, visual appearance features and glyph structure features are considered as conditions. Visual appearance features ensure that the color, texture, font, and lighting of the generated content match the appearance represented by those features. Since these visual appearance features are derived from the first text itself, the appearance of the generated content matches the appearance of the first text. Glyph structure features ensure that the stroke direction and positional relationships between the strokes of the generated content match the structure represented by those features. Since these glyph structure features are derived from the rendering result of the second text, the text content presented in the generated content is the second text. Both conditions work together in the same generation process, resulting in a replacement text image that presents the second text while maintaining an appearance consistent with the first text.

[0077] Step 205: Replace the image content containing the first text in the first image with a replacement text image to obtain the second image.

[0078] The second image refers to the image obtained after the text replacement is completed.

[0079] Optionally, the portion of the replacement text image corresponding to the location of the first text can be overlaid onto the corresponding location of the first image; alternatively, the size of the replacement text image can be adjusted to match the area where the first text is located before overlaying.

[0080] During the replacement process, the portion of the first image outside the content containing the first text is not replaced, so the second image remains consistent with the first image in that portion.

[0081] In the aforementioned text editing method based on scene images, a first text image is obtained by acquiring the image content containing the first text in a first image. This preserves the original pixels of the first text during processing, allowing the extracted visual appearance features to originate directly from the first text itself, rather than from speculation about the surrounding image. A second text image is obtained by rendering the second text, allowing the stroke shapes of the second text to be presented in pixel form. The extracted character structure features can then characterize the stroke direction of the second text. By constraining the visual appearance and character structure of the generated content with these two features respectively, the resulting replacement text image not only presents the content of the second text but also matches the appearance of the first text. After replacing the image content containing the first text in the first image with this replacement text image, the text content in the original position in the second image has been replaced with the second text, while its color, texture, font, and lighting remain consistent with the original first text. Therefore, the consistency of the visual appearance before and after text replacement can be improved.

[0082] In one exemplary embodiment, such as Figure 3 As shown, step 202 includes steps 301 to 304. Wherein:

[0083] Step 301: Determine the outer graph of the region where the first text is located in the first image.

[0084] In this context, a circumscribed figure refers to a regular shape that can completely encompass the area containing the first text, such as a circumscribed rectangle, a circumscribed parallelogram, or a circumscribed circle. Taking a circumscribed rectangle as an example, it can be the rectangle with the smallest area that can contain the region.

[0085] The shape of the area containing the first text usually conforms to the character outline, with irregular and winding edges. Taking its circumscribed shape yields a regularly shaped area, which facilitates the extraction of image content within this area from the first image and also facilitates the uniform processing of the content within this area.

[0086] Step 302: Obtain the image content located within the bounded graphic area of ​​the first image to obtain the first text image.

[0087] The first text image retains the rendering attributes of the first text in the first image, and the rendering attributes include at least one of color, texture, font and lighting.

[0088] Among them, presentation attributes refer to the various appearance attributes that the first text possesses when it is presented in the first image.

[0089] Color refers to the color of the strokes of the first text, which can be a single color or a gradient; texture refers to the surface texture of the strokes of the first text, such as metallic texture, wear marks, or fabric texture; font refers to the writing style of each character in the first text, including the thickness of the strokes, the presence or absence of serifs, and whether they are slanted; lighting refers to the lighting conditions of the location of the first text, including the location of highlights, the direction and intensity of shadows, etc.

[0090] Since the image content is obtained directly from the first image, the pixels are the same as the pixels of the first text in the first image. All the above-mentioned presentation attributes are retained along with the pixels, without the need for separate extraction or description.

[0091] Step 303: Determine the stroke geometry of the second text.

[0092] Among them, stroke geometry refers to which strokes constitute each character of the second text, how each stroke moves, and the relative positions between the strokes.

[0093] For example, the geometric structure of Chinese characters can be represented by the direction of strokes such as horizontal, vertical, left-falling, and right-falling strokes and their relative positions; for Latin letters, it can be represented by the direction of straight line segments and arc segments and their relative positions.

[0094] Optionally, the outline of each character in the second text can be found based on a preset character library, thereby determining its stroke geometry.

[0095] Step 304: Render the geometric structure of the strokes according to the preset arrangement to obtain the second text image.

[0096] The preset layout refers to how the characters of the second text are arranged on the image, such as in a single line, multiple lines, or along a specified baseline. During rendering, each stroke is drawn as a pixel on the image based on the stroke geometry, and the position of each character on the image is determined according to the preset layout, thus obtaining the second text image.

[0097] Optionally, during rendering, the character regions of the second text can be differentiated in brightness from their background, for example, by drawing light-colored characters on a dark background or dark-colored characters on a light background. The more pronounced the brightness difference between the character regions and the background, the clearer the stroke boundaries will be in the image, and the more accurately the glyph structure features extracted from the second text image will reflect the direction of the strokes.

[0098] In this embodiment, the first text image is obtained verbatim from the image content within the bounded graphic area of ​​the first image, preserving the color, texture, font, and lighting attributes of the first text, thus providing a basis for the appearance of the first text itself. The second text image is obtained by rendering the stroke geometry of the second text, providing a structural basis at the stroke level rather than a category basis at the character level. Since strokes are the basic building blocks common to all languages, as long as the second text can be rendered, its stroke geometry can be represented. Therefore, the second text can contain rare characters or belong to a language different from the first text. This method is not limited by a preset character set in terms of character types.

[0099] As mentioned above, the first text image and the second text image are both images and belong to the same category in form, so they need to be treated differently when extracting features. The following embodiments of this application will describe this processing method.

[0100] In one exemplary embodiment, such as Figure 4 As shown, after step 202, the method further includes steps 401 to 402. Wherein:

[0101] Step 401: Add first explanatory information to the first text image; wherein the first explanatory information is used to instruct the first text image to provide a visual appearance.

[0102] Step 402: Add second explanatory information to the second text image; wherein the second explanatory information is used to instruct the second text image to provide glyph structure.

[0103] The explanatory information refers to the information used to indicate the purpose of an image in this replacement. It can be expressed in text form and participates in the feature extraction process along with the image.

[0104] Both the first and second text images are images, and their data formats are identical. However, in this replacement, we need to obtain appearance information from the first text image and structural information from the second text image; these two purposes are different. Furthermore, the first text image contains not only the appearance of the first text but also the glyphs of the first text itself. If processed without differentiation, the content obtained from the first text image may also include the glyphs of the first text, causing the generated content to still present the first text instead of the second text.

[0105] Adding descriptive information indicating the purpose of each image can differentiate them during feature extraction: information about appearance is obtained from the first text image, while information about structure is obtained from the second text image.

[0106] Optionally, a description template can be pre-set, which contains a label that corresponds one-to-one with each image; the first description information is associated with the label corresponding to the first text image, and the second description information is associated with the label corresponding to the second text image, thus completing the addition.

[0107] The first explanatory information can be described as follows: the image is taken from the area of ​​the text to be replaced in the image to be edited, and is used to provide a visual style; the second explanatory information can be described as follows: the image is a glyph template of the text to be written, and is used to provide the glyph structure.

[0108] In this embodiment, by adding explanatory information indicating the purpose of the first text image and the second text image respectively, two images that are similar in form can be treated differently when extracting features, thereby obtaining information on appearance and structure respectively, and avoiding the use of the glyphs of the first text contained in the first text image.

[0109] The preceding text explained how to add descriptive information indicating the purpose of two images. Building upon this, the following text further explains the process of extracting two features based on this descriptive information.

[0110] In one exemplary embodiment, such as Figure 5 As shown, step 203 includes steps 501 to 504. Wherein:

[0111] Step 501: Determine the region where the first text is located in the first image.

[0112] Step 502: Remove the image content located in the area of ​​the first image to obtain the background image.

[0113] In this context, the image content outside this area in the background image remains consistent with the first image.

[0114] Here, the background image is the first image after the image content within the area containing the first text has been removed. During removal, the original image content within that area is no longer displayed; for example, the pixels within that area are set to preset values.

[0115] It should be noted that the area determined here is the same area used in step 301 when determining the external graphic. This area serves two purposes in this method: extracting image content and removing image content. Extracting the image content within this area yields the first text image, thus preserving the pixels of the first text and providing a reference for its appearance. Removing the image content within this area yields the background image, freeing up space for new text content to be generated. These two aspects are complementary; without extraction, there is no reference for the appearance; without removal, the first text content remains on the screen, leaving no place for new text content.

[0116] Step 503: Combine the first text image, the second text image, and the background image to obtain a composite visual input.

[0117] Among them, composite visual input refers to the visual data obtained by merging the above three images and participating in feature extraction as a whole.

[0118] The background image provides the image to be edited, the first text image provides the appearance, and the second text image provides the structure. Replacement cannot be completed without any one of these three; therefore, all three need to be processed together in the same process. By stitching the three images together, they can be received by the same process without needing to set up separate processing paths for the first and second text images. This avoids situations where the results obtained from independent paths do not match the data in the main path in terms of form.

[0119] Optionally, the three images can be stitched together in the channel direction or in the spatial direction; before stitching, the dimensions of the first and second text images can be processed accordingly to make them fit the background image.

[0120] Step 504: According to the instructions of the first explanatory information, extract the visual appearance features of the first text from the composite visual input, and according to the instructions of the second explanatory information, extract the glyph structure features of the second text from the composite visual input.

[0121] During extraction, information of the relevant aspect is obtained from the composite visual input according to the intended use indicated by the explanatory information.

[0122] The first explanatory information indicates that the first text image provides a visual appearance, and thus the data derived from the first text image in the composite visual input is information about appearance; the second explanatory information indicates that the second text image provides a glyph structure, and thus the data derived from the second text image in the composite visual input is information about structure.

[0123] In this embodiment, the same area is used for both extracting the image content within the area to preserve the appearance basis of the first text and removing the image content within the area to make room for the generation of new content. These two uses complement each other. By stitching the two reference images and the background image into a composite visual input, all three are processed together by the same process. There is no need to set up separate processing paths to provide appearance and structural basis, which can avoid the situation where the results obtained by the independent path do not match the data of the main path in form.

[0124] The preceding text explained how to stitch two reference images together with a background image to form a composite visual input, and how to extract two features from it according to the instructions. Building upon the above, the following section describes a specific implementation of this extraction process.

[0125] In one exemplary embodiment, such as Figure 6 As shown, step 504 includes steps 601 to 603. Wherein:

[0126] Step 601: Map the composite visual input to a preset representation space to obtain a visual representation.

[0127] The representation space is a space used to carry data features. Data in this space exists in the form of features, and its dimension is usually lower than that of the original pixel data. During mapping, the composite visual input in pixel form is transformed into data in the form of features in this representation space, and the resulting features are the visual representations.

[0128] Visual representation refers to the features corresponding to a composite visual input in a representation space after mapping. Pixel-level data has high dimensionality and a lot of redundancy, and processing it directly requires a large amount of computation. After mapping it to the representation space, the content in the image is contained in a more compact form, and feature extraction and content generation can be performed in this space.

[0129] It should be noted that since the background image, the first text image, and the second text image have been stitched together as a whole, they all enter the same representation space through the same mapping. Therefore, the features corresponding to the three are in the same representation space and are mutually compatible in form.

[0130] Step 602: Encode the text information containing the first explanatory information and the second explanatory information to obtain text features.

[0131] The text information refers to a segment of text that includes the two explanatory information items mentioned above. During encoding, this segment of text is converted into data in feature form, and the resulting features are the text features.

[0132] It should be noted that the explanatory information itself is in text form and cannot directly interact with the visual representation in feature form. After it is encoded as text features, the text features and visual representation can participate in the same processing, and the purpose indicated by the explanatory information can then be applied to the visual representation.

[0133] Optionally, in addition to the two explanatory information items mentioned above, the text information may also include the second text itself. Encoding the second text itself can provide additional semantic and syntactic information about the text segment for generating the replacement text image, contributing to the completeness and readability of the generated content at the text level.

[0134] Step 603: Perform cross-modal processing on the visual representation using text features to obtain the visual appearance features of the first text and the glyph structure features of the second text.

[0135] Textual features originate from descriptive information, which carries the purpose of each image; visual representations originate from composite visual input, which carries the content of each image. In cross-modal processing, textual features influence visual representations, that is, based on the purpose of each image, relevant information is extracted from the content of each image.

[0136] Thus, the visual representation derived from the first text image, under the influence of the text features corresponding to the first explanatory information, acquires information about appearance, resulting in visual appearance features; the visual representation derived from the second text image, under the influence of the text features corresponding to the second explanatory information, acquires information about structure, resulting in glyph structure features.

[0137] In this embodiment, the composite visual input enters the same representation space through the same mapping, so that the features corresponding to the three images are mutually compatible in form. There is no need to set up separate feature extraction paths for the first text image and the second text image, thus avoiding the situation where the features obtained by the independent path do not match the data in the representation space in form. The explanatory information, after being encoded, participates in the calculation together with the visual representation in the same processing process, so that the purpose indicated by the explanatory information can actually be applied to the feature extraction, thereby obtaining visual appearance features and character structure features respectively.

[0138] The above section explained the role of cross-modal processing. Building upon that, the following section further describes a specific implementation method for cross-modal processing.

[0139] In one exemplary embodiment, such as Figure 7 As shown, step 603 includes steps 701 to 703. Wherein:

[0140] Step 701: While keeping the visual representation and text features independent of each other, perform cross-attention processing on the visual representation and text features to obtain aligned visual representation and aligned text features.

[0141] In the cross-attention process, visual representations and textual features are used to query and exchange information with each other.

[0142] In this process, visual representations and textual features remain in separate groups, and their data are not merged with each other, thus maintaining their independence. After this processing, a connection is established between the purpose indicated by the explanatory information and the content of the corresponding image, that is, alignment is completed.

[0143] Visual representations carry detailed information such as texture and color in an image, while textual features carry semantic information about usage. These two types of information are fundamentally different. If visual representations and textual features are merged before the information is aligned, the details of the image can easily be altered due to interference from semantic information. Therefore, in this step, keeping them independent and exchanging information only through cross-attention allows for the establishment of a connection while preventing the other from changing the information.

[0144] Step 702: Concatenate the aligned visual representation with the aligned text features to obtain the feature sequence.

[0145] Among them, the feature sequence is the feature obtained by merging visual representation and text features, and participates in self-attention processing as a whole.

[0146] Step 703: Perform self-attention processing on the feature sequence to obtain the visual appearance features of the first text and the glyph structure features of the second text.

[0147] Here, during self-attention processing, within the feature sequence, the parts query and merge with each other.

[0148] After step 701 is completed, the association between the purpose and the content of the image has been established. At this point, by placing the visual representation and text features in the same set of features for in-depth integration, the resulting features can be coordinated as a whole, thereby making the generated content coherent and natural on the image.

[0149] In this embodiment, cross-modal processing includes cross-attention processing and self-attention processing. In cross-attention processing, visual representations and text features remain independent of each other and only exchange information, ensuring that image details are not interfered with by semantic information. In self-attention processing, visual representations and text features are merged and deeply integrated, making the various features coordinated as a whole. Thus, it ensures both the accurate establishment of the association between purpose and image content and the coherence of the generated content on the screen.

[0150] The extraction process of visual appearance features and character structure features has been explained above. Building upon this, the process of generating replacement text images using these two features will be further explained below.

[0151] In one exemplary embodiment, such as Figure 8 As shown, step 204 includes steps 801 to 803. Wherein:

[0152] Step 801: Determine the portion of the visual representation that corresponds to the region within the background image to obtain the representation to be processed.

[0153] The representation to be processed refers to the part of the visual representation that corresponds to the area in the background image where the image content has been removed, that is, the part that needs to be used to generate new content.

[0154] The content of the background image outside this area does not need to be changed, thus limiting the processing to this part. The rest of the image will not be changed during processing, thereby ensuring that the content outside this area in the second image remains consistent with the first image.

[0155] Step 802: Constrain the visual appearance of the representation to be processed with visual appearance features, and constrain the glyph structure of the representation to be processed with glyph structure features, and perform iterative processing on the representation to be processed to obtain the target representation.

[0156] During iterative processing, the representation to be processed is repeatedly processed in multiple rounds. Each round of processing changes it one step in the direction that meets the constraints. After several rounds, a result that meets the constraints is obtained, which is the target representation.

[0157] In each round of processing, both visual appearance features and glyph structure features are used as constraints. Visual appearance features cause the results of this round to change in terms of color, texture, font, and lighting in a direction consistent with the first text, while glyph structure features cause the results of this round to change in terms of stroke direction and stroke position relationships in a direction consistent with the second text.

[0158] Step 803: Decode the target representation to obtain the replacement text image.

[0159] The image containing the replacement text contains the second text.

[0160] Here, during decoding, the data in the feature form in the representation space is restored to the image in the form of pixels. This process is the reverse of the mapping process in step 601.

[0161] It should be noted that the text content presented in the replacement text image is the second text, not the first text. This is because, in the iterative processing, the glyph structure features constrain the stroke direction and positional relationship, and this feature is taken from the second text image; meanwhile, the visual appearance features constrain color, texture, font, and lighting, but do not include stroke direction. These two constraints operate independently, therefore the glyphs of the first text are not included in the generated content along with their appearance.

[0162] In this embodiment, by limiting the scope of the iterative processing to the part of the representation corresponding to the area of ​​the removed content, the image outside the area is not changed during the processing, so that the part outside the area in the second image is consistent with the first image, and the replaced image does not leave any trace of modification; by constraining the visual appearance and the glyph structure with two features respectively, the resulting replaced text image not only presents the content of the second text, but its appearance also matches the first text.

[0163] The foregoing has described each step of the method provided in the embodiments of this application and its specific implementation. For ease of understanding, the complete implementation process of the method provided in the embodiments of this application will be described below with reference to a specific model structure.

[0164] In one exemplary embodiment, a text editing method based on scene images is provided, which can be executed by a multimodal generative model. The structure of the model will be described first, followed by an explanation of how the data input to the model is constructed, how the model processes the input data, and how the model is trained.

[0165] This multimodal generative model can adopt a multimodal diffusion transformer (MMDiT) structure for image inpainting tasks, such as using the FLUX-Fill model as its basic structure. This structure processes image editing as a conditional rectified flow (CRF) process in the latent space (i.e., the representation space mentioned above), and performs joint reasoning on image content, spatial structure, and textual semantics under multimodal conditions.

[0166] In this structure, the image encoding and decoding part can employ an encoder and decoder from a Variational Auto-Encoder (VAE). The encoder maps the pixel-based image to the latent space, obtaining latent variables, i.e., the visual representation mentioned earlier; the decoder then restores the latent variables back to the pixel-based image. In addition to the image, a mask indicating the region where the text to be replaced is located can also be converted into an embedding vector, providing spatial prior information to the model, enabling the model to perceive the location of the region to be modified during processing.

[0167] Complementing the image encoding and decoding part is the text encoding part, which can be composed of two text encoders in parallel. One text encoder can be a text-to-text transfer transformer (T5) encoder, used to process complete natural language prompts, capturing high-level semantic intent and contextual information. Its output embedding vector is injected into the model through a cross-attention layer to guide the overall content generation and scene-level consistency. The other text encoder can be a contrastive language-image pre-training (CLIP) encoder, used to encode concise text descriptions that emphasize visual alignment, thereby enhancing the alignment between vision and language and stabilizing the correspondence between the generated content and the screen context.

[0168] After the above encoding, both visual and textual data enter the backbone network. This backbone network consists of alternating dual-stream Transformer blocks and single-stream Transformer blocks. In the dual-stream Transformer block, visual tokens and text tokens first undergo self-attention processing in their respective independent streams, and then information exchange between modalities is achieved through a cross-attention mechanism. This allows the model to maintain the feature structure of each modality while aligning textual semantics with spatially based visual representations. In the single-stream Transformer block, visual tokens and text tokens are concatenated into a unified token sequence for deep joint processing.

[0169] Furthermore, during model training, the parameters of the image encoder, image decoder, and two text encoders can be frozen, and only the parameters of the multimodal diffusion Transformer backbone network can be updated. This approach preserves the original visual and linguistic representation capabilities of the pre-trained model, focusing parameter updates on how to achieve appearance-consistent reconstruction using reference images.

[0170] The structure used in this model has been explained above. Based on this, the construction method of the data input to the model will be explained below. For ease of explanation, let the first image to be edited be I, and the binary mask indicating the region where the first text is located be M.

[0171] First, a background image Im is constructed. The first image I is then processed according to a mask M, causing the image content within the area containing the first text to be removed, while the image content outside this area is fully preserved. That is:

[0172] I m = I ⊙ (1-M).

[0173] Here, ⊙ represents element-wise multiplication.

[0174] Next, the first text image Is is constructed, which in this embodiment can also be referred to as a visual style cue. During construction, the maximum bounding rectangle of the area indicated by the mask M is first calculated, and then an image block within the range of this rectangle is directly cropped from the first image I and used as Is. This image block naturally contains specific appearance information of the area, including color, texture, font features, and local lighting conditions, thus providing the model with an intuitive visual reference to preserve the original appearance.

[0175] In contrast, the second text image Ig in this embodiment can also be referred to as a visual glyph cue. During construction, an image processing library (such as the Python image processing library Pillow) is invoked to render the second text as a single-line high-contrast image, such as an image of light-colored characters on a dark background. This high-contrast image representation can accurately preserve the geometric structure at the stroke level. This design not only provides the model with clear glyph structure guidance but also reduces the difficulty for the model to learn cross-linguistic and complex character structures from scratch.

[0176] After the above three components are constructed, they are spliced ​​together to obtain the composite visual input Iinput:

[0177] Iinput = Concat(Ig, Is, Im).

[0178] Here, Concat represents the stitching operation, which can be performed along the channel direction. This composite visual input is then fed into a frozen image encoder to extract latent variables, and then enters a multimodal diffusion Transformer backbone network for joint processing.

[0179] In addition to visual input, structured text instructions also need to be constructed, namely, the text information containing the first and second explanatory information mentioned above. This text instruction is not a simple natural language description, but rather a templated text that assigns a purpose to each image, containing identifiers that correspond one-to-one with each image. Its content can be expressed as follows: the image corresponding to identifier IMAGE1 is the actual scene image to be edited; the image corresponding to identifier IMAGE2 is a template image rendered with light-colored characters on a dark background, containing the text to be written; the image corresponding to identifier IMAGE3 is a cropped area taken from the original scene, used to provide visual style reference.

[0180] Thus, this text instruction binds the concatenated composite visual input to a specific editing intent: the identifier IMAGE1 corresponds to the background image Im, which informs the model that this part of the data is the real scene background that needs to be repaired; the identifier IMAGE2 corresponds to the second text image Ig, which clearly indicates that this part of the data is the geometric structure template for the text to be written, and the model needs to follow its stroke direction to generate new content; the identifier IMAGE3 corresponds to the first text image Is, which indicates that this part of the data is a visual style reference, and the model needs to extract appearance information such as color, texture, and font from it and apply it to the generated area.

[0181] In addition to the text instructions mentioned above, auxiliary text conditions can also be constructed. For example, the string of the second text itself can be input into the first text encoder mentioned above to provide the model with information on the high-level semantics and syntax of the text; or, the concise text describing the internal style of the target text region can be encoded by the second text encoder mentioned above to help the model understand and retain the style of the region.

[0182] The construction method of the input data has been explained above. Based on this, the process of the model processing this input data is described below, which can be divided into four stages: input encoding, cross-modal interaction, joint processing, and decoding output.

[0183] During the input encoding stage, the composite visual input, which is formed by concatenating the background image Im, the second text image Ig, and the first text image Is, is fed into the frozen image encoder and mapped into visual markers in the latent space; the structured text instructions are fed into the frozen text encoder to obtain text markers.

[0184] After encoding is complete, the cross-modal interaction phase begins. Visual and textual tags enter the dual-stream Transformer block, where visual and textual features remain independent and exchange and align information through a cross-attention mechanism, ensuring that the purpose assigned in the textual instructions can accurately apply to its corresponding visual features.

[0185] After the information is aligned, the joint processing stage begins. The aligned multimodal tags are concatenated and fed into a single-stream Transformer block, where deep feature fusion and joint inference are performed through a self-attention mechanism. During this process, latent variables corresponding to the region to be edited are iteratively denoised.

[0186] Finally, there is the decoding and output stage. The latent variables after iterative processing enter the frozen image decoder and are restored to pixel-level images; then, through a cropping operation, the generated target text regions are stitched back to their corresponding positions in the original image, thus obtaining the edited second image.

[0187] The processing procedure of this model has been explained above. This model can be trained. To enable the model to achieve consistent text editing with limited paired data and to overcome the problem of the model excessively copying the original screen content, a two-stage training strategy can be adopted, with corresponding datasets constructed for each stage. The following section first explains how the datasets are constructed, and then describes the training process for each of the two stages.

[0188] Training data can be divided into two categories: large-scale self-supervised data and high-quality paired data. The large-scale self-supervised pre-training dataset is used to enable the model to possess basic multilingual scene text rendering and image inpainting capabilities. This part of the data can use the AnyWord-3M dataset as the core large-scale benchmark dataset, whose training set contains 1.6 million Chinese images, 1.39 million English images, and 10,000 images covering Japanese, Korean, Arabic, Bengali, and Hindi. To enhance multilingual generalization ability, public datasets such as ICDAR-19, RusTitW, ThaiOCRBench, and Swahili-STR can also be aggregated to construct a multilingual dataset, with 80% of it randomly allocated for training. Multilingual data can be used in combination during training without employing specific language-based scheduling strategies.

[0189] In contrast, high-quality paired datasets are used to teach models how to preserve the original appearance. Related text recognition datasets mask the target region, resulting in the loss of the original text's appearance information, necessitating the construction of separate paired datasets. During construction, an instruction-based image editing model can be used to generate modified images according to explicit editing instructions. Each pair of data contains one original image and one corresponding edited image, with only the target text region modified, while the remaining regions remain strictly unchanged. Subsequent manual selection criteria include: non-target regions remaining unchanged, semantically accurate generated text content, and high consistency between the edited text and the original text in style, color, and texture. After selection, 4000 high-quality image pairs can be obtained.

[0190] Based on the completion of the two types of datasets mentioned above, training can be carried out in two stages. The first stage is the self-supervised pre-training stage, whose training objective is to enable the model to have basic text repair capabilities while avoiding overfitting to specific text styles. Specifically, training can be performed on the aforementioned large-scale self-supervised dataset for one training epoch. In this stage, the input includes multimodal visual and textual conditions, and the optimization objective follows the standard conditional corrected flow loss function. Let z0 be the clean latent variable extracted by the image encoder, and z1 be the sampled standard Gaussian noise. At time step t, the model constructs a noisy latent variable zt through corrected flow interpolation. The training objective is to enable the model to predict the velocity field connecting zt and z0, and its loss function can be defined as:

[0191] .

[0192] in, θ represents the velocity field predicted by the model, E represents the expectation, and c represents the multimodal conditional input, which includes the aforementioned visual labels, text embeddings, and spatial mask information.

[0193] After completing the self-supervised pre-training, the model enters the second stage, the cooling-off training stage. The problem to be solved in this stage is that the first text image contains the first text itself. If it is directly used as a style reference for training, the model is prone to degenerate optimization during the optimization process. That is, the features of the region where the first text is located excessively dominate the generation process, causing the model to not only copy the original style, but also copy the original glyph content, thus failing to generate new text.

[0194] To restrict the learning signal to local text transformations and encourage the model to completely decouple style preservation from content regeneration during optimization, this stage employs a target-region-oriented training objective. Specifically, let z0(src) and z0(tgt) be the latent variable representations of the source image and its corresponding edited image, respectively, and construct a time-dependent interpolated latent variable z. t :

[0195] z t = (1-σ t )×z0 src + σ t ×z0 tgt .

[0196] Where, σ t∈[0,1] represents the temporal interpolation coefficients. As can be seen from this formula, the starting point of the interpolation latent variable is no longer random noise, but rather the latent variable of the source image; its endpoint is the latent variable of the edited image. In this stage, the original text region is used as meta-information input through style prompts, and the model is trained for 10 training epochs on the aforementioned 4000 pairs of paired data. The training objective is to enable the model to predict the velocity field connecting the source latent variable and the target latent variable; the loss function minimized is designed specifically for the aforementioned interpolation process.

[0197] In this design, the model no longer learns to draw a complete image from random noise, but rather the range of changes in the latent variables of the source image that points to the latent variables of the target image. Since the only difference between the paired data is the target text region, this range of changes precisely corresponds to the change in text content, while the changes in other parts are zero. If the model simply copies the original glyphs from the source image, its predicted changes will be zero, which contradicts the training objective and results in a penalty. Thus, degenerate optimization is avoided, and style preservation and content regeneration are separated during the optimization process.

[0198] After the two-stage training process described above, even with a small amount of paired training data, the model can still learn how to align the generated text with the style of the original image. Text accuracy can be measured using sentence accuracy and normalized edit distance (NED); visual fidelity can be measured using Fréchet inception distance (FID) and learned perceptual image patch similarity (LPIPS).

[0199] In this embodiment, visual style cues are obtained by directly cropping from the target text region of the image to be edited. The model can directly reference the color, texture, and font of the original text at the latent variable level without introducing a separate style encoder. Therefore, there is no representation mismatch between the features extracted by the independent encoder and the latent space of the image. As a result, the generated text can retain the color, font, and texture details specific to the original region, rather than borrowing style cues from the surrounding background. By rendering the text to be written as a high-contrast glyph as a visual glyph cue, the model learns the direction at the stroke level rather than the category at the character level. Therefore, it is not limited by the preset character set in terms of character types. It is also applicable to languages ​​with complex and diverse writing systems. Furthermore, it does not produce negative transfer when continuously introducing new languages. The model's generalization ability can be further enhanced by sharing the underlying stroke structure. By using a training objective oriented towards the target region for cool-down training, the model completely decouples style preservation and content regeneration during optimization. Therefore, it can still achieve good results even with a limited amount of paired data.

[0200] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0201] Based on the same inventive concept, this application also provides a scene-image-based text editing apparatus for implementing the above-described scene-image-based text editing method. The solution provided by this apparatus is similar to the implementation described in the above-described method; therefore, the specific limitations in one or more scene-image-based text editing apparatus embodiments provided below can be found in the limitations of the scene-image-based text editing method described above, and will not be repeated here.

[0202] In one exemplary embodiment, such as Figure 9 As shown, a text editing device based on scene images is provided, including: an acquisition module, an image construction module, a feature extraction module, a generation module, and a replacement module, wherein:

[0203] The acquisition module is used to acquire the first image to be edited, and to determine the first text to be replaced in the first image and the second text to replace the first text.

[0204] The image construction module is used to obtain the image content where the first text is located in the first image to obtain the first text image; and to render the second text according to the stroke geometry of the second text to obtain the second text image.

[0205] The feature extraction module is used to extract the visual appearance features of the first text from the first text image and the glyph structure features of the second text from the second text image.

[0206] The generation module is used to constrain the visual appearance with visual appearance features and constrain the glyph structure with glyph structure features to generate replacement text images.

[0207] The replacement module is used to replace the image content containing the first text in the first image with a replacement text image to obtain the second image.

[0208] In one embodiment, the image construction module is further configured to: determine the circumscribed graph of the region where the first text is located in the first image; obtain the image content within the circumscribed graph range in the first image to obtain the first text image; wherein the first text image retains the presentation attributes of the first text in the first image, and the presentation attributes include at least one of color, texture, font and lighting; determine the stroke geometry of the second text; and render the stroke geometry according to a preset arrangement to obtain the second text image.

[0209] In one embodiment, the apparatus further includes a descriptive information adding module for adding first descriptive information to a first text image, wherein the first descriptive information is used to indicate that the first text image provides a visual appearance; and for adding second descriptive information to a second text image, wherein the second descriptive information is used to indicate that the second text image provides a glyph structure.

[0210] In one embodiment, the feature extraction module is further configured to: determine the region where the first text is located in the first image; remove the image content located in the region in the first image to obtain a background image; wherein the image content outside the region in the background image is consistent with the first image; stitch together the first text image, the second text image and the background image to obtain a composite visual input; extract the visual appearance features of the first text from the composite visual input according to the instructions of the first explanatory information; and extract the glyph structure features of the second text from the composite visual input according to the instructions of the second explanatory information.

[0211] In one embodiment, the feature extraction module is further configured to: map the composite visual input to a preset representation space to obtain a visual representation; encode the text information containing the first explanatory information and the second explanatory information to obtain text features; and perform cross-modal processing on the visual representation using the text features to obtain the visual appearance features of the first text and the glyph structure features of the second text.

[0212] In one embodiment, the feature extraction module is further configured to: perform cross-attention processing on the visual representation and text features while keeping the visual representation and text features independent of each other, to obtain aligned visual representation and aligned text features; concatenate the aligned visual representation and aligned text features to obtain a feature sequence; and perform self-attention processing on the feature sequence to obtain the visual appearance features of the first text and the glyph structure features of the second text.

[0213] In one embodiment, the generation module is further configured to: determine the portion of the visual representation corresponding to the region within the background image to obtain the representation to be processed; constrain the visual appearance of the representation to be processed with visual appearance features and constrain the glyph structure of the representation to be processed with glyph structure features, iteratively process the representation to be processed to obtain the target representation; decode the target representation to obtain a replacement text image; wherein the replacement text image presents a second text.

[0214] The modules in the aforementioned scene image-based text editing device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the operations corresponding to each module.

[0215] In one exemplary embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 10As shown, the computer device includes a processor, memory, input / output interfaces, a communication interface, a display unit, and an input device. The processor, memory, and input / output interfaces are connected via a system bus, and the communication interface, display unit, and input device are also connected to the system bus via the input / output interfaces. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The input / output interfaces are used for exchanging information between the processor and external devices. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wireless Fidelity (WIFI), mobile cellular networks, Near Field Communication (NFC), or other technologies. When executed by the processor, the computer program implements a text editing method based on scene images. The display unit is used to form a visually visible image and can be a display screen, a projection device, or a virtual reality imaging device. The display screen can be an LCD screen or an e-ink screen. The input device of the computer device can be a touch layer covering the display screen, or buttons, trackballs, or touchpads set on the casing of the computer device, or external keyboards, touchpads, or mice, etc.

[0216] Those skilled in the art will understand that Figure 10 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0217] In an exemplary embodiment, a computer device is provided, including a memory and a processor. The memory stores a computer program, and the processor executes the computer program to perform the following steps: acquiring a first image to be edited, and determining a first text to be replaced in the first image and a second text to replace the first text; acquiring the image content where the first text is located in the first image to obtain a first text image; and rendering the second text according to the stroke geometry of the second text to obtain a second text image; extracting visual appearance features of the first text from the first text image, and extracting glyph structure features of the second text from the second text image; constraining the visual appearance with the visual appearance features and constraining the glyph structure with the glyph structure features to generate a replacement text image; and replacing the image content where the first text is located in the first image with the replacement text image to obtain the second image.

[0218] In one embodiment, when the processor executes the computer program, it further performs the following steps: determining the circumscribed graph of the region where the first text is located in the first image; acquiring the image content within the circumscribed graph range in the first image to obtain the first text image; wherein the first text image retains the presentation attributes of the first text in the first image, and the presentation attributes include at least one of color, texture, font, and lighting; determining the stroke geometry of the second text; rendering the stroke geometry according to a preset arrangement to obtain the second text image.

[0219] In one embodiment, when the processor executes the computer program, it further performs the following steps: adding first explanatory information to a first text image; wherein the first explanatory information is used to indicate that the first text image provides a visual appearance; adding second explanatory information to a second text image; wherein the second explanatory information is used to indicate that the second text image provides a glyph structure.

[0220] In one embodiment, when the processor executes the computer program, it further performs the following steps: determining the region where the first text is located in the first image; removing the image content located in the region from the first image to obtain a background image; wherein the image content outside the region in the background image is consistent with the first image; stitching together the first text image, the second text image, and the background image to obtain a composite visual input; extracting the visual appearance features of the first text from the composite visual input according to the instructions of the first explanatory information; and extracting the glyph structure features of the second text from the composite visual input according to the instructions of the second explanatory information.

[0221] In one embodiment, when the processor executes the computer program, it further performs the following steps: mapping the composite visual input to a preset representation space to obtain a visual representation; encoding text information containing first explanatory information and second explanatory information to obtain text features; and performing cross-modal processing on the visual representation using the text features to obtain the visual appearance features of the first text and the glyph structure features of the second text.

[0222] In one embodiment, when the processor executes the computer program, it further performs the following steps: while keeping the visual representation and text features independent of each other, performing cross-attention processing on the visual representation and text features to obtain aligned visual representation and aligned text features; concatenating the aligned visual representation and aligned text features to obtain a feature sequence; and performing self-attention processing on the feature sequence to obtain the visual appearance features of the first text and the glyph structure features of the second text.

[0223] In one embodiment, when the processor executes the computer program, it further performs the following steps: determining a portion of the visual representation corresponding to the region within the background image to obtain a representation to be processed; constraining the visual appearance of the representation to be processed with visual appearance features and constraining the glyph structure of the representation to be processed with glyph structure features, iteratively processing the representation to be processed to obtain a target representation; decoding the target representation to obtain a replacement text image; wherein the replacement text image presents a second text.

[0224] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, performs the following steps: acquiring a first image to be edited, and determining a first text to be replaced in the first image and a second text to replace the first text; acquiring the image content containing the first text in the first image to obtain a first text image; and rendering the second text according to the stroke geometry of the second text to obtain a second text image; extracting visual appearance features of the first text from the first text image and extracting glyph structure features of the second text from the second text image; constraining the visual appearance with the visual appearance features and constraining the glyph structure with the glyph structure features to generate a replacement text image; and replacing the image content containing the first text in the first image with the replacement text image to obtain the second image.

[0225] In one embodiment, when the computer program is executed by the processor, it further performs the following steps: determining the circumscribed graph of the region where the first text is located in the first image; acquiring the image content within the circumscribed graph range in the first image to obtain the first text image; wherein the first text image retains the presentation attributes of the first text in the first image, and the presentation attributes include at least one of color, texture, font, and lighting; determining the stroke geometry of the second text; rendering the stroke geometry according to a preset arrangement to obtain the second text image.

[0226] In one embodiment, when the computer program is executed by a processor, it further performs the following steps: adding first explanatory information to a first text image; wherein the first explanatory information is used to instruct the first text image to provide a visual appearance; adding second explanatory information to a second text image; wherein the second explanatory information is used to instruct the second text image to provide a glyph structure; determining the region where the first text is located in the first image; removing image content located in the region from the first image to obtain a background image; splicing the first text image, the second text image, and the background image to obtain a composite visual input; extracting the visual appearance features of the first text from the composite visual input according to the instructions of the first explanatory information; and extracting the glyph structure features of the second text from the composite visual input according to the instructions of the second explanatory information.

[0227] In one embodiment, when the computer program is executed by the processor, it further performs the following steps: mapping the composite visual input to a preset representation space to obtain a visual representation; encoding text information containing first explanatory information and second explanatory information to obtain text features; performing cross-modal processing on the visual representation using the text features to obtain visual appearance features of the first text and glyph structure features of the second text; determining the portion of the visual representation corresponding to the region within the background image to obtain a representation to be processed; constraining the visual appearance of the representation to be processed with the visual appearance features and constraining the glyph structure of the representation to be processed with the glyph structure features, iteratively processing the representation to be processed to obtain a target representation; and decoding the target representation to obtain a replacement text image.

[0228] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, performs the following steps: acquiring a first image to be edited, and determining a first text to be replaced in the first image and a second text to replace the first text; acquiring the image content containing the first text in the first image to obtain a first text image; and rendering the second text according to the stroke geometry of the second text to obtain a second text image; extracting visual appearance features of the first text from the first text image and extracting glyph structure features of the second text from the second text image; constraining the visual appearance with the visual appearance features and constraining the glyph structure with the glyph structure features to generate a replacement text image; and replacing the image content containing the first text in the first image with the replacement text image to obtain the second image.

[0229] In one embodiment, when the computer program is executed by a processor, it further performs the following steps: adding first explanatory information to a first text image; adding second explanatory information to a second text image; determining the region where the first text is located in the first image; removing image content located in that region from the first image to obtain a background image; stitching together the first text image, the second text image, and the background image to obtain a composite visual input; extracting the visual appearance features of the first text from the composite visual input according to the instructions of the first explanatory information; and extracting the glyph structure features of the second text from the composite visual input according to the instructions of the second explanatory information.

[0230] In one embodiment, when the computer program is executed by the processor, it further performs the following steps: mapping the composite visual input to a preset representation space to obtain a visual representation; encoding text information containing first explanatory information and second explanatory information to obtain text features; performing cross-attention processing on the visual representation and text features while keeping the visual representation and text features independent of each other to obtain aligned visual representation and aligned text features; concatenating the aligned visual representation and aligned text features to obtain a feature sequence; and performing self-attention processing on the feature sequence to obtain the visual appearance features of the first text and the glyph structure features of the second text.

[0231] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0232] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0233] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, artificial intelligence (AI) processors, etc., and are not limited to these.

[0234] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0235] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A text editing method based on scene images, characterized in that, The method includes: Obtain a first image to be edited, and determine the first text to be replaced in the first image and the second text to replace the first text; The image content containing the first text is obtained from the first image to obtain the first text image; and the second text is rendered according to the stroke geometry of the second text to obtain the second text image. Extract the visual appearance features of the first text from the first text image, and extract the glyph structure features of the second text from the second text image; The visual appearance is constrained by the visual appearance features, and the glyph structure is constrained by the glyph structure features to generate a replacement text image; The image content containing the first text in the first image is replaced with the replacement text image to obtain the second image.

2. The method according to claim 1, characterized in that, The steps of obtaining the image content containing the first text in the first image to obtain the first text image, and rendering the second text according to the stroke geometry of the second text to obtain the second text image include: Determine the bounding graph of the region where the first text is located in the first image; The image content located within the bounding graphic area of ​​the first image is obtained to obtain the first text image; wherein the first text image retains the presentation attributes of the first text in the first image, and the presentation attributes include at least one of color, texture, font and lighting. Determine the stroke geometry of the second text; The stroke geometry is rendered according to a preset layout to obtain the second text image.

3. The method according to claim 1, characterized in that, After obtaining the image content containing the first text in the first image to obtain the first text image; and rendering the second text according to the stroke geometry of the second text to obtain the second text image, the method further includes: Add first explanatory information to the first text image; wherein the first explanatory information is used to instruct the first text image to provide a visual appearance; Add second explanatory information to the second text image; wherein the second explanatory information is used to indicate that the second text image provides a glyph structure.

4. The method according to claim 3, characterized in that, The steps of extracting the visual appearance features of the first text from the first text image and extracting the glyph structure features of the second text from the second text image include: Determine the region where the first text is located in the first image; Remove the image content located within the defined region from the first image to obtain a background image; wherein, the image content outside the defined region in the background image remains consistent with the first image; By stitching together the first text image, the second text image, and the background image, a composite visual input is obtained. As instructed by the first explanatory information, the visual appearance features of the first text are extracted from the composite visual input; and as instructed by the second explanatory information, the glyph structure features of the second text are extracted from the composite visual input.

5. The method according to claim 4, characterized in that, The visual appearance features of the first text are extracted from the composite visual input according to the instructions of the first explanatory information. And, according to the instructions of the second explanatory information, the step of extracting the glyph structure features of the second text from the composite visual input includes: The composite visual input is mapped to a preset representation space to obtain a visual representation; Encode the text information containing the first explanatory information and the second explanatory information to obtain text features; The visual representation is processed across modalities using the text features to obtain the visual appearance features of the first text and the glyph structure features of the second text.

6. The method according to claim 5, characterized in that, The step of performing cross-modal processing on the visual representation using the text features to obtain the visual appearance features of the first text and the glyph structure features of the second text includes: While keeping the visual representation and the text features independent of each other, cross-attention processing is performed on the visual representation and the text features to obtain aligned visual representation and aligned text features. The aligned visual representation is concatenated with the aligned text features to obtain a feature sequence; Self-attention processing is performed on the feature sequence to obtain the visual appearance features of the first text and the glyph structure features of the second text.

7. The method according to claim 5, characterized in that, The step of constraining the visual appearance with the visual appearance features and constraining the glyph structure with the glyph structure features to generate a replacement text image includes: The portion of the visual representation corresponding to the region within the background image is determined to obtain the representation to be processed. The visual appearance of the representation to be processed is constrained by the visual appearance features, and the glyph structure of the representation to be processed is constrained by the glyph structure features. The representation to be processed is iteratively processed to obtain the target representation. The target representation is decoded to obtain the replacement text image; wherein the replacement text image contains the second text.

8. A text editing device based on scene images, characterized in that, The device includes: The acquisition module is used to acquire a first image to be edited, and to determine the first text to be replaced in the first image and the second text to replace the first text; An image construction module is used to obtain the image content where the first text is located in the first image to obtain a first text image; and to render the second text according to the stroke geometry of the second text to obtain a second text image. The feature extraction module is used to extract the visual appearance features of the first text from the first text image and the glyph structure features of the second text from the second text image. The generation module is used to constrain the visual appearance with the visual appearance features and constrain the glyph structure with the glyph structure features to generate a replacement text image. The replacement module is used to replace the image content containing the first text in the first image with the replacement text image to obtain the second image.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

Images