Image processing method and device, equipment and storage medium

Abstract

The invention provides an image processing method and device, equipment and a storage medium, and relates to the field of image processing. The method comprises the steps of obtaining a to-be-processed image, wherein the to-be-processed image comprises a text; performing multi-dimensional recognition processing on a text in the to-be-processed image to obtain a text noise feature of the to-be-processed image; obtaining a text editing prompt word; and performing image de-noising and redrawing according to the text editing prompt words and the text noise features, and generating a target image after the text is edited. According to the method, feature extraction can be comprehensively carried out on the text in the to-be-processed image from multiple dimensions to obtain the text noise features, the text noise features serve as the basis of noise pixel elimination, the obtained text editing prompt words serve as the basis of generation of the redrawing content, and the target image after the text is edited is generated.

Description

Technical Field

[0001] This disclosure relates to the field of image processing, and more particularly to an image processing method, apparatus, device, and storage medium. Background Technology

[0002] With the rapid development of computer technology and image processing technology, people's needs for image processing are becoming increasingly diversified, among which text editing within images has become an important requirement. Editing text within images can greatly improve the work efficiency of users in various fields and provide them with a more diverse and personalized image editing experience.

[0003] It should be noted that the information disclosed in the background section above is only used to enhance the understanding of the background of this disclosure, and therefore may include information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0004] The purpose of this disclosure is to provide an image processing method, apparatus, device, and storage medium.

[0005] According to a first aspect of the present disclosure, an image processing method is provided, comprising: acquiring an image to be processed, the image to be processed including text; performing multidimensional recognition processing on the text in the image to be processed to obtain text noise features of the image to be processed; acquiring text editing prompts; and performing image denoising and redrawing based on the text editing prompts and the text noise features to generate a target image after the text has been edited.

[0006] In some embodiments, multidimensional recognition processing is performed on the text in the image to be processed to obtain text noise features of the image to be processed, including: performing text recognition on the image to be processed to obtain text features of the image to be processed; performing position recognition on the text in the image to be processed to obtain a region segmentation feature map of the image to be processed; performing edge detection on the text in the image to be processed to obtain a text edge feature map of the image to be processed; and fusing the text features, the region segmentation feature map, and the text edge feature map to obtain text noise features of the image to be processed.

[0007] In some implementations, image denoising and redrawing based on the text editing prompts and the text noise features to generate a target image with the text edited includes: inputting the text editing prompts and the text noise features into an image generation model; wherein the image generation model is obtained by adjusting and training an initial stable diffusion model; performing semantic understanding on the text editing prompts through the image generation model to determine the semantic features for text editing and the target text to be edited in the text; and processing the target text in the image based on the text noise features and the semantic features through the image generation model to denoise and redraw the corresponding region of the target text in the image to be processed, thereby generating a target image with the target text edited.

[0008] In some embodiments, the image processing method further includes: performing morphological detection on the text in the image to be processed to obtain the text morphological features of the image to be processed; inputting the text morphological features into the image generation model; and using the image generation model to use the text morphological features as guiding information for denoising and redrawing to generate a target image after the target text has been edited.

[0009] In some implementations, morphological detection is performed on the text in the image to be processed to obtain the text morphological features of the image to be processed, including: extracting text contours from the text in the image to be processed to obtain text contour extraction results; expanding the text contour extraction results to obtain contour expansion results; and encoding the contour expansion results to obtain the text morphological features.

[0010] In some embodiments, the image generation model includes a cloud-based replacement model, an edge-side replacement model, and an edge-side elimination model; wherein, the cloud-based replacement model is obtained by adjusting and training the initial stable diffusion model using a second sample image containing text; the edge-side replacement model is obtained by compressing, encoding, and quantizing the cloud-based replacement model; and the edge-side elimination model is obtained by distilling the initial stable diffusion model.

[0011] In some embodiments, the image processing method further includes: in response to receiving a text replacement instruction, invoking the cloud-based replacement model or the edge-side replacement model to generate the target image; and in response to receiving a text removal instruction, invoking the edge-side removal model to generate the target image.

[0012] In some implementations, obtaining text editing prompts includes: in response to receiving a text replacement instruction, obtaining a text replacement prompt input by the user as the text editing prompt; and in response to receiving a text deletion instruction, obtaining a preset text deletion prompt as the text editing prompt.

[0013] In some embodiments, the image processing method further includes: generating a target image after the text has been replaced in response to the text editing prompt being a text replacement prompt; wherein the text replacement prompt indicates the text information in the image to be replaced; and generating a target image after the text has been removed in response to the text editing prompt being a text removal prompt; wherein the text removal prompt indicates quality description information for the target image.

[0014] In some embodiments, the image processing method further includes: displaying a plurality of text regions identified in the image to be processed; and, in response to a selection operation and an editing operation of a target text region in the plurality of text regions, displaying a target image after the text in the target text region has been edited.

[0015] In some implementations, the text features are output by a trained text feature extraction model; the text noise features are output by a trained feature fusion model; wherein, the image processing method further includes: acquiring a first sample image containing text, sample prompt words, and a label image matching the first sample image and the sample prompt words; performing text recognition on the first sample image using an initial text feature extraction model to obtain sample text features of the first sample image; performing position recognition on the text in the first sample image to obtain a sample region segmentation feature map of the first sample image; performing edge detection on the text in the first sample image to obtain a sample text edge feature map of the first sample image; fusing the sample text features, the sample region segmentation feature map, and the sample text edge feature map using an initial feature fusion model to obtain sample text noise features of the first sample image; performing image denoising and redrawing based on the sample prompt words and the sample text noise features to generate a predicted image; constructing a loss function based on the predicted image and the label image, and using the loss function to train the initial text feature extraction model and the initial feature fusion model to obtain a trained text feature extraction model and a trained feature fusion model.

[0016] According to a second aspect of the present disclosure, an image processing apparatus is provided, comprising: an acquisition unit for acquiring an image to be processed, the image to be processed including text; a noise feature generation unit for performing multi-dimensional recognition processing on the text in the image to be processed to obtain text noise features of the image to be processed; the acquisition unit is further configured to acquire text editing prompts; and an image generation unit for performing image denoising and redrawing based on the text editing prompts and the text noise features to generate a target image after the text has been edited.

[0017] In some embodiments, the noise feature generation unit performs multi-dimensional recognition processing on the text in the image to be processed to obtain text noise features of the image to be processed, including: performing text recognition on the image to be processed to obtain text features of the image to be processed; performing position recognition on the text in the image to be processed to obtain a region segmentation feature map of the image to be processed; performing edge detection on the text in the image to be processed to obtain a text edge feature map of the image to be processed; and fusing the text features, the region segmentation feature map, and the text edge feature map to obtain text noise features of the image to be processed.

[0018] In some embodiments, the image generation unit performs image denoising and redrawing based on the text editing prompt and the text noise features to generate a target image after the text has been edited. This includes: inputting the text editing prompt and the text noise features into an image generation model; wherein the image generation model is obtained by adjusting and training an initial stable diffusion model; performing semantic understanding on the text editing prompt through the image generation model to determine the semantic features for text editing and the target text to be edited; and processing the target text in the image to be processed based on the text noise features and the semantic features using the image generation model to denoise and redraw the corresponding region of the target text in the image to be processed, thereby generating a target image after the target text has been edited.

[0019] In some embodiments, the image processing apparatus further includes a shape detection unit, which is used to perform shape detection on the text in the image to be processed to obtain the text shape features of the image to be processed; the image generation unit is further used to input the text shape features into the image generation model; and the image generation model uses the text shape features as guiding information for denoising and redrawing to generate a target image after the target text has been edited.

[0020] In some implementations, the morphology detection unit performs morphology detection on the text in the image to be processed to obtain the text morphology features of the image to be processed, including: extracting text contours from the text in the image to be processed to obtain a text contour extraction result; expanding the text contour extraction result to obtain a contour expansion result; and encoding the contour expansion result to obtain the text morphology features.

[0021] In some embodiments, the image generation model includes a cloud-based replacement model, an edge-side replacement model, and an edge-side elimination model; wherein, the cloud-based replacement model is obtained by adjusting and training the initial stable diffusion model using a second sample image containing text; the edge-side replacement model is obtained by compressing, encoding, and quantizing the cloud-based replacement model; and the edge-side elimination model is obtained by distilling the initial stable diffusion model.

[0022] In some embodiments, the image processing apparatus further includes a model invocation unit, configured to: invoke the cloud-based replacement model or the edge-side replacement model to generate the target image in response to receiving a text replacement instruction; and invoke the edge-side elimination model to generate the target image in response to receiving a text elimination instruction.

[0023] In some embodiments, the acquisition unit acquires text editing prompts by: in response to receiving a text replacement instruction, acquiring a text replacement prompt input by the user as the text editing prompt; and in response to receiving a text deletion instruction, acquiring a preset text deletion prompt as the text editing prompt.

[0024] In some embodiments, the image generation unit is further configured to: generate a target image after the text has been replaced in response to the text editing prompt being a text replacement prompt; wherein the text replacement prompt indicates the text information in the image to be replaced; and generate a target image after the text has been removed in response to the text editing prompt being a text removal prompt; wherein the text removal prompt indicates quality description information for the target image.

[0025] In some embodiments, the image processing apparatus further includes a display unit for: displaying a plurality of text regions identified in the image to be processed; and, in response to a selection operation and an editing operation of a target text region in the plurality of text regions, displaying a target image after the text in the target text region has been edited.

[0026] In some embodiments, the text features are output by a trained text feature extraction model; the text noise features are output by a trained feature fusion model; the image processing device further includes a training unit; wherein, the acquisition unit is further configured to: acquire a first sample image containing text, sample prompt words, and a label image matching the first sample image and the sample prompt words; the noise feature generation unit is further configured to perform text recognition on the first sample image using the initial text feature extraction model to obtain sample text features of the first sample image; the noise feature generation unit is further configured to perform position recognition on the text in the first sample image to obtain a sample region segmentation feature map of the first sample image; the noise feature generation unit is further configured to... The method involves performing edge detection on the text in the first sample image to obtain a sample text edge feature map of the first sample image; the noise feature generation unit is further configured to fuse the sample text features, the sample region segmentation feature map, and the sample text edge feature map using an initial feature fusion model to obtain sample text noise features of the first sample image; the image generation unit is further configured to perform image denoising and redrawing based on the sample prompt words and the sample text noise features to generate a prediction image; the training unit is configured to: construct a loss function based on the prediction image and the label image, and use the loss function to train the initial text feature extraction model and the initial feature fusion model to obtain the trained text feature extraction model and the trained feature fusion model.

[0027] According to a third aspect of the present disclosure, an electronic device is provided, characterized in that it includes: a processor; a memory for storing processor-executable instructions; wherein the processor is configured to implement the image processing method described above.

[0028] According to a fourth aspect of the present disclosure, a non-transitory computer-readable storage medium is provided, wherein when instructions in the storage medium are executed by a processor of a mobile terminal, the mobile terminal is enabled to execute an image processing method, the method comprising: acquiring an image to be processed, the image to be processed including text; performing multi-dimensional recognition processing on the text in the image to be processed to obtain text noise features of the image to be processed; acquiring text editing prompts; and performing image denoising and redrawing based on the text editing prompts and the text noise features to generate a target image after the text has been edited.

[0029] According to a fifth aspect of the present disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements the image processing method described above.

[0030] The technical solutions provided by the embodiments of this disclosure may include the following beneficial effects:

[0031] This disclosure can perform multi-dimensional recognition processing on the text in the image to be processed. It comprehensively extracts features from the text in the image from multiple dimensions and then obtains text noise features to achieve a more comprehensive understanding and processing of the text in the image to be processed based on the text noise features. On the other hand, it can obtain text editing prompts and apply them together with the text noise features to the image denoising and redrawing process. Specifically, the text noise features can be used as the basis for processing text pixels, and the original text-related pixels in the image to be processed can be eliminated as noise pixels. At the same time, based on the semantic information corresponding to the text editing prompts, the pixel content that conforms to the semantic information can be redrawn on the text area in the image to be processed, and finally the target image after the original text has been accurately edited can be generated, thereby realizing the editing of the text in the image to be processed.

[0032] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0033] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure.

[0034] Figure 1 This is a flowchart illustrating an image processing method according to some embodiments of the present disclosure.

[0035] Figure 2 This is a flowchart illustrating a text noise feature obtained in an image processing method according to some embodiments of the present disclosure.

[0036] Figure 3 This is a schematic diagram illustrating an image processing method according to some embodiments of the present disclosure, in which an image to be processed is processed into a text edge feature map.

[0037] Figure 4 This is a schematic diagram of a model processing framework in an image processing method according to some embodiments of the present disclosure.

[0038] Figure 5 This is a flowchart illustrating yet another image processing method according to some embodiments of the present disclosure.

[0039] Figure 6 This is a schematic diagram illustrating an image processing method according to some embodiments of the present disclosure, in which an image to be processed is processed into text morphological features.

[0040] Figure 7 This is a schematic diagram of a model processing framework in another image processing method according to some embodiments of the present disclosure.

[0041] Figure 8 This is an interactive process for replacing text in an image in an image processing method according to some embodiments of the present disclosure.

[0042] Figure 9 This is an interactive process for removing text from an image in another image processing method according to some embodiments of the present disclosure.

[0043] Figure 10 This is a block diagram illustrating an image processing apparatus according to some embodiments of the present disclosure.

[0044] Figure 11 This is a block diagram illustrating an apparatus for image processing according to some embodiments of the present disclosure. Detailed Implementation

[0045] Exemplary embodiments of this disclosure will be described in detail herein, examples of which are illustrated in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings denote the same or similar elements unless otherwise indicated. Various changes, modifications, and equivalents of the methods, apparatus, and / or systems described herein will become apparent upon understanding this disclosure. For example, the order of operations described herein is merely illustrative and is not limited to those orders set forth herein, but can be changed as will become apparent upon understanding this disclosure, except for operations that must be performed in a particular order. Furthermore, for clarity and brevity, descriptions of features known in the art may be omitted.

[0046] The embodiments described below, which are examples of some of the embodiments of this disclosure, do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0047] In related technologies, scene text detection methods can generally be divided into two categories: regression-based methods and segmentation-based methods. Regression-based methods treat text detection as a unique object detection task, aiming to locate text regions by predicting offsets from anchor points or pixels. However, compared to general objects, text exhibits significant variations in scale and orientation. While regression-based methods perform well with quadrilateral text, they struggle to adapt to text of arbitrary shapes.

[0048] Segmentation-based methods treat text detection as a dense binary prediction task. Differentiable binarization is introduced into the segmentation network, allowing for adaptive threshold prediction. Text segmentation methods typically rely on thresholding or low-level features to binarize scene text images. However, these methods often struggle with text images containing complex colors and textures, resulting in poor performance.

[0049] The specific implementation methods of the embodiments of this disclosure will now be described in detail with reference to the accompanying drawings.

[0050] Figure 1 This is a flowchart illustrating an image processing method according to some embodiments of the present disclosure, such as... Figure 1 As shown, the image processing method can be applied to a terminal device and may include the following steps.

[0051] In step S110, an image to be processed is obtained, which includes text.

[0052] In this embodiment of the disclosure, the image to be processed contains text content, which is embedded in the image as pixels. The image to be processed can come from various sources, such as text in scanned documents, photos, screenshots, etc.

[0053] In step S120, multidimensional recognition processing is performed on the text in the image to be processed to obtain the text noise features of the image to be processed.

[0054] In this embodiment of the disclosure, multi-dimensional recognition processing is performed on the text in the image to be processed, which can extract detailed features of the text in the image, namely text noise features. Text noise features can be used to describe the characteristics of the text itself and the region in the image to be processed, and can be used in subsequent denoising and redrawing processes.

[0055] Among them, the text noise features obtained through multi-dimensional recognition processing can comprehensively describe the text in the image to be processed from multiple dimensions. By integrating these multi-dimensional characteristics, the text noise features can be used to more comprehensively understand and process the text in the image to be processed.

[0056] In an exemplary embodiment, the content, layout, position, outline, font, size, color, or other visual features of the text in the image to be processed can be identified, and these features can be used together to constitute text noise features.

[0057] In an exemplary embodiment, the text noise feature may be a noise image whose size may be the same as the size of the image to be processed.

[0058] In step S130, the text editing prompt is obtained.

[0059] In this embodiment of the disclosure, the text editing prompt can be a keyword or phrase used to guide the generation of the target image; that is, the text editing prompt can be used to guide the subsequent text editing operations.

[0060] In this embodiment of the disclosure, the text editing prompt can be entered by the user according to actual needs, and may include the text content that the user wishes to edit and its related instructions; the text editing prompt can also be a preset piece of text information.

[0061] In an exemplary embodiment, the text editing prompts can indicate the modifications that need to be made to the text in the image, such as replacing certain words, adjusting the text format, adding or deleting text, etc. Specifically, they can include specific words or phrases to be replaced or modified, or more general instruction phrases about text style, format layout, or image quality requirements, such as: "Replace 'The weather is nice today' in the image with 'Let's go on a trip'", "Display 'The weather is nice today' in the image in blue", etc.

[0062] In step S140, the image is denoised and redrawn based on the text editing prompts and the text noise features to generate the target image after the text has been edited.

[0063] In this embodiment, a denoising and redrawing operation can be performed based on the previously collected information (i.e., text noise features and text editing prompts). During the denoising and redrawing process, on the one hand, based on the text noise features, pixels originally containing text in the image to be processed can be eliminated as noise pixels; on the other hand, by understanding the semantic information of the text editing prompts, this semantic information can be used as a guide to redraw pixel content that conforms to the semantic information on the text region in the image to be processed, ultimately generating a target image with precisely edited text, thereby achieving the editing of text in the image to be processed.

[0064] In an exemplary embodiment, a functional module containing a trained noise predictor can be configured, which can perform noise removal based on text noise features to generate a new image.

[0065] As can be seen from the above steps, the image processing method provided in this disclosure can perform multi-dimensional recognition processing on the text in the image to be processed, obtain text noise features related to the text in the image to be processed, and then perform image denoising and redrawing based on the obtained text editing prompts and the obtained text noise features to generate a new image with the original text area redrawn. It is evident that, through the embodiments of this disclosure, on the one hand, multi-dimensional recognition processing can be performed on the text in the image to be processed, comprehensively extracting features from multiple dimensions to obtain text noise features, so as to achieve a more comprehensive understanding and processing of the text in the image to be processed based on the text noise features; on the other hand, text editing prompts can be obtained, and the text editing prompts and text noise features can be applied together in the image denoising and redrawing process. Specifically, the text noise features can be used as the basis for processing text pixels, eliminating the original text-related pixels in the image to be processed as noise pixels. Simultaneously, based on the semantic information corresponding to the text editing prompts, pixel content conforming to the semantic information can be redrawn on the text area in the image to be processed, ultimately generating a target image with the original text precisely edited, thereby realizing the editing of the text in the image to be processed.

[0066] Figure 2 This is a flowchart illustrating a text noise feature obtained in an image processing method according to some embodiments of the present disclosure.

[0067] like Figure 2 As shown, in some embodiments of this disclosure, performing multidimensional recognition processing on the text in the image to be processed to obtain the text noise features of the image to be processed may include the following steps.

[0068] Step S210: Perform text recognition on the image to be processed to obtain the text features of the image to be processed.

[0069] In this embodiment, optical character recognition (OCR) technology can be used to identify the text content in the image to be processed, and then a trained text feature extraction model can be used to process the text content to obtain the text features of the image to be processed. The text features can carry information such as the type, font, size, and color of the characters.

[0070] In an exemplary embodiment, step S210 may involve: performing text recognition on the image to be processed to obtain text words; performing sequence transformation on the text words to obtain a sequence array; performing multiple downsampling on the sequence array to obtain multiple intermediate features; and concatenating the multiple intermediate features to obtain the text features of the image to be processed.

[0071] Step S220: Perform position recognition on the text in the image to be processed to obtain the region segmentation feature map of the image to be processed.

[0072] In this embodiment of the disclosure, position recognition is performed on the text in the image to be processed to determine the specific location of the text in the image. Furthermore, corresponding color information can be determined based on the identified position, and the position information and color information are included in the region segmentation feature map for use in the fusion generation of text noise features.

[0073] In an exemplary embodiment, the size of the region segmentation feature map can be the same as the size of the image to be processed. The region segmentation feature map may include text selection information, where the selected area is the text region corresponding to the text. The text region can be an irregular shape selected around the perimeter of a continuous text (such as a complete sentence or paragraph) as a whole.

[0074] In an exemplary embodiment, the image segmentation function module configured on the terminal side can be invoked to locate the text region through image processing techniques such as connected component analysis and projection methods, and obtain the region segmentation feature map of the image to be processed. This feature map can represent the boundary between the text region and the background region.

[0075] Step S230: Perform edge detection on the text in the image to be processed to obtain the text edge feature map of the image to be processed.

[0076] In this embodiment of the disclosure, edge detection algorithms such as the Scribble model can be used to detect the edges of text characters, thereby obtaining a text edge feature map of the image to be processed. This feature map can highlight the outline and detailed information of the text characters.

[0077] In an exemplary embodiment, the size of the text edge feature map can be the same as the size of the image to be processed.

[0078] In an exemplary embodiment, step S230 may involve: extracting text edges from the text in the image to be processed to obtain a text edge extraction result; and encoding the text edge extraction result to obtain a text edge feature map of the image to be processed. The text edge extraction result may be an irregular shape selected by bounding the pixels involved in the text strokes.

[0079] In an exemplary embodiment, VAE-Encoder can be used to quantize the INT4 model to encode the text edge extraction results.

[0080] Figure 3 This is a schematic diagram illustrating an image processing method according to some embodiments of the present disclosure, in which an image to be processed is processed into a text edge feature map.

[0081] like Figure 3 As shown, the image includes a target image 301 and a first intermediate image 302. The first intermediate image 302 is obtained by extracting text edges from the target image 301, and the text edge feature map is obtained by encoding the first intermediate image 302.

[0082] Step S240: The text features, the region segmentation feature map, and the text edge feature map are fused to obtain the text noise features of the image to be processed.

[0083] In this embodiment, the text features, region segmentation feature maps, and text edge feature maps obtained in the preceding steps can be fused. The purpose of feature fusion is to integrate feature information from different dimensions to form a more comprehensive and richer feature representation. The fusion process can be implemented using methods such as weighted superposition, feature concatenation, or feature fusion layers in deep learning. By fusing these features, the text noise features of the image to be processed can be obtained. In this way, the text noise features can contain both the basic attributes of the text and reflect the spatial location and edge details of the text in the image.

[0084] In an exemplary embodiment, step S240 may involve: first, performing size normalization on the region segmentation feature map to obtain an adjusted segmentation feature map; then, performing downsampling and compression processing on the adjusted segmentation feature map to obtain a compressed segmentation feature map; then, using a self-attention mechanism to concatenate the text features, the compressed segmentation feature map, and the text edge feature map to obtain a concatenated vector; and finally, performing convolution processing on the concatenated vector to obtain the text noise features of the image to be processed.

[0085] In the self-attention mechanism, the key vector can be determined based on the text features, the value vector can be determined based on the compressed segmentation feature map, and the query vector can be determined based on the text edge feature map.

[0086] Through the embodiments of this disclosure, multi-dimensional recognition processing, such as text recognition, location recognition and region segmentation, edge detection, and feature fusion, can be performed on the image to be processed to obtain the text noise features of the image. These features provide strong support for subsequent image processing tasks (including text denoising, image generation, etc.).

[0087] In some embodiments of this disclosure, the text features are output by a trained text feature extraction model; the text noise features are output by a trained feature fusion model.

[0088] In this embodiment of the disclosure, the text feature extraction model can be a deep learning-based model, such as a recurrent neural network (RNN), long short-term memory network (LSTM), or Transformer structure, to capture complex patterns and relationships in the text.

[0089] Feature fusion models can include multiple convolutional layers and sampling layers to process and concatenate various types of features, ensuring that the different features after concatenation can be distributed in the same feature space.

[0090] Figure 4 This is a schematic diagram of a model processing framework in an image processing method according to some embodiments of the present disclosure.

[0091] like Figure 4 As shown, it includes a text detection module 401, a text feature extraction model 402, a region segmentation module 403, an edge detection module 404, a feature fusion model 405, and an image generation module 406.

[0092] refer to Figure 4 The text detection module 401, text feature extraction model 402, and region segmentation module 403 can process the image to be processed respectively, and obtain the text features, region segmentation feature map, and text edge feature map of the image to be processed. Then, the text features, region segmentation feature map, and text edge feature map can be input into the feature fusion model 405. After processing these three features, the feature fusion model 405 can output the text noise features of the image to be processed. Finally, the obtained text editing prompts and text noise features can be input into the image generation module 406. The image generation module 406 performs image denoising and redrawing based on the text editing prompts and text noise features to generate the target image.

[0093] In some embodiments of this disclosure, the image processing method may further include training a text feature extraction model and a feature fusion model in the following manner.

[0094] The process involves: acquiring a first sample image containing text, sample prompt words, and a label image matching the first sample image and the sample prompt words; performing text recognition on the first sample image using an initial text feature extraction model to obtain sample text features of the first sample image; performing position recognition on the text in the first sample image to obtain a sample region segmentation feature map of the first sample image; performing edge detection on the text in the first sample image to obtain a sample text edge feature map of the first sample image; fusing the sample text features, the sample region segmentation feature map, and the sample text edge feature map using an initial feature fusion model to obtain sample text noise features of the first sample image; performing image denoising and redrawing based on the sample prompt words and the sample text noise features to generate a predicted image; constructing a loss function based on the predicted image and the label image, and using the loss function to train the initial text feature extraction model and the initial feature fusion model to obtain the trained text feature extraction model and the trained feature fusion model.

[0095] In this embodiment of the disclosure, after multiple iterations of training, both the trained text feature extraction model and the feature fusion model can be obtained simultaneously. Combined with... Figure 4 During the training process described above, the text feature extraction model 402 and the feature fusion model 405 can be jointly trained.

[0096] After training, the text feature extraction model can accurately and effectively extract text features from images, and the feature fusion model can learn how to effectively integrate feature information from these different sources and generate a text noise feature containing multi-dimensional information.

[0097] This disclosure provides a method for joint training. The text feature extraction model and the feature fusion model can collaborate and jointly optimize their performance. Since the two models can depend on each other during training, they can mutually constrain each other, thereby reducing the risk of overfitting during the training of a single model and contributing to improved accuracy and robustness in image processing.

[0098] Figure 5 This is a flowchart illustrating yet another image processing method according to some embodiments of the present disclosure.

[0099] In this embodiment of the disclosure, Figure 5 In the image processing method shown, steps S510, S520, and S530 are respectively related to... Figure 1 Steps S110, S120, and S130 in the image processing method shown correspond to each other and will not be repeated here.

[0100] In this embodiment of the disclosure, Figure 1Based on the image processing method shown, Figure 5 The image processing method shown may also include the following steps.

[0101] In step S540, the text editing prompt and the text noise features are input into the image generation model; wherein the image generation model is obtained by adjusting and training an initial stable diffusion model.

[0102] In this embodiment of the disclosure, the stable diffusion model can be the SD1.5 model (Stable Diffusion v1.5), and the image generation model can be a model finely tuned based on SD1.5. The SD1.5 model is a version of the Stable Diffusion series of models, designed to generate high-quality images from noise. The SD1.5 model can generate realistic images by progressively reducing noise based on the theory of the diffusion process.

[0103] In this embodiment of the disclosure, the image generation model obtained by fine-tuning the stable diffusion model can also have the ability to generate high-quality images from noise. After fine-tuning and training the stable diffusion model, the image generation model can have the ability to generate target images of edited text based on text noise features.

[0104] In step S550, the image generation model is used to perform semantic understanding on the text editing prompts to determine the semantic features for text editing and the target text to be edited in the text.

[0105] In this embodiment of the disclosure, a text editing prompt refers to a keyword or phrase used in the input to guide the SD1.5 model to generate a new pixel image. Therefore, the image generation model fine-tuned based on the SD1.5 model can understand the semantic information corresponding to the text editing prompt, and thus determine the text content (i.e., the target text) indicated by the text editing prompt and the related editing instructions (such as replacement, addition, deletion, etc.).

[0106] In step S560, the image generation model processes the text noise features and semantic features to denoise and redraw the region corresponding to the target text in the image to be processed, thereby generating the target image after the target text has been edited.

[0107] In this embodiment, the image generation model can locate the region corresponding to the target text in the image to be processed, and then denoise and redraw the region based on the text noise features to generate the target image. The target image visually remains consistent with the original image, but in the region corresponding to the target text, the original text (i.e., the target text) has been modified accordingly based on text editing prompts, presenting an effect of editing the target text in the image.

[0108] Through the embodiments of this disclosure, an image generation model finely tuned based on SD1.5 can be used to achieve efficient and accurate image text editing.

[0109] In some embodiments of this disclosure, the image processing method further includes: performing morphological detection on the text in the image to be processed to obtain the text morphological features of the image to be processed; inputting the text morphological features into the image generation model; and using the image generation model to use the text morphological features as guiding information for denoising and redrawing to generate a target image after the target text has been edited.

[0110] In this embodiment of the disclosure, edge morphology detection algorithms such as Canny and Sobel can be used to detect text in the image to be processed and obtain text morphology features.

[0111] In an exemplary embodiment, existing edge detection algorithms such as Canny and Sobel can be retrained using training images labeled with detailed text edge contours to improve the morphological detection capability of the edge morphology detection algorithms and ensure that the trained algorithm model can accurately identify the edge morphology of the text. The text morphological features can carry information about the text contour extraction results, which can be irregular shapes that more precisely outline the pixels involved in the text strokes.

[0112] In this embodiment of the disclosure, during the denoising and redrawing process using the image generation model, the text morphological features can be used as positional guides for denoising and redrawing.

[0113] Through the embodiments of this disclosure, after inputting text morphological features into the image generation model, the text morphological features can serve as a guiding feature in the generation of the target image, instructing the image generation model to more accurately denoise the pixels of the original text and redraw the area where the original text is located.

[0114] Furthermore, in an exemplary embodiment, during the training of the text feature extraction model and the feature fusion model, a step can be added to generate text morphological features and input the text morphological features into the image generation model as a guide, thereby training a text feature extraction model and a feature fusion model with accurate processing capabilities and ensuring the coordination of collaborative image generation between various modules.

[0115] In some embodiments of this disclosure, morphological detection is performed on the text in the image to be processed to obtain the text morphological features of the image to be processed, including: extracting text contours from the text in the image to be processed to obtain text contour extraction results; expanding the text contour extraction results to obtain contour expansion results; and encoding the contour expansion results to obtain the text morphological features.

[0116] In this embodiment of the disclosure, since many handwritten characters have traces of pauses or outlines at the beginning and end of strokes, the removal of residues may occur after morphological detection of the text in the image. Therefore, an image dilation algorithm can be used to expand the text contour extraction results (e.g., 0.1%, 0.12%) to obtain text morphological features with better application effects.

[0117] The contour expansion results can be encoded using the same encoding method as the text edge extraction results. Alternatively, the contour expansion results can be encoded using VAE-Encoder quantization of the INT4 model.

[0118] Figure 6 This is a schematic diagram illustrating the process of processing an image into text morphological features in an image processing method according to some embodiments of this disclosure. For example... Figure 6 As shown, the image includes a to-be-processed image 601, a second intermediate image 602, and a third intermediate image 603. The second intermediate image 602 is obtained by extracting the text contours from the to-be-processed image 601, the third intermediate image 603 is obtained by expanding the contours of the second intermediate image 602, and the text morphological features are obtained by encoding the second intermediate image 602.

[0119] In some embodiments of this disclosure, the image generation model includes a cloud-based replacement model, an edge-side replacement model, and an edge-side elimination model; wherein, the cloud-based replacement model is obtained by adjusting and training the initial stable diffusion model using a second sample image containing text; the edge-side replacement model is obtained by compressing, encoding, and quantizing the cloud-based replacement model; and the edge-side elimination model is obtained by distilling the initial stable diffusion model.

[0120] In this embodiment of the disclosure, the cloud-based replacement model and the edge-side replacement model can be used to modify the replacement editing type of text in an image. The edge-side elimination model can be used to modify the elimination editing type of text in an image.

[0121] Among them, the cloud-based replacement model can retain the SD1.5 model's multiple iterations of denoising in the generated image process. The cloud-based replacement model can be deployed in the cloud, enabling it to access more powerful computing resources online and achieve high-precision generation results.

[0122] Compression encoding and quantization of the cloud-based replacement model can reduce its size and computational complexity, resulting in a smaller edge-side replacement model. This allows it to run on resource-constrained terminal devices such as mobile devices and embedded devices, making edge-side deployment of the replacement model possible. In practical applications, due to its small size, the edge-side replacement model can also run offline, providing users with better privacy while ensuring image generation functionality.

[0123] Distilling the cloud-based replacement model yields an edge-side elimination model. Specifically, since the edge-side elimination model removes text from images, its target is more explicit than text replacement, making it easier to implement. Therefore, the multi-iteration image generation process of the cloud-based replacement model can be distilled into a single-step image generation process. This allows the edge-side elimination model to learn to generate the target image with removed text in just one iteration, and to output the elimination result with only one inference step. This design ensures edge-side deployment while accelerating the function's operation, eliminating long iteration times and providing a better user experience.

[0124] This disclosure provides three different models, all derived from an initial stable diffusion model through adjustments and optimizations. These models are adaptable to various application scenarios and computing resources, offering users more choices in different application environments. This design also allows the image generation model to be flexibly applied to both cloud and terminal devices, meeting the needs of diverse users.

[0125] Figure 7 This is a schematic diagram of a model processing framework in another image processing method according to some embodiments of the present disclosure.

[0126] like Figure 7 As shown, it includes a text detection module, a text feature extraction model, a region segmentation module, an edge detection module, a morphology detection module, a feature fusion model, a cloud-based replacement model, an edge-side replacement model, and an edge-side elimination model.

[0127] refer to Figure 7This disclosure illustrates an image processing method that may include the following steps.

[0128] Step 1: Text detection and feature extraction.

[0129] This step involves Figure 7 The text detection module and text feature extraction model in the text.

[0130] For user-input images, text detection is first performed by a text detection module (such as an OCR function module), which mainly captures global text information. In the usage stage, by modeling the global text in a single step, the subsequent process can respond to the user's region selection and edit the text at different locations without having to select and model it separately, which effectively improves the execution efficiency of the function.

[0131] For text content recognized based on OCR technology, tokenization can be performed first, dividing each word into multiple token representations. For example, the recognized phrase "Xiaoming, the weather is really nice today" can be divided into phrases such as "Xiaoming", ",", "today", "weather", and "really nice". Each phrase can be converted into a corresponding integer sequence according to a predefined word list, and the integer sequence mapped to each phrase can be called a token.

[0132] Next, these tokens are fed into a text feature extraction model (such as a Transformer-structured module) for modeling. Five groups of Transformer-blocks at different scales can be designed. Each group reduces the size of the data features by downsampling and further extracts the content effectively through Attention. For multi-level features, the feature dimensions at different scales can be obtained by connecting them, i.e., Fd = {Fd1, Fd2, Fd3, Fd4}, where Fdi ∈ Ci × Hi × Wi represents the feature of the i-th layer of the backbone. All features are then connected using the Concat() function to obtain the text features of the image to be processed.

[0133] In this step, by constructing a Transformer-based structural modeling method, the detected text can be quickly modeled. Furthermore, after model quantization, this module has a smaller computational load, making it more conducive to deployment.

[0134] Step 2: Text region segmentation.

[0135] This step involves Figure 7 The region segmentation module in the system.

[0136] The text in the image to be processed can be segmented by calling the segmentation interface (i.e., the interface of the segmentation function structure already integrated on the device) on the device (i.e., the device on which this solution is implemented, such as a mobile phone or tablet), and the segmentation result, i.e., the region segmentation feature map of the image to be processed, can be obtained. Region segmentation features Figure 1 It can be used for modeling its own segmented regions, and it can also be sent to the edge detection module for use.

[0137] For text segmentation results, only the position information of the text and the color information of that position can be carried. This can greatly reduce the difficulty of modeling, so a smaller model structure can be designed to model this part.

[0138] In this step, the following objectives can be achieved: (1) By performing image segmentation on the text content, the text is regarded as part of the image, and its position is determined according to the principle of segmentation, so as to ensure that the content can be redrawn in the accurate position (i.e., the area corresponding to the text) during the subsequent redrawing process; (2) The segmented content can assist other modules (i.e., the edge detection module) in performing edge recognition and detection of the text content, which can avoid the error caused by directly detecting the whole image, and can avoid introducing additional computation, thereby improving the accuracy of edge detection and the processing efficiency of edge detection.

[0139] Step 3: Extract text edges.

[0140] This step involves Figure 7 The edge detection module and shape detection module in the middle.

[0141] Two detection methods can be used in this step. First, edge detection is achieved through an edge detection module. The Scribble model can be used to extract the text edges. The text edge extraction results are more blurred and broader than the results output by the morphology detection module. They include not only the contour edges within the original text but also the shadows related to the text content. In other words, the final result is content with relatively blurred edges. The text edge extraction results can be encoded using a VAE-Encoder quantization INT4 model to obtain the text edge feature map of the image to be processed. In subsequent steps, the text edge feature map can be fused with the region segmentation feature map and text features. This approach allows different text-related features to synergistically influence each other, ensuring the comprehensiveness and accuracy of the obtained text noise features.

[0142] Secondly, accurate edge extraction is achieved using a morphological detection module. This can be achieved by retraining the Canny model based on its structure. During training, inaccurate edge values ​​in the dataset can be corrected, ensuring the trained model accurately recognizes text outlines. To address the issue of pen punctuation marks or streaks at the beginning and end of strokes in handwritten text, which can cause residual text removal problems, an image dilation algorithm can be used to expand the edges by 0.1%, resulting in an expanded outline. This expanded outline can then be encoded using the same encoder, specifically a VAE-Encoder quantized INT4 model, to obtain text morphological features. These features are then used as guiding features in the image generation model, serving as a guide for image generation.

[0143] In traditional techniques, when text contains shadows or the handwritten text is abstract or distorted, the removal results often leave residual text. For example, when training the model with a large number of regular fonts, if the text in the image has residual shadows or the handwritten text is abstract, shape mismatch can cause removal failure.

[0144] This step achieves the following objectives: (1) to extract accurate text edge positions; and (2) to resolve the shadow issues that may occur in some handwritten text and generated image text content at this stage. In other words, this step can avoid the aforementioned problems.

[0145] Step 4: Feature fusion to generate text noise features.

[0146] This step involves Figure 7 The feature fusion model in the text integrates text features, region segmentation feature maps, and text edge feature maps.

[0147] First, the size of the region segmentation feature map is adjusted to 256x256. After two downsampling (DS) operations, it can be compressed to a scale of 32x32. Then, the text features extracted in the above process are concatenated with the text edge feature map. After concatenation, a feature size of 512x512 is obtained. Two convolutions are then used for transition to ensure that the concatenated features are distributed in the same feature space. This concatenated feature is the text noise feature, which can be used to remove noise in the subsequent image generation process.

[0148] During the concatenation process, Self-Attention can be used to combine the three sets of features. In Self-Attention, Q (query vector), K (key vector), and V (value vector) are obtained by calculating different features. To reduce computational cost, a space reduction operation with small K and small V can be introduced. That is, K and V are the smaller text features and the compressed region segmentation feature map, respectively, and the text edge feature map is treated as the query. This allows for more efficient and accurate image generation guided by text features and region segmentation feature maps.

[0149] Step 5: Image generation.

[0150] This step involves Figure 7 The cloud-based replacement model, the edge-side replacement model, and the edge-side elimination model are included.

[0151] This step can be divided into two parts: elimination and redrawing.

[0152] For the redraw function: Firstly, a finely tuned SD1.5 model is utilized, primarily through fine-tuning images containing text content to make the model more sensitive to text generation. During use, users can input the text prompt to be regenerated, such as "It's raining today," thus replacing the original content with new content.

[0153] For generating more text, the model can adaptively fill segmented regions with text size to achieve new image text content. This part of the model is implemented through both cloud and on-device deployment. The main reason is that text redrawing is difficult, and the original SD1.5 model's iterative denoising process can be retained, so the original model is deployed in the cloud for better generation results. On the device side, through compression encoding and quantization, the model inherently suffers some accuracy loss, but it can run entirely on on-device devices, achieving offline operation and better privacy. Furthermore, the fused text noise features can be used as initial noise for each model during image generation. For the model's Cross-Attention part, in addition to the input of text editing prompts, text morphological features can also be combined as guiding features to achieve more accurate position and content guidance, resulting in better text replacement effects.

[0154] For the image removal function: The implementation difficulty of the image removal function is relatively lower than that of the redraw function. Therefore, to ensure that this function is fully edge-based, a distillation process can be used on the SD1.5 series model. This process distills the generated images from multiple iterations into a single-step image generation process, achieving single-step diffusion denoising. Given an image and reference features, the model only needs to infer once to output the removal result. This design ensures edge-based deployment while accelerating the function's operation, eliminating long iteration times and providing a better user experience. Furthermore, the single-step inference process also uses text noise feature maps as noise for model initialization and text morphological features as guiding features. Additionally, the image removal function does not require user input of a prompt; the prompt can default to pre-defined vocabulary (such as image quality constraints).

[0155] Furthermore, in the model training process of this technical solution, two datasets can be used: Chinese and English datasets. For the English text dataset, ICDAR13FST, COCO_TS, MLT_S, Total-Text, etc., can be used; for the bilingual text dataset, BTS can be used. Model training can employ the AdamW optimizer, with an initial learning rate of 0.0005, weight decay of 0.01, and a batch size of 16. Additionally, data augmentation operations such as random pruning and flipping can be used during the training phase. All modules are jointly trained.

[0156] In some embodiments of this disclosure, the image processing method further includes: in response to receiving a text replacement instruction, invoking the cloud-based replacement model or the edge-side replacement model to generate the target image; and in response to receiving a text elimination instruction, invoking the edge-side elimination model to generate the target image.

[0157] In this embodiment of the disclosure, different function buttons may be displayed to the user for selection. For example, the function buttons may include a text replacement function button and a text elimination function button.

[0158] In response to the user's selection of the text replacement function, a text replacement instruction can be generated. At this time, either a cloud-based replacement model or a device-based replacement model can be invoked to generate the target image. The user can also be shown selection controls corresponding to the cloud-based and device-based replacement models, and the model invoked will be determined based on the user's selection.

[0159] In response to the user's selection of the text removal function, a text removal instruction can be generated, at which point the edge removal model can be invoked to generate the target image.

[0160] The embodiments disclosed herein provide users with a convenient image processing interaction method, enabling users to edit text in images through simple operations.

[0161] In some embodiments of this disclosure, obtaining text editing prompts includes: in response to receiving a text replacement instruction, obtaining a text replacement prompt input by the user as the text editing prompt; and in response to receiving a text deletion instruction, obtaining a preset text deletion prompt as the text editing prompt.

[0162] In this embodiment of the disclosure, in response to receiving a text replacement instruction, a prompt word input box can be displayed to the user, who can input a text replacement prompt word through the prompt word input box.

[0163] Through the embodiments of this disclosure, corresponding text editing prompts (user-inputted replacement prompts or preset removal prompts) can be obtained based on the received instruction type (text replacement or text removal), enabling the system to accurately understand the user's editing intent and perform corresponding image processing operations accordingly. This design improves the interactivity and flexibility of the system, allowing users to more conveniently edit and modify text in images.

[0164] In some embodiments of this disclosure, the image processing method further includes: generating a target image after the text has been replaced in response to the text editing prompt being a text replacement prompt; wherein the text replacement prompt indicates the text information in the image to be replaced; and generating a target image after the text has been removed in response to the text editing prompt being a text removal prompt; wherein the text removal prompt indicates quality description information for the target image.

[0165] In this embodiment of the disclosure, when the obtained text editing prompt is a text replacement prompt, it means that the user wants to replace a certain text content in the image with other content. The model will generate the target image after the text has been replaced, and then return it to the terminal for display. When the obtained text editing prompt is a text removal prompt, it means that the user wants to remove a certain text content from the image. The model will generate the target image after the text has been removed, and then return it to the terminal for display.

[0166] Through the embodiments of this disclosure, by accurately responding to different types of text editing prompts (text replacement prompts or text removal prompts) and performing corresponding image processing operations (replacement or removal) based on the specific content of these prompts, this flexible and precise image processing method can efficiently meet users' diverse needs for text editing in images and enhance users' image editing experience.

[0167] In some embodiments of this disclosure, the image processing method further includes: displaying a plurality of text regions identified in the image to be processed; and, in response to a selection operation and an editing operation of a target text region in the plurality of text regions, displaying a target image after the text in the target text region has been edited.

[0168] In this embodiment of the disclosure, after identifying text in an image, multiple identified text regions can be displayed to the user. The user can select the text region to be edited via buttons or menus on the interface, and can also select the desired editing function, such as text replacement and text removal functions. If the text replacement function is selected, the user can then input the desired text editing prompt. Then, after generating the target image based on the user's input and instructions by calling the corresponding model, the target image is displayed to the user.

[0169] Figure 8 This is an interactive process for replacing text in an image in an image processing method according to some embodiments of the present disclosure.

[0170] like Figure 8 As shown, the interactive process of replacing text in an image may include the following steps.

[0171] Step S810: Receive the user's selection of the text replacement function and generate a text replacement instruction.

[0172] Step S820: In response to the text replacement instruction, a prompt input box is displayed to obtain the text replacement prompt entered by the user through the prompt input box; wherein, the text replacement prompt indicates the text information in the image to be replaced.

[0173] Step S830: In response to the text replacement instruction, call the cloud-based replacement model or the terminal-side replacement model.

[0174] Step S840: Generate the target image after text replacement based on the text replacement prompt words using the called model.

[0175] Step S850: Display the target image.

[0176] The specific implementation methods for each of the above steps have been described in detail in the embodiments of the method, and will not be elaborated here.

[0177] Figure 9 This is an interactive process for removing text from an image in another image processing method according to some embodiments of the present disclosure.

[0178] like Figure 9 As shown, the interactive process for removing text from an image may include the following steps.

[0179] Step S910: Receive the user's selection of the text elimination function and generate a text elimination instruction.

[0180] Step S920: In response to the text removal instruction, a preset text removal prompt word is obtained; wherein the text removal prompt word indicates quality description information for the target image.

[0181] Step S930: In response to the text elimination instruction, the cloud-based replacement terminal elimination model is invoked.

[0182] Step S940: Generate the target image after text removal based on the text removal prompt words using the invoked model.

[0183] Step S950: Display the target image.

[0184] The specific implementation methods for each of the above steps have been described in detail in the embodiments of the method, and will not be elaborated here.

[0185] It should be noted that the above figures are merely illustrative representations of the processes included in methods according to some embodiments of this disclosure, and are not intended to be limiting. It is readily understood that the processes shown in the above figures do not indicate or limit the temporal order of these processes. Furthermore, it is readily understood that these processes may be executed synchronously or asynchronously, for example, in multiple modules.

[0186] The following are embodiments of the apparatus disclosed herein, which can be used to execute embodiments of the method disclosed herein. For details not disclosed in the apparatus embodiments of this disclosure, please refer to the embodiments of the method disclosed herein.

[0187] Figure 10 This is a block diagram illustrating an image processing apparatus according to some embodiments of the present disclosure. (Refer to...) Figure 10 The device includes: an acquisition unit 1001, a noise feature generation unit 1002, an image generation unit 1003, a morphology detection unit 1004, a model calling unit 1005, a display unit 1006, and a training unit 1007.

[0188] The acquisition unit 1001 is used to acquire an image to be processed, which includes text; the noise feature generation unit 1002 is used to perform multi-dimensional recognition processing on the text in the image to be processed to obtain the text noise features of the image to be processed; the acquisition unit 1001 is also used to acquire text editing prompts; the image generation unit 1003 is used to perform image denoising and redrawing based on the text editing prompts and the text noise features to generate the target image after the text has been edited.

[0189] In some embodiments of this disclosure, the noise feature generation unit 1002 performs multi-dimensional recognition processing on the text in the image to be processed to obtain text noise features of the image to be processed, including: performing text recognition on the image to be processed to obtain text features of the image to be processed; performing position recognition on the text in the image to be processed to obtain a region segmentation feature map of the image to be processed; performing edge detection on the text in the image to be processed to obtain a text edge feature map of the image to be processed; and fusing the text features, the region segmentation feature map, and the text edge feature map to obtain text noise features of the image to be processed.

[0190] In some embodiments of this disclosure, the image generation unit 1003 performs image denoising and redrawing based on the text editing prompt and the text noise features to generate a target image after the text has been edited. This includes: inputting the text editing prompt and the text noise features into an image generation model; wherein the image generation model is obtained by adjusting and training an initial stable diffusion model; performing semantic understanding on the text editing prompt through the image generation model to determine the semantic features for text editing and the target text to be edited; and processing the target text based on the text noise features and the semantic features through the image generation model to denoise and redraw the corresponding region of the target text in the image to be processed, thereby generating a target image after the target text has been edited.

[0191] In some embodiments of this disclosure, the morphology detection unit 1004 is used to perform morphology detection on the text in the image to be processed to obtain the text morphology features of the image to be processed; the image generation unit 1003 is further used to input the text morphology features into the image generation model; and the image generation model uses the text morphology features as guiding information for denoising and redrawing to generate the target image after the target text has been edited.

[0192] In some embodiments of this disclosure, the morphology detection unit 1004 performs morphology detection on the text in the image to be processed to obtain the text morphology features of the image to be processed, including: extracting text contours from the text in the image to be processed to obtain a text contour extraction result; expanding the text contour extraction result to obtain a contour expansion result; and encoding the contour expansion result to obtain the text morphology features.

[0193] In some embodiments of this disclosure, the image generation model includes a cloud-based replacement model, an edge-side replacement model, and an edge-side elimination model; wherein, the cloud-based replacement model is obtained by adjusting and training the initial stable diffusion model using a second sample image containing text; the edge-side replacement model is obtained by compressing, encoding, and quantizing the cloud-based replacement model; and the edge-side elimination model is obtained by distilling the initial stable diffusion model.

[0194] In some embodiments of this disclosure, the image processing apparatus further includes a model invocation unit 1005, configured to: in response to receiving a text replacement instruction, invoke the cloud-based replacement model or the edge-side replacement model to generate the target image; and in response to receiving a text elimination instruction, invoke the edge-side elimination model to generate the target image.

[0195] In some embodiments of this disclosure, the acquisition unit 1001 acquires text editing prompts, including: in response to receiving a text replacement instruction, acquiring a text replacement prompt input by the user as the text editing prompt; and in response to receiving a text elimination instruction, acquiring a preset text elimination prompt as the text editing prompt.

[0196] In some embodiments of this disclosure, the image generation unit 1003 is further configured to: generate a target image after the text has been replaced in response to the text editing prompt being a text replacement prompt; wherein the text replacement prompt indicates the text information in the image to be replaced; and generate a target image after the text has been removed in response to the text editing prompt being a text removal prompt; wherein the text removal prompt indicates quality description information for the target image.

[0197] In some embodiments of this disclosure, the display unit 1006 is configured to: display a plurality of text regions identified in the image to be processed; and, in response to a selection operation and an editing operation of a target text region in the plurality of text regions, display a target image after the text in the target text region has been edited.

[0198] In some embodiments of this disclosure, the text features are output by a trained text feature extraction model; the text noise features are output by a trained feature fusion model; wherein, the acquisition unit 1001 is further configured to: acquire a first sample image containing text, sample prompt words, and a label image matching the first sample image and the sample prompt words; the noise feature generation unit 1002 is further configured to: perform text recognition on the first sample image using the initial text feature extraction model to obtain sample text features of the first sample image; the noise feature generation unit 1002 is further configured to: perform position recognition on the text in the first sample image to obtain a sample region segmentation feature map ... Edge detection is performed on the text in the first sample image to obtain the sample text edge feature map of the first sample image; the noise feature generation unit 1002 is further used to fuse the sample text features, the sample region segmentation feature map and the sample text edge feature map using the initial feature fusion model to obtain the sample text noise features of the first sample image; the image generation unit 1003 is further used to perform image denoising and redrawing based on the sample prompt words and the sample text noise features to generate a prediction image; the training unit 1007 is used to: construct a loss function based on the prediction image and the label image, and use the loss function to train the initial text feature extraction model and the initial feature fusion model to obtain the trained text feature extraction model and the trained feature fusion model.

[0199] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0200] Figure 11 This is a block diagram illustrating an apparatus 1100 for image processing according to some embodiments of the present disclosure. For example, apparatus 1100 may be a mobile phone, computer, digital broadcasting terminal, messaging device, game console, tablet device, medical device, fitness equipment, personal digital assistant, etc.

[0201] Reference Figure 11 The device 1100 may include one or more of the following components: a processing component 1102, a memory 1104, a power component 1106, a multimedia component 1108, an audio component 1110, an input / output (I / O) interface 1112, a sensor component 1114, and a communication component 1116.

[0202] Processing component 1102 typically controls the overall operation of device 1100, such as operations associated with display, telephone calls, data communication, camera operation, and recording operations. Processing component 1102 may include one or more processors 1120 to execute instructions to perform all or part of the steps of the methods described above. Furthermore, processing component 1102 may include one or more modules to facilitate interaction between processing component 1102 and other components. For example, processing component 1102 may include a multimedia module to facilitate interaction between multimedia component 1108 and processing component 1102.

[0203] Memory 1104 is configured to store various types of data to support the operation of device 1100. Examples of such data include instructions for any application or method operating on device 1100, contact data, phonebook data, messages, pictures, videos, etc. Memory 1104 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0204] The power supply component 1106 provides power to the various components of the device 1100. The power supply component 1106 may include a power management system, one or more power sources, and other components associated with generating, managing, and distributing power to the device 1100.

[0205] Multimedia component 1108 includes a screen that provides an output interface between the device 1100 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touchscreen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensors may sense not only the boundaries of the touch or swipe action but also the duration and pressure associated with the touch or swipe operation. In some embodiments, multimedia component 1108 includes a front-facing camera and / or a rear-facing camera. When the device 1100 is in an operating mode, such as a shooting mode or a video mode, the front-facing camera and / or the rear-facing camera may receive external multimedia data. Each front-facing camera and rear-facing camera may be a fixed optical lens system or have focal length and optical zoom capabilities.

[0206] Audio component 1110 is configured to output and / or input audio signals. For example, audio component 1110 includes a microphone (MIC) configured to receive external audio signals when device 1100 is in an operating mode, such as call mode, recording mode, and voice recognition mode. The received audio signals may be further stored in memory 1104 or transmitted via communication component 1116. In some embodiments, audio component 1110 also includes a speaker for outputting audio signals.

[0207] I / O interface 1112 provides an interface between processing component 1102 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include, but are not limited to, home buttons, volume buttons, power buttons, and lock buttons.

[0208] Sensor assembly 1114 includes one or more sensors for providing status assessments of various aspects of device 1100. For example, sensor assembly 1114 may detect the on / off state of device 1100, the relative positioning of components such as the display and keypad of device 1100, changes in the position of device 1100 or a component of device 1100, the presence or absence of user contact with device 1100, the orientation or acceleration / deceleration of device 1100, and temperature changes of device 1100. Sensor assembly 1114 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. Sensor assembly 1114 may also include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, sensor assembly 1114 may also include an accelerometer, a gyroscope, a magnetometer, a pressure sensor, or a temperature sensor.

[0209] Communication component 1116 is configured to facilitate wired or wireless communication between device 1100 and other devices. Device 1100 can access wireless networks based on communication standards, such as WiFi, 3G, 4G, 5G, other communication standards, or combinations thereof. In some embodiments of this disclosure, communication component 1116 receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In some embodiments of this disclosure, communication component 1116 further includes a near-field communication (NFC) module to facilitate short-range communication. For example, the NFC module may be implemented based on radio frequency identification (RFID) technology, Infrared Data Association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.

[0210] In some embodiments of this disclosure, the apparatus 1100 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the methods described above.

[0211] In some embodiments of this disclosure, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 1104 including instructions, which can be executed by a processor 1120 of device 1100 to perform the above-described method. For example, the non-transitory computer-readable storage medium may be a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device, etc.

[0212] A non-transitory computer-readable storage medium, when instructions in the storage medium are executed by a processor of a mobile terminal, enables the mobile terminal to perform an image processing method, the method comprising: acquiring an image to be processed, the image to be processed including text; performing multi-dimensional recognition processing on the text in the image to be processed to obtain text noise features of the image to be processed; acquiring text editing prompts; and performing image denoising and redrawing based on the text editing prompts and the text noise features to generate a target image after the text has been edited.

[0213] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the following claims.

[0214] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. An image processing method, characterized in that, include: Obtain an image to be processed, wherein the image to be processed includes text; Multidimensional recognition processing is performed on the text in the image to be processed to obtain the text noise features of the image to be processed; Get text editing prompts; Based on the text editing prompts and the text noise features, the image is denoised and redrawn to generate the target image after the text has been edited.

2. The method according to claim 1, characterized in that, Multidimensional recognition processing is performed on the text in the image to be processed to obtain the text noise features of the image to be processed, including: Text recognition is performed on the image to be processed to obtain the text features of the image to be processed; The text in the image to be processed is located to obtain a region segmentation feature map of the image to be processed. Edge detection is performed on the text in the image to be processed to obtain the text edge feature map of the image to be processed; The text features, the region segmentation feature map, and the text edge feature map are fused to obtain the text noise features of the image to be processed.

3. The method according to claim 1, characterized in that, Based on the text editing prompts and the text noise features, image denoising and redrawing are performed to generate the target image after the text has been edited, including: The text editing prompts and text noise features are input into the image generation model; wherein the image generation model is obtained by adjusting and training an initial stable diffusion model; The image generation model is used to perform semantic understanding of the text editing prompts to determine the semantic features of the text to be edited and the target text to be edited in the text. The image generation model processes the text noise features and semantic features to denoise and redraw the region corresponding to the target text in the image to be processed, thereby generating a target image after the target text has been edited.

4. The method according to claim 3, characterized in that, The method further includes: The text in the image to be processed is subjected to morphological detection to obtain the text morphological features of the image to be processed; The text morphological features are input into the image generation model; The image generation model uses the text morphological features as guiding information for denoising and redrawing to generate the target image after the target text has been edited.

5. The method according to claim 4, characterized in that, The text in the image to be processed is subjected to morphological detection to obtain the text morphological features of the image to be processed, including: Text contour extraction is performed on the text in the image to be processed to obtain the text contour extraction result; The extracted text contours are expanded to obtain expanded contours. The outline expansion result is encoded to obtain the text morphological features.

6. The method according to claim 3, characterized in that, The image generation model includes a cloud-based replacement model, an edge-side replacement model, and an edge-side elimination model; The cloud-based replacement model is obtained by adjusting and training the initial stable diffusion model using a second sample image containing text. The edge-side replacement model is obtained by compressing, encoding, and quantizing the cloud-based replacement model; The end-side elimination model is obtained by distilling the initial stable diffusion model.

7. The method according to claim 6, characterized in that, The method further includes: In response to receiving a text replacement instruction, the target image is generated by invoking the cloud-based replacement model or the edge-side replacement model. In response to receiving a text removal instruction, the end-side removal model is invoked to generate the target image.

8. The method according to claim 1, characterized in that, Get text editing prompts, including: In response to receiving a text replacement instruction, the system obtains the text replacement suggestion word entered by the user and uses it as the text editing suggestion word; In response to receiving a text deletion instruction, a preset text deletion prompt word is obtained and used as the text editing prompt word.

9. The method according to claim 1 or 8, characterized in that, The method further includes: In response to the text editing prompt being a text replacement prompt, a target image after the text has been replaced is generated; wherein, the text replacement prompt indicates the text information in the image to be replaced. In response to the text editing prompt being a text removal prompt, a target image after the text has been removed is generated; wherein the text removal prompt indicates quality description information for the target image.

10. The method according to claim 1, characterized in that, The method further includes: Multiple text regions identified in the image to be processed are displayed; In response to selection and editing operations of target text regions in the plurality of text regions, the target image after the text in the target text region has been edited is displayed.

11. The method according to claim 2, characterized in that, The text features are output by a trained text feature extraction model; the text noise features are output by a trained feature fusion model. The method further includes: Obtain a first sample image containing text, a sample prompt word, and a label image that matches the first sample image and the sample prompt word; The first sample image is used to perform text recognition using an initial text feature extraction model to obtain sample text features of the first sample image; the position of the text in the first sample image is identified to obtain sample region segmentation feature map of the first sample image; and edge detection is performed on the text in the first sample image to obtain sample text edge feature map of the first sample image. The initial feature fusion model is used to fuse the sample text features, the sample region segmentation feature map, and the sample text edge feature map to obtain the sample text noise features of the first sample image; Based on the sample prompt words and the sample text noise features, the image is denoised and redrawn to generate a predicted image; A loss function is constructed based on the predicted image and the labeled image. The initial text feature extraction model and the initial feature fusion model are trained using the loss function to obtain the trained text feature extraction model and the trained feature fusion model.

12. An image processing apparatus, characterized in that, include: An acquisition unit is used to acquire an image to be processed, wherein the image to be processed includes text; The noise feature generation unit is used to perform multi-dimensional recognition processing on the text in the image to be processed to obtain the text noise features of the image to be processed. The acquisition unit is also used to acquire text editing prompts; The image generation unit is used to perform image denoising and redrawing based on the text editing prompts and the text noise features to generate the target image after the text has been edited.

13. An electronic device, characterized in that, include: processor; Memory used to store processor-executable instructions; The processor is configured to implement the steps of the method according to any one of claims 1-11.

14. A non-transitory computer-readable storage medium, wherein instructions in the storage medium, when executed by a processor of a mobile terminal, enable the mobile terminal to perform an image processing method, the method comprising: Obtain an image to be processed, wherein the image to be processed includes text; Multidimensional recognition processing is performed on the text in the image to be processed to obtain the text noise features of the image to be processed; Get text editing prompts; Based on the text editing prompts and the text noise features, the image is denoised and redrawn to generate the target image after the text has been edited.

Images