Image editing method and device, storage medium and program product

Abstract

The embodiment of the invention provides an image editing method and device, a storage medium and a program product. In the embodiment of the invention, a plurality of first interface elements pointing to different style reference information and style generation modes and a plurality of second interface elements pointing to different copywriting reference information and copywriting generation modes are provided on an image editing page; the style reference information and the style generation mode are obtained through the first interface element, the copywriting reference information and the text style generation mode are obtained through the second interface element, flexible selection is provided for users, and different user requirements are met; the image-text editing model is called from the two dimensions of the text style and the copywriting content for text editing, automation of text editing is achieved, consumed time is short, and the text editing effect is good.

Description

Technical Field

[0001] The present application relates to the field of Internet technology, and in particular to an image editing method, device, storage medium and program product. Background Art

[0002] With the rapid development of e-commerce, product visuals have become a key factor influencing consumer purchasing decisions. The effective integration of marketing copy and product imagery can effectively convey a product's core selling points and differentiation. Especially during e-commerce promotion cycles and seasonal operations, merchants have a marketing need to frequently update product images.

[0003] Currently, merchants on e-commerce platforms manually edit text information in product images using photo editing software, which is time-consuming and requires professional skills. In addition, the text templates and text styles provided by existing product image production tools are relatively simple and not flexible enough. Summary of the Invention

[0004] Various aspects of the present application provide an image editing method, device, storage medium, and program product to enhance the flexibility of image editing in terms of text style and content.

[0005] An embodiment of the present application provides an image editing method, including: displaying an image editing page for editing at least one original image, the image editing page including: multiple first interface elements and multiple second interface elements; different first interface elements point to different style reference information and style generation methods, and different second interface elements point to different copy reference information and copy generation methods; in response to a first user operation on a target first interface element, obtaining target style reference information and target style generation method pointed to by the target first interface element, and generating a target style image according to the target style reference information and target style generation method; in response to a second user operation on a target second interface element, obtaining target copy reference information and target copy generation method pointed to by the target second interface element, and generating target copy content according to the target copy reference information and target copy generation method; calling a graphic and text editing model according to the target copy content and the target text style in the target style image, and performing text editing processing on the to-be-edited areas in the at least one original image respectively to obtain at least one target image.

[0006] An embodiment of the present application also provides an electronic device, comprising: a memory and a processor; the memory is used to store a computer program; the processor is coupled to the memory and is used to execute the computer program to implement each step in the image editing method provided in the embodiment of the present application.

[0007] An embodiment of the present application also provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, the processor implements the steps of the image editing method provided in the embodiment of the present application.

[0008] An embodiment of the present application also provides a computer program product, including a computer program / instruction. When the computer program / instruction is executed by a processor, the processor implements the steps of the image editing method provided in the embodiment of the present application.

[0009] In an embodiment of the present application, a plurality of first interface elements pointing to different style reference information and style generation methods, as well as a plurality of second interface elements pointing to different text reference information and text generation methods are provided on the image editing page; the style reference information and style generation method are obtained through the first interface elements, and the text reference information and text style generation method are obtained through the second interface elements, providing users with flexible choices to adapt to different user needs; the graphic editing model is called to perform text editing from two dimensions of text style and text content, realizing the automation of text editing, which saves time and has better text editing effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0011] Figure 1 A flowchart of an image editing method provided by an exemplary embodiment of the present application;

[0012] Figure 2 A schematic diagram of selecting a region to be edited for an original image provided by an exemplary embodiment of the present application;

[0013] Figure 3 A schematic structural diagram of a graphic and text editing model provided for an exemplary embodiment of the present application;

[0014] Figure 4 A flowchart of another image editing method provided by an exemplary embodiment of the present application;

[0015] Figure 5 A schematic structural diagram of an electronic device provided as an exemplary embodiment of the present application. DETAILED DESCRIPTION

[0016] To make the purpose, technical solutions, and advantages of this application more clear, the technical solutions of this application will be clearly and completely described below in conjunction with the specific embodiments of this application and the corresponding drawings. Obviously, the embodiments described are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0017] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0018] The various models involved in this application (including but not limited to language models or large models) are in compliance with relevant laws and standards.

[0019] Currently, traditional text editing tools have some limitations:

[0020] 1) Operational expertise barriers: Merchants on e-commerce platforms manually use photo editing tools to add / modify text in product images, which is time-consuming and requires professional skills, and cannot meet high-frequency marketing needs.

[0021] 2) Lack of style customization: There is a lack of intelligent text style adaptation, and visual elements such as fonts, colors, and layouts cannot be dynamically matched with product characteristics and store tone.

[0022] 3) Lack and limitations of copywriting styles: When merchants create copywriting for product images, they are often constrained by both the creation tools and their own creativity. Furthermore, product image creation tools provide uniform copywriting templates and style options, but are unable to intelligently recommend copywriting information for different product images or different stores. This lacks flexibility and makes it difficult to meet the diverse and personalized needs of merchants.

[0023] 4) Insufficient batch processing capabilities: It is impossible to batch modify text on multiple images (for example, replacing promotional dates for multiple product images), which affects the image editing efficiency of marketing activities.

[0024] In an embodiment of the present application, a plurality of first interface elements pointing to different style reference information and style generation methods, as well as a plurality of second interface elements pointing to different text reference information and text generation methods are provided on the image editing page; the style reference information and style generation method are obtained through the first interface elements, and the text reference information and text style generation method are obtained through the second interface elements, providing users with flexible choices to adapt to different user needs; the graphic editing model is called to perform text editing from two dimensions of text style and text content, realizing the automation of text editing, which saves time and has better text editing effect.

[0025] Furthermore, when using a multimodal approach to generate candidate copy content, the multimodal model is invoked based on the original copy content in the original image, the original product information in the original image, and its associated product information to intelligently recommend candidate copy content that is compatible with the original image. This allows for more targeted, personalized copy recommendations based on the original copy information and product information in the original image. Furthermore, by incorporating other product information associated with the original product, a variety of copy recommendations can be made to users, thus meeting the personalized and diverse needs of merchants.

[0026] Similarly, when using a multimodal generation method to generate candidate text styles, based on the original copy information and product information in the original image, text styles can be recommended to users in a more targeted manner. In addition, by introducing other product information associated with the original product, a variety of text styles can be recommended to users, thus meeting the personalized and diversified needs of merchants.

[0027] Furthermore, for multiple original images, when the target text content and target style image are determined, text editing can be performed in batches to improve processing efficiency and simplify user operations.

[0028] A solution provided by an embodiment of the present application is described in detail below with reference to the accompanying drawings.

[0029] Figure 1 This is a flow chart of a method for editing text and images provided by an exemplary embodiment of the present application. Figure 1 As shown, the method includes:

[0030] 101. Display an image editing page for editing at least one original image, the image editing page comprising: a plurality of first interface elements and a plurality of second interface elements; different first interface elements point to different style reference information and style generation methods, and different second interface elements point to different text reference information and text generation methods.

[0031] 102. In response to a first user operation on a target first interface element, obtain target style reference information and a target style generation method pointed to by the target first interface element, and generate a target style image according to the target style reference information and the target style generation method.

[0032] 103. In response to a second user operation on the target second interface element, obtain target copy reference information and a target copy generation method pointed to by the target second interface element, and generate target copy content according to the target copy reference information and the target copy generation method.

[0033] 104. Based on the target text content and the target text style in the target style image, call the image-text editing model to perform text editing processing on the to-be-edited areas in the at least one original image to obtain at least one target image.

[0034] In this embodiment, the original image is an image with image editing requirements. The original image includes a text area to be edited, referred to as the area to be edited, and the area to be edited is used to carry text information. For example, before image editing is performed on the original image, the area to be edited may carry original text information. By executing the image editing method on the original image, the original text information in the area to be edited can be replaced with target text information. The target text information includes target copy content and target text style. The text style refers to a collection of a series of text attributes that give the text a specific visual effect during the design and typesetting process. The text style may include but is not limited to: font attributes, color attributes, and typesetting attributes. The font attributes may include but are not limited to: font family, font weight, and font style. The color features may include but are not limited to: red, green, and blue (RGB) values, gradient mode, and transparency. The typesetting parameters may include but are not limited to: character spacing, line height, and alignment.

[0035] The target copy content has a copy style, which is a semantic expression method and linguistic characteristics of the content. For example, the copy style may include but is not limited to: emotional tendency, sentence structure (short slogan type / scenario description type), keyword distribution (promotional type / functional type / emotional type), etc. Among them, emotional tendency may include but is not limited to: cheerful, professional or friendly, etc. Sentence structure may include but is not limited to: short slogan type or scenario description type, etc. Keyword distribution may include but is not limited to: promotional type, functional type and emotional type, etc. The target copy content may include a combination of one or more of words, numbers, letters and symbols.

[0036] In this embodiment, an image editing page for editing at least one original image is displayed. The implementation of the image editing page is not limited. For example, the image editing page can be implemented as an application page of an e-commerce application, or as a web page. The image editing page can include an upload control, and a user can initiate an image upload operation using the upload control. Accordingly, the image editing page can respond to the image upload operation and obtain at least one original image uploaded by the user.

[0037] In this embodiment, the image editing page includes: multiple first interface elements and multiple second interface elements. The implementation method of the first interface element is not limited. For example, the first interface element can be implemented as: a button, a control, a card, a slider, an input box, and a pop-up box, etc. The implementation method of the second interface element is also not limited. For example, the first interface element can be implemented as: a button, a control, a selector, a card, a slider, an input box, and a pop-up box, etc. The implementation methods of the first page element and the second interface element can be the same or different, and there is no limitation on this. The number of first interface elements can be 1 or multiple, such as 2 or 3. The number of second interface elements can be 1 or multiple, such as 3 or 5.

[0038] In this embodiment, multiple style generation methods are provided. Different style generation methods can be based on different style reference information. A style generation method is a method for indicating how to generate a style image based on the style reference information. A style image is an image that can reflect a specific text style. For example, a specific text style can include font style, size, or color. Different first interface elements point to different style reference information and style generation methods. For example, first interface element A1 points to style reference information B1 and style generation method C1; first interface element A2 points to style reference information B2 and style generation method C2.

[0039] In this embodiment, users can initiate user actions on the image editing page. Depending on the application scenario, the user may be different. For example, in an e-commerce scenario, the user may be a merchant; in a lifestyle service platform, the user may be a service provider. User actions include, but are not limited to, touch, gesture, voice, head movement, eye movement, and other interaction methods. Touch actions include, but are not limited to, single-click, double-click, long-press, and swipe.

[0040] In this embodiment, the user can initiate a first user operation on a target first interface element among multiple first interface elements as desired. For ease of distinction and description, the first interface element on which the first user operation is executed is referred to as the target first interface element. The first user operation can be any one of the above-mentioned user operation examples, without limitation. In response to the first user operation on the target first interface element, the image editing page can obtain the target style reference information and target style generation method pointed to by the target first interface element, and generate a target style image based on the target style reference information and the target style generation method, where the target style image has the target text style. For example, the target style image is generated based on the target style reference information as indicated by the target style generation method.

[0041] In this embodiment, multiple copywriting generation methods are also provided. Each copywriting generation method utilizes different copywriting reference information, and the copywriting generation method indicates how to generate the target copywriting content based on the copywriting reference information. Different second interface elements point to different copywriting reference information and copywriting generation methods. For example, second interface element A3 points to copywriting reference information B3 and copywriting generation method C3; second interface element A4 points to copywriting reference information B4 and copywriting generation method C4.

[0042] In this embodiment, the user can perform a second user operation on a target second interface element among multiple second interface elements according to his or her own wishes. For the sake of distinction and description, the second interface element on which the second user operation is to be performed is referred to as the target second interface element. The second user operation can be any one of the above-mentioned user operation examples, and this is not limited to this. The image editing page can respond to the second user operation on the target second interface element, obtain the target copy reference information and the target copy generation method pointed to by the target second interface element, and generate the target copy content according to the target copy reference information and the target copy generation method. For example, according to the instructions of the target copy generation method, the target copy content is generated according to the target copy reference information.

[0043] This embodiment provides a target image-text editing model that can perform text editing on a to-be-edited region of an original image, replacing the original text in the to-be-edited region with target text, or adding target text to the target text region. The image-text editing model can implement controllable editing of the to-be-edited region using an inpainting algorithm.

[0044] The implementation of the image-text editing model is not limited. For example, the image-text editing model can be implemented as a Stable Diffusion model or a Stable Diffusion eXtended Large (SDXL) model. The target text content can be injected into the image-text editing model as text conditional information to guide the model in learning text morphology and understanding text content. The target text style in the target style image can also be injected into the image-text editing model as style conditional information to help the model understand text style.

[0045] In this embodiment, according to the target text content and the target text style in the target style image, the graphic editing model is called to perform text editing processing on the to-be-edited areas in at least one original image to obtain at least one target image.

[0046] In the glyph dimension, the target text content is used to ensure the accuracy of image and text editing; in the style dimension, by injecting a target style image, the target image adopts the target text style, improving the usability of image editing. Glyph control across these two dimensions, target text content and target text style, improves the quality of the resulting target image. This not only supports the editing of numbers and English letters, but also has excellent generation effects in Chinese and other languages. Accurate glyph control and high fidelity ensure the accuracy of the generated text glyphs, and the newly generated text is highly integrated with the original image, creating a natural and seamless effect.

[0047] Optionally, the image and text editing model includes: an image feature extraction network, a multi-scale feature extraction network, and an image feature conversion network.

[0048] The image feature extraction network can be any network capable of extracting image features, and the image feature conversion network is a network capable of restoring image features to an image. The image feature extraction network and the image feature conversion network can be implemented as an encoder and decoder, respectively. For example, the encoder and decoder involved in models such as Convolutional Neural Networks (CNN), Residual Networks (ResNet), and Variational Autoencoders (VAE).

[0049] The multi-scale feature extraction network can be any network capable of performing multi-scale feature extraction. For example, the multi-scale feature extraction network can be a downsampling-upsampling architecture based on a skip connections mechanism, such as a Unet network.

[0050] exist Figure 3In the figure, the internal structure of the initial image and text editing model during the model training process is illustrated as an example, wherein the image feature extraction network is a VAE encoder, the image feature conversion network is a VAE decoder, and the multi-scale feature extraction network is a Unet model as an example, but it is not limited to this.

[0051] besides, Figure 3 The text encoding model for extracting text encoding features and the style feature extraction model for extracting text style features are included. Detailed descriptions of extracting text encoding features and text style features can be found in the subsequent embodiments and will not be repeated here.

[0052] In an embodiment of the present application, a plurality of first interface elements pointing to different style reference information and style generation methods, as well as a plurality of second interface elements pointing to different text reference information and text generation methods are provided on the image editing page; the style reference information and style generation method are obtained through the first interface elements, and the text reference information and text style generation method are obtained through the second interface elements, providing users with flexible choices to adapt to different user needs; the graphic editing model is called to perform text editing from two dimensions of text style and text content, realizing the automation of text editing, which saves time and has better text editing effect.

[0053] In an optional embodiment, in response to a second user operation on a target second interface element, the implementation method for obtaining the target copy reference information and the target copy generation method pointed to by the target second interface element is not limited. Depending on the implementation method of the second interface element, the implementation method for obtaining the target copy reference information and the target copy generation method pointed to by the target second interface element in response to the second user operation on the target second interface element may also vary.

[0054] Exemplarily, the plurality of second interface elements include: a text generation control and an input box control. The second user operation can be implemented as a touch operation and an input operation, etc. The touch operation includes but is not limited to: a single operation, a double-click operation, a long press operation, and a sliding operation.

[0055] The text generation control refers to the multimodal information corresponding to at least one original image and the first multimodal generation method. For ease of distinction and description, the method of generating text content using multimodal information is referred to as the first multimodal generation method. Multimodal information may include, but is not limited to, image information and text information. For example, the multimodal information corresponding to at least one original image includes at least one original image and text information included in the at least one original image.

[0056] The input box control refers to the target text content and text input method. The target text content refers to the text content entered by the user through the input box control. For ease of distinction and description, the text generation method that generates the target copy content based on the target text content entered by the user is referred to as the text input method.

[0057] The user can initiate a second user operation on the text generation control or the input box control according to their own wishes or needs. In the case where the user initiates a touch operation on the text generation control, the image editing page responds to the touch operation on the text generation control and uses the multimodal information corresponding to at least one original image pointed to by the text generation control and the first multimodal generation method as the target text reference information and the target text generation method; or, in the case where the user initiates a touch operation on the input box control, the image editing page responds to the input operation on the input box control and obtains the target text content input into the input box control and the text input method pointed to by the input box control, and uses the target text content and the text input method as the target text reference information and the target text generation method.

[0058] Among them, different second interface elements are provided to users. Based on the first multimodal generation method, rich, diverse and creative candidate copy content can be generated for users, and diversified copy content can be generated. Based on the text input method, the copy content required by the user can be quickly generated, allowing users to choose as needed, thereby improving the flexibility of image editing.

[0059] Optionally, the implementation method of generating the target copy content is not limited according to the target copy reference information and the target copy generation method. The implementation method of generating the target copy content is also different according to the target copy reference information and the target copy generation method. The following exemplifies an implementation method of generating the target copy content according to the target copy reference information and the target copy generation method, including: when the target copy generation method is the first multimodal generation method, based on the multimodal information corresponding to at least one original image, calling the first multimodal model to generate at least one candidate copy content, outputting at least one candidate copy content for the user to select, and using the candidate copy content selected by the user as the target copy content.

[0060] For ease of distinction and description, the multimodal generative model that generates candidate copy content is referred to as the first multimodal generative model. A first multimodal model is one that can process and fuse data from different modalities. Data from different modalities can include text, images, audio, or video. The first multimodal model receives data from both image and text modalities, integrating visual and textual information to improve the compatibility of generated candidate copy content with the original image.

[0061] The implementation of the first multimodal generative model is not limited. For example, the first multimodal generative model may include, but is not limited to: Generative Adversarial Networks (GAN), Convolutional Neural Networks (CNN), Recurrent Neural Networks (RNN), Deep Neural Networks (DNN), Residual Networks, Encoder-Only architecture, Decoder-Only architecture, Encoder-Decoder architecture, and architecture based on self-attention mechanism, etc.

[0062] The internal implementation structure of the first multimodal generation model is not limited. For example, the first multimodal model includes: a feature extraction network, a feature fusion network, and a copy generation network. The feature extraction network is used to extract features from different modal information separately. For input information in the text modality, natural language processing technology can be used; for input information in the image modality, convolutional neural networks can be used. The feature fusion network is used to fuse the features extracted from different modalities to obtain fused features. The copy generation network is used to generate candidate copy information based on the aforementioned fused features.

[0063] When the target copy generation mode is a text input mode, the target text content input by the user is parsed to obtain the to-be-filled text content, and the to-be-filled text content is used as the target copy content according to the text input mode. The to-be-filled text content refers to the text information that the user wishes to fill in the original image.

[0064] The above-mentioned method uses the first multimodal generation model to more accurately capture the nuances of user needs in the context, generating copy that is more appropriate to the situation. By displaying a variety of candidate copy for users to choose from, it increases interactivity and flexibility, allowing users to select or further customize according to their preferences. The text content to be filled in is directly parsed from user input, ensuring that the target copy content is highly personalized and accurate, directly reflecting the user's intentions and specific requirements. For users who know exactly what copy content they need, this straightforward approach can quickly obtain satisfactory results and save time costs.

[0065] Further optionally, the original image includes one or more commodities. For the sake of distinction, the commodities included in the original image are referred to as first commodities, and other commodities associated with the first commodity are referred to as second commodities. The meaning of "association" may include at least one of the following: 1) The first commodity and the second commodity belong to the same store, the same brand or the same series, etc. 2) The first commodity and the second commodity are similar in category, function, use or other attributes. 3) The association of purchasing behavior between the first commodity and the second commodity, for example, when purchasing the first commodity (such as a coffee machine), the second commodity (such as coffee beans or filter paper) is also tended to be purchased. 4) "Association" may mean that the first commodity and the second commodity are bundled together for promotion. For example, buying a first commodity (such as a top) gives away a second commodity (such as a scarf).

[0066] Based on this, before using the multimodal information corresponding to at least one original image and the first multimodal generation method as the target copy reference information and target copy generation method, at least one original image, information about the first product included in each of the at least one original image, information about the second product associated with the first product, and the original copy content contained in each of the at least one original image can also be obtained as the multimodal information corresponding to the at least one original image. This allows the multimodal model to integrate image semantic recognition (such as product category, visual focus), text semantics (existing copy keywords), and store knowledge (common features of products in the same store) for joint analysis. For example, when identifying pet food images, combining the high-frequency words "natural" and "no additives" in the store's historical copy to recommend a matching affinity copy style.

[0067] At least one original image is information in image mode, and the information about the first product, the information about the second product associated with the first product, and the original text content contained in at least one original image are information in text mode. The information about the first product may include, but is not limited to, the basic attributes, specifications, functional characteristics, price information, and user reviews of the first product. Basic attributes may include, but are not limited to, name, brand, model, category, and keywords, and specifications may include, but are not limited to, size, color, material, and capacity. Functional characteristics may include, but are not limited to, descriptions of the product's main functions and features, such as the resolution and processor speed of electronic products. Pricing information may include, but is not limited to, market price, sales price, promotional price, and whether discounts are available. User reviews may include, but are not limited to, ratings, comments, and feedback from users who have purchased the product. Information about the second product can be found in the introduction to the first product and will not be repeated here.

[0068] The original text included in the original image is not limited and may include, but is not limited to, product name and brand identification information, product function information, promotion information, user review summary, application scenario description information, price information, and after-sales service commitment information.

[0069] Among them, by combining the original image and the text information it contains, we can better understand the content context of the image, thereby generating copy that is more in line with the actual scene. For example, for a picture showing a certain product, identifying the text description and product features in the image can help to more accurately describe the product. Using product information and its association with other products, you can refer to the copy on other products to generate targeted and guiding copy content, thereby increasing users' purchasing intention and satisfaction. This method of integrating multiple information sources can provide richer and in-depth content descriptions, making the copy content not just a simple introduction, but containing more practical information and emotional resonance points, thereby enhancing users' reading experience and sense of identification with the brand.

[0070] Furthermore, the method for parsing the target text content to be filled in from the target text content is not limited. An exemplary explanation is given below. If the target text content is text content description information, the text content description information is input into an artificial intelligence-based language parsing model for text parsing to obtain the text content to be filled in; or, if the target text content includes text content input based on a target input control, the text content input based on the target input control is used as the text content to be filled in.

[0071] Specifically, the input box control includes: a first input box control and a second input box control. Optionally, the input box control also includes a third input box control. The first input box control is used for a user to enter text content description information, i.e., text information described in natural language, which can also be called editing instructions. The second input box control is used for a user to enter text content to be filled in, and the third input box control is used for a user to enter text content to be modified.

[0072] On this basis, the user can input the required modifications to the original image into the first input box control in the form of natural language description. In response to the input operation on the first input box control, the image editing page obtains the text content description information input by the user, and inputs the text content description information into the language parsing model based on artificial intelligence for text parsing to obtain the text content to be filled in. Optionally, the language parsing model based on artificial intelligence can also parse the text content to be modified from the text content description information. For example, the text content description information is "Change 2024 in the image to 2025", the text content to be filled in obtained by parsing the language parsing model based on artificial intelligence is "2025", and the text content to be modified obtained by parsing is "2024".

[0073] Among them, the language parsing model based on artificial intelligence can be a language model (LanguageModel, LM) based on artificial intelligence. The embodiments of the present application do not limit the number of model parameters supported by the language model, and the goal is to meet application requirements. If the model parameters are relatively large, the scale of the language model will be relatively large, and the model performance will be relatively better. Of course, it will consume more time and resources during the reasoning and training process; if the model parameters are relatively small, the scale of the language model will be relatively small. When the performance meets the requirements, the model is more lightweight and consumes relatively less time and resources during the reasoning and training process. The language model is used to process and generate a deep learning model for natural language text. The model is based on a neural network architecture and can be trained on a large amount of text data to capture the complex patterns and structures of the language. In an optional embodiment, the language model includes an encoder, a decoder, a self-attention layer, and a feed-forward neural network. The encoder is used to understand the semantics of the input text. The decoder is used to generate the output text. The self-attention layer allows words at each position to pay attention to words at other positions in the sentence, thereby better capturing contextual information. The feedforward neural network performs a nonlinear transformation on the representation of each position. In another optional embodiment, the language model can be implemented using a decoder-only architecture, which mainly includes a decoder structure; in this architecture, the generation of the text content to be filled in can be completed through two stages: pre-filling and decoding. Regardless of which model architecture is adopted, when text needs to be generated, the language model can generate the next most likely word from left to right based on a given prompt. Different sampling strategies can be applied in this process, such as greedy search, beam search, or temperature-adjusted sampling to control the diversity of the generated text.

[0074] Among them, the user can enter text content in the second input box control and the third input box respectively. In response to the input operation on the second input box control, the image editing page uses the input target text content as the text content to be filled in; in response to the input operation on the third input box control, the input text content is used as the text content to be modified.

[0075] Among them, the use of an AI-based language parsing model to understand text content description information (such as user instructions) can adapt to various forms of free expression and describe needs in one's own language without being restricted by a fixed format. The text content to be modified and the text content to be filled in are automatically parsed from the text content description information, and the text editing operation is automatically completed. In addition, through direct input, misunderstandings caused by users' casual expressions are reduced, and the accuracy of copy generation is improved.

[0076] In an optional embodiment, the text content to be modified can be parsed from the target text content input into the input box control, and the parsing method can refer to the above embodiment. When the text content to be modified is identified, at least a portion of the original image including the text content to be modified is identified from at least one original image, wherein the entire at least one original image may include the text content to be modified, or a portion of the at least one original image may include the text content to be modified; and the area of ​​the at least part of the original image containing the text content to be modified is used as the area to be edited.

[0077] By analyzing the text content to be modified, the area to be edited in the original image can be quickly identified. This precise positioning avoids the tedious manual search and selection process, significantly improving work efficiency.

[0078] In addition, the image editing page can also perform text recognition on at least one original image, identifying the characters within the original image and the location of each character within the original image. Users can select characters within the original image, and the image editing page, in response to the user's selection, will select the area containing the selected characters as the editable area. The selected characters can be one or more, without limitation.

[0079] Among them, the method of performing character recognition on the original image is not limited. For example, field recognition and character recognition can be performed on the original image through optical character recognition (OCR) technology to determine the global bounding box (bbox) of each field and the layout bounding box bbox of the characters included in each field. For example, "heart and heart are connected" can be considered as a field, and each of "heart", "and", "heart", "phase" and "connect" is a character. Furthermore, the area where the selected character is located is the area defined by the local bounding box of the selected character. As Figure 2 As shown, the local bounding box corresponding to “heart to heart” is exemplarily displayed.

[0080] Furthermore, the local bounding boxes of the characters included in each field are modified based on the global bounding box of the field. For example, the local bounding box is modified based on information such as the height, width, and aspect ratio of the global bounding box of the field. For characters with missing local bounding boxes, their local bounding boxes are estimated, and the maximum effective padding area of ​​each character is calculated and used as the local bounding box of the character.

[0081] In this embodiment, in the above method, the area where the local boundary box corresponding to the character selected by the user is located is used as the area to be edited. In addition, if the user hopes to add target text information to the area in the original image that is not filled with original text information, the user can select the area to be edited on the original image. For example, the user selects a rectangular area through a rectangular box on the original image, and the image editing page responds to the user's selection operation and uses the rectangular area selected by the user through the rectangular box as the area to be edited.

[0082] In an optional embodiment, in response to a first user operation on a target first interface element, the method for obtaining the target style reference information and the target style generation method pointed to by the target first interface element is not limited. Depending on the implementation method of the first interface element, the method for obtaining the target style reference information and the target style generation method pointed to by the target first interface element in response to the first user operation on the target first interface element may also vary. An exemplary description is provided below.

[0083] The plurality of first interface elements include: an original style control, a reference style control, and a multimodal style control. The first user operation on the first interface element can be implemented as a touch operation or an upload operation, and the touch operation includes but is not limited to: a single operation, a double-click operation, a long press operation, and a sliding operation.

[0084] Among them, the original style control points to the original image and the original style generation method, and the original style generation method refers to the method of generating text style based on the original image. The reference style control points to the reference image and the reference style generation method, and the reference image refers to the image provided by the user to assist in generating style information, and the reference style generation method refers to the method of generating text style based on the reference image. The multimodal style control points to the multimodal information of at least one original image and the second multimodal generation method. Among them, for the sake of convenience in distinction and description, the method of generating text style using multimodal information is referred to as the second multimodal generation method. For the introduction of multimodal information, please refer to the aforementioned embodiment and will not be repeated here.

[0085] The user may initiate a first user operation on the original style control, the reference style control, and the multimodal style control according to their own preferences or needs. When the user initiates a touch operation on the original style control, the image editing page, in response to the touch operation on the original style control, uses the original style generation method associated with at least one original image and the original style control as target style reference information and the target style generation method; or, when the user initiates an upload operation on the reference style control, the image editing page, in response to the upload operation on the reference style control, obtains the uploaded reference image and uses the reference image and the reference style generation method associated with the reference style control as target style reference information and the target style generation method; or, when the user initiates a touch operation on the multimodal style control, the image editing page, in response to the touch operation on the multimodal style control, obtains multimodal information of at least one original image and a second multimodal generation method associated with the multimodal style control as target style reference information and the target style generation method.

[0086] Among them, different first interface elements are provided to users, and diversified or personalized text styles can be generated for users based on the second multimodal generation method. The text style required by the user can be quickly generated based on the original style generation method and the reference style generation method, so that users can choose as needed, thereby improving the flexibility of image editing.

[0087] Optionally, the method for generating a target style image having a target text style based on the target style reference information and the target style generation method is not limited. The following example provides an example: if the target style generation method is the second multimodal generation method, based on the multimodal information corresponding to at least one original image, a second multimodal model is called to generate at least one text style label; and a target style image is selected from at least one style image corresponding to the at least one style label; if the target style generation method is the original style generation method, a target original image is determined from at least one original image, an area to be modified in the target original image is identified, and a target style image is generated based on the area to be modified, the target style image having the original style; if the target style generation method is the reference style generation method, a reference image is used as the target style image, the target style image having the reference style.

[0088] Among them, the second multimodal model refers to a model that can process and fuse data of different modalities. Data of different modalities may include: text, image, audio or video, etc. The second multimodal model receives data of image modality and text modality, and by integrating visual and text information, improves the adaptability of the generated candidate copy content to the original image. The implementation method of the second multimodal model is not limited. For a detailed introduction, please refer to the introduction to the first multimodal model, which will not be repeated here.

[0089] The second multimodal model is used to generate at least one text style label. A text style label is a type of categorical information that can reflect different text styles. Different text style labels correspond to different style images. For example, text style label D1 corresponds to style image E1, which includes bold text, fixed spacing of 22 points, and underlined text. Text style label D2 corresponds to style image E2, which includes text in italic, red, and 80% transparency.

[0090] In the case where the target style generation method is the original style generation method, if the number of original images is one, that original image can be used as the target original image. If there are multiple original images, the method for selecting the target original image from the multiple original images is not limited. For example, one original image can be randomly selected from the multiple original images as the target image. For another example, in response to a selection operation on the multiple original images, the selected original image can be used as the target original image. For another example, the first uploaded original image or the last uploaded original image can be used as the target original image.

[0091] Regardless of whether the original image is one or multiple, the area to be modified in the target original image can be identified. The detailed implementation method can be found in the aforementioned embodiment and will not be repeated here. The implementation method of generating the target style image based on the area to be modified is not limited. For example, the area to be modified in the original image can be directly used as the target style image. Alternatively, setting processing is performed on the area to be modified. For example, the setting processing may include but is not limited to: edge enhancement, texture generation, and color adjustment, etc., which are not limited to this.

[0092] Among them, when the target style generation method is the reference style generation method, the target style image is generated according to the reference image. For example, the entire reference image is directly used as the target style image. For another example, the reference image may include multiple text styles, and text recognition can be performed on the reference image to identify the characters in the reference image and the position information of each character in the reference image. The user can select characters on the reference image, and in response to the user's selection operation on the reference image, the target style image is generated according to the area where the selected characters are located. For a detailed introduction to character recognition, please refer to the aforementioned embodiment and will not be repeated here.

[0093] Based on multimodal information, the system can generate multiple style tags, providing users with a rich selection. Users can flexibly select based on their needs, satisfying the aesthetic preferences of different scenarios and audiences. Modifications are made based on the stylistic features of the original image to ensure that the generated target image maintains the overall style of the original image. Users can explicitly specify a target style by providing a reference image. By combining the advantages of the three approaches, a flexible and efficient style generation method can be constructed to meet the needs of different scenarios while improving user experience and work efficiency. Furthermore, optionally, the target style image can be selected from at least one style image corresponding to at least one text style tag. The implementation of selecting the target style image is not limited. A specific example is provided below, which includes providing a text style library that maintains a correspondence between style tags and style images; matching at least one style image corresponding to at least one text style tag from the text style library; displaying at least one style image; and, in response to a selection operation on the at least one style image, using the selected style image as the target style image. The implementation of displaying the at least one style image is not limited. For example, it can be displayed in a list, a grid layout, a carousel, etc., without limitation.

[0094] The system automatically matches at least one style image corresponding to at least one style tag from the text style library to ensure that the selected image is consistent with the original content or the style desired by the user. For example, if the style tag is "retro", the system will select images with retro style characteristics.

[0095] In an optional embodiment, based on the target text content and the target text style in the target style image, a graphic editing model is called to perform text editing processing on the to-be-edited area in at least one original image to obtain at least one target image. The implementation method is not limited. The following illustrative implementation method is provided, in which a first glyph image corresponding to the target text information is generated based on a preset glyph library, and the first glyph image reflects the character morphological features in the target text content; text style features adapted to the to-be-edited area are extracted from the target style image; based on the text style features, character morphological features, and text encoding features of the target text content, a graphic editing model is called to perform text editing processing on the to-be-edited area in at least one original image to obtain the target image.

[0096] In this embodiment, in terms of glyphs, considering that the characters included in the target copy content may be words, numbers or letters, and the words contain a variety of complex glyph strokes, the accuracy of glyph generation can be guaranteed by glyph images. Specifically, a first glyph image corresponding to the target copy content is generated based on a preset glyph library, and the first glyph image reflects the character morphological characteristics in the target copy content. Among them, the glyph library includes glyphs of various characters, and the glyph of each character in the target copy content can be obtained from the glyph library, and the glyph of each character is displayed on a solid color background. For example, the first glyph image can be an image of black text on a white background including the target copy content in a set font. Among them, the set font can include: Songti, Heiti or Kaiti, etc. As Figure 3 As shown, the first glyph image is exemplarily displayed, and the glyph image includes "GALAXY" in black letters on a white background, but is not limited thereto.

[0097] In this embodiment, in terms of font style, in order to maintain the consistency between the target text content and the text information included in the original image, text style features that are adapted to the target text area can be extracted from the original image. Alternatively, the user can extract text style features that are adapted to the target text area from the associated image of the original image according to his or her own wishes, so that the target text content is the same or similar in style to that in the associated image. Among them, the associated image of the original image is an image that assists in extracting text style features. The associated image can be provided by the user, or the user can select an image that suits his or her wishes from multiple images provided by the terminal device as the associated image of the original image. Among them, text style features refer to the attribute information that the target text content has when it is visually presented, such as font, color, weight (thickness), underline, shadow, background color, and border.

[0098] In an optional embodiment, the method for extracting text style features that are adapted to the region to be edited from the target style image is not limited. Depending on the implementation of the target style image, the method for extracting text style features that are adapted to the region to be edited may also vary.

[0099] The embodiment of extracting text style features adapted to the target text region from an original image or an associated image of the original image is not limited. The associated image of the original image can be understood as the aforementioned reference image. An exemplary description is given below.

[0100] When the target text area includes original text information, text style features that are adapted to the target text area are extracted from the original image, allowing the model to retain the font style in the original image to the greatest extent possible.

[0101] When the target text area includes original text information, the text style features of the original text information are extracted and used as the text style features adapted to the target text area. This style injection method can make the text style of the generated target copy content consistent with the style of the original text information. Even without additional reference text, the original style can still be maintained, making the final target image harmonious and unified on the background.

[0102] Based on this, an implementation method for extracting text style features that are adapted to the area to be edited from the original image when the area to be edited includes original text information includes: extracting a local image where the original text information is located (such as a local image of the area to be edited) from the original image, and generating a second glyph image corresponding to the original text information based on a preset glyph library; inputting the local image and the second glyph image into a style feature extraction model for feature extraction, respectively, to obtain first feature information and second feature information; the first feature information includes text style features and text content features of the original text information, and the second feature information includes text content features of the original text information; according to the second feature information, obtaining the text style features of the original text information from the first feature information as text style features that are adapted to the area to be edited.

[0103] Among them, the implementation method of extracting the local image where the original text information is located from the original image includes: taking the area where the entire original text information belongs to the original image as the local image, or, when the character style included in the original text information is consistent, taking the area where some characters of the original text information belong to as the local image.

[0104] The implementation method of generating the second glyph image corresponding to the original text information based on the preset glyph library can refer to the aforementioned implementation method of generating the first glyph image for the target text content, which will not be repeated here.

[0105] The style feature extraction model can be any model that extracts features from images. For example, style feature extraction models may include, but are not limited to, self-supervised visual representation learning models. This model is a pre-trained feature extraction model that can extract multi-level features from images. These features are extracted in intermediate layers (rather than the final layer) because the features in the intermediate layers retain certain spatial information while also possessing high-level abstraction capabilities.

[0106] Among them, it can be considered that the original text information in the second glyph image belongs to the standard glyph, and the second feature information extracted for the second glyph image does not carry text style features, but only includes text content features. Feature extraction is performed on the original text information, and the first feature information obtained may include text content features and text style features. According to the second feature information, the text style features of the original text information are obtained from the first feature information. As an implementation method of the text style features adapted to the area to be edited, a subtraction operation can be performed on the first feature information and the second feature information, and the text style features of the original text information are obtained by subtracting the second feature information from the first feature information.

[0107] The text style features adapted to the area to be edited are extracted from the associated image of the original image.

[0108] For a detailed introduction to the associated images of the original image, please refer to the aforementioned embodiments. For ease of distinction and description, the text information included in the associated images of the original image is referred to as associated text information. An implementation method for extracting text style features adapted to the area to be edited from the associated images of the original image includes: extracting a local image where the associated text information is located from the associated images of the original image, and generating a third glyph image corresponding to the associated text information based on a preset glyph library; inputting the local image and the third glyph image into a style feature extraction model for feature extraction, respectively, to obtain third feature information and fourth feature information; the third feature information includes text style features and text content features of the associated text information, and the fourth feature information includes text content features of the associated text information; according to the fourth feature information, obtaining the text style features of the associated text information from the third feature information as text style features adapted to the area to be edited.

[0109] For the target style image determined by the second multimodal generation method, the implementation method of extracting text style features therefrom can refer to the extraction process for the original image and its associated images, which will not be repeated here. In an optional embodiment, how to convert the target text content into a form that can be understood by the graphic editing model is crucial. For example, in some feature encoding models, the text is segmented from a semantic level, such as treating some words with complete meanings as a whole. For example, "I" and "we" will be treated as a processing unit (token), which helps to capture the meaning at the vocabulary level. However, in an embodiment of the present application, it is hoped that the text encoding model has recognition capabilities for each character (such as Chinese characters). According to the above method of segmenting the text from the semantic level, semantic adhesion problems may be caused. Semantic Conflation generally refers to semantic ambiguity or confusion caused by improper use of vocabulary. Therefore, the text encoding model provided in the embodiment of the present application performs text encoding from the character dimension, and the text encoding model has the ability to distinguish for each character. The text encoding model can also be called a text-word separation encoding model. Text-word separation refers to text encoding from the character dimension, rather than text encoding from the semantic dimension.

[0110] Based on this, the image editing method provided in the embodiment of the present application further includes: generating a text encoding feature of the target text content. Specifically, the target text content is input into a text encoding model, the characters and character positions in the target text content are jointly encoded to obtain at least one character code; and the at least one character code is concatenated to obtain a text encoding feature of the target text content.

[0111] Inputting the target text content into the text encoding model specifically refers to inputting at least one character included in the target text content into the text encoding model respectively, and the text encoding model outputs a character code for each character.

[0112] Optionally, an implementation method for jointly encoding the characters and character positions in the target text content to obtain at least one character code includes: searching the character code feature of at least one character included in the target text content from a character database; performing position coding on at least one character according to the order of at least one character in the target text content to obtain the position coding feature of at least one character; generating the character code of at least one character according to the character coding feature and position coding feature of at least one character. The character database includes character coding features of various characters (such as Chinese characters, numbers or letters, etc.), and the character database can be implemented as a code book. The character coding feature can be a character embedding vector. The position coding mainly assists the target graphic editing model to understand the relative or absolute position relationship between the characters in the target text content. An implementation method for generating the character code of at least one character according to the character coding feature and position coding feature of at least one character can be to splice each character coding feature and its position coding feature to obtain an intermediate feature, input the intermediate feature into a text encoder for text encoding, and obtain a character code, such as Figure 3 Alternatively, the intermediate feature is used as a character code.

[0113] In an optional embodiment, a method for calling a graphic editing model based on text style features, character morphological features and text encoding features of target copy content to perform text editing processing on the to-be-edited area in at least one original image to obtain a target image includes: inputting text style features, text encoding features, a first glyph image and an original image, a first mask image and a second mask image into the target graphic editing model, the first mask image being obtained by masking the target text area in the original image, and the second mask image being a mask image for extracting the image area where the target copy content is located; generating initial image features based on the first glyph image, the original image, the first mask image and the second mask image; based on a cross-attention mechanism, repairing the initial image features using text style features and text encoding features to obtain target image features, the target image features integrating text style features, character morphological features and text layout features; generating a target image based on the target image features.

[0114] The first mask image is obtained by masking the area to be edited in the original image. For example, in the first mask image, the area to be edited is masked with black pixels, while other areas remain unchanged. The layout features can reflect information such as the font size, line spacing, character spacing, and alignment of the target content.

[0115] The region of interest of the second mask image corresponds to the region to be edited, the second mask image may be a binary image, and the region of interest may be represented by 0 or 1. When the region of interest is represented by 1, the region of interest is a white area.

[0116] Optionally, an implementation method for generating initial image features based on a first glyph image, an original image, a first mask image and a second mask image includes: using an image feature extraction network to extract features from the first glyph image, the original image and the first mask image respectively to obtain character morphological features, basic image features and background image features respectively; downsampling the second mask image to obtain mask features; and splicing the mask features, character morphological features, basic image features and background image features to obtain initial image features.

[0117] Optionally, an implementation method based on a cross-attention mechanism, which uses text style features and text encoding features to repair initial image features to obtain target image features, includes: based on the cross-attention layer in the image and text editing model, performing cross-attention calculations on the initial image features with the text style features and text encoding features respectively to obtain a first cross-attention result and a second cross-attention result; generating target image features based on the first cross-attention result and the second cross-attention result.

[0118] For example, a target image feature is generated based on a weighted sum of the first cross-attention result and the second cross-attention result. For another example, a self-attention calculation, a spatial attention calculation, a convolution or a residual calculation is performed on the first cross-attention result and the second cross-attention result to obtain a target output feature.

[0119] Among them, the cross-attention layer in the image and text editing model includes multiple, for example, the first cross-attention layer performs cross-attention calculation on the initial image features and text style features to obtain a first cross-attention result; the second cross-attention layer performs cross-attention calculation on the initial image features and text encoding features to obtain a second cross-attention result. Optionally, the image and text editing model includes: a multi-scale feature extraction network, the multi-scale feature extraction network includes a first cross-attention layer and a second cross-attention layer of multiple scales, and cross-attention calculation can be performed at each scale by the first cross-attention layer and the second cross-attention layer. For an introduction to the multi-scale feature extraction network, please refer to the aforementioned embodiment and will not be repeated here.

[0120] Further optionally, the text style features and the text encoding features have an initial scale, and the initial image features also have the same initial scale. In this case, an implementation method is provided for performing cross-attention calculations on the initial image features, the text style features, and the text encoding features based on a cross-attention layer, respectively, to obtain a first cross-attention result and a second cross-attention result, including: generating text style features of multiple scales and text encoding features of multiple scales based on the text style features and text encoding features with the initial scale, combined with multiple scale information supported by the multi-scale feature extraction network in the target image and text editing model; utilizing the cross-attention layer in the multi-scale feature extraction model, performing cross-attention calculations on the input features of any scale in the multi-scale feature extraction network, respectively, with the text style features and text encoding features of any scale, to obtain a first cross-attention result and a second cross-attention result; wherein, the input features on the first scale are the initial image features, and the output features on any scale are the input features on the next scale; generating output features on any scale based on the first cross-attention result and the second cross-attention result; and using the output features on the last scale as the target image features.

[0121] Among them, the implementation method of generating multiple-scale text style features and multiple-scale text encoding features based on the text style features and text encoding features with the initial scale, combined with the multiple-scale information of the multi-scale feature extraction network in the target image-text editing model is not limited. For example, a conditional injection model can be used to generate multiple-scale text style features and multiple-scale text encoding features based on the text style features and text encoding features with the initial scale, combined with the multiple-scale information of the target image-text editing model, and the text style features on multiple scales and the text encoding features on multiple scales are input into the target image-text editing model. Among them, the conditional injection model can be an implicit potential alignment (IPA) model or a control model ControlNet).

[0122] Among them, the Implicit Latent Alignment Model is a method for introducing conditional information into the target image-text editing model. By aligning image features at different scales in the latent space, IPA can effectively inject text style features or text encoding features into the target image-text editing model. ControlNet is a method for introducing conditional information into the target image-text editing model. This method introduces a control module into the target image-text editing model. This module receives text style features or text encoding features, converts them into different scales, and performs cross-attention calculations on the input features to guide the generation of the target image.

[0123] Among them, for any scale, the text style features and input features at the scale are cross-attention calculated to obtain a first cross-attention result; the text encoding features and input features at the scale are cross-attention calculated to obtain a second cross-attention result; based on the first cross-attention result and the second cross-attention result, the output features at any scale are generated.

[0124] Optionally, an implementation method for generating a target image based on target image features includes: decoding the target image features to obtain a predicted image, the predicted image including target text content with text style features, and directly using the predicted image as the target image.

[0125] Alternatively, another embodiment of generating a target image based on target image features includes: decoding the target image features to obtain a predicted image, the predicted image including target text content having text style features, character morphology features, and text layout features; extracting the local image region containing the target text content from the predicted image based on a second mask image, wherein the region to be edited is the region of interest in the second mask image; and replacing the region to be edited in the original image with the local image region to obtain the target image. The second mask image may be a binary image, and the region of interest may be represented by either 0 or 1. When the region of interest is represented by 1, the region of interest is a white area.

[0126] In the above two implementations, the target image features can be decoded by an image feature conversion network to obtain a predicted image. The introduction of the image feature conversion network can be found in the above embodiments and will not be repeated here.

[0127] Furthermore, to make the local image region more aligned with the original image, a transparency setting is applied to the edge of the region of interest. When the edge of the region of interest has a certain degree of transparency, it can display part of the original image's background information when synthesized with the original image, rather than abruptly switching from the original image's background to the local image region. This smooth transition reduces the hard boundary at the edge of the region of interest, making the local image region appear more like part of the original image rather than a rigidly superimposed area. Based on this, an embodiment of extracting the local image region containing the target text content from a predicted image based on a mask image of the region to be edited includes: adjusting the transparency of an associated region of the region of interest in a second mask image to obtain an edge-optimized second mask image; and extracting the local image region containing the target text content from the predicted image based on the edge-optimized second mask image. The associated region of the edge position can be an area with a set distance threshold from the edge position, where the distance threshold can be 1px, 2px, or 5px, etc., where px represents pixels. The transparency adjustment can be to set the transparency of the pixels in the associated region to a uniform value, such as 80% or 50%, where 100% is fully transparent and 0% is opaque. Alternatively, the transparency adjustment may be to reduce the transparency in order from near to far order of the pixel positions and edge positions in the associated area.

[0128] Scenario-based implementation examples:

[0129] like Figure 4 As shown, the image editing method includes 3 branches, the e-commerce copywriting intelligent recommendation and dynamic editing solution includes 3 branches, and each branch can become an independent system.

[0130] Branch 1:

[0131] 1. In response to the original image upload operation, obtain the original image uploaded by the merchant; perform character recognition on the original image. In response to the merchant selecting a character in the original image, the area containing the selected character is used as the area to be edited. For the method of determining the target style image, see step 2 or step 3.

[0132] 2. If there is original text information in the area to be edited, a candidate style image is generated based on the area to be edited, and the candidate style image includes the text style corresponding to the original text information.

[0133] 3. Alternatively, in response to the reference image upload operation, the reference image is used as a candidate style image, where the candidate style image includes the text style of the reference text information.

[0134] 4. Obtain the merchant's store information (information about multiple products included in the store), the current original image and the text information it contains, and the information about the products included in the current original image as multimodal information. Based on the multimodal information, call a third multimodal generation model to generate candidate style tags and at least one candidate copy content; based on the style tag, match the style image corresponding to the style tag from the text style library as the candidate style image. The third multimodal model can generate both style tags and candidate copy content. For an introduction to the third multimodal generation model, please refer to the aforementioned introduction to the first and second multimodal generation models, and will not be repeated here.

[0135] 5. In response to an input operation on an input box control, the copy content input by the merchant is obtained as the target copy content, or, in response to a selection operation on at least one candidate copy content, the selected copy content is used as the target copy content; and, in response to the merchant's selection operation on a candidate style image, the selected candidate style image is used as the target style image, or, if the merchant does not select a candidate style image, the original image is used as the target style image.

[0136] 6. Based on the target style image and target copy content, call the image and text editing model to perform text editing on the to-be-edited area in the original image uploaded by the merchant to obtain the target image.

[0137] Branch 2

[0138] 1. In response to an image upload operation, multiple original images are obtained; in response to an input operation on a first input box control, the input text content description information (e.g., editing instructions) is obtained. For example, the editing instruction may be "change '2024' in the original image to '2025'".

[0139] 2. The language parsing model based on artificial intelligence parses the text content description information, and parses out the content to be modified (e.g., 2024) and the content to be filled in (e.g., 2025), and uses the content to be filled in as the target content.

[0140] 3. Perform character recognition on the original image based on the content of the text to be modified to determine the area to be edited in the original image.

[0141] 4. Use the local image of the area to be edited as the target style image, or, in response to the uploading operation of the reference image, use the reference image as the target style image.

[0142] 5. Based on the target text content and the target text style in the target style image, the image-text editing model is called to batch edit the text in the target areas of multiple original images to generate multiple target images. For example, "2024" in multiple original images can be changed to "2025".

[0143] Branch Three

[0144] 1. In response to an image upload operation, multiple original images are obtained; in response to an input operation on a second input box control, text content to be filled in is obtained; in response to an input operation on a third input box control, text content to be modified is obtained.

[0145] 2. Perform character recognition on the original image based on the content of the text to be modified to determine the area to be edited in the original image.

[0146] 3. Use the local image of the area to be edited as the target style image, or, in response to the uploading operation of the reference image, use the reference image as the target style image.

[0147] 4. Based on the target copy content and the target text style in the target style image, call the image and text editing model to batch edit the text in the edited areas of multiple original images to obtain multiple target images. For example, batch modify "2024" to "2025" in multiple original images. For example, in an e-commerce scenario, take "modifying end-of-season promotion images for a women's clothing store" as an example:

[0148] 1. In response to the image upload operation, five original product images are obtained, including one or more products in the five original product images. For example, the five original product images include the same product.

[0149] 2. The product information in the original product image can be a dress or chiffon material, and the store's historical copywriting includes high-frequency words such as "light" and "romantic". The store's historical copywriting refers to the copywriting information of other product information in the store. The five original product images, the original copywriting content included in the original product images, and the store's historical copywriting are used as multimodal information and input into the first multimodal generation model to generate candidate copywriting content such as "elegant and intellectual".

[0150] 3. In response to the merchant's upload operation on the "text reference style" control, obtain the previous season's promotional image as the target style image, and the target style image includes a gold gradient font feature.

[0151] 4. In response to the merchant's input operation, a batch processing instruction is obtained, which is: change "End of Season Clearance" to "Elegant and Intellectual", determine the area to be edited in the original product image based on "End of Season Clearance", and use "Elegant and Intellectual" as the target copy content.

[0152] 5. Based on the target copy content and the target text style in the target style image, the image and text editing model is invoked to batch edit the to-be-edited areas of multiple original product images, thereby generating five target product images. The five target product images can be displayed on an image editing page, which includes a publish control. After the merchant manually verifies that there are no issues, the five target product images can be published with a single click in response to a touch operation on the publish control.

[0153] In the embodiment of the present application, style injection is performed through the graphic editing model, and different text styles can be referenced for generation and editing, solving the problem of no style reference and few style patterns. By enhancing the batch processing capability through branches two and three, the processing efficiency is improved, solving the problem of not being able to meet high-frequency image processing. By understanding the current original image, related products, related stores, and the high-quality text style and copywriting style of e-commerce scenarios, suitable text styles and copywriting are recommended for the current editor.

[0154] Regarding the embodiments of this application Figure 1 The detailed implementation and beneficial effects of each step in the method have been described in detail in the aforementioned embodiments and will not be elaborated on here.

[0155] It should be noted that the execution entity of each step of the method provided in the above embodiment can be the same device, or the method can be executed by different devices. For example, the execution entity of steps 101 to 103 can be device A; for another example, the execution entity of steps 101 and 102 can be device A, and the execution entity of step 103 can be device B; and so on.

[0156] In addition, some of the processes described in the above embodiments and the accompanying drawings include multiple operations that appear in a specific order, but it should be clearly understood that these operations may not be executed in the order in which they appear in this article or may be executed in parallel. The serial numbers of the operations, such as 101, 102, etc., are only used to distinguish between different operations, and the serial numbers themselves do not represent any order of execution. In addition, these processes may include more or fewer operations, and these operations may be executed in sequence or in parallel. It should be noted that the descriptions of "first", "second", etc. in this article are used to distinguish different messages, devices, modules, etc., and do not represent a sequential order, nor do they limit "first" and "second" to different types.

[0157] Figure 5 A schematic diagram of the structure of an electronic device provided by an exemplary embodiment of the present application is shown in FIG. Figure 5 As shown, the device includes a memory 54 and a processor 55 .

[0158] The memory 54 is used to store computer programs and can be configured to store various other data to support operations on the electronic device. Examples of such data include instructions for any application or method used to operate on the electronic device.

[0159] The processor 55 is coupled to the memory 54 and is used to execute the computer program in the memory 54 for: executing the image editing method described in the aforementioned embodiments. For details, please refer to the description of the aforementioned embodiments and will not be described further here.

[0160] Further, if Figure 5 As shown, the electronic device also includes: a communication component 56, a display 57, a power component 58, an audio component 59 and other components. Figure 5 Only some components are shown schematically, which does not mean that the electronic device only includes Figure 5 In addition, Figure 5 The components in the dotted box are optional components, not mandatory components, and the specific components may depend on the product form of the electronic device. The electronic device of this embodiment can be implemented as a terminal device such as a desktop computer, a laptop computer, a smart phone or an IOT (Internet of Things) device and a smart wearable device (such as a smart watch, a smart bracelet), or a conventional server, a cloud server or a server array. If the electronic device of this embodiment is implemented as a terminal device such as a desktop computer, a laptop computer, a smart phone, etc., it can include Figure 5 If the electronic device of this embodiment is implemented as a conventional server, cloud server or server array and other server-side devices, it may not include Figure 5 Components within the dotted box.

[0161] The above-mentioned memory can be implemented by any type of volatile or non-volatile memory device or a combination thereof, such as static random-access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0162] The communication component is configured to facilitate wired or wireless communication between the device in which the communication component resides and other devices. The device in which the communication component resides can access a wireless network based on a communication standard, such as a 2G, 3G, 4G / LTE, 5G, or other mobile communication network, or a combination thereof. In an exemplary embodiment, the communication component receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel.

[0163] The above-mentioned display includes a screen, which may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor can not only sense the boundary of the touch or slide action, but also detect the duration and pressure associated with the touch or slide operation.

[0164] The power supply assembly provides power to various components of the device in which the power supply assembly is located. The power supply assembly may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the device in which the power supply assembly is located.

[0165] The above-mentioned audio component can be configured to output and / or input audio signals. For example, the audio component includes a microphone (MIC), and when the device where the audio component is located is in an operating mode, such as call mode, recording mode, and voice recognition mode, the microphone is configured to receive external audio signals. The received audio signal can be further stored in a memory or sent via a communication component. In some embodiments, the audio component also includes a speaker for outputting audio signals.

[0166] Accordingly, an embodiment of the present application also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, enables the processor to implement the steps in the above method embodiment. The computer-readable storage medium includes volatile or non-volatile or a combination thereof, and may be removable or non-removable. Examples of computer-readable storage media include, but are not limited to, phase-change random access memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), flash memory or other memory technology, CD-ROM, digital versatile disc (DVD) or other optical storage, magnetic cassette, tape disk storage or other magnetic storage device or any other non-transmission medium.

[0167] Accordingly, the present application embodiment also provides a computer program product, which includes a computer program or instructions. When the computer program or instructions are executed by a processor, the processor is enabled to implement the steps in the above-mentioned method embodiment. It should be understood that each process or a combination of multiple processes in the above-mentioned method flow can be implemented by a computer program or instruction. In addition, these computer programs or instructions can be applied to a processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing device, so that the processor of the general-purpose computer, the special-purpose computer, the embedded processor or other programmable data processing device can be implemented as a device for implementing the corresponding functions in the above-mentioned method embodiment.

[0168] It should also be noted that the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, commodity, or apparatus that includes a series of elements includes not only those elements but also other elements not explicitly listed, or includes elements inherent to such process, method, commodity, or apparatus. In the absence of further limitations, an element defined by the phrase "comprises a ..." does not exclude the presence of other identical elements in the process, method, commodity, or apparatus that includes the element.

[0169] The above are merely embodiments of the present application and are not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application should all be included within the scope of the claims of the present application.

Claims

1. An image editing method, characterized in that: include: Displaying an image editing page for editing at least one original image, the image editing page comprising: a plurality of first interface elements and a plurality of second interface elements; different first interface elements pointing to different style reference information and style generation methods, and different second interface elements pointing to different copy reference information and copy generation methods; In response to a first user operation on a target first interface element, obtaining target style reference information and a target style generation method pointed to by the target first interface element, and generating a target style image according to the target style reference information and the target style generation method; In response to a second user operation on a target second interface element, obtaining target copy reference information and a target copy generation method pointed to by the target second interface element, and generating target copy content according to the target copy reference information and the target copy generation method; According to the target text content and the target text style in the target style image, a graphic and text editing model is called to perform text editing processing on the to-be-edited areas in the at least one original image to obtain at least one target image.

2. The method according to claim 1, characterized in that The plurality of second interface elements include: a text generation control and an input box control. In response to a second user operation on the target second interface element, obtaining target text reference information and a target text generation method pointed to by the target second interface element includes: In response to a touch operation on the text generation control, using the multimodal information and the first multimodal generation method corresponding to the at least one original image as the target text reference information and the target text generation method; or, In response to an input operation on the input box control, the input target text content and the text input method are acquired as the target copy reference information and the target copy generation method.

3. The method according to claim 2, characterized in that Generate target copy content according to the target copy reference information and the target copy generation method, including: If the target copywriting generation mode is the first multimodal generation mode, generating at least one candidate copywriting content by calling the first multimodal model based on the multimodal information corresponding to the at least one original image; and outputting the at least one candidate copywriting content for selection by the user, and using the candidate copywriting content selected by the user as the target copywriting content; If the target copy generation method is a text input method, the text content to be filled in is parsed from the target text content, and the text content to be filled in is used as the target copy content according to the text input method.

4. The method according to claim 3, characterized in that Also includes: The at least one original image, information about a first product included in each of the at least one original image, information about a second product associated with the first product, and original text content included in each of the at least one original image are obtained as multimodal information corresponding to the at least one original image.

5. The method according to claim 3, characterized in that Parsing the target text content to obtain the text content to be filled in, including: If the target text content is text content description information input based on the first input box control, the text content description information is input into a language parsing model based on artificial intelligence to perform text parsing to obtain the text content to be filled in; or, If the target text content includes text content input based on the second input box control, the text content input based on the second input control is used as the text content to be filled in.

6. The method according to claim 2, characterized in that Also includes: Parsing the target text content to obtain the text content to be modified; identifying, from the at least one original image, at least a portion of the original image including the text content to be modified; The area where the text content to be modified is located in at least part of the original image is used as the area to be edited.

7. The method according to claim 1, characterized in that The first interface element includes: an original style control, a reference style control, and a multimodal style control. In response to a first user operation on the target first interface element, obtaining target style reference information and a target style generation method pointed to by the target first interface element includes: In response to a touch operation on the original style control, using the original style generation method associated with the at least one original image and the original style control as target style reference information and the target style generation method; or, In response to an upload operation on the reference style control, obtaining the uploaded reference image, and using the reference image and the reference style generation method associated with the reference style control as target style reference information and the target style generation method; or, In response to a touch operation on the multimodal style control, multimodal information of the at least one original image and a second multimodal generation mode associated with the multimodal style control are acquired as target style reference information and a target style generation mode.

8. The method according to claim 7, characterized in that Generating a target style image according to the target style reference information and the target style generation method includes: If the target style generation mode is the second multimodal generation mode, generating at least one style label by calling the second multimodal model according to the multimodal information corresponding to the at least one original image; and selecting the target style image from the at least one style image corresponding to the at least one style label; If the target style generation method is the original style generation method, determining a target original image from at least one original image, identifying a region to be modified in the target original image, and generating a target style image based on the region to be modified; If the target style generation method is a reference style generation method, a target style image is generated according to the reference image.

9. The method according to claim 8, characterized in that Selecting the target style image from at least one style image corresponding to the at least one style label includes: matching at least one style image corresponding to the at least one style tag from a text style library; Display at least one style image; In response to a selection operation on at least one style image, the selected style image is used as a target style image.

10. The method according to any one of claims 1 to 9, characterized in that According to the target text content and the target text style in the target style image, calling the image-text editing model to perform text editing processing on the to-be-edited areas in the at least one original image to obtain at least one target image, including: generating a first glyph image corresponding to the target text information based on a preset glyph library, wherein the first glyph image reflects the character morphological features in the target text content; Extracting text style features that are adapted to the area to be edited from the target style image; According to the text style features, the character morphology features and the text encoding features of the target text content, a graphic editing model is called to perform text editing processing on the to-be-edited area in the at least one original image to obtain a target image.

11. An electronic device, characterized in that: include: memory and processor; The memory is used to store a computer program; the processor is coupled to the memory and is used to execute the computer program to implement the steps in the method according to any one of claims 1 to 10.

12. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the processor is caused to implement the steps of the method according to any one of claims 1 to 10.

13. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the processor is caused to implement the steps of the method according to any one of claims 1 to 10.

Images