An image processing method and related apparatus

Abstract

This application discloses an image processing method and related apparatus, relating to the field of artificial intelligence technology. The method includes: inputting an image to be processed and a target mask into a trained first deep learning model; using target cue information to guide the first deep learning model to remove target modification elements within the mask region of the target mask; and outputting a target image. The target image is then repaired to obtain a repaired image. The first deep learning model is obtained by retraining a redrawing model using low-rank adaptive techniques, and the training data includes a first image, a sample mask, and a second image. The second image is obtained by removing sample modification elements from the mask region of the sample mask from the first image. Therefore, this method effectively adjusts the redrawing model to a first deep learning model with element removal capabilities through fine-tuning training techniques, improving the element removal effect, achieving natural removal of modification elements, and obtaining a high-quality repaired image.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to an image processing method and related apparatus. Background Technology

[0002] In today's highly digitalized market environment, the efficiency of marketing, design, and film and television content production is a key component of a company's marketing strategy. In particular, during automated image editing, defects in the removal of decorative elements such as text, subtitles, watermarks, and icons result in low image quality, directly impacting the efficiency of image editing. Summary of the Invention

[0003] In view of the above problems, this application provides an image processing method and related apparatus to improve the removal effect of decorative elements and improve image quality. The specific solution is as follows:

[0004] The first aspect of this application provides an image processing method, comprising:

[0005] Obtain the image to be processed and the target mask, wherein the mask area of ​​the target mask is the image area corresponding to the target modification element to be removed;

[0006] The image to be processed and the target mask are input into the trained first deep learning model. Using the target cue information, the first deep learning model is guided to remove the target modification elements in the mask area of ​​the target mask and output the target image.

[0007] The first deep learning model is obtained by retraining the redrawing model using low-rank adaptive technology, and the training data includes a first image, a sample mask, and a second image. The second image is obtained by removing sample modification elements from the mask region of the sample mask from the first image.

[0008] The target image is repaired to obtain a repaired image.

[0009] In one possible implementation, obtaining the image to be processed and the target mask includes:

[0010] Acquire initial input data, which includes the image to be processed, the initial mask, and the initial prompt information;

[0011] Based on the mask area of ​​the initial mask, a sub-image of a preset shape is extracted from the image to be processed;

[0012] The sub-image to be processed and the initial prompt information are input into a pre-trained second deep learning model to guide the second deep learning model to identify the target modification element and output the element localization region;

[0013] The target modification element is identified within the element localization region using a semantic image segmentation model, and a candidate mask region is output.

[0014] The target mask is generated based on the candidate mask region.

[0015] In one possible implementation, generating the target mask based on the candidate mask region includes:

[0016] Calculate the distance between the edge of the candidate mask region and the edge of the element positioning region. If the distance is less than a preset distance threshold, determine that the candidate mask region is correctly positioned, and use the candidate mask region as the mask region to generate the target mask.

[0017] If the distance is not less than the distance threshold, the candidate mask region is expanded according to the edge of the element positioning region and used as the mask region to generate the target mask.

[0018] In one possible implementation, the image to be processed and the target mask are input into a trained first deep learning model. Using target cue information, the first deep learning model is guided to remove the target modification elements within the mask region of the target mask, and the target image is output, including:

[0019] Generate formatted target prompt messages;

[0020] The target mask and the image to be processed are scaled to a preset size, respectively.

[0021] The target mask, the image to be processed, and the target prompt information are input into the first deep learning model, which guides the first deep learning model to remove the target modification elements in the mask area of ​​the target mask from the image to be processed, and outputs the target image.

[0022] In one possible implementation, the image processing method also includes:

[0023] Obtain multiple first images containing various sample modification elements and sample masks corresponding to the sample modification elements of each first image;

[0024] Obtain a second image by removing sample modification elements from the mask region of the first image;

[0025] The first image, sample mask, and second image are combined to obtain the element image dataset;

[0026] Using the element image dataset as training data, the pre-trained redraw model is fine-tuned using low-rank adaptive techniques to obtain the first deep learning model.

[0027] In one possible implementation, the target image is repaired to obtain a repaired image, including:

[0028] The target image is input into the image restoration model, and the image restoration model is used to perform initial restoration on the target image to obtain a candidate image.

[0029] Based on the target mask, the region to be redrawn in the candidate image is extracted, and the region to be redrawn is not smaller than the mask area of ​​the target mask;

[0030] The area to be redrawn is input into the detail redrawing model, and the detail redrawing model is used to perform fine repair on the area to be redrawn to obtain the repaired area image.

[0031] The repaired region image is backfilled into the candidate image to obtain the repaired image.

[0032] A second aspect of this application provides an image processing apparatus, comprising:

[0033] The input acquisition unit is used to acquire the image to be processed and the target mask, wherein the mask area of ​​the target mask is the image area corresponding to the target modification element to be removed;

[0034] An element removal unit is used to input the image to be processed and the target mask into a trained first deep learning model, and use target cue information to guide the first deep learning model to remove the target modification elements in the mask area of ​​the target mask, and output the target image;

[0035] The first deep learning model is obtained by retraining the redrawing model using low-rank adaptive technology, and the training data includes a first image, a sample mask, and a second image. The second image is obtained by removing sample modification elements from the mask region of the sample mask from the first image.

[0036] An image restoration unit is used to restore the target image to obtain a restored image.

[0037] A third aspect of this application provides a computer program product including computer-readable instructions that, when executed on an electronic device, cause the electronic device to implement the image processing method described in the first aspect or any implementation thereof.

[0038] A fourth aspect of this application provides an electronic device, including at least one processor and a memory connected to the processor, wherein:

[0039] The memory is used to store computer programs;

[0040] The processor is used to execute the computer program so that the electronic device can implement the image processing method of the first aspect or any implementation thereof.

[0041] The fifth aspect of this application provides a computer storage medium carrying one or more computer programs, which, when executed by an electronic device, enable the electronic device to perform the image processing method described in the first aspect or any implementation thereof.

[0042] By employing the above technical solution, this application provides an image processing method that obtains an image to be processed and a target mask, wherein the mask region of the target mask is the image region corresponding to the target modifier element to be removed. The image to be processed and the target mask are input into a trained first deep learning model. Using target cue information, the first deep learning model is guided to remove the target modifier element within the mask region of the target mask, and the target image is output. The target image is then repaired to obtain a repaired image. The first deep learning model is obtained by retraining the redrawing model using LoRA technology, and the training data includes a first image, a sample mask, and a second image. The second image is obtained by removing the sample modifier element from the mask region of the sample mask in the first image. Therefore, this method effectively adjusts the redrawing model into a first deep learning model with element removal capabilities through LoRA fine-tuning training technology, improving the element removal effect, achieving natural removal of modifier elements, and further repairing the target image to obtain a high-quality repaired image. Attached Figure Description

[0043] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic, and the originals and elements are not necessarily drawn to scale.

[0044] Figure 1 A system architecture diagram provided for this application;

[0045] Figure 2 A flowchart illustrating an image processing method provided in this application;

[0046] Figure 3 A schematic flowchart illustrating another image processing method provided in an embodiment of this application;

[0047] Figure 4 A schematic flowchart illustrating another image processing method provided in an embodiment of this application;

[0048] Figure 5 A schematic flowchart illustrating another image processing method provided in an embodiment of this application;

[0049] Figure 6 This is a schematic diagram of the structure of an image processing system provided in an embodiment of this application;

[0050] Figure 7 This is a schematic diagram of the structure of an image processing apparatus provided in an embodiment of this application;

[0051] Figure 8 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0052] The embodiments of this application are described below with reference to the accompanying drawings. The terminology used in the implementation section of this application is for explaining specific embodiments only and is not intended to limit the scope of this application.

[0053] The embodiments of this application will now be described with reference to the accompanying drawings. Those skilled in the art will recognize that, with technological advancements and the emergence of new scenarios, the technical solutions provided in the embodiments of this application are equally applicable to similar technical problems.

[0054] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar elements and are not necessarily used to describe a specific order or sequence. It should be understood that such terms are interchangeable where appropriate; this is merely a way of distinguishing elements with the same properties in the description of embodiments of this application. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion, so that a process, method, system, product, or apparatus that comprises a series of units is not necessarily limited to those units, but may include other units not explicitly listed or inherent to those processes, methods, products, or apparatuses.

[0055] This application can be applied to the field of image processing technology, specifically to the removal of modifier elements during image editing. These modifier elements include, but are not limited to, text, subtitles, watermarks, and icons. For example, in image editing of video frames, a modifier element might be a playback source identifier, which falls under the category of icons. In fields such as film, advertising, e-commerce, marketing, and publishing, the characteristics of different types of modifier elements (text, subtitles, watermarks, and icons, etc.) vary significantly, and different image subjects also differ greatly, greatly increasing the difficulty and cost of removing modifier elements from images.

[0056] Any image design and production process often involves the removal of numerous decorative elements. Different types of decorative elements (text, subtitles, watermarks, icons, etc.) have significantly different characteristics, greatly increasing the difficulty and cost of removal. For example, in e-commerce image creation, it's necessary to remove existing brand logos, slogans, and other text or icon elements to replace them with new brand information, thereby improving brand image dissemination and market competitiveness. In the post-production of films and television series, the removal of subtitles and icons is a common requirement. In some international co-productions, subtitles in different languages ​​need to be replaced or removed for localized dubbing and translation. Furthermore, to meet the requirements of different broadcasting platforms, some icon elements also need to be removed. Statistics show that in the post-production of a medium-sized television series, the cost related to the removal of decorative elements accounts for approximately 15% of the total production cost. In book cover and interior page design, it is sometimes necessary to remove text, watermarks, and other decorative elements from images to achieve better visual effects and comply with copyright requirements. In images involving personal privacy, such as ID photos and medical images, it is necessary to remove decorative elements such as text or icons containing sensitive personal information.

[0057] Therefore, current image processing methods using deep learning algorithms to remove decorative elements have shortcomings in terms of removal effect, especially for decorative elements with large mask areas. The removed image is often damaged and cannot meet the requirements of the removal task. Designers usually need to manually repair the image, which results in a lot of time consumption and low efficiency.

[0058] To address the aforementioned problems, this application provides an image processing method aimed at improving the removal effect of modifier elements, thereby enhancing image quality. The image processing method of this application embodiment will be described in detail below with reference to the accompanying drawings.

[0059] See Figure 1 , Figure 1 A schematic diagram of a system architecture is shown. The system may include a terminal 100 and a server 200. The server 200 can provide the methods provided in the embodiments of this application to one or more terminals.

[0060] The terminal 100 may have an image processing application installed. The application and the webpage can provide an interface. The terminal 100 can receive relevant parameters input by the user on the image processing interface and send the parameters to the server 200. The server 200 can obtain the processing result based on the received parameters and return the processing result to the terminal 100.

[0061] It should be understood that in some optional implementations, the terminal 100 can also complete the action of obtaining the processing result based on the received parameters on its own, without the need for the server to cooperate. This application embodiment is not limited to this.

[0062] The following description Figure 1 The product form of the mid-terminal 100;

[0063] The terminal 100 in this application embodiment can be a mobile phone, tablet computer, wearable device, vehicle device, augmented reality (AR) / virtual reality (VR) device, laptop computer, ultra-mobile personal computer (UMPC), netbook, personal digital assistant (PDA), etc., and this application embodiment does not impose any restrictions on it.

[0064] Terminal 100 may include a radio frequency unit, memory, input unit, display unit, camera (optional), audio circuitry (optional), speaker (optional), microphone (optional), headphone jack (optional), processor, external interface, power supply, and other components. Those skilled in the art will understand that the above-mentioned components are merely examples and do not constitute a limitation on the terminal or multifunctional device; it may include more or fewer components, or a combination of certain components, or different components.

[0065] The input unit can be used to receive input numeric or character information, and to generate key signal inputs related to user settings and function control of the portable multi-functional device. Specifically, the input unit may include a touchscreen (optional) and / or other input devices. Other input devices may include, but are not limited to, one or more of a physical keyboard, function keys (such as volume control buttons, power buttons, etc.), trackball, mouse, joystick, etc.

[0066] Among them, the input device can receive input data, etc.

[0067] The display unit can be used to display information input by the user or information provided to the user, various menus of the terminal, interactive interfaces, file display, and / or playback of any multimedia file. In the embodiments of this application, the display unit can be used to display the image processing interface, processing results, etc.

[0068] The memory can be used to store software code related to the image processing method, the processor can execute the steps of the image processing method, and can also schedule other units (such as the input unit and display unit mentioned above) to achieve the corresponding functions.

[0069] This radio frequency unit (optional) can be used to receive and send signals during information transmission or calls.

[0070] In this embodiment of the application, the radio frequency unit can send data to the server 200 and receive the processing results sent by the server 200.

[0071] It should be understood that this radio frequency unit is optional and can be replaced with other communication interfaces, such as a network port.

[0072] Terminal 100 also includes a power source (such as a battery) for supplying power to the various components.

[0073] Terminal 100 also includes an external interface, which can be a standard Micro USB interface or a multi-pin connector, which can be used to connect terminal 100 to other devices for communication or to connect a charger to charge terminal 100.

[0074] Server 200 includes a bus, a processor, a communication interface, and memory. The processor, memory, and communication interface communicate with each other via the bus.

[0075] The memory can be used to store software code related to image processing methods, the processor can execute the steps of the chip's image processing methods, and it can also schedule other units to achieve corresponding functions.

[0076] Reference Figure 2 , Figure 2 This is a flowchart illustrating an image processing method provided in an embodiment of this application, such as... Figure 2 As shown in the figure, an image processing method provided in this application embodiment may include steps S201 to S203, which are described in detail below.

[0077] 201. Obtain the image to be processed and the target mask.

[0078] In this embodiment, the mask area of ​​the target mask is the image area corresponding to the target modification element to be removed.

[0079] 202. Input the image to be processed and the target mask into the trained first deep learning model. Use the target cue information to guide the first deep learning model to remove the target modification elements in the mask area of ​​the target mask and output the target image.

[0080] In this embodiment, the first deep learning model is obtained by retraining the redrawing model using LoRA (Low-Rank Adaptation) technology, and the training data includes a first image, a sample mask, and a second image. The second image is obtained by removing sample modification elements from the mask area of ​​the sample mask in the first image.

[0081] 203. Repair the target image to obtain the repaired image.

[0082] In this embodiment, an artificial intelligence image restoration tool is used to restore the target image. For example, FoocusInpaint is used to redraw the model and restore the mask area in the target image based on the target mask to obtain the restored image.

[0083] As can be seen from the above technical solution, the image processing method provided in this application involves acquiring an image to be processed and a target mask, wherein the mask region of the target mask is the image region corresponding to the target modifier element to be removed. The image to be processed and the target mask are input into a trained first deep learning model. Using target cue information, the first deep learning model is guided to remove the target modifier element within the mask region of the target mask, and the target image is output. The target image is then repaired to obtain a repaired image. The first deep learning model is obtained by retraining the redrawing model using LoRA technology, and the training data includes a first image, a sample mask, and a second image. The second image is obtained by removing the sample modifier element from the mask region of the sample mask in the first image. Therefore, this method effectively adjusts the redrawing model into a first deep learning model with element removal capabilities through LoRA fine-tuning training technology, improving the element removal effect, achieving natural removal of modifier elements, and further repairing the target image to obtain a high-quality repaired image.

[0084] Based on the above embodiments, see Figure 3 , Figure 3 This is a schematic flowchart illustrating another image processing method provided in an embodiment of this application. Figure 3 This illustrates S201, the specific implementation process for acquiring the image to be processed and the target mask, as follows: Figure 3 As shown in the figure, an image processing method provided in this application embodiment may include steps S301 to S305, which are described in detail below.

[0085] S301. Obtain initial input data.

[0086] In this embodiment, the initial input data includes an initial mask, an image to be processed, and initial prompts. The mask area of ​​the initial mask is the area to be edited as input by the user.

[0087] Specifically, the system obtains the image to be processed input by the user, and an initial mask generated by painting the modification elements to be removed on the image to be processed. It also obtains the initial prompt information input by the user, which describes the modification elements to be removed, i.e. the target modification elements. The mask area of ​​the initial mask is the area painted by the user.

[0088] S302. Based on the mask region of the initial mask, extract a sub-image of a preset shape from the image to be processed.

[0089] In this embodiment, the preset shape is a rectangle, and the outer rectangular region of the mask area of ​​the initial mask is obtained. The outer rectangular region is then expanded to obtain an extended rectangular region. Based on the extended rectangular region, a sub-image to be processed is extracted from the image to be processed.

[0090] S303. Input the sub-image to be processed and the initial prompt information into the pre-trained second deep learning model, guide the second deep learning model to identify target modification elements, and output the element location region.

[0091] In this embodiment, the second deep learning model is a pre-trained large-scale visual-language (VL) model, specifically the Qwen-VL model. The sub-image to be processed and the initial prompt information are input into the Qwen-VL model, which guides the Qwen-VL model to locate the target modifier element on the sub-image to be processed and outputs the element location region that encloses the target modifier element.

[0092] S304. Use a semantic image segmentation model to identify target modification elements within the element localization region and output candidate mask regions.

[0093] In this embodiment, the semantic image segmentation model can specifically be the SAM (Segment Anything Model) model.

[0094] S305. Generate the target mask based on the candidate mask region.

[0095] In this embodiment, the distance between the edge of the candidate mask region and the edge of the element positioning region is calculated. If the distance is less than a preset distance threshold, the candidate mask region is determined to be correctly positioned. The candidate mask region is then used as the mask region to generate the target mask.

[0096] If the distance is not less than the distance threshold, the target mask is generated after expanding the candidate mask region according to the edge of the element location region.

[0097] It should be noted that both the candidate mask region and the element positioning region are rectangular regions with edges including the four sides of the rectangle. Therefore, the distances to the corresponding edges of the candidate mask region and the element positioning region are calculated respectively. After obtaining the edge distances, based on the distances corresponding to the four edges and the distance threshold, it is determined whether the candidate mask region is accurately positioned, thereby determining whether the candidate mask region should be adjusted.

[0098] As can be seen from the above technical solutions, the image processing method provided in this application uses the Qwen-VL model combined with prompt word engineering to identify the element location region, and then uses a semantic image segmentation model to locate the candidate mask region within the element location region, thereby improving the mask quality. The accuracy of the candidate mask region is determined by the edge detection method. If it is inaccurate, the mask quality is further improved by expanding the mask, making the mask fit the element edge of the target modification element better, expanding the mask region to generate a transition area, etc., and adapting to the characteristics of the first deep learning model's strong processing capability for large area masks.

[0099] Based on the above embodiments, see Figure 4 , Figure 4 This is a schematic flowchart illustrating another image processing method provided in an embodiment of this application. Figure 4 The following illustrates S202, which involves inputting the image to be processed and the target mask into the trained first deep learning model, using target cue information to guide the first deep learning model to remove the target modification elements corresponding to the target mask, and outputting the target image. Figure 4 As shown in the embodiment of this application, the image processing method may further include steps S401 to S403, which will be described in detail below.

[0100] S401. Generate formatted target prompt information.

[0101] In this embodiment, if the user inputs initial prompt information, a formatted target prompt information is generated based on the initial prompt information, so that the target prompt information conforms to the prompt information format of the first deep learning model. If the user does not input initial prompt information, that is, there is no initial prompt information in the initial input information, a formatted target prompt information is directly generated. For example, based on the identified target modifier element, a prompt word is generated to describe the target modifier element, and combined with the pre-configured formatted information of the prompt information, the target prompt information is obtained.

[0102] S402. Scale the target mask and the image to be processed to the preset size respectively.

[0103] S403. Input the target mask, the image to be processed, and the target prompt information into the pre-trained first deep learning model, guide the first deep learning model to remove the target modification elements corresponding to the target mask from the image to be processed, and output the target image.

[0104] In this embodiment, the first deep learning model is obtained by fine-tuning the pre-trained Flux1 Fill model. After the Flux Fill model erases the corresponding region of the image to be processed according to the target mask, it is encoded, spliced ​​with the image to be processed and the target mask, and then spliced ​​with the noise latent space. The image is then entered into the Transformer module to gradually denoise and generate the redrawn image.

[0105] In this embodiment, the first deep learning model is obtained by fine-tuning the pre-trained Flux1 Fill model. The process of constructing the first deep learning model includes:

[0106] Obtain the element image dataset.

[0107] Specifically, a first image with multiple types of modifiers, a sample mask, and a corresponding second image after removing the modifiers are collected. For example, the first image includes e-commerce images with icons and text, movie screenshots with subtitles, and various poster images with icons and text. The second image corresponding to the removal of modifiers from the first image based on the sample mask is then obtained.

[0108] As can be seen, this step specifically collects data such as e-commerce images with icons and text, movie screenshots with subtitles, and various poster images with icons and text to create an element image dataset.

[0109] Based on the element image dataset, the Flux1 Fill model was fine-tuned using LoRA technology to obtain the first deep learning model.

[0110] Specifically, small-scale counterfactual datasets enhance the capabilities of the first deep learning model more than large-scale ordinary datasets. Therefore, we construct a small-scale counterfactual dataset, namely an element image dataset, and use LoRA technology to fine-tune the image generation process in the Tansformer module.

[0111] Specifically, based on the Transformer weight matrix Extract the low-rank decomposition matrix to represent the updated parameters, i.e. Where A is a reduced-dimensional matrix and B is an increased-dimensional matrix, the dimension of the LoRA module is much smaller than that of the matrix. Dimensions.

[0112] As can be seen from the above technical solutions, the image processing method provided in this application utilizes LoRA technology to train a first deep learning model based on the Flux Fill model for image embellishment elements such as text, icons, subtitles, and watermarks. By using an element image dataset and LoRA fine-tuning training technology, the target removal capability of Flux Fill is stimulated, effectively converting the local redrawing capability of Flux Fill into the removal capability of embellishment elements. The trained first deep learning model improves the effect of element removal, achieving natural removal of embellishment elements.

[0113] Based on this, see Figure 5 , Figure 5 This is a schematic flowchart illustrating another image processing method provided in an embodiment of this application. Figure 5 The diagram illustrates S203, which involves repairing the target image to obtain the repaired image. Figure 5 As shown in the embodiment of this application, the image processing method may further include steps S501 to S504, which will be described in detail below.

[0114] S501. Input the target image into the image restoration model, and use the image restoration model to perform initial restoration on the target image to obtain candidate images.

[0115] In this embodiment, the image inpainting model can be the Foocus Inpaint repainting model. The Foocus Inpaint repainting model is an image inpainting tool based on a small convolutional network design. It achieves efficient local repainting by compressing the number of channels and can enhance the inpainting capabilities of any model without relying on a dedicated model.

[0116] S502. Based on the target mask, extract the region to be redrawn in the candidate image.

[0117] In this embodiment, the area to be redrawn is not smaller than the mask area of ​​the target mask. Optionally, the sub-image areas in the candidate image are extracted using the edge of the mask area of ​​the target mask as the dividing line to obtain the area to be redrawn.

[0118] S503. Input the area to be redrawn into the detail redrawing model, and use the detail redrawing model to perform fine repair on the area to be redrawn to obtain the repaired area image.

[0119] In this embodiment, the detailed redrawing model is the Flux Fill redrawing model, which is used for local redrawing and map expansion.

[0120] S504. The repaired area image is backfilled into the candidate image to obtain the repaired image.

[0121] That is, the area to be redrawn in the candidate image is replaced by the image of the repaired area to obtain the repaired image.

[0122] As can be seen from the above technical solution, the overall image is repaired using Foocus inpaint. The overall texture structure generated by the Foocus inpaint model ensures that Flux Fill does not generate unnecessary detailed structures, guaranteeing the integrity of the image contour and structural level after removal. Then, based on the target mask, the FluxFill model, which has stronger detail redrawing capabilities, is used for detail repair, ensuring the image detail quality of the removed area. After two repairs, a high-quality image material with the target modification elements removed is finally obtained. At the same time, the first deep learning model is fine-tuned based on Flux Fill, so Flux Fill redrawing repair can also ensure higher redrawing stability. In summary, the two-layer repair structure ensures the stability of image repair, achieving a virtually seamless effect.

[0123] As can be seen from the above embodiments, the image processing method provided in this application can quickly remove target modifier elements and repair the image after removal, improving the removal effect and ensuring the quality of the removed image. Specifically, this method can be applied to an image processing system, see [link to relevant documentation]. Figure 6 , Figure 6 This is a schematic diagram of the structure of an image processing system provided in an embodiment of this application, such as... Figure 6 As shown, the image processing system includes a preprocessing module 601, a removal processing module 602, and a repair processing module 603.

[0124] In this embodiment, the preprocessing module 601 preprocesses the user-input image to be processed, the initial mask, and the initial prompt information. The preprocessing module pre-configures a second deep learning model, specifically the Qwen-VL model. The initial mask is generated based on the image region selected by the user. For example, after the user selects (by smearing or selecting by box) the region of the target modification element in the image to be processed, the initial mask is generated.

[0125] The preprocessing process includes:

[0126] Guided by initial prompts, a second deep learning model is used to identify target modifiers based on the image to be processed and an initial mask, thereby locating the image region of the target modifier and obtaining the element location region.

[0127] Based on the element location region, the initial mask is optimized to obtain the target mask. By optimizing the initial mask, the mask region of the target mask is made to fit the element edge of the target modified element more closely, and the transition region is generated by expanding the mask region.

[0128] The initial prompt message is optimized to generate the target prompt message.

[0129] It should be noted that traditional image processing methods rely on an initial mask input by the user for removing decorative elements, which poses a significant threat to system stability, has a high failure rate, and severely impacts user experience. The image processing system provided in this application optimizes the initial input mask through a preprocessing module, utilizes a large model to understand user intent, repairs and improves the initial user input mask, reduces the dependence on user input for removing decorative elements, and improves the accuracy and success rate of removal.

[0130] In this embodiment, the removal processing module 602 performs target modification element removal processing on the image to be processed. The removal processing process includes:

[0131] Scale the target mask and the image to be processed to the preset size respectively.

[0132] The target mask, the image to be processed, and the target prompt information are input into a pre-trained first deep learning model, which is then guided to remove the target modification elements corresponding to the target mask from the image to be processed, and output the target image.

[0133] The first deep learning model was obtained by fine-tuning the pre-trained Flux1 Fill model. The process of constructing the first deep learning model includes:

[0134] To obtain the element image dataset, specifically, collect e-commerce images with icons and text, movie screenshots with subtitles, and various poster images with icons and text, and create an image retouching and element removal dataset, also known as the element image dataset.

[0135] Based on the element image dataset, the Flux1 Fill model was fine-tuned using LoRA technology to obtain the modified first deep learning model.

[0136] In summary, the target removal module accurately removes decorative elements such as text captions, icons, and image watermarks based on user input. Specifically, it intelligently identifies the target decorative elements to be removed, generates prompts, and guides a first deep learning model built on the target removal algorithm to remove the decorative elements in the user-specified area, ensuring image quality.

[0137] The target removal algorithm is an image embellishment removal algorithm based on Flux1 Fill. It utilizes LoRA (Low-Rank Adaptation) technology to fine-tune the Flux1Fill model for removing embellishment elements such as text, subtitles, watermarks, and icons. During the fine-tuning process, a large number of sample images containing the first type of embellishment elements (text and icons), sample movie and TV screenshots containing the second type of embellishment elements (subtitles), and sample promotional images containing the third type of embellishment elements (text and watermarks) are selectively added to the regular ObjectRemove dataset. This trains a first deep learning model with a stronger effect on removing image embellishment elements.

[0138] In this embodiment, the repair processing module 603 integrates an image repair model and a detail repainting model. Specifically, the image repair model is the Foocus inpaint repainting model, and the detail repainting model is the Flux Fill repainting model. The repair processing module performs repair processing on the target image, and the repair processing methods include:

[0139] The image to be repaired, i.e. the target image, is input into the Fooocus inpaint repainting model. The Fooocus inpaint repainting model performs initial repair on the target image to obtain candidate images, which restore the integrity of the image at the level of general outline and structure.

[0140] Based on the target mask, the region to be redrawn in the candidate image is extracted.

[0141] The area to be redrawn is input into the Flux Fill redrawing model, which is then used to perform fine-tuning on the area to be redrawn, resulting in a repaired area image.

[0142] The repaired area image is backfilled into the candidate image to obtain the repaired image.

[0143] As can be seen, after the image restoration module performs initial restoration on the target image after removing the modifier elements, it uses Flux1 Fill to perform fine restoration on the area to be redrawn in the target image to address any potential damage to details that may occur after removing the modifier elements. Because Flux1 Fill has stronger fine restoration capabilities, it further refines the texture and other details of the restoration area, achieving efficient and accurate image restoration, further improving the removal effect of the modifier elements, and thus obtaining a high-quality source image.

[0144] In summary, the image processing system provided in this application embodiment enables image preprocessing, removal of modifier elements, and restoration, efficiently completing the task of removing image modifier elements and quickly obtaining high-quality image materials. It should be noted that the specific functional implementation methods of each module of the image processing system can be found in the above embodiments. Specifically, this image processing system can achieve the following technical effects:

[0145] First, improve the removal effect of decorative elements, thereby improving image quality.

[0146] This application optimizes the image retouching task from two aspects: retouching and secondary restoration. On the one hand, existing image retouching algorithms rarely consider the removal of retouching elements, resulting in poor performance for text, captions, icons, and other retouching elements. Furthermore, manual removal by designers is extremely difficult and time-consuming. This system constructs a retouching element removal dataset and trains a fine-tuned LoRA model for retouching element removal, thereby achieving better retouching element removal results. On the other hand, to address the potential loss of detail in the removed image, a collaborative workflow of the Foocus inpaint repainting model and the Flux Fill repainting model is adopted, perfectly ensuring the high quality of the final image and effectively improving the system's generalization and stability.

[0147] Second, improve image processing efficiency.

[0148] The image processing system provided by this solution is based on a large model and can complete the removal of image modification elements in a stable and short time, which greatly shortens the image processing cycle and quickly responds to a large number of image processing needs.

[0149] Third, reduce the cost and complexity of system modification.

[0150] In traditional image processing methods, companies typically need to purchase third-party image processing software or interfaces, as well as the hardware to run the software. They also need to adapt their existing image processing systems based on the purchased equipment. In this solution, the image processing system can directly remove modifiers by training and configuring various models, eliminating additional procurement costs and reducing system modification costs and complexity.

[0151] Furthermore, traditional image processing methods require hiring professional designers, and in many cases, even these designers are unable to remove certain elements. In contrast, this system requires only simple system training and maintenance costs to easily complete the task of removing decorative elements, significantly reducing reliance on high-cost creative professionals and thus drastically lowering overall design costs.

[0152] The above describes an image processing method provided by an embodiment of this application. The following describes the apparatus for performing the above image processing method.

[0153] Please see Figure 7 , Figure 7 This is a schematic diagram of the structure of an image processing apparatus provided in an embodiment of this application. Figure 7 As shown, the image processing apparatus 700 includes:

[0154] The input acquisition unit 701 is used to acquire the image to be processed and the target mask, wherein the mask area of ​​the target mask is the image area corresponding to the target modification element to be removed;

[0155] The element removal unit 702 is used to input the image to be processed and the target mask into the trained first deep learning model, and use the target cue information to guide the first deep learning model to remove the target modification elements in the mask area of ​​the target mask and output the target image;

[0156] The first deep learning model is obtained by retraining the redrawing model using low-rank adaptive technology, and the training data includes a first image, a sample mask, and a second image. The second image is obtained by removing sample modification elements from the mask region of the sample mask from the first image.

[0157] The image restoration unit 703 is used to restore the target image to obtain a restored image.

[0158] In one possible implementation, when the input acquisition unit acquires the image to be processed and the target mask, it is specifically used for:

[0159] Acquire initial input data, which includes the image to be processed, the initial mask, and the initial prompt information;

[0160] Based on the mask area of ​​the initial mask, a sub-image of a preset shape is extracted from the image to be processed;

[0161] The sub-image to be processed and the initial prompt information are input into a pre-trained second deep learning model to guide the second deep learning model to identify the target modification element and output the element localization region;

[0162] The target modification element is identified within the element localization region using a semantic image segmentation model, and a candidate mask region is output.

[0163] The target mask is generated based on the candidate mask region.

[0164] In one possible implementation, when the input acquisition unit generates the target mask based on the candidate mask region, it is specifically used for:

[0165] Calculate the distance between the edge of the candidate mask region and the edge of the element positioning region. If the distance is less than a preset distance threshold, determine that the candidate mask region is correctly positioned, and use the candidate mask region as the mask region to generate the target mask.

[0166] If the distance is not less than the distance threshold, the candidate mask region is expanded according to the edge of the element positioning region and used as the mask region to generate the target mask.

[0167] In one possible implementation, the element removal unit is used to input the image to be processed and the target mask into a trained first deep learning model, and guide the first deep learning model to remove the target modification elements within the mask region of the target mask using target cue information. Specifically, when outputting the target image, it is used to:

[0168] Generate formatted target prompt messages;

[0169] The target mask and the image to be processed are scaled to a preset size, respectively.

[0170] The target mask, the image to be processed, and the target prompt information are input into the first deep learning model, which guides the first deep learning model to remove the target modification elements in the mask area of ​​the target mask from the image to be processed, and outputs the target image.

[0171] In one possible implementation, the image processing apparatus further includes a model building unit for:

[0172] Obtain multiple first images containing various sample modification elements and sample masks corresponding to the sample modification elements of each first image;

[0173] Obtain a second image by removing sample modification elements from the mask region of the first image;

[0174] The first image, sample mask, and second image are combined to obtain the element image dataset;

[0175] Using the element image dataset as training data, the pre-trained redraw model is fine-tuned using low-rank adaptive techniques to obtain the first deep learning model.

[0176] In one possible implementation, the image inpainting unit is used to repair the target image, and when obtaining the repaired image, it is specifically used for:

[0177] The target image is input into the image restoration model, and the image restoration model is used to perform initial restoration on the target image to obtain a candidate image.

[0178] Based on the target mask, the region to be redrawn in the candidate image is extracted, and the region to be redrawn is not smaller than the mask area of ​​the target mask;

[0179] The area to be redrawn is input into the detail redrawing model, and the detail redrawing model is used to perform fine repair on the area to be redrawn to obtain the repaired area image.

[0180] The repaired region image is backfilled into the candidate image to obtain the repaired image.

[0181] This application also provides an electronic device in its embodiments. (See reference...) Figure 8 The diagram illustrates a structural schematic suitable for implementing the electronic device in the embodiments of this application. The electronic device in the embodiments of this application may include, but is not limited to, fixed terminals such as mobile phones, laptops, PDAs (personal digital assistants), PADs (tablet computers), desktop computers, etc. Figure 8 The electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.

[0182] like Figure 8 As shown, the electronic device may include a processing unit (e.g., a central processing unit, a graphics processing unit, etc.) 801, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 802 or a program loaded from a storage device 808 into a random access memory (RAM) 803. When the electronic device is powered on, the RAM 803 also stores various programs and data required for the operation of the electronic device. The processing unit 801, ROM 802, and RAM 803 are interconnected via a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0183] Typically, the following devices can be connected to I / O interface 805: input devices 806 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 807 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 808 including, for example, memory cards, hard drives, etc.; and communication devices 809. Communication device 809 allows electronic devices to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 8 Electronic devices with various devices are shown, but it should be understood that it is not required to implement or have all of the devices shown. More or fewer devices may be implemented or have alternatively.

[0184] This application also provides a computer program product including computer-readable instructions, which, when executed on an electronic device, cause the electronic device to implement any of the image processing methods provided in this application.

[0185] This application also provides a computer-readable storage medium that carries one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device can implement any of the image processing methods provided in this application.

[0186] It should also be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. In addition, in the device embodiment drawings provided in this application, the connection relationship between modules indicates that they have a communication connection, which can be implemented as one or more communication buses or signal lines.

[0187] Through the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware, or it can be implemented by special-purpose hardware including application-specific integrated circuits, special-purpose CPUs, special-purpose memory, special-purpose components, etc. Generally, any function performed by a computer program can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be diverse, such as analog circuits, digital circuits, or special-purpose circuits. However, for this application, software program implementation is more often the preferred implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a readable storage medium, such as a computer floppy disk, USB flash drive, mobile hard disk, ROM, RAM, magnetic disk, or optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, training equipment, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0188] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product.

[0189] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, training device, or data center to another website, computer, training device, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a training device or data center that integrates one or more available media. The available media may be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media (e.g., solid-state drives (SSDs)).

Claims

1. An image processing method, characterized by, The method comprises the following steps: acquiring initial input data, the initial input data comprising a to-be-processed image, an initial mask, and initial prompt information; using a second deep learning model to identify a target decoration element based on the to-be-processed image and the initial mask under the guidance of the initial prompt information, locating an image region of the target decoration element, and obtaining an element positioning region; using a semantic image segmentation model to identify the target decoration element in the element positioning region and output a candidate mask region; calculating the distance between the edge of the candidate mask region and the edge of the element positioning region, determining whether the candidate mask region is accurately positioned based on the distance and a distance threshold, determining whether to adjust the candidate mask region, and further generating a target mask, wherein the mask region of the target mask is an image region corresponding to the target decoration element to be removed; inputting the to-be-processed image and the target mask into a trained first deep learning model, using target prompt information to guide the first deep learning model to remove the corresponding region of the to-be-processed image according to the target mask, encoding, splicing the to-be-processed image and the target mask, splicing with a noise latent space, inputting into a Transformer module for gradual denoising to remove the target decoration element in the mask region of the target mask, and outputting a target image; wherein the first deep learning model is obtained by retraining a redraw model through a low-rank adaptive technology, and the training data comprises a first image, a sample mask, and a second image, wherein the second image is obtained by removing a sample decoration element in the mask region of the sample mask from the first image; inputting the target image into an image inpainting model, using the image inpainting model to perform preliminary repair on the target image to obtain a candidate image, and the image inpainting model is a Fooocus Inpaint redraw model; based on the target mask, extracting a to-be-redrawn region in the candidate image, wherein the to-be-redrawn region is not less than the mask region of the target mask; inputting the to-be-redrawn region into a detail redraw model, using the detail redraw model to perform fine repair on the to-be-redrawn region to obtain a repair region image, and the detail redraw model is a Flux Fill redraw model; backfilling the repair region image to the candidate image to obtain a repair image.

2. The image processing method of claim 1, wherein, The method comprises the following steps: based on the mask region of the initial mask, cutting a to-be-processed sub-image of a preset shape from the to-be-processed image; inputting the to-be-processed sub-image and the initial prompt information into a pre-trained second deep learning model to guide the second deep learning model to identify the target decoration element and output an element positioning region.

3. The image processing method of claim 1, wherein, The method comprises the following steps: If the distance is less than a preset distance threshold, it is determined that the candidate mask region is correctly positioned, and the target mask is generated with the candidate mask region as the mask region. If the distance is not less than the distance threshold, the target mask is generated with the candidate mask region after the edge of the element positioning region is expanded as the mask region.

4. The image processing method of claim 1, wherein, The target prompt information is generated in a formatted manner. The target mask and the to-be-processed image are respectively scaled to a preset size. The target mask, the to-be-processed image, and the target prompt information are input into the first deep learning model, and the first deep learning model is guided to remove the target modification element in the mask region of the target mask by encoding and splicing the to-be-processed image and the target mask after the corresponding region of the to-be-processed image is removed according to the target mask, splicing with a noise latent space, inputting into a Transformer module for gradual denoising, and outputting the target image. The image processing method further comprises:

5. The image processing method of claim 1, wherein, a plurality of first images containing various types of sample modification elements and sample masks corresponding to sample modification elements of each first image are respectively acquired; a second image obtained by removing sample modification elements in the mask region of the sample mask from the first image is acquired; corresponding first images, sample masks, and second images are collected to obtain an element image dataset; the first deep learning model is obtained by fine-tuning a pre-trained redrawing model using low-rank adaptive technology with the element image dataset as training data. It comprises:

6. An image processing apparatus characterized by comprising: an input acquisition unit configured to acquire initial input data, the initial input data comprising a to-be-processed image, an initial mask, and initial prompt information; a target modification element is identified based on the to-be-processed image and the initial mask under the guidance of the initial prompt information using a second deep learning model, an image region of the target modification element is positioned, an element positioning region is obtained, the target modification element is identified in the element positioning region using a semantic image segmentation model, a candidate mask region is output, a distance between an edge of the candidate mask region and an edge of the element positioning region is calculated, and whether the candidate mask region is accurately positioned is determined based on the distance and a distance threshold, whether the candidate mask region is adjusted is determined, and a target mask is generated, wherein a mask region of the target mask is an image region corresponding to a target modification element to be removed. ​ An element removing unit is configured to input the to-be-processed image and the target mask into a trained first deep learning model, guide the first deep learning model to remove the corresponding region of the to-be-processed image according to the target mask, and input the to-be-processed image and the target mask after encoding and splicing into a noise latent space and a Transformer module for gradual denoising, so as to remove the target decoration elements in the mask region of the target mask and output a target image. The first deep learning model is obtained by retraining a redraw model through a low-rank adaptive technology, and training data includes a first image, a sample mask, and a second image, wherein the second image is obtained by removing sample decoration elements in a mask region of the sample mask from the first image. An image repairing unit is configured to input the target image into an image repairing model, perform preliminary repairing on the target image by using the image repairing model to obtain a candidate image, and extract a to-be-redrawn region in the candidate image based on the target mask, wherein the to-be-redrawn region is not less than the mask region of the target mask. The to-be-redrawn region is input into a detail redraw model, fine repairing is performed on the to-be-redrawn region by using the detail redraw model to obtain a repaired region image, and the detail redraw model is a Flux Fill redraw model. The repaired region image is backfilled into the candidate image to obtain a repaired image.

7. A computer program product, characterised in that, The computer readable instructions, when executed on an electronic device, cause the electronic device to implement the image processing method of any one of claims 1-5.

8. An electronic device, comprising: The memory is configured to store computer programs. The processor is configured to execute the computer programs to enable the electronic device to implement the image processing method of any one of claims 1-5. The storage medium carries one or more computer programs, which, when executed by an electronic device, enable the electronic device to implement the image processing method of any one of claims 1-5.

9. A computer storage medium, characterized in that ​

Images