Picture material one-key localization translation and content reconstruction method and system

Abstract

The present application relates to the field of image processing, and more particularly to a picture material one-key localization translation and content reconstruction method and system. The method first responds to the input of an original picture, uses a preset multi-modal semantic analysis model to receive and analyze the original picture to extract image element information and text information, and then generates fusion prompt features based on the image element information and the text information. The fusion prompt features are input into a preset image segmentation model to obtain segmented images of multiple layers. The segmented images are input into a preset image defect completion model to obtain completed images of multiple layers. During this process, the text information of different layers is extracted according to a preset layered art configuration and input into a preset multi-language translation and typesetting model to obtain reconstructed text of multiple different layers. Finally, the completed images and the reconstructed text are reorganized to obtain a reconstructed image. The method can obtain a reconstructed image with better quality and stronger aesthetic adaptability.

Description

Technical Field

[0001] This invention relates to the field of image processing technology. More specifically, this invention relates to a method and system for one-click localization translation and content reconstruction of image materials. Background Technology

[0002] In overseas advertising campaigns, the linguistic or cultural limitations of advertising image materials are a major factor restricting advertising effectiveness. Generally, advertising images exported from domestic to overseas markets only undergo cultural character translation. These images are only understood by users speaking the local language and cannot effectively reach the target audience in overseas regions, significantly diminishing the advertising's reach and brand influence. Traditionally, image decoupling methods based on RGB / CMYK color channel separation and layer masks are used. This method separates the foreground and background of the image using RGB / CMYK color channel splitting, selection tools, and layer masking techniques. Then, relying on pixel transparency and selection rules, layers are manually decoupled, and the image is then manually enhanced or modified. While accurate and lossless, this method is only suitable for traditional layer editing, requiring mostly manual operation and resulting in extremely low efficiency. Therefore, to balance efficiency and effectively reach the target audience in overseas regions, several solutions have been proposed in existing technologies, including: One approach is layer decoupling based on multimodal large model semantic understanding. This method utilizes the recently emerging multimodal semantic understanding model to decouple layers, which has significant advantages in image segmentation and understanding. However, when there are multiple overlapping elements (text or patterns) in an image, it may not be able to accurately handle the relationship between different layers, resulting in unclear layer boundaries and blurred segmentation, which seriously affects the final image reconstruction / translation quality and the overall output image is not aesthetically pleasing.

[0003] The second approach is a text rendering enhancement-based typesetting optimization method. This method enhances text rendering and typesetting to address issues such as blurry text, typos, and chaotic layout. However, when dealing with less commonly spoken or low-resource languages, it may encounter adaptation difficulties due to incomplete training data and differences in language features, resulting in unsatisfactory translation and typesetting effects.

[0004] Therefore, existing technologies mainly suffer from poor image reconstruction quality and low aesthetic adaptability. Summary of the Invention

[0005] To address the technical problems of poor image reconstruction quality and low aesthetic adaptability in existing technologies, this invention discloses a method and system for one-click localization translation and content reconstruction of image materials.

[0006] In a first aspect, this invention discloses a method for one-click localization translation and content reconstruction of image materials, including: In response to the input of the original image, a pre-defined multimodal semantic parsing model is used to receive and parse the original image in order to extract image element information and text information; Based on image element information and text information, a fusion prompt feature is generated; the fusion prompt feature is input into a preset image segmentation model to obtain segmented images of multiple layers; the segmented images are input into a preset image incompleteness completion model to obtain completed images of multiple layers. Based on the preset layered art configuration, the text information of different layers is extracted and input into the preset multilingual translation and typesetting model to obtain reconstructed text of multiple different layers; The completed image and reconstructed text are combined to obtain the reconstructed image.

[0007] Beneficial Effects: For the translation and reconstruction of original images, this invention employs a multimodal semantic parsing model for analysis, then initiates a concurrent process. For image processing, cue word fusion technology guides the image segmentation model for accurate segmentation, and an image incompleteness completion model performs hierarchical completion based on segmentation to overcome the technical challenges of unclear layer boundaries and ambiguous segmentation. For text translation and reconstruction, this invention automatically adapts to layered art configurations, accurately translating and formatting the text content in the image to meet customized aesthetic requirements. Finally, the completed image and reconstructed text are combined to obtain a reconstructed image of higher quality and stronger aesthetic adaptability.

[0008] Preferably, based on image element information and text information, fused prompt features are generated, including: Image element information and / or text information are input into a preset weighted fusion algorithm to calculate fusion prompt features.

[0009] Preferred weighted fusion algorithms include:

[0010] in, It is a concept c Fusion prompt features, concepts c It refers to the target object that needs to be segmented from the image element information; It is aimed at the concept c Extracted semantic features of text information; It is aimed at the concept c Extracted spatial features of visual cues for points / boxes; These are reference features of the reference image; , and For adaptive weight values, .

[0011] Preferably, the target audience includes the main body of the advertisement, a person, the background and / or the foreground.

[0012] Preferably, the image incompleteness completion model includes an image inpainting algorithm based on a two-stream multimodal diffusion architecture, a spatially gated attention region perception mechanism, a text instruction-enhanced completion generation mechanism, and a total completion loss function; Among them, the image completion algorithm is used to repair image defects in the segmented image to obtain the image to be completed; the spatial gating attention region perception mechanism is used to adjust the completion attention weight of the image completion algorithm; and the text instruction enhanced completion generation mechanism is used to complete the image to be completed and output the completed image.

[0013] Preferably, the complete total loss function is as follows:

[0014] in, It is the total loss function; It is a loss due to incomplete repair; It is the over-scoring loss; It is the loss in noise reduction; It's a loss of coloring; , , , This is the loss weight for the corresponding task.

[0015] Preferably, if the target object includes a human figure, the method further includes the following steps before recombining the completed image and reconstructed text: Input the portrait layer in the completed image into the population reconstructor to reconstruct the style of the human figure in the portrait layer so that it is adapted to the pre-configured target deployment area.

[0016] Preferably, the method of the present invention further includes: Determine the target translation language based on the target geographical area.

[0017] Preferably, the layered art configuration includes attention to layout balance, visual hierarchy, color coordination, element regularity, and / or area white space.

[0018] Secondly, the present invention also discloses a system for one-click localization translation and content reconstruction of image materials, including a processor and a memory. The memory stores computer program instructions, and when the computer program instructions are executed by the processor, the method for one-click localization translation and content reconstruction of image materials described in the first aspect is implemented.

[0019] The beneficial effects of this invention are as follows: (1) Compared with the prior art, the method of the present invention can obtain reconstructed images with better quality and more aesthetic and adaptability.

[0020] (2) Compared with the prior art, the method of the present invention can automatically adapt to the target delivery area, automatically change the character image style and text language, and has a strong adaptive capability. Attached Figure Description

[0021] The above and other objects, features, and advantages of exemplary embodiments of the present invention will become readily apparent upon reading the following detailed description with reference to the accompanying drawings. In the drawings, several embodiments of the invention are illustrated by way of example and not limitation, and like or corresponding reference numerals denote like or corresponding parts, wherein: Figure 1 This is a flowchart of the one-click localization translation and content reconstruction method for image materials in Embodiment 1 of the present invention; Figure 2 This is a schematic diagram of the system for one-click localization translation and content reconstruction of image materials in Embodiment 2 of the present invention. Detailed Implementation

[0022] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0023] The specific embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0024] Example 1 like Figure 1 As shown, this embodiment discloses a method for one-click localization translation and content reconstruction of image materials, including: S10: In response to the input of the original image, the system uses a pre-defined multimodal semantic parsing model to receive and parse the original image in order to extract image element information and text information.

[0025] In this embodiment, the original images are generally taken from a local material library, which supports batch import. The multimodal semantic parsing model can adopt a semantic understanding model based on the Qwen series. Image element information can include information about people, advertising subjects, background information, foreground information, and other information besides text content.

[0026] It should be noted that different text information may reside on different layers. For example, text information such as labels, logos, or advertising slogans may be located at the top layer, while text information such as watermarks may be located at the bottom layer. As for text information that has already been combined and packaged, it is embedded within the image itself.

[0027] Specifically, in step S10, a multimodal semantic parsing model is used to identify and parse elements such as text, subject, and people in the original image, and generate label information such as layer number, occupied area, text outline color / fill color / font style / size, and subject / person description. In this way, each element and text will be labeled and classified in detail to ensure that the image content can be accurately extracted and expressed; while labeling the layer order can ensure that the original occlusion relationship can be maintained during reconstruction.

[0028] After step S10 is executed, in order to improve the efficiency of image reconstruction, the method of this embodiment concurrently executes the image segmentation-complete process of steps S20-S40 and the text reconstruction process of step S50.

[0029] S20: Generate fused prompt features based on image element information and text information.

[0030] Specifically, image element information and / or text information are input into a preset weighted fusion algorithm to calculate fused prompt features. The aforementioned weighted fusion algorithm can be:

[0031] in, It is a concept c Fusion prompt features, concepts c It refers to the target object that needs to be segmented from the image element information; It is aimed at the concept c Extracted semantic features of text information; It is aimed at the concept c Extracted spatial features of visual cues for points / boxes; It is a reference feature of the reference instance image; , and For adaptive weight values, .

[0032] It should be further explained that the target objects mentioned above include the advertising subject, the portrait, the background and / or the foreground. Compared with the existing technology, the fusion prompt features obtained by the above algorithm realize the dynamic fusion of text, visual and example multimodal prompts. It can support pixel-level accurate cutout of the advertising subject, portrait, background and foreground and other elements, and enhance the image segmentation model's ability to process complex image displays.

[0033] S30: Input the fusion prompt features into the preset image segmentation model to obtain segmented images with multiple layers.

[0034] In this embodiment, the image segmentation model described above can adopt the existing open-source SAM3 model. In order to improve the accuracy of its decoupled segmentation, this embodiment also introduces a set of decoupled concept segmentation scoring algorithms to guide it to perform accurate segmentation.

[0035] Specifically, the above decoupled concept segmentation scoring algorithm is as follows:

[0036] in, Indicates the first The concept of matching one reference feature c The final segmentation score; Indicates the first Each reference feature is for the concept. c The localization confidence can be output by the localization branch of the SAM3 model to focus on pixel-level boundary accuracy; This represents the global concept existence probability output by the Presence Head, which is used to determine the concept's existence. c Does it appear entirely in the segmented image?

[0037] S40: Input the segmented image into the preset image incompleteness completion model to obtain a complete image with multiple layers.

[0038] In this embodiment, the above-mentioned image incompleteness completion model includes an image inpainting algorithm based on a two-stream multimodal diffusion architecture, a spatially gated attention region perception mechanism, a text instruction-enhanced completion generation mechanism, and a total completion loss function.

[0039] Among them, the image completion algorithm is used to repair image defects in the segmented image to obtain the image to be completed; the spatial gating attention region perception mechanism is used to adjust the completion attention weight of the image completion algorithm; and the text instruction enhanced completion generation mechanism is used to complete the image to be completed and output the completed image.

[0040] Furthermore, the core focus of image completion algorithms lies in resolving the balance between "precise control" and "natural generation" in instruction-based image editing, which is particularly evident in the fields of image incompleteness repair and old photo restoration. More specifically, the expression for the aforementioned image completion algorithm is:

[0041] in, The output image to be completed can be represented directly, even without the need for a text-based enhanced completion generation mechanism. A segmented image indicating the presence of image defects; This represents a mask for the damaged area, which can be generated through manual interactive annotation or AI visual inspection technology. Indicates user text commands; Indicates the concatenation function; This represents a text instruction feature extraction function, which is used to convert natural language instructions into vectors that can be aligned with visual features; This represents a visual feature extraction function, which is used to simultaneously extract global semantic features and local features of masked regions from damaged images; These represent the model parameters of the two-stream multimodal diffusion Transformer.

[0042] The above image completion algorithm uses The function achieves deep fusion of text and visual features, and then generates the image to be completed or the completed image through a diffusion model, which solves the problem of "semantics and instructions being disconnected" in traditional models.

[0043] As for the spatial gating attention region perception mechanism, its algorithmic expression is as follows:

[0044] in, Indicates the input image; Indicates a corrupted mask; The query vector representing the image; This represents the key vector that focuses only on the damaged region covered by the mask. Indicates the transpose symbol; A value vector representing an image; This represents the dimension of the key vector, used for normalization to avoid gradient vanishing. This represents a spatial gating function that dynamically adjusts attention weights through a broken mask, making the non-broken region ( The attention weight of (=0) approaches 0, and the damaged area ( Maximize the attention weights of (=1) to avoid semantic drift in non-target regions during repair; This represents a soft maximum function, used to transform any real input into a probability distribution, with output values ​​between 0 and 1.

[0045] By utilizing the aforementioned spatial gating attention region perception mechanism, the technical effect of "only modifying the damaged area without touching the complete area" can be achieved, greatly improving the image damage repair accuracy of the method in this embodiment.

[0046] Furthermore, the algorithmic expression for the text instruction enhanced completion generation mechanism is as follows:

[0047] in, This represents the final image after completion; This represents a feature fusion function used to combine text features. Visual features of damaged images Perform cross-modal alignment and fusion; This represents a generator function that generates complete content based on fused features and a broken mask. This represents the weighting coefficient of the text instruction, with a value ranging from 0.1 to 0.9. It is used to adjust the degree of influence of the text instruction on the completion result. The larger the coefficient, the more closely the completed content matches the text description.

[0048] By introducing the aforementioned text instruction-enhanced completion generation mechanism, personalized and intention-based incomplete completion techniques can be achieved.

[0049] Furthermore, to enable the method in this embodiment to simultaneously optimize image sharpness, purity, and color consistency while completing incomplete repair, thereby achieving a one-click repair process, it is necessary to introduce a total repair loss function, the mathematical expression of which is:

[0050] in, It is the total loss function; It is a loss due to incomplete repair; It is the over-scoring loss; It is the loss in noise reduction; It's a loss of coloring; , , , This is the loss weight for the corresponding task.

[0051] It should be noted that after completing the image completion of each layer using the above image incompleteness completion model, for the portrait layer, the method in this embodiment further includes: S400: Input the portrait layer in the completed image into the population reconstructor to reconstruct the style of the human figure in the portrait layer to adapt it to the pre-configured target delivery area.

[0052] In this embodiment, the aforementioned population reconstructor is used to explicitly decouple facial identity and ethnic style features. Specifically, based on the traditional multi-domain image translation framework, a dedicated ethnic style extractor is designed to learn and separate ethnic-related style codes from the data in an unsupervised manner, ensuring that the generation process only modifies ethnic attributes without destroying identity information. Simultaneously, a deep learning-based facial recognition algorithm and its identity consistency loss are introduced, along with adversarial loss, domain classification loss, and cycle consistency loss, to achieve multi-domain ethnic conversion without a reference image under a single generator and single discriminator structure. This significantly improves the accuracy of ethnic feature transfer and the ability to preserve facial identity.

[0053] Furthermore, the population style encoder of the aforementioned population reconstructor is configured as follows:

[0054]

[0055] in, This represents a population style extractor, which is capable of extracting styles from input face images. and corresponding racial domain tags Unsupervised extraction of low-dimensional racial style codes Racial style codes include racial characteristics such as skin color, facial contours, and facial proportions; It is the identity feature of a facial image; It is a decoupled latent space vector, which is formed by concatenating ethnic style encoding and identity features.

[0056] By configuring the population style encoder described above, the method in this embodiment achieves explicit separation of identity features and ethnic style features in the latent space, ensuring that only ethnic style is modified during the generation process while identity information is preserved.

[0057] Furthermore, to quantify the distance between the identity feature vectors of the input image and the transformed image, thereby constraining the generator to modify ethnic features without compromising facial identity, this embodiment also introduces an identity consistency loss function, the algorithm of which is described as follows:

[0058] in, This represents the identity consistency loss value; the smaller the value, the higher the degree of matching between the identity features of the input image and the transformed image. This represents the mathematical expectation of the input face image x and the target race domain label y; This represents a pre-weighted feature extractor responsible for outputting a facial identity feature vector. This represents a generator that takes the original image and the target ethnic domain label as input and outputs a face image with ethnic conversion. This represents the L1 norm.

[0059] Preferably, the method in this embodiment further includes: Based on the target delivery region mentioned above, the target translation language is determined. That is, if there is a human image completion process, the target delivery region is extracted during the region adaptation process to determine the target translation language, such as English, Spanish, or French.

[0060] S50: Based on the preset layered art configuration, extract the text information of different layers and input it into the preset multilingual translation and typesetting model to obtain reconstructed text of multiple different layers.

[0061] It should be further noted that the aforementioned layered art configuration includes attention to layout balance, visual hierarchy, color harmony, element regularity, and / or area white space. The multilingual translation and typesetting model includes a translation module and a typesetting module. For the translation module, existing text recognition and text translation mapping methods can be used to achieve text translation. For the typesetting module, this embodiment introduces a task distillation and unified reward feedback mechanism, which enables the typesetting module to simultaneously perform multiple tasks, including partial poster editing and global creation.

[0062] Specifically, the aforementioned task distillation and unified reward feedback mechanism includes a basic diffusion generation algorithm, a regional feature alignment algorithm, a spatial layout constraint loss, and an aesthetic scoring algorithm.

[0063] Among them, the basic diffusion generation algorithm constrains the diffusion iteration range through region-aware conditions, ensuring that content is generated only within the user-defined area. It forms the underlying generation foundation for local typography. The region feature alignment algorithm achieves feature alignment between local typographic content and the reference layout and overall environment through dual constraints. It is the core constraint that ensures typographic consistency and style uniformity. The spatial layout constraint loss precisely constrains the spatial position, size, and spacing of elements within the region through DIoU loss. It is the key loss that ensures the spatial accuracy and layout rationality of local typography. The aesthetic scoring algorithm constructs a professional aesthetic evaluation system through multi-dimensional weighted scoring. It serves as the core scoring basis for the model to optimize the visual aesthetics of local typography.

[0064] More specifically, the expression for the basic diffusion generation algorithm described above is:

[0065] in, The representative diffusion model in the 1st The hidden features of the step, The representative diffusion model in the 1st The hidden features of the step; It is the first The noise scale of the step decreases as the number of diffusion steps increases, and is used to control the randomness and stability of the generation process, gradually guiding the noise to transition into clear content. It is the region-aware condition input, which includes the user-defined region mask, the reference layout / content features within the region, and the surrounding environment features outside the region. It is a key constraint that limits the generation range. This represents the unified diffusion generation function, which is responsible for learning the mapping from noise to the target typographic content.

[0066] More specifically, the region feature alignment algorithm is as follows:

[0067] in, It is a high-dimensional feature of the content generated within the defined area, covering core dimensions such as layout structure, texture style, and lighting and color tone; It is a user-specified reference layout or content feature within the area, used to constrain the layout and style alignment of the generated content; It refers to the characteristics of the surrounding environment outside the defined area, used to ensure the consistency between the local layout and the overall poster style; It is a balance coefficient used to adjust the weight of the alignment of the reference layout with the integration of the environmental style, so as to avoid excessive bias towards one side; It is the L2 norm, which measures the difference between the generated features and the reference features; This represents the cosine similarity function, used to measure the distributional similarity between generated features and environmental features.

[0068] More specifically, the aforementioned spatial layout constraint losses are as follows:

[0069]

[0070] in, This is the spatial layout loss value; It is the bounding box of elements within the region generated by the diffusion model; It is the corresponding reference bounding box, that is, the layout balance attention, which can be the user's expected layout position and size; It is the ratio of distance to intersection to union; It is the bounding box intersection-union ratio, used to measure the reasonableness of element coverage and overlap; It is the Euclidean distance between the center of the generated box and the center of the reference box, used to constrain the position offset of elements; It is the minimum center distance between the generated box and the reference box, used to constrain the spatial scale of the overall layout.

[0071] As for the most important aesthetic scoring algorithm, its algorithmic description is as follows:

[0072] in, This is the aesthetic loss value, used to quantify the aesthetic quality deviation of the generated typography within the defined area; The total number of core dimensions for evaluating poster aesthetics covers five major professional design considerations: layout balance, visual hierarchy, color harmony, element regularity, and reasonable use of white space. For the first The weighted coefficients of each aesthetic dimension are obtained by training with manually labeled aesthetic data and are used to strengthen the constraint weights of key aesthetic indicators. For the first A single-item aesthetic scoring function for each dimension; For the first Structural features of local layout in each dimension; For the first Style characteristics of partial layout in each dimension; For the first The harmony features of local layout in each dimension; the score range for each feature is [0,1], and the higher the score, the better the aesthetic performance of that dimension.

[0073] Specifically, by weighting the scores of each dimension and taking the average, and then subtracting the average from 1, the final aesthetic loss is obtained. The smaller the loss value, the higher the local typographic aesthetic quality within the defined area.

[0074] S60: Combine the completed image and the reconstructed text to obtain the reconstructed image.

[0075] It should be noted that the above image segmentation-completion process and text reconstruction process only change the image or text content of the corresponding layer, without changing its original layer number. When executing step S60, the layers are reorganized according to their layer numbers to restore the original occlusion relationship between the layers.

[0076] Through the above technical description, unlike existing technologies, the method of this embodiment can obtain reconstructed images of higher quality and with stronger aesthetic adaptability, specifically reflected in the following five dimensions: Firstly, it can utilize the Qwen series of multimodal models for semantic understanding, guide the division of the text-subject-background layer order, generate structured semantic labels, and then use the SAM3 model for zero-shot semantic segmentation, thereby achieving pixel-level accurate image matting.

[0077] Secondly, the entire process can be driven by natural language, enabling collaborative processing of multiple models without the need to execute mask or Diffusion Transformer global semantic generation.

[0078] Third, the facial feature conversion can preserve the original facial identity features and adapt to local appearance characteristics (skin color / facial features / contours), making the entire transition process smoother and more natural.

[0079] Fourth, it can support customized text conversion for over a hundred languages, with controllable translation quality.

[0080] Fifth, it can automatically optimize the layout based on the aesthetic scoring algorithm, and achieve adaptive typesetting for multiple languages ​​and automatic adaptation of text length.

[0081] Example 2 like Figure 2 As shown, this embodiment discloses a system for one-click localization translation and content reconstruction of image materials, including a processor and a memory. The memory stores computer program instructions. When the computer program instructions are executed by the processor, the method for one-click localization translation and content reconstruction of image materials described in Embodiment 1 is implemented.

[0082] The system in this embodiment also includes other components well known to those skilled in the art, such as communication interfaces. Their settings and functions are known in the art, and therefore will not be described in detail here.

[0083] In this invention, the aforementioned memory can be any tangible medium containing or storing a program that can be used or combined with an instruction execution system, apparatus, or device. For example, a computer-readable storage medium can be any suitable magnetic or magneto-optical storage medium, such as Resistive Random Access Memory (RRAM), Dynamic Random Access Memory (DRAM), Static Random Access Memory (SRAM), Enhanced Dynamic Random Access Memory (EDRAM), High-Bandwidth Memory (HBM), Hybrid Memory Cube (HMC), etc., or any other medium that can be used to store desired information and can be accessed by an application, module, or both. Any such computer storage medium can be part of a device or accessible to or connected to a device. Any application or module described in this invention can be implemented using computer-readable / executable instructions that can be stored or otherwise maintained by such a computer-readable medium.

[0084] In the description of this specification, "multiple" means at least two, such as two, three or more, etc., unless otherwise expressly and specifically defined.

[0085] While this specification has shown and described numerous embodiments of the invention, it will be apparent to those skilled in the art that such embodiments are provided by way of example only. Many modifications, alterations, and alternatives will occur to those skilled in the art without departing from the spirit and essence of the invention. It should be understood that various alternatives to the embodiments of the invention described herein may be employed in the practice of this invention.

Claims

1. A method for one-click localization translation and content reconstruction of image materials, characterized in that, include: In response to the input of the original image, a preset multimodal semantic parsing model is used to receive and parse the original image to extract image element information and text information; Based on the image element information and the text information, a fusion prompt feature is generated; the fusion prompt feature is input into a preset image segmentation model to obtain segmented images with multiple layers; The segmented image is input into a preset image incompleteness completion model to obtain a completed image with multiple layers; Based on the preset layered art configuration, the text information of different layers is extracted and input into the preset multilingual translation and typesetting model to obtain reconstructed text of multiple different layers; The completed image and the reconstructed text are combined to obtain the reconstructed image.

2. The method for one-click localization translation and content reconstruction of image materials according to claim 1, characterized in that, Based on the image element information and the text information, a fused prompt feature is generated, including: The image element information and / or the text information are input into a preset weighted fusion algorithm to calculate the fusion prompt features.

3. The method for one-click localization translation and content reconstruction of image materials according to claim 2, characterized in that, The weighted fusion algorithm includes: in, It is a concept c Fusion prompt features, concepts c It refers to the target object that needs to be segmented from the image element information; It is aimed at the concept c Extracted semantic features of text information; It is aimed at the concept c Extracted spatial features of visual cues for points / boxes; These are reference features of the reference image; , and For adaptive weight values, .

4. The method for one-click localization translation and content reconstruction of image materials according to claim 3, characterized in that, The target objects include the advertising subject, the portrait, the background and / or the foreground.

5. The method for one-click localization translation and content reconstruction of image materials according to claim 1, characterized in that, The image incompleteness completion model includes an image inpainting algorithm based on a two-stream multimodal diffusion architecture, a spatially gated attention region perception mechanism, a text instruction-enhanced completion generation mechanism, and a total completion loss function; The image completion algorithm is used to repair image defects in the segmented image to obtain an image to be completed; the spatial gating attention region perception mechanism is used to adjust the completion attention weight of the image completion algorithm; and the text instruction enhanced completion generation mechanism is used to complete the image to be completed and output the completed image.

6. The method for one-click localization translation and content reconstruction of image materials according to claim 5, characterized in that, The complete total loss function is specifically as follows: in, It is the total loss function; It is a loss due to incomplete repair; It is the over-scoring loss; It is the loss in noise reduction; It's a loss of coloring; , , , This is the loss weight for the corresponding task.

7. The method for one-click localization translation and content reconstruction of image materials according to claim 4, characterized in that, If the target object includes a human figure, the method further includes, before recombining the completed image and the reconstructed text: The portrait layer in the completed image is input into the population reconstructor to reconstruct the style of the human figure in the portrait layer so that it is adapted to the pre-configured target deployment area.

8. The method for one-click localization translation and content reconstruction of image materials according to claim 7, characterized in that, Also includes: Based on the target delivery area, determine the target translation language.

9. The method for one-click localization translation and content reconstruction of image materials according to claim 1, characterized in that, The layered art configuration includes attention to layout balance, visual hierarchy, color coordination, element regularity, and / or area white space.

10. A system for one-click localization translation and content reconstruction of image materials, characterized in that, It includes a processor and a memory, wherein the memory stores computer program instructions, and when the computer program instructions are executed by the processor, the one-click localization translation and content reconstruction method for image materials as described in any one of claims 1-9 is implemented.

Images