Image processing method and apparatus

Abstract

The application relates to the technical field of image processing, and provides an image processing method and device. The method comprises the following steps: acquiring a background image and a text description of a target to be synthesized; obtaining target region indication information based on visual features of the background image and semantic features of the text description, wherein the target region indication information comprises position information and scale information of the target to be synthesized in the background image; processing the background image based on the target region indication information to generate a space constraint control condition; and generating a target image fused with the background image based on the space constraint control condition and the text description. The image processing method and device provided by the application can accurately determine the position and scale of the target to be synthesized in the background image by fusing the visual features and the semantic features, and can generate a space constraint control condition, so that the image synthesis process is effectively constrained, and the generated target image is more accurate and reasonable in terms of spatial layout, scale ratio and fusion degree with the background.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to an image processing method and apparatus. Background Technology

[0002] With the development of computer vision and artificial intelligence technologies, image generation and editing technologies are increasingly widely used in fields such as industrial inspection and creative design. Traditional image generation methods mainly include simple transformations based on data augmentation and synthesis methods based on generative models.

[0003] However, simple transformations based on data augmentation struggle to simulate the shape and physical properties of real targets. While generative model-based synthesis methods can achieve a degree of controllable generation, they still suffer from issues such as imprecise text instruction control, unnatural blending of targets and backgrounds, and poor cross-scene adaptability. For example, mask-based inpainting methods rely on local context for content filling, which can lead to semantic inconsistencies and abrupt texture transitions in complex scenes. Meanwhile, instruction-based editing methods often suffer from target position shifts, scale misalignment, or background distortion due to insufficient spatial awareness.

[0004] Therefore, existing technologies still have significant shortcomings in terms of precise spatial control of text commands, natural integration of generated content with the scene, and cross-scene adaptability. There is an urgent need for a method that can achieve high-precision and high-naturalness image editing and generation. Summary of the Invention

[0005] This application provides an image processing method and apparatus to address the technical defects in the prior art where the content generated by image editing does not blend naturally with the scene, thereby achieving high-precision, high-naturalness, and scene-adaptive image editing and compositing.

[0006] In a first aspect, embodiments of this application provide an image processing method, comprising the following steps:

[0007] Obtain the background image and the text description of the target to be synthesized, wherein the background image is the image obtained after target region elimination processing based on the original image;

[0008] Based on the visual features of the background image and the semantic features of the text description, target region indication information is obtained, which includes the position and scale information of the target to be synthesized in the background image.

[0009] Based on the target area indication information, the background image is processed to generate spatial constraint control conditions;

[0010] Based on spatial constraints and textual descriptions, a target image is generated that is fused with the background image.

[0011] In one embodiment, target region indication information is obtained based on the visual features of the background image and the semantic features of the text description, which can be specifically implemented as follows:

[0012] The background image and text description are input into a pre-trained region localization model to obtain the target region indication information output by the region localization model.

[0013] In yet another embodiment, the background image and text description are input into a pre-trained region localization model to obtain target region indication information output by the region localization model, which can be specifically implemented as follows:

[0014] Based on the visual features of the background image and the semantic features of the text description, cross-modal fusion and parsing are performed to obtain fused features;

[0015] Heatmaps and predicted scale values ​​are generated based on fusion features. The heatmaps are used to characterize the reasonableness score of placing the target to be synthesized at various locations in the background image.

[0016] Based on the heat map and predicted scale values, location and scale information are determined.

[0017] In another embodiment, the background image is processed based on the target region indication information to generate spatial constraint control conditions, which can be specifically implemented as follows:

[0018] Filter at least one candidate region with a score higher than a preset threshold from the heat map;

[0019] The bounding box of the target to be synthesized in the background image is determined based on the candidate region. The bounding box is used to define the location and scale information.

[0020] Generate a control graph containing the mask corresponding to the region inside the bounding box, which serves as the spatial constraint control condition.

[0021] In yet another embodiment, a target image fused with the background image is generated based on spatial constraint control conditions and text description, which can be specifically implemented as follows:

[0022] Spatial constraints and textual descriptions are input into a pre-trained image generation model to generate a target image that is fused with the background image.

[0023] In yet another embodiment, the region localization model is trained in the following manner:

[0024] Construct a training sample set, where each training sample includes: the original image, the corresponding background image, the control graph, and the text description;

[0025] The background image is obtained by image processing of the labeled target regions in the original image; the control chart is obtained by masking the labeled target regions in the original image.

[0026] The background image and text description are used as input to the model, and the location and scale of the labeled target area are used as supervision signals to train the region localization model.

[0027] Secondly, embodiments of this application provide an image processing apparatus, including the following modules:

[0028] The image acquisition module is used to acquire a background image and a text description of the target to be synthesized. The background image is obtained by removing the target region from the original image.

[0029] The image localization module is used to obtain target region indication information based on the visual features of the background image and the semantic features of the text description. The target region indication information includes the position information and scale information of the target to be synthesized in the background image.

[0030] The image processing module is used to process the background image based on the target area indication information and generate spatial constraint control conditions;

[0031] The image generation module is used to generate a target image that is fused with the background image based on spatial constraint control conditions and text description.

[0032] Thirdly, embodiments of this application provide an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the image processing method of the first aspect.

[0033] Fourthly, embodiments of this application provide a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the image processing method of the first aspect.

[0034] Fifthly, embodiments of this application provide a computer program product, including a computer program that, when executed by a processor, implements the steps of the image processing method of the first aspect.

[0035] The image processing method and apparatus provided in this application locate the target region by fusing visual and semantic features, enabling precise determination of the position and scale of the target to be synthesized in a background image. Based on the spatial constraint control conditions generated from the target region indication information, effective constraints can be imposed on the image synthesis process, making the generated target image more accurate and reasonable in terms of spatial layout, scale ratio, and integration with the background, thereby improving the quality and realism of the synthesized image. Attached Figure Description

[0036] To more clearly illustrate the technical solutions in this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0037] Figure 1 A flowchart illustrating the image processing method provided in this application.

[0038] Figure 2 A flowchart illustrating the method for implementing the regional positioning model provided in this application.

[0039] Figure 3 A flowchart illustrating the training method for the regional positioning model provided in this application.

[0040] Figure 4 A schematic diagram of the image processing apparatus provided in this application.

[0041] Figure 5 A schematic diagram of the structure of the electronic device provided in this application. Detailed Implementation

[0042] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0043] With the rapid development of industrial intelligence and computer vision technology, the demand for visual anomaly detection and creative image editing is becoming increasingly urgent in various closed and open environments. This encompasses scenarios such as quality inspection, equipment fault diagnosis, and security monitoring in the power, rail transportation, and manufacturing industries, as well as image editing scenarios including artistic element synthesis, automatic image retouching (such as intelligent retouching, flaw repair, and content replacement), and creative image stitching. High-quality target images (including anomaly images from various closed and open environments, and creatively edited images) serve as the core support for model training or direct application, and their quality and diversity directly determine the application effect of related technologies. However, real-world target images suffer from scarcity, high acquisition costs (requiring manual screening and labeling), and complex and varied types, making it difficult to fully cover potential needs through real-world acquisition. Therefore, high-quality target image generation technology has become a key means to overcome data bottlenecks.

[0044] Current mainstream target image generation methods can be divided into two categories: one is traditional data augmentation methods, which generate "pseudo-target images" through simple transformations such as rotation, cropping, and noise addition, but these methods struggle to simulate the morphological features and physical properties of real targets and lack precise control over target type, location, and style; the other is synthesis methods based on generative models, which mainly include two technical approaches:

[0045] 1. Masking Inpainting: This method fills in content based on the surrounding context of the target area to generate a visually consistent image. However, it is prone to problems such as inconsistent semantic logic and non-fusion of texture features in complex scenes. For example, the currently popular image inpainting model, Stable Diffusion Inpainting, often results in blurred edges and harsh transitions in editing scenarios such as compositing minor anomalies like loose solder joints on circuit boards or scratches on bearing surfaces, or embedding artistic elements and repairing image defects due to insufficient local texture capture.

[0046] 2. Instruction Editing: Deep learning models, such as InstructPix2Pix, can adjust target attributes or insert new targets through text prompts. However, due to insufficient spatial awareness, the generated results often have problems such as positional offset (misalignment of edited elements), scale imbalance (mismatch in proportions, unbalanced size of creative elements), and background distortion (disjointed style of the edited scene). It is difficult to meet the requirements of various closed and open environments for target generation accuracy and naturalness.

[0047] In summary, existing technologies still have significant shortcomings in areas such as precise control of text commands, natural blending of targets and backgrounds, and targeted adaptation to various closed and open environments. To address these issues, this application proposes an image processing method that locates potential target regions through text semantics and performs repairs in designated areas based on prior scene knowledge. This improves the controllability and realism of target image generation, adapting to the application needs of various closed and open environments and multiple image editing scenarios.

[0048] The following is combined Figures 1 to 5 The image processing methods and apparatus provided in the embodiments of this application will be described in detail.

[0049] It should be noted that the core of the image processing method proposed in this application lies in a two-stage processing framework: In the first stage, the region localization model intelligently predicts the position and scale of the target to be synthesized based on the input background image and text description, and generates spatial constraint control conditions based on the position and scale of the target to be synthesized; In the second stage, the image generation model uses the spatial constraint control conditions as spatial guidance and combines them with the input text description to generate a high-quality target image that blends naturally with the background in the specified region.

[0050] Figure 1A flowchart illustrating the image processing method provided in this application is shown below. Figure 1 As shown, the method includes the following:

[0051] Step 101: Obtain the background image and the text description of the target to be synthesized.

[0052] The background image is the image obtained after target region elimination processing based on the original image.

[0053] For example, the background image could be a clean street scene image obtained by removing specified pedestrians or vehicles from an original street photograph containing such images. Alternatively, it could be an image of a intact, defect-free bearing surface obtained by removing the scratched areas from an original image of a bearing with surface scratches using image inpainting techniques.

[0054] In one embodiment, target region removal can be achieved through a pre-trained image inpainting model. The image inpainting model receives the original image and the location information of the target region to be removed (e.g., the location information can be represented in the form of a rectangular bounding box), and performs content filling and inpainting on the target region based on the contextual information surrounding the target region (such as texture, structure, and color continuity).

[0055] The target to be composited refers to the visual entity object that needs to be composited and added to the background image.

[0056] Text descriptions are natural language phrases or statements used to describe the category, attributes, state, or appearance of the target to be synthesized. Text descriptions provide high-level semantic information to guide the model in understanding "what kind of target needs to be generated".

[0057] In one embodiment, a background image of an empty street and the text description "a pedestrian with an umbrella" are used, and "a pedestrian with an umbrella" is the target to be synthesized.

[0058] In another embodiment, given a background image of an intact wall and the text description "a graffiti", then "a graffiti" is the target to be synthesized.

[0059] Step 102: Based on the visual features of the background image and the semantic features of the text description, obtain the target area indication information.

[0060] In this embodiment of the application, the background image and text description are input into a pre-trained region localization model to obtain the target region indication information output by the region localization model.

[0061] The target region indication information includes the location and scale information of the target to be synthesized in the background image.

[0062] In this embodiment of the application, the location information refers to the center point coordinates or boundary range of the area where the target to be synthesized appears in the background image.

[0063] For example, location information could be a two-dimensional coordinate (x, y) used to define the center point of the target bounding box; or a four-dimensional vector (x, y). min ,y min ,x max ,y max The `<boundary>` property is used to directly define the coordinates of the top-left and bottom-right corners of the bounding box, without any specific restrictions.

[0064] Scale information is used to quantify the expected size or coverage of the target to be synthesized.

[0065] For example, scale information can be the width and height of the predicted bounding box, or a scaling factor (a percentage relative to the background image size or a normalized value), without any specific restrictions.

[0066] It is understandable that, combining location and scale information, the target area indication information can be expressed as a bounding box. For example: Box=(x,y,w,h).

[0067] Where (x,y) are the coordinates of the center point, and (w,h) are the width and height of the bounding box.

[0068] Specifically, the region localization model maps multimodal inputs to target region indication information through the following process: First, it performs cross-modal fusion and global parsing of the visual features of the background image and the semantic features of the text description to obtain fused features that imply image-text correlation. Then, based on these fused features, it performs the tasks of generating heatmaps and predicting scale values ​​in parallel.

[0069] Heatmaps are used to characterize the reasonableness score of placing the target to be synthesized at various locations in the background image. For example, heatmaps can visually characterize the reasonableness of placing the target to be synthesized at each spatial location in the background image in the form of a score (usually a continuous value between 0 and 1).

[0070] The predicted scale value is used to define the reasonable size of the target.

[0071] Furthermore, by analyzing the region with the highest response in the heat map, the location information of the target is determined, and combined with the predicted scale value, a complete and quantified target area indication information is constructed.

[0072] Step 103: Based on the target area indication information, process the background image to generate spatial constraint control conditions.

[0073] Specifically, at least one candidate region with a score higher than a preset threshold is selected from the heat map; the bounding box of the target to be synthesized in the background image is determined based on the candidate region, and a control map containing the mask corresponding to the region inside the bounding box is generated as a spatial constraint control condition.

[0074] The bounding box is used to define location and scale information.

[0075] For example, the heatmap generated based on step 102 can be represented as By using a preset threshold, one or more candidate areas with the highest heat values ​​are selected.

[0076] Specifically, the optimal placement location is determined by searching for global or local maximum points on the heatmap. and optimal prediction scale value For the selected candidate regions, the spatial coordinates and range of the target region in the original image are accurately determined by fitting bounding boxes, thereby clarifying the specific location of the target object in the background image.

[0077] Based on the bounding box information of the target region, the background image The process involves processing to create precise spatial constraint control conditions.

[0078] Specifically, a control graph is generated by setting the pixel value of the image region inside the bounding box to 1. The control chart The overall structure and texture information of the background image are preserved, but its pixel values ​​in the target area are set to black, thus providing a clear and visually recognizable spatial constraint signal for the subsequent target image generation model.

[0079] This control condition ensures that the generation process can accurately focus on the target region, guiding the model to perform conditional generation within that region, while avoiding unnecessary modifications in other regions.

[0080] Step 104: Based on spatial constraint control conditions and text description, generate a target image that is fused with the background image.

[0081] Specifically, spatial constraints and textual descriptions are input into a pre-trained image generation model to obtain a target image fused with the background image.

[0082] In one embodiment, the image generation model is constructed and trained as follows:

[0083] 1. Model architecture: Based on the Flux architecture and combined with the ControlNet control mechanism, a dual-channel generation network consisting of a generation path and a control path is constructed.

[0084] Generation path: As the backbone network, it is responsible for modeling the global semantics and overall consistency of the image, and completing the denoising process from noise to a clear image.

[0085] Control Path: As a conditional control network, it takes the control graph C as input, extracts its spatial structure features, and injects spatial constraints into multiple intermediate layers of the generation path through cross-layer connections, so as to achieve precise and fine-grained control over the position, shape and structure of the generated content.

[0086] 2. Training method: For each training sample, the original image is known. Control charts and the corresponding target category text description Its training samples are represented as follows: That is, the image-control-text triplet.

[0087] The training process aims to optimize the model parameters θ, enabling the image generation model to reconstruct images that are consistent with the actual image given a control graph C and a text description T. Highly consistent target images .

[0088] Control conditions for the target area and text description Input the trained image generation model:

[0089]

[0090] From the above formula, it can be understood that a well-trained image generation model Control conditions of the target area The text description T is used as the joint input condition. After one forward propagation, the final target image Î is directly output. Target region control conditions. It provides spatial location and range constraints; the text description T provides target category and attribute constraints to ensure semantic consistency of the generated results.

[0091] Final output target image The spatial location, semantic attributes, and scene adaptability are all consistent with the preset target, thus ensuring that the generated results have high quality and practical value.

[0092] The image processing method provided in this application locates the target region by fusing visual and semantic features, enabling precise determination of the position and scale of the target to be synthesized within the background image. The spatial constraint control conditions generated based on this target region indication information effectively constrain the image synthesis process, making the generated target image more accurate and reasonable in terms of spatial layout, scale ratio, and integration with the background, thereby improving the quality and realism of the synthesized image.

[0093] Figure 2 A flowchart illustrating the regional positioning model execution method provided in this application is shown below. Figure 2 As shown, the method includes the following:

[0094] It should be noted that the execution method of the regional localization model, namely the specific content in step 102, is the specific execution process of inputting the background image and text description into the pre-trained regional localization model to obtain the target area indication information output by the regional localization model.

[0095] Step 201: Based on the visual features of the background image and the semantic features of the text description, perform cross-modal fusion and parsing to obtain fused features.

[0096] Specifically, the regional positioning model adopts a dual encoder architecture and a Transformer fusion module to achieve cross-modal fusion and parsing.

[0097] Optionally, the dual encoder architecture includes a visual encoder and a semantic encoder.

[0098] A visual encoder is used to extract visual features from a background image. Optionally, visual features include, but are not limited to, texture information, structural information, color information, and spatial layout information.

[0099] A semantic encoder is used to extract semantic features from text descriptions. Optionally, semantic features include, but are not limited to, category information, attribute information, etc.

[0100] In this embodiment of the application, the background image is... The input is fed into a visual encoder, which performs layer-by-layer feature transformations through its multi-layer network structure, and outputs visual features. ,in, The spatial dimension of visual features The channel dimension for visual features.

[0101] Then, the text description T is input into the semantic encoder, and after word segmentation and embedding operations, it is processed through its network structure to output semantic features. ,in, As a dimension of semantic features, this vector represents the overall semantic meaning of the text description.

[0102] Furthermore, visual features are incorporated through a learnable linear projection layer. and semantic features Perform dimensional transformations separately.

[0103] For example, let the linear projection function be... ,in, This is the weight matrix. Let be the bias vector. Then the aligned multimodal features are represented as:

[0104]

[0105] in, The aligned feature dimensions, that is, the aligned visual features with consistent dimensions, are generated. and alignment semantic features That is, through two independent learnable linear transformations ( and ), which will incorporate visual features derived from branches and different dimensions. and semantic features This is mapped to a joint semantic space of uniform dimension (all d-dimensional), thereby generating and W v b v These are the weight matrix and bias vector for the visual features, respectively; W s b s These are the weight matrix and bias vector for the semantic features, respectively.

[0106] Then, after aligning visual and semantic features, a Transformer association model is constructed.

[0107] Specifically, a Transformer architecture containing an L-layer encoder is constructed as a cross-modal association module. This aligns visual features. and alignment semantic features As input to this module. The feature vector of each spatial location is treated as an image patch label, and a two-dimensional sinusoidal positional code is added to it to preserve spatial information; As a semantic marker.

[0108] The self-attention and cross-attention mechanisms in this Transformer architecture establish a global association between visual and semantic tags.

[0109] Furthermore, in each Transformer encoder layer, a multi-head self-attention mechanism is used to process all input tags (including image patch tags and semantic tags). This mechanism can capture long-distance dependencies between spatial locations within visual features and model the bidirectional interaction between semantic features and all visual features.

[0110] Specifically, for the first The input feature set of the layer Its self-attention output The calculation process can be represented as follows: First, Through a learnable linear projection matrix The matrix is ​​converted into a query matrix, a key matrix, and a value matrix, respectively. Then, the dot product of the query and key matrices is calculated and scaled, and attention weights are obtained using the Softmax function. Finally, the weights are multiplied by the value matrix to obtain the output. This output is then residually connected to the input and normalized, and a nonlinear transformation is performed through a multilayer perceptron to obtain the final output of that layer. Through iterative processing at layer L, the deep connection between text semantics and image scenes is continuously optimized and strengthened.

[0111] Finally, after layer-by-layer processing by the L-layer Transformer encoder, all visual tags fully interact with semantic tags in the cross-attention mechanism, realizing a global information mapping from text semantics to image space. Through the dynamic allocation mechanism of attention weights, the Transformer can adaptively focus on semantically relevant regions in the image according to the semantic cues, enabling semantic information to form an explicit response distribution in the spatial dimension, thereby establishing a semantically guided spatial attention pattern.

[0112] It's important to note that this fused feature doesn't directly generate a definite location and scale, but rather is learned gradually through subsequent supervised training. During training, the Transformer continuously adjusts its internal attention distribution and feature weights through joint optimization of scale prediction and location heatmap generation tasks, establishing a stable mapping relationship between semantic cues and image spatial location and region scale. As training progresses, the Transformer gradually acquires explicit spatial localization and scale reasoning capabilities, automatically focusing on the most reasonable region in the image and predicting its scale based on the input semantics.

[0113] Finally, the fused features are output. ,in, , ), These represent the spatial size and feature dimension of the feature map, respectively. This fused feature, while maintaining global semantic consistency, contains the spatial response structure and scale distribution features learned through training, providing unified and interpretable input support for subsequent prediction scale values ​​and heatmap generation modules.

[0114] Step 202: Generate a heatmap and predicted scale value based on the fusion features. The heatmap is used to characterize the reasonableness score of placing the target to be synthesized at each position in the background image.

[0115] Specifically, the steps for generating heatmaps and predicted scale values ​​based on fused features include the following sub-steps:

[0116] Step 1: Construct the prediction scale branch: Combine the fused features output by the Transformer. Input is fed into the lightweight prediction scaling module. First, global average pooling is performed on the fused features to obtain a global feature vector that integrates global contextual information:

[0117]

[0118] in, Represents the global feature vector. This represents the pooling operation function. It is used to average all spatial locations (H×W) of each channel (d channels in total) of the fused feature F, thereby compressing the two-dimensional spatial feature into a one-dimensional vector and aggregating the global context.

[0119] Step 2: Optimal prediction scale: Scale the global feature vector... The input is processed by a multilayer perceptron, undergoing nonlinear transformation and activation function, and the output is the optimal predicted scale value of the target. Optimal prediction scale A reasonable size range for quantifying the target region:

[0120]

[0121] in, The MLP stands for Core Decoder, which represents the optimal prediction scale and is typically composed of fully connected layers, activation functions, and possibly normalization layers stacked sequentially. It is responsible for... Mapped to the target scale.

[0122] During the training phase, a scale supervision signal, namely the true scale value, is introduced. The true scale value is derived from the rectangular bounding boxes B labeled in the training set. Specifically, for each bounding box labeled with a target region, the side length of the bounding box is defined as the true scale value of that sample. Its geometric center is defined as the location of the true target region. The difference between the predicted scale value and the true scale value is measured by the mean squared error loss.

[0123]

[0124] in, To determine the error between the predicted scale value and the true scale value. This represents the number of samples. Let be the predicted scale value for the i-th sample. Let be the true scale value of the i-th sample. This formula can effectively constrain the predicted scale value output by the MLP to approximate the true scale value, thereby improving the accuracy and stability of target region size estimation.

[0125] Step 3: Constructing the heatmap generation branch: fusing features from the Transformer Compared with the true scale value Common input decoder. The decoder consists of multiple layers of "convolution-upsampling" modules, which progressively map high-dimensional features into a heatmap:

[0126]

[0127] Where H represents the heatmap, and the dimension is... H and W represent the height and width of the heatmap, respectively. Each pixel value H(x,y) in the heatmap represents the image location (x,y) at a scale of [missing information]. The placement of the target to be composited is rated on its reasonableness; the higher the score, the more reasonable the placement. denoted as decoder, and F represents fused feature.

[0128] This heatmap provides location-scale dual-dimensional guidance for subsequent modules, thereby achieving accuracy and interpretability in target detection and localization.

[0129] During the training phase, the location of the real target area is set as follows: The corresponding heatmap score should be significantly higher than other locations. Therefore, a sparse contrastive loss function is used as a constraint:

[0130]

[0131] in, H(x,y) represents the heatmap loss value, and H(x,y) represents the reasonableness score of the location (x,y) in the heatmap. Indicates the actual location of the target area Reasonableness score, This represents all candidate spatial locations in the heatmap, i.e., each pixel or each grid point. For the set boundary value, the neighborhood of the real location ( The value is set to 0, and the value for the far-from-the-target area is set to a constant, such as 0.1. This maximizes the score for the true target area, while ensuring that the score for locations far from the true target area does not exceed the true value minus the set boundary value. A range constraint term is also introduced:

[0132]

[0133] in, Indicates the range loss value. The score indicating the constrained true position approaches 1. This indicates that the minimum value of the constrained heatmap approaches 0.

[0134] This ensures that the score range is within [0,1] and has discriminative power. The final heatmap loss is:

[0135]

[0136] in, This represents the total heat map loss value. Indicates the range loss value. This represents the heatmap loss value.

[0137] in, , These represent the weights for the two types of sub-losses. This loss design highlights the saliency of the true target location-scale while allowing the retention of multiple reasonable candidate regions, thereby improving the robustness and interpretability of target localization and predicted scale values.

[0138] Step 203: Determine location information and scale information based on the heat map and predicted scale values.

[0139] Specifically, the heatmap H is analyzed to determine the placement locations of one or more high-confidence targets.

[0140] In one embodiment, the optimal target center coordinates (x, y) can be directly determined by finding the global maximum point of the heatmap.

[0141] In another embodiment, to accommodate multiple possible reasonable locations or improve robustness, a preset confidence threshold can be set to filter out all pixels or connected regions with thermal values ​​higher than the threshold. Then, one or more candidate location coordinates {(x...} can be obtained through clustering or calculating region centroids. i ,y i )}.

[0142] The scale information is directly taken from the predicted scale value output by the predicted scale value branch. This value is a numerical measure of the reasonable size of the target, such as representing the side length, area, or a normalized scaling factor of the bounding box. When multiple candidate locations exist, the same prediction scale value can be shared. Alternatively, the corresponding scale can be generated by fine-tuning based on the local context of each candidate region.

[0143] The specific form of combining the parsed location information with the determined scale information is to convert it into one or more bounding boxes.

[0144] For example, a shape centered at position (x, y) with width and height determined by... The determined rectangle. Therefore, it is defined as Box=(x,y,w,h) or a similar structured representation.

[0145] The image processing method provided in this application achieves a deep association between text semantics and visual scene through cross-modal fusion, enabling the generated heat map and predicted scale value to collaboratively indicate the most reasonable position and size of the target. This effectively overcomes the common problems of positioning offset and scale mismatch in traditional methods, and significantly improves the spatial accuracy and semantic controllability of image editing.

[0146] Figure 3 A flowchart illustrating the training method for the regional positioning model provided in this application is shown below. Figure 3 As shown, the method includes the following:

[0147] Step 301: Construct a training sample set; where each training sample includes: the original image, the corresponding background image, the control graph, and the text description.

[0148] Specifically, the background image is obtained by image processing of the labeled target areas in the original image; the control map is obtained by masking the labeled target areas in the original image.

[0149] A target region refers to an image region in the original image that corresponds to a specific target object. In the embodiments of this application, the target region can be represented in the form of a rectangular bounding box.

[0150] For example, the original image can be represented as That is, the original image It is an RGB color image with a height of H and a width of W.

[0151] For example, a rectangular bounding box is denoted as B, and B is used to specify the spatial location and extent of the target in the image (such as abnormal areas of industrial equipment, blemish areas in an image, etc.).

[0152] Specifically, the original image The image is input into the image inpainting model. Within the area corresponding to the rectangular bounding box B, the image inpainting model performs a target region elimination operation and outputs a background image I without the target.

[0153] For example, background image I can be represented as: .

[0154] In the original image Fill the rectangular bounding box with black areas to generate a control chart.

[0155] For example, control chart C can be represented as: ,in, This indicates that the area to be filled with black is located at the bounding box of the rectangle.

[0156] The control chart explicitly encodes the spatial location information of the target. Simultaneously, a textual description T of the target category is assigned to each sample, providing semantic constraints.

[0157] Ultimately, each training sample can be represented as a quadruple:

[0158] ,

[0159] Where D represents the training sample quadruple and the original image Preserve target features, background image Provides a target-free reference, control chart Provide spatial constraints and text descriptions It provides semantic constraints. This achieves dual control over both space and semantics, ensuring the accuracy and scalability of training samples.

[0160] Step 302: Use the background image and text description as model input, and the location and scale of the labeled target area as supervision signal to train the region localization model.

[0161] Specifically, the background image I and the text description T are used as inputs to the region localization model. The region localization model outputs the predicted target scale. A heatmap H representing the probability of the target appearing.

[0162] During training, the annotation information of the training samples is used to provide supervision: the center coordinates of the labeled bounding box B are extracted as the true location. ), and extract its size information as the true scale value. .

[0163] The region localization model is optimized by designing a joint loss function. For example, the joint loss function may include:

[0164] Prediction Scale Loss: Measures the prediction scale value Compared with the true scale value The error between them.

[0165] Heatmap supervision loss: constrain the heatmap H to its true location ( The highest response was received at ().

[0166] The region localization model updates all its parameters by minimizing the aforementioned joint loss using the backpropagation algorithm. After iterative training, a region localization model is finally obtained that can automatically and accurately infer the reasonable location and scale of a target based on images and text.

[0167] The image processing method provided in this application, through a sample structure of original image, background image, control chart, and text description, combined with position and scale supervision signals, enables the model to learn the spatial constraints of the target and incorporate semantic associations. This allows the model to accurately grasp the reasonable position and scale of the target in the scene, improve the accuracy of region localization and scene adaptability, and lay a reliable foundation for subsequent image generation.

[0168] The image processing apparatus provided in this application is described below. The image processing apparatus described below can be referred to in correspondence with the image processing method described above.

[0169] Figure 4 A schematic diagram of the image processing apparatus provided in this application. Figure 4 As shown, the device includes the following:

[0170] The image acquisition module 401 is used to acquire a background image and a text description of the target to be synthesized, wherein the background image is an image obtained after target region elimination processing based on the original image;

[0171] The image localization module 402 is used to obtain target region indication information based on the visual features of the background image and the semantic features of the text description. The target region indication information includes the position information and scale information of the target to be synthesized in the background image.

[0172] Image processing module 403 is used to process the background image based on target area indication information to generate spatial constraint control conditions;

[0173] Image generation module 404 is used to generate a target image that is fused with the background image based on spatial constraint control conditions and text description.

[0174] In one embodiment, the image positioning module 402 is specifically used for:

[0175] The background image and text description are input into a pre-trained region localization model to obtain the target region indication information output by the region localization model.

[0176] In yet another embodiment, the image positioning module 402 is specifically used for:

[0177] Based on the visual features of the background image and the semantic features of the text description, cross-modal fusion and parsing are performed to obtain fused features;

[0178] Heatmaps and predicted scale values ​​are generated based on fusion features. The heatmaps are used to characterize the reasonableness score of placing the target to be synthesized at various locations in the background image.

[0179] Based on the heat map and predicted scale values, location and scale information are determined.

[0180] In yet another embodiment, the image processing module 403 is specifically used for:

[0181] Filter at least one candidate region with a score higher than a preset threshold from the heat map;

[0182] The bounding box of the target to be synthesized in the background image is determined based on the candidate region. The bounding box is used to define the location and scale information.

[0183] Generate a control graph containing the mask corresponding to the region inside the bounding box, which serves as the spatial constraint control condition.

[0184] In yet another embodiment, the image processing module 403 is specifically used for:

[0185] Spatial constraints and textual descriptions are input into a pre-trained image generation model to generate a target image that is fused with the background image.

[0186] In yet another embodiment, the region localization model is trained in the following manner:

[0187] Construct a training sample set, where each training sample includes: the original image, the corresponding background image, the control graph, and the text description;

[0188] The background image is obtained by image processing of the labeled target regions in the original image; the control chart is obtained by masking the labeled target regions in the original image.

[0189] The background image and text description are used as input to the model, and the location and scale of the labeled target area are used as supervision signals to train the region localization model.

[0190] Figure 5 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 5As shown, the electronic device may include a processor 510, a communications interface 520, a memory 530, and a communication bus 540, wherein the processor 510, communications interface 520, and memory 530 communicate with each other via the communication bus 540. The processor 510 can call logical instructions in the memory 530 to execute an image processing method, which includes: acquiring a background image and a text description of the target to be synthesized, wherein the background image is an image obtained after target region elimination processing based on the original image; obtaining target region indication information based on the visual features of the background image and the semantic features of the text description, the target region indication information including the position and scale information of the target to be synthesized in the background image; processing the background image based on the target region indication information to generate spatial constraint control conditions; and generating a target image fused with the background image based on the spatial constraint control conditions and the text description.

[0191] Furthermore, the logical instructions in the aforementioned memory 530 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0192] On the other hand, this application also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can perform the image processing methods provided by the above methods. The method includes: acquiring a background image and a text description of the target to be synthesized, wherein the background image is an image obtained after target region elimination processing based on the original image; obtaining target region indication information based on the visual features of the background image and the semantic features of the text description, wherein the target region indication information includes the position information and scale information of the target to be synthesized in the background image; processing the background image based on the target region indication information to generate spatial constraint control conditions; and generating a target image fused with the background image based on the spatial constraint control conditions and the text description.

[0193] In another aspect, this application also provides a non-transitory computer-readable storage medium storing a computer program thereon. When executed by a processor, the computer program is implemented to perform the image processing methods provided by the above methods. The method includes: acquiring a background image and a text description of a target to be synthesized, wherein the background image is an image obtained after target region elimination processing based on the original image; obtaining target region indication information based on the visual features of the background image and the semantic features of the text description, wherein the target region indication information includes the position information and scale information of the target to be synthesized in the background image; processing the background image based on the target region indication information to generate spatial constraint control conditions; and generating a target image fused with the background image based on the spatial constraint control conditions and the text description.

[0194] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0195] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., including several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods of various embodiments or some parts of embodiments.

[0196] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. An image processing method, characterized in that, The method includes: Obtain a background image and a text description of the target to be synthesized, wherein the background image is an image obtained after target region elimination processing based on the original image; Based on the visual features of the background image and the semantic features of the text description, cross-modal fusion and parsing are performed to obtain fused features; a heatmap and a predicted scale value are generated based on the fused features; target region indication information is obtained based on the heatmap and the predicted scale value; the heatmap is used to characterize the reasonableness score of placing the target to be synthesized at each position in the background image, and the target region indication information includes the position information and scale information of the target to be synthesized in the background image; Based on the target region indication information, the background image is processed to generate spatial constraint control conditions, including: selecting at least one candidate region with a score higher than a preset threshold from the heat map; determining the bounding box of the target to be synthesized in the background image based on the candidate region, the bounding box being used to define the location information and the scale information; and generating a control map containing the mask corresponding to the region inside the bounding box as the spatial constraint control conditions. Based on the spatial constraint control conditions and the text description, a target image is generated that is fused with the background image.

2. The image processing method according to claim 1, characterized in that, The step of generating a target image fused with the background image based on the spatial constraint control conditions and the text description includes: The spatial constraint control conditions and the text description are input into a pre-trained image generation model to generate the target image fused with the background image.

3. The image processing method according to claim 1, characterized in that, The step of obtaining the target area indication information is performed by an area positioning model, which is trained in the following way: Construct a training sample set, where each training sample includes: the original image, the corresponding background image, the control graph, and the text description; The background image is obtained by image processing the marked target regions in the original image; the control map is obtained by masking the marked target regions in the original image. The background image and the text description are used as model inputs, and the location and scale of the labeled target area are used as supervision signals to train the region localization model.

4. An image processing apparatus, characterized in that, The device includes: The image acquisition module is used to acquire a background image and a text description of the target to be synthesized, wherein the background image is an image obtained after target region elimination processing based on the original image; An image localization module is used to perform cross-modal fusion and parsing based on the visual features of the background image and the semantic features of the text description to obtain fused features; generate a heatmap and a predicted scale value based on the fused features; and obtain target region indication information based on the heatmap and the predicted scale value. The heatmap is used to characterize the reasonableness score of placing the target to be synthesized at each position in the background image, and the target region indication information includes the position information and scale information of the target to be synthesized in the background image. The image processing module is used to process the background image based on the target region indication information to generate spatial constraint control conditions, including: filtering at least one candidate region with a score higher than a preset threshold from the heat map; determining the bounding box of the target to be synthesized in the background image based on the candidate region, the bounding box being used to define the position information and the scale information; and generating a control map containing the mask corresponding to the region inside the bounding box as the spatial constraint control conditions. An image generation module is used to generate a target image that is fused with the background image based on the spatial constraint control conditions and the text description.

5. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the image processing method as described in any one of claims 1 to 3.

6. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the image processing method as described in any one of claims 1 to 3.

7. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the image processing method as described in any one of claims 1 to 3.

Images