An adaptive image editing control system based on multi-dimensional feature perception

Abstract

The application discloses a kind of adaptive image editing control systems based on multidimensional feature perception, belong to artificial intelligence technical field.The system of the present application includes front-end interaction layer, logic control layer and reasoning execution layer;Wherein front-end interaction layer receives and visualizes display to be edited image and natural language instruction;Logic control layer extracts image physical resolution, instance total number and density by multidimensional feature sensing unit, semantic analysis comparison unit extracts semantic transformation span and complexity, complexity arbitration unit generates joint complexity index according to above parameter, and dynamic routing controller executes workflow dynamic routing selection according to joint complexity index;Reasoning execution layer receives the signal of logic control layer, and global guide intensity and sampling iteration step number of model reasoning engine are dynamically adjusted by inference parameter adaptive mapper using joint complexity index, so as to realize the optimization adaptation of image editing effect in different complexity scenarios.

Description

Technical Field

[0001] This invention relates to an adaptive image editing control system, specifically an adaptive image editing control system based on multi-dimensional feature perception, belonging to the field of artificial intelligence technology. Background Technology

[0002] With the rapid development of deep learning technology, image editing techniques based on diffusion models have become a research hotspot in the field of computer vision. Users can modify, redraw, or transfer style of image content through natural language commands. However, most existing image editing tools adopt fixed processing workflows and static parameter configurations, which cannot be dynamically adjusted according to the complexity of the input image itself and the degree of user intent.

[0003] In practical applications, when the input image has high resolution, densely distributed objects, or the user's instructions differ significantly from the original image's semantics, fixed-weight editing models often suffer from semantic leakage, blurred local details, or low instruction compliance. Furthermore, due to the lack of pre-quantitative evaluation of image features, fixed workflows still utilize redundant high-performance nodes when handling simple tasks, leading to a waste of computing resources. Therefore, a control system capable of sensing multi-dimensional features of the input image and automatically adjusting the computational logic is urgently needed. Summary of the Invention

[0004] This invention aims to provide an image editing control system capable of sensing the deviations in image physical and instance features and semantic intent, and dynamically optimizing workflow or model weights accordingly. This addresses the problem that existing technologies require manual adjustment of global guidance strength and sampling iteration steps when handling complex editing tasks, and are difficult to precisely match with image complexity. Specifically, this invention aims to solve the following technical problems: (1) solving the problem of editing failure caused by the difficulty in quantifying the span of semantic intent; (2) solving the problem of resource mismatch between static workflow and heterogeneous task requirements.

[0005] To achieve the above objectives, the present invention provides an adaptive image editing control system based on multi-dimensional feature perception, comprising a front-end interaction layer, a logic control layer, and an inference execution layer;

[0006] The front-end interaction layer is used to receive user-uploaded images to be edited, natural language editing instructions, and visual displays;

[0007] The logic control layer includes a multi-dimensional feature sensing unit, a semantic analysis and comparison unit, a complexity arbitration unit, and a dynamic routing controller, which are responsible for the extraction and processing of all features.

[0008] The complexity arbitration unit generates a joint complexity index based on the physical resolution, total number of instances, and instance density extracted by the multi-dimensional feature sensing unit, as well as the semantic transformation span and semantic complexity extracted by the semantic analysis and comparison unit; the dynamic routing controller performs dynamic routing selection of the workflow based on the joint complexity index.

[0009] The inference execution layer includes a generative model inference engine, a dynamically mounted node library, and an inference parameter adaptive mapper, which are used to receive signals from the logic control layer and perform parameter calculations.

[0010] The inference parameter adaptive mapper dynamically adjusts the global guidance strength and sampling iteration steps of the generative model inference engine based on the joint complexity index.

[0011] Furthermore, the specific implementation process of the multi-dimensional feature sensing unit includes:

[0012] First, the physical resolution of the image is obtained by parsing the image sensor data stream. ;

[0013] Subsequently, the image was pixel-level labeled using an instance segmentation model, and the total number of instances was determined by counting the number of independent connected components in the non-background category. ;

[0014] Finally, the image is divided into feature grids using a spatial geometry analyzer, the pixel percentage of instance masks within each grid is calculated, and the instance density is determined through spatial distribution feature statistics. .

[0015] Furthermore, the formula for calculating the instance density is expressed as follows:

[0016] ,

[0017] in, Indicates an active grid. To calculate the total number of active grid cells, For pixel area, For the first Instance mask, For the input image, and These represent the image width and height, respectively.

[0018] Furthermore, the specific implementation process of the semantic analysis and comparison unit includes:

[0019] The image and editing instructions are processed using a pre-trained visual-text encoder to obtain image feature vectors and instruction feature vectors;

[0020] The cosine distance between the image feature vector and the instruction feature vector is obtained through the vector space calculation module and defined as the semantic transformation span. ;

[0021] The semantic complexity is calculated by parsing the instructions using a natural language processor. .

[0022] Furthermore, the formula for calculating the semantic transformation span is expressed as follows:

[0023] ,

[0024] in, Represents absolute value. Represents the image feature vector. Represents the instruction feature vector;

[0025] The formula for calculating semantic complexity is as follows:

[0026] ,

[0027] in, This represents the logarithmic operation with base 2. Indicates the depth of the instruction syntax tree.

[0028] Furthermore, the process by which the complexity arbitration unit generates the joint complexity index includes:

[0029] The system receives the physical resolution, total number of instances, and instance density extracted by the multi-dimensional feature sensing unit, as well as the semantic transformation span and semantic complexity extracted by the semantic analysis and comparison unit.

[0030] The received parameters are normalized and then input into the nonlinear coupling function to generate a joint complexity index. ;

[0031] The joint complexity index is mapped to the control signal domain by using the Sigmoid activation function, and then used as the criterion for dynamic routing controllers.

[0032] Furthermore, the expression for the nonlinear coupling function is:

[0033] ,

[0034] in, Indicates exponentiation. , , , and These are the weighting coefficients for physical resolution, total number of instances, instance density, semantic complexity, and semantic transformation span, respectively.

[0035] Furthermore, the dynamic routing controller performs path selection within a preset decision matrix based on the value range of the joint complexity index.

[0036] Furthermore, the inference parameter adaptive mapper dynamically outputs a guidance strength rescale signal based on the joint complexity index; its correction logic is as follows: when the joint complexity index increases, the global guidance strength is automatically increased to ensure instruction compliance, and the number of sampling iteration steps is increased simultaneously to ensure the generation accuracy of high-density regions.

[0037] Furthermore, the dynamic mounting node library includes super-resolution nodes, redraw nodes, and LoRA loading nodes.

[0038] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0039] 1. Through a pioneering complexity index coupling mechanism, the system transforms abstract image physical constraints (such as instance density and instance count) and intuitive semantic transformation spans into standardized control signals. When processing high-density objects or tasks requiring significant modifications, the system automatically increases the guiding weight to ensure that the generation logic can effectively overcome the inertia of the original image texture and accurately align complex instructions. This effectively solves the technical pain points of traditional solutions in complex tasks, such as instruction failure and logical inconsistency.

[0040] 2. This system departs from the rigid approach of traditional tools that use fixed sampling steps, instead employing a complexity arbitration unit to quantitatively predict the "generation difficulty" of a task. For simple local fine-tuning tasks, the system automatically reduces the number of sampling iterations, achieving adaptive image editing parameters with relatively low computational cost. For highly complex scenarios, the system automatically increases the sampling depth to compensate for detail accuracy, thus ensuring high-quality output while significantly reducing the system's average response latency and resource consumption.

[0041] 3. This invention transforms the parameter tuning process, which originally required users to possess deep domain knowledge (such as repeatedly trying the best combination of guide values ​​and steps), into a closed-loop automated process driven by sensing devices. Users do not need to pay attention to the underlying inference configuration; the system can automatically match the optimal parameter scheme based on the multidimensional features of the input image, achieving an intelligent "input-and-get" experience. This allows non-professional users to obtain professional-level generation results in complex editing scenarios. Attached Figure Description

[0042] Figure 1 This is a system architecture diagram of the present invention;

[0043] Figure 2 This is the node graph of the present invention running in ComfyUI;

[0044] Figure 3 This is a workflow example in ComfyUI;

[0045] Figure 4 A comparison chart showing the different ClipScores of 4 test cases under two different conditions;

[0046] Figure 5 The graphs show the comparison of workflow time for four test cases under two different scenarios.

[0047] Figure 6 This is a diagram illustrating the dataset, corresponding prompts, and the generated results. Detailed Implementation

[0048] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments, and the advantages of the present invention above and in other aspects will become clearer.

[0049] This invention provides an adaptive image editing control system based on multi-dimensional feature perception, the system architecture of which is as follows: Figure 1 As shown, it specifically includes the following:

[0050] Front-end interaction layer: Responsible for receiving user-uploaded images to be edited and natural language editing instructions, and visually displaying them in a workflow node-based format. Figure 2 This is the node graph of the present invention running in ComfyUI, which is the external display front end of the adaptive image editing control system proposed in this invention. It runs on a node-based open-source generative AI application, ComfyUI, and achieves highly customizable content generation by combining models and operations through a visual node workflow.

[0051] Figure 2 The `clip` node is the entry point for loading the clip model, `seg_model` is the entry point for the segmentation model, `image` is the entry point for the image, `data` is the entry point for the prompt words, `path_A`, `path_B`, and `path_C` are the path presets, `final_steps` is the final number of steps to calculate, `select` is the selected path, and `cfg` is the global guidance strength.

[0052] The logic control layer, acting as the core arbitration mechanism of the system, is responsible for the extraction and processing of all features. Internally, it integrates a multi-dimensional feature sensing unit, a semantic analysis and comparison unit, a complexity arbitration unit, and a dynamic routing controller.

[0053] (1) The multidimensional feature sensing unit integrates image preprocessing operators and lightweight instance segmentation models, such as the YOLO series; its specific image preprocessing implementation process includes:

[0054] First, image data streams from different sources are parsed into standard pixel matrices (such as RGB format) to obtain the physical resolution of the original image. ;

[0055] Subsequently, the image was pixel-level labeled using an instance segmentation model, and the total number of instances was determined by counting the number of independent connected components in the non-background category. ;

[0056] Finally, using a spatial geometry analyzer, the image is divided into... The feature grid is used to calculate the pixel proportion of the instance mask within each grid, and the instance density is obtained through spatial distribution feature statistics. .

[0057] The instance mask in this embodiment is extracted using an instance segmentation model, such as YOLOv8, which originates from Jocher, G., Chaurasia, A., & Qiu, J. "Ultralytics YOLOv8." GitHub, version 8.0.0, 2023.

[0058] The formula for calculating the instance density is as follows:

[0059] ,

[0060] in, The active grid is defined as a feature grid cell whose percentage of pixels covered by the instance mask exceeds a preset threshold (e.g., 5%). To calculate the total number of active grid cells, For pixel area, For the first Instance mask, For the input image, and These represent the image width and height, respectively.

[0061] (2) The semantic analysis and comparison unit adopts multimodal representation alignment technology, and extracts the global semantic vector of the image through, for example, the Qwen open source Qwen2.5-VL-7B-Instruct visual coding model, which is derived from Bai, Jinze et al. Qwen-VL: A VersatileVision-Language Model for Understanding, Localization, Text Reading, and Beyond. arXiv preprint arXiv:2308.12966, 2023. It extracts the feature vector of editing instructions through its text encoder. Calculate the cosine distance between the two, which is the semantic transformation span. , is represented as:

[0062] ,

[0063] in, Representing absolute values, semantic transformation span is used to quantify the intent span.

[0064] Simultaneously, through a dependency parser, the complexity of the predicate verb and object of the instruction is parsed and mapped to semantic complexity. ;

[0065] The dependency parser used in this embodiment is provided by the syntactic analysis module of the Language Technology Platform (LTP) developed by the Social Computing and Information Retrieval Research Center of Harbin Institute of Technology.

[0066] The formula for calculating semantic complexity is as follows:

[0067] ,

[0068] in, This represents the logarithmic operation with base 2. Indicates the depth of the instruction syntax tree.

[0069] In this embodiment, the instruction is: "Replace the sofa in the picture with the green sofa".

[0070] (3) The process by which the complexity arbitration unit generates a joint complexity index based on the physical resolution, total number of instances, and instance density extracted by the multi-dimensional feature sensing unit, as well as the semantic transformation span and semantic complexity extracted by the semantic analysis and comparison unit, is as follows:

[0071] The system receives the physical resolution, total number of instances, and instance density extracted by the multi-dimensional feature sensing unit, as well as the semantic transformation span and semantic complexity extracted by the semantic analysis and comparison unit.

[0072] The received parameters are normalized and then input into the nonlinear coupling function to generate a joint complexity index. ;

[0073] The joint complexity index is mapped to the control signal domain by using the Sigmoid activation function, and then used as the criterion for dynamic routing controllers.

[0074] The normalization expression is: , For input features, and The minimum and maximum values ​​of the feature are... For normalized characteristics; the expression for the nonlinear coupling function is: ,

[0075] in, Indicates exponentiation. , , , and These are the weighting coefficients for physical resolution, total number of instances, instance density, semantic complexity, and semantic transformation span, respectively.

[0076] (4) The dynamic routing controller performs dynamic routing selection of the workflow based on the joint complexity index; such as Figure 2 Choose the appropriate workflow from pathA and pathB.

[0077] Inference Execution Layer: Its core is a generative model inference engine (such as Nunchaku-SDXL) with parameter intervention interfaces. This layer contains a dynamically mountable node library (such as super-resolution nodes, redraw nodes, LoRA loading nodes, etc.) and an inference parameter mapper, which directly receives signals from the logic control layer and performs parameter calculations.

[0078] The dynamic routing controller receives the joint complexity index from the complexity arbitration unit. The controller has a preset multi-level decision matrix, based on... The real-time value range enables dynamic allocation of workflow paths, such as... Figure 2 The specific mechanisms for path_A, path_B, and path_C are as follows:

[0079] The dynamically mounted node library acts as a "plugin pool" for the dynamic routing controller, providing it with the computing power components required for heterogeneous tasks. This node library is built by the user. The dynamic routing controller, based on... The dynamic routing controller determines which nodes are activated and orchestrated into the current workflow sequence based on the judgment results. Specific workflow nodes are added by the user. The final output of the dynamic routing controller is a topology-orchestrated task pipeline, which, along with components from the node library, is input to the inference engine to determine the workflow's path.

[0080] Decision matrix mechanism: It is divided into three threshold ranges: low, medium, and high. If... In the low range, the route points to the "basic editing path," and only lightweight nodes are mounted; if In the high-complexity range, the route points to the "enhanced processing path." At this point, the controller will call a Super-Resolution node or load a node from the dynamically mounted node library to compensate for the loss of detail in high-complexity scenarios. The following example uses threshold partitioning:

[0081] low complexity range : Perform basic editing path, retaining only the core diffusion sampling nodes;

[0082] Medium complexity range : Perform standard path redrawing and add local attention enhancement nodes;

[0083] High complexity interval : Execute the enhanced processing path and force the mounting of super-resolution nodes and deep LoRA loading nodes.

[0084] (5) The inference parameter adaptive mapper dynamically outputs a guidance strength rescale signal based on the joint complexity index, which is asynchronously controlled by the complexity arbitration unit in the same source as the dynamic routing controller. Its correction logic is as follows: when the joint complexity index increases, the global guidance strength is automatically increased to ensure instruction compliance, and the number of sampling iteration steps is increased synchronously to ensure the generation accuracy of high-density regions, and the output is... Figure 2 The `final_steps` parameter has the following two parameters that adjust the number of steps:

[0085] Guidance Rescale: The mapper uses a linear or piecewise function to rescale the map. Mapped to the Guidance Rescale signal (typically with a value range of...) The formula is as follows:

[0086] ,

[0087] in, To guide the intensity rescaling of the output signal. This is the default rescaling signal for the model. This parameter, representing sensitivity gain, is adjustable by the user. This signal is used to dynamically correct the global guidance strength (Classifier-Free Guidance Scale) to prevent overexposure at high sensitivity levels. In scenarios with large semantic spans, the generated content deviates from the instructions. The pruning function is the dot product of semantics and the joint complexity index. Limited to the range of 0-1.

[0088] Sampling step number adaptation: mapper based on Calculate the number of incremental sampling steps The logic is: when (Density) or When the semantic complexity is high, As the number of sampling steps increases, the mapper outputs a higher sampling step signal, ensuring that the inference engine has sufficient iterations to converge complex spatial features, as shown in the following formula:

[0089]

[0090]

[0091] in, It is a rounding function. Step adjustment factor Nonlinear correction exponent (usually) (used to rapidly increase the number of steps under extremely high complexity). and , which are the user's estimated minimum and maximum Step values, respectively.

[0092] Ultimately, the workflow will select the appropriate path and K-sampler parameters. While keeping the diffusion model unchanged, the image generation time and generation quality are balanced by automatically fine-tuning cfg (global guidance strength) and step number (iteration step number).

[0093] Additionally, this embodiment also provides a workflow example from ComfyUI, which can be found in the following reference. Figure 3The simplified process is as follows: First, the original image and the "change to the green sofa" instruction are received through the image input and CLIP text encoding nodes; then, the segmentation model loader and CLIP loader provide the system with pixel-level feature perception and semantic understanding capabilities; the core node, the adaptive sampling path router (i.e., the front-end window of this system), comprehensively considers image complexity and semantic span, calculates and outputs the optimal number of sampling steps (final_steps) and path selection signal in real time; the UNet / LoRA loader and VAE encoder prepare the generation model inference environment, and the K sampler performs precise denoising processing according to adaptive parameters; finally, the VAE decoder outputs an edited result that balances instruction compliance and generation quality, as shown in the image. Figure 3 As shown in the bottom right corner.

[0094] Compared with conventional workflows, this invention has the following advantages:

[0095] Execution path optimization: For simple tasks, the system can avoid excessive GPU memory usage and model loading time, such as by appropriately reducing the number of iteration steps and switching generation paths. Figure 5 Comparison of generation time.

[0096] Automated robustness: reduces repeated manual debugging by users Figure 3 In the workflow, the cfg (global guiding strength) and number of steps (iteration steps) of the K sampler are determined within a specified range according to... Automatic fine-tuning still ensures the quality of the generated product, such as Figure 4 The CLIPS score (a metric for evaluating the relevance of content generated from an image) is similar to the others.

[0097] The following table shows the results of four images selected as queue tasks, comparing those without this solution node (baseline) and those with this solution node. The purpose is to verify its ability to automatically fine-tune to reduce total time when executing complex and simple combined tasks. Comparisons of ClipScore and workflow time for the four test cases in the two scenarios can be found in the reference table. Figure 4 and Figure 5 The performance results and generation results can be referred to Table 1 and Table 2 respectively. Figure 6 As can be seen, with the quality remaining basically unchanged, the time consumption has decreased compared to the solution without this node, while the generation quality of the solution with this node has improved.

[0098] Table 1. Comparison of performance results between baseline and node

[0099]

[0100] This invention provides an adaptive image editing control system based on multi-dimensional feature perception. Many methods and approaches exist for implementing this technical solution; the above description is merely a preferred embodiment. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of this invention, and these improvements and modifications should also be considered within the scope of protection of this invention. All components not explicitly stated in this embodiment can be implemented using existing technologies.

Claims

1. An adaptive image editing control system based on multi-dimensional feature perception, characterized in that, It includes a front-end interaction layer, a logic control layer, and an inference and execution layer; The front-end interaction layer is used to receive user-uploaded images to be edited, natural language editing instructions, and visual displays; The logic control layer is responsible for the extraction and processing of all features, including a multi-dimensional feature sensing unit, a semantic analysis and comparison unit, a complexity arbitration unit, and a dynamic routing controller. The complexity arbitration unit generates a joint complexity index based on the physical resolution, total number of instances, and instance density extracted by the multi-dimensional feature sensing unit, as well as the semantic transformation span and semantic complexity extracted by the semantic analysis and comparison unit; the dynamic routing controller performs dynamic routing selection of the workflow based on the joint complexity index. The inference execution layer is used to receive signals from the logic control layer and perform parameter calculations, including a generative model inference engine, a dynamically mounted node library, and an inference parameter adaptive mapper. The inference parameter adaptive mapper dynamically adjusts the global guidance strength and sampling iteration steps of the generative model inference engine based on the joint complexity index.

2. The adaptive image editing control system based on multi-dimensional feature perception according to claim 1, characterized in that, The specific implementation process of the multi-dimensional feature sensing unit includes: First, the physical resolution of the image is obtained by parsing the image sensor data stream. ; Subsequently, the image was pixel-level labeled using an instance segmentation model, and the total number of instances was determined by counting the number of independent connected components in the non-background category. ; Finally, the image is divided into feature grids using a spatial geometry analyzer, the pixel percentage of instance masks within each grid is calculated, and the instance density is determined using a spatial distribution entropy algorithm. .

3. The adaptive image editing control system based on multi-dimensional feature perception according to claim 2, characterized in that, The formula for calculating the instance density is as follows: ， in, For active grids, To calculate the total number of active grid cells, For pixel area, For the first Instance mask, For the input image, and These represent the image width and height, respectively.

4. The adaptive image editing control system based on multi-dimensional feature perception according to claim 1, characterized in that, The specific implementation process of the semantic analysis and comparison unit includes: The image and editing instructions are processed using a pre-trained visual-text encoder to obtain image feature vectors and instruction feature vectors; The cosine distance between the image feature vector and the instruction feature vector is obtained through the vector space calculation module and defined as the semantic transformation span. ; The semantic complexity is calculated by parsing the instructions using a natural language processor. .

5. The adaptive image editing control system based on multi-dimensional feature perception according to claim 4, characterized in that, The formula for calculating the semantic transformation span is expressed as follows: ， in, Represents absolute value. Represents the image feature vector. Represents the instruction feature vector; The formula for calculating semantic complexity is as follows: ， in, This represents the logarithmic operation with base 2. Indicates the depth of the instruction syntax tree.

6. The adaptive image editing control system based on multi-dimensional feature perception according to claim 1, characterized in that, The process by which the complexity arbitration unit generates the joint complexity index includes: The system receives the physical resolution, total number of instances, and instance density extracted by the multi-dimensional feature sensing unit, as well as the semantic transformation span and semantic complexity extracted by the semantic analysis and comparison unit. The received parameters are normalized and then input into the nonlinear coupling function to generate a joint complexity index. ; The joint complexity index is mapped to the control signal domain by using the Sigmoid activation function, and then used as the criterion for dynamic routing controllers.

7. The adaptive image editing control system based on multi-dimensional feature perception according to claim 6, characterized in that, The expression for the nonlinear coupling function is: ， in, Indicates exponentiation. , , , and These are the weighting coefficients for physical resolution, total number of instances, instance density, semantic complexity, and semantic transformation span, respectively.

8. The adaptive image editing control system based on multi-dimensional feature perception according to claim 1, characterized in that, The dynamic routing controller performs path selection within a preset decision matrix based on the value range of the joint complexity index.

9. The adaptive image editing control system based on multi-dimensional feature perception according to claim 1, characterized in that, The inference parameter adaptive mapper dynamically outputs a guidance strength rescaling signal based on the joint complexity index. Its correction logic is as follows: when the joint complexity index increases, the global guidance strength is automatically increased to ensure instruction compliance, and the number of sampling iteration steps is increased simultaneously to ensure the generation accuracy of high-density regions.

10. The adaptive image editing control system based on multi-dimensional feature perception according to claim 1, characterized in that, The dynamic mounting node library includes super-resolution nodes, redraw nodes, and LoRA loading nodes.

Images