Image enhancement method using visual language model (VLM) and electronic device and recording medium utilizing the same

Abstract

An image enhancement method utilizing a visual-language model (VLM), an electronic device utilizing the same, and a recording medium are provided. An image enhancement method according to one embodiment receives an input image, analyzes the input image using a multimodal artificial intelligence model including a visual-language model (VLM) to generate defect information regarding defects in the input image, determines execution conditions for enhancement processing on the input image based on the defect information, performs image processing on the input image according to the execution conditions to generate an enhanced image, performs verification regarding whether the enhanced image satisfies a predetermined standard, and controls the output of the enhanced image or the re-execution of image processing according to the verification result.

Description

Technology Field

[0001] The present invention relates to an image enhancement method utilizing a visual-language model (VLM), and to an electronic device and a recording medium utilizing the same. Background Technology

[0002] Due to digital transformation and the proliferation of mobile environments, images are utilized as key information that determines the first impression of products and services across various channels, including menus, kiosks, e-commerce, delivery apps, and social media. However, shooting environments involve numerous variables—such as lighting, backlighting, color temperature, camera shake, focus misalignment, and loss during compression and transmission—leading to frequent quality degradation issues like noise, blur, color distortion, reduced contrast, smudges, shadows, and the inclusion of unnecessary background objects. Such quality degradation can negatively impact user perception and purchasing decisions, and there is a growing need for management in terms of brand consistency and screen readability.

[0003] Traditionally, methods to improve image quality have been widely used, such as photo editing experts performing post-processing or utilizing automatic correction functions in applications (adjusting brightness, contrast, saturation, and white balance, sharpening, noise reduction, etc.). Additionally, in the field of computer vision, techniques such as filter-based correction, feature point / edge-based sharpening, histogram equalization, and color correction have been proposed to mitigate defects. Recently, the scope of application has also expanded to include noise reduction, shake recovery, super-resolution, color correction, and background separation using deep learning. However, as shooting conditions and subjects become more diverse, it becomes difficult to reliably process all defects using a single correction rule or a single model. Furthermore, excessive correction can lead to issues such as increased visual artificiality or the degradation of important content in the original image. Consequently, in environments involving mass image processing, there is a continuously growing need for image quality management technologies capable of handling various types of defects while reducing limitations in terms of cost, time, and consistency. The problem to be solved

[0004] Although images are utilized across various channels as critical information determining the quality of products and services, in reality, heterogeneous defects such as noise, blur, uneven lighting, color distortion, reflections and smudges, and the inclusion of unnecessary background objects frequently occur due to differences in shooting environments and operator skill levels. These defects undermine the communicative power and consistency of images and can negatively impact customer trust and conversion rates. Nevertheless, conventional automatic correction is effective only for specific defects or focuses on batch correction, resulting in low consistency in processing tailored to specific defect types; meanwhile, manual correction is unsuitable for mass processing due to the high cost and time burden.

[0005] Accordingly, the problem that the present invention aims to solve is to provide a technical means capable of automatically diagnosing defects present in an input image, determining the type of defect and the scope requiring improvement, and performing improvement on the entire image or selective improvement on a specific area.

[0006] In addition, an additional task is to structure and provide processing results and related information to enable the quality of a large volume of images to be maintained and managed according to certain standards from a service operation perspective, while suppressing issues such as excessive distortion of the core meaning of the original images (form, composition, and identity of the main subject) or the generation of artificial outputs during the improvement process.

[0007] Ultimately, the objective of the present invention is to provide an image enhancement method, an electronic device, and a recording medium that simultaneously satisfy automation, consistency, and scalability for images containing various defects.

[0008] The problems of the present invention are not limited to those mentioned above, and other unmentioned technical problems will be clearly understood by a person skilled in the art from the description below. means of solving the problem

[0009] An image enhancement method according to an embodiment of the present invention for solving the above problem receives an input image, analyzes the input image using a multimodal artificial intelligence model including a visual-language model (VLM) to generate defect information regarding defects in the input image, determines execution conditions for enhancement processing on the input image based on the defect information, performs image processing on the input image according to the execution conditions to generate an enhanced image, performs verification regarding whether the enhanced image satisfies a predetermined standard, and controls outputting the enhanced image or re-performing the image processing according to the verification result.

[0010] The above defect information may include at least one of parameters regarding the type of defect, the severity of the defect, the location of the defect, the extent of the defect, and the correction strength.

[0011] In the step of receiving the input image, text metadata corresponding to the input image may be further received along with the input image, and the multimodal artificial intelligence model may further utilize the text metadata to generate the defect information.

[0012] The step of determining the above execution conditions may include a step of selecting at least one of global improvement processing and local improvement processing based on the above defect information.

[0013] The above execution condition may include area information or mask information indicating an area to be edited in the input image, and the image processing may be selectively performed on a part of the input image based on the area information or mask information.

[0014] The above image processing may include at least one of a global restoration track and a local inpainting track, and the global restoration track and the local inpainting track may be performed sequentially.

[0015] The above global restoration track can restore the input image using an image-to-image generation model, and can maintain the structural consistency of the input image by using structural information extracted from the input image as a control input for the generation process.

[0016] The above-mentioned local inpainting track can generate a mask using a segmentation model based on location information included in the defect information, and perform inpainting on the area designated by the mask.

[0017] The above inpainting can be performed using negative prompts or prohibition rules to suppress the creation of specific objects or specific types.

[0018] The above verification may include at least one of (i) semantic preservation verification based on embedding similarity between the input image and the improved image, and (ii) text consistency verification that determines whether the response to a text-based query and text metadata match by re-inputting the improved image into the multimodal artificial intelligence model.

[0019] If the above verification result is determined not to satisfy the above predetermined criteria, the above re-execution may be performed by changing at least one of the correction strength parameter, the strength of the structural control, the prompt, the mask information, or the execution order of the track.

[0020] An image enhancement method according to an embodiment of the present invention for solving the above problem comprises receiving an input image, searching for at least one defect region in the input image to calculate defect information and a defect degree corresponding to the defect region, setting a restoration region to include the defect region, determining a spatially variable restoration index within the restoration region such that (i) the restoration index increases as the region with a greater degree of defect based on the defect degree, and (ii) the restoration index decreases as it approaches the boundary of the restoration region based on the distance from the boundary of the restoration region, and performing restoration processing on the input image while controlling the intensity and / or amount of computation of image processing performed in the restoration region based on the spatially variable restoration index to generate a corrected image, and generating an output image by blending the corrected image with the input image or a reference image generated from the input image using weights based on the spatially variable restoration index to reduce the unnaturalness of the boundary between the corrected image and the input image.

[0021] The above defect map is a defect map representing the degree of defect corresponding to the x-coordinate of the input image. Calculated as, and the above spatial variable restoration index is the distance to the boundary of the above-mentioned restoration area Taper function based on and the above defect map By combining, it is determined by the following mathematical formula 1, and <Mathematical Formula 1> , the above taper function is defined by the following mathematical formula 2, and <Mathematical Formula 2> , in the above mathematical formula 2, w is the boundary buffer width, and It could be.

[0022] An image enhancement method according to an embodiment of the present invention for solving the above problem comprises receiving an input image, searching for a plurality of defect regions in the input image to calculate defect information and a defect degree for the plurality of defect regions, clustering the plurality of defect regions based on spatial proximity and / or similarity to generate a plurality of defect clusters, setting a plurality of concentrated restoration regions to each include each defect cluster, setting the remaining region excluding the plurality of concentrated restoration regions as a general restoration region, determining a spatially variable restoration index for each of the plurality of concentrated restoration regions such that (i) the restoration index increases as the region with a greater degree of defect based on the defect degree, and (ii) the restoration index decreases as it approaches the boundary of the concentrated restoration region based on the distance from the boundary of the concentrated restoration region, setting a minimum restoration index smaller than the spatially variable restoration index for the general restoration region, and performing restoration processing on the input image to generate a corrected image while differentially controlling the intensity and / or amount of image processing performed in the plurality of concentrated restoration regions and the general restoration region based on the spatially variable restoration index and the minimum restoration index, and, to reduce unnaturalness at the boundary between the concentrated restoration region and the general restoration region, An output image is generated by blending the correction image and the input image or the reference image generated from the input image using weights based on a spatially variable restoration index.

[0023] The k-th intensive restoration area among the plurality of intensive restoration areas mentioned above is the cluster mask corresponding to the k-th defect cluster The outer veil of It is set to the following Equation 3 by the expansion operation for, and <Equation 3> , expansion radius is the cluster average defect rate Based on, it is determined by the following mathematical formula 4, and <Mathematical Formula 4> , the k-th intensive restoration area above Spatial variable recovery index for The defects and the above intensive restoration area Distance to the boundary of Based on, it is calculated by the following mathematical formula 5, and <Mathematical Formula 5> Total restoration index is determined by the following mathematical formula 6, and <Mathematical Formula 6> , intensity control variable of the above restoration process is <Mathematical Formula 7> It can be set to.

[0024] An electronic device according to an embodiment of the present invention for solving the above problem comprises: a communication interface for performing wired or wireless communication with an external device; a memory for storing one or more commands; and one or more processors for executing the one or more commands stored in the memory. The one or more processors include an input / output function in which they receive an input image to be improved and metadata corresponding to the input image through the communication interface and transmit an improved image and a processing result; an analysis function that analyzes the input image and generates defect information using a multimodal artificial intelligence model including a visual-language model (VLM), and calculates editing instructions and / or parameters based on the defect information; a mask generation function that generates or refines a mask based on region of interest information included in the defect information; a generative editing function that generates an improved image by performing image processing including global correction and / or local correction based on the input image, editing instructions and parameters and the mask; a verification function that determines whether quality criteria and / or semantic preservation criteria are satisfied based on a comparison between the input image and the improved image; a policy control function that controls image processing based on the verification result or adjusts editing instructions and / or parameters and / or masks to trigger reprocessing; and a plurality of improved image candidates under the control of the policy control function. It is configured to perform an image generation function that creates images or generates derived images that conform to service channel specifications.

[0025] A computer-readable program is stored therein that executes a command to communicate with an external device to receive an input image and metadata corresponding to the input image and to transmit an improved image and processing results; a command to analyze the input image using a multimodal artificial intelligence model including a visual-language model (VLM), generate defect information, and calculate editing instructions and / or parameters based on the defect information; a command to generate or refine a mask based on region of interest information included in the defect information; a command to generate an improved image by performing image processing including global correction and / or local correction based on at least one of the input image, the editing instructions, the parameters, and the mask; a command to determine whether quality criteria and / or semantic preservation criteria are satisfied based on a comparison of the input image and the improved image; a command to control accepting the image processing or adjusting the editing instructions and / or the parameters and / or the mask to trigger reprocessing based on the determination result; and a command to generate multiple improved image candidates or generate a derived image conforming to service channel specifications according to the control. Effects of the invention

[0026] According to an embodiment of the present invention, based on the reality that defects in input images occur differently depending on the type, the improvement path and processing scope can be automatically determined according to the defect diagnosis results, thereby reducing the problems of over-correction and under-correction that occurred in batch correction using a single rule. As a result, defects affecting the entire image, such as noise, blur, illumination, and color issues, are processed through global improvement, while defects limited to specific areas, such as foreign objects, local contamination, and partial damage, are processed through selective improvement, thereby improving the consistency of improvement tailored to the characteristics of the defects. Furthermore, in a mass image processing environment, the cost and time of existing workflows that relied on expert manual work can be reduced, and result variations due to operator proficiency can be decreased, thereby increasing the consistency of quality.

[0027] Furthermore, the present invention can be designed to suppress the risk of unnecessary alteration of the core meaning of the original image during the improvement process, which is advantageous for maintaining the identity of important subjects and ensuring readability on service screens.

[0028] In addition, since the processing steps and results can be provided as structured information, it becomes possible to conduct operations that allow for tracking and auditing which defects were identified by which criteria and which improvements were performed.

[0029] As a result, the level of automation for image quality management is enhanced across various applications, such as kiosk menus, e-commerce product images, and promotional materials. Furthermore, even if new channels or defect types are added, the system can be extended within the same framework, contributing to the improvement of overall service competitiveness.

[0030] The effects according to the embodiments are not limited to those exemplified above, and a wider variety of effects are included in this specification. Brief explanation of the drawing

[0031] FIG. 1 is a block diagram schematically showing the overall configuration of an image enhancement system utilizing a visual-language model (VLM) according to one embodiment of the present invention. FIG. 2 is a block diagram illustrating functional components (e.g., input / output module, VLM engine, mask generation module, generative editing module, verification module, image generation module, policy engine) that may be included in a server according to one embodiment of the present invention. FIG. 3 is a flowchart showing the steps of an image enhancement method according to one embodiment of the present invention. FIG. 4 is a flowchart including a track-based (global restoration track and local inpainting track) image enhancement method and a semantic preservation verification and re-execution loop according to another embodiment of the present invention. FIG. 5 is a diagram illustrating a partial restoration method that sets a restoration area to include a plurality of defective areas according to an embodiment of the present invention, and variably controls the restoration strength according to a spatially variable restoration index within the restoration area. FIG. 6 is a diagram illustrating a total restoration method that, according to an embodiment of the present invention, clusters a plurality of defect regions to set a plurality of concentrated restoration regions and general restoration regions, and differentially controls the restoration intensity and / or computational amount for each region. FIG. 7 is a schematic diagram illustrating the hardware configuration of a server according to one embodiment of the present invention. Specific details for implementing the invention

[0032] The advantages and features of the present invention and the methods for achieving them will become clear by referring to the embodiments described below in detail together with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below but may be implemented in various different forms. These embodiments are provided merely to ensure that the disclosure of the present invention is complete and to fully inform those skilled in the art of the scope of the invention, and the present invention is defined only by the scope of the claims.

[0033] Although terms such as "first," "second," etc., are used to describe various components, it goes without saying that these components are not limited by these terms. These terms are used merely to distinguish one component from another. Therefore, it goes without saying that the first component mentioned below may be the second component within the technical scope of the present invention. A singular expression includes a plural expression unless the context clearly indicates otherwise.

[0034] Hereinafter, embodiments of the present invention will be described in detail with reference to the attached drawings. Identical or similar reference numerals are used for identical components in the drawings.

[0035] FIG. 1 is a block diagram schematically showing the overall configuration of an image enhancement system utilizing a visual-language model (VLM) according to one embodiment of the present invention.

[0036] Referring to FIG. 1, an image enhancement system utilizing a visual-language model (VLM) (hereinafter referred to as the "system") may include a server (10), an administrator terminal (20), and a service terminal (30), and these components are connected to enable mutual data transmission and reception through a wired or wireless communication network.

[0037] The server (10) is the primary computational entity that performs image enhancement in the present invention. The server (10) may include software modules and computational resources for performing analysis and enhancement processing on input images, and in particular, it may execute or call a visual-language model (VLM) to interpret the state of an image and generate information necessary for subsequent processing. The server (10) may perform functions such as storage, preprocessing, processing history management, result generation, and result distribution for target images received from the outside, and depending on the embodiment, it may be configured to include a database or storage unit to store and manage original images, corrected images, verification results, application history, setting parameters, etc. Additionally, the server (10) may include an API, an authentication module, and an access control module for linking with an administrator terminal (20) and / or a service terminal (30), thereby providing security and traceability in an operating environment.

[0038] The administrator terminal (20) is a terminal that provides a user interface for an operator or administrator to request image improvement from the server (10), receive the results generated from the server (10), and review, approve, and apply them.

[0039] The administrator terminal (20) can be implemented in various forms, such as a smartphone, tablet, PC, or POS terminal, and can request improvement processing by selecting a captured image or a previously stored image and transmitting it to the server (10). Additionally, the administrator terminal (20) can receive comparison verification results along with the corrected image from the server (10), and can display, for example, the quality change of the improved version compared to the original, the changed area, whether the verification passed or failed, and whether reprocessing is required. Furthermore, the administrator terminal (20) can input image application settings to the service terminal (30), such as “which image to apply to which menu / product,” “when to apply,” and “which of the multiple candidate images to adopt,” and transmit them to the server (10). Accordingly, the operator can set various operational policies, such as batch application, phased application, or A / B test-based application, for multiple stores or multiple devices.

[0040] The service terminal (30) is a terminal that actually displays an improved image and is a contact device to which the service to which the present invention is applied is provided. The service terminal (30) can be implemented, for example, as a kiosk, a table order device, digital signage, a store display, an order and payment terminal, etc.

[0041] The service terminal (30) may be configured to receive an improved image from the server (10) and display it on a screen, or to refer to an improved image stored in the server (10). Depending on the embodiment, the server (10) may operate by “pushing” the image to the service terminal (30) or by the service terminal (30) periodically “pulling” the image from the server (10). Additionally, the service terminal (30) may be configured to send operational logs, such as the version of the applied image, the time of application, and whether the display was successful, back to the server (10) so that the server (10) can manage the application status in an integrated manner.

[0042] The operator selects an image improvement target (e.g., menu image) through the administrator terminal (20) and transmits it to the server (10), and the server (10) generates the result of improving the image (corrected image) and the comparison verification result and provides them to the administrator terminal (20). The operator confirms the received result and finalizes the application settings, and the server (10) distributes the improved image to the service terminal (30) or provides it for the service terminal (30) to refer to according to the finalized settings. This structure allows the image quality management task to be processed in an integrated manner by the central server (10), while enabling the field operator to review the results through the administrator terminal (20) and control the application to the service terminal (30), thereby providing a system configuration that enables consistent image operation even in a multi-device / multi-store environment.

[0043] Additionally, some functions of the server (10), the administrator terminal (20), and the service terminal (30) may be integrated into a single device or, conversely, distributed by function depending on the embodiment. According to the embodiment, the functions of the administrator terminal (20) may not be separated into a separate terminal but may be implemented integrally in the service terminal (30). For example, the service terminal (30) may be configured to include a UI for administrators so that an operator can directly input an image improvement request on the service terminal (30), check the corrected image and comparison verification results received from the server (10), and determine the application settings (application timing, menu / product mapping, candidate selection, etc.). In this case, the administrator terminal (20) may exist only as a logical function block and may be implemented physically as an integrated form in the service terminal (30).

[0044] In addition, according to the embodiment, all or part of the functions of the server (10) may be integrally implemented in the service terminal (30). In this case, the service terminal (30) may be referred to as an electronic device while including the functions of the server (10) itself. That is, the electronic device may be a computing device such as the server (10) or the service terminal (30).

[0045] A service terminal (30) may be configured to include computational resources (GPU, NPU, etc.) capable of locally executing at least some of the following: a VLM engine, a generative editing module, a verification module, etc., so as to perform image analysis and correction processing without an external server and immediately reflect the results on the screen of the service terminal (30). In this case, the server (10) may be omitted as a physical device or reduced to an auxiliary server that performs only management functions such as updates, policy distribution, and log collection, and the configuration of FIG. 1 may be interpreted on the premise that it is a conceptual configuration including such integrated and distributed implementation.

[0046] FIG. 2 is a diagram illustrating functional components that may be included in a server according to an embodiment of the present invention in a block form.

[0047] Referring to FIG. 2, the server (10) includes an input / output module (210), a VLM engine (220), a mask generation module (230), a generative editing module (240), a verification module (250), an image generation module (260), and a policy engine (270) to perform a series of processes from receiving an input image to analysis, correction, verification, and providing results. Each component may be implemented in software and may be implemented to operate integrally within the server by combining GPU / accelerator-based computing resources and storage, or distributed in the form of microservices and called from each other.

[0048] The input / output module (210) serves as an external interface of the server (10) and performs the role of receiving images to be improved and related metadata (e.g., menu name, category, shooting conditions, required quality standards, application channels) from an administrator terminal or a service terminal, and transmitting processing results. The input / output module (210) transmits original images and processing parameters to an internal pipeline on a request basis, and provides the resulting corrected images, candidate image sets, verification reports, change area information, applicability, etc., to the outside. Additionally, depending on the embodiment, it may be configured to perform identifier assignment for request tracking, version control, caching, and log collection. The VLM engine (220) is an analysis module that performs semantic understanding of the input image. The VLM engine (220) analyzes the input image and, depending on the embodiment, can generate defect information regarding defects in the input image by using text metadata corresponding to the input image together. The defect information may include at least one of parameters regarding the type of defect, the severity of the defect, the location of the defect, the range of the defect, and the correction strength. Additionally, the VLM engine (220) may generate editing instructions and / or parameters to be used for image processing based on the defect information. The editing instructions may include at least one selection of global improvement processing or local improvement processing, text prompts, negative prompts, defect descriptions, and constraints for structure preservation, and the parameters may include at least one of region of interest coordinates, correction strength, guidance strength, and sampling-related control values. The output of the VLM engine (220) may be generated as a strategy output in a structured format that can be interpreted by a subsequent module.

[0049] As an example, the strategy output may include a defect list (type, severity, rationale), region of interest information, an edit instruction, and a correction strength parameter (β). The policy engine (270) and the generative edit module (240) may be controlled to select at least one of a global correction path and a local correction path based on the strategy output.

[0050] When local correction is required, the mask generation module (230) generates or refines a mask representing the area to be edited based on the region of interest information included in the defect information. The mask generation module (230) receives at least one of the bounding box, point, defect location or polygon candidate, or text-based instruction provided by the VLM engine (220) as input, and can generate a pixel-unit mask using a segmentation model or traditional image processing technique. Additionally, the mask generation module (230) can generate a final mask (Mi) by performing mask dilation / erosion, boundary smoothing, multi-mask candidate generation, and scoring.

[0051] The generative editing module (240) is a module that performs image correction according to the strategy output of the VLM engine (220) and the control of the policy engine (270). The generative editing module (240) can generate an improved image by performing image processing including global correction and / or local correction based on at least one of an input image, editing instructions, parameters, and a mask. When a global correction path is selected, the generative editing module (240) operates to improve the overall quality while maintaining the composition and meaning of the input image, and when a local correction path is selected, it can operate to selectively perform inpainting or partial restoration on the area designated by the final mask (Mi).

[0052] The verification module (250) determines whether the result of generative editing or image generation satisfies quality standards and semantic preservation conditions. The verification module (250) can evaluate low-level quality indicators of the corrected version relative to the original (e.g., sharpness, noise level, contrast, color distribution stability, etc.) and simultaneously evaluate whether there is semantic match. Semantic match evaluation can be implemented, for example, by comparing the degree of match after performing VLM-based captioning or attribute extraction on the original image and the corrected image, or by checking whether they are “the same product / the same subject” based on embedding similarity. Additionally, the verification module (250) can determine as a negative case cases where elements not present in the original are inserted (hallucination) or the core subject is deformed due to excessive generation, and can feed back to the policy engine (270) or VLM engine (220) whether reprocessing is necessary and the direction of reprocessing (reduction of correction intensity, strengthening of structural constraints, mask modification, etc.).

[0053] The image generation module (260) is a module that operates when “expressiveness enhancement” or “candidate image generation” is required as an extension of correction. The image generation module (260) can generate multiple style / composition candidates in an image-to-image manner based on an input image, or generate derivative images that meet specifications (resolution, margins, background tone, etc.) suitable for a service channel. Additionally, since visual emphasis points may vary depending on the image type, such as food, products, or people, the image generation module (260) may be configured to generate candidates by reflecting the type determination and emphasis point information provided by the VLM engine (220). However, the present invention is not limited to trademarks or implementations of specific generation models and can be replaced with various generation models or pipelines.

[0054] The policy engine (270) is a control module that orchestrates the entire pipeline and consistently applies processing conditions and safety devices. The policy engine (270) manages global / local path selection rules, acceptable change ranges, number of iterations, thresholds, and candidate selection criteria for each defect type, and determines the final result by combining them with the channel policy of the service terminal (e.g., limiting excessive style changes for kiosks, preserving menu name / price display areas, etc.). Additionally, the policy engine (270) can be configured to trigger a reprocessing loop based on the result of the verification module (250) and to gradually adjust correction strength or constraints to converge in a direction that “increases quality while maintaining meaning.” Consequently, the structure of FIG. 2 can be designed to ensure scalability and operability even in various defects and various service environments by modularizing the flow of image reception → analysis → path determination → mask / editing or creation → verification → output.

[0055] FIG. 3 is a flowchart illustrating an image enhancement method according to an embodiment of the present invention.

[0056] Referring to FIG. 3, an image enhancement method according to one embodiment may include a step pipeline of “User / System → Analysis (VLM) → Mask Generation / Refinement → Generative Editing → Verification”.

[0057] This embodiment emphasizes that the input image is not treated merely as a target for simple image quality correction, but rather the VLM semantically analyzes defects to generate a “defect list (type, severity, basis for judgment),” refines a mask based on defect location candidates, performs generative editing with an editing request including a mask (Mi), editing instructions, and correction strength parameters (β), and finally determines whether to accept or retry through comparative verification (before, after).

[0058] In step S101, the administrator terminal (20) or service terminal (30) inputs the image to be improved (input image) into the input / output module (210) of the server (10). For example, the input image may be a menu or product image to be displayed on a kiosk / table order, and may contain various defects such as blur, noise, uneven illumination, color distortion, reflected light, and background obstructions that occurred during the shooting, compression, and transmission processes. The administrator terminal (20) or service terminal (30) may additionally transmit policy information along with the image, such as the application channel, target specifications, and prohibited modification targets (e.g., logo, price tag, text area).

[0059] In this embodiment, the input is not limited to images but may consist of multimodal inputs. That is, the user / system may additionally provide text metadata along with the food image, and the text metadata may include, for example, menu name, food category, attributes to be displayed such as allergy / country of origin labeling, shooting time / store information, applicable channel (kiosk / table order), and target specifications. The server (e.g., input / output module) combines the image and text metadata on a request basis to parse the context, and may set internal processing parameters or constraints so that the defect diagnosis and correction process in a subsequent step reflects the corresponding context (e.g., menu identity, expression standards by category, prohibited change elements).

[0060] The input / output module (210) can transmit the input image to the VLM engine (220).

[0061] In step S102, the VLM engine (220) analyzes the input image to generate a list of defects. The list of defects can be generated as structured data that includes at least the type and severity of the defects, and the rationale or explanation thereof. For example, by summarizing the defects, such as “global defocus (medium severity)” or “background objects around food (high severity),” and providing the rationale for why the defect is a problem and what improvements are needed, it enables the editing path and editing intensity to be rationally determined in subsequent steps. At this time, the VLM engine (220) also determines whether the defect is suitable for global improvement (global correction) or local improvement (local inpainting) and performs the role of branching the subsequent flow.

[0062] In this embodiment, defect diagnosis based on the VLM engine (220) may go beyond simple labeling to include "strategy formulation." That is, the VLM engine (220) can diagnose defects by including items such as global image sharpness (blur due to shaking / focus mismatch), global noise level (grain / color noise due to low-light / high-sensitivity shooting), local obstructions (e.g., surrounding objects or shadow projections such as tissues, cutlery, fingers, shadows), and lens contamination (local smudge or contrast reduction due to fingerprints / stains). The VLM engine (220) may be configured to determine parameters to be used in a subsequent restoration (global correction / local inpainting) pipeline based on the diagnosis results.

[0063] Additionally, the VLM engine (220) can return structured data in JSON format as a “strategy output” that includes defect type, severity, rationale, as well as restoration strength (e.g., Denoising Strength) and location information of the target for removal. For example, the JSON may include a value indicating the strength of global correction (denoising_strength or an equivalent control variable), masking coordinates (bounding box / polygon / point) for objects requiring local removal, processing priority per defect, global / local path selection flags, etc. Thus, the JSON-based strategy output produced in step S102 is directly utilized in subsequent steps (S103 to S105) for mask generation and refinement and for configuring generative editing commands, thereby ensuring that the connection between defect diagnosis and restoration execution is maintained consistently.

[0064] In step S103, the VLM engine (220) calculates defect locations or polygon candidates for at least some of the defects included in the defect list and transmits them to the mask generation module (230). Here, “polygon candidates” are candidate representations of a region of interest (ROI) for specifying defect areas, and can take various forms such as bounding boxes, polygon coordinates, or sets of points. If multiple defects exist, multiple polygon candidates may be generated for each defect, and these are used as inputs to reliably determine the final editing range in the subsequent mask refinement step.

[0065] In step S104, the mask generation module (230) generates a final mask (Mi) based on the defect location / polygon candidate. The generated final mask (Mi) can be transmitted to an administrator terminal (20) or a service terminal (30) via an input / output module (210).

[0066] The final mask (Mi) defines the editing target area in pixel units, and it is desirable to include a refinement process rather than simply adopting the defect candidate areas as they are. For example, morphological corrections such as boundary smoothing and dilation / erosion, the generation and selection of multiple candidate masks, and post-processing to reduce omission or overextension of defect boundaries may be performed. Additionally, since this embodiment may incorporate constraints to exclude prohibited areas (e.g., text, logos, price tags, specific parts of key subjects, etc.), the final mask (Mi) can be determined by subtracting the "prohibited areas" from the "editable areas." Such exclusion of prohibited areas functions as a safeguard to enhance the reliability of the output and its applicability to services.

[0067] In step S105, the administrator terminal (20) or service terminal (30) configures an edit request to be transmitted to the generative edit module (240) via the input / output module (210) and transmits it to the generative edit module (240). The edit request may be configured based on the strategy output of the VLM engine (220) and may include at least an input image, a final mask (Mi), an edit instruction, and a correction strength parameter (β).

[0068] In step S106, the generative editing module (240) receives the editing request as input and generates a corrected image (after). In this case, if global correction is required, it operates to improve the overall quality while maintaining the composition and meaning of the input image, and if local correction is required, it selectively performs inpainting or partial restoration on the area designated as the final mask (Mi). Additionally, in this embodiment, it may be configured to generate a plurality of candidate (after) images as needed and select the optimal candidate in subsequent verification.

[0069] The candidate image can be transmitted to the administrator terminal (20) or service terminal (30) via the input / output module (210). Additionally, the candidate image can be transmitted to the verification module (250).

[0070] In step S107, the verification module (250) determines whether quality criteria and / or semantic preservation criteria are met based on a comparison of the input image and the improved image. The verification may include, for example, (i) semantic preservation verification based on embedding similarity between the input image and the improved image and / or (ii) text consistency verification in which the improved image is re-input into a multimodal artificial intelligence model to determine whether the response to a text-based query matches the text metadata. If the verification result does not meet the criteria, the policy engine (270) may trigger reprocessing by adjusting the editing instructions and / or parameters and / or masks. Specifically, at least one of the following may be performed: adjusting β, adjusting the structural control strength, modifying the prompt, redefining the forbidden area or mask (Mi), adjusting the execution order of the global correction path and the local correction path, or re-executing the defect judgment of the VLM engine (220).

[0071] delete

[0072] FIG. 4 is a flowchart illustrating an image enhancement method according to another embodiment of the present invention.

[0073] Referring to FIG. 4, this embodiment is based on the basic flow (S101–S104) of FIG. 3, but is an extended embodiment in which “Track A / B-based generative restoration / editing” and “semantic preservation verification and re-execution loop” are clearly revealed, particularly in steps S105–S107. In this embodiment, steps S105–S107 can be concretized beyond the level of receiving a simple correction request / result, into steps of structuring and parameterizing the analysis results of VLM into editing commands, executing global restoration and local inpainting in sequence, and then determining acceptance, rejection, and re-execution by determining the semantic preservation of the result through quantitative and qualitative combined verification.

[0074] In step S105, the input / output module (210) and the VLM engine (220) use the defect list (type, severity, rationale) generated in step S102 and the mask (Mi) confirmed in steps S103–S104 to construct an editing command in a form that the generative editing module (240) can execute immediately. This editing command consists of pipeline instructions including “whether to perform global restoration (Track A),” “whether to perform local inpainting (Track B),” “execution order (e.g., Track A followed by Track B),” and control parameters for each track. In particular, for Track A, it includes a denoising strength or control variable (β) corresponding to the restoration strength value calculated by the VLM engine (220) to suppress excessive regeneration and limit restoration to a refinement level. Additionally, for Track B, editing instructions can be configured to suppress the illusion of toppings / ingredients that were not in the original being generated in the removed area, by including a removal target area designated as a mask (Mi) and a negative prompt or prohibition rule such as “Do not generate food.” Consequently, step S105 is a step of replacing the semantic diagnosis of the VLM engine (220) with a generative editing execution condition, and is the point where the “analysis-editing link” of the present invention is specifically implemented.

[0075] In step S106, the present embodiment emphasizes a flow in which the generative editing module (240) generates a correction image according to the editing command configured in S105, while executing Track A and Track B sequentially. First, in Track A, structural control can be applied to prevent hallucination while performing diffusion-based Img2Img restoration. For example, structural constraints such as a contour map or depth map are extracted from the input image and input into a ControlNet-based adapter to guide the generation process so that it does not excessively deviate from the original geometric shape and object placement. Subsequently, the denoising intensity (or β) proposed by the VLM engine (220) is set to a relatively low value so that the restoration is performed in a direction of “restoring and refining” the quality of sharpness, noise, illumination, and color, rather than in a direction of “redrawing” the image. Then, in Track B, the VLM engine (220) uses the result image from Track A as input and selectively performs inpainting on a mask (Mi) that reflects a specific obstruction area. At this point, SAM-based precision masking results can be utilized, and a negative prompt is used to naturally fill the removed area with the table texture / background context. Consequently, step S106 is a step that combines global quality improvement and local defect removal to produce an improved image suitable for service application while maintaining the layout and shape of the original.

[0076] In step S107, the verification module (250) compares and verifies the image before correction (before) and the image after correction (after) to determine whether to accept it, and if there is a discrepancy or signs of danger, it triggers Reject and re-execution. In the embodiment of FIG. 4, the verification of semantic preservation is key, and can combine (i) quantitative verification based on CLIP embedding similarity (distance) and (ii) text consistency verification based on VLM engine (220) material. For example, it checks whether the embedding similarity between the original and the result is above a threshold to check whether the object identity / composition change is excessive, and at the same time, the result image is re-input into the VLM engine (220) to check whether the response to a query such as “What is this food?” matches the initial input metadata (menu name / category). If a discrepancy occurs, the result is rejected, and steps S105–S106 can be configured to be repeated by selecting a method such as lowering β (denoising intensity), strengthening structural constraint weights, refining the mask (Mi), or strengthening the negative prompt as a re-execution condition. This iterative loop functions as a safeguard to converge to a result that satisfies not only simple image quality improvement but also “meaning preservation + policy compliance.”

[0077] FIG. 5 is a drawing for explaining a partial restoration method according to an embodiment of the present invention.

[0078] Referring to FIG. 5, there are first to third defect regions (901 to 903) within the input image (IM), and the server sets a restoration region (910) to include all of these defect regions (901 to 903), and then performs partial restoration by spatially varying the restoration index (restoration strength) within the restoration region (910). The key is to achieve fast processing and natural boundary bonding simultaneously by (i) increasing the restoration strength closer to the center of the defect and (ii) minimizing the restoration strength closer to the boundary of the restoration region (910).

[0079] First, in the defect detection phase, the coordinates of the input image (IM) Set it as such, defect (severity) map Calculate the defect mask corresponding to the defect region (901~903). At this time, respectively If left as such, the entire defect area is It is defined as. Here It is set to a value closer to 1 for larger defects within the defect region and closer to 0 for smaller defects (e.g., a combination of the probability and intensity of blur / noise / contamination / obstruction presence).

[0080] Next, the restoration area (910) is the entire defect area Since it must be set to completely include, the restoration area is the dilation or outer envelope of the defect mask. Defines. For example It can be set as, and the radius of expansion It can be configured to be larger as the defect becomes more severe, so that the "context surrounding the defect is also restored." For example, the defect is also average Depending on If left as is, the restoration area (910) is wider when the defects of 901~903 are generally severe, and the restoration area (910) is not unnecessarily large when the defects are minor, thereby improving the processing speed.

[0081] Next, to ensure “minimization of restoration index at the boundary” within the restoration area (910), the restoration area The boundary of Distance to " Calculate " and boundary damping (taper) function Defined as. Here is the boundary buffer width, is the damping gradient at the boundary. Then, the final restoration index It is designed to "significantly reflect defect levels and converge to 0 at boundaries." It can be set to. If you increase it, in areas with high defect severity (e.g., high-severity defects like 901) As it rapidly increases, the restoration index is maximized, and at the boundary of the restoration area (910) This minimizes the restoration index. As a result, the restoration strength is more concentrated for the "larger defect" among 901 to 903, and the area outside or near the boundary of the restoration area (910) is almost preserved from the original, reducing seams.

[0082] It is directly linked to the parameters of generative restoration / inpainting. For example, the denoising intensity (or correction intensity) in diffusion-based Img2Img or inpainting cast If left as is, the center of the defect (inside 901~903) is As it grows larger, strong restoration is performed, and at the boundary of the restoration area (910) It is set to suppress excessive changes. In addition, the number of iterations / sampling steps are set for processing speed. do Vary it to, and In-pixel enables fast ROI-based processing by omitting computations (or applying only low-cost filters).

[0083] For “natural handling with areas where restoration was not performed,” the restoration result and original Blend with the restoration index. If left as is, even within the restoration area (910), as it approaches the boundary As a result, the original naturally becomes dominant, and in the center The restoration results are fully reflected. At this time Applying Gaussian smoothing once makes boundary changes smoother, further reducing visual dissonance.

[0084] To summarize, after searching for defect areas 901 to 903 in FIG. 5, a restoration area (910) including all of them is set, and the defects and boundary distance Restoration index combining By spatially distributing the restoration strength (β) and computational resources (N), it is possible to achieve "critical defect intensive restoration + boundary minimal restoration + fast processing + natural joining" simultaneously.

[0085] FIG. 6 is a drawing for explaining a partial restoration method according to an embodiment of the present invention.

[0086] Referring to FIG. 6, the input image (IM) has first to third defect regions (901 to 903) and fourth to sixth defect regions (904 to 906) separated from each other, and a total restoration method can be configured by clustering these defects to set multiple concentrated restoration regions (910, 920) and treating the other regions as general restoration regions (930).

[0087] The following describes specific method examples regarding "defect detection → clustering → setting intensive restoration areas → applying restoration indices → minimizing general restoration → natural boundary joining."

[0088] The server (10) calculates a severity map S(x) ∈ [0,1] for each coordinate x ∈ Ω of the input image (IM). S(x) can be defined as a value that integrates global / local noise levels, sharpness degradation, signs of lens contamination, and the possibility of obstruction presence. A defect candidate mask D(x) is formed using a threshold value τ. Defined as such, it forms a set of pixels corresponding to defect areas (901~906).

[0089] The server performs spatial distance-based clustering on defect objects (connected components) or sets of defect pixels derived from the defect candidate mask D(x). For example, clusters can be formed by applying methods such as DBSCAN based on the density of the center points or boundary points of defect objects, or by applying connected component merging rules. The clustering results are the defect mask and K cluster masks. decomposing into It can be expressed in the form of.

[0090] Each cluster For this, a concentrated restoration area is set to include the entire cluster. Since the concentrated restoration area is preferably the envelope of the cluster mask plus a safety margin, for example, It can be defined as. Here The outer boundary surrounding the cluster (e.g., convex hull or bounding polygon), is an expansion operation, and It can be varied depending on the degree of defect. For example, the cluster average defect rate cast Defined as, If set to this value, the more severe the defect in the cluster, the wider the concentrated restoration area is set, allowing context restoration to be performed stably. In Fig. 6 This first intensive restoration area (910), It corresponds to the second concentrated restoration area (920).

[0091] Now, to satisfy the requirement of "large restoration indices based on the degree of defect within the intensive restoration area, and small ones at the boundaries," cluster-specific restoration indices Defines. First, the concentrated restoration area boundary Distance conversion to Let it be, and the boundary taper (damping) function is It is defined as follows. Here, the clip function is a function that clamps a value so that it does not go outside a specified range, keeping it within that range. It is usually written in the form clip(z, a, b), and it forcibly restricts the input value z to be between the minimum value a and the maximum value b. As a formula, It is defined as.

[0092] Here is the boundary buffer width, is an index that determines how quickly the restoration index drops to zero at the boundary. Then, the restoration index of the concentrated restoration area is calculated by multiplying the defect emphasis term and the boundary attenuation term. It can be placed as. Increasing it further maximizes the restoration index in areas with large defects, and In As a result, the restoration index is minimized at the boundary of the concentrated restoration area. Consequently, spatially variable restoration is implemented with strong restoration at the center of the 910 / 920 interior and weak restoration at the boundary.

[0093] The general restoration area (930) is the complement of the concentrated restoration areas It can be defined as follows. In the general recovery region, since fast processing must be achieved by minimizing the recovery index, it can be a constant or a very small value. ( It can be defined as (or a value close to 0). The total restoration index is You can combine them (even if there is overlap, the strongest restoration index is applied first).

[0094] Total restoration index This is directly linked to the control of intensity and computational load in the generative restoration pipeline. For example, the denoising intensity (or intensity control variable) of diffusion-based Img2Img / inpainting cast When set to this, β increases at the center of the defect in the concentrated restoration area (910, 920) so that strong restoration is performed, and in the general restoration area (930) since In effect, only minimal correction is performed. To further increase processing speed, the sampling steps / iteration count is also Let it be, In the general restoration area, "fast processing" is achieved by configuring it to omit generative restoration or apply only low-cost filters (light denoise / tone correction).

[0095] Finally, for natural processing at the boundary between the intensive restoration area and the general restoration area, the intensive restoration result and general processing results Blend with restoration index-based weights.

[0096] If set to this, the center of the concentrated restoration area In this area, most of the restoration results are reflected, and the boundary and general restoration areas In this case, general processing / originals become dominant, reducing seams. If necessary, Apply Gaussian smoothing to By making the spatial changes smoother, the unnaturalness of the boundaries can be further reduced.

[0097] FIG. 7 is a diagram illustrating the hardware configuration of a server according to one embodiment of the present invention.

[0098] Referring to FIG. 7, the server (10) may include a processor (110), memory (120), a transmitting / receiving device (130), an input interface device (140), an output interface device (150), a storage device (160), and a bus (170). The processor (110) executes program instructions stored in memory (120) or storage device (160) to perform the functional configuration of FIG. 2 (e.g., VLM-based analysis, mask generation, generative editing, verification, and policy control, etc.) and the image enhancement method of FIG. 3 or FIG. 4. Memory (120) includes ROM and RAM to store programs, processing parameters, intermediate results, etc., and storage device (160) may store original images, corrected images, verification reports, logs, and learning / policy data. The transmitting / receiving device (130) may receive input images and metadata through wired / wireless communication with an administrator terminal or a service terminal, and transmit enhanced images and verification results. The input / output interface device (140, 150) supports user input and result display, and the camera module (180) can acquire an image to be improved by providing image capture or video input according to the embodiment. The bus (170) is connected to transmit data and control signals between the components.

[0099] An image enhancement method according to one embodiment of the present invention (e.g., steps according to FIGS. 3 and 4) and / or functional components of a server (10) (e.g., functions performed by the input / output module, VLM engine, mask generation module, generative editing module, verification module, image generation module, and policy engine of FIG. 2) may be implemented as a combination of program instructions executable by the processor (110) of the server (10). The program instructions may be stored in a computer-readable recording medium, and the recording medium may include ROM, RAM, flash memory, hard disk, SSD, optical disk, magnetic disk, or a storage area downloadable over a network.

[0100] For example, the above program commands may include a command to cause the server (10) to perform input / output functions of receiving and transmitting input images and metadata, a command to analyze the input image using a visual-language model (VLM) and generate defect information (at least one of type / severity / location / reason), a command to generate or refine a mask based on the defect information, a command to generate an improved image by performing global correction and / or local correction based on editing instructions and parameters, a command to perform quality and / or semantic preservation verification on the improved image, and a command to control output or re-execution based on the verification results. Accordingly, each module of FIG. 2 is not limited to a component fixed in hardware, but can be implemented as a program command executed by the processor (110) or as a modularized software configuration.

[0101] Additionally, the above program commands may cause the server (10) to perform receiving an input image (S101), generating a VLM-based defect list (S102), calculating defect location candidates (S103), generating / refining a mask (S104), configuring an edit request (S105), generating a corrected image through generative editing (S106), and verifying and deciding acceptance / retry based on before-and-after comparison (S107). Depending on the embodiment, the above commands may cause multiple tracks including Track A (global restoration) and Track B (local inpainting) to be executed sequentially or in parallel, or to perform a reprocessing loop by changing at least one of the parameters / masks / prompts according to the result of semantic preservation verification. Accordingly, the method of the present invention may be implemented as commands stored on a computer-readable recording medium.

[0102] Although embodiments according to the technical concept of the present invention have been described above with reference to the attached drawings, those skilled in the art will understand that the present invention may be implemented in other specific forms without changing its technical concept or essential features. The embodiments described above should be understood as illustrative in all respects and not restrictive. Explanation of the symbols

[0103] 10: Server 110: Processor 120: Memory 130: Transmitter / Receiver 140: Input interface device 150: Output interface device 160: Storage device 170: Bus 180: Camera module 20: Administrator Terminal 210: Input / Output Module 220: VLM engine 230: Mask generation module 240: Generative Edit Module 250: Verification Module 260: Image generation module 270: Policy Engine 30: Service terminal 901~903: 1st~3rd defect areas 910: 1st Intensive Restoration Zone 920: 2nd Intensive Restoration Zone 930: General recovery area IM: Input Image

Claims

Claim 1 In an image enhancement method performed by an electronic device, the method receives an input image and text metadata corresponding to the input image, analyzes the input image using a multimodal artificial intelligence model including a visual-language model (VLM) to generate defect information regarding defects in the input image, determines execution conditions for enhancement processing on the input image based on the defect information, and performs image processing on the input image according to the execution conditions to generate an enhanced image, wherein the defect information includes at least one of parameters regarding the type of defect, the severity of the defect, the location of the defect, the range of the defect, and the correction strength, and determining the execution conditions includes selecting at least one of global enhancement processing and local enhancement processing based on the defect information, and determining region information or mask information indicating an area to be edited in the input image, wherein the image processing includes at least one of a global restoration track that uses structural information extracted from the input image using an image-to-image generation model as a control input for the generation process, and a local inpainting track that performs inpainting using a negative prompt or prohibition rule on an area designated by a mask generated using a segmentation model based on location information included in the defect information, and wherein the global restoration track and the local The inpainting track is performed sequentially, and for the improved image, a verification is performed including at least one of (i) semantic preservation verification based on embedding similarity between the input image and the improved image, and (ii) text consistency verification in which the improved image is re-input into the multimodal artificial intelligence model to determine whether the response to a text-based query matches the text metadata; if the verification result satisfies the criteria, the improved image is output, and if the verification result does not satisfies the criteria, the correction strength parameter, the strength of the structural control, the prompt, the mask information,or controlling to re-perform the image processing by changing at least one of the execution order of the global restoration track and the local inpainting track, Image improvement methods. Claim 2 An image enhancement method according to claim 1, wherein the output of the multimodal artificial intelligence model is generated in a structured format that can be interpreted by a subsequent module, and the structured format includes at least one of a defect label, a severity score, region of interest coordinates, and a recommended parameter. Claim 3 An image enhancement method according to claim 2, wherein the structured format is a JSON format including defect type, severity, basis for judgment, restoration strength, location information of the removal target, and global / local path selection flags. Claim 4 An image enhancement method according to claim 1, wherein the multimodal artificial intelligence model, based on the defect information, produces at least one of an editing instruction including performing global correction, performing specific area inpainting, and recommending re-shooting, and at least one of a text prompt, a negative prompt, a defect description, a guide regarding correction intensity, and a constraint for structural preservation. Claim 5 An image enhancement method according to claim 1, wherein the mask information is generated or refined based on region of interest information provided by the multimodal artificial intelligence model, and the region of interest information includes at least one of a bounding box, a point, a text-based instruction, and a polygon candidate. Claim 6 An image enhancement method according to claim 5, wherein the mask information is generated as a pixel-unit mask using at least one of a traditional image processing technique including a segmentation model or edge, color, brightness-based separation and morphology operations. Claim 7 An image enhancement method according to claim 1, wherein an editing request for performing the image processing comprises the input image, a final mask, an editing instruction, and a correction intensity parameter. Claim 8 An image enhancement method according to claim 1, wherein the verification additionally determines, in addition to whether the image quality indicator is improved, whether the identity of the core subject is maintained, whether the prohibited area is damaged, whether unnecessary objects are created, and whether the quality standard for each channel is satisfied. Claim 9 An image enhancement method according to claim 1, wherein the re-execution further comprises modifying editing instructions, re-regulating prohibited areas or masks, or re-executing defect judgment by the multimodal artificial intelligence model. Claim 10 An image enhancement method according to claim 1, wherein the verification comprises at least one of (i) semantic preservation verification based on embedding similarity between the input image and the enhanced image, and (ii) text consistency verification in which the enhanced image is re-input into the multimodal artificial intelligence model to determine whether the response to a text-based query matches text metadata. Claim 11 An image improvement method according to claim 1, wherein if the verification result satisfies the criteria, the improved image and verification report are provided to an administrator terminal or a service terminal, and subsequent distribution or configuration is performed so that the improved image can be applied to the service terminal. Claim 12 A method for image enhancement performed by an electronic device comprises: receiving an input image and text metadata corresponding to the input image; analyzing the input image and the text metadata using a multimodal artificial intelligence model including a visual-language model (VLM) to generate region of interest information representing at least one defect area and defect information including the type, severity, and location of a defect corresponding to the at least one defect area; calculating a defect map representing the degree of defect corresponding to the coordinate x of the input image based on the severity and location information included in the defect information; setting a restoration area to include the at least one defect area; determining a spatially variable restoration index within the restoration area, wherein (i) the restoration index increases as the region with a greater degree of defect is based on the defect map, and (ii) the restoration index decreases as it approaches the boundary of the restoration area based on the distance from the boundary of the restoration area; and generating a corrected image by performing restoration processing on the input image while controlling the intensity and / or amount of computation of image processing performed in the restoration area based on the spatially variable restoration index. and to reduce the unnaturalness of the boundary between the correction image and the input image, the method includes the step of generating an output image by blending the correction image and the input image or a reference image generated from the input image using weights based on the spatially variable restoration index, wherein the spatially variable restoration index is the distance to the boundary of the above-mentioned restoration area Taper function based on and the above defect map By combining, it is determined by the following mathematical formula 1, and <Mathematical Formula 1> , the above taper function is defined by the following mathematical formula 2, and <Mathematical Formula 2> ,in the above mathematical formula 2, w is the boundary buffer width, and Person, image improvement method. Claim 13 An image enhancement method according to claim 12, wherein Gaussian smoothing is applied to the spatially variable restoration index, and then the corrected image and the input image or the reference image are blended using weights based on the spatially variable restoration index to which the Gaussian smoothing is applied. Claim 14 A method for image enhancement performed by an electronic device comprises: receiving an input image and text metadata corresponding to the input image; analyzing the input image and the text metadata using a multimodal artificial intelligence model including a visual-language model (VLM) to generate region of interest information representing a plurality of defect regions and defect information including the type, severity, and location of a defect for each of the plurality of defect regions; calculating a defect map for each of the plurality of defect regions based on location information and severity included in the defect information; clustering the plurality of defect regions based on spatial proximity and / or similarity to generate a plurality of defect clusters; setting a plurality of concentrated restoration regions to each include a defect cluster, and setting the remaining regions excluding the plurality of concentrated restoration regions as general restoration regions; determining a spatially variable restoration index for each of the plurality of concentrated restoration regions such that (i) the restoration index increases as the region with a greater degree of defect increases based on the defect map, and (ii) the restoration index decreases as it approaches the boundary of the concentrated restoration region based on the distance from the boundary of the concentrated restoration region; and setting a minimum restoration index smaller than the spatially variable restoration index for the general restoration region. A method for image enhancement comprising: a step of generating a corrected image by performing restoration processing on the input image while differentially controlling the intensity and / or amount of image processing performed in the plurality of concentrated restoration areas and the general restoration area based on the spatially variable restoration index and the minimum restoration index; and a step of generating an output image by blending the corrected image with the input image or a reference image generated from the input image using weights based on the spatially variable restoration index to reduce unnaturalness at the boundary between the concentrated restoration area and the general restoration area. Claim 15 In claim 14, the k-th concentrated restoration area among the plurality of concentrated restoration areas is the cluster mask corresponding to the k-th defect cluster The outer veil of It is set to the following Equation 3 by the expansion operation for, and <Equation 3> ,expansion radius is the cluster average defect rate Based on, it is determined by the following mathematical formula 4, and <Mathematical Formula 4> , the k-th intensive restoration area above Spatial variable recovery index for The defects and the above intensive restoration area Distance to the boundary of Based on, it is calculated by the following mathematical formula 5, and <Mathematical Formula 5> Total restoration index is determined by the following mathematical formula 6, and <Mathematical Formula 6> , intensity control variable of the above restoration process is set by the following mathematical formula 7, <Mathematical Formula 7> ,Image improvement method. Claim 16 In an electronic device that performs enhancement processing on an input image, a communication interface; a memory; The system includes one or more processors that execute one or more instructions stored in the memory, wherein the one or more processors receive an input image and text metadata corresponding to the input image through the communication interface, and generate defect information regarding defects in the input image by analyzing the input image using a multimodal artificial intelligence model including a visual-language model (VLM), determine execution conditions for improvement processing on the input image based on the defect information, and generate an improved image by performing image processing on the input image according to the execution conditions, wherein the defect information includes at least one of parameters regarding the type of defect, the severity of the defect, the location of the defect, the range of the defect, and the correction strength, and determining the execution conditions includes selecting at least one of global improvement processing and local improvement processing based on the defect information, and determining region information or mask information representing an area to be edited in the input image, wherein the image processing includes a global restoration track that uses structural information extracted from the input image using an image-to-image generation model as a control input for the generation process, and a local track that performs inpainting using a negative prompt or prohibition rule on an area designated by a mask generated using a segmentation model based on location information included in the defect information. The method includes at least one of the inpainting tracks, wherein the global restoration track and the local inpainting track are performed sequentially, and for the improved image, the method performs a verification including at least one of (i) semantic preservation verification based on embedding similarity between the input image and the improved image, and (ii) text consistency verification in which the improved image is re-input into the multimodal artificial intelligence model to determine whether the response to a text-based query matches the text metadata.An electronic device that outputs the improved image when the verification result meets the criteria, and controls the image processing to be re-performed by changing at least one of the correction strength parameter, the strength of the structural control, the prompt, the mask information, or the execution order of the global restoration track and the local inpainting track when the verification result does not meet the criteria. Claim 17 In a computer-readable recording medium, the recording medium enables an electronic device to receive an input image and text metadata corresponding to the input image, analyze the input image using a multimodal artificial intelligence model including a visual-language model (VLM) to generate defect information regarding defects in the input image, determine execution conditions for improvement processing on the input image based on the defect information, and perform image processing on the input image according to the execution conditions to generate an improved image, wherein the defect information includes at least one of parameters regarding the type of defect, the severity of the defect, the location of the defect, the range of the defect, and the correction strength, and determining the execution conditions includes selecting at least one of global improvement processing and local improvement processing based on the defect information, and determining region information or mask information indicating an area to be edited in the input image, wherein the image processing includes at least one of a global restoration track that uses structural information extracted from the input image using an image-to-image generation model as a control input for the generation process, and a local inpainting track that performs inpainting using a negative prompt or prohibition rule on an area designated by a mask generated using a segmentation model based on location information included in the defect information. Includes one, wherein the global restoration track and the local inpainting track are performed sequentially, and for the improved image, a verification is performed including at least one of (i) semantic preservation verification based on embedding similarity between the input image and the improved image, and (ii) text consistency verification in which the improved image is re-input into the multimodal artificial intelligence model to determine whether the response to a text-based query matches the text metadata, and if the verification result satisfies the criteria, the improved image is output, and if the verification result does not satisfies the criteria, the correction strength parameter, the strength of the structural control,A computer-readable recording medium storing a program that controls the re-execution of the image processing by changing at least one of a prompt, mask information, or the execution order of the global restoration track and the local inpainting track.

Images