An image restoration method based on self-evolution experience pool and related equipment

Abstract

The application discloses an image restoration method based on self-evolution experience pool and related equipment, and the method comprises the steps of identifying the degradation type of a degraded image to obtain a degradation type set; based on the degradation type set and user preference requirements, searching in a hierarchical experience pool to obtain a target restoration planning scheme, and then restoring the degraded image to obtain a first restored image; evaluating the degradation condition of the first restored image to determine whether each degradation type in the degradation type set has been eliminated; if all have been eliminated, the first restored image is taken as a final restored image; if there is a degradation type that has not been eliminated, the target restoration planning scheme is corrected, and the operation of restoring the degraded image is performed again until each degradation type has been eliminated, and a final restored image is obtained. The application can improve the restoration planning efficiency while retaining the high compatibility of the training-free method, and can be widely applied in the field of image processing technology.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to an image restoration method and related equipment based on a self-evolving experience pool. Background Technology

[0002] Image restoration agents aim to leverage the image understanding and planning capabilities of multimodal large language models to plan the removal order of degradations and select restoration tools based on the perceived degradation type. However, in the absence of experience-driven approaches, general multimodal large language models often incur high trial-and-error costs when facing zero-shot planning in order to achieve satisfactory restoration results.

[0003] However, existing image restoration methods can be broadly categorized into training-based and training-free methods. Training-based methods fine-tune a general multimodal large language model to enable it to perceive and plan degradation for image restoration tasks. These methods embed experience into the model parameters, offering high inference efficiency but lacking compatibility with new tools or degradation types, requiring retraining for expansion. Training-free methods, on the other hand, focus more on system design, emphasizing workflow orchestration and coordinating interactions between different modules. While these methods offer high compatibility, the lack of effective experience-guided mechanisms means that in complex scenarios with multiple coupled degradation types, extensive sequential searches and tool trials are still necessary during the planning phase, resulting in lower inference efficiency. Furthermore, existing image restoration methods do not consider users' subjective preferences for the restored image, failing to dynamically adjust the degradation removal order and tool selection based on user preferences, making it difficult to meet the personalized requirements of different application scenarios. Summary of the Invention

[0004] The main objective of this application is to propose an image restoration method and related equipment based on a self-evolving experience pool, which can improve the restoration planning efficiency while retaining the high compatibility of the training-free method, and enable the restoration results to meet the personalized requirements of users in different application scenarios.

[0005] To achieve the above objectives, one aspect of this application proposes an image restoration method based on a self-evolving experience pool, comprising: Acquire degraded images and perform degradation identification on the degraded images to obtain a set of degradation types; Based on the set of degradation types and user preference requirements, a target restoration plan is obtained by searching a pre-built hierarchical experience pool through key-value matching. The target restoration plan includes a first degradation removal order of the set of degradation types and a first restoration tool corresponding to each degradation type. According to the first degradation removal order, the first restoration tool corresponding to each degradation type is called sequentially to perform restoration operations on the degraded image to obtain the first restored image; The first restored image is subjected to a degradation assessment to determine whether each degradation type in the set of degradation types has been eliminated; If each of the degradation types in the set of degradation types has been eliminated, the first restored image is taken as the final restored image; If at least one of the degradation types in the set of degradation types has not been eliminated, the target restoration plan is modified, and the modified restoration plan is used as the target restoration plan. The restoration operation on the degraded image is then performed again until each of the degradation types in the set of degradation types has been eliminated, and the final restored image is obtained.

[0006] In some embodiments, the hierarchical experience pool includes a first granularity layer, a second granularity layer, and an insight layer; The step of retrieving a target restoration plan based on the set of degradation types and user preference requirements by using key-value matching in a pre-built hierarchical experience pool includes: Using the set of degradation types and user preference requirements as the first query key, key-value matching retrieval is performed in the first granularity layer. If a first key value matching the first query key exists in the first granularity layer, a significance test is performed on the capability parameter corresponding to the first key value. If the significance test passes, the degradation order corresponding to the first key value is used as the first degradation removal order. If the significance test fails, the target pattern profile is determined based on the set of degradation types and the user preference requirements; The target pattern profile is used as the second query key. Key-value matching retrieval is performed in the second granularity layer. If there is a second key value in the second granularity layer that matches the second query key, the degradation order corresponding to the second key value is used as the first degradation removal order. If there is no first key value matching the first query key in the first granularity layer or no second key value matching the second query key in the second granularity layer, the user preference requirement is used as the third query key, and key value matching retrieval is performed in the insight layer to determine the third key value matching the third query key. The first degradation removal order is determined based on the text experience corresponding to the third key value; Obtain the first restoration tool corresponding to each degradation type, and based on the first degradation removal order and the first restoration tool corresponding to each degradation type, obtain the target restoration plan.

[0007] In some embodiments, determining the target pattern profile based on the set of degradation types and the user preference requirements includes: Obtain multiple pre-stored pattern profiles from the second granularity layer; The degraded image and each of the pattern portraits are encoded to obtain the image encoding features corresponding to the degraded image and the pattern encoding features corresponding to each of the pattern portraits; Calculate the similarity between the image coding features and each of the pattern coding features, and determine a candidate pattern profile set from the multiple pattern profiles based on the similarity. The semantic anchor text corresponding to each candidate pattern profile in the candidate pattern profile set and the degraded image are input into a preset multimodal large language model for matching and evaluation processing to obtain the target pattern profile.

[0008] In some embodiments, determining the first degradation removal order based on the textual experience corresponding to the third key value includes: Retrieve preset prompts; Based on the preset guiding prompts, the textual experience, and the set of degradation types, a reasoning and planning process is performed to obtain the first degradation removal order.

[0009] In some embodiments, revising the target restoration plan and using the revised restoration plan as the target restoration plan, and then returning to perform a restoration operation on the degraded image, includes: Obtain priority experience guidance data from the hierarchical experience pool; Based on the priority experience guidance data, the first degradation removal order in the target restoration planning scheme is modified; The modified degradation removal order and the first restoration tool corresponding to each degradation type are used as the modified first restoration plan scheme; Using the modified first restoration plan as the target restoration plan, return to the step of restoring the degraded image to obtain the second restored image; The second restored image is subjected to degradation assessment. If there are still unresolved degradation types in the set of degradation types, the first restoration tool corresponding to each degradation type in the target restoration plan is modified based on the priority experience guidance data to obtain the modified restoration tool for each degradation type. The modified degradation removal order and the modified restoration tool for each degradation type are used as the modified second restoration plan scheme; Using the modified second restoration plan as the target restoration plan, return to the step of performing the restoration operation on the degraded image until all degradation types are eliminated.

[0010] In some embodiments, the method further includes: Obtain the set of degradation types and the set of restoration tools corresponding to the degraded image; Based on the set of degradation types and the set of restoration tools, multiple restoration planning paths are determined for the degraded image. When the number of degradation types in the set of degradation types is 1, each restoration planning path includes a second restoration tool corresponding to the degradation type. When the number of degradation types in the set of degradation types is greater than 1, each restoration planning path includes a second degradation removal order of the set of degradation types and a second restoration tool matched for each degradation type under the second degradation removal order. Using each of the restoration planning paths, a restoration operation is performed on the degraded image to obtain a third restored image corresponding to each restoration planning path; Based on the image quality evaluation index corresponding to the user preference requirements, the quality of each of the third restored images is evaluated, and the win rate ranking of each second restoration tool is determined based on the evaluation results. Based on the degraded image, the set of degradation types, the set of restoration tools, and the win rate ranking, an atomic experience record is generated; The atomic experience records are stored. When the number of atomic experience records stored in the same set of degradation types accumulates to a preset batch size, the experience in the hierarchical experience pool is evolved using all the atomic experience records in the batch. The evolved hierarchical experience pool is used to guide the retrieval of restoration planning schemes, the correction of planning schemes, and the selection of optimal restoration tools for other degraded images.

[0011] To achieve the above objectives, another aspect of this application proposes an image restoration apparatus based on a self-evolving experience pool, the apparatus comprising: A degradation identification module is used to acquire degraded images and perform degradation identification on the degraded images to obtain a set of degradation types; The solution retrieval module is used to retrieve a target restoration plan solution from a pre-built hierarchical experience pool based on the set of degradation types and user preference requirements through key-value matching. The target restoration plan solution includes a first degradation removal order of the set of degradation types and a first restoration tool corresponding to each degradation type. The restoration operation module is used to sequentially call the first restoration tool corresponding to each degradation type to perform restoration operations on the degraded image according to the first degradation removal order, so as to obtain the first restored image; The degradation assessment module is used to perform degradation assessment on the first restored image to determine whether each degradation type in the degradation type set has been eliminated; A first image restoration module is used to take the first restored image as the final restored image when each of the degradation types in the set of degradation types has been eliminated. The second image restoration module is used to modify the target restoration plan scheme when at least one of the degradation types in the degradation type set has not been eliminated, and to use the modified restoration plan scheme as the target restoration plan scheme, and return to perform restoration operation on the degraded image until each of the degradation types in the degradation type set has been eliminated, so as to obtain the final restored image.

[0012] To achieve the above objectives, another aspect of this application provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the method described above.

[0013] To achieve the above objectives, another aspect of the embodiments of this application proposes a computer-readable storage medium storing a computer program that, when executed by a processor, implements the methods described above.

[0014] This application also discloses a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device can read the computer instructions from the computer-readable storage medium and execute the computer instructions, causing the computer device to perform the aforementioned method.

[0015] The embodiments of this application include at least the following beneficial effects: This application provides an image restoration method based on a self-evolving experience pool. This method first acquires degraded images and performs degradation identification on the degraded images to obtain a set of degradation types. Based on the set of degradation types and user preference requirements, a target restoration plan is obtained by searching a pre-constructed hierarchical experience pool using key-value matching. The target restoration plan includes a first degradation removal order for the set of degradation types and a first restoration tool corresponding to each degradation type. Following the first degradation removal order, the first restoration tool corresponding to each degradation type is sequentially called to restore the degraded image. The process involves performing a restoration operation to obtain a first restored image; performing a degradation assessment on the first restored image to determine whether each degradation type in the degradation type set has been eliminated; if each degradation type in the degradation type set has been eliminated, the first restored image is used as the final restored image; if at least one degradation type in the degradation type set has not been eliminated, the target restoration plan is modified, and the modified restoration plan is used as the target restoration plan. The process then returns to perform a restoration operation on the degraded image until each degradation type in the degradation type set has been eliminated, resulting in the final restored image. This application retrieves the target restoration plan based on a set of degradation types and user preferences within a hierarchical experience pool. This approach eliminates the need for fine-tuning of the large language model, thus avoiding retraining and ensuring good compatibility with new tools or degradation types. Furthermore, by using key-value matching in the hierarchical experience pool, restoration plans can be quickly matched directly to the current set of degradation types, allowing for compatibility with new tools or degradation types without model retraining. This maintains the high compatibility of the training-free method while improving restoration planning efficiency. Additionally, by using the user preference as one of the query keys in the hierarchical experience pool, the retrieved degradation removal order and tool selection can be dynamically adjusted according to the user's preferences, ensuring the restoration results meet the personalized requirements of users in different application scenarios. Attached Figure Description

[0016] Figure 1 This is a schematic diagram of an implementation environment provided in an embodiment of this application; Figure 2 This is a flowchart of an image restoration method based on a self-evolving experience pool provided in an embodiment of this application; Figure 3 This is a schematic diagram of the inference workflow of the training-free image restoration agent at each stage provided in the embodiments of this application; Figure 4 This is a schematic diagram of the evolutionary process of the self-evolutionary mechanism provided in the embodiments of this application; Figure 5 This is a schematic diagram of the structure of an image restoration device based on a self-evolving experience pool provided in an embodiment of this application; Figure 6 This is a schematic diagram of the hardware structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0017] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of this application and are not intended to limit it. In the following description, when referring to the accompanying drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with those of this application; they are merely examples of apparatuses and methods consistent with some aspects of the embodiments of this application.

[0018] It is understood that the terms "first," "second," "third," "fourth," etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0019] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0020] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0021] Existing image restoration methods can be broadly categorized into training-based and training-free methods. Training-based methods fine-tune a general multimodal large language model to enable degradation perception and planning capabilities for image restoration tasks. These methods embed experience into the model parameters, offering high inference efficiency but lacking compatibility with new tools or degradation types, requiring retraining for expansion. Training-free methods, on the other hand, focus more on system design, emphasizing workflow orchestration and coordinating interactions between different modules. While offering high compatibility, the lack of effective experience-guided mechanisms means that in complex scenarios with multiple coupled degradation types, extensive sequential searches and tool trials are still necessary during the planning phase, resulting in lower inference efficiency. Furthermore, existing image restoration methods do not consider users' subjective preferences for the restored image, failing to dynamically adjust the degradation removal order and tool selection based on user preferences, making it difficult to meet the personalized requirements of different application scenarios.

[0022] In view of this, this application provides an image restoration method and related device based on a self-evolving experience pool. The method first acquires degraded images and performs degradation identification to obtain a set of degradation types. Based on the set of degradation types and user preference requirements, a target restoration plan is obtained by searching a pre-constructed hierarchical experience pool using key-value matching. The target restoration plan includes a first degradation removal order for the set of degradation types and a first restoration tool corresponding to each degradation type. Following the first degradation removal order, the first restoration tool corresponding to each degradation type is sequentially called to restore the degraded image. Perform a restoration operation to obtain a first restored image; perform a degradation assessment on the first restored image to determine whether each degradation type in the degradation type set has been eliminated; if each degradation type in the degradation type set has been eliminated, use the first restored image as the final restored image; if at least one degradation type in the degradation type set has not been eliminated, modify the target restoration plan and use the modified restoration plan as the target restoration plan, then return to perform a restoration operation on the degraded image until each degradation type in the degradation type set has been eliminated, and obtain the final restored image. This application retrieves the target restoration plan based on a set of degradation types and user preferences within a hierarchical experience pool. This approach eliminates the need for fine-tuning of the large language model, thus avoiding retraining and ensuring good compatibility with new tools or degradation types. Furthermore, by using key-value matching in the hierarchical experience pool, restoration plans can be quickly matched directly to the current set of degradation types, allowing for compatibility with new tools or degradation types without model retraining. This maintains the high compatibility of the training-free method while improving restoration planning efficiency. Additionally, by using the user preference as one of the query keys in the hierarchical experience pool, the retrieved degradation removal order and tool selection can be dynamically adjusted according to the user's preferences, ensuring the restoration results meet the personalized requirements of users in different application scenarios.

[0023] The image restoration method based on a self-evolving experience pool provided in this application relates to the field of image processing technology. This image restoration method based on a self-evolving experience pool can be applied to a terminal, a server, or software running on a terminal or server. In some embodiments, the terminal can be a smartphone, tablet, laptop, desktop computer, smart speaker, smartwatch, or in-vehicle terminal, but is not limited to these. The server can be configured as an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The server can also be a node server in a blockchain network. The software can be an application implementing the image restoration method based on a self-evolving experience pool, but is not limited to the above forms.

[0024] This application can be used in a wide variety of general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics devices, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices. This application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0025] like Figure 1 The diagram shown is a schematic representation of an implementation environment provided in an embodiment of this application. (Refer to...) Figure 1 The implementation environment includes at least one terminal 102 and a server 101. The terminal 102 and the server 101 can be connected via a network, either wirelessly or via a wired connection, to complete data transmission and exchange.

[0026] Server 101 can be a standalone physical server, a server cluster or distributed system consisting of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms.

[0027] Additionally, server 101 can also be a node server in a blockchain network. Blockchain is a novel application model of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanisms, and encryption algorithms.

[0028] Terminal 102 can be a smartphone, tablet, laptop, desktop computer, smart speaker, smartwatch, etc. It can also be a vehicle-mounted terminal of the various device types described above, but is not limited to these. Terminal 102 and server 101 can be directly or indirectly connected via wired or wireless communication, and this embodiment does not impose any limitations.

[0029] For example, based on Figure 1 The implementation environment shown in this application embodiment provides an image restoration method based on a self-evolving experience pool. The following description uses the application of this image restoration method based on a self-evolving experience pool in server 101 as an example. It can be understood that this method can also be applied to terminal 102.

[0030] Reference Figure 2 , Figure 2 This is a flowchart illustrating an image restoration method based on a self-evolving experience pool applied to a server, provided as an embodiment of this application. The execution subject of this method can be any of the aforementioned computer devices (including servers or terminals). See also... Figure 2 The method may include the following steps: S100. Obtain the degraded image and perform degradation identification on the degraded image to obtain a set of degradation types.

[0031] In this embodiment of the application, a degraded image to be restored is acquired and input into a preset multimodal large language model. The image understanding capability of the preset multimodal large language model is used to identify the degradation type of the degraded image, thereby obtaining a set of degradation types. The set of degradation types includes one or more degradation types such as rain streaks, haze, low light, noise, or blur.

[0032] In the embodiments of this application, the multimodal large language model can be Qwen3.5, GLM5.1, or Llama3.2, etc.

[0033] S200. Based on the set of degradation types and user preference requirements, a target restoration plan is obtained by searching a pre-built hierarchical experience pool using key-value matching. The target restoration plan includes a first degradation removal order of the set of degradation types and a first restoration tool corresponding to each degradation type.

[0034] The hierarchical experience pool includes a first granularity layer (i.e., a coarse-grained layer), a second granularity layer (i.e., a fine-grained layer), and an insight layer.

[0035] It should be noted that the user preference requirements in this embodiment refer to the user's visual quality requirements for the image. Furthermore, this embodiment utilizes the current experience in the hierarchical experience pool for retrieval.

[0036] For example, the steps of obtaining the target restoration plan scheme by retrieving from a pre-built hierarchical experience pool using current experience through key-value matching based on the set of degradation types and user preference requirements include S210-S260: S210. Using the set of degradation types and user preference requirements as the first query key, perform key-value matching retrieval in the first granularity layer. If there is a first key value in the first granularity layer that matches the first query key, perform a significance test on the capability parameter corresponding to the first key value. If the significance test passes, use the degradation order corresponding to the first key value as the first degradation removal order. S220. If the significance test fails, determine the target pattern profile based on the set of degradation types and the user preference requirements. The steps include S221-S224: S221. Obtain multiple pattern profiles pre-stored in the second granularity layer; Each pattern profile Includes a set of degraded image support sets and its corresponding semantic anchor text Its expression is:

[0037] S222. Encode the degraded image and each of the pattern portraits to obtain the image coding features corresponding to the degraded image and the pattern coding features corresponding to each of the pattern portraits; For example, embodiments of this application may employ a CLIP encoder to encode degraded images and to encode images of various modes.

[0038] S223. Calculate the similarity between the image coding features and each of the pattern coding features, and determine a candidate pattern profile set from the multiple pattern profiles based on the similarity. In the specific implementation process, the cosine similarity between the image coding features and each of the pattern coding features is calculated to measure the similarity between them. The formula for calculating the cosine similarity is as follows:

[0039] in, For degraded images, For pattern coding features, Image encoding features, Create a pattern profile.

[0040] Furthermore, based on the calculated cosine similarity, the patterns are sorted, and the top-ranked pattern profiles are selected to form a candidate pattern profile set. S224. Input the semantic anchor text corresponding to each candidate pattern image in the candidate pattern image set and the degraded image into a preset multimodal large language model for matching and evaluation processing to obtain the target pattern image.

[0041] Specifically, the multimodal large language model evaluates the matching degree between the semantic anchor text of each candidate pattern profile and the input degraded image, and comprehensively judges which candidate pattern profile can most accurately describe the degradation mode of the current input degraded image; then, based on the evaluation results of the multimodal large language model, the pattern profile with the highest matching degree with the degraded image is determined from the candidate pattern profile set as the target pattern profile. .

[0042] S230. Using the target pattern profile as the second query key, perform key-value matching retrieval in the second granularity layer. If there is a second key value in the second granularity layer that matches the second query key, use the degradation order corresponding to the second key value as the first degradation removal order. S240. If there is no first key value matching the first query key in the first granularity layer or no second key value matching the second query key in the second granularity layer, the user preference requirement is used as the third query key, and key value matching retrieval is performed in the insight layer to determine the third key value matching the third query key. S250. Determine the first degradation removal order based on the text experience corresponding to the third key value. The steps include S251-S252: S251. Obtain preset guidance prompts; Understandably, guiding prompts are structured cue texts used in the insight layer to guide the large language model in performing degeneracy removal sequence planning.

[0043] S252. Based on the preset guiding prompts, the text experience, and the set of degradation types, perform reasoning and planning to obtain the first degradation removal order.

[0044] Specifically, in this embodiment of the application, the first degeneration removal order can be obtained by inputting the preset guiding prompts, text experience and degeneration type set into the preset large language model for reasoning planning.

[0045] S260. Obtain the first restoration tool corresponding to each degradation type, and obtain the target restoration plan based on the first degradation removal order and the first restoration tool corresponding to each degradation type.

[0046] This application's embodiments automatically select the optimal tool for each degradation factor under the constraint of preference requirements at a coarse-grained hierarchical level. This compresses the original joint optimization search space into a permutation and combination problem centered on degradation type under the constraint of preference requirements, thereby significantly reducing the computational complexity of the planning stage and improving efficiency. In addition, user preferences are introduced as search conditions or planning constraints at each level of retrieval, which can dynamically adjust the degradation removal order and tool selection according to different user preferences, making the restoration results more in line with actual usage needs.

[0047] S300. Following the first degradation removal order, the first restoration tool corresponding to each degradation type is called sequentially to perform restoration operations on the degraded image, thereby obtaining the first restored image.

[0048] S400. Perform a degradation assessment on the first restored image to determine whether each degradation type in the degradation type set has been eliminated.

[0049] In this embodiment, the first restored image is input into a preset multimodal large language model. The image understanding capability of the multimodal large language model is used to perform degradation assessment on the first restored image; that is, the multimodal large language model identifies the degradation type of the first restored image. Specifically, a set of degradation types is used as a list of degradation types to be verified, and each degradation type in the set is evaluated one by one to determine whether it still exists in the first restored image. For example, if the set of degradation types D = {rain removal, low light}, the multimodal large language model is used to evaluate whether rain streaks and low light conditions still exist in the first restored image.

[0050] S500, if each of the degradation types in the set of degradation types has been eliminated, the first restored image is taken as the final restored image.

[0051] Specifically, if the evaluation results indicate that every degradation type in the degradation type set has been eliminated, it means that the target restoration plan currently being implemented has successfully eliminated all degradation types in the degradation image, and the first restoration image has achieved the expected restoration effect. In this case, the first restoration image is output as the final restoration image.

[0052] S600: If at least one of the degradation types in the degradation type set has not been eliminated, the target restoration plan is modified, and the modified restoration plan is used as the target restoration plan. The restoration operation on the degraded image is then performed until each of the degradation types in the degradation type set has been eliminated, and the final restored image is obtained.

[0053] In this embodiment, after performing degradation assessment on the first restored image as described above, if the assessment result indicates that at least one degradation type in the degradation type set has not been eliminated, it means that the currently executed target restoration planning scheme has failed to completely eliminate all degradation types in the degraded image, and the first restored image has not achieved the expected restoration effect. At this time, a rollback operation is triggered (i.e., rollback to the planning scheme stage) to correct the target restoration planning scheme, and the restoration operation is re-executed based on the corrected planning scheme. It should be noted that the rollback operation first attempts to improve the restoration effect by correcting the degradation removal order; if there are still uneliminated degradation types after correcting the order, the restoration tools corresponding to each degradation type are further adjusted. For example, the steps of correcting the target restoration planning scheme, using the corrected restoration planning scheme as the target restoration planning scheme, and returning to perform the restoration operation on the degraded image include S610-S670: S610. Obtain priority experience guidance data from the hierarchical experience pool; Priority-based experience guidance data refers to multiple restoration planning schemes in the hierarchical experience pool, ranked from highest to lowest priority, corresponding to the current set of degradation types and user preference needs. This means that when the target restoration plan needs to be revised, the next-priority restoration planning scheme is retrieved from the hierarchical experience pool. The next-priority scheme refers to the candidate scheme with the second-highest priority among the current search results. For example, if the currently tried scheme is the highest priority scheme, the second-highest priority scheme is retrieved as priority-based experience guidance data; if the second-highest priority scheme still fails, even lower priority schemes are retrieved.

[0054] S620. Based on the priority experience guidance data, the first degradation removal order in the target restoration planning scheme is modified; Specifically, in this application embodiment, the degradation removal order is modified first, without changing the first restoration tool corresponding to each degradation type. That is, the first restoration tool corresponding to each degradation type remains unchanged, and only the first degradation removal order in the target restoration planning scheme is replaced with the degradation removal order in the next priority planning scheme.

[0055] S630. The modified degradation removal order and the first restoration tool corresponding to each degradation type are used as the modified first restoration plan scheme. S640. Using the modified first restoration plan as the target restoration plan, return to the step of performing the restoration operation on the degraded image to obtain the second restored image; Specifically, the modified first restoration plan is taken as the current target restoration plan, and the process returns to step S300. That is, according to the modified degradation removal order, the first restoration tool corresponding to each degradation type is called in sequence to perform restoration operations on the degraded image, and the second restored image can be obtained.

[0056] S650. Perform a degradation assessment on the second restored image. If there are still unresolved degradation types in the degradation type set, modify the first restoration tool corresponding to each degradation type in the target restoration plan based on the priority experience guidance data to obtain the modified restoration tool for each degradation type. Specifically, a degradation assessment is performed on the second restored image to determine whether each degradation type in the degradation type set has been eliminated in the second restored image. If every degradation type in the second restored image has been eliminated, it indicates that the revised planning scheme is effective, and the second restored image is output as the final restored image. If at least one degradation type in the degradation type set remains uneliminated, it indicates that simply revising the degradation removal order is insufficient to eliminate all degradation types. In this case, while maintaining the current revised degradation removal order, the restoration tools corresponding to each degradation type are further revised. Specifically, based on priority-based empirical guidance data, for each degradation type in the degradation type set, the next priority restoration tool is selected from the candidate tool set corresponding to that degradation type, replacing the currently used first restoration tool, thus obtaining the revised restoration tool for each degradation type.

[0057] S660. The modified degradation removal order and the modified restoration tool for each degradation type are used as the modified second restoration plan scheme; S670. Using the modified second restoration plan as the target restoration plan, return to the step of performing the restoration operation on the degraded image until all degradation types are eliminated.

[0058] Specifically, the modified second restoration plan is used as the current target restoration plan, and the process returns to step S300. That is, according to the modified degradation removal order, the modified restoration tool corresponding to each degradation type is called in sequence to perform restoration operations on the degraded image to obtain a new restored image. The degradation of the new restored image is evaluated again to determine whether each degradation type in the degradation type set has been eliminated. The above modification and restoration operation process is repeated. That is, if the modification of the first degradation removal order is ineffective, the restoration tool is further modified until each degradation type in the degradation type set is eliminated. When each degradation type in the degradation type set has been eliminated, the currently obtained restored image is output as the final restored image.

[0059] The embodiments of this application can explore alternative solutions step by step in the manner of first correcting the order and then correcting the tools when the initial target planning scheme is not performing well. It makes full use of the priority experience guidance in the hierarchical experience pool and obtains the restoration result that meets the user's preference needs as much as possible within the limited computational overhead.

[0060] It is understood that steps S100 to S600 above belong to the reasoning stage of the restoration planning scheme. This embodiment of the application also includes an experience acquisition stage. In the reasoning stage, planning schemes are generated only based on existing experience in the hierarchical experience pool, without exploring planning paths. In the experience acquisition stage, all planning paths are explored for the degraded images, generating atomic experience records. The hierarchical experience pool is then updated through an experience evolution mechanism to guide the retrieval of restoration planning schemes for other degraded images, the correction of planning schemes, and the selection of optimal restoration tools. Exemplarily, this embodiment of the application also includes the following steps: (1) Obtain the set of degradation types and the restoration toolset corresponding to the degraded image; In this embodiment of the application, the image restoration intelligent agent system automatically caches degraded images that appear during the inference stage. and its corresponding set of degeneracy types Understandably, this is a set of degenerate types. The set of degradation types identified in step S100, the restoration toolset For use in processing degenerate type collections The set of all candidate tools for each degradation type, for the set of degradation types Each degradation type in There exists a corresponding subset of candidate tools. , Includes features that can be used to eliminate this type of degradation. All candidate restoration tools, for example, if the set of degradation types Then the restoration toolset Includes all candidate tools for rain removal (such as tool A, tool B, and tool C) and all candidate tools for low light (such as tool D and tool E).

[0061] (2) Based on the set of degradation types and the set of restoration tools, determine multiple restoration planning paths corresponding to the degraded image, wherein each restoration planning path includes a second degradation removal order of the set of degradation types and a second restoration tool matched for each degradation type under the second degradation removal order; Specifically, based on the set of degradation types The number of degradation types determines the number of planned paths. :

[0062] in, Represents the set of degeneracy types The number of degenerate types, This represents the total number of candidate tools in the restoration toolkit.

[0063] when When the degraded image contains only one type of degradation, the degradation removal order is unique (there is only one type of degradation, and there is no change in the order). Therefore, the number of planned paths is limited. Equal to the number of candidate tools in the candidate toolset Each planned path includes a second restoration tool. When When a degraded image contains multiple types of degradation, different degradation removal orders will lead to different restoration effects. Therefore, the number of planned paths... Equal to the factorial of the number of degenerate types Each planning path includes a second degradation removal order for the set of degradation types and a second restoration tool matched for each degradation type under the second degradation removal order.

[0064] (3) Using each of the restoration planning paths, perform a restoration operation on the degraded image to obtain a third restored image corresponding to each of the restoration planning paths; Specifically, each using Planned paths for degraded images Perform the restoration operation to obtain These three third-restored images constitute a set of third-restored images:

[0065] in, Indicates according to the first The third restored image obtained after performing the restoration operation on the planned path.

[0066] (4) Based on the image quality evaluation index corresponding to the user preference requirements, evaluate the quality of each of the third restored images, and determine the win rate ranking of each second restoration tool based on the evaluation results; In this embodiment of the application, user preference requirements are obtained. Corresponding image quality assessment index set Among them, the image quality assessment index set It includes at least one image quality assessment metric, which is used to quantitatively evaluate the visual quality of the restored image. Based on the positive or negative correlation between the metric value and image quality, image quality assessment metrics are divided into "higher is better" type metrics. and "the smaller the better" type indicators .

[0067] Furthermore, regarding the result obtained in step (3) Each pair of third-restored images is compared in pairs. For any two third-restored images... and According to the image quality assessment index set Each evaluation indicator A comparison is made. Specifically, for every two third-restored images, an index is calculated. Preference voting Its expression is:

[0068] in, Indicators Below, the third restored image Compared to the third restored image Whether it is judged as superior.

[0069] The rules for determining the value are as follows:

[0070] That is: if the evaluation indicators For the "bigger the better" type and the first The index value of the third restored image is greater than that of the first. The index value of the third restored image, then , indicating in the indicator Below image Superior to image If the evaluation indicators For the "smaller the better" type and the first The index value of the third restored image is less than that of the first. The index value of the third restored image, then ;otherwise .

[0071] Therefore, the win rate between every two third restored images can be derived. And the statistics of wins / losses / draws , and .in, Represents the third restored image Superior Throw to One vote; Represents the third restored image Superior Throw to One vote; This indicates a tie.

[0072] Based on the results of the pairwise comparisons above, the average win rate corresponding to each third restored image is calculated. :

[0073] The average win rate of each third restored image was obtained. Then, all third-party restored images are sorted from highest to lowest according to their average win rate, resulting in a ranking for each third-party restored image. :

[0074] in, This indicates a descending order sorting operation, meaning the higher the average win rate, the higher the ranking.

[0075] Since each third restored image corresponds to a planned path, and each planned path contains a second degradation removal order and a second restoration tool matching each degradation type under that order, the ranking of each third restored image essentially reflects the relative merits of the combination of the second degradation removal order and the second restoration tool corresponding to that path.

[0076] (5) Generate atomic experience records based on the degraded image, the set of degradation types, the set of restoration tools, and the win rate ranking; It should be noted that atomic empirical records are for a specific set of degenerate types. and user preferences and needs Below is a complete record of the exploration results of all planned paths and their ranking in terms of merit. This record includes the following information: a set of degradation types. This is used to identify the degradation scenario to which the experience applies; degradation image. Restoration toolset This is used to identify all candidate tools in this degradation scenario; user preference requirements. The first image is used to identify the preference constraints corresponding to the experience; multiple third-reconstruction images and their corresponding rankings are used to characterize the relative merits of each planning path.

[0077] (6) Store the atomic experience records. When the number of atomic experience records stored in the same degeneration type set reaches the preset batch size, use all atomic experience records in the batch to evolve the experience in the hierarchical experience pool. The evolved hierarchical experience pool is used to guide the retrieval of restoration planning schemes, the correction of planning schemes and the selection of restoration tools for other degeneration images.

[0078] In this embodiment, the atomic experience records generated in step (5) are stored in the experience cache of the image restoration intelligent agent system. In the experience cache, the atomic experience records are organized according to a set of degradation types. Categorized storage, i.e., collections belonging to the same degradation type. All atomic experience records are grouped together; whenever a new atomic experience record is generated and stored in the experience cache, the set of the current degeneracy type in the experience cache is counted. The number of corresponding atomic experience records, when compared with a certain set of degenerate types The corresponding number of atomic experience records reaches the preset batch size. At that time, triggering the set of degradation types The experience-based evolution process. This includes a preset batch size. For a pre-set threshold, for example , or etc.

[0079] In the process of empirical evolution, the set of acquisition and degeneration types is obtained. The corresponding batch (total) (Item) Atomic experience record, within this batch Each atomic experience record contains win / loss / draw statistics for each restored image under multiple planning paths. These statistics are merged with previous counts, and using the merged statistics, a Bradley-Terry-Davidson (BTD) statistical model is introduced to parameterize the win / loss / draw relationship in a probabilistic manner. The expression is as follows: ;

[0080] in, This indicates the strength parameter of a draw. Indicates capability parameters, and They represent the first The first planning scheme, the first The capability parameters represent the restoration performance of each planning scheme. These parameters quantify the relative quality of each scheme; a higher capability parameter value indicates a better restoration effect. The capability parameters can be estimated by maximizing the log-likelihood function through numerical calculation.

[0081] After estimating the capability parameters Then, sort them in descending order to obtain the latest priority function for the coarse-grained layer:

[0082] in, This indicates the number of candidate planning schemes, and the ranking result serves as the priority order of each planning scheme.

[0083] Furthermore, in order to determine whether it is necessary to introduce fine-grained experience, the estimated capability parameters are... Perform a significance test. Specifically, at the significance level... Below, the top-ranked candidate parameters In a unilateral Wald test, the capability gap between the two candidate capability parameters is determined by... A metric is considered significant if it meets the following conditions:

[0084] in, Represents capability parameters The standard error, Indicates significance level The critical value is determined by the following criteria. If this significance condition is met, it indicates a statistically significant difference between the top-ranked candidate parameter and the second-best candidate parameter, meaning the optimal solution has a clear advantage, and there is no need to introduce more fine-grained experience. Otherwise, it indicates that the priority distinction in the coarse-grained layer is insufficient, requiring additional fine-grained empirical modeling. If fine-grained empirical modeling is triggered, then in the fine-grained layer, the experience needs to focus on specific degradation patterns. Specifically, the fine-grained layer follows the pattern profiling described earlier. The definition, namely, each pattern profile It contains a set of degraded image support sets and their semantic anchor text. However, The categories are not predefined, therefore a learning mechanism is needed to dynamically build and optimize pattern profiles. An effective pattern profile... The following two requirements must be met: Semantic consistency Pattern profiling Image group in In semantic space, they should be close to each other to form compact feature representations, which facilitates retrieval by the image encoder; Ranking consistency Mapping to the same pattern profile The fine-grained priority function should be statistically comparable, which ensures the profiling of similar patterns in the experience pool. The fine-grained priority function can be reused.

[0085] Based on the above two consistency requirements, this application presents an agent-driven pattern profiling learning system to fully utilize the understanding capabilities of a multimodal large language model. This pattern profiling learning system leverages the latest... Ranking of in-round cache Learning is performed in mini-batch mode. For each degraded image in the mini-batch... First, a degenerate description is generated using a multimodal large language model. Subsequently, a multi-agent debate paradigm is adopted to partition the current small batches and construct new pattern profiles: ;

[0086] in, The specific roles of each included agent are specified; It encapsulates some predefined bimodal executable operations that can process both image data and text data; This refers to the above-mentioned dual consistency requirement (i.e., semantic consistency). Consistency with ranking This is implemented as the belief (i.e., constraint) of each agent.

[0087] In defining new pattern profiles Then, an iterative process is used to profile the existing patterns. Further optimization and iteration are carried out, with this iterative process driven by an agent to maintain semantic consistency. The soft update, and the Spearman rank correlation coefficient Drivers used to maintain ranking consistency Alternate between hard constraints: ;

[0088] in, This indicates that the intelligent agent has a pattern profile. The operation functions include the following meta-operations: add, replace, update, and delete. Through an iterative process, the agent appends new support set images. and correction mode portrait A consensus is gradually being reached on textual expression.

[0089] After being subject to semantic consistency Consistency with ranking After multiple mini-batch learning steps with regularization constraints, pattern profiling is achieved. The information within tends to stabilize. Therefore, clustering is more effective in profiling specific patterns. The historical cache rankings associated with images in the dataset also converge. Using these stable rankings, we can deduce the bindings to pattern profiles. Final priority function As a priority level under fine-grained layers:

[0090] in, Indicates the first Stable ranking after rounds of iterations This indicates the operation of deriving the final priority function based on a stable ranking.

[0091] Furthermore, in the insight layer, based on the estimated capability parameters The pairwise win / loss / draw relationships can be derived from the aforementioned parametric modeling expressions. After this step, the derived relationships are concatenated into text form and combined with guiding prompts. The textual experience is then extracted through a large language model and used as the latest priority function at the insight granularity.

[0092] Here, Guidance represents the preset guiding prompt; Concat represents the operation of concatenating the win rate and draw relationships into text.

[0093] The embodiments of this application introduce a self-evolutionary mechanism into the experience pool through the above-described experience evolution process. The experience pool is iteratively optimized using accumulated records. This continuous evolutionary capability enables a better Pareto balance between restoration quality and inference efficiency as it is used. Extensive experiments have shown that, thanks to its self-evolutionary experience-driven learning capability, it has successfully achieved results superior to existing methods in multi-degenerate coupled scenarios.

[0094] To explain in detail the principles of the technical solution of this application, the overall process of this application will be described below with reference to some specific embodiments. It is easy to understand that the following is an explanation of the technical principles of this application and should not be regarded as a limitation of this application.

[0095] In a specific embodiment, the image restoration method based on a self-evolving experience pool of this application is applied to the image restoration agent EvoIR-Agent. The image restoration process based on the self-evolving experience pool performed by the image restoration agent EvoIR-Agent includes: like Figure 3 The diagram shown illustrates the inference workflow of the training-free image reconstruction agent at each stage. (See attached image reconstruction agent documentation.) Figure 3 Specifically, given a degraded image In the image restoration agent's perception stage, the image understanding capability of a multimodal large language model is utilized to first identify the degradation type and record it as... Furthermore, based on each degradation type, the planning phase determines the degradation removal sequence. The order in which decisions are made is defined as follows: A permutation of, i.e. Following this plan, the implementation phase utilizes methods targeting specific degradation. A series of restoration tools (defined as) To obtain the restored image The subsequent reflection phase, using the same approach as the perception phase, utilizes a multimodal large language model to evaluate the image. All degradation types on If the image has been effectively eliminated, the process terminates and the image is output; otherwise, a "rollback" phase is triggered. By introducing an experience pool, the image restoration agent can gain robust experience guidance during the planning and rollback phases, thereby promoting the understanding of the image. and Specifically, during the rollback, the image restoration agent system will prioritize correcting the removal order. Then adjust the tool selection To ensure the integrity of the experience-based guidance.

[0096] Furthermore, the steps for developing a restoration plan during the planning phase include: This application embodiment pre-constructs an experience pool to provide experiential guidance for the planning and rollback phases of the image restoration agent. The process of constructing the experience pool is as follows: First, since different tool choices can result in drastically different visual qualities in the restored image, this application's embodiments conceptualize these differences as a preference requirement. Among them, this preference demand This represents a subjective preference interpretation that can be specified by humans and supported by different image quality assessment metrics.

[0097] Based on this, the guidance provided by experience can be mathematically expressed as a priority-driven mapping function. For tool selection, the objective function is:

[0098] in, Representation and degradation factor Corresponding candidate toolsets, It includes historical information from the reasoning phase. The objective function for tool selection can be understood as, given user preferences and needs... Current degradation factors and historical information Under the condition of priority-driven mapping function From candidate toolsets Choose the tool with the highest priority As the optimal tool.

[0099] Furthermore, in more complex degradation coupling scenarios, the degradation removal order... The objective function should be based on the priority-driven mapping function. Combined with relevant restoration tools Together they are established, and their expression is:

[0100] in, This represents the optimal degeneracy removal order; this degeneracy removal order The meaning of the objective function can be understood as, given user preference requirements... Degeneration type set Restoration toolset and historical information Under the condition of priority-driven mapping function Choose the optimal order from all possible degeneration removal orders. .

[0101] Furthermore, regarding the construction of an empirical foundation, with the degeneration type... and restoration tools As the degradation removal order increases, the number of permutations and combinations grows exponentially. Directly modeling based on experience is inefficient and redundant. Therefore, this application proposes to simplify the empirical modeling conditions: considering image restoration in degradation-coupled scenarios, when a preference requirement is specified, the system will utilize the best tool. To explore the order of degradation removal, this process will... Joint optimization of search space compression to meet preference requirements Purely degenerate type set under constraints This problem, centered on permutations and combinations, significantly reduces the computational complexity of the planning phase. Based on this, a hierarchical experience pool is constructed, where the optimal... and The degradation depends not only on the type of degradation but also on the specific degradation mode. Therefore, this application adopts a hierarchical experience pool structure, which can effectively balance information at different granularities. Specifically, the hierarchical experience pool includes the following three levels: Insight Layer: This layer is oriented towards preference requirements and is the highest level. At this level, the priority-driven mapping function is instantiated by the large language model. That is, given the preference requirements and the set of degradation types, the large language model uses the refined textual experience to guide the planning of the degradation removal order.

[0102] Coarse-grained layer: The coarse-grained layer is oriented towards degenerate classes. This layer instantiates the priority-driven mapping function as a direct key-value mapping, using the preference requirements and the set of degenerate types as the query key, and the corresponding key-value is the degenerate removal order.

[0103] Fine-grained layer: The fine-grained layer is geared towards degradation patterns, where a degradation type may cluster into different degradation patterns. This layer also instantiates priority-driven mapping functions as key-value maps, but incorporates retrieval strategies. The core principle is that similar degradation patterns will exhibit similar priority functions.

[0104] Specifically, the fine-grained layer first uses pattern profiling to anchor specific degradation patterns. Each pattern profiling... Defined as:

[0105] in, It is a set of degraded image support sets that characterize this pattern. For the number of images supported in this set, Use it as a semantic anchor.

[0106] During the reasoning process, given the input degraded image... In addition to preference demand Q and degradation type set D, a two-stage cascaded retrieval is performed at the fine-grained layer. The first stage: using the CLIP encoder. Based on cosine similarity, the top-K relevant pattern profiles are recalled. The cosine similarity calculation formula is as follows:

[0107] in, The pattern encoding features corresponding to the pattern profile. These are the image coding features corresponding to the degraded image.

[0108] The second stage involves a multimodal large language model evaluating the semantic anchor text of the Top-K candidate pattern profiles against the input degraded image to identify the pattern profile with the highest matching degree. Ultimately, As a query condition, its corresponding key value is the order of degradation removal.

[0109] It should be noted that the above three levels have a common feature: under the guidance of the coarse-grained level, the system can automatically select the highest priority tool for each degradation factor according to the preference requirements. In other words, regardless of which level the degradation removal order is obtained from, the system selects the optimal tool that matches each degradation type according to the preference requirements.

[0110] This application's embodiments explicitly define that the experiential elements in an image restoration agent should include three aspects: tool selection, degradation removal compliance, and visual quality (i.e., user preference requirements). It also constructs a coarse-to-fine granularity retrieval mode for the experience pool: insight (visual quality), coarse granularity (degradation type), and fine granularity (degradation mode). Furthermore, this application's embodiments introduce an experience evolution mechanism to continuously update the priority-driven mapping function. This experience evolution mechanism specifically includes an experience acquisition stage and an experience evolution stage, such as... Figure 4 As shown, its evolutionary process includes: (a) Experience acquisition stage: The experience acquisition phase is separate from the aforementioned reasoning phase and does not belong to the reasoning phase. In the reasoning phase, the system only generates planning schemes based on existing experience in the hierarchical experience pool, without exploring planning paths; while in the experience acquisition phase, the system explores all planning paths for the degenerate images that appear, generating atomic experience records to guide subsequent evolutionary phases.

[0111] Specifically, the system automatically caches degraded images that appear during the inference phase. For the cached degraded images... The system thoroughly explores all potential planning paths to establish a comprehensive prioritization model, thereby generating a record called the atomic experience record.

[0112] A single atomic experience record contains a degenerate image of a set of degenerate types and its corresponding toolset. It was put into use, thus producing a third set of restored images:

[0113] Among them, the number of planned paths The determination method is as follows:

[0114] That is: when the set of degenerate types When only one degradation type is included, the number of planned paths Equal to candidate toolset The number of candidate tools; when the set of degenerate types When multiple degradation types are included, the number of path combinations It is equal to the factorial of the number of degenerate types.

[0115] Subsequently, based on user preferences, the aforementioned third-restored images were evaluated using image quality assessment metrics. Specifically, for each of these third-restored images, a pairwise comparison was performed on each assessment metric. For any two third-restored images... and Calculation indicators Preference voting :

[0116] in:

[0117] in and These represent evaluation indicator types that respectively indicate "the larger the better" and "the smaller the better"; Represents the third restored image Superior Throw to One vote; Represents the third restored image Superior Throw to One vote; This indicates a tie.

[0118] Based on the above comparison results, the win rate between each pair of restored images can be derived. And the statistics of wins / losses / draws , and .

[0119] After that, calculate the average win rate for each tool:

[0120] And by sorting, their specific ranking is obtained:

[0121] in, This indicates a descending order sorting operation; the higher the average win rate, the higher the ranking.

[0122] Degradation type set, candidate toolset User preferences and needs Multiple third-party restored images and their rankings. Associated storage generates an atomic experience record and stores it in the experience cache.

[0123] (II) The stage of empirical evolution: In the process of empirical evolution, the set of acquisition and degeneration types is obtained. The corresponding batch (total) (Item) Atomic experience record, within this batch Each atomic experience record contains win / loss / draw statistics for each restored image under multiple planning paths. These statistics are merged with previous counts, and using the merged statistics, a Bradley-Terry-Davidson (BTD) statistical model is introduced to parameterize the win / loss / draw relationship in a probabilistic manner. The expression is as follows: ;

[0124] in, This indicates the strength parameter of a draw. Indicates capability parameters, and They represent the first The first planning scheme, the first The capability parameters represent the restoration performance of each planning scheme. These parameters quantify the relative quality of each scheme; a higher capability parameter value indicates a better restoration effect. The capability parameters can be estimated by maximizing the log-likelihood function through numerical calculation.

[0125] After estimating the capability parameters Then, sort them in descending order to obtain the latest priority function for the coarse-grained layer:

[0126] in, This indicates the number of candidate planning schemes, and the ranking result serves as the priority order of each planning scheme.

[0127] Furthermore, in order to determine whether it is necessary to introduce fine-grained experience, the estimated capability parameters are... Perform a significance test. Specifically, at the significance level... Below, the top-ranked candidate parameters In a unilateral Wald test, the capability gap between the two candidate capability parameters is determined by... A metric is considered significant if it meets the following conditions:

[0128] in, Represents capability parameters The standard error, Indicates significance level The critical value is determined by the following criteria. If this significance condition is met, it indicates a statistically significant difference between the top-ranked candidate parameter and the second-best candidate parameter, meaning the optimal solution has a clear advantage, and there is no need to introduce more fine-grained experience. Otherwise, it indicates that the priority distinction in the coarse-grained layer is insufficient, requiring additional fine-grained empirical modeling. If fine-grained empirical modeling is triggered, then in the fine-grained layer, the experience needs to focus on specific degradation patterns. Specifically, the fine-grained layer follows the pattern profiling described earlier. The definition, namely, each pattern profile It contains a set of degraded image support sets and their semantic anchor text. However, The categories are not predefined, therefore a learning mechanism is needed to dynamically build and optimize pattern profiles. An effective pattern profile... The following two requirements must be met: Semantic consistency Pattern profiling Image group in In semantic space, they should be close to each other to form compact feature representations, which facilitates retrieval by the image encoder; Ranking consistency Mapping to the same pattern profile The fine-grained priority function should be statistically comparable, which ensures the profiling of similar patterns in the experience pool. The fine-grained priority function can be reused.

[0129] Based on the above two consistency requirements, this application presents an agent-driven pattern profiling learning system to fully utilize the understanding capabilities of a multimodal large language model. This pattern profiling learning system leverages the latest... Ranking of in-round cache Learning is performed in mini-batch mode. For each degraded image in the mini-batch... First, a degenerate description is generated using a multimodal large language model. Subsequently, a multi-agent debate paradigm is adopted to partition the current small batches and construct new pattern profiles: ;

[0130] in, The specific roles of each included agent are specified, with different agents undertaking different functional responsibilities; It encapsulates some predefined bimodal executable operations that can process both image data and text data; This refers to the above-mentioned dual consistency requirement (i.e., semantic consistency). Consistency with ranking This is implemented as the belief (i.e., constraint) of each agent.

[0131] In defining new pattern profiles Then, an iterative process is used to profile the existing patterns. Further optimization and iteration are carried out, with this iterative process driven by an agent to maintain semantic consistency. The soft update, and the Spearman rank correlation coefficient Drivers used to maintain ranking consistency Alternate between hard constraints: ;

[0132] in, This indicates that the intelligent agent has a pattern profile. The operation functions include the following meta-operations: add, replace, update, and delete. Through an iterative process, the agent appends new support set images. and correction mode portrait A consensus is gradually being reached on textual expression.

[0133] After being subject to semantic consistency Consistency with ranking After multiple mini-batch learning steps with regularization constraints, pattern profiling is achieved. The information within tends to stabilize. Therefore, clustering is more effective in profiling specific patterns. The historical cache rankings associated with images in the dataset also converge. Using these stable rankings, we can deduce the bindings to pattern profiles. Final priority function As a priority level under fine-grained layers:

[0134] in, Indicates the first Stable ranking after rounds of iterations This indicates the operation of deriving the final priority function based on a stable ranking.

[0135] Furthermore, in the insight layer, based on the estimated capability parameters The pairwise win / loss / draw relationships can be derived from the aforementioned parametric modeling expressions. After this step, the derived relationships are concatenated into text form and combined with guiding prompts. The textual experience is then extracted through a large language model and used as the latest priority function at the insight granularity.

[0136] Here, Guidance represents the preset guiding prompt; Concat represents the operation of concatenating the win rate and draw relationships into text.

[0137] This application's embodiments establish a self-evolutionary mechanism for the experience pool, enabling it to better adapt to data sources as the accumulated samples increase, thereby gradually improving model recovery performance and inference efficiency. In summary, this application provides an image restoration method and related device based on a self-evolving experience pool. The method first acquires degraded images and performs degradation identification to obtain a set of degradation types. Based on the set of degradation types and user preference requirements, a target restoration plan is obtained by searching a pre-constructed hierarchical experience pool using key-value matching. The target restoration plan includes a first degradation removal order for the set of degradation types and a first restoration tool corresponding to each degradation type. Following the first degradation removal order, the first restoration tool corresponding to each degradation type is sequentially called to restore the degraded image. Perform a restoration operation to obtain a first restored image; perform a degradation assessment on the first restored image to determine whether each degradation type in the degradation type set has been eliminated; if each degradation type in the degradation type set has been eliminated, use the first restored image as the final restored image; if at least one degradation type in the degradation type set has not been eliminated, modify the target restoration plan and use the modified restoration plan as the target restoration plan, then return to perform a restoration operation on the degraded image until each degradation type in the degradation type set has been eliminated, and obtain the final restored image. This application retrieves the target restoration plan based on a set of degradation types and user preferences within a hierarchical experience pool. This approach eliminates the need for fine-tuning of the large language model, thus avoiding retraining and ensuring good compatibility with new tools or degradation types. Furthermore, by using key-value matching in the hierarchical experience pool, restoration plans can be quickly matched directly to the current set of degradation types, allowing for compatibility with new tools or degradation types without model retraining. This maintains the high compatibility of the training-free method while improving restoration planning efficiency. Additionally, by using the user preference as one of the query keys in the hierarchical experience pool, the retrieved degradation removal order and tool selection can be dynamically adjusted according to the user's preferences, ensuring the restoration results meet the personalized requirements of users in different application scenarios.

[0138] like Figure 5 As shown in the figure, this application embodiment also provides a structural schematic diagram of an image restoration device based on a self-evolving experience pool. This device can implement the above-described method and may include: The degradation identification module 21 is used to acquire a degraded image and perform degradation identification on the degraded image to obtain a set of degradation types; The solution retrieval module 22 is used to retrieve a target restoration plan scheme by searching a pre-built hierarchical experience pool based on the set of degradation types and user preference requirements through key-value matching. The target restoration plan scheme includes a first degradation removal order of the set of degradation types and a first restoration tool corresponding to each degradation type. The restoration operation module 23 is used to sequentially call the first restoration tool corresponding to each degradation type to perform restoration operations on the degraded image according to the first degradation removal order, so as to obtain the first restored image; The degradation assessment module 24 is used to perform degradation assessment on the first restored image to determine whether each degradation type in the degradation type set has been eliminated; The first image restoration module 25 is used to take the first restored image as the final restored image when each of the degradation types in the set of degradation types has been eliminated. The second image restoration module 26 is used to modify the target restoration plan scheme when at least one of the degradation types in the degradation type set has not been eliminated, and to use the modified restoration plan scheme as the target restoration plan scheme, and return to perform restoration operation on the degraded image until each of the degradation types in the degradation type set has been eliminated, so as to obtain the final restored image.

[0139] It is understood that the content of the above method embodiments is applicable to the present device embodiments. The specific functions implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0140] This application also provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the above-described image restoration method based on a self-evolving experience pool. This electronic device can be any smart terminal, including tablet computers, in-vehicle computers, etc.

[0141] It is understood that the content of the above method embodiments is applicable to this device embodiment. The specific functions implemented by this device embodiment are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0142] Please see Figure 6 , Figure 6 The hardware structure of an electronic device according to another embodiment is illustrated. The electronic device includes: The processor 901 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this application. The memory 902 can be implemented as a read-only memory (ROM), static storage device, dynamic storage device, or random access memory (RAM). The memory 902 can store the operating system and other applications. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 902 and is called and executed by the processor 901 using the image restoration method based on a self-evolving experience pool according to the embodiments of this application. The input / output interface 903 is used to implement information input and output; The communication interface 904 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.). Bus 905 transmits information between various components of the device (e.g., processor 901, memory 902, input / output interface 903, and communication interface 904); The processor 901, memory 902, input / output interface 903, and communication interface 904 are connected to each other within the device via bus 905.

[0143] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described image restoration method based on a self-evolving experience pool.

[0144] It is understood that the content of the above method embodiments is applicable to this storage medium embodiment. The specific functions implemented in this storage medium embodiment are the same as those in the above method embodiments, and the beneficial effects achieved are also the same as those achieved in the above method embodiments.

[0145] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0146] The embodiments described in this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided by the embodiments of this application. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems.

[0147] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of this application, and may include more or fewer steps than shown, or combine certain steps, or different steps.

[0148] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0149] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0150] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of the units described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0151] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0152] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0153] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0154] The preferred embodiments of the present application have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims of the present application. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and substance of the embodiments of the present application shall be within the scope of the claims of the present application.

Claims

1. An image restoration method based on a self-evolving experience pool, characterized in that, include: Acquire degraded images and perform degradation identification on the degraded images to obtain a set of degradation types; Based on the set of degradation types and user preference requirements, a target restoration plan is obtained by searching a pre-built hierarchical experience pool through key-value matching. The target restoration plan includes a first degradation removal order of the set of degradation types and a first restoration tool corresponding to each degradation type. According to the first degradation removal order, the first restoration tool corresponding to each degradation type is called sequentially to perform restoration operations on the degraded image to obtain the first restored image; The first restored image is subjected to a degradation assessment to determine whether each degradation type in the set of degradation types has been eliminated; If each of the degradation types in the set of degradation types has been eliminated, the first restored image is taken as the final restored image; If at least one of the degradation types in the set of degradation types has not been eliminated, the target restoration plan is modified, and the modified restoration plan is used as the target restoration plan. The restoration operation on the degraded image is then performed again until each of the degradation types in the set of degradation types has been eliminated, and the final restored image is obtained.

2. The image restoration method based on a self-evolving experience pool according to claim 1, characterized in that, The hierarchical experience pool includes a first granularity layer, a second granularity layer, and an insight layer; The step of retrieving a target restoration plan based on the set of degradation types and user preference requirements by searching a pre-built hierarchical experience pool using key-value matching includes: Using the set of degradation types and user preference requirements as the first query key, key-value matching retrieval is performed in the first granularity layer. If a first key value matching the first query key exists in the first granularity layer, a significance test is performed on the capability parameter corresponding to the first key value. If the significance test passes, the degradation order corresponding to the first key value is used as the first degradation removal order. If the significance test fails, the target pattern profile is determined based on the set of degradation types and the user preference requirements; The target pattern profile is used as the second query key. Key-value matching retrieval is performed in the second granularity layer. If there is a second key value in the second granularity layer that matches the second query key, the degradation order corresponding to the second key value is used as the first degradation removal order. If there is no first key value matching the first query key in the first granularity layer or no second key value matching the second query key in the second granularity layer, the user preference requirement is used as the third query key, and key value matching retrieval is performed in the insight layer to determine the third key value matching the third query key. The first degradation removal order is determined based on the text experience corresponding to the third key value; Obtain the first restoration tool corresponding to each degradation type, and based on the first degradation removal order and the first restoration tool corresponding to each degradation type, obtain the target restoration plan.

3. The image restoration method based on a self-evolving experience pool according to claim 2, characterized in that, The step of determining the target pattern profile based on the set of degradation types and the user preference requirements includes: Obtain multiple pre-stored pattern profiles from the second granularity layer; The degraded image and each of the pattern portraits are encoded to obtain the image encoding features corresponding to the degraded image and the pattern encoding features corresponding to each of the pattern portraits; Calculate the similarity between the image coding features and each of the pattern coding features, and determine a candidate pattern profile set from the multiple pattern profiles based on the similarity. The semantic anchor text corresponding to each candidate pattern profile in the candidate pattern profile set and the degraded image are input into a preset multimodal large language model for matching and evaluation processing to obtain the target pattern profile.

4. The image restoration method based on a self-evolving experience pool according to claim 2, characterized in that, The step of determining the first degradation removal order based on the text experience corresponding to the third key value includes: Retrieve preset prompts; Based on the preset guiding prompts, the textual experience, and the set of degradation types, a reasoning and planning process is performed to obtain the first degradation removal order.

5. The image restoration method based on a self-evolving experience pool according to claim 1, characterized in that, The process of revising the target restoration plan and using the revised restoration plan as the target restoration plan, then returning to perform the restoration operation on the degraded image, includes: Obtain priority experience guidance data from the hierarchical experience pool; Based on the priority experience guidance data, the first degradation removal order in the target restoration planning scheme is modified; The modified degradation removal order and the first restoration tool corresponding to each degradation type are used as the modified first restoration plan scheme; Using the modified first restoration plan as the target restoration plan, return to the step of restoring the degraded image to obtain the second restored image; The second restored image is subjected to degradation assessment. If there are still unresolved degradation types in the set of degradation types, the first restoration tool corresponding to each degradation type in the target restoration plan is modified based on the priority experience guidance data to obtain the modified restoration tool for each degradation type. The modified degradation removal order and the modified restoration tool for each degradation type are used as the modified second restoration plan scheme; The modified second restoration plan is used as the target restoration plan, and the process of restoring the degraded image is repeated until all degradation types are eliminated.

6. The image restoration method based on a self-evolving experience pool according to claim 1, characterized in that, The method further includes: Obtain the set of degradation types and the set of restoration tools corresponding to the degraded image; Based on the set of degradation types and the set of restoration tools, multiple restoration planning paths are determined for the degraded image. When the number of degradation types in the set of degradation types is 1, each restoration planning path includes a second restoration tool corresponding to the degradation type. When the number of degradation types in the set of degradation types is greater than 1, each restoration planning path includes a second degradation removal order of the set of degradation types and a second restoration tool matched for each degradation type under the second degradation removal order. Using each of the restoration planning paths, a restoration operation is performed on the degraded image to obtain a third restored image corresponding to each restoration planning path; Based on the image quality evaluation index corresponding to the user preference requirements, the quality of each of the third restored images is evaluated, and the win rate ranking of each second restoration tool is determined based on the evaluation results. Based on the degraded image, the set of degradation types, the set of restoration tools, and the win rate ranking, an atomic experience record is generated; The atomic experience records are stored. When the number of atomic experience records stored in the same set of degradation types accumulates to a preset batch size, the experience in the hierarchical experience pool is evolved using all the atomic experience records in the batch. The evolved hierarchical experience pool is used to guide the retrieval of restoration planning schemes, the correction of planning schemes, and the selection of optimal restoration tools for other degraded images.

7. An image restoration device based on a self-evolving experience pool, characterized in that, The device includes: A degradation identification module is used to acquire degraded images and perform degradation identification on the degraded images to obtain a set of degradation types; The solution retrieval module is used to retrieve a target restoration plan solution from a pre-built hierarchical experience pool based on the set of degradation types and user preference requirements through key-value matching. The target restoration plan solution includes a first degradation removal order of the set of degradation types and a first restoration tool corresponding to each degradation type. The restoration operation module is used to sequentially call the first restoration tool corresponding to each degradation type to perform restoration operations on the degraded image according to the first degradation removal order, so as to obtain the first restored image; The degradation assessment module is used to perform degradation assessment on the first restored image to determine whether each degradation type in the degradation type set has been eliminated; A first image restoration module is used to take the first restored image as the final restored image when each of the degradation types in the set of degradation types has been eliminated. The second image restoration module is used to modify the target restoration plan scheme when at least one of the degradation types in the degradation type set has not been eliminated, and to use the modified restoration plan scheme as the target restoration plan scheme, and return to perform restoration operation on the degraded image until each of the degradation types in the degradation type set has been eliminated, so as to obtain the final restored image.

8. An electronic device, characterized in that, Including the processor and memory; The memory is used to store programs; The processor executes the program to implement the image restoration method based on a self-evolving experience pool as described in any one of claims 1 to 6.

9. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the image restoration method based on a self-evolving experience pool as described in any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the image restoration method based on a self-evolving experience pool as described in any one of claims 1 to 6.

Images