Image enhancement method, device, server and storage medium
By determining the degradation factors based on the image frequency and selecting corresponding enhancement modules for cascading processing, the problem of poor enhancement effect in the existing technology is solved, and a highly targeted image enhancement effect is achieved.
Patent Information
- Application Number
- CN202111573741.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-12-21
- Publication Date
- 2024-11-12
- Estimated Expiration
- 2041-12-21
AI Technical Summary
The prior art does not have good effect when performing image enhancement processing on low-quality images, and it is difficult to effectively process targeted different degradation factors.
By receiving the image to be enhanced, degradation factors are determined according to its image frequency, and selecting corresponding enhancement modules from the pre-trained enhancement model for processing, using feature information and spatial point coordinates for image enhancement, and using cascade mode of multiple enhancement modules Perform multiple processing to improve image quality.
Targeted enhancement processing for different degradation factors is achieved, which significantly improves the quality and visual effect of the image.
Smart Images

Figure CN114418868B_ABST
Abstract
Description
Technical Field
[0001] The present application belongs to the field of computer technology, and in particular, relates to an image enhancement method, device, server and storage medium. Background Art
[0002] Image enhancement usually aims to emphasize the overall or local features of an image, such as improving the brightness, contrast, resolution, etc. of the image, making an unclear image clear, or highlighting important features in the image to improve the visual effect of the image.
[0003] With the development of computer vision technology, high-quality images have gradually become a demand of people. In real production and life, images are often degraded due to factors such as rain, fog, low brightness, low resolution, high exposure, color cast, etc. Therefore, in related technologies, it is necessary to perform image enhancement processing on low-quality degraded images. Summary of the invention
[0004] The embodiments of the present application provide an image enhancement method, device, server, and storage medium, which can solve the problem in the related art that the effect of image enhancement processing on low-quality degraded images is not good enough.
[0005] A first aspect of an embodiment of the present application provides an image enhancement method, comprising:
[0006] receiving an image to be enhanced, and determining a degradation factor of the image to be enhanced according to an image frequency of the image to be enhanced;
[0007] According to the degradation factor, a target enhancement module corresponding to the image to be enhanced is determined, wherein the target enhancement module belongs to a pre-trained enhancement model, the enhancement model includes a plurality of enhancement modules, each enhancement module corresponds to a degradation factor, and each enhancement module is used to enhance the image with the corresponding degradation factor;
[0008] The target enhancement module performs enhancement processing on the image to be enhanced to obtain an enhanced result image, and outputs the enhanced result image.
[0009] Further, the image to be enhanced is enhanced according to the target enhancement module to obtain an enhanced result image, including:
[0010] Extract feature information of the image to be enhanced and record it as target feature information;
[0011] Determine the spatial point coordinates corresponding to the target feature information according to the mapping relationship between the preset feature information and the spatial point coordinates in the preset space;
[0012] The target feature information and the spatial point coordinates corresponding to the target feature information are input into the enhancement module to obtain an enhanced result image.
[0013] Furthermore, when there are multiple degradation factors, there are multiple target enhancement modules; and
[0014] The image to be enhanced is enhanced according to the target enhancement module to obtain an enhanced result image, including:
[0015] Sort multiple target enhancement modules according to the image frequency corresponding to each target enhancement module to obtain a module sequence;
[0016] The first target enhancement module in the module sequence is used as the target module, the image to be enhanced is used as the target image, and the following enhancement processing operations are performed on the target image:
[0017] Extracting feature information of the target image, determining the spatial point coordinates corresponding to the extracted feature information according to a mapping relationship between the preset feature information and the spatial point coordinates in a preset space, and inputting the extracted feature information and the determined spatial point coordinates into a target module to obtain an enhanced result image;
[0018] The target image is replaced with the obtained result image, and the next target enhancement module in the module sequence is used as the target module. The enhancement processing operation is continued until multiple target enhancement modules in the module sequence are executed, and the obtained result image is determined as the enhanced result image.
[0019] Furthermore, the enhanced model is trained as follows:
[0020] Acquire a test sample set, wherein the test samples in the test sample set include a sample original image, a sample degradation factor, and a sample degraded image obtained by processing the sample original image using the sample degradation factor;
[0021] For each initial enhancement module in the initial enhancement model, according to the degradation factor corresponding to the initial enhancement module, a test sample matching the initial enhancement module is selected from the test sample set, feature information is extracted from the sample degraded image in the selected test sample to obtain sample feature information, and the obtained sample feature information and the sample original image are input into the initial enhancement module to obtain the spatial point coordinates of the input sample feature information and the sample original image;
[0022] According to the spatial point coordinates corresponding to each initial enhancement module, a target distribution is determined, the degree of similarity between the target distribution and the Gaussian distribution is determined according to a preset loss function, and the adjustment state of each initial enhancement module is determined according to the determined degree of similarity, and parameters of the corresponding initial enhancement module are adjusted based on the adjustment state of each initial enhancement module, wherein the target distribution is the distribution of spatial points of at least part of the initial enhancement modules in the initial enhancement model for each test sample, and the adjustment state includes a state that needs to be adjusted and a state that does not need to be adjusted;
[0023] When the adjustment states of the initial enhancement modules in the initial enhancement model are all states that do not require adjustment, the enhancement model is obtained.
[0024] Further, adjusting parameters of the corresponding initial enhancement modules based on the adjustment states of the initial enhancement modules includes:
[0025] For the initial enhancement module whose corresponding adjustment state is a state that does not require adjustment, corresponding training completion indication information is generated, indicating that the initial enhancement module does not need to be tested in the next round of testing.
[0026] Further, adjusting parameters of the corresponding initial enhancement modules based on the adjustment states of the initial enhancement modules includes:
[0027] For the initial enhancement module whose corresponding adjustment state is a state that needs to be adjusted, the parameters of the initial enhancement module are adjusted according to the back propagation algorithm.
[0028] Further, receiving the image to be enhanced, and determining the degradation factor of the image to be enhanced according to the image frequency of the image to be enhanced, including:
[0029] The image frequency of the image to be enhanced is input into a pre-trained classification model to obtain a degradation factor corresponding to the image frequency, wherein the classification model is used to characterize the corresponding relationship between the image frequency and the degradation factor.
[0030] A second aspect of an embodiment of the present application provides an image enhancement device, including:
[0031] An image receiving unit, used for receiving the image to be enhanced, and determining a degradation factor of the image to be enhanced according to an image frequency of the image to be enhanced;
[0032] A module determination unit, used to determine a target enhancement module corresponding to the image to be enhanced according to the degradation factor, wherein the target enhancement module belongs to a pre-trained enhancement model, the enhancement model includes a plurality of enhancement modules, each enhancement module corresponds to a degradation factor, and each enhancement module is used to perform enhancement processing on the image having the corresponding degradation factor;
[0033] The image enhancement unit is used to perform enhancement processing on the image to be enhanced according to the target enhancement module to obtain an enhanced result image, and output the enhanced result image.
[0034] A third aspect of an embodiment of the present application provides a server, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, each step of the image enhancement method provided in the first aspect is implemented.
[0035] A fourth aspect of an embodiment of the present application provides a storage medium, which stores a computer program. When the computer program is executed by a processor, each step of the image enhancement method provided in the first aspect is implemented.
[0036] The image enhancement method, device, server and storage medium provided by the embodiments of the present application have the following beneficial effects: when enhancing an image, since images with different degradation factors have different corresponding image frequencies, the image degradation factor can be determined according to the image frequency. In addition, the corresponding enhancement module is determined from the enhancement model according to the degradation factor, and the image is enhanced by the enhancement module, so that the image can be processed by the enhancement module corresponding to the degradation factor, which is more targeted and helps to improve the image enhancement effect. BRIEF DESCRIPTION OF THE DRAWINGS
[0037] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0038] Figure 1 is a flowchart of an implementation of an image enhancement method provided by an embodiment of the present application;
[0039] Figure 2 is a flowchart of another image enhancement method provided by an embodiment of the present application;
[0040] Figure 3 is a flowchart of an implementation method of an image enhancement method having multiple degradation factors provided by an embodiment of the present application;
[0041] Figure 4 is a flowchart of an implementation of an image enhancement method with four degradation factors provided by an embodiment of the present application;
[0042] Figure 5 is a flowchart for implementing a training enhancement model provided by an embodiment of the present application;
[0043] Figure 6 is a structural block diagram of an image enhancement device provided by an embodiment of the present application;
[0044] Figure 7 It is a structural block diagram of a server provided in an embodiment of the present application. DETAILED DESCRIPTION
[0045] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application may also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present application.
[0046] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, wholes, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or combinations thereof.
[0047] As used in the specification and appended claims of this application, the term "if" can be interpreted as "when" or "uponce" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrase "if it is determined" or "if [described condition or event] is detected" can be interpreted as meaning "uponce it is determined" or "in response to determining" or "uponce [described condition or event] is detected" or "in response to detecting [described condition or event]", depending on the context.
[0048] References to "one embodiment" or "some embodiments" etc. described in the specification of this application mean that one or more embodiments of the present application include specific features, structures or characteristics described in conjunction with the embodiment. Therefore, the statements "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. that appear in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways.
[0049] In order to illustrate the technical solution of the present application, the following embodiments are used for illustration.
[0050] See also Figure 1 , Figure 1 A flowchart of an image enhancement method according to an embodiment of the present application is shown, including:
[0051] Step 101: receiving an image to be enhanced, and determining a degradation factor of the image to be enhanced according to an image frequency of the image to be enhanced.
[0052] The image to be enhanced is usually a low-quality image whose quality is reduced due to degradation factors, for example, a low-brightness image with dim light.
[0053] Among them, image frequency is usually an indicator that represents the intensity of grayscale changes in an image, and is the gradient of grayscale in a plane space.
[0054] Among them, degradation factors are factors that cause image degradation. In reality, image degradation factors usually include: rain, fog, low brightness, low resolution, high exposure, color cast, etc.
[0055] In this embodiment, the execution subject of the above-mentioned image enhancement method is usually a server. It should be noted that the server can be hardware or software. When the server is hardware, it can be implemented as a distributed server cluster consisting of multiple servers, or it can be implemented as a single server. When the server is software, it can be implemented as multiple software or software modules, or it can be implemented as a single software or software module. No specific limitation is made here.
[0056] In practice, the above execution subject may receive the image to be enhanced from a terminal device in communication connection, for example, may receive the image to be enhanced scanned by a scanning device from a scanning device.
[0057] Since the image frequencies corresponding to low-quality images with different degradation factors are usually different, after obtaining the image to be enhanced, the execution subject may use the image frequency of the image to be enhanced to determine the degradation factor corresponding to the image to be enhanced. For example, the execution subject may use the image frequency to find the degradation factor corresponding to the image frequency of the image to be enhanced from a pre-established image frequency-degradation factor correspondence table. The image frequency-degradation factor correspondence table may be a correspondence table pre-established by the execution subject and storing a plurality of correspondences between image frequencies and degradation factors.
[0058] Step 102: determining a target enhancement module corresponding to the image to be enhanced according to the degradation factor.
[0059] The target enhancement module belongs to a pre-trained enhancement model, and the enhancement model includes a plurality of enhancement modules, each enhancement module corresponds to a degradation factor, and each enhancement module is used to perform enhancement processing on an image having a corresponding degradation factor.
[0060] In practice, the target enhancement module corresponding to the image to be enhanced is determined according to the degradation factors. Usually, the target enhancement module is determined by the degradation factors based on the corresponding relationship between the degradation factors and the enhancement modules. For example, a low-quality image with rain can correspond to a deraining enhancement module, and a low-quality image with low brightness can correspond to a brightening enhancement module.
[0061] Step 103: Perform enhancement processing on the image to be enhanced according to the target enhancement module to obtain an enhanced result image, and output the enhanced result image.
[0062] Among them, the enhanced result image is usually an image obtained after being processed by the target enhancement module.
[0063] Among them, the target enhancement module performs enhancement processing on the image to be enhanced, and usually the target enhancement module performs image enhancement processing on the part of the image to be enhanced where the image quality is reduced due to the corresponding degradation factor. For example, the rain removal enhancement module processes the part of the low-quality image with rain where the image quality is reduced due to the degradation factor rain. It should be pointed out that by enhancing the part of the image degraded due to the degradation factor rain, the influence of the degradation factor rain on the image can be removed, thereby improving the image quality.
[0064] The enhanced result image is obtained and the enhanced result image is outputted, usually, the image to be enhanced is enhanced by the target enhancement module and then the obtained image is outputted as the enhanced result image.
[0065] The image enhancement method provided in this embodiment, in the process of enhancing the image to be enhanced, according to the different image frequencies corresponding to different degradation factors, the corresponding degradation factor is determined by the frequency of the image to be enhanced, and then the corresponding enhancement module is determined by the degradation factor. The enhancement module performs image enhancement processing on the image to be enhanced, removes the influence of the degradation factor on the image to be enhanced, obtains the enhanced image and outputs it, and selects a suitable enhancement module to enhance the image according to the degradation factor of the image, which helps to improve the image enhancement effect.
[0066] See also Figure 2 , Figure 2 A flowchart of another image enhancement method provided by an embodiment of the present application is shown, including:
[0067] Step 201: receiving an image to be enhanced, and determining a degradation factor of the image to be enhanced according to an image frequency of the image to be enhanced.
[0068] Step 202: Determine a target enhancement module corresponding to the image to be enhanced according to the degradation factor.
[0069] In this embodiment, the specific operations of steps 201-202 are the same as Figure 1The operations of steps 101 - 102 in the illustrated embodiment are substantially the same and will not be described in detail herein.
[0070] Step 203: extract feature information of the image to be enhanced, and record it as target feature information.
[0071] The feature information of the image to be enhanced is usually information used to describe the features of the image to be enhanced. In practice, the feature information can usually describe the degradation factors corresponding to the image to be enhanced.
[0072] In practice, the feature information of the image to be enhanced can be extracted by inputting the image to be enhanced into an autoencoder, and extracting the feature information in the image to be enhanced through the autoencoder. Here, in order to facilitate the distinguishing description, the extracted feature information is recorded as target feature information.
[0073] It should be pointed out that an autoencoder is an artificial neural network that can learn an efficient representation of input data through unsupervised learning. This efficient representation of the input data is called encodings, and its dimension is generally much smaller than the input data, making the autoencoder useful for dimensionality reduction. More importantly, autoencoders can be used as powerful feature detectors for pre-training of deep neural networks.
[0074] Step 204 , determining the spatial point coordinates corresponding to the target feature information according to the mapping relationship between the preset feature information and the spatial point coordinates in the preset space.
[0075] Here, different feature information usually corresponds to different spatial points. After obtaining the feature information of the image to be enhanced, the above-mentioned execution subject can determine the spatial point in the preset space corresponding to the feature information based on the correspondence between the feature information and the spatial point coordinates in the preset space. For example, the above-mentioned execution subject can use the feature information to find the spatial point coordinates corresponding to the feature information of the image to be enhanced from a pre-established feature information-spatial point coordinate correspondence table. Among them, the above-mentioned feature information-spatial point coordinate correspondence table can be a correspondence table pre-established by the execution subject, storing the correspondence between multiple feature information and spatial point coordinates. In practice, Gaussian space can be used as a preset space, multiple feature information can be mapped to Gaussian space, and a correspondence table of the correspondence between multiple feature information and spatial point coordinates in Gaussian space can be established.
[0076] Step 205: input the target feature information and the spatial point coordinates corresponding to the target feature information into an enhancement module to obtain an enhanced result image, and output the enhanced result image.
[0077] In practice, target feature information and the spatial point coordinates corresponding to the target feature information are usually input into an enhancement module. The enhancement module enhances the image according to the received target feature information and the spatial point coordinates in a preset space to obtain an enhanced result image and output the enhanced result image.
[0078] The image enhancement method provided in this embodiment, in the process of enhancing the image to be enhanced, determines the corresponding spatial point coordinates according to the mapping relationship between the feature information of the image to be enhanced and the spatial point coordinates in a preset space, and then inputs the feature information and the spatial point coordinates into the enhancement module at the same time. The enhancement module enhances the image to be enhanced according to the received feature information and spatial point coordinates.
[0079] See also Figure 3 , Figure 3 The flowchart of an implementation method of an image enhancement method with multiple degradation factors provided by an embodiment of the present application is shown, including:
[0080] Step 301: receiving an image to be enhanced, and determining a degradation factor of the image to be enhanced according to an image frequency of the image to be enhanced.
[0081] Step 302: Determine a target enhancement module corresponding to the image to be enhanced according to the degradation factor.
[0082] In this embodiment, the specific operations of steps 301-302 are the same as Figure 1 The operations of steps 101 - 102 in the illustrated embodiment are substantially the same and will not be described in detail herein.
[0083] Step 303: sort the multiple target enhancement modules according to the image frequency corresponding to each target enhancement module to obtain a module sequence.
[0084] The module sequence is usually a sequence composed of identifiers of multiple target enhancement modules, and the identifier of the target enhancement module may be information for identifying the target enhancement module, such as the identifier of the target enhancement module may be "001", indicating a rain removal enhancement module, or "defogging enhancement module", indicating a defogging enhancement module. As an example, the obtained module sequence may be: "rain removal enhancement module-defogging enhancement module-super-resolution enhancement module", indicating that the target enhancement modules involved are, in sequence: rain removal enhancement module, defogging enhancement module, and super-resolution enhancement module.
[0085] In practice, the execution subject may sort the identifiers of multiple target enhancement modules in descending order of the image frequencies corresponding to the target modules. For example, if the image frequency of the rain removal enhancement module is f1, the image frequency of the fog removal enhancement module is f2, the image frequency of the brightening enhancement module is f3, and the image frequency of the super-resolution enhancement module is f4, and f1>f2>f3>f4, then sort the identifiers of the multiple target enhancement modules, and the resulting module sequence may be: "rain removal enhancement module-fog removal enhancement module-brightening enhancement module-super-resolution enhancement module".
[0086] Step 304, taking the first target enhancement module in the module sequence as the target module, taking the image to be enhanced as the target image, extracting feature information of the target image, determining the spatial point coordinates corresponding to the extracted feature information according to the mapping relationship between the preset feature information and the spatial point coordinates in the preset space, and inputting the extracted feature information and the determined spatial point coordinates into the target module to obtain an enhanced result image.
[0087] In this embodiment, the target module corresponds to Figure 2 In the target enhancement module of the embodiment shown, the target image corresponds to Figure 2 The image to be enhanced in the embodiment shown. The basic operations of extracting feature information of the target image, determining the spatial point coordinates corresponding to the extracted feature information according to the mapping relationship between the preset feature information and the spatial point coordinates in the preset space, and inputting the extracted feature information and the determined spatial point coordinates into the target module to obtain the enhanced result image are as follows: Figure 2 The operations of steps 203-205 in the illustrated embodiment are substantially the same and will not be described in detail herein.
[0088] Step 305, replace the target image with the obtained result image, take the next target enhancement module in the module sequence as the target module, continue to perform the enhancement processing operation until multiple target enhancement modules in the module sequence are executed, determine the obtained result image as the enhanced result image, and output the enhanced result image.
[0089] Here, the multiple target enhancement modules indicated by the above module sequence are usually combined in a cascade manner. The target image is replaced with the obtained result image, and the next target enhancement module in the module sequence is used as the target module to continue to perform the enhancement processing operation. Usually, the result image processed by the target enhancement module of the previous level is input into the target enhancement module of the next level to continue the image enhancement processing. For example, for the three-level module sequence of rain removal enhancement module-defogging enhancement module-super-resolution enhancement module, the result image processed by the rain removal enhancement module is used as the target image and input into the defogging enhancement module, and the defogging enhancement module performs further enhancement processing operations on the image.
[0090] Among them, until multiple target enhancement modules in the module sequence are executed, the obtained result image is determined as the enhanced result image, usually the image to be enhanced is input into the first-level enhancement module in the module sequence, the first-level enhancement module performs enhancement processing operation on the image to be enhanced, and then the result image enhanced by the first-level enhancement module is input into the next-level enhancement module in the module sequence, and the next-level enhancement module continues to perform enhancement processing operation on the image until the image to be enhanced is enhanced by all enhancement modules in the corresponding module sequence, and the result image enhanced by the last-level enhancement module in the module sequence is used as the enhanced result image, and the enhanced result image is output.
[0091] To give a further example, for the three-level module sequence of "rain removal enhancement module-defogging enhancement module-super-resolution enhancement module", the target image is first input into the first-level rain removal enhancement module, and the rain removal enhancement module performs rain removal enhancement processing on the target image. Then, the result image after the rain removal enhancement processing is used as the target image and input into the next-level defogging enhancement module, and the defogging enhancement module performs defogging enhancement processing on the target image. After that, the result image after the defogging enhancement processing is used as the target image and input into the next-level super-resolution enhancement module, and the super-resolution enhancement module performs resolution enhancement processing on the image, uses the result image after the resolution enhancement processing as the enhanced result image, and outputs the enhanced result image.
[0092] As an example, see Figure 4 , Figure 4A flowchart for implementing an image enhancement method with four degradation factors provided by an embodiment of the present application is shown. The image to be enhanced includes four degradation factors: rain, fog, low brightness, and low resolution. The corresponding four-level module sequence is "rain removal enhancement module-fog removal enhancement module-brightening enhancement module-super-resolution enhancement module". First, extract rain feature information from the image to be enhanced (rain, fog, low brightness, low resolution), determine the spatial point g1 corresponding to the rain feature information according to the mapping relationship between the preset feature information and the spatial point coordinates in the preset Gaussian space, and then input the rain feature information and the spatial point g1 into the rain removal enhancement module, and the rain removal enhancement module performs enhancement processing on the image to be enhanced (rain, fog, low brightness, low resolution) to remove the influence of the degradation factor rain on the image, and obtain the image to be enhanced (fog, low brightness, low resolution) after the rain removal enhancement processing; extract fog feature information from the image to be enhanced (fog, low brightness, low resolution), determine the spatial point g2 corresponding to the fog feature information according to the mapping relationship between the preset feature information and the spatial point coordinates in the preset Gaussian space, and then input the fog feature information and the spatial point g2 into the defogging enhancement module, and the defogging enhancement module performs enhancement processing on the image to be enhanced (fog, low brightness, low resolution) to remove the influence of the degradation factor fog on the image, and obtain the image to be enhanced (low brightness) after the defogging enhancement processing. , low resolution); extract low brightness feature information from the image to be enhanced (low brightness, low resolution), determine the spatial point g3 corresponding to the low brightness feature information according to the mapping relationship between the preset feature information and the spatial point coordinates in the preset Gaussian space, and then input the low brightness feature information and the spatial point g3 into the brightening enhancement module, and the brightening enhancement module performs enhancement processing on the image to be enhanced (low brightness, low resolution) to remove the influence of the degradation factor low brightness on the image, and obtain the image to be enhanced (low resolution) after brightness enhancement processing; extract low resolution feature information from the image to be enhanced (low resolution), determine the spatial point g4 corresponding to the low resolution feature information according to the mapping relationship between the preset feature information and the spatial point coordinates in the preset Gaussian space, and then input the low resolution feature information and the spatial point g4 into the super-resolution enhancement module, and the super-resolution enhancement module performs enhancement processing on the image to be enhanced (low resolution) to remove the influence of the degradation factor low resolution on the image, and obtain the enhanced result image after resolution enhancement processing.
[0093] See also Figure 5 , Figure 5 The following is a flowchart of an implementation of training an enhanced model provided by an embodiment of the present application, including:
[0094] Step 501: Obtain a test sample set.
[0095] The test samples in the test sample set include sample original images, sample degradation factors, and sample degraded images obtained by processing the sample original images with the sample degradation factors.
[0096] Here, the execution subject may obtain the test sample set locally or from other electronic devices connected by communication.
[0097] Step 502, for each initial enhancement module in the initial enhancement model, according to the degradation factor corresponding to the initial enhancement module, select a test sample matching the initial enhancement module from the test sample set, extract feature information of the sample degraded image in the selected test sample to obtain sample feature information, and input the obtained sample feature information and the sample original image into the initial enhancement module to obtain the spatial point coordinates of the input sample feature information and the sample original image.
[0098] Since each enhancement module corresponds to a degradation factor, for example, the degradation factor corresponding to the rain removal enhancement module is rain, and the degradation factor corresponding to the brightening enhancement module is low brightness. Therefore, for each initial enhancement module, the execution subject can use the degradation factor corresponding to the initial enhancement module, and select training samples whose corresponding sample degradation factors are the same as the degradation factors corresponding to the initial enhancement module from the training sample set as test samples corresponding to the initial enhancement module.
[0099] In practice, for each initial enhancement module, after selecting a test sample of the initial enhancement module, the execution entity may extract feature information from the sample degraded image in the selected test sample to obtain sample feature information. For example, the sample degraded image may be input into an autoencoder, and the autoencoder may extract feature information from the sample degraded image to obtain the sample feature information.
[0100] For each initial enhancement module, after obtaining the sample feature information of the initial enhancement module, the above-mentioned execution entity can input the obtained sample feature information and the sample original image into the initial enhancement module to obtain the spatial point coordinates of the input sample feature information and the sample original image.
[0101] Step 503, determine the target distribution according to the spatial point coordinates corresponding to each initial enhancement module, determine the similarity between the target distribution and the Gaussian distribution according to a preset loss function, and determine the adjustment state of each initial enhancement module according to the determined similarity, and adjust the parameters of the corresponding initial enhancement module based on the adjustment state of each initial enhancement module.
[0102] The target distribution is the distribution of the spatial points of at least part of the initial enhancement modules in the initial enhancement model for each test sample, and the adjustment state includes a state that needs adjustment and a state that does not need adjustment.
[0103] The target distribution is determined according to the spatial point coordinates corresponding to each initial enhancement module, and the spatial point coordinates corresponding to each initial enhancement module may be fitted into a curve, and then the target distribution of the spatial point is obtained according to the fitted curve.
[0104] In practice, the loss function is usually determined by the maximum likelihood estimation method, and then the similarity between the target distribution and the Gaussian distribution is determined based on the determined loss function. The similarity can be expressed by a similarity parameter, and the value of the similarity parameter is usually greater than 0 and less than 1.
[0105] According to the determined similarity, the adjustment state of each initial enhancement module is determined by comparing a similarity parameter used to represent the similarity between the target distribution and the Gaussian distribution with a preset similarity threshold, and the value of the similarity threshold is usually greater than 0 and less than 1. If the similarity parameter is less than the similarity threshold, the adjustment state of the initial enhancement module is a state that needs to be adjusted. If the similarity parameter is greater than or equal to the similarity threshold, the adjustment state of the initial enhancement module is a state that does not need to be adjusted.
[0106] In practice, the parameters of the corresponding initial enhancement modules are adjusted based on the adjustment status of each initial enhancement module. Usually, when the target distribution does not satisfy the Gaussian distribution, the parameters of the initial enhancement module are adjusted to adjust the distribution of the spatial points corresponding to the initial enhancement module so that the target distribution satisfies the Gaussian distribution.
[0107] Step 504: When the adjustment status of each initial enhancement module in the initial enhancement model is a status that does not require adjustment, an enhancement model is obtained.
[0108] In practice, when the adjustment states of the initial enhancement modules in the initial enhancement model are all in a state where no adjustment is required, the similarity parameter between the target distribution of the spatial points corresponding to the initial enhancement model and the Gaussian distribution is greater than or equal to the preset similarity threshold, that is, the target distribution conforms to the Gaussian distribution, and the enhancement model is obtained. Among them, the spatial points corresponding to the initial enhancement modules correspond to different regions on the Gaussian distribution, and the distribution of the spatial points corresponding to the initial enhancement modules together constitutes the Gaussian distribution of the initial enhancement model.
[0109] In practice, each enhancement module in the enhancement model is reversible, that is, feature information and the original image can be input and the corresponding spatial point coordinates can be output, or feature information and spatial point coordinates can be input and the corresponding original image can be output.
[0110] In some optional implementations of this embodiment, adjusting parameters of corresponding initial enhancement modules based on adjustment states of respective initial enhancement modules includes:
[0111] For the initial enhancement module whose corresponding adjustment state is a state that does not require adjustment, corresponding training completion indication information is generated, indicating that the initial enhancement module does not need to be tested in the next round of testing.
[0112] In practice, the distribution of the area to which the spatial points corresponding to the initial enhancement module corresponding to the state that does not require adjustment belongs conforms to the distribution corresponding to the area in the Gaussian distribution. The initial enhancement module is a trained enhancement module and can generate indication information that the training is completed. There is no need to test the initial enhancement module in the next round of testing, thereby improving the efficiency of the initial enhancement model training.
[0113] In some optional implementations of this embodiment, adjusting parameters of corresponding initial enhancement modules based on adjustment states of respective initial enhancement modules includes:
[0114] For the initial enhancement module whose corresponding adjustment state is a state that needs to be adjusted, the parameters of the initial enhancement module are adjusted according to the back propagation algorithm.
[0115] Here, the parameters of the initial enhancement module are adjusted according to the back propagation algorithm, which usually means that the target distribution must conform to the Gaussian distribution. The parameters of the initial enhancement module corresponding to the area that does not conform to the Gaussian distribution are adjusted through the back propagation algorithm, so that the distribution of the spatial points corresponding to the initial enhancement module conforms to the Gaussian distribution.
[0116] In some optional implementations of this embodiment, receiving an image to be enhanced, and determining a degradation factor of the image to be enhanced according to an image frequency of the image to be enhanced includes:
[0117] The image frequency of the image to be enhanced is input into a pre-trained classification model to obtain a degradation factor corresponding to the image frequency.
[0118] The classification model is used to characterize the correspondence between the image frequency and the degradation factor. Here, the classification model can be a model obtained by training an initial model (such as a convolutional neural network (CNN), a residual network (ResNet), etc.) based on the training samples using a machine learning method.
[0119] In some optional implementations of this embodiment, the above classification model is generated through the following steps:
[0120] The first step is to obtain a sample information set, where the sample information in the sample information set includes sample frequency information and a sample degradation factor corresponding to the sample frequency information.
[0121] Here, the execution subject may obtain the sample information set locally or from other electronic devices connected by communication.
[0122] In the second step, the sample frequency information in the sample information set is taken as input, and the sample degradation factor corresponding to the input sample frequency information is taken as the expected output to train and obtain the classification model.
[0123] Here, the above-mentioned execution subject can input the sample frequency information included in each sample information in the sample information set into the initial classification model to obtain the actual output corresponding to each input sample frequency information, and then compare the actual output corresponding to each sample frequency information with the expected output corresponding to the corresponding sample frequency information for consistency. If the two are consistent, the classification is considered accurate, otherwise, the classification is considered inaccurate. In this way, the above-mentioned execution subject can obtain the accuracy of training the initial classification model. If the accuracy is greater than the preset accuracy threshold, the initial classification model is used as the above-mentioned classification model. Among them, the accuracy threshold is usually a pre-set value, and the value of the accuracy threshold is usually greater than 0 and less than 1. The initial classification model can be various neural network models, such as a convolutional neural network model, a recurrent neural network model, a support vector machine model, etc.
[0124] See also Figure 6 , Figure 6 : is a structural block diagram of a message sending device provided in an embodiment of the present application. The message sending device 600 of this embodiment includes:
[0125] The image receiving unit 601 is used to receive the image to be enhanced, and determine the degradation factor of the image to be enhanced according to the image frequency of the image to be enhanced.
[0126] The module determination unit 602 is used to determine the target enhancement module corresponding to the image to be enhanced according to the degradation factor, wherein the target enhancement module belongs to a pre-trained enhancement model, the enhancement model includes multiple enhancement modules, each enhancement module corresponds to the degradation factor, and each enhancement module is used to enhance the image with the corresponding degradation factor.
[0127] The image enhancement unit 603 is used to perform enhancement processing on the image to be enhanced according to the target enhancement module to obtain an enhanced result image, and output the enhanced result image.
[0128] As an embodiment of the present application, the device further includes a feature extraction unit (not shown in the figure). The feature extraction unit is used to:
[0129] Extract feature information of the image to be enhanced and record it as target feature information;
[0130] Determine the spatial point coordinates corresponding to the target feature information according to the mapping relationship between the preset feature information and the spatial point coordinates in the preset space;
[0131] The target feature information and the spatial point coordinates corresponding to the target feature information are input into the enhancement module to obtain an enhanced result image.
[0132] As an embodiment of the present application, the device further includes an enhancement processing unit (not shown in the figure). The enhancement processing unit is used to:
[0133] Sort multiple target enhancement modules according to the image frequency corresponding to each target enhancement module to obtain a module sequence;
[0134] The first target enhancement module in the module sequence is used as the target module, the image to be enhanced is used as the target image, and the following enhancement processing operations are performed on the target image:
[0135] Extracting feature information of the target image, determining the spatial point coordinates corresponding to the extracted feature information according to a mapping relationship between the preset feature information and the spatial point coordinates in a preset space, and inputting the extracted feature information and the determined spatial point coordinates into a target module to obtain an enhanced result image;
[0136] The target image is replaced with the obtained result image, and the next target enhancement module in the module sequence is used as the target module. The enhancement processing operation is continued until multiple target enhancement modules in the module sequence are executed, and the obtained result image is determined as the enhanced result image.
[0137] As an embodiment of the present application, the device further includes a model training unit (not shown in the figure). The model training unit is used to:
[0138] Acquire a test sample set, wherein the test samples in the test sample set include a sample original image, a sample degradation factor, and a sample degraded image obtained by processing the sample original image using the sample degradation factor;
[0139] For each initial enhancement module in the initial enhancement model, according to the degradation factor corresponding to the initial enhancement module, a test sample matching the initial enhancement module is selected from the test sample set, feature information is extracted from the sample degraded image in the selected test sample to obtain sample feature information, and the obtained sample feature information and the sample original image are input into the initial enhancement module to obtain the spatial point coordinates of the input sample feature information and the sample original image;
[0140] According to the spatial point coordinates corresponding to each initial enhancement module, a target distribution is determined, the degree of similarity between the target distribution and the Gaussian distribution is determined according to a preset loss function, and the adjustment state of each initial enhancement module is determined according to the determined degree of similarity, and parameters of the corresponding initial enhancement module are adjusted based on the adjustment state of each initial enhancement module, wherein the target distribution is the distribution of spatial points of at least part of the initial enhancement modules in the initial enhancement model for each test sample, and the adjustment state includes a state that needs to be adjusted and a state that does not need to be adjusted;
[0141] When the adjustment states of the initial enhancement modules in the initial enhancement model are all states that do not require adjustment, the enhancement model is obtained.
[0142] As an embodiment of the present application, the device further includes an information generation unit (not shown in the figure). The information generation unit is used to:
[0143] For the initial enhancement module whose corresponding adjustment state is a state that does not require adjustment, corresponding training completion indication information is generated, indicating that the initial enhancement module does not need to be tested in the next round of testing.
[0144] As an embodiment of the present application, the device further includes a parameter adjustment unit (not shown in the figure). The parameter adjustment unit is used to:
[0145] For the initial enhancement module whose corresponding adjustment state is a state that needs to be adjusted, the parameters of the initial enhancement module are adjusted according to the back propagation algorithm.
[0146] As an embodiment of the present application, the device further includes a frequency classification unit (not shown in the figure). The frequency classification unit is used to:
[0147] The image frequency of the image to be enhanced is input into a pre-trained classification model to obtain a degradation factor corresponding to the image frequency, wherein the classification model is used to characterize the corresponding relationship between the image frequency and the degradation factor.
[0148] In the device provided in this embodiment, when performing image enhancement processing on an image, since images with different degradation factors correspond to different image frequencies, the image degradation factor can be determined according to the image frequency. In addition, a corresponding enhancement module is determined from the enhancement model according to the degradation factor, and the image is enhanced by the enhancement module, so that the image can be processed by the enhancement module corresponding to the degradation factor, which is more targeted and helps to improve the image enhancement effect.
[0149] It should be understood that Figure 6 In the structural block diagram of the image enhancement device shown in FIG. 1 , each unit is used to perform Figure 1 , Figure 2 , Figure 3The steps in the corresponding embodiments, and for Figure 1 , Figure 2 , Figure 3 Each step in the corresponding embodiment has been explained in detail in the above embodiment. Figure 1 , Figure 2 , Figure 3 as well as Figure 1 , Figure 2 , Figure 3 The relevant descriptions in the corresponding embodiments are not repeated here.
[0150] See also Figure 7 , Figure 7 700 of the embodiment of the present application includes: a processor 701, a memory 702, and a computer program 703 stored in the memory 702 and executable on the processor 701, such as an image enhancement program. When the processor 701 executes the computer program 703, the steps in each of the above-mentioned image enhancement embodiments are implemented, such as Figure 1 Steps 101 to 103 shown, or Figure 2 201 to 205 as shown, or Figure 3 Alternatively, the processor 701 executes the computer program 703 to implement the above Figure 6 The functions of each unit in the corresponding embodiment are, for example, Figure 6 For details on the functions of the units 601 to 603 shown, please refer to Figure 6 The relevant descriptions in the corresponding embodiments are not repeated here.
[0151] Exemplarily, the computer program 703 may be divided into one or more units, one or more units are stored in the memory 702, and are executed by the processor 701 to complete the present application. One or more units may be a series of computer program instruction segments capable of completing specific functions, and the instruction segments are used to describe the execution process of the computer program 703 in the server 700. For example, the computer program 703 may be divided into an image receiving unit, a module determination unit, and an image enhancement unit, and the specific functions of each unit are as described above.
[0152] The server may include, but is not limited to, a processor 701 and a memory 702. Those skilled in the art will appreciate that Figure 7 It is only an example of server 700 and does not constitute a limitation of server 700. It may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, the turntable device may also include input and output devices, network access devices, buses, etc.
[0153] The processor 701 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.
[0154] The memory 702 may be an internal storage unit of the server 700, such as a hard disk or memory of the server 700. The memory 702 may also be an external storage device of the server 700, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the server 700. Further, the memory 702 may also include both an internal storage unit of the server 700 and an external storage device. The memory 702 is used to store computer programs and other programs and data required by the turntable device. The memory 702 may also be used to temporarily store data that has been output or is to be output.
[0155] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.
[0156] If the integrated module is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, and the computer program that can be completed by instructing the relevant hardware through a computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include: any entity or device that can carry computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electric carrier signal, telecommunication signal and software distribution medium. It should be noted that the content contained in the computer-readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, the computer-readable medium does not include electric carrier signals and telecommunication signals.
[0157] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.
[0158] The embodiments described above are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.
Claims
1. An image enhancement method, characterized in that: include: receiving an image to be enhanced, and determining a degradation factor of the image to be enhanced according to an image frequency of the image to be enhanced; Determining a target enhancement module corresponding to the image to be enhanced according to the degradation factor, wherein the target enhancement module belongs to a pre-trained enhancement model, the enhancement model includes a plurality of enhancement modules, each enhancement module corresponds to a degradation factor, and each enhancement module is used to enhance the image having the corresponding degradation factor; The target enhancement module performs enhancement processing on the image to be enhanced to obtain an enhanced result image, including: extracting feature information of the image to be enhanced and recording it as target feature information; determining the spatial point coordinates corresponding to the target feature information according to a mapping relationship between the preset feature information and the spatial point coordinates in a preset space; inputting the target feature information and the spatial point coordinates corresponding to the target feature information into the enhancement module to obtain the enhanced result image; The enhanced result image is output.
2. The image enhancement method according to claim 1, characterized in that: When there are multiple degradation factors, there are multiple target enhancement modules; as well as The step of performing enhancement processing on the image to be enhanced according to the target enhancement module to obtain an enhanced result image includes: Sort multiple target enhancement modules according to the image frequency corresponding to each target enhancement module to obtain a module sequence; The first target enhancement module in the module sequence is used as the target module, the image to be enhanced is used as the target image, and the following enhancement processing operation is performed on the target image: Extracting feature information of the target image, determining the spatial point coordinates corresponding to the extracted feature information according to a mapping relationship between the preset feature information and the spatial point coordinates in a preset space, and inputting the extracted feature information and the determined spatial point coordinates into the target module to obtain an enhanced result image; The target image is replaced by the obtained result image, and the next target enhancement module in the module sequence is used as the target module. The enhancement processing operation is continued until multiple target enhancement modules in the module sequence are executed, and the obtained result image is determined as the enhanced result image.
3. The image enhancement method according to claim 1, characterized in that: The enhanced model is trained in the following way: Acquire a test sample set, wherein the test samples in the test sample set include a sample original image, a sample degradation factor, and a sample degraded image obtained by processing the sample original image using the sample degradation factor; For each initial enhancement module in the initial enhancement model, according to the degradation factor corresponding to the initial enhancement module, a test sample matching the initial enhancement module is selected from the test sample set, feature information of the sample degraded image in the selected test sample is extracted to obtain sample feature information, and the obtained sample feature information and the sample original image are input into the initial enhancement module to obtain the spatial point coordinates of the input sample feature information and the sample original image; According to the spatial point coordinates corresponding to each initial enhancement module, a target distribution is determined, the similarity between the target distribution and the Gaussian distribution is determined according to a preset loss function, and the adjustment state of each initial enhancement module is determined according to the determined similarity, and the parameters of the corresponding initial enhancement module are adjusted based on the adjustment state of each initial enhancement module, wherein the target distribution is the distribution of the spatial points of at least part of the initial enhancement modules in the initial enhancement model for each test sample, and the adjustment state includes a state that needs to be adjusted and a state that does not need to be adjusted; When the adjustment states of the initial enhancement modules in the initial enhancement model are all states that do not require adjustment, the enhancement model is obtained.
4. The image enhancement method according to claim 3, characterized in that: The step of adjusting parameters of the corresponding initial enhancement modules based on the adjustment states of the initial enhancement modules includes: For the initial enhancement module whose corresponding adjustment state is a state that does not require adjustment, corresponding training completion indication information is generated, indicating that the initial enhancement module does not need to be tested in the next round of testing.
5. The image enhancement method according to claim 3, characterized in that: The step of adjusting parameters of the corresponding initial enhancement modules based on the adjustment states of the initial enhancement modules includes: For the initial enhancement module whose corresponding adjustment state is a state that needs to be adjusted, the parameters of the initial enhancement module are adjusted according to the back propagation algorithm.
6. The image enhancement method according to any one of claims 1 to 5, characterized in that: The receiving the image to be enhanced and determining the degradation factor of the image to be enhanced according to the image frequency of the image to be enhanced includes: The image frequency of the image to be enhanced is input into a pre-trained classification model to obtain the degradation factor corresponding to the image frequency, wherein the classification model is used to characterize the corresponding relationship between the image frequency and the degradation factor.
7. An image enhancement device, characterized in that: include: An image receiving unit, configured to receive an image to be enhanced, and determine a degradation factor of the image to be enhanced according to an image frequency of the image to be enhanced; a module determination unit, configured to determine a target enhancement module corresponding to the image to be enhanced according to the degradation factor, wherein the target enhancement module belongs to a pre-trained enhancement model, the enhancement model includes a plurality of enhancement modules, each enhancement module corresponds to a degradation factor, and each enhancement module is used to perform enhancement processing on an image having a corresponding degradation factor; The image enhancement unit is used to perform enhancement processing on the image to be enhanced according to the target enhancement module to obtain an enhanced result image, including: extracting feature information of the image to be enhanced and recording it as target feature information; determining the spatial point coordinates corresponding to the target feature information according to a mapping relationship between the preset feature information and the spatial point coordinates in a preset space; inputting the target feature information and the spatial point coordinates corresponding to the target feature information into the enhancement module to obtain the enhanced result image; The image enhancement unit is also used to output the enhanced result image.
8. A server comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the image enhancement method according to any one of claims 1 to 6 is implemented.
9. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the image enhancement method according to any one of claims 1 to 6 is implemented.
Citation Information
Cited By
A method and system for dynamic routing adaptive enhancement for degraded physical images
CN122798646A
