Method for training image processing model, apparatus, device, and storage medium

Abstract

Disclosed in the present invention are a method for training an image processing model, an apparatus, a device, and a storage medium. The method comprises: acquiring a training sample set, wherein any training sample in the training sample set takes a masked image as a sample feature, and takes a high-quality image as a sample label, in the same training sample, the masked image and the high-quality image have the same image content and same image size, but have different image quality, and the image quality of a masked part in the masked image is lower than the image quality of the same position area in the high-quality image; acquiring an initial image processing model; and training the initial image processing model on the basis of the training sample set to obtain a trained result image processing model, wherein the result image processing model is used for improving the image quality.

Description

Image processing model training method, device, equipment and storage medium Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a training method, apparatus, device and storage medium for an image processing model. Background Art

[0002] With the popularization of computer technology, image processing technology has been widely used in many fields, such as object detection and tracking in images, or content recognition in images.

[0003] However, the images to be processed may have low image quality. For example, images may be blurry, foggy, rainy, or in low-light conditions. Directly processing these low-quality images often results in inaccurate results.

[0004] In order to improve the effect of image processing, it is usually necessary to improve the image quality first.

[0005] However, the current method for improving image quality generally uses a fixed algorithm to increase contrast or brightness, which has a poor effect in improving image quality.

[0006] Summary of the Invention

[0007] The present invention provides a training method, apparatus, device and storage medium for an image processing model to address deficiencies in related technologies.

[0008] According to a first aspect of an embodiment of the present invention, a method for training an image processing model is provided, comprising:

[0009] Obtain a training sample set;

[0010] For any training sample in the training sample set, the mask image is used as a sample feature and the high-quality image is used as a sample label; in the same training sample, the mask image and the high-quality image have the same image content and image size but different image quality, and the image quality of the masked portion in the mask image is less than the image quality of the same position area in the high-quality image;

[0011] Obtaining an initial image processing model;

[0012] The initial image processing model is trained based on the training sample set to obtain a trained result image processing model; the result image processing model is used to improve image quality.

[0013] Optionally, the pixel values ​​of the non-masked portion in the mask image are the same as the pixel values ​​of the same position area in the high-quality image.

[0014] Optionally, the method for constructing the training sample includes:

[0015] Obtain high-quality images and mask images;

[0016] The obtained mask image and the high-quality image have the same image content and image size but different image quality; the image quality of the masked portion of the obtained mask image is lower than the image quality of the same position area in the obtained high-quality image; the pixel values ​​of the non-masked portion of the obtained mask image are the same as the pixel values ​​of the same position area in the obtained high-quality image;

[0017] The obtained mask image is used as the sample feature and the obtained high-quality image is used as the sample label to construct the training sample.

[0018] Optionally, obtaining a high-quality image and a mask image includes:

[0019] Acquire high-quality images;

[0020] A mask portion is determined for the acquired high-quality image, and the image quality of the determined mask portion is reduced to obtain a mask image.

[0021] Optionally, obtaining a high-quality image and a mask image includes:

[0022] Acquire a low-quality image and a high-quality image; the acquired low-quality image and the acquired high-quality image have the same image content and image size but different image quality, and the image quality of the acquired low-quality image is less than the image quality of the acquired high-quality image;

[0023] A mask image is obtained based on the acquired low-quality image and high-quality image.

[0024] Optionally, acquiring the low-quality image and the high-quality image includes:

[0025] Acquire high-quality images;

[0026] For the acquired high-quality image, the image quality is reduced to obtain a low-quality image.

[0027] Optionally, obtaining a mask image based on the acquired low-quality image and high-quality image includes:

[0028] determining a mask portion based on the acquired low-quality image and the acquired high-quality image;

[0029] The mask image is obtained by replacing the pixel values ​​of the masked part determined in the high-quality image with the pixel values ​​of the same position area in the low-quality image; or, replacing the pixel values ​​of the non-masked part determined in the low-quality image with the pixel values ​​of the same position area in the high-quality image; or, combining the pixel values ​​of the non-masked part determined in the high-quality image and the pixel values ​​of the masked part in the low-quality image.

[0030] Optionally, determining the mask portion based on the acquired low-quality image and high-quality image includes:

[0031] Divide the acquired low-quality image into several image blocks;

[0032] For at least one image block, a corresponding mask part is determined respectively.

[0033] Optionally, determining a corresponding mask part for at least one image block includes:

[0034] For at least one image block, perform the following steps respectively:

[0035] An image quality level corresponding to the image block is determined, and a corresponding mask portion is determined based on a mask determination method corresponding to the determined image quality level.

[0036] Optionally, determining the image quality level corresponding to the image block includes:

[0037] Determine the image quality level corresponding to the image block based on a preset classification model;

[0038] or

[0039] Determine the difference between an image block in the acquired low-quality image and an image block at the same position in the acquired high-quality image; and determine an image quality level corresponding to the image block based on the determined difference.

[0040] Optionally, the lower the image quality represented by the image quality level, the smaller the area proportion of the determined mask portion in the corresponding image block.

[0041] Optionally, the mask determination method corresponding to the image quality level includes:

[0042] The image block is further divided into image sub-blocks corresponding to the number of image quality levels;

[0043] Among image blocks with the same image quality level, image sub-blocks with corresponding proportions of the image quality levels are determined as mask parts.

[0044] Optionally, the lower the image quality represented by the image quality level, the smaller the corresponding number;

[0045] The lower the image quality represented by the image quality level, the smaller the corresponding ratio.

[0046] Optionally, the initial image processing model includes: a dimensionality reduction encoding layer and a dimensionality increase decoding layer;

[0047] The dimensionality reduction coding layer is used to: perform dimensionality reduction convolution on the input image to obtain a feature map;

[0048] The dimension-raising decoding layer is used to upsample the input feature map and output a predicted image.

[0049] Optionally, the initial image processing model further includes: a fully connected layer;

[0050] The fully connected layer is used to predict and output a predicted feature map of the same size as the input feature map.

[0051] Optionally, the fully connected layer is used to:

[0052] Divide the input feature map into a preset number of feature image blocks;

[0053] Based on the preset number of feature image blocks, respectively predicting and obtaining the preset number of prediction image blocks;

[0054] The obtained predicted image blocks are spliced ​​into a predicted feature map.

[0055] Optionally, the fully connected layer is used to:

[0056] Dividing the feature map obtained from the dimensionality reduction coding layer into a preset number of feature image blocks;

[0057] Based on the preset number of feature image blocks, respectively predicting and obtaining the preset number of prediction image blocks;

[0058] The obtained predicted image blocks are spliced ​​into a predicted feature map.

[0059] Optionally, the initial image processing model is used to:

[0060] For the input mask image, perform dimensionality reduction convolution to obtain the feature map;

[0061] The obtained feature map is upsampled and a predicted image is output; the predicted image and the input mask image have the same size.

[0062] Optionally, the method further includes:

[0063] Constructing encoding layers of other image processing task models based on the encoding layers in the result image processing model;

[0064] or

[0065] For any image, the image quality is improved based on the result image processing model.

[0066] Optionally, the image quality is characterized by the degree to which it is affected by environmental factors.

[0067] Optionally, the loss function of the initial image processing model is used to:

[0068] The loss is determined for the masked portion of the predicted image output by the initial image processing model and the same location area in the high-quality image of the corresponding sample.

[0069] According to a second aspect of an embodiment of the present invention, another method for training an image processing model is provided, comprising:

[0070] Acquire a plurality of image pairs; each image pair includes a low-quality image and a high-quality image; in each image pair, the low-quality image and the high-quality image have the same image content and image size but different image quality, and the image quality of the low-quality image is less than the image quality of the high-quality image;

[0071] Acquire an initial image processing model, and determine the acquired initial image processing model as the current image processing model;

[0072] The following steps are executed in a loop until the preset loop stop condition is met:

[0073] Based on the acquired image pairs, training samples are constructed; for any constructed training sample, the mask image is used as a sample feature and the high-quality image is used as a sample label; in the same training sample, the mask image and the high-quality image have the same image content and image size but different image quality, the image quality of the masked portion in the masked image is lower than the image quality of the same position area in the high-quality image, and the pixel values ​​of the non-masked portion in the masked image are the same as the pixel values ​​of the same position area in the high-quality image;

[0074] Training the current image processing model based on the constructed training samples, and re-determining the image processing model after training and updating as the current image processing model;

[0075] After the loop ends, the current image processing model is determined as the result image processing model.

[0076] Optionally, constructing a training sample based on the acquired image pair includes:

[0077] For any set of image pairs, the low-quality image is divided into several image blocks, and the mask part of at least one image block is determined using a preset classification model; in the acquired low-quality image, the pixel values ​​of the determined non-masked part are replaced with the pixel values ​​of the same position area in the high-quality image to obtain a mask image; the mask image is used as the sample feature and the high-quality image is used as the sample label to construct a training sample.

[0078] Optionally, in each cycle, the encoding layer parameters of the preset classification model are updated to the encoding layer parameters of the current image processing model.

[0079] According to a third aspect of an embodiment of the present invention, another method for training an image processing model is provided, comprising:

[0080] Obtain a training sample set;

[0081] For any training sample in the training sample set, the mask image is used as a sample feature and the high-quality image is used as a sample label; in the same training sample, the mask image and the high-quality image have the same image content and image size but different fog influence levels, and the fog influence level of the masked portion in the mask image is greater than the fog influence level of the same position area in the high-quality image;

[0082] Obtaining an initial image processing model;

[0083] The initial image processing model is trained based on the training sample set to obtain a trained result image processing model; the result image processing model is used to reduce the degree of fog influence.

[0084] According to a fourth aspect of an embodiment of the present invention, there is provided a training device for an image processing model, comprising:

[0085] A sample acquisition unit, used to acquire a training sample set;

[0086] For any training sample in the training sample set, the mask image is used as a sample feature and the high-quality image is used as a sample label; in the same training sample, the mask image and the high-quality image have the same image content and image size but different image quality, and the image quality of the masked portion in the mask image is less than the image quality of the same position area in the high-quality image;

[0087] A model acquisition unit, used for acquiring an initial image processing model;

[0088] A training unit is used to train the initial image processing model based on the training sample set to obtain a trained result image processing model; the result image processing model is used to improve image quality.

[0089] Optionally, the pixel values ​​of the non-masked portion in the mask image are the same as the pixel values ​​of the same position area in the high-quality image.

[0090] Optionally, the method for constructing the training sample includes:

[0091] Obtain high-quality images and mask images;

[0092] The obtained mask image and the high-quality image have the same image content and image size but different image quality; the image quality of the masked portion of the obtained mask image is lower than the image quality of the same position area in the obtained high-quality image; the pixel values ​​of the non-masked portion of the obtained mask image are the same as the pixel values ​​of the same position area in the obtained high-quality image;

[0093] The obtained mask image is used as the sample feature and the obtained high-quality image is used as the sample label to construct the training sample.

[0094] Optionally, obtaining a high-quality image and a mask image includes:

[0095] Acquire high-quality images;

[0096] A mask portion is determined for the acquired high-quality image, and the image quality of the determined mask portion is reduced to obtain a mask image.

[0097] Optionally, obtaining a high-quality image and a mask image includes:

[0098] Acquire a low-quality image and a high-quality image; the acquired low-quality image and the acquired high-quality image have the same image content and image size but different image quality, and the image quality of the acquired low-quality image is less than the image quality of the acquired high-quality image;

[0099] A mask image is obtained based on the acquired low-quality image and high-quality image.

[0100] Optionally, acquiring the low-quality image and the high-quality image includes:

[0101] Acquire high-quality images;

[0102] For the acquired high-quality image, the image quality is reduced to obtain a low-quality image.

[0103] Optionally, obtaining a mask image based on the acquired low-quality image and high-quality image includes:

[0104] determining a mask portion based on the acquired low-quality image and the acquired high-quality image;

[0105] The mask image is obtained by replacing the pixel values ​​of the masked part determined in the high-quality image with the pixel values ​​of the same position area in the low-quality image; or, replacing the pixel values ​​of the non-masked part determined in the low-quality image with the pixel values ​​of the same position area in the high-quality image; or, combining the pixel values ​​of the non-masked part determined in the high-quality image and the pixel values ​​of the masked part in the low-quality image.

[0106] Optionally, determining the mask portion based on the acquired low-quality image and high-quality image includes:

[0107] Divide the acquired low-quality image into several image blocks;

[0108] For at least one image block, a corresponding mask part is determined respectively.

[0109] Optionally, determining a corresponding mask part for at least one image block includes:

[0110] For at least one image block, perform the following steps respectively:

[0111] An image quality level corresponding to the image block is determined, and a corresponding mask portion is determined based on a mask determination method corresponding to the determined image quality level.

[0112] Optionally, determining the image quality level corresponding to the image block includes:

[0113] Determine the image quality level corresponding to the image block based on a preset classification model;

[0114] or

[0115] Determine the difference between an image block in the acquired low-quality image and an image block at the same position in the acquired high-quality image; and determine an image quality level corresponding to the image block based on the determined difference.

[0116] Optionally, the lower the image quality represented by the image quality level, the smaller the area proportion of the determined mask portion in the corresponding image block.

[0117] Optionally, the mask determination method corresponding to the image quality level includes:

[0118] The image block is further divided into image sub-blocks corresponding to the number of image quality levels;

[0119] Among image blocks with the same image quality level, image sub-blocks with corresponding proportions of the image quality levels are determined as mask parts.

[0120] Optionally, the lower the image quality represented by the image quality level, the smaller the corresponding number;

[0121] The lower the image quality represented by the image quality level, the smaller the corresponding ratio.

[0122] Optionally, the initial image processing model includes: a dimensionality reduction encoding layer and a dimensionality increase decoding layer;

[0123] The dimensionality reduction coding layer is used to: perform dimensionality reduction convolution on the input image to obtain a feature map;

[0124] The dimension-raising decoding layer is used to upsample the input feature map and output a predicted image.

[0125] Optionally, the initial image processing model further includes: a fully connected layer;

[0126] The fully connected layer is used to predict and output a predicted feature map of the same size as the input feature map.

[0127] Optionally, the fully connected layer is used to:

[0128] Divide the input feature map into a preset number of feature image blocks;

[0129] Based on the preset number of feature image blocks, respectively predicting and obtaining the preset number of prediction image blocks;

[0130] The obtained predicted image blocks are spliced ​​into a predicted feature map.

[0131] Optionally, the fully connected layer is used to:

[0132] Dividing the feature map obtained from the dimensionality reduction coding layer into a preset number of feature image blocks;

[0133] Based on the preset number of feature image blocks, respectively predicting and obtaining the preset number of prediction image blocks;

[0134] The obtained predicted image blocks are spliced ​​into a predicted feature map.

[0135] Optionally, the initial image processing model is used to:

[0136] For the input mask image, perform dimensionality reduction convolution to obtain the feature map;

[0137] The obtained feature map is upsampled and a predicted image is output; the predicted image and the input mask image have the same size.

[0138] Optionally, the device further comprises:

[0139] Application units for:

[0140] Constructing encoding layers of other image processing task models based on the encoding layers in the result image processing model;

[0141] or

[0142] For any image, the image quality is improved based on the result image processing model.

[0143] Optionally, the image quality is characterized by the degree to which it is affected by environmental factors.

[0144] Optionally, the loss function of the initial image processing model is used to:

[0145] The loss is determined for the masked portion of the predicted image output by the initial image processing model and the same location area in the high-quality image of the corresponding sample.

[0146] According to the above embodiments, by using mask images and high-quality images to train the image processing model, the trained image processing model can be used to improve image quality, which is more flexible than a fixed algorithm and is convenient for better improving image quality.

[0147] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0148] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0149] FIG1 is a schematic flow chart of a method for training an image processing model according to an embodiment of the present invention;

[0150] FIG2 is a schematic diagram showing a principle of combining images according to an embodiment of the present invention;

[0151] FIG3 is a schematic diagram showing a principle of image division according to an embodiment of the present invention;

[0152] FIG4 is a flow chart of another method for training an image processing model according to an embodiment of the present invention;

[0153] FIG5 is a schematic flow chart of another method for training an image processing model according to an embodiment of the present invention;

[0154] FIG6 is a schematic diagram showing the principle of a pre-training method according to an embodiment of the present invention;

[0155] FIG7 is a schematic diagram showing a segmented image according to an embodiment of the present invention;

[0156] FIG8 is a schematic diagram showing the principle of a method for determining fog concentration according to an embodiment of the present invention;

[0157] FIG9 is a schematic diagram showing a principle of determining a mask according to an embodiment of the present invention;

[0158] FIG10 is a schematic diagram showing another principle of combining images according to an embodiment of the present invention;

[0159] FIG11 is a schematic diagram of a pre-training model structure according to an embodiment of the present invention;

[0160] FIG12 is a schematic diagram of a Block structure according to an embodiment of the present invention;

[0161] FIG13 is a schematic diagram showing a principle of determining model loss according to an embodiment of the present invention;

[0162] FIG14 is a schematic structural diagram of a training device for an image processing model according to an embodiment of the present invention;

[0163] FIG15 is a schematic structural diagram of another image processing model training device according to an embodiment of the present invention;

[0164] FIG16 is a schematic structural diagram of another image processing model training device according to an embodiment of the present invention;

[0165] FIG17 is a schematic diagram of the hardware structure of a computer device configured with the method according to an embodiment of the present invention. DETAILED DESCRIPTION

[0166] Exemplary embodiments will be described in detail herein, examples of which are illustrated in the accompanying drawings. In the following description, when referring to the drawings, like numbers in different figures represent like or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments are not intended to represent all possible embodiments consistent with the present invention. Rather, they are merely examples of apparatus and methods consistent with certain aspects of the present invention, as detailed in the appended claims.

[0167] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in the embodiments of the present invention are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0168] With the popularization of computer technology, image processing technology has been widely used in many fields, such as object detection and tracking in images, or content recognition in images.

[0169] However, the images to be processed may have low image quality. For example, images may be blurry, foggy, rainy, or in low-light conditions. Directly processing these low-quality images often results in inaccurate results.

[0170] In order to improve the effect of image processing, it is usually necessary to improve the image quality first.

[0171] However, the current method for improving image quality generally uses a fixed algorithm to increase contrast or brightness, which has a poor effect in improving image quality.

[0172] An embodiment of the present invention discloses a method for training an image processing model.

[0173] In this method, a machine learning approach can be used to train an image processing model to improve image quality. Specifically, training samples can be constructed using images of varying quality to train the image processing model and improve image quality.

[0174] The image processing model is obtained through model training to improve image quality. Compared with fixed algorithms, it can be more flexible and convenient to better improve image quality.

[0175] In the specific process of constructing training samples, low-quality images and high-quality images can be obtained to form training samples, with low-quality images as sample features and high-quality images as sample labels for training image processing models.

[0176] The obtained low-quality image and high-quality image may have the same image content and image size but different image quality. The image quality of the low-quality image may be lower than that of the high-quality image.

[0177] Specifically, the image quality of the low-quality image can be controlled to be lower than the image quality of the high-quality image in the form of a mask.

[0178] The low-quality image may specifically include a mask portion, and the image quality of the mask portion in the low-quality image may be specifically set to be lower than the image quality of the region at the same position in the high-quality image.

[0179] The non-masked portion in the low-quality image can be consistent with the same position area in the high-quality image. Specifically, the pixel values ​​of the non-masked portion in the low-quality image can be the same as the pixel values ​​of the same position area in the high-quality image.

[0180] During the training process, since the difference between low-quality images and high-quality images mainly lies in the mask part, it is possible to help improve the image quality of the mask part in the low-quality image based on at least the non-mask part in the low-quality image, and learn the association between different parts of the low-quality image to improve the overall image quality of the low-quality image.

[0181] With the help of masks, the image quality of low-quality images can be easily controlled. In the process of model training, the image quality of the masked part can be focused on, which facilitates learning the relationship between different parts of the image and improves the training effect of the image processing model.

[0182] Moreover, since the non-masked part of the low-quality image can be consistent with the same position area of ​​the high-quality image, it can facilitate the training convergence of the image processing model and improve the training efficiency of the model.

[0183] This method mainly relies on model training to train an image processing model for improving image quality, and makes improvements on the training samples. Based on the form of masks, it is convenient to control the quality of the image in the training sample features, facilitate learning the relationship between different parts of the image, and improve the training effect of the image processing model.

[0184] The following is a detailed explanation of a training method for an image processing model provided by an embodiment of the present invention.

[0185] As shown in FIG1 , FIG1 is a flow chart of a training method for an image processing model according to an embodiment of the present invention.

[0186] The embodiment of the present invention does not limit the execution subject of the method process. Optionally, the execution subject can be any computing device, such as an image processing terminal, an image processing server, an image processing client, etc.

[0187] The method may include the following steps.

[0188] S101: Obtain a training sample set.

[0189] Among them, any training sample in the training sample set uses the mask image as the sample feature and the high-quality image as the sample label.

[0190] In the same training sample, the mask image and the high-quality image have the same image content and image size but different image quality. The image quality of the masked part in the mask image may be lower than the image quality of the same position area in the high-quality image.

[0191] S102: Acquire an initial image processing model.

[0192] S103: Training an initial image processing model based on the training sample set to obtain a trained result image processing model; the result image processing model is used to improve image quality.

[0193] The above method process uses mask images and high-quality images to train the image processing model, so that the trained image processing model can be used to improve image quality. Compared with fixed algorithms, it is more flexible and convenient to better improve image quality.

[0194] Furthermore, the above method flow can facilitate learning the association between various parts of the image during the training process by setting an image mask, thereby improving the training effect of the image processing model and facilitating better improvement of image quality.

[0195] Furthermore, the pixel values ​​in the unmasked portion of the masked image can be the same as those in the same location in the high-quality image. Since the unmasked portions of the masked and high-quality images remain consistent, this facilitates loss calculation, improves the training efficiency of image processing models, and accelerates the training convergence of image processing models. This allows for a more focused focus on improving the image quality of the masked portion, allowing for better learning of the relationships between image components and ultimately enhancing the training effectiveness of image processing models.

[0196] Of course, the non-masked portion in the mask image may also be different from the region at the same position in the high-quality image.

[0197] The following is a detailed explanation of various aspects of the above method flow.

[0198] 1. Specific scenarios for improved image quality.

[0199] This method does not limit the specific measurement method of image quality.

[0200] Optionally, the image quality can be determined based on the attributes of the image itself, specifically, the image quality can be measured based on at least one of the following: clarity, brightness, contrast, exposure, noise, artifacts, focus, flash, texture, color, hand shake, etc.

[0201] For example, blurred images, noisy images, dark images, etc.

[0202] In a specific example, in the training sample set of this method, for any training sample, the masked portion of the mask image may have less clarity than the region at the same location in the high-quality image, or may have more noise than the region at the same location in the high-quality image, etc. Training an image processing model based on such a training sample set can be used to improve image quality based on the image's inherent properties, specifically by increasing clarity, brightness, or reducing noise.

[0203] In addition, image quality is also affected by other factors, especially factors during image capture or transmission.

[0204] For example, the weather, lighting, hand shaking when taking the image, the hardware limitations of the camera itself, or the compression when transmitting the image, etc.

[0205] For example, images taken on rainy days are easily affected by the rain, images taken on foggy days are easily reduced in clarity, and images taken at night are easily reduced in quality due to insufficient light, making them unusable for image processing such as target detection.

[0206] For example, during the transmission of an image, the compression algorithm loses a lot of information or adds a lot of noise, resulting in a decrease in image quality.

[0207] Optionally, the image quality can be characterized and measured by the degree of influence of environmental factors. The higher the degree of influence of environmental factors, the lower the image quality. Specific environmental factors are not limited. Environmental factors may include at least one of the following: weather, lighting, non-photographed objects, and the like. The weather may specifically be fog, rain, snow, dust, hail, and the like. The influence of lighting is mainly due to the lack of information caused by the lack of light. For example, images taken at dusk or at night are often difficult to distinguish. Non-photographed objects may specifically be objects in the environment that affect the shooting, such as steam, water mist, glass, and the like.

[0208] It is understood that improving image quality can specifically mean reducing the degree to which images are affected by environmental factors. For example, reducing the impact of fog on captured images to achieve a defogging effect; reducing the impact of dusty weather on captured images to achieve a dust removal effect.

[0209] In a specific example, in the training sample set of the present method, for any training sample, the mask portion of the mask image may be more affected by environmental factors than the same position region in the high-quality image.

[0210] Specifically, it can be the mask part of the mask image, which is affected by fog to a greater extent than the area at the same position in the high-quality image, that is, the mask part has more fog than the area at the same position in the high-quality image; it can also be the mask part of the mask image, which is affected by rainy days to a greater extent than the area at the same position in the high-quality image, that is, the mask part has more raindrops than the area at the same position in the high-quality image; it can also be the mask part of the mask image, which is affected by sand and dust to a greater extent than the area at the same position in the high-quality image, that is, the mask part has more sand and dust than the area at the same position in the high-quality image; it can also be the mask part of the mask image, which is affected by night to a greater extent than the area at the same position in the high-quality image, that is, the mask part has less light than the area at the same position in the high-quality image.

[0211] Training the image processing model based on the above training sample set can reduce the degree to which the image is affected by environmental factors, thereby improving the image quality.

[0212] For ease of understanding, in some specific scenarios, training samples can be constructed based on foggy images, and the above-mentioned method flow can be used to train an image processing model for achieving the defogging effect; training samples can also be constructed based on rainy images, and the above-mentioned method flow can be used to train an image processing model for achieving the deraining effect.

[0213] Of course, it is also possible to integrate multiple images of different image qualities and use the above method flow to train an image processing model for comprehensively improving image quality.

[0214] By explaining the specific scenario of improving the specific image quality, the above scenario can be combined with the following description.

[0215] 2. How to construct training samples.

[0216] 1. About the constructed training samples.

[0217] First, explain the constructed training samples from the perspective of the construction results.

[0218] The following explanation is given for any training sample in the training sample set. It can be understood that the following explanation can be referred to for each training sample in the training sample set.

[0219] Any constructed training sample may include a mask image and a high-quality image. Specifically, the mask image may be used as a sample feature, and the high-quality image may be used as a sample label.

[0220] Among them, in any training sample, the mask image and the high-quality image have the same image content and image size but different image quality. The image quality of the mask part in the mask image is lower than the image quality of the same position area in the high-quality image.

[0221] In any training sample, the pixel values ​​of the non-masked part of the mask image can be the same as the pixel values ​​of the same position area in the high-quality image. Of course, there can also be differences in the non-masked part.

[0222] For the specific explanation of image quality, please refer to the above explanation.

[0223] Regarding the mask portion of the mask image, there is no specific limitation on the method for determining the mask portion, and details are explained below. In the same training sample, the mask image and the high-quality image may differ in the mask portion, while being identical in the non-mask portion.

[0224] The non-masked part in the mask image can be kept consistent with the same position area in the high-quality image in the same training sample.

[0225] Specifically, the value of any pixel in the non-masked part of the mask image can be the same as the value of the pixel at the same position in the high-quality image in the same training sample; or the value of each pixel in the non-masked part of the mask image can be the same as the value of the pixel at the same position in the high-quality image in the same training sample.

[0226] Furthermore, the image sizes of any two different training samples can be different or the same. For training samples with different image sizes, the image sizes can be pre-standardized or unified when training the image processing model. Therefore, it is permitted for the training sample set to contain different training samples with different image sizes.

[0227] 2. About the specific construction method of training samples.

[0228] The process of this method does not limit the specific construction method of the training samples, as long as the training samples that meet the above form can be constructed.

[0229] Optionally, the specific construction of the training sample requires a mask image and a high-quality image that conform to the above format. Specifically, the mask image and the high-quality image that conform to the above format can be directly obtained to construct the training sample.

[0230] Since the quality difference between the mask image and the high-quality image that conforms to the above form mainly exists in the mask part, it is possible to obtain a high-quality image and set a mask to obtain a mask image, or to obtain a mask image and set a mask to improve the image quality to obtain a high-quality image, or to obtain two images of different qualities and combine them to obtain a mask image, and determine the image with higher quality as the high-quality image.

[0231] This embodiment can use masks to help increase the number of training samples and reduce the difficulty of constructing training samples.

[0232] Optionally, the method for constructing a training sample may specifically include: obtaining a high-quality image and a mask image; the obtained mask image and the high-quality image have the same image content and image size but different image quality; the image quality of the masked part in the obtained mask image is less than the image quality of the same position area in the obtained high-quality image; the pixel values ​​of the non-masked part in the obtained mask image are the same as the pixel values ​​of the same position area in the obtained high-quality image; and constructing a training sample using the obtained mask image as a sample feature and the obtained high-quality image as a sample label.

[0233] Specifically, a set of image pairs can be obtained, wherein the two images in the image pair have the same image content and image size but different image quality. Specifically, the quality difference can be in the masked portion, while the quality difference is the same in the non-masked portion, and the pixel values ​​at the same position are the same.

[0234] Furthermore, the image with higher quality in the set of image pairs can be determined as the high-quality image, and the image with lower quality can be determined as the mask image.

[0235] This embodiment does not limit the method for obtaining the above-mentioned image pairs. A pre-stored image pair may be directly obtained, or an image pair conforming to the above-mentioned format may be constructed.

[0236] This embodiment does not limit the specific method for obtaining a high-quality image, nor does it limit the specific method for obtaining a mask image. This embodiment also does not limit the specific method for constructing a high-quality image, nor does it limit the specific method for constructing a mask image.

[0237] Alternatively, an image can be directly acquired and determined to be a high-quality image, and a mask image can be generated based on the high-quality image. Acquiring a high-quality image and a mask image can specifically include: acquiring a high-quality image; determining a mask portion of the acquired high-quality image; and reducing the image quality of the determined mask portion to obtain the mask image. This embodiment does not limit the specific method for reducing image quality; specific methods for reducing image quality include blurring, adding noise, increasing fog, adding raindrops, and so on.

[0238] Of course, it is also possible to directly acquire an image and determine it as a mask image, and then further generate a high-quality image based on the mask image. Acquiring a high-quality image and a mask image can specifically include: acquiring a mask; determining a masked portion of the acquired mask; and improving the image quality of the determined masked portion to obtain a high-quality image. This embodiment does not limit the specific method for improving image quality; specifically, image quality can be improved using a fixed algorithm.

[0239] It can be seen that the above processing is based on the mask part of an image to obtain another image, thereby obtaining a set of image pairs that conform to the above form, wherein the mask image and the high-quality image can be regarded as two different copies of the same image, and the image quality of the two copies is different.

[0240] It is understandable that for the same high-quality image, multiple different mask parts can be determined, and multiple different mask images can be determined, so that different samples can be constructed respectively (specifically, the sample features can be different, but the sample labels are the same).

[0241] Similarly, for the same mask image, multiple different mask parts can be determined, and multiple different high-quality images can be determined, so that different samples can be constructed respectively (specifically, the sample features can be the same, but the sample labels can be different).

[0242] Therefore, this embodiment can conveniently increase the number of training samples and reduce the difficulty of constructing training samples.

[0243] In addition, optionally, a mask image and a high-quality image can be obtained by combining two images with different qualities.

[0244] Specifically, a group of image pairs can be obtained, where the image pairs include low-quality images and high-quality images; in any group of image pairs, the low-quality images and the high-quality images have the same image content and image size but different image quality, and the image quality of the low-quality images is less than the image quality of the high-quality images.

[0245] The two images in the image pair can be different copies of the same image obtained through different processing, or can be obtained by taking separate photos, as long as they meet the above format.

[0246] Specifically, the image may be first acquired and determined to be a low-quality image, and then the image quality of the low-quality image may be improved as a whole to obtain a high-quality image. Alternatively, the image may be first acquired and determined to be a high-quality image, and then the image quality of the high-quality image may be reduced as a whole to obtain a low-quality image.

[0247] Of course, two images may be taken at the same time and the same position for the same object using different methods and respectively determined as a low-quality image and a high-quality image based on the image quality.

[0248] Therefore, optionally, obtaining a high-quality image and a mask image can specifically be: obtaining a low-quality image and a high-quality image; the obtained low-quality image and the obtained high-quality image have the same image content and the same image size but different image quality, and the image quality of the obtained low-quality image is less than the image quality of the obtained high-quality image; and obtaining a mask image based on the obtained low-quality image and the obtained high-quality image.

[0249] There is no limitation on the method of obtaining a low-quality image or the method of constructing a low-quality image. There is no limitation on the method of obtaining a high-quality image or the method of constructing a high-quality image.

[0250] Optionally, obtaining a low-quality image and a high-quality image may specifically include: obtaining a high-quality image; and reducing the quality of the obtained high-quality image to obtain a low-quality image. This embodiment does not limit the specific method for reducing image quality, and the image quality may be reduced by blurring, adding noise, adding fog, adding raindrops, etc.

[0251] Of course, optionally, obtaining a low-quality image and a high-quality image may also be performed by: obtaining a low-quality image; and improving the quality of the obtained low-quality image to obtain a high-quality image. This embodiment does not limit the specific method of improving image quality, and the image quality may be improved by a fixed algorithm.

[0252] Alternatively, different cameras can be used to capture the same object at the same location and angle at the same time, resulting in images of varying quality. Alternatively, different cameras can be used to capture the same object at the same location and angle at different times, resulting in images of varying quality. Different cameras may have different hardware configurations, resulting in varying image quality.

[0253] The low-quality image and the high-quality image can be regarded as two different copies of the same image, and the image quality of the two copies is different.

[0254] It is understandable that for the high-quality images obtained, different methods or different degrees of quality reduction can be used to obtain different low-quality images, which are used to combine to obtain mask images, thereby increasing the number of training samples, improving training effects, and reducing the difficulty of constructing training samples.

[0255] For example, the acquired high-quality image can be subjected to image quality reduction processing (specifically, blurring or adding fog) to obtain a first image. The first image is then subjected to image quality reduction processing to obtain a second image. Three image pairs can then be obtained: (first image, high-quality image), (second image, first image), and (second image, high-quality image). These pairs can be combined to obtain a mask image, which is then used to generate training samples.

[0256] The present method does not limit the method for combining the low-quality image and the high-quality image to obtain the mask image. Specifically, the combination can be performed by combining the mask portion, or the combination can be performed first to obtain the mask image, or the combination can be performed directly to obtain the mask image.

[0257] The specific method for determining the mask part can be found below.

[0258] For the low-quality image and the high-quality image in the same image pair, a portion of pixel values ​​can be taken from each of them and combined into a mask image.

[0259] Specifically, the pixel values ​​may be replaced based on a low-quality image, or the pixel values ​​may be replaced based on a high-quality image.

[0260] Optionally, a mask image is obtained based on the acquired low-quality image and high-quality image, specifically: based on the acquired low-quality image and high-quality image, a mask part is determined; the pixel values ​​of the mask part determined in the high-quality image are replaced with the pixel values ​​of the same position area in the low-quality image to obtain a mask image.

[0261] In addition, optionally, a mask image is obtained based on the acquired low-quality image and high-quality image, which can be specifically: based on the acquired low-quality image and high-quality image, the mask part is determined; the pixel values ​​of the non-masked part determined in the low-quality image are replaced with the pixel values ​​of the same position area in the high-quality image to obtain the mask image.

[0262] Of course, optionally, the mask part can also be determined based on the acquired low-quality image and high-quality image; the pixel values ​​of the non-masked part determined in the high-quality image and the pixel values ​​of the masked part in the low-quality image are combined to obtain a mask image.

[0263] The specific method for determining the mask portion may be found below.

[0264] By using the above combination method, the mask image can be obtained quickly and conveniently, thereby improving the efficiency of combining images and obtaining the mask image.

[0265] Moreover, different mask images can be obtained through different combinations or different mask parts, so that different training samples can be constructed, which increases the number of training samples, reduces the difficulty of constructing training samples, and improves the training effect.

[0266] For ease of understanding, as shown in FIG2 , FIG2 is a schematic diagram showing a principle of a combined image according to an embodiment of the present invention.

[0267] Among them, images taken in clear weather can be determined as high-quality images, and images taken in foggy weather can be determined as low-quality images. For the convenience of display, the images here are represented by simple geometric lines, and a simple gray color is used to represent the fog effect in foggy weather.

[0268] It is understandable that the low-quality image and the high-quality image have the same image content and image size but different image quality. The image quality of the low-quality image is lower than that of the high-quality image.

[0269] In addition, it is determined that there is a mask part, which is displayed in the form of black squares.

[0270] The high-quality image and the low-quality image may then be combined according to the distribution of the mask portion to obtain a combined image, and the combined image may be determined as the mask image.

[0271] Afterwards, training samples can be constructed based on the mask image and the high-quality image, with the mask image as the sample feature and the high-quality image as the sample label.

[0272] It can be understood that different mask images can be determined based on different mask parts, and different training samples can be constructed.

[0273] 3. How to determine the mask part.

[0274] 1. Regarding the determined mask part.

[0275] In this method, the mask portion is not limited to a specific shape and size.

[0276] For example, the mask portion may be a regular rectangle, circle, triangle, etc., or an irregular shape, specifically, it may fit the outline of an object or a target.

[0277] For another example, the size of the mask portion may be a fixed size, a size with a fixed area ratio, or a non-fixed size that changes with the shape.

[0278] It should be emphasized that, since the mask portion exists in the mask image, the mask portion can be determined in the process of constructing the training sample.

[0279] For different images, the same or different mask parts can be determined. Please refer to the above explanation for details.

[0280] 2. Specifically determine the method for the mask part.

[0281] The method does not limit the specific way of determining the mask part. Optionally, the mask can be determined randomly or according to a fixed mask format.

[0282] For example, the mask may be randomly determined for a high-quality image, or the mask may be determined for a low-quality image according to a preset fixed mask format.

[0283] The preset fixed mask format may specifically be setting fixed-size areas at the four corners of the image as masks, and the like.

[0284] In an optional embodiment, the mask portion may be determined for the entire image, or the mask portion may be determined for each partial area of ​​the image.

[0285] When determining the mask portion for the high-quality image, the mask portion may be determined for the entire high-quality image, or may be determined for each partial region of the high-quality image.

[0286] Therefore, optionally, a mask part is determined for the acquired high-quality image, which may specifically be: dividing the acquired high-quality image into several image blocks; and determining a corresponding mask part for at least one image block.

[0287] Of course, the mask part may also be randomly determined for the acquired high-quality image.

[0288] Furthermore, when determining a mask portion for a low-quality image, the mask portion may be determined for the entire low-quality image, or the mask portion may be determined for each partial region of the low-quality image.

[0289] Optionally, the mask part is determined for the acquired low-quality image, which may specifically be: dividing the acquired low-quality image into a plurality of image blocks; and determining a corresponding mask part for at least one image block.

[0290] When determining the mask part for a low-quality image and a high-quality image, the mask part can be determined by combining the entire low-quality image and the entire high-quality image. The mask part can also be determined separately for each partial area of ​​the low-quality image, or the mask part can be determined separately for each partial area of ​​the high-quality image.

[0291] Optionally, the mask part is determined based on the acquired low-quality image and high-quality image. Specifically, the acquired low-quality image is divided into several image blocks; and corresponding mask parts are determined for at least one image block.

[0292] In this embodiment, the image may be divided to obtain image blocks and the mask part may be determined for the image blocks, thereby improving the dispersion and flexibility of the mask part.

[0293] The above embodiments are further summarized below to explain how to divide an image into image blocks and determine mask parts for the image blocks.

[0294] This embodiment does not limit the specific method, number, and size of image blocks obtained by dividing the image.

[0295] Alternatively, the image may be segmented using a target segmentation method or a fixed-size segmentation method. The image may be segmented to obtain one or more image blocks. Furthermore, the sizes of the image blocks obtained by segmentation may be the same or different.

[0296] For ease of understanding, in a specific example, a target in an image may be determined by a target detection algorithm, and the area where the target is located may be determined as a masked portion or a non-masked portion.

[0297] In another specific example, the image may also be divided into a number of image blocks of a fixed size or the same size, specifically 16 image blocks of the same size.

[0298] Optionally, corresponding mask parts may be determined for one or more image blocks, respectively; corresponding mask parts may also be determined for all or part of the image blocks, respectively.

[0299] This embodiment does not limit the method of determining the mask portion for the image block, and may be determined randomly or according to a fixed preset format.

[0300] For example, the mask portion may be randomly determined for each image block, or the area at a fixed position at the upper left corner of each image block may be determined as the mask portion.

[0301] In an optional embodiment, since the image is divided into several image blocks and considering that the image quality between different image blocks is different, the mask part can be determined based on the properties of the different image blocks themselves, thereby meeting the property requirements of the image blocks and improving the flexibility of the mask part.

[0302] To facilitate understanding, the following explanation is given using environmental factors as an example.

[0303] For a foggy image, usually nearby objects are less affected by the fog and can be distinguished relatively easily, but distant objects are often greatly affected by the fog and are difficult to distinguish.

[0304] After dividing the foggy image into several image blocks, the degree of fog influence on different image blocks is different. Therefore, the mask part can be determined in different ways according to the degree of fog influence on each image block, thereby improving the flexibility of the mask part.

[0305] It should be noted that the setting of the mask part is mainly to help the image processing model learn the relationship between the various parts of the image during the training process, so as to improve the image quality. A more flexible setting of the mask part can facilitate the image processing model to better learn the relationship between the various parts of the image and better improve the image quality.

[0306] This embodiment does not limit the attributes of a specific image block. Optionally, the attributes of an image block may specifically include at least one of the following: the image quality level of the image block itself, the difficulty of image reconstruction, the degree of influence by environmental factors, etc.

[0307] Corresponding to different attribute values ​​of the image block, there can be different mask determination methods.

[0308] For example, when environmental factors specifically affect image quality, the degree of impact varies across different parts of the image. For example, on a foggy day, nearby objects are less affected by the fog, while distant objects are more affected. Accordingly, since nearby objects are less affected by the fog, their image quality is higher and contains more information, making it easier to improve image quality. This can be achieved by setting a larger mask. On the other hand, distant objects are less affected by the fog, resulting in lower image quality and less information, which can be improved by setting a smaller mask.

[0309] Therefore, the image blocks can be divided and the mask parts can be determined based on different mask determination methods according to the image block attributes.

[0310] Optionally, for at least one image block, a corresponding mask part is determined respectively. Specifically, for at least one image block, the following steps are performed respectively: an image quality level corresponding to the image block is determined, and a corresponding mask part is determined based on a mask determination method corresponding to the determined image quality level.

[0311] Of course, the image quality level here is just one example of an image block attribute.

[0312] It can be understood that the image block attributes may include at least one of the following: the image quality level of the image block itself, the difficulty of image reconstruction, the degree of influence by environmental factors, and the like.

[0313] Correspondingly, for at least one image block, the corresponding mask part is determined respectively. Specifically, the following steps can be performed respectively for at least one image block: determining the image reconstruction difficulty corresponding to the image block, and determining the corresponding mask part based on the mask determination method corresponding to the determined image reconstruction difficulty.

[0314] In addition, for at least one image block, the corresponding mask part is determined respectively. Specifically, the following steps can be performed respectively for at least one image block: determining the degree of influence of environmental factors corresponding to the image block, and determining the corresponding mask part based on the mask determination method corresponding to the determined degree of influence of environmental factors.

[0315] This embodiment can determine the corresponding mask determination method by combining the attributes of the image block itself, thereby improving the flexibility of the mask part, facilitating the image processing model to better learn the relationship between the various parts of the image, and facilitating better improvement of image quality.

[0316] The following sections explain the image quality levels and the corresponding mask determination methods. For other image block attributes, such as the difficulty of image reconstruction and the degree of environmental influence, as well as the corresponding mask determination methods, please refer to the following explanations.

[0317] This method does not limit the specific form and determination method of the image quality level.

[0318] The image quality level can be used to measure the quality of an image. In a specific example, the image quality level can be divided into 1-5 levels, where level 1 has the lowest image quality and level 5 has the highest image quality.

[0319] It is understood that a specific calculation formula can be used to determine the image quality level, for example, based on multiple image attributes. Accordingly, when determining the image quality level of an image block, multiple attributes of the image block can be used for calculation. Specifically, for example, the image quality level can be calculated based on the clarity, brightness, contrast, etc. of the image block. Specifically, the clarity of the image block can be comprehensively normalized to determine the image quality level.

[0320] Of course, other methods can also be used to calculate the image quality level.

[0321] Machine learning can be considered, using a trained model to determine the image quality level for each image block. Specifically, corresponding training samples can be used, for example, using images or image blocks as sample features and corresponding set image quality levels as sample labels. The image quality levels set in the training samples can be manually set or calculated according to the above-mentioned calculation formula.

[0322] Since the above embodiment can be for a low-quality image, dividing the image blocks and determining the image quality levels of each block for determining the mask part, it is also possible to consider the difference between the low-quality image and the high-quality image to determine the image quality levels of the image blocks in the low-quality image.

[0323] For example, the difficulty of reconstructing a low-quality image into a high-quality image, or the image reconstruction difficulty, can be determined based on the difference in image blocks at the same location between the low-quality image and the high-quality image. Specifically, the image blocks at the same location between the low-quality image and the high-quality image can be foggy and fog-free, respectively. Based on the difference in image blocks, the image reconstruction difficulty, or the image dehazing difficulty, can be determined, which is used to measure and determine the image quality level.

[0324] Therefore, optionally, determining the image quality level corresponding to the image block can specifically be at least one of the following: 1) determining the image quality level corresponding to the image block based on a preset classification model; 2) determining the difference between the image block in the acquired low-quality image and the image block at the same position in the acquired high-quality image; and determining the image quality level corresponding to the image block based on the determined difference.

[0325] This embodiment does not limit the specific structure of the preset classification model. Optionally, the preset classification model can be an image classification model, and the specific structure can be an image convolutional network, an image neural network, etc. The preset classification model can be pre-trained.

[0326] In a specific example, the preset classification model can be a graph convolutional neural network, which can include a dimensionality reduction convolution layer and a fully connected layer for predicting the output image quality level.

[0327] This embodiment does not limit the specific preset classification model training method. Specifically, training can be performed using training samples with images or image blocks as sample features and corresponding set image quality levels as sample labels. Of course, other forms of training samples can also be used for training.

[0328] The preset classification model can be used to predict the corresponding image quality level for the input image or image block, and thus can play a role in predicting image quality.

[0329] Correspondingly, the image block can be directly input into the preset classification model to obtain the image quality level predicted by the preset classification model as the image quality level corresponding to the image block.

[0330] For ease of understanding, in a specific example, the image quality level can be determined for an image on a foggy day, specifically by identifying different fog densities. Correspondingly, the image quality level can be in the form of a fog density level.

[0331] The fog concentration levels are explained below.

[0332] ① The fog concentration is large, more information is needed for mask reconstruction, the image quality is low, and the image quality level is low.

[0333] ② The fog density is average, and certain information is required for mask reconstruction. The image quality is medium, and the image quality level is medium.

[0334] ③ The fog concentration is small, less information is required for mask reconstruction, the image quality is higher, and the image quality level is higher.

[0335] Therefore, the fog concentration levels can be divided into three categories. For the convenience of description, they are referred to as fog concentration levels 1-3. Level 1 has the highest fog concentration, and level 3 has the lowest fog concentration.

[0336] Afterwards, based on the above-mentioned division method of fog concentration levels, sample annotation can be performed and samples can be constructed.

[0337] It should be noted that in the same foggy image, different image blocks may have different fog densities due to the influence of near and far scenes. For example, near scenes are clearer and have lower fog density, while far scenes are blurry and have higher fog density.

[0338] Therefore, the fog concentration levels can be marked for different image blocks divided into foggy images, and training samples can be constructed to train the preset classification model.

[0339] The trained preset classification model can be used to predict the fog concentration level for images or image blocks.

[0340] Therefore, the fog concentration level of the input image block can be automatically determined according to the preset classification model.

[0341] This embodiment does not limit the specific method for determining the difference between image blocks. Optionally, the difference between image blocks can be determined based on the similarity between the image blocks. Specifically, the similarity between an image block in a low-quality image obtained and an image block at the same position in a high-quality image obtained can be determined. The higher the similarity, the smaller the difference. Thus, the difference can be determined based on the similarity. Specifically, the similarity can be inversely proportional to the difference. This embodiment does not specifically limit the method for determining the similarity between image blocks. Specifically, the similarity between image blocks can be determined using mean square error, peak signal-to-noise ratio, structural similarity algorithm, etc.

[0342] Of course, you can also consider using machine learning model training, or other formulas, or the difference between image blocks, etc. to determine the differences between image blocks.

[0343] This embodiment does not limit the manner in which the image quality level of an image block is determined based on the difference between the image blocks. Alternatively, the greater the difference between the image blocks, the lower the image quality represented by the image quality level.

[0344] This embodiment does not limit the specific mask determination method.

[0345] Optionally, the mask determination method may specifically be to randomly determine the mask for the image block, or to determine the mask at a fixed position for the image block. For example, the fixed position may be the upper left corner or the lower right corner of the image block.

[0346] Of course, the mask determination method may also be to further divide the image block into a plurality of image sub-blocks, and randomly determine a preset proportion of image sub-blocks as the mask part.

[0347] Since there is a corresponding relationship between the image quality level and the mask determination method, this embodiment does not limit the specific corresponding relationship.

[0348] Alternatively, the correspondence may be random without specific settings. For example, an image quality level representing higher image quality may be set to correspond to a random mask determination method, while an image quality level representing lower image quality may be set to correspond to a fixed position mask determination method.

[0349] In an optional embodiment, the image quality level may be flexibly set according to the image quality represented by the image quality level.

[0350] Specifically, considering that image blocks with lower image quality often require more image information when improving image quality, a better image quality improvement effect can be achieved. Therefore, a relatively small mask portion can be set.

[0351] Similarly, considering image blocks with higher image quality, when improving image quality, less image information can often be used, resulting in a better image quality improvement effect. Therefore, the proportion of the mask portion can be flexibly set, and the mask portion can be set larger or smaller.

[0352] Therefore, optionally, the lower the image quality represented by the image quality level is, the smaller the area proportion of the determined mask part in the corresponding image block is.

[0353] Of course, when the image quality represented by the image quality level is lower than the preset image quality, the lower the image quality represented by the image quality level, the smaller the area proportion of the determined mask part in the corresponding image block.

[0354] For the case where the image quality represented by the image quality level is higher than the preset image quality, it can be set more flexibly. A fixed mask area ratio or a random mask area ratio can be used to better learn the association between various parts in the image.

[0355] This embodiment does not limit the specific flexible setting method. For ease of understanding, in an optional embodiment, it can be considered to further divide the image block into more detailed image sub-blocks, so that the mask part can be determined for the image sub-blocks.

[0356] Specifically, for each image sub-block divided in a single image block, a designated image sub-block may be determined as the mask part.

[0357] This embodiment does not limit the specific method, number, and size of dividing the image block to obtain image sub-blocks.

[0358] Optionally, the image can be segmented using a target segmentation method or a fixed-size segmentation method. The image can be segmented into one or more sub-blocks. Furthermore, the sizes of the sub-blocks can be the same or different.

[0359] In a specific example, the image block may be divided into a number of image sub-blocks of a fixed size or the same size, specifically 16 image sub-blocks of the same size.

[0360] It should be noted that, when specifically dividing the image blocks, further division may not be performed and the image blocks themselves may be directly retained.

[0361] Furthermore, considering that the mask portion is determined for each image block individually, the distribution is relatively uniform. To further enhance flexibility, the mask portion can be determined jointly for multiple image blocks. Specifically, for multiple image blocks, some image sub-blocks can be comprehensively determined as the mask portion. Specifically, the multiple image blocks can be image blocks with the same image quality level.

[0362] Furthermore, the image quality level may be associated with the way in which the image block is divided into image sub-blocks.

[0363] This embodiment does not limit the manner of dividing the image sub-blocks associated with the specific image quality level.

[0364] Optionally, different image quality levels may correspond to different ways of dividing an image block into image sub-blocks. Specifically, different image quality levels may correspond to different numbers or ratios of image sub-blocks obtained by division. Of course, different image quality levels may also correspond to ways of determining image sub-blocks by object detection, or ways of dividing an image block into image sub-blocks of fixed sizes.

[0365] Specifically, the image sub-blocks may be obtained by randomly dividing the image quality levels in correspondence with each other, without setting a specific correspondence relationship.

[0366] Taking into account the image quality, flexible settings can be made according to the image quality represented by the image quality level.

[0367] Specifically, considering image blocks with lower image quality, more continuous image information is often required when improving image quality, which can have a better image quality improvement effect. Therefore, fewer image sub-blocks can be set as mask parts, or continuous image sub-blocks can be set as non-mask parts.

[0368] Similarly, considering image blocks with higher image quality, improving image quality often requires less or discontinuous image information, while still achieving a better image quality improvement effect. Therefore, the number and continuity of image sub-blocks can be flexibly set. Specifically, a larger or smaller number of image sub-blocks can be used as masking portions, continuous or discontinuous image sub-blocks can be used as non-masking portions, and so on.

[0369] The continuous image sub-blocks may specifically be adjacent image sub-blocks.

[0370] In a specific example, an image block can be divided into image sub-blocks of equal size, and different image quality levels can correspond to different numbers of image sub-blocks obtained by division. The greater the number of divided image sub-blocks, the more likely discontinuous image sub-blocks (i.e., non-adjacent image sub-blocks) are to be determined as non-masked portions when randomly determining image sub-blocks as mask portions.

[0371] Therefore, optionally, the method for determining the mask corresponding to the image quality level can be: for the image block, further dividing it into a number of image sub-blocks corresponding to the image quality level; between image blocks with the same image quality level, determining the image sub-blocks with a proportion corresponding to the image quality level as the mask part.

[0372] This embodiment does not limit the specific correspondence between the image quality level and the corresponding quantity. Optionally, there may be a flexible correspondence between the image quality level and the corresponding quantity, and there is no positive or negative correlation. Optionally, the lower the image quality represented by the image quality level, the smaller the corresponding quantity can be, and the fewer image sub-blocks can be obtained by division. In the case where the divided image sub-blocks need to maintain the same size, the fewer the divided image sub-blocks are and the larger the size is, the more continuous image information can be retained as the non-mask portion. Of course, the lower the image quality represented by the image quality level, the larger the corresponding quantity can be.

[0373] Similarly, this embodiment does not limit the specific correspondence between image quality levels and corresponding ratios. Alternatively, a flexible correspondence between image quality levels and corresponding ratios can exist, without a positive or negative correlation. Alternatively, the lower the image quality represented by the image quality level, the smaller the corresponding ratio can be, meaning that fewer image sub-blocks can be determined as masked portions. Of course, the lower the image quality represented by the image quality level, the larger the corresponding ratio can be.

[0374] Optionally, the lower the image quality represented by the image quality level, the smaller the corresponding number may be; and the lower the image quality represented by the image quality level, the smaller the corresponding ratio may be.

[0375] In addition, it is understandable that the above-mentioned image quality level may also be replaced by other attributes of the image block, such as the difficulty of image reconstruction, or the degree of influence by environmental factors.

[0376] This embodiment can determine the corresponding mask determination method and the more specific method of dividing the image sub-blocks by combining the attributes of the image block itself, thereby improving the flexibility of the mask part, facilitating the image processing model to better learn the relationship between the various parts of the image, and facilitating better improvement of image quality.

[0377] For ease of understanding, a specific example of foggy image segmentation is given below.

[0378] As shown in FIG3 , FIG3 is a schematic diagram showing a principle of image division according to an embodiment of the present invention.

[0379] The foggy image may be first divided into 4*4 image blocks of the same size.

[0380] Furthermore, a preset classification model can be used to determine the fog density level of each image block.

[0381] The fog concentration levels are explained below.

[0382] ① The fog concentration is large, more information is needed for mask reconstruction, the image quality is low, and the image quality level is low.

[0383] ② The fog density is average, and certain information is required for mask reconstruction. The image quality is medium, and the image quality level is medium.

[0384] ③ The fog concentration is small, less information is required for mask reconstruction, the image quality is higher, and the image quality level is higher.

[0385] Therefore, the fog concentration levels can be divided into three categories. For the convenience of description, they are referred to as fog concentration levels 1-3. Level 1 has the highest fog concentration, and level 3 has the lowest fog concentration.

[0386] For fog density level 1, the image block size can be kept without further division.

[0387] For level 2 fog density, the image can be further divided into 2*2 image sub-blocks of the same size.

[0388] For level 3 fog density, the image can be further divided into 4*4 image sub-blocks of the same size.

[0389] According to the divided image blocks, it can be determined that the image blocks in the first row are of level 1 fog density, the image blocks in the second row are of level 2 fog density, and the image blocks in rows 3-4 are of level 3 fog density, so that the image sub-blocks can be divided accordingly.

[0390] Then, for each image sub-block with the same fog density, the corresponding proportion or number of image sub-blocks can be randomly determined as the mask part. Specifically, this is displayed by filling it with gray.

[0391] In addition, it should be noted that the above embodiment explains the method of determining the mask part.

[0392] It is understood that there are many different ways to determine the mask portion, and many different possible results.

[0393] The same mask portion determination method may also determine the same or different mask portions.

[0394] Therefore, different mask part determination methods can be used to obtain multiple different mask images, or the same mask part determination method can be used multiple times to obtain multiple different mask images, which are used to form training samples, thereby increasing the number of training samples, reducing the difficulty of obtaining training samples, and improving the training effect of the image processing model.

[0395] The above embodiments explain how to determine the mask portion. It is understandable that the above embodiments can be applied in the process of constructing training samples to determine the mask portion and further determine the mask image.

[0396] Specifically, the mask part may be determined for the high-quality image to obtain the mask image, or the mask part may be determined for the low-quality image to obtain the mask image by combining the low-quality image with the high-quality image. For specific explanations, please refer to the above.

[0397] 4. S101: Obtain a training sample set.

[0398] Among them, any training sample in the training sample set uses the mask image as the sample feature and the high-quality image as the sample label.

[0399] In the same training sample, the mask image and the high-quality image have the same image content and image size but different image quality. The image quality of the masked part in the mask image is lower than the image quality of the same position area in the high-quality image. The pixel values ​​of the non-masked part in the mask image are the same as the pixel values ​​of the same position area in the high-quality image.

[0400] The above embodiment explains the construction of training samples and the method of determining the mask part. The present method process does not limit the form and content of the training sample set, as long as it includes the training samples constructed in the above embodiment.

[0401] The present method does not limit the number of training samples included in the training sample set. The training sample set may include one or more training samples, specifically training samples constructed according to the above embodiment.

[0402] The present method does not limit the method for obtaining the training sample set. Optionally, the training sample set can be pre-constructed by training samples, stored in a storage area in advance, and retrieved from the storage area when needed; or the training sample set can be constructed in real time, specifically, the training sample set can be constructed in real time.

[0403] It should be noted that, since the above embodiments describe various different mask portion determination methods, various mask portion determination methods can be employed to determine various mask portions for use in constructing training samples. Of course, different mask portion determination methods can also be employed to determine various mask portions for use in constructing training samples during different real-time training sample set constructions.

[0404] 5. S102: Obtain an initial image processing model.

[0405] 1. About the initial image processing model.

[0406] The method flow does not limit the specific structure of the initial image processing model.

[0407] Optionally, the initial image processing model can be a graph convolutional network, a graph neural network, a residual network, etc. Of course, the model structure included in the initial image processing model is not limited, and can include an encoding layer, a decoding layer, a fully connected layer, a convolutional layer, etc.

[0408] In a specific example, the initial image processing model may be a unet model.

[0409] It is understandable that the initial image processing model can be used to improve image quality, which means that the image quality needs to be improved for the input image, and the predicted output quality is higher than the predicted image of the input image.

[0410] While the image sizes of the mask image and the high-quality image in the training samples are the same, the image sizes between the input image and the predicted image of the initial image processing model can also be the same.

[0411] 2. This method does not limit the specific way of obtaining the initial image processing model.

[0412] Optionally, a pre-stored initial image processing model may be directly obtained, or the initial image processing model may be constructed and obtained in real time. The initial image processing model may be a pre-trained model or an untrained model.

[0413] 3. For ease of understanding, a specific embodiment of the initial image processing model is given below.

[0414] In an optional embodiment, in order to reduce the amount of calculation and improve the prediction efficiency of the image processing model, the structure in the initial image processing model can be flexibly set.

[0415] Among them, the overall image processing process of first reducing the dimension and then increasing the dimension can be adopted to help extract image features and reduce the amount of calculation.

[0416] Optionally, in the initial image processing model, dimensionality reduction processing can be performed on the image first to extract image features. Since the amount of feature map data obtained after dimensionality reduction processing is smaller than the amount of feature map data obtained by directly extracting image features without dimensionality reduction processing 3, the amount of calculation can be reduced.

[0417] This embodiment does not limit the specific dimensionality reduction processing and image feature extraction methods.

[0418] Specifically, the dimensionality of the input image can be reduced first, and then the features can be extracted. That is, the dimensionality of the input image can be reduced by downsampling first, and then the feature map can be extracted. Alternatively, a convolution kernel can be used to extract the feature map of the input image, and the dimension of the extracted feature map is smaller than that of the input image. Specifically, the feature map can be smaller than the input image in terms of resolution (width and height), so that dimensionality reduction and feature extraction can be achieved simultaneously.

[0419] Correspondingly, in order to output a predicted image, the extracted feature map may be further subjected to dimensionality upgrading processing, specifically, the dimensionality upgrading processing may be performed in combination with information in multiple channels of the feature map.

[0420] This embodiment does not limit the specific dimensionality increase processing method. Optionally, the dimensionality increase processing can be performed by upsampling or deconvolution.

[0421] Therefore, optionally, the initial image processing model may include: a dimensionality reduction coding layer and a dimensionality increase decoding layer; the dimensionality reduction coding layer may be used to: perform dimensionality reduction convolution on the input image to obtain a feature map; the dimensionality increase decoding layer may be used to: upsample the input feature map to output a predicted image.

[0422] This embodiment does not limit the specific dimensionality reduction convolution method, nor does it limit the specific upsampling method, and deconvolution may be used.

[0423] Among them, the output of the dimensionality reduction coding layer can be cascaded with the input of the dimensionality increase decoding layer, so that the output feature map of the dimensionality reduction coding layer can be input into the dimensionality increase decoding layer for processing.

[0424] Of course, the input feature map of the dimension-raising decoding layer can also be the feature map output by other model structures in the initial image processing model. For details, please refer to the following explanation.

[0425] This embodiment does not limit the specific structures of the dimensionality reduction coding layer and the dimensionality increase decoding layer.

[0426] Optionally, the dimensionality reduction encoding layer may include one or more convolutional layers, one or more fully connected layers, one or more residual layers, one or more feature extraction layers, one or more downsampling layers, etc. Similarly, the dimensionality increase decoding layer may include one or more deconvolutional layers, one or more upsampling layers, one or more fully connected layers, one or more residual layers, one or more feature extraction layers, etc.

[0427] In addition, optionally, the mask part in the mask image is used in the process of this method to help the image processing model learn the relationship between the various parts of the image. In order to facilitate further learning of the relationship between the various parts of the image, the initial image processing model can include a fully connected layer.

[0428] Among them, the fully connected layer can be used to learn the association between various parts of the image.

[0429] Optionally, the initial image processing model may include a fully connected layer; the fully connected layer may be used to predict and output a predicted feature map of the same size as the input feature map.

[0430] This embodiment does not limit the specific prediction method of the fully connected layer. Optionally, the fully connected layer can be used to predict the values ​​of each pixel in the output feature map based on the entire input feature map, or it can be used to predict each part of the output feature map based on the entire input feature map.

[0431] Specifically, it can be understood that according to the values ​​of all pixels in the input feature map, predictions are made for each pixel value in the output feature map, thereby achieving fully connected predictions and exploring the associations between different pixel values ​​in the feature map, which makes it easier to learn the associations between different parts of the image.

[0432] It can also be understood that according to each part of the input feature map, prediction is made for each part of the output feature map respectively, thereby realizing full connection prediction, and exploring the relationship between different parts in the feature map, which is convenient for learning the relationship between the various parts of the image.

[0433] For ease of understanding, the input feature map of the fully connected layer can contain 9 pixels. Based on the values ​​of these 9 pixels, the values ​​of the 9 pixels in the output feature map can be predicted. The value of each pixel in the output feature map can be determined based on the values ​​of all pixels in the input feature map, thus achieving fully connected prediction.

[0434] The feature map of the input fully connected layer can also be divided into 9 regions, and the 9 regions in the output feature map can be predicted based on these 9 regions. Each region in the output feature map can be determined based on the 9 regions in the input feature map, thus achieving fully connected prediction.

[0435] Therefore, optionally, the fully connected layer can be used to: divide the input feature map into a preset number of feature image blocks; based on the preset number of feature image blocks, respectively predict a preset number of predicted image blocks; and splice the obtained predicted image blocks into a predicted feature map.

[0436] The size of the predicted feature map can be the same as the size of the input feature map.

[0437] The sizes of the divided feature image blocks may be the same, the sizes of the predicted image blocks may be the same, and the sizes of the feature image blocks and the predicted image blocks may be the same.

[0438] This embodiment is not limited to the feature map of the input fully connected layer. Specifically, it can be the input image of the initial image processing model, or it can be the feature map output by other model structures in the initial image processing model, such as the feature map output by the dimensionality reduction coding layer.

[0439] Therefore, the fully connected layer can optionally be used to: divide the feature map obtained by the dimensionality reduction coding layer into a preset number of feature image blocks; predict a preset number of predicted image blocks based on the preset number of feature image blocks; and concatenate the obtained predicted image blocks into a predicted feature map. The predicted feature map can then be input into the dimensionality increase decoding layer.

[0440] This embodiment can be based on the fully connected layer in the initial image processing model to facilitate better learning of the association between various parts in the image and improve the training effect of the image processing model.

[0441] This embodiment does not limit the specific processing flow of the initial image processing model.

[0442] Optionally, the specific image processing process of the initial image processing model can be: the initial image processing model is used to: perform dimensionality reduction convolution on the input mask image to obtain a feature map; upsample the obtained feature map to output a predicted image; the predicted image and the input mask image can have the same size.

[0443] The specific image processing process of the initial image processing model can also be: the initial image processing model is used to: perform dimensionality reduction convolution on the input mask image to obtain a feature map; input the obtained feature map into the fully connected layer to obtain a fully connected feature map; upsample the obtained fully connected feature map to output a predicted image; the predicted image and the input mask image can have the same size.

[0444] Of course, it can also be combined with the structure of the residual network. The initial image processing model can be used to: perform dimensionality reduction convolution on the input mask image to obtain a feature map; upsample the obtained feature map to obtain an upsampled image, and the upsampled image and the input mask image have the same size; combine the upsampled image and the input mask image (specifically, they can be added) to obtain a predicted image and output it; the predicted image and the input mask image can have the same size.

[0445] The loss can then be determined based on the predicted image and the high-quality image (i.e., sample label) corresponding to the input mask image to train the initial image processing model.

[0446] 6. S103: Train the initial image processing model based on the training sample set to obtain a trained result image processing model; the result image processing model is used to improve image quality.

[0447] The above embodiment explains a specific initial image processing model, and the present method flow is not limited to a specific training method.

[0448] The specific training method can be to input the training samples in the training sample set into the initial image processing model in batches, update the model parameters according to the model loss, until the preset training stop condition is met, and obtain the final result image processing model.

[0449] This embodiment does not limit the specific preset training stop conditions, which may be that the number of training times reaches a preset number, the number of training cycles reaches a preset number, the model loss is less than a preset model loss threshold, the difference between the updated model parameters and the previous model parameters is less than a preset difference, etc.

[0450] This embodiment does not limit the specific method for calculating the model loss. Specifically, the model loss may be calculated using image similarity or image distinction, or using methods such as mean square error. Specifically, the difference between the predicted image and the high-quality image (i.e., the sample label) in the training sample may be calculated as the model loss.

[0451] Optionally, since the mask image and the high-quality image can be consistent in the non-mask part in the same training sample, we can only focus on improving the image quality of the mask part, and thus only calculate the loss of the mask part.

[0452] Optionally, the loss function of the initial image processing model can be used to determine the loss for the mask portion in the predicted image output by the initial image processing model and the same position area in the high-quality image in the corresponding sample.

[0453] This embodiment can reduce the amount of model loss calculation, and can focus on improving the image quality of the mask part, thereby facilitating the improvement of the convergence speed of the image processing model.

[0454] Of course, the loss of the entire image can also be calculated directly. Specifically, the loss can be determined for the predicted image output by the initial image processing model and the high-quality image in the corresponding sample.

[0455] The specific method of determining the loss is not limited, and it can be determined based on image similarity or difference, or by calculating the mean square error.

[0456] 7. How to apply the model.

[0457] After the training is completed, the resulting image processing model can be obtained.

[0458] This method does not limit the specific way of using the result image processing model.

[0459] In an optional embodiment, based on the above explanation of specific scenarios for improving image quality, this method process can be applied to a variety of different scenarios, such as defogging scenes, deraining scenes, desanding scenes, reducing motion blur, improving night shooting effects, etc.

[0460] Specifically, the resulting image processing model can be directly used to improve image quality in the above scenarios. Of course, you can also consider using more concentrated training samples to optimize the effect of the resulting image processing model in specific scenarios.

[0461] For example, in defogging scenarios, foggy images are mainly used as mask images, and images taken in clear weather are used as high-quality images to construct training samples for training image processing models, thereby improving the defogging effect of the resulting image processing model and applying it in scenarios where defogging is required.

[0462] Similarly, for scenes shot at night, we can focus on using images with insufficient lighting as mask images and images with sufficient lighting as high-quality images to construct training samples for training the image processing model. The resulting image processing model can be used to optimize night shooting effects and can be applied to scenes shot at night.

[0463] For blurred images caused by hand shaking, images taken through glass, images with water mist on the lens, etc., training samples in specific scenarios can also be used for optimization training, so that the resulting image processing model can be applied in specific scenarios.

[0464] In an optional embodiment, since the resulting image processing model can be used to improve image quality, it can also be used to process tasks that require improving image quality.

[0465] In a specific example, for the task of image target detection, the captured image is generally used directly for target detection. However, due to the timing of the shooting or the weather, blur, shaking or insufficient lighting may occur. Therefore, the resulting image processing model trained using the above method embodiment can be used to improve the image quality, and target detection can be performed on the image with improved quality, thereby improving the efficiency and accuracy of target detection.

[0466] Similarly, in other examples, for other image processing tasks, such as image detection, image classification, image segmentation, etc., the input image can first be trained using the image processing model obtained by the above method embodiment to improve the image quality, and then the improved image can be processed accordingly.

[0467] Furthermore, for tasks that simply require improving image quality, the resulting image processing model can be used directly to improve image quality. For example, when displaying images to users or when users need to process images, quality improvement functionality can be provided and the resulting image processing model can be used to improve image quality.

[0468] In another optional embodiment, since the resulting image processing model can be used to improve image quality, knowledge that can improve image quality is learned, so that the parameters in the resulting image processing model can be used to help build and train other image processing task models to improve the effectiveness of other models.

[0469] For example, for the target detection model, the parameters in the resulting image processing model can be combined to improve image quality and perform corresponding processing, and the association between different parts of the image can also be learned.

[0470] Therefore, the parameters in the resulting image processing model can be used to help build or train other image processing task models.

[0471] The process of this method is not limited to a specific method. The encoding layer in the result image processing model can be directly used as the encoding layer in other image processing task models, or the decoding layer in the result image processing model can be used as the decoding layer in other image processing task models, and so on.

[0472] Of course, the parameters of the resulting image processing model can also be added to other image processing task models. Specifically, the encoding layer, decoding layer, or fully connected layer in the resulting image processing model can be added.

[0473] Therefore, optionally, the method flow may also include: constructing the coding layer of other image processing task models based on the coding layer in the result image processing model; or improving the image quality of any image based on the result image processing model.

[0474] Of course, there may be other ways to use the result image processing model, and the above embodiments are for illustrative purposes only.

[0475] 8. The embodiment of the present invention also provides another training method process for an image processing model.

[0476] The above method flow mainly explains the specific process of single training, and this manual also provides another training method for an image processing model, which can be performed multiple times. During each training, the mask part can be reset and the training samples can be reconstructed for training the image processing model to improve the training effect.

[0477] This method does not limit the way of constructing training samples. Please refer to the explanation of the above method.

[0478] For the convenience of description, this method process chooses to explain the method of constructing training samples by combining low-quality images and high-quality images, which can be replaced by other training sample construction methods.

[0479] The following is a detailed explanation of another image processing model training method provided by an embodiment of the present invention.

[0480] As shown in FIG4 , FIG4 is a flow chart of another method for training an image processing model according to an embodiment of the present invention.

[0481] The embodiment of the present invention does not limit the execution subject of the method process. Optionally, the execution subject can be any computing device, such as an image processing terminal, an image processing server, an image processing client, etc.

[0482] The method may include the following steps.

[0483] S201: Acquire several groups of image pairs.

[0484] Any set of image pairs includes low-quality images and high-quality images; in any set of image pairs, the low-quality images and the high-quality images have the same image content and image size but different image quality, and the image quality of the low-quality images is less than the image quality of the high-quality images.

[0485] S202: Acquire an initial image processing model, and determine the acquired initial image processing model as the current image processing model.

[0486] S203: Execute the following steps in a loop until a preset loop stop condition is met:

[0487] Based on the acquired image pairs, construct training samples;

[0488] For any training sample constructed, the mask image is used as the sample feature and the high-quality image is used as the sample label. In the same training sample, the mask image and the high-quality image have the same image content and image size but different image quality. The image quality of the masked part in the mask image is lower than the image quality of the same position area in the high-quality image. The pixel values ​​of the non-masked part in the mask image are the same as the pixel values ​​of the same position area in the high-quality image.

[0489] The current image processing model is trained based on the constructed training samples, and the image processing model after training and updating is re-determined as the current image processing model.

[0490] S204: After the loop ends, the current image processing model is determined as the result image processing model.

[0491] The above method process uses mask images and high-quality images to train the image processing model, so that the trained image processing model can be used to improve image quality. Compared with fixed algorithms, it is more flexible and convenient to better improve image quality.

[0492] The above method flow can also construct training samples multiple times in a cycle and perform multiple trainings to improve the training effect of the image processing model.

[0493] Furthermore, the above method flow can facilitate learning the association between various parts of the image during the training process by setting an image mask, thereby improving the training effect of the image processing model and facilitating better improvement of image quality.

[0494] In addition, since the non-masked part of the masked image remains consistent with the high-quality image, it is convenient to calculate the loss, improve the efficiency of model training, focus on improving the image quality of the masked part, and better learn the relationship between the various parts of the image, thereby improving the model training effect.

[0495] It should be noted that this method does not limit the specific method of constructing training samples in each cycle. Training samples can be constructed based on the image pairs obtained in S201, or other methods can be used to construct training samples. For details, please refer to the explanation of S101-S103 of the above method flow.

[0496] The process of this method does not limit the specific training method and training stop conditions in each cycle.

[0497] Specifically, the loss of the current image processing model can be determined based on the currently constructed training sample set, and the model parameters can be further updated. The training stopping condition can be that the number of model parameter updates exceeds a preset threshold, or the determined loss is less than a preset loss.

[0498] The method flow does not limit the loop stopping condition, which can be specifically when the number of loops reaches a preset number, or when the difference between the current image processing models of the two previous and next loops is less than a preset difference.

[0499] In addition, it is understandable that, in each cycle of the present method, the training samples constructed may be newly constructed, but the training samples constructed in the previous cycle may also be used during the training process to further improve the training effect.

[0500] In an optional embodiment, the image pairs acquired in S201 may be combined to construct training samples.

[0501] Optionally, training samples are constructed based on the acquired image pairs, specifically: for any group of image pairs, the low-quality image is divided into several image blocks, and the mask part of at least one image block is determined using a preset classification model; in the acquired low-quality image, the pixel values ​​of the determined non-masked part are replaced with the pixel values ​​of the same position area in the high-quality image to obtain a mask image; and the training sample is constructed using the mask image as the sample feature and the high-quality image as the sample label.

[0502] For the specific method of determining the mask part by the preset classification model, please refer to the explanation of the above method flow S101-S103.

[0503] Specifically, a preset classification model can determine the image quality level for each image block in the low-quality image. The mask portion of each image block is then determined based on the mask determination method corresponding to the image quality level. The preset classification model can include a coding layer and a fully connected layer to predict the image quality level of the image block. For more details, please refer to the above.

[0504] Among them, the preset classification model can be trained together with the loop.

[0505] Optionally, in each cycle, the coding layer parameters of the preset classification model may be updated to the coding layer parameters of the current image processing model, so that the preset classification model and the current image processing model can share the coding layer parameters.

[0506] For an explanation of the process of this method, please refer to the explanation of the above process S101-S103.

[0507] 9. The embodiment of the present invention also provides an image processing model training method process for a specific defogging scenario.

[0508] The following is a detailed explanation of a training method for an image processing model provided by an embodiment of the present invention.

[0509] As shown in FIG5 , FIG5 is a flow chart of another method for training an image processing model according to an embodiment of the present invention.

[0510] The embodiment of the present invention does not limit the execution subject of the method process. Optionally, the execution subject can be any computing device, such as an image processing terminal, an image processing server, an image processing client, etc.

[0511] The method may include the following steps.

[0512] S301: Obtain a training sample set.

[0513] Among them, any training sample in the training sample set uses the mask image as the sample feature and the high-quality image as the sample label.

[0514] In the same training sample, the mask image and the high-quality image have the same image content and image size but different degrees of fog influence. The degree of fog influence on the masked part in the mask image is greater than the degree of fog influence on the same position area in the high-quality image.

[0515] S302: Acquire an initial image processing model.

[0516] S303: Train an initial image processing model based on the training sample set to obtain a trained result image processing model; the result image processing model is used to reduce the degree of fog influence.

[0517] It is understandable that reducing the impact of fog can improve image quality.

[0518] Optionally, the pixel values ​​of the non-masked portion in the mask image are the same as the pixel values ​​of the same position area in the high-quality image.

[0519] Optionally, the method for constructing a training sample includes: obtaining a high-quality image and a mask image;

[0520] The obtained mask image and the high-quality image have the same image content and image size but different degrees of fog influence; the degree of fog influence on the masked portion of the obtained mask image is greater than the degree of fog influence on the same location area in the obtained high-quality image; the pixel values ​​of the non-masked portion of the obtained mask image are the same as the pixel values ​​of the same location area in the obtained high-quality image;

[0521] The obtained mask image is used as the sample feature and the obtained high-quality image is used as the sample label to construct the training sample.

[0522] Optionally, obtaining a high-quality image and a mask image includes: obtaining a high-quality image; determining a mask portion for the obtained high-quality image, and increasing the fog influence degree of the determined mask portion to obtain the mask image.

[0523] Optionally, obtaining a high-quality image and a mask image includes: obtaining a low-quality image and a high-quality image; the obtained low-quality image and the obtained high-quality image have the same image content and image size but different degrees of fog influence, and the degree of fog influence of the obtained low-quality image is greater than the degree of fog influence of the obtained high-quality image; and obtaining a mask image based on the obtained low-quality image and the obtained high-quality image.

[0524] Optionally, obtaining a low-quality image and a high-quality image includes: obtaining a high-quality image; and increasing the degree of fog influence on the obtained high-quality image to obtain a low-quality image.

[0525] Optionally, obtaining a mask image based on the acquired low-quality image and high-quality image includes:

[0526] determining a mask portion based on the acquired low-quality image and the acquired high-quality image;

[0527] The mask image is obtained by replacing the pixel values ​​of the masked part determined in the high-quality image with the pixel values ​​of the same position area in the low-quality image; or, replacing the pixel values ​​of the non-masked part determined in the low-quality image with the pixel values ​​of the same position area in the high-quality image; or, combining the pixel values ​​of the non-masked part determined in the high-quality image and the pixel values ​​of the masked part in the low-quality image.

[0528] Optionally, determining a mask portion based on the acquired low-quality image and high-quality image includes:

[0529] Divide the acquired low-quality image into several image blocks;

[0530] For at least one image block, a corresponding mask part is determined respectively.

[0531] Optionally, for at least one image block, a corresponding mask part is determined respectively, including: for at least one image block, the following steps are performed respectively: determining the fog impact level corresponding to the image block, and determining the corresponding mask part based on a mask determination method corresponding to the determined fog impact level.

[0532] Optionally, determining the fog impact level corresponding to the image block includes: determining the fog impact level corresponding to the image block based on a preset classification model; or determining the difference between the image block in the acquired low-quality image and the image block at the same position in the acquired high-quality image; and determining the fog impact level corresponding to the image block based on the determined difference.

[0533] Optionally, the lower the image quality represented by the fog impact level, the smaller the area proportion of the determined mask portion in the corresponding image block.

[0534] Optionally, the method for determining the mask corresponding to the fog impact level includes: further dividing the image block into a number of image sub-blocks corresponding to the fog impact level; and determining the image sub-blocks corresponding to the proportion of the fog impact level between image blocks with the same fog impact level as the mask part.

[0535] Optionally, the lower the image quality represented by the fog impact level, the smaller the corresponding number; the lower the image quality represented by the fog impact level, the smaller the corresponding ratio.

[0536] Optionally, the initial image processing model includes: a dimensionality reduction coding layer and a dimensionality increase decoding layer; the dimensionality reduction coding layer is used to: perform dimensionality reduction convolution on the input image to obtain a feature map; the dimensionality increase decoding layer is used to: upsample the input feature map to output a predicted image.

[0537] Optionally, the initial image processing model further includes: a fully connected layer; the fully connected layer is used to: predict and output a predicted feature map of the same size as the input feature map.

[0538] Optionally, the fully connected layer is used to: divide the input feature map into a preset number of feature image blocks; based on the preset number of feature image blocks, respectively predict a preset number of predicted image blocks; and splice the obtained predicted image blocks into a predicted feature map.

[0539] Optionally, the fully connected layer is used to: divide the feature map obtained by the dimensionality reduction coding layer into a preset number of feature image blocks; based on the preset number of feature image blocks, respectively predict a preset number of predicted image blocks; and splice the obtained predicted image blocks into a predicted feature map.

[0540] Optionally, the initial image processing model is used to: perform dimensionality reduction convolution on the input mask image to obtain a feature map; upsample the obtained feature map to output a predicted image; and the predicted image and the input mask image have the same size.

[0541] Optionally, the above method flow may further include: constructing coding layers of other image processing task models based on the coding layers in the result image processing model; or reducing the degree of fog influence based on the result image processing model for any image.

[0542] Optionally, the loss function of the initial image processing model is used to determine the loss for a mask portion in a predicted image output by the initial image processing model and a region at the same position in a high-quality image in a corresponding sample.

[0543] For an explanation of the process of this method, please refer to the explanation of the above process S101-S103.

[0544] 10. Specific applications

[0545] The embodiment of the present invention also provides a specific image defogging method process.

[0546] The image defogging method process may be based on an image processing model trained in any of the above method embodiments to perform image defogging.

[0547] Specifically, the following steps may be included: inputting the target image into a preset image processing model to obtain a defogging image output by the preset image processing model.

[0548] The preset image processing model may be pre-trained based on the training method of the image processing model in any of the above embodiments.

[0549] It should be noted that the preset image processing model can be used to improve image quality. For images containing fog or affected by foggy weather, the impact of fog or foggy weather on image quality can be reduced by improving image quality.

[0550] In the above method embodiment (the above method process S301-S303), the image processing model trained for the defogging scene is also explained. Using this image processing model can better perform image defogging and improve the image defogging effect.

[0551] Of course, it is understood that embodiments of the present invention may also provide image rain removal methods, image dust removal methods, image steam removal methods, etc. Specifically, the target image may be input into a preset image processing model to achieve corresponding image quality improvement, or an image processing model trained for a specific scenario may be used.

[0552] For ease of understanding, the above image defogging method can be applied in a variety of specific scenarios.

[0553] Optionally, the image defogging method can be applied to at least one of the following: an outdoor foggy vehicle display task, an outdoor foggy vehicle detection task, and an outdoor foggy pedestrian detection task.

[0554] Specifically, in the case of outdoor foggy weather, the on-board display task can usually display images of the vehicle's surrounding environment, thereby facilitating assisted driving. However, foggy weather has an impact on the quality of the displayed image, so the image processing model trained in the above method embodiment can be used to improve the image quality. Specifically, the above-mentioned image defogging method can be used to defog the image before displaying it, thereby improving the quality of the displayed image and facilitating better assisted driving.

[0555] For outdoor foggy vehicle detection tasks, specifically, the task of capturing images and detecting vehicles in foggy conditions can be performed. Since foggy conditions affect the quality of captured images, the image processing model trained in the above method embodiments can be used to improve image quality. Specifically, the above image defogging method can be used to defog the image before performing target (vehicle) detection, thereby improving the success rate and accuracy of vehicle detection.

[0556] Similarly, for outdoor foggy pedestrian detection tasks, specifically, the task of detecting pedestrians in images captured outdoors under foggy conditions can be performed. Since foggy weather affects the quality of captured images, the image processing model trained in the above method embodiment can be used to improve image quality. Specifically, the above image defogging method can be used to defog the image before performing target (pedestrian) detection, thereby improving the success rate and accuracy of pedestrian detection.

[0557] It is understandable that other image processing tasks on other outdoor foggy days can also be processed using the above-mentioned result image processing model or the above-mentioned image defogging method process.

[0558] 11. Application examples

[0559] For ease of understanding, the embodiment of the present invention further provides an application embodiment.

[0560] Target detection, segmentation, and classification in foggy conditions have always been important research topics in the security and surveillance field. Due to the low visibility in foggy conditions, heavy fog significantly increases the difficulty of these downstream tasks.

[0561] The industry now widely utilizes deep learning methods, using convolutional neural networks (CNNs) to train downstream foggy tasks (such as foggy object detection and foggy segmentation). However, due to the complexity of these tasks, each downstream task requires extensive training and dataset annotation. Weights between these tasks are not shared, making model training extremely expensive.

[0562] Existing image mask modeling pre-training methods are almost all designed for common image classification, segmentation, and detection tasks.

[0563] Regarding the downstream tasks in foggy scenarios mentioned in this example, there are currently no specialized fog image mask modeling methods designed for downstream tasks with varying fog concentrations. Instead, directly applying the aforementioned image mask modeling methods designed for general image classification, segmentation, and detection tasks to downstream tasks in foggy conditions can actually reduce the accuracy of these downstream tasks.

[0564] Therefore, it is particularly important to design a pre-training algorithm that is suitable for mask modeling of images with different fog concentrations.

[0565] This embodiment designs a CNN-based pre-training algorithm for adaptive mask modeling based on fog concentration. To address the problems that ① existing foggy downstream tasks (foggy target detection, foggy segmentation, etc.) require a large amount of training cost and annotation cost respectively, and ② current pre-training algorithms are basically not designed for CNN networks and foggy days with different concentrations, this patent innovatively designs a CNN-based pre-training method for random mask modeling based on fog concentration. Only one pre-training model needs to be trained, and then fine-tuned on each foggy downstream task at a small cost, which can significantly improve the convergence speed and accuracy of each downstream task under different fog concentrations.

[0566] 1. Introduction to the overall process.

[0567] The overall process of the proposed algorithm is shown in FIG6 , which is a schematic diagram showing the principle of a pre-training method according to an embodiment of the present invention.

[0568] First, the pre-trained model is trained based on the image containing the mask part, and then the encoding layer parameters in the pre-trained model are frozen and used as the encoding layer of the downstream task model for fine-tuning training.

[0569] The pre-trained model may be an image processing model, specifically used to improve image quality.

[0570] The specific process is as follows:

[0571] 1) First, the fog image Ffoggy in the open source fog image dataset is segmented into image blocks of fixed size. As shown in FIG7 , FIG7 is a schematic diagram illustrating a segmented image according to an embodiment of the present invention.

[0572] 2) Then, the foggy image block obtained by the above segmentation is sent to the classification model to determine the segmentation level according to the fog density of the current image block.

[0573] Take the three categories as an example:

[0574] ① If the fog density is high and more information is needed for mask reconstruction, the original image block size is maintained, which is 256*256 here.

[0575] ② If the fog density is average and certain information is needed for mask reconstruction, a 2x split is performed, specifically 128*128 here.

[0576] ③ If the fog density is small and less information is needed for mask reconstruction, a 4-fold split is performed, specifically 64*64 here.

[0577] Therefore, according to the classification model, the level of splitting can be automatically and adaptively determined according to the fog density of the current image block.

[0578] More detailed fog concentration levels can be divided here, but due to the complexity of data production and dataset annotation, dividing it into three categories can achieve obvious results in actual use.

[0579] As shown in Figure 8, which is a schematic diagram illustrating the principle of a method for determining fog density according to an embodiment of the present invention, the classification model structure includes a coding layer and a fully connected layer, which can be used to predict the fog density level for an input image block.

[0580] 3) Next, the fog image Ffoggy is segmented based on the segmentation levels of each image block obtained above. A corresponding mask Fmask is generated accordingly. The mask Fmask is used to guide how the fog image Ffoggy and the corresponding clear image Fclean are combined within each image block.

[0581] As shown in Figure 9, Figure 9 is a schematic diagram showing a principle of determining a mask according to an embodiment of the present invention. For the specific segmentation process and the process of determining the mask, please refer to Figure 3.

[0582] 4) Again, based on the mask Fmask obtained above, the fog image Ffoggy and the corresponding clear image Fclean in the open source fog image dataset are combined to obtain a combined image Finput.

[0583] For example, if the segmentation level is 3, a 256*256 image block is split 4-fold, specifically into 16 64*64 blocks. Then, based on the preset mask ratio (here set to 0.5), eight of these 64*64 blocks are randomly selected and the foggy image Ffoggy is used to replace the clear image Fclean in the corresponding areas. This process continues until all 256*256 image blocks in the image are combined, resulting in the combined image Fininput.

[0584] In defogging training, the input composite image Finput is a mixture of the foggy image Ffoggy and the corresponding clear image Fclean. In this way, the clear area can be used to guide the defogging of the foggy area, effectively improving the defogging effect.

[0585] FIG10 is a schematic diagram illustrating another principle for combining images according to an embodiment of the present invention. Specifically, based on the mask Fmask, Ffoggy and Fclean are combined to obtain a combined image Finput (i.e., a mask image), which is then input into a pre-trained model for training. The specific combination process can be seen in FIG2.

[0586] 5) Finally, the combined image Fininput is fed into the designed pre-trained model NETdefog, which is trained with dehazing as the optimization objective. It's important to note that the weights of the encoder in NETdefog can be shared with those of the classification model during training, reducing the training burden and improving training efficiency.

[0587] After pre-training, the encoder weights in NETdefog are "frozen," and then a decoder adapted to different downstream tasks is added. With minimal fine-tuning, the convergence speed and accuracy of downstream tasks under varying fog concentrations can be significantly improved. For example, these pre-trained weights can be used for tasks such as outdoor foggy vehicle segmentation, outdoor foggy vehicle classification, and outdoor foggy pedestrian detection.

[0588] 2. Introduction to the network structure of the pre-trained model NETdefog and the classification model.

[0589] The structure of the main Encoder part can be replaced with commonly used networks such as Resnet and Refinetnet.

[0590] As shown in Figure 11, Figure 11 is a schematic diagram of a pre-training model structure according to an embodiment of the present invention. The specific process is as follows.

[0591] 1) First, the composite image Finin (3 × H × W, where 3 represents the RGB channels, H represents the height of the video frame, and W represents the width of the video frame) is used as the input to NETdefog. Initial feature extraction is performed through a 2D convolutional neural network layer to obtain F2D-init, with dimensions B × C × H × W (where C is the designed number of channels and B is the batch size set during training).

[0592] 2) Secondly, F2D-init is input into four blocks (as shown in Figure 9) for image compression. Each block contains a two-dimensional convolutional neural network layer with a stride of 2, resulting in a Fencoder with a size of B×C×(H / / 16)×(W / / 16).

[0593] At this point, the structure introduction of the Encoder part is over

[0594] 3) Again, the Fencoder is input into four deconvolution layers (ConvTransposed) for recovery with a step size of 2 to obtain the Fdecoder with a size of B×C×H×W.

[0595] 4) Finally, Fdecoder is input into a 2D convolutional neural network layer for fusion and channel dimension reduction, and the dehazed image Fout (size B×3×H×W) is finally obtained.

[0596] Specifically, a residual network structure can be used to combine the input image with the output image of the two-dimensional convolutional neural network layer (specifically, they can be added) to obtain a defogged image Fout.

[0597] As shown in FIG12 , FIG12 is a schematic diagram of a Block structure according to an embodiment of the present invention.

[0598] It contains 2 2D convolutional neural network layers and 1 2D convolutional neural network layer with a stride of 2. A residual network structure is also set up to combine the output of the last layer with the input to obtain the final output.

[0599] The classification model shares the weights of the Encoder part with NETdefog, but changes the Decoder part to the commonly used fully connected layer FC layer for classification, so it will not be described in detail.

[0600] 3. Introduction to pre-training loss function.

[0601] As shown in FIG13 , FIG13 is a schematic diagram showing a principle of determining model loss according to an embodiment of the present invention.

[0602] Specifically, the masked parts of the clear image Fclean and the defogging image Fout can be extracted, and the L2 loss can be calculated only for the masked parts. In this way, NETdefog only optimizes the defogging performance, and the optimization of the classification model is implicit in the calculated L2 loss, so there is no need to design loss specifically.

[0603] It should be noted that L2 loss is calculated on the feature maps of different resolutions output by the four blocks, and the final loss is obtained by adding them together.

[0604] L2loss is specifically the mean square error (MSE), the average of the squares of the differences between the model prediction value f(x) and the true value y of the sample. The formula is as follows:

[0605] The various technical features in the above embodiments can be arbitrarily combined as long as there is no conflict or contradiction between the combinations of features. However, due to space limitations, they are not described one by one. Therefore, the arbitrary combination of the various technical features in the above embodiments also falls within the scope of the present invention.

[0606] Corresponding to the above method embodiments, the embodiments of the present invention also provide corresponding device embodiments.

[0607] As shown in FIG14 , FIG14 is a structural diagram of a training device for an image processing model according to an embodiment of the present invention.

[0608] The apparatus may include the following units.

[0609] The sample acquisition unit 401 is used to acquire a training sample set.

[0610] For any training sample in the training sample set, the mask image is used as the sample feature and the high-quality image is used as the sample label. In the same training sample, the mask image and the high-quality image have the same image content and image size but different image quality. The image quality of the masked part in the mask image is lower than the image quality of the same position area in the high-quality image. The pixel values ​​of the non-masked part in the mask image are the same as the pixel values ​​of the same position area in the high-quality image.

[0611] The model acquisition unit 402 is used to acquire an initial image processing model.

[0612] The training unit 403 is used to train the initial image processing model based on the training sample set to obtain a trained result image processing model; the result image processing model is used to improve image quality.

[0613] Optionally, the method for constructing a training sample includes:

[0614] Obtain high-quality images and mask images;

[0615] The obtained mask image and the high-quality image have the same image content and image size but different image quality; the image quality of the masked portion of the obtained mask image is lower than the image quality of the same position area in the obtained high-quality image; the pixel values ​​of the non-masked portion of the obtained mask image are the same as the pixel values ​​of the same position area in the obtained high-quality image;

[0616] The obtained mask image is used as the sample feature and the obtained high-quality image is used as the sample label to construct the training sample.

[0617] Optionally, obtain high-quality images and mask images, including:

[0618] Acquire high-quality images;

[0619] A mask portion is determined for the acquired high-quality image, and the image quality of the determined mask portion is reduced to obtain a mask image.

[0620] Optionally, obtain high-quality images and mask images, including:

[0621] Acquire a low-quality image and a high-quality image; the acquired low-quality image and the acquired high-quality image have the same image content and image size but different image quality, and the image quality of the acquired low-quality image is less than the image quality of the acquired high-quality image;

[0622] A mask image is obtained based on the acquired low-quality image and high-quality image.

[0623] Optionally, acquire low-quality and high-quality images, including:

[0624] Acquire high-quality images;

[0625] For the acquired high-quality image, the image quality is reduced to obtain a low-quality image.

[0626] Optionally, obtaining a mask image based on the acquired low-quality image and high-quality image includes:

[0627] determining a mask portion based on the acquired low-quality image and the acquired high-quality image;

[0628] The mask image is obtained by replacing the pixel values ​​of the masked part determined in the high-quality image with the pixel values ​​of the same position area in the low-quality image; or, replacing the pixel values ​​of the non-masked part determined in the low-quality image with the pixel values ​​of the same position area in the high-quality image; or, combining the pixel values ​​of the non-masked part determined in the high-quality image and the pixel values ​​of the masked part in the low-quality image.

[0629] Optionally, determining a mask portion based on the acquired low-quality image and high-quality image includes:

[0630] Divide the acquired low-quality image into several image blocks;

[0631] For at least one image block, a corresponding mask part is determined respectively.

[0632] Optionally, for at least one image block, determining a corresponding mask portion respectively includes:

[0633] For at least one image block, perform the following steps respectively:

[0634] An image quality level corresponding to the image block is determined, and a corresponding mask portion is determined based on a mask determination method corresponding to the determined image quality level.

[0635] Optionally, determining an image quality level corresponding to the image block includes:

[0636] Determine the image quality level corresponding to the image block based on a preset classification model;

[0637] or

[0638] Determine the difference between an image block in the acquired low-quality image and an image block at the same position in the acquired high-quality image; and determine an image quality level corresponding to the image block based on the determined difference.

[0639] Optionally, the lower the image quality represented by the image quality level, the smaller the area proportion of the determined mask portion in the corresponding image block.

[0640] Optionally, a method for determining a mask corresponding to an image quality level includes:

[0641] The image block is further divided into image sub-blocks corresponding to the number of image quality levels;

[0642] Among image blocks with the same image quality level, image sub-blocks with corresponding proportions of the image quality levels are determined as mask parts.

[0643] Optionally, the lower the image quality represented by the image quality level, the smaller the corresponding number;

[0644] The lower the image quality represented by the image quality level, the smaller the corresponding ratio.

[0645] Optionally, the initial image processing model includes: a dimensionality reduction encoding layer and a dimensionality increase decoding layer;

[0646] The dimensionality reduction coding layer is used to perform dimensionality reduction convolution on the input image to obtain a feature map;

[0647] The dimensionality-increasing decoding layer is used to upsample the input feature map and output the predicted image.

[0648] Optionally, the initial image processing model further includes: a fully connected layer;

[0649] The fully connected layer is used to predict the predicted feature map of the same size as the input feature map.

[0650] Optionally, fully connected layers are used to:

[0651] Divide the input feature map into a preset number of feature image blocks;

[0652] Based on a preset number of feature image blocks, respectively predicting a preset number of prediction image blocks;

[0653] The obtained predicted image blocks are spliced ​​into a predicted feature map.

[0654] Optionally, fully connected layers are used to:

[0655] The feature map obtained from the dimensionality reduction coding layer is divided into a preset number of feature image blocks;

[0656] Based on a preset number of feature image blocks, respectively predicting a preset number of prediction image blocks;

[0657] The obtained predicted image blocks are spliced ​​into a predicted feature map.

[0658] Optionally, an initial image processing model is used to:

[0659] For the input mask image, perform dimensionality reduction convolution to obtain the feature map;

[0660] The obtained feature map is upsampled and a predicted image is output; the predicted image and the input mask image have the same size.

[0661] Optionally, the apparatus further includes an application unit 404, configured to:

[0662] Based on the coding layer in the resulting image processing model, construct the coding layers of other image processing task models;

[0663] or

[0664] For any image, the image quality is improved based on the resulting image processing model.

[0665] Optionally, the image quality is characterized by the degree to which it is affected by environmental factors.

[0666] Optionally, the loss function of the initial image processing model is used to:

[0667] The loss is determined for the masked portion of the predicted image output by the initial image processing model and the same location area in the high-quality image of the corresponding sample.

[0668] For detailed explanation, please refer to the above method embodiment.

[0669] As shown in FIG15 , FIG15 is a structural diagram of another image processing model training device according to an embodiment of the present invention.

[0670] The apparatus may include the following units.

[0671] The image pair acquisition unit 501 is used to acquire several groups of image pairs; any group of image pairs includes a low-quality image and a high-quality image; in any group of image pairs, the low-quality image and the high-quality image have the same image content and image size but different image quality, and the image quality of the low-quality image is less than the image quality of the high-quality image.

[0672] An initial model acquisition unit 502 is configured to acquire an initial image processing model and determine the acquired initial image processing model as a current image processing model;

[0673] The loop unit 503 is configured to loop through the following steps until a preset loop stop condition is met:

[0674] Based on the acquired image pairs, training samples are constructed; for any constructed training sample, the mask image is used as a sample feature and the high-quality image is used as a sample label; in the same training sample, the mask image and the high-quality image have the same image content and image size but different image quality, the image quality of the masked portion in the masked image is lower than the image quality of the same position area in the high-quality image, and the pixel values ​​of the non-masked portion in the masked image are the same as the pixel values ​​of the same position area in the high-quality image;

[0675] The current image processing model is trained based on the constructed training samples, and the image processing model after training and updating is re-determined as the current image processing model.

[0676] The stopping unit 504 is configured to determine the current image processing model as the result image processing model after the loop ends.

[0677] Optionally, the circulation unit 503 is configured to:

[0678] For any set of image pairs, the low-quality image is divided into several image blocks, and the mask part of at least one image block is determined using a preset classification model; in the acquired low-quality image, the pixel values ​​of the determined non-masked part are replaced with the pixel values ​​of the same position area in the high-quality image to obtain a mask image; the mask image is used as the sample feature and the high-quality image is used as the sample label to construct a training sample.

[0679] Optionally, in each cycle, the encoding layer parameters of the preset classification model are updated to the encoding layer parameters of the current image processing model.

[0680] For detailed explanation, please refer to the above method embodiment.

[0681] As shown in FIG16 , FIG16 is a structural diagram of another image processing model training device according to an embodiment of the present invention.

[0682] The apparatus may include the following units.

[0683] The defogging sample unit 601 is used to obtain a training sample set.

[0684] For any training sample in the training sample set, the mask image is used as the sample feature and the high-quality image is used as the sample label. In the same training sample, the mask image and the high-quality image have the same image content and image size but different degrees of fog influence. The degree of fog influence on the masked part in the mask image is greater than the degree of fog influence on the same position area in the high-quality image.

[0685] The pixel values ​​of the non-masked portion in the mask image can be the same as the pixel values ​​of the same position area in the high-quality image.

[0686] The defogging model unit 602 is used to obtain an initial image processing model.

[0687] The defogging training unit 603 is used to train the initial image processing model based on the training sample set to obtain a trained result image processing model; the result image processing model is used to reduce the degree of fog influence.

[0688] For detailed explanation, please refer to the above method embodiment.

[0689] An embodiment of the present invention further provides a computer device, which comprises at least a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps of any of the above method embodiments are implemented.

[0690] An embodiment of the present invention also provides an electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions that can be executed by the one processor, and the instructions are executed by the at least one processor so that the at least one processor can perform the steps of any of the above-mentioned method embodiments.

[0691] FIG17 is a schematic diagram of the hardware structure of a computer device configured to implement the method according to an embodiment of the present invention, according to an embodiment of the present invention. The device may include: a processor 1010, a memory 1020, an input / output interface 1030, a communication interface 1040, and a bus 1050. The processor 1010, the memory 1020, the input / output interface 1030, and the communication interface 1040 are connected to each other within the device via the bus 1050.

[0692] The processor 1010 can be implemented using a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided by the embodiments of the present invention.

[0693] The memory 1020 can be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory), static storage devices, dynamic storage devices, etc. The memory 1020 can store an operating system and other application programs. When the technical solutions provided by the embodiments of the present invention are implemented through software or firmware, the relevant program codes are stored in the memory 1020 and are called and executed by the processor 1010.

[0694] The input / output interface 1030 is used to connect input / output modules to implement information input and output. The input / output modules can be configured as components within the device (not shown in the figure) or can be externally connected to the device to provide corresponding functions. Input devices may include a keyboard, mouse, touch screen, microphone, various sensors, etc., and output devices may include a display, speaker, vibrator, indicator light, etc.

[0695] The communication interface 1040 is used to connect to a communication module (not shown) to enable communication between the device and other devices. The communication module can communicate via a wired method (such as USB, network cable, etc.) or a wireless method (such as mobile network, WiFi, Bluetooth, etc.).

[0696] The bus 1050 comprises a path for transmitting information between the various components of the device (eg, the processor 1010 , the memory 1020 , the input / output interface 1030 , and the communication interface 1040 ).

[0697] It should be noted that although the above device only shows the processor 1010, the memory 1020, the input / output interface 1030, the communication interface 1040, and the bus 1050, in a specific implementation, the device may also include other components necessary for normal operation. In addition, those skilled in the art will understand that the above device may only include the components necessary to implement the embodiments of the present invention, and does not necessarily include all the components shown in the figure.

[0698] An embodiment of the present invention further provides a computer program product, comprising a computer program / instruction, which implements the steps of any of the above method embodiments when executed by a processor.

[0699] An embodiment of the present invention further provides a computer-readable storage medium having a computer program stored thereon, which implements the steps of any of the above method embodiments when executed by a processor.

[0700] An embodiment of the present invention further provides a computer-readable storage medium storing a computer program, wherein the computer program implements the steps of any of the above method embodiments when executed by a processor.

[0701] Computer-readable media includes permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. The information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic disk storage or other magnetic storage devices, or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include transitory media such as modulated data signals and carrier waves.

[0702] Through the description of the above embodiments, it can be seen that those skilled in the art can clearly understand that the embodiments of the present invention can be implemented by means of software plus the necessary general hardware platform. Based on this understanding, the technical solution of the embodiments of the present invention can essentially or in other words, the contributing part can be embodied in the form of a software product. The computer software product can be stored in a storage medium such as ROM / RAM, a magnetic disk, an optical disk, etc., and includes a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in various embodiments or certain parts of the embodiments of the present invention.

[0703] The systems, devices, modules, or units described in the above embodiments may be implemented by computer chips or entities, or by products having certain functions. A typical implementation device is a computer, which may be in the form of a personal computer, laptop computer, cellular phone, camera phone, smartphone, personal digital assistant, media player, navigation device, email transceiver, game console, tablet computer, wearable device, or any combination of these devices.

[0704] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment. The device embodiment described above is merely illustrative, wherein the modules described as separate components may or may not be physically separated, and when implementing the embodiment of the present invention, the functions of each module can be implemented in the same one or more software and / or hardware. It is also possible to select some or all of the modules according to actual needs to achieve the purpose of the embodiment. A person of ordinary skill in the art can understand and implement it without paying any creative work.

[0705] The above description is only a specific implementation of the embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the embodiment of the present invention. These improvements and modifications should also be regarded as protection for the embodiment of the present invention.

[0706] In the present invention, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance. The term "plurality" refers to two or more, unless otherwise clearly defined.

[0707] Other embodiments of the present invention will readily occur to those skilled in the art after considering the specification and practicing the disclosure disclosed herein. The present invention is intended to cover any variations, uses, or adaptations of the present invention that follow the general principles of the invention and include common knowledge or customary techniques in the art not disclosed herein.

[0708] It will be understood that the present invention is not limited to the exact construction that has been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof.

Claims

1. A training method for an image processing model, characterized in that Including: Obtain a training sample set; For any training sample in the training sample set, use the masked image as the sample feature and the high-quality image as the sample label; In the same training sample, the image content, image size are the same between the masked image and the high-quality image, but the image quality is different. The image quality of the masked part in the masked image is less than the image quality of the same position area in the high-quality image; Obtain an initial image processing model; Train the initial image processing model based on the training sample set to obtain a trained result image processing model; the result image processing model is used to improve the image quality.

2. The method according to claim 1, wherein In the same training sample, the pixel values of the non-masked part in the masked image are the same as the pixel values of the same position area in the high-quality image.

3. The method according to claim 1, characterized in that The construction method of the training sample includes: Obtain a high-quality image and a masked image; The obtained masked image and high-quality image have the same image content and image size, but different image qualities; the image quality of the masked part in the obtained masked image is less than the image quality of the same position area in the obtained high-quality image; the pixel values of the non-masked part in the obtained masked image are the same as the pixel values of the same position area in the obtained high-quality image; Use the obtained masked image as the sample feature and the obtained high-quality image as the sample label to construct a training sample.

4. The method according to claim 3, characterized in that, The obtaining of the high-quality image and the masked image includes: Obtain a high-quality image; For the obtained high-quality image, determine the masked part and reduce the image quality of the determined masked part to obtain a masked image.

5. The method according to claim 3, wherein The obtaining of the high-quality image and the masked image includes: Obtain a low-quality image and a high-quality image; between the obtained low-quality image and high-quality image, the image content and image size are the same, but the image quality is different. The image quality of the obtained low-quality image is less than the image quality of the obtained high-quality image; Based on the obtained low-quality image and high-quality image, obtain a masked image.

6. The method according to claim 5, characterized in that, The obtaining of the low-quality image and the high-quality image includes: Obtain a high-quality image; For the obtained high-quality image, reduce the image quality to obtain a low-quality image.

7. The method according to claim 5, characterized in that, The obtaining of the masked image based on the obtained low-quality image and high-quality image includes: Based on the obtained low-quality image and high-quality image, determine the masked part; Replace the pixel values of the determined masked part in the high-quality image with the pixel values of the same position area in the low-quality image to obtain a masked image; or, replace the pixel values of the determined non-masked part in the low-quality image with the pixel values of the same position area in the high-quality image to obtain a masked image; or, combine the pixel values of the determined non-masked part in the high-quality image and the pixel values of the masked part in the low-quality image to obtain a masked image.

8. The method according to claim 7, wherein The determining of the masked part based on the obtained low-quality image and high-quality image includes: Divide the obtained low-quality image into several image blocks; For at least one image block, respectively determine the corresponding mask part, including: For at least one image block, respectively perform the following steps: Determine the image quality level corresponding to the image block, and based on the mask determination method corresponding to the determined image quality level, determine the corresponding mask part.

9. The method according to claim 1, wherein The initial image processing model includes: a dimensionality reduction encoding layer and a dimensionality increase decoding layer; The dimensionality reduction encoding layer is used for: performing dimensionality reduction convolution on the input image to obtain a feature map; The dimensionality increase decoding layer is used for: performing upsampling on the input feature map and outputting a predicted image.

10. The method according to claim 9, wherein The initial image processing model further includes: a fully connected layer; The fully connected layer is used for: predicting and outputting a predicted feature map of the same size for the input feature map.

11. According to the method of claim 9, wherein The initial image processing model is used for: Performing dimensionality reduction convolution on the input masked image to obtain a feature map; Performing upsampling on the obtained feature map and outputting a predicted image; the predicted image has the same size as the input masked image.

12. The method according to claim 1, wherein It further includes: Based on the encoding layer in the result image processing model, construct the encoding layer of other image processing task models; Or For any image, improve the image quality based on the result image processing model.

13. The method according to claim 1, characterized in that The image quality is characterized by the degree of influence of environmental factors.

14. The method according to claim 1, wherein The loss function of the initial image processing model is used for: Determine the loss for the mask part in the predicted image output by the initial image processing model and the same position area in the high-quality image in the corresponding sample.

15. A training method for an image processing model, characterized in that, It includes: Obtain several groups of image pairs; Any group of image pairs includes a low-quality image and a high-quality image; In any group of image pairs, the image content is the same, the image size is the same, but the image quality is different between the low-quality image and the high-quality image, and the image quality of the low-quality image is less than the image quality of the high-quality image; Obtain an initial image processing model and determine the obtained initial image processing model as the current image processing model; Loop and execute the following steps until the preset loop stop condition is met: Based on the obtained image pairs, construct training samples; for any constructed training sample, use the masked image as the sample feature and the high-quality image as the sample label; in the same training sample, the image content is the same, the image size is the same, but the image quality is different between the masked image and the high-quality image, the image quality of the mask part in the masked image is less than the image quality of the same position area in the high-quality image, and the pixel values of the non-mask part in the masked image are the same as the pixel values of the same position area in the high-quality image; Train the current image processing model based on the constructed training samples and re-determine the image processing model after the training update is completed as the current image processing model; After the loop ends, determine the current image processing model as the result image processing model.

16. A training method for an image processing model, characterized in that, It includes: Obtain a training sample set; For any training sample in the training sample set, use the masked image as the sample feature and the high-quality image as the sample label; In the same training sample, the image content is the same, the image size is the same, but the degree of fog influence is different between the masked image and the high-quality image, and the degree of fog influence of the mask part in the masked image is greater than the degree of fog influence of the same position area in the high-quality image; Obtain an initial image processing model; Train the initial image processing model based on the training sample set to obtain a trained result image processing model; the result image processing model is used to reduce the degree of fog influence.

17. An image defogging method, characterized in that Including: Input the target image into a preset image processing model to obtain a defogged image output by the preset image processing model; The preset image processing model is pre-trained in advance based on the training method of the image processing model described in any one of claims 1 to 16.

18. The method according to claim 17, wherein The image defogging method is applied to at least one of the following: in-vehicle display tasks on outdoor foggy days, vehicle detection tasks on outdoor foggy days, and pedestrian detection tasks on outdoor foggy days.

19. A training device for an image processing model, characterized in that, Including: A sample acquisition unit for acquiring a training sample set; Any training sample in the training sample set uses a masked image as the sample feature and a high-quality image as the sample label; in the same training sample, the masked image and the high-quality image have the same image content, the same image size but different image qualities, and the image quality of the masked part in the masked image is less than the image quality of the same position area in the high-quality image; A model acquisition unit for acquiring an initial image processing model; A training unit for training the initial image processing model based on the training sample set to obtain a trained result image processing model; the result image processing model is used to improve the image quality.

20. An electronic device, characterized in that, Including: At least one processor; And, A memory communicatively connected to the at least one processor; wherein, The memory stores instructions executable by the one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the steps of the method described in any one of claims 1 to 18.

21. A computer-readable storage medium storing a computer program, characterized in that, The computer program, when executed by a processor, implements the steps of the method described in any one of claims 1 to 18.

22. A computer program product comprising a computer program / instructions, characterized in that, The computer program / instructions, when executed by a processor, implement the steps of the method described in any one of claims 1 to 18.

Images