Image processing method and device, electronic equipment, computer storage medium and product

Abstract

The invention relates to an image processing method and device, electronic equipment, a computer storage medium and a product, and relates to the technical field of image processing. The method comprises the following steps: acquiring a to-be-processed image; carrying out fog sense recognition on the to-be-processed image, and obtaining a fog sense recognition result of the to-be-processed image; the fog sense recognition result comprises a first type of images or a second type of images, and the fog sense concentration of the first type of images is smaller than the fog sense concentration of the second type of images; performing image processing on the to-be-processed image according to the fog sense recognition result to obtain a target image; image processing of the first type of images is different from image processing of the second type of images, and image processing of the second type of images comprises image defogging processing. By the adoption of the method, different image processing strategies can be intelligently switched according to different fog sense recognition results, accurate processing of images with different fog sense concentrations is achieved, and the image processing effect is improved.

Description

Technical Field

[0001] The present application relates to the field of image processing technologies, and in particular, to an image processing method, apparatus, electronic device, computer storage medium, and product. Background Art

[0002] Fog is a natural phenomenon formed by the combined action of Mie scattering and Rayleigh scattering of suspended particles (such as water droplets, ice crystals, etc.) in the atmosphere, which has a significant degradation effect on optical imaging systems.

[0003] Compared with the images taken in a non-foggy scene, the images taken in a foggy scene often have serious quality problems such as a significant decrease in contrast and a large loss of color details. When the foggy scene images are actually applied in fields such as video surveillance, autonomous driving, and computer vision target recognition, they will have a negative impact. Therefore, it is necessary to process the images, such as image dehazing processing, to restore clear images from the input captured by the camera.

[0004] However, the effects of traditional image processing methods are not ideal. After some images are processed, problems such as color deviation with distorted contrast and saturation often occur. Summary of the Invention

[0005] The present application provides an image processing method, apparatus, electronic device, computer storage medium, and computer program product that can improve the image processing effect.

[0006] The present application provides an image processing method, the method comprising:

[0007] Obtaining an image to be processed;

[0008] Performing fog sense recognition on the image to be processed, and obtaining a fog sense recognition result of the image to be processed; the fog sense recognition result includes a first type of image or a second type of image, and the fog sense concentration of the first type of image is less than that of the second type of image;

[0009] Performing image processing on the image to be processed according to the fog sense recognition result to obtain a target image; the image processing of the first type of image is different from that of the second type of image, and the image processing of the second type of image includes image dehazing processing.

[0010] The present application provides an image processing apparatus, the apparatus comprising:

[0011] An obtaining module, configured to obtain an image to be processed;

[0012] An identification module is configured to perform fog-sensation identification on the image to be processed and obtain a fog-sensation identification result of the image to be processed. The fog-sensation identification result includes a first type of image or a second type of image, and the fog-sensation concentration of the first type of image is less than that of the second type of image.

[0013] A processing module is configured to perform image processing on the image to be processed according to the fog-sensation identification result to obtain a target image. The image processing methods for the first type of image and the second type of image are different, and the image processing for the second type of image includes defogging processing.

[0014] This application provides an electronic device, including a memory and a processor. The memory stores a computer program, and when the processor executes the computer program, the steps of the method described above are implemented.

[0015] This application provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the method described above are implemented.

[0016] This application provides a computer program product, including a computer program. When the computer program is executed by a processor, the steps of the method described above are implemented.

[0017] The above-mentioned image processing method, device, electronic device, computer storage medium, and computer program product obtain an image to be processed, perform fogginess recognition on the image to be processed, obtain the fogginess recognition result of the image to be processed, and perform image processing on the image to be processed according to the fogginess recognition result to obtain a target image. Among them, the fogginess recognition result includes a first type of image and a second type of image. The fogginess concentration of the first type of image is less than that of the second type of image, and the image processing of the first type of image is different from that of the second type of image. The image processing of the second type of image includes image defogging processing. In this way, the recognition of the image in the fogginess dimension is realized, and different image processing strategies can be intelligently switched according to different fogginess recognition results, realizing the precise processing of images with different fogginess concentrations, improving the image processing ability and image processing effect for complex and diverse images, meeting the image processing requirements in different fogginess concentration scenarios, and balancing the image processing effects in different fogginess concentration scenarios. Compared with the problem of color deviation caused by defogging all images in the related technology, the present application can adopt different image processing methods for different types of images, that is, not perform defogging processing on the first type of image and perform defogging processing on the second type of image. This not only solves the problems such as the decrease in image contrast and the loss of color details caused by fogging in the foggy scene, but also takes into account the image processing requirements in ordinary scenes such as non-fog scenes and slightly foggy scenes, avoiding the color deviation problems such as contrast and saturation distortion that may be caused by defogging non-fog images and images with low fogginess concentration, that is, non-extremely foggy images. Thus, high-quality image effects can be maintained in different scenes, and the image processing quality is improved. In addition, the present application performs fogginess recognition on the entire image to be processed. Compared with cutting the image into multiple image blocks for block detection, it makes full use of the global information of the image, avoids the problem of uneven defogging effect in the subsequent image processing process caused by block detection, and improves the accuracy of fogginess recognition and the overall effect of image processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 is one of the schematic flowcharts of the image processing method in an embodiment;

[0019] Figure 2 is the second schematic flowchart of the image processing method in an embodiment;

[0020] Figure 3 is the third schematic flowchart of the image processing method in an embodiment;

[0021] Figure 4 is the fourth schematic flowchart of the image processing method in an embodiment;

[0022] Figure 5 is the schematic structural diagram of the fogginess detection model in an embodiment;

[0023] Figure 6 FIG5 is a fifth flow chart of an image processing method in one embodiment;

[0024] Figure 7 FIG6 is a sixth flow chart of an image processing method in an embodiment;

[0025] Figure 8 FIG7 is a flow chart of an image processing method according to an embodiment;

[0026] Figure 9 is a structural block diagram of an image processing device in one embodiment;

[0027] Figure 10 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0028] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0029] It is understandable that in extreme foggy scenes such as hardware-induced glare, strong backlight, and severe fogging, images often have serious quality problems such as significant contrast reduction and loss of color details. In order to effectively restore the image quality and visual effects in these extreme scenes, it is often necessary to implement targeted image optimization strategies such as defogging and contrast adjustment.

[0030] Single image dehazing technology has always been one of the most representative low-level visual tasks in the field of computer vision. The relevant technical solutions are rich and diverse, ranging from traditional methods based on physical models and prior knowledge, to end-to-end mapping methods based on convolutional neural networks (CNN), to advanced methods based on Transformer and generative adversarial networks (GAN). These methods have shown good generalization ability and dehazing effects in different types of foggy scenes. Among them, Transformer is a neural network architecture based on an attention mechanism, which is used to process sequence data, such as text in natural language processing tasks.

[0031] However, whether it is a dehazing algorithm based on CNN or Transformer, it usually can only process hazy images. However, in practical applications, it is expected that the dehazing model can intelligently perform corresponding processing according to the state of the input image: for non-hazy images, the output result should be as consistent with the input as possible, and only hazy images are dehazed, so as to achieve the ability to balance the processing of non-hazy and hazy images. Unfortunately, the dehazing algorithms in the related technologies have obvious deficiencies in this regard. After the non-hazy images are processed by the dehazing model, color deviation problems such as contrast and saturation distortion often occur.

[0032] Therefore, the embodiments of the present application provide an image processing method, device, computer device, medium and product, which can identify extremely hazy images and non-extremely hazy images, and flexibly switch corresponding image optimization strategies according to different scenarios (whether extremely hazy or non-extremely hazy), so as to better meet the diverse image optimization requirements.

[0033] In some embodiments, as Figure 1 shown, an image processing method is provided. In this embodiment, taking the application of this method to a terminal as an example, it can be understood that this method can also be applied to a server, and can also be applied to a system including a terminal and a server, and is implemented through the interaction between the terminal and the server. Among them, the terminal can be a device with a shooting function. For example, an electronic device with a camera, a camera, etc., which is not limited here. In this embodiment, this method includes steps S102 to S106.

[0034] S102: Obtain the image to be processed.

[0035] The image to be processed refers to the image to be optimized. Among them, image optimization processing includes but is not limited to image dehazing processing, pixel depth conversion, brightening, sharpening, etc., and can be set accordingly according to the image quality requirements, which is not limited here.

[0036] In applications, the image to be processed can be an image obtained by the terminal shooting a specific scene.

[0037] S104: Perform fog sense recognition on the image to be processed, and obtain the fog sense recognition result of the image to be processed.

[0038] The fog sense recognition result is used to represent the image category of the image to be processed classified in the fog sense dimension. The fog sense recognition result includes the first type of image or the second type of image. Among them, the fog sense concentration of the first type of image is less than that of the second type of image. The fog sense concentration is used to characterize the degree of haziness of the image. The greater the fog sense concentration, the greater the degree of haziness of the image, and the lower the quality of the image; the smaller the fog sense concentration, the smaller the degree of haziness of the image, and the higher the quality of the image.

[0039] For example, the first type of image is a non-extreme fog image, and the second type of image is an extreme fog image. Among them, the extreme fog image can be understood as an image taken by the terminal in an extreme fog scene, and correspondingly, the non-extreme fog image can be understood as an image taken by the terminal in a non-extreme fog scene.

[0040] Extreme foggy scenes refer to scenes with high fog density, low visibility, and significant atmospheric scattering effects, or scenes with high fog density in images captured by the terminal. Correspondingly, non-extreme foggy scenes can be understood as scenes with high fog density and fair visibility, or scenes with low fog density in images captured by the terminal.

[0041] Exemplarily, the first category of images at least includes non-fog images (i.e., non-fog images), for example, the first category of images includes non-fog images and first-level foggy images (such as slightly foggy images); the second category of scenes includes at least second-level foggy images; wherein the fog density of the first-level foggy images is less than the fog density of the second-level foggy images. For example, taking the first category of images as non-extreme foggy images and the second category of images as non-extreme foggy images as an example, wherein the non-extreme foggy images may include non-foggy images and slightly foggy images; the extreme foggy images may include moderate foggy scenes, heavy foggy scenes, extreme dense foggy scenes, etc., such as images shot in scenes such as glare, strong backlight, and foggy scenes.

[0042] In addition, for foggy images, the above uses the first-level foggy images and the second-level foggy images as two levels, and divides the first-level foggy images into the first category of images, and divides the second-level foggy images into the second category of images as examples for explanation. In practical applications, foggy images with different foggy concentrations can be divided into more levels according to actual needs, and different levels of foggy images correspond to different foggy concentrations, which are not excessively limited here.

[0043] S106: Perform image processing on the image to be processed according to the fog recognition result to obtain a target image.

[0044] The image processing of the first type of images is different from that of the second type of images, wherein the image processing of the second type of images includes image defogging processing.

[0045] Exemplarily, the image processing of the first category of images does not include image defogging processing, that is, when the fog recognition result of the image to be processed is the first category of images, image defogging processing is not performed on the image to be processed; when the fog recognition result of the image to be processed is the second category of images, image defogging processing is performed on the image to be processed.

[0046] Taking the first type of image as a non-extremely foggy image and the second type of image as an extremely foggy image as an example, when the fogginess recognition result of the image to be processed is a non-extremely foggy image, the image to be processed is not de-fogged; when the fogginess recognition result of the image to be processed is an extremely foggy image, the image to be processed is de-fogged.

[0047] It should be noted that the image processing of the first type of image and the second type of image may both include other image processing processes. For example, pixel depth conversion, brightening, sharpening, etc. Specifically, it can be set accordingly according to the image quality requirements and will not be limited here.

[0048] The image processing method provided in the above embodiment obtains the image to be processed, performs fogginess recognition on the image to be processed, obtains the fogginess recognition result of the image to be processed, and performs image processing on the image to be processed according to the fogginess recognition result to obtain the target image. Among them, the fogginess recognition result includes the first type of image and the second type of image. The fogginess concentration of the first type of image is less than that of the second type of image, and the image processing of the first type of image is different from that of the second type of image. The image processing of the second type of image includes image de-fogging. In this way, the recognition of the image in the fogginess dimension is realized, and different image processing strategies can be intelligently switched according to different fogginess recognition results, realizing the precise processing of images with different fogginess concentrations, improving the processing ability and image processing effect of complex and diverse images, meeting the image processing requirements in different fogginess concentration scenarios, and balancing the image processing effects in different fogginess concentration scenarios. Compared with the problem of color deviation caused by de-fogging all images in the related art, the present application can adopt different image processing methods for different types of images, that is, not de-fog the first type of image and de-fog the second type of image, which not only solves the problems such as the decrease in image contrast and the loss of color details caused by fog in the foggy scene, but also takes into account the image processing requirements in ordinary scenes such as non-fog scenes and slightly foggy scenes, avoiding the color deviation problems such as contrast and saturation distortion that may be caused by de-fogging non-fog images and images with low fogginess concentration, that is, non-extremely foggy images. Therefore, high-quality image effects can be maintained in different scenes, and the image processing quality is improved. In addition, the present application performs fogginess recognition on the whole image to be processed. Compared with detecting by cutting the image into multiple image blocks, it makes full use of the global information of the image and avoids the problem of uneven de-fogging effect in the subsequent image processing process caused by block detection, improving the accuracy of fogginess recognition and the overall effect of image processing.

[0049] In some embodiments, the fogginess recognition result is a binary fogginess recognition result. That is, the first type of image and the second type of image are image categories obtained by binary classification of the image to be processed from the fogginess dimension. For example, the first type of image is a non-extremely foggy image, and the second type of image is an extremely foggy image. In this way, using binary classification for fogginess recognition of images simplifies the image recognition process and computational complexity compared to complex multi-classification recognition, improves the image recognition efficiency, and at the same time flexibly adjusts subsequent image processing strategies based on the fogginess recognition result, can adapt to the image processing requirements in different scenarios, and improves the image processing quality in various fogginess concentration scenarios (including non-fog scenarios and foggy scenarios).

[0050] In some embodiments, as Figure 2 shown, S104, perform fogginess recognition on the image to be processed and obtain the fogginess recognition result of the image to be processed, including the following steps S202 and S204.

[0051] S202: Use the trained fogginess detection model to perform fogginess recognition on the image to be processed and obtain the fogginess concentration probability of the image to be processed.

[0052] The fogginess detection model, which can also be referred to as a fogginess detector, is used to perform fogginess recognition on the image to be processed. The fogginess detection model is a pre-trained model. The input of the fogginess detection model is the image to be processed, and the output of the fogginess detection model is the fogginess concentration probability of the image to be processed.

[0053] The fogginess concentration probability is used to represent the degree of fogginess of the image to be processed. The lower the fogginess concentration probability, the lower the degree of fogginess of the image to be processed; the higher the fogginess concentration probability, the higher the degree of fogginess of the image to be processed. Exemplarily, the fogginess concentration probability includes the probability that the image to be processed belongs to the first type of image, such as a non-extremely foggy image, and / or the fogginess concentration probability includes the probability that the image to be processed belongs to the second type of image, such as an extremely foggy image.

[0054] It should be noted that in the scenario where the fogginess recognition result is binary, the sum of the probability that the image to be processed belongs to the first type of image and the probability that the image to be processed belongs to the second type of image is 1. In applications, the fogginess concentration probability output by the fogginess detection model can be set according to the actual situation, and no further limitation is made here.

[0055] In applications, the terminal can pre-store the trained fogginess detection model, or can obtain the trained fogginess detection model from the cloud through the network. No further limitation is made on the source of the fogginess detection model here. After the terminal captures the image to be processed, it can directly input the entire image to be processed into the trained fogginess detection model, perform fogginess recognition on the image to be processed through the fogginess detection model, and obtain the corresponding fogginess concentration probability.

[0056] S204: Obtain the fogginess recognition result of the image to be processed according to the fogginess concentration probability and the preset fogginess concentration probability threshold.

[0057] The fogginess concentration probability threshold is preset and appropriate values can be set according to the actual scenario. It can be understood that the value range of the fogginess concentration probability is 0 to 1, and the corresponding fogginess concentration probability threshold can be greater than 0 and less than 1.

[0058] The fogginess concentration probability threshold is used to determine whether the image to be processed belongs to the first type of image or the second type of image. The fogginess concentration probability threshold can be understood as the fogginess classification boundary. The size of the fogginess concentration probability threshold will affect the fogginess recognition result of the image.

[0059] The fogginess concentration probability threshold is correlated with the fogginess concentration probability output by the fogginess detection model. If the fogginess concentration probability output by the fogginess detection model includes the probability that the image to be processed belongs to the second type of image, then under the condition of the same fogginess concentration probability, the larger the fogginess concentration probability threshold, the smaller the possibility that the image to be processed belongs to the second type of image; the smaller the fogginess concentration probability threshold, the greater the possibility that the image to be processed belongs to the second type of image. In applications, the fogginess concentration probability output by the fogginess detection model and the corresponding fogginess concentration probability threshold can be set according to the actual scenario, and no more limitations are made here.

[0060] In applications, taking the fogginess concentration probability output by the fogginess detection model including the probability that the image to be processed belongs to the second type of image as an example, the terminal can use the trained fogginess detection model to obtain the fogginess concentration probability that the image to be processed belongs to the second type of image, and compare this fogginess concentration probability with the corresponding fogginess concentration probability threshold. If the fogginess concentration probability is less than the fogginess concentration probability threshold, it can be determined that the fogginess recognition result of the image to be processed is the first type of image; if the fogginess concentration probability is greater than or equal to the fogginess concentration probability threshold, it can be determined that the fogginess recognition result of the image to be processed is the second type of image.

[0061] The image processing method provided in the above embodiments detects the fogginess degree of the image to be processed through the fogginess detection model, and outputs the fogginess concentration probability of the image to be processed, and compares the fogginess concentration probability with the preset fogginess concentration probability threshold, so as to realize the fogginess recognition of the image to be processed. In this way, according to the effects and requirements of subsequent image processing strategies, the fogginess classification boundary, that is, the fogginess concentration probability threshold, can be flexibly adjusted to accurately guide the subsequent image processing process, realize a more targeted image processing strategy, and improve the flexibility and adaptability of image processing.

[0062] In some embodiments, as Figure 3 shown, the image processing method further includes the following steps S302 and S304.

[0063] S302: Construct a sample set.

[0064] S304: Use the sample set to train the initial fog detection model and obtain the trained fog detection model.

[0065] The sample set includes multiple sample data pairs. Each sample data pair includes two sample data. Each sample data includes a sample image and a corresponding fog classification label.

[0066] Each sample data pair includes a first-class sample image and a second-class sample image. That is, in each sample data pair, one sample data includes a sample image and the corresponding fog classification label is the first-class image, and the other sample data includes another sample image and the corresponding fog classification label is the second-class image. Exemplarily, the fog classification label of the first-class image such as a non-extreme foggy image (including a non-fog image and a slightly foggy image) is 0, and the fog classification label of the second-class image such as an extreme foggy image is 1.

[0067] In each sample data pair, the first-class sample image includes a non-fog sample image or a first-level foggy sample image, and the second-class sample image includes at least a second-level foggy sample image, where the fog concentration of the first-level foggy sample image is less than the fog concentration of the second-level foggy sample image.

[0068] It can be understood that the above divides the sample images with a fog concentration greater than that of the first-level foggy sample image into the second-class sample images. In applications, the second-class sample images can also be divided into multiple levels according to the fog concentration. For example, the second-class sample images include second-level foggy sample images, and can also include third-level foggy sample images, fourth-level foggy sample images, fifth-class foggy sample images, etc. with gradually increasing fog concentrations. This is not limited here.

[0069] Taking the first-class sample image as a non-extreme foggy sample image and the second-class image as an extreme foggy sample image; correspondingly, the first-class sample image can include a non-fog sample image and a slightly foggy sample image, and the second-class sample image can include an extreme foggy sample image; where the fog concentrations of slightly foggy images, moderately foggy images, severely foggy images, and extremely concentrated images increase in sequence.

[0070] The sample images can be obtained by shooting in a real scene. For example, extreme foggy sample images are obtained by shooting in extreme scenes such as extreme fog, glare, or strong backlighting, and non-extreme foggy sample images are obtained in non-fog scenes and slightly foggy scenes.

[0071] The sample images can also be constructed using real images obtained by shooting. For example, for the extremely foggy sample images and non-extremely foggy sample images obtained by shooting, the extremely foggy information of the extremely foggy sample images can be extracted. This extremely foggy information is used to represent the foggy characteristics of the extremely foggy images, and the extremely foggy sample images are generated based on the extremely foggy information and the non-extremely foggy sample images.

[0072] It should be noted that the sample set can be constructed according to the actual scenario and image processing requirements, and the ratio between the non-fog images and the foggy images with different fog concentrations in the sample dataset can be set to meet the training requirements of the fog sense detection model. Here, the source of the sample images and the number of images of each category in the sample set are not overly limited.

[0073] The image processing method provided in the above embodiments provides data support for the training of the fog sense detection model by constructing sample data pairs of the first type of sample images such as non-extremely foggy images (including non-fog sample images and slightly foggy sample images) and the second type of sample images such as extremely foggy images, enabling the fog sense detection model to learn and master the foggy information of the images, thereby providing the possibility for the fog sense detection model to perform fog sense recognition on the images and obtain the fog concentration probability. Among them, in the constructed sample data pairs, the non-fog sample images and the first-level foggy sample images are uniformly classified as the first type of sample images, and the second-level foggy sample images are classified as the second type of sample images. Therefore, through threshold classification, two types of images, namely the first type of images and the second type of images, can be obtained, which actually covers three different data distribution situations. It should be noted that the more types the sample images are divided into, the more data distribution situations are actually covered. Here, the types of sample images are not overly limited.

[0074] In some embodiments, as Figure 4 shown in S304, the pre-established fog sense detection model is trained using the sample set, and the trained fog sense detection model is obtained, including the following steps S402 to S406.

[0075] S402: Input the sample set into the initial fog sense detection model to obtain the predicted fog concentration probability correspondingly.

[0076] S404: Input the predicted fog concentration probability into the multi-class focal loss function to obtain the loss value correspondingly.

[0077] S406: Adjust the parameters of the initial fog sense detection model according to the loss value, and obtain the trained fog sense detection model when the model training conditions are met.

[0078] In the case where the fog sense recognition results include the first type of images and the second type of images, the multi-class focal loss function (Focal Loss) can be understood as the two-class focal loss function.

[0079] The multi-class focal loss function includes a focal parameter γ and a balancing weight α t . Among them, the focal parameter γ is used to adjust the attention degree of the multi-class focal loss function to easy and difficult samples. The focal parameter γ is a hyperparameter and can be set according to experiments or experience. For example, the focal parameter γ can be set to 2 or other appropriate values, and no more limitations are made here.

[0080] The balancing weight α t is used to balance positive and negative samples. Among them, the positive sample refers to the second-class sample image such as an extremely foggy image, and the negative sample refers to the first-class sample image such as a non-extremely foggy image. For positive samples, α t =α, and for negative samples, α t =1-α. Among them, α is a hyperparameter and can be set according to experiments or experience. For example, α can be set to 0.25 or other appropriate values, and no more limitations are made here.

[0081] The multi-class focal loss function is positively correlated with the focal parameter γ, and the multi-class focal loss function is t negatively correlated with the balancing weight α. Exemplarily, the multi-class focal loss function can be expressed by the formula:

[0082] f(p t ) = -α t (1 - p t ) γ log(p t ) (1)

[0083] Among them, f(p t ) represents the loss value; p t represents the prediction probability that the multi-class focal loss function treats the image to be processed as belonging to the target class (the first-class image and / or the second-class image), that is, the fogginess concentration prediction probability. For positive samples, p t =p; for negative samples, p t =1 - p; γ represents the focal parameter; α t represents the balancing weight.

[0084] Based on the above formula (1), it can be seen that the focal parameter γ adjusts the loss value through the dynamic scaling factor (1 - p t ) γ . On the one hand, it can reduce the weight of easy-to-classify samples: when the prediction probability p t of the sample image's fogginess concentration is close to 1 (that is, the sample image is easily classified correctly), (1 - p t ) γwill approach 0, thus significantly reducing the contribution of this sample to the total image loss. This means that the model will pay less attention to these easily classifiable sample images. On the other hand, it can increase the weight of difficult-to-classify samples: when p t is small (i.e., the sample image is difficult to classify), (1 - p t ) γ is close to 1, and the loss contribution is relatively large. This enables the model to pay more attention to these difficult-to-classify sample images, thereby improving the model's learning ability for minority classes and complex sample images.

[0085] In addition, the balancing weight α t is used to adjust the weight ratio between positive and negative samples, alleviating the imbalance between different classes. In a dataset with class imbalance, negative samples (the first type of sample images such as non-extremely foggy sample images) are usually much more numerous than positive samples (the second type of sample images such as extremely foggy sample images). By setting α t , the balancing weight of the majority class (usually negative samples) can be reduced, while the weight of the minority class (usually positive samples) can be increased. Exemplarily, setting α t of the positive samples (the second type of sample images such as extremely foggy sample images) to 0.6 can increase the weight of the positive samples, thus enabling the model to pay more attention to the learning of positive samples.

[0086] The image processing method provided by the above embodiments trains the fog-sensing detection model by introducing a multi-class focal loss function. The multi-class focal loss function can solve the class imbalance problem through the focal parameter and the balancing weight, enabling the model to pay more attention to the learning of the second type of images such as extremely foggy images, improving the detection accuracy of the model for extremely foggy scenarios, enhancing the model's attention to difficult-to-classify samples, and thus achieving the goal of efficient and accurate fog-sensing detection.

[0087] In some embodiments, as Figure 5 shown, the fog-sensing detection model includes an input layer, a hidden layer, and an output layer. Among them, the input layer is used to receive the input image. The size of the image received by the input layer can be set according to the actual situation and will not be limited here too much.

[0088] The hidden layer includes at least a first residual layer, a second residual layer, and a third residual layer connected in sequence. The first residual layer is used to connect to the input layer, and the third residual layer is used to connect to the output layer. The first residual layer, the second residual layer, and the third residual layer are respectively used to extract fog-sensing features from the input image.

[0089] The output layer is used to output the fog-sensing concentration probability.

[0090] Exemplarily, the hidden layer includes a convolutional (Conv) layer, a batch normalization (BatchNormalization, BN) layer, an activation layer, a max pooling (Max Pool) layer, a first residual layer, a second residual layer, a third residual layer, a global average pooling (Global Average Pool, GAP) layer, and a fully connected layer that are cascaded in sequence.

[0091] Among them, the initial convolutional layer is connected to the input layer and is used to extract features from the input image. The size of the convolutional kernel of the initial convolutional layer can be set according to actual needs. For example, the convolutional kernel can be set to 7x7, or it can be other suitable sizes such as 5x5, etc., and no excessive limitation is made here.

[0092] The batch normalization layer can accelerate model training, stabilize gradients, and reduce overfitting by normalizing the input distribution. The activation layer can endow the network with the ability to learn complex patterns through nonlinear transformation. Exemplarily, the activation function adopted by the activation layer includes ReLU (Rectified Linear Unit). Among them, the batch normalization layer and the activation layer are co-designed. The batch normalization layer can provide a stable input for the activation layer, and the activation layer introduces nonlinearity. The two jointly build an efficient and robust deep learning model.

[0093] The max pooling layer can reduce the data dimension, reduce the computational amount through downsampling, retain the local maximum value, highlight the texture and key features, suppress noise. In addition, it is insensitive to input position offset, improves the robustness of the model, can also reduce overfitting, and enhances the generalization ability of the model.

[0094] The first residual layer, the second residual layer, and the third residual layer are connected in sequence, that is, three cascaded residual blocks. In this way, by stacking three residual blocks, the expression ability of the deep network is enhanced, more complex features are learned, and at the same time, the gradient problem caused by depth is avoided. While ensuring the learning ability, the model structure can be simplified; in addition, the skip connection of the residual layer provides a direct path for the gradient, which can alleviate gradient disappearance and explosion, and support the training of deeper networks; in addition, the residual layer can simplify the learning task, accelerate convergence and optimization, make the network easier to find effective solutions, and improve the generalization ability and reduce the risk of overfitting. Exemplarily, the structures of the first residual layer are different from those of the second residual layer and the third residual layer respectively, and the structures of the second residual layer and the third residual layer are the same.

[0095] The global average pooling layer can retain the global statistical information of the feature map by calculating the global average value of each channel, and achieve dimensionality reduction and feature compression.

[0096] The fully connected layer can linearly combine the features of the previous layer through matrix operations and implement non-linear modeling in combination with activation functions. Exemplarily, the fully connected layer can include a flattening layer and a GEMM (General Matrix Multiply) operation.

[0097] In the image processing method provided in the above embodiments, the fog perception detection model receives a complete image, that is, directly inputs the entire image for holistic fog perception detection, rather than cutting the image into multiple image blocks for block-by-block detection. In this way, not only can the global information of the image be fully utilized, avoiding the problem of uneven defogging effect that may be caused by block-by-block detection, but also the accuracy of fog perception detection and the overall effect of image optimization can be improved. In addition, the fog perception detection model in this application can learn the global information of the image. Using three residual layers can have good fog perception detection ability. Compared with the models with more than three, such as four residual layers, in the related art, the fog perception detection model provided in this application has good fog perception detection performance, and the structure is simpler and the complexity is lower, which helps to improve the fog perception detection efficiency.

[0098] Please continue to refer to Figure 5 , in some embodiments, the first residual layer includes two convolutional layers, two batch normalization layers and one activation layer. The first residual layer includes a simple residual block, including a main branch and an auxiliary branch with residual connection, and an activation layer. The main branch of the first residual layer includes a convolutional layer (connected to a max pooling layer), a batch normalization layer, an activation layer, a convolutional layer and a batch normalization layer connected in sequence, and the max pooling layer and the batch normalization layer at the last stage of the main branch are output through an activation layer (such as Relu) after being connected by residual connection. Among them, the convolution kernel of the convolutional layer of the first residual layer can be set according to actual needs. For example, it can be 3x3, or other appropriate values, which will not be limited too much here.

[0099] The second residual layer and the third residual layer both include a complex residual block. The main branches of the second residual layer and the third residual layer both include two convolutional layers, two batch normalization layers and one activation layer. The auxiliary branches of the second residual layer and the third residual layer both include one convolutional layer and one batch normalization layer.

[0100] Among them, the second residual layer includes a main branch and an auxiliary branch with residual connections, and an activation layer. The main branch of the second residual layer includes a convolutional layer (connected to the first residual layer), a batch normalization layer, an activation layer, a convolutional layer, and a batch normalization layer connected in sequence. The auxiliary branch of the second residual layer includes a convolutional layer (connected to the first residual layer) and a batch convolutional layer connected in sequence. The batch normalization layer of the main branch of the second residual layer and the last-level batch normalization layer of the auxiliary branch are connected by residual connection and then output through an activation layer (such as Relu). Among them, the convolution kernel of the convolutional layer of the second residual layer can be set according to actual needs; for example, the convolution kernel of the main branch can be 3x3, and the convolution kernel of the auxiliary branch can be 1x1; the convolution kernel of the convolutional layer of the second residual layer can also be other appropriate values, which are not limited here.

[0101] Among them, the third residual layer includes a main branch and an auxiliary branch with residual connections, and an activation layer. The main branch of the third residual layer includes a convolutional layer (connected to the second residual layer), a batch normalization layer, an activation layer, a convolutional layer, and a batch normalization layer connected in sequence. The auxiliary branch of the third residual layer includes a convolutional layer (connected to the second residual layer) and a batch convolutional layer connected in sequence. The batch normalization layer of the main branch of the third residual layer and the last-level batch normalization layer of the auxiliary branch are connected by residual connection and then output through an activation layer (such as Relu). Among them, the convolution kernel of the convolutional layer of the third residual layer can be set according to actual needs; for example, the convolution kernel of the main branch can be 3x3, and the convolution kernel of the auxiliary branch can be 1x1; the convolution kernel of the convolutional layer of the third residual layer can also be other appropriate values, which are not limited here.

[0102] For the image processing method provided in the above embodiments, the first residual layer adopts a simple residual block structure, which can perform shallow fog feature extraction. The second residual layer and the third residual layer both adopt a complex residual block structure, which can perform deep fog feature extraction respectively, helping to improve the model's ability to extract fog features from images and simplify the model structure; compared with the model structure in the related art where each residual layer adopts multiple residual blocks, the fog detection model provided in this application is a lightweight ResNet model, or called ResNet8 model. Compared with the ResNet model in the related art, this application simplifies the network depth and complexity, and while ensuring the accuracy of image fog detection, improves the model detection efficiency.

[0103] In some embodiments, as Figure 6 shown, the image processing method further includes the following steps S602 and step S604.

[0104] S602: Perform object recognition on the image to be processed and obtain the object recognition result of the image to be processed.

[0105] S604: Determine a fogginess concentration probability threshold from a preset set of fogginess concentration probability thresholds according to the target recognition result.

[0106] Among them, the target recognition result includes a target image and a non-target image. The target recognition includes at least one of portrait recognition and day-night recognition.

[0107] The fogginess concentration probability threshold of the target image is less than that of the non-target image.

[0108] In the target image, the first type of image includes a non-fog image, and the second type of image includes a first-level foggy image and a second-level foggy image. In the non-target image, the first type of image includes a non-fog image and a first-level foggy image, and the second type of image includes a second-level foggy image.

[0109] For example, if the target recognition includes portrait recognition, the image processing method may further include: performing portrait recognition on the image to be processed, obtaining the portrait recognition result of the image to be processed, and determining a fogginess concentration threshold from a preset set of fogginess concentration thresholds according to the portrait recognition result. Among them, the portrait recognition result includes a portrait image and a non-portrait image. The fogginess concentration probability threshold of the portrait image is less than that of the non-portrait image. In the portrait image, the first type of image, such as a non-extremely foggy image, includes a non-fog image, and the second type of image, such as an extremely foggy image, includes a first-level foggy image (such as a slightly foggy image) and a second-level foggy image. In the non-portrait image, the first type of image (such as a non-extremely foggy image) includes a non-fog image and a first-level foggy image (such as a slightly foggy image), and the second type of image, such as an extremely foggy image, includes a second-level foggy image. Among them, any suitable portrait recognition method can be used for portrait recognition, and no further limitation is made here.

[0110] Another example, if the target recognition includes day-night recognition, the image processing method may further include: performing day-night recognition on the image to be processed, obtaining the day-night recognition result of the image to be processed, and determining a fogginess concentration threshold from a preset set of fogginess concentration thresholds according to the day-night recognition result; among them, the day-night recognition result includes a day image and a night image. The fogginess concentration probability threshold of the night image is less than that of the day. In the night image, the first type of image, such as a non-extremely foggy image, includes a non-fog image, and the second type of image, such as an extremely foggy image, includes a first-level foggy image (such as a slightly foggy image) and a second-level foggy image. In the day image, the first type of image (such as a non-extremely foggy image) includes a non-fog image and a first-level foggy image (such as a slightly foggy image), and the second type of image, such as an extremely foggy image, includes a second-level foggy image. Among them, any suitable day-night recognition method can be used for day-night recognition, and no further limitation is made here.

[0111] It can be understood that in the scenario of portrait recognition, since the fog on the portrait has a great impact on the visual effect, it is usually necessary to remove the fog in the portrait area as much as possible to ensure the clarity and detail performance of the portrait. Therefore, in the portrait recognition scenario, the fog classification boundary of the portrait image, that is, the fog concentration probability threshold, can be appropriately lowered, and some slightly foggy images can be reclassified into the extreme fog category. In this way, combined with the subsequent fog removal strategy for the portrait image, more sufficient and accurate fog removal processing can be performed on the portrait image, improving the processing effect of the image.

[0112] In the day-night recognition scenario, since the fog at night has a great impact on the visual effect, it is usually necessary to remove the fog in the night image as much as possible to ensure the clarity and detail performance of the night image. Therefore, in the day-night recognition scenario, the fog classification boundary of the night image, that is, the fog concentration probability threshold, can be appropriately lowered, and some slightly foggy images can be reclassified into the extreme fog category. In this way, combined with the subsequent fog removal strategy for the night image, more sufficient and accurate fog removal processing can be performed on the night image, improving the processing effect of the image.

[0113] In addition, the image processing method may further include: performing portrait recognition and day-night recognition on the image to be processed, obtaining the portrait recognition result and the day-night recognition result of the image to be processed, and determining the fog concentration threshold from a preset set of fog concentration thresholds according to the portrait recognition result and the day-night recognition result. Among them, the fog concentration probability threshold of the portrait night image is respectively less than the fog concentration probability thresholds of the portrait day image, the non-portrait day image, and the non-portrait night image, and the fog concentration probability threshold of the non-portrait day image is respectively greater than the fog concentration probability thresholds of the portrait day image and the non-portrait day image.

[0114] In the application, the recognition type of the image to be processed can be selected according to the actual scenario and the image processing requirements, and no excessive limitation is made here.

[0115] The image processing method provided by the above embodiment performs target recognition on the image to be processed, obtains the target recognition result of the image to be processed, and determines the fog concentration probability threshold from a preset set of fog concentration probability thresholds according to the target recognition result. In this way, by comparing the fog concentration probability threshold and the fog concentration probability of the image to be processed, the fog classification of the image to be processed is realized; among them, the target recognition includes at least one of portrait recognition and day-night recognition. In this way, while performing fog recognition on the image to be processed, the portrait scene recognition and day-night scene recognition of the image to be processed are added, and according to the portrait recognition result and / or the day-night scene recognition, the fog classification boundary of the image to be processed, that is, the fog concentration probability threshold, is adaptively adjusted, which can provide more accurate guidance for subsequent image fog removal processing or non-image fog removal processing of the image to be processed, balance the image processing effects in different scenarios, and improve the flexibility and adaptability of image processing.

[0116] Based on the above, by flexibly adjusting the fogginess classification boundary, it is possible to guide the subsequent image processing flow according to the specific requirements of different scenarios and implement more targeted optimization strategies. When the image to be processed is classified as the first type of image, such as a non-extremely foggy image, the subsequent processing can adopt the image processing flow of TMC; while when the input image is classified as the second type of image, such as an extremely foggy image, the subsequent processing can adopt the image dehazing processing strategy. By organically combining this classification with the image processing strategy, it is possible to more accurately meet the image optimization requirements in various scenarios and significantly improve the processing ability for complex and diverse images.

[0117] In some embodiments, as Figure 7 shown, S106, perform image optimization processing on the image to be processed according to the fogginess recognition result to obtain the target image, including the following steps S702 and S704.

[0118] S702: In the case where the fogginess recognition result is the second type of image, determine the corresponding image dehazing algorithm from the image dehazing algorithm library according to the fogginess recognition result and the fogginess concentration probability.

[0119] S704: Use the image dehazing algorithm of the image to be processed to perform image dehazing processing on the image to be processed to obtain the target image.

[0120] Based on the foregoing, the first type of image, such as a non-extremely foggy image, may include a non-fog image and the first extremely foggy image, that is, a slightly foggy image, and the second type of image, such as an extremely foggy image, may include the second-level foggy image. Thus, although the fogginess recognition result of the image to be processed includes two categories, the first type of image and the second type of image, it includes an image distribution with three different fog concentrations.

[0121] In this regard, in the case where the fogginess recognition result is the second type of image, the fogginess recognition result and the non-fog concentration probability can be combined to select a suitable image dehazing algorithm from the non-fog dehazing algorithm library, and use this image dehazing algorithm to perform image dehazing processing on the image to be processed.

[0122] Among them, the image dehazing algorithm library includes at least one image dehazing algorithm, and the image dehazing algorithm is used to perform dehazing processing on the image. The image dehazing algorithm includes, but is not limited to, an end-to-end mapping method based on a convolutional neural network, a dehazing method based on Transformer and a generative adversarial network, and will not be limited too much here.

[0123] The degree of image defogging processing for a to-be-processed image with a high probability of fogginess concentration is greater than that of a to-be-processed image with a low probability of fogginess concentration. It can be understood that the higher the probability of fogginess concentration, the higher the fogginess concentration of the to-be-processed image, and the worse the indicators such as clarity, contrast, and saturation of the to-be-processed image. In this regard, an image defogging algorithm with stronger defogging performance can be used to perform image defogging processing on the to-be-processed image to perform a higher degree of image defogging processing on the to-be-processed image. The lower the probability of fogginess concentration, the lower the fogginess concentration of the to-be-processed image, and the better the indicators such as clarity, contrast, and saturation of the to-be-processed image. In this regard, an image defogging algorithm with lower defogging performance can be used to perform image defogging processing on the to-be-processed image to perform a lower degree of image defogging processing on the to-be-processed image.

[0124] It should be noted that in the case where the fogginess recognition result is the first type of image, it indicates that the fogging degree of the to-be-processed image is very low, and the indicators such as clarity, contrast, and saturation of the to-be-processed image meet the image requirements. In this regard, image defogging processing may not be performed on this type of image to avoid problems such as distortion of contrast and saturation caused by performing defogging processing on the first type of image, and to improve the balanced processing ability for non-fog images and fogged images.

[0125] For the image processing method provided in the above embodiment, in the case where the fogginess recognition result is the second type of image, according to the fogginess recognition result and the probability of fogginess concentration, a corresponding image defogging algorithm is determined from the image defogging algorithm library, and the to-be-processed image is processed by using the image defogging algorithm of the to-be-processed image to obtain a target image. In this way, it is possible to adaptively adjust the degree of image defogging processing for the to-be-processed image in combination with the probability of fogginess concentration, balance the image defogging effects of fogged images with different fog concentrations, and have strong flexibility.

[0126] In some embodiments, the image processing method further includes: performing target recognition on the to-be-processed image and obtaining the target recognition result of the to-be-processed image; wherein, the target recognition includes at least one of portrait recognition and day-night recognition. For specific details, reference can be made to the relevant introduction above and will not be elaborated here.

[0127] Correspondingly, S704, determining a corresponding image defogging algorithm from the image defogging algorithm library according to the fogginess recognition result and the probability of fogginess concentration includes: the step of determining a corresponding image defogging algorithm from the image defogging algorithm library according to the target recognition result, the fogginess recognition result, and the probability of fogginess concentration. Wherein, the target recognition result includes at least one of the portrait recognition result and the day-night recognition result.

[0128] Based on the foregoing, in portrait images, the first type of images includes non-foggy images, and the second type of images includes first-level foggy images and second-level foggy images; for example, non-extremely foggy images include non-foggy images, and extremely foggy images include foggy images (including slightly foggy and extremely foggy); in this regard, the corresponding image defogging algorithm can be determined from the image defogging algorithm library according to the portrait recognition result, the fog sense recognition result, and the fog sense concentration probability, so as to perform image defogging processing on the image to be processed according to the image defogging algorithm. Among them, in the second type of images such as extremely foggy images, the degree of image defogging processing for portrait images with a high fog sense concentration probability is greater than that for portrait images with a low fog sense concentration probability. In this way, for portrait foggy images that need to be subjected to image defogging processing, the corresponding image defogging algorithm can be adaptively adjusted in combination with the fog sense concentration probability, which can balance the image defogging effects of foggy images with different fog concentrations in portrait images and helps to improve the overall processing performance of the images.

[0129] In night images, the first type of images includes non-foggy images, and the second type of images includes first-level foggy images and second-level foggy images; for example, non-extremely foggy images include non-foggy images, and extremely foggy images include foggy images (including slightly foggy and extremely foggy); in this regard, the corresponding image defogging algorithm can be determined from the image defogging algorithm library according to the day-night recognition result, the fog sense recognition result, and the fog sense concentration probability, so as to perform image defogging processing on the image to be processed according to the image defogging algorithm. Among them, in the second type of images such as extremely foggy images, the degree of image defogging processing for night images with a high fog sense concentration probability is greater than that for night images with a low fog sense concentration probability. In this way, for night foggy images that need to be subjected to image defogging processing, the corresponding image defogging algorithm can be adaptively adjusted in combination with the fog sense concentration probability, which can balance the image defogging effects of foggy images with different fog concentrations in night images and helps to improve the overall processing performance of the images.

[0130] In some embodiments, in combination with the foregoing Figures 1 to 7 , such as Figure 8 shown, a method for image processing is provided, and the method includes the following steps S802 to step S818.

[0131] S802: Construct a sample set.

[0132] The sample set includes a plurality of sample data pairs, and the sample data pairs include non-extremely foggy images and extremely foggy images, where the non-extremely foggy images include non-foggy images and slightly foggy images.

[0133] S804: Use the sample set and the multi-class focal loss function to train the initial fog sense detection model to obtain a trained fog sense detection model.

[0134] The multi-class focal loss function can be specifically referred to the aforementioned formula (1). For the specific structure and training process of the fog detection model, please refer to the aforementioned Figure 5 and related introductions, which will not be elaborated here.

[0135] S806: Obtain the image to be processed.

[0136] S808: Use the trained fog detection model to perform fog recognition on the image to be processed, and obtain the fog concentration probability of the image to be processed.

[0137] S810: Perform human face recognition on the image to be processed, and obtain the human face recognition result of the image to be processed.

[0138] S812: Determine the fog concentration probability threshold of the image to be processed from the preset fog concentration threshold set according to the human face recognition result.

[0139] S814: Compare the fog concentration probability of the image to be processed with the fog concentration probability threshold, and obtain the fog recognition result of the image to be processed.

[0140] S816: In the case where the fog recognition result is a non-extremely foggy image, perform image color mapping processing on the image to be processed.

[0141] S818: In the case where the fog recognition result is an extremely foggy image, perform image de-fogging processing on the image to be processed.

[0142] The image processing method provided by the above embodiments effectively overcomes key quality problems such as significant decrease in image contrast and massive loss of color details in extremely foggy scenes through the deep integration of the fog detector and the image processing strategy. This solution realizes intelligent fog recognition and effect optimization, and can accurately select the corresponding image optimization processing or targeted de-fogging optimization strategy according to the specific state of the input image. Whether in daily ordinary scenes (such as non-fog and slightly foggy scenes) or extremely challenging extremely foggy scenes, the system can flexibly switch to the corresponding image processing strategy according to the fog recognition result, so as to ensure stable output of high-quality image effects in different scenes and provide users with a consistent high-quality visual experience.

[0143] It should be understood that although the steps in the flowcharts involved in the above embodiments are shown in sequence according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise clearly stated in this article, there is no strict order limit for the execution of these steps, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above embodiments may include multiple steps or multiple stages. These steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be executed alternately or in turn with at least a part of other steps or steps or stages in other steps.

[0144] Based on the same inventive concept, an embodiment of the present application also provides an image processing apparatus for implementing the above-mentioned image processing method. The solution provided by this apparatus for solving problems is similar to the solution described in the above method. Therefore, the specific limitations in one or more embodiments of the following image processing apparatus can refer to the limitations on the image processing method in the above text and will not be repeated here.

[0145] In some embodiments, as Figure 9 shown, an image processing apparatus 900 is provided, including an acquisition module 901, an identification module 902, and a processing module 903. Among them, the acquisition module 901 is used to acquire an image to be processed. The identification module 902 is used to perform fog perception identification on the image to be processed and obtain the fog perception identification result of the image to be processed; the fog perception identification result includes a first type of image or a second type of image, and the fog concentration of the first type of image is less than that of the second type of image. The processing module 903 is used to perform image processing on the image to be processed according to the fog perception identification result to obtain a target image; the image processing methods for the first type of image and the second type of image are different, and the image processing of the second type of image includes defogging processing.

[0146] The image processing device 900 provided by the above embodiments obtains the image to be processed through the acquisition module 901, performs fog-sensation recognition on the image to be processed through the recognition module 902, and obtains the fog-sensation recognition result of the image to be processed. Then, through the processing module 903, image processing is performed on the image to be processed according to the fog-sensation recognition result to obtain the target image. Among them, the fog-sensation recognition result includes a first type of image and a second type of image. The fog-sensation concentration of the first type of image is less than that of the second type of image, and the image processing of the first type of image is different from that of the second type of image. The image processing of the second type of image includes image defogging processing. In this way, the recognition of the image in the fog-sensation dimension is realized, and different image processing strategies can be intelligently switched according to different fog-sensation recognition results, realizing the precise processing of images with different fog-sensation concentrations, improving the processing ability and image processing effect of complex and diverse images, meeting the image processing requirements in different fog-sensation concentration scenarios, and balancing the image processing effects in different fog-sensation concentration scenarios. Compared with the problem of color deviation caused by defogging all images in the related art, the present application can adopt different image processing methods for different types of images, that is, not perform defogging processing on the first type of image and perform defogging processing on the second type of image. This not only solves the problems such as the decrease in image contrast and the loss of color details caused by fog in the foggy scene, but also takes into account the image processing requirements in ordinary scenes such as non-fog scenes and slightly foggy scenes, avoiding the color deviation problems such as contrast and saturation distortion that may be caused by defogging non-fog images and images with low fog-sensation concentration (i.e., non-extremely foggy images). Thus, high-quality image effects can be maintained in different scenes, and the image processing quality is improved. In addition, the present application performs fog-sensation recognition on the entire image to be processed. Compared with cutting the image into multiple image blocks for block detection, it makes full use of the global information of the image, avoids the problem of uneven defogging effect in the subsequent image processing process caused by block detection, and improves the accuracy of fog-sensation recognition and the overall effect of image processing.

[0147] In some embodiments, the acquisition module is further configured to perform fog-sensation recognition on the image to be processed by using a trained fog-sensation detection model, and obtain the fog-sensation concentration probability of the image to be processed; and obtain the fog-sensation recognition result of the image to be processed according to the fog-sensation concentration probability and a preset fog-sensation concentration probability threshold.

[0148] In some embodiments, the image processing device further includes a construction module and a training module. The construction module is used to construct a sample set; the sample set includes a plurality of sample data pairs, and each sample data pair includes a first type of sample image and a second type of sample image; in the sample data pair, the first type of sample image includes a non-fog sample image or a first-level foggy sample image, and the second type of sample image includes at least a second-level foggy sample image; the fogginess concentration of the first-level foggy sample image is less than that of the second-level foggy sample image. The training module is used to train an initial fogginess detection model using the sample set and obtain a trained fogginess detection model.

[0149] In some embodiments, the training module is further used to input the sample set into the initial fogginess detection model to obtain a predicted fogginess concentration probability; input the predicted fogginess concentration probability into a multi-class focal loss function to obtain a loss value; adjust the parameters of the initial fogginess detection model according to the loss value, and obtain a trained fogginess detection model when the model training conditions are met; the multi-class focal loss function includes a focal parameter and a balance weight, the multi-class focal loss function is positively correlated with the focal parameter, and the multi-class focal loss function is negatively correlated with the balance weight.

[0150] In some embodiments, the recognition module is further used to perform target recognition on the image to be processed and obtain the target recognition result of the image to be processed; the target recognition result includes a target image and a non-target image, and the target recognition includes at least one of person recognition and day-night recognition.

[0151] The image processing device further includes a determination module, and the determination module is used to determine a fogginess concentration probability threshold from a preset set of fogginess concentration probability thresholds according to the target recognition result; wherein, the fogginess concentration probability threshold of the target image is less than that of the non-target image; in the target image, the first type of image includes a non-fog image, and the second type of image includes a first-level foggy image and a second-level foggy image; in the non-target image, the first type of image includes a non-fog image and a first-level foggy image, and the second type of image includes a second-level foggy image.

[0152] In some embodiments, the processing module is further used to, when the fogginess recognition result is a second-type image, determine a corresponding image defogging algorithm from the image defogging algorithm library according to the fogginess recognition result and the fogginess concentration probability; use the image defogging algorithm of the image to be processed to perform image defogging processing on the image to be processed to obtain a target image; the image defogging processing degree of the image to be processed with a high fogginess concentration probability is greater than that of the image to be processed with a low fogginess concentration probability.

[0153] In some embodiments, the recognition module is further configured to perform object recognition on the image to be processed and obtain the object recognition result of the image to be processed; the object recognition result includes a target image and a non-target image, and the object recognition includes at least one of portrait recognition and day-night recognition. The determination module is further configured to determine a corresponding image defogging algorithm from the image defogging algorithm library according to the object recognition result, the fogginess recognition result, and the fogginess concentration probability.

[0154] Each module in the above image processing device can be implemented in whole or in part by software, hardware, and their combination. The above modules can be embedded in the processor of the computer device in hardware form or independent of it, or stored in the memory of the computer device in software form, so that the processor can call and execute the operations corresponding to the above modules.

[0155] In some embodiments, an electronic device is provided. The electronic device can be a server, and its internal structure diagram can be as Figure 10 shown. The electronic device includes a processor, a memory, and a network interface connected through a system bus. Among them, the processor of the electronic device is used to provide computing and control capabilities. The memory of the electronic device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the electronic device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, it implements an image processing method.

[0156] Those skilled in the art can understand that Figure 10 the structure shown in

[0157] is only a block diagram of some structures related to the solution of this application, and does not constitute a limitation on the electronic device to which the solution of this application is applied. The specific electronic device may include more or fewer components than those shown in the figure, or combine some components, or have a different component layout.

[0158] In some embodiments, an electronic device is provided, including a memory and a processor. A computer program is stored in the memory, and when the processor executes the computer program, the steps of the foregoing method are implemented.

[0159] In some embodiments, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by the processor, the steps of the foregoing method are implemented.

[0160] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties.

[0161] Those of ordinary skill in the art can understand that all or part of the processes in the methods of the above embodiments can be completed by instructing relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above methods. Among them, any reference to a memory, database, or other medium used in the embodiments provided in this application can include at least one of non-volatile and volatile memories. Non-volatile memories can include read-only memory (ROM), magnetic tapes, floppy disks, flash memories, optical memories, high-density embedded non-volatile memories, resistive random access memories (ReRAM), magnetoresistive random access memories (MRAM), ferroelectric random access memories (FRAM), phase change memories (PCM), graphene memories, etc. Volatile memories can include random access memory (RAM) or external cache memories, etc. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc. The databases involved in the embodiments provided in this application can include at least one of relational databases and non-relational databases. Non-relational databases can include distributed databases based on blockchain, etc., without limitation. The processors involved in the embodiments provided in this application can be general-purpose processors, central processors, graphics processors, digital signal processors, programmable logic devices, data processing logics based on quantum computing, etc., without limitation.

[0162] The technical features of the above embodiments can be combined arbitrarily. For the sake of brevity of description, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, it should be considered as the scope recorded in this specification.

[0163] The above-described embodiments merely represent several implementation manners of the present application. The description thereof is relatively specific and detailed, but it should not be construed as a limitation to the patent scope of the present application. It should be noted that for those of ordinary skill in the art, without departing from the concept of the present application, several modifications and improvements can still be made, and these all fall within the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the appended claims.

Claims

1. An image processing method, characterized in that, The method includes: Obtaining an image to be processed; Performing fogginess recognition on the image to be processed, and obtaining a fogginess recognition result of the image to be processed; the fogginess recognition result includes a first type of image or a second type of image, and the fogginess concentration of the first type of image is less than that of the second type of image; Performing image processing on the image to be processed according to the fogginess recognition result to obtain a target image; the image processing of the first type of image is different from that of the second type of image, and the image processing of the second type of image includes image dehazing processing.

2. The method according to claim 1, characterized in that, The performing fogginess recognition on the image to be processed and obtaining the fogginess recognition result of the image to be processed includes: Performing fogginess recognition on the image to be processed by using a trained fogginess detection model, and obtaining a fogginess concentration probability of the image to be processed; Obtaining the fogginess recognition result of the image to be processed according to the fogginess concentration probability and a preset fogginess concentration probability threshold.

3. The method according to claim 2, wherein The method further includes: Constructing a sample set; the sample set includes a plurality of sample data pairs, and each sample data pair includes a first type of sample image and a second type of sample image; in the sample data pair, the first type of sample image includes a fog-free sample image or a first-level foggy sample image, and the second type of sample image includes at least a second-level foggy sample image; the fogginess concentration of the first-level foggy sample image is less than that of the second-level foggy sample image; Training an initial fogginess detection model by using the sample set, and obtaining a trained fogginess detection model.

4. The method according to claim 3, wherein The training the pre-established fogginess detection model by using the sample set and obtaining a trained fogginess detection model includes: Inputting the sample set into the initial fogginess detection model to correspondingly obtain a fogginess concentration prediction probability; Inputting the fogginess concentration prediction probability into a multi-class focal loss function to correspondingly obtain a loss value; Adjusting the parameters of the initial fogginess detection model according to the loss value, and obtaining a trained fogginess detection model when the model training conditions are met; the multi-class focal loss function includes a focal parameter and a balance weight, the multi-class focal loss function is positively correlated with the focal parameter, and the multi-class focal loss function is negatively correlated with the balance weight.

5. The method according to any one of claims 2-4, characterized in that, The fogginess detection model includes: An input layer for receiving an input image; A hidden layer including at least a first residual layer, a second residual layer, and a third residual layer connected in sequence, wherein the first residual layer is used to connect with the input layer, and the first residual layer, the second residual layer, and the third residual layer are respectively used to extract fogginess features of the input image; An output layer for connecting with the third residual layer and used to output a fogginess concentration probability.

6. The method according to claim 5, wherein The first residual layer includes two convolutional layers, two batch normalization layers, and one activation layer; The main branches of the second residual layer and the third residual layer each include two convolutional layers, two batch normalization layers, and one activation layer, and the auxiliary branches of the second residual layer and the third residual layer each include one convolutional layer and one batch normalization layer.

7. The method according to any one of claims 2-4, characterized in that, The method further includes: Perform object recognition on the image to be processed, and obtain the object recognition result of the image to be processed; the object recognition result includes a target image and a non-target image, and the object recognition includes at least one of human recognition and day / night recognition; Determine the fogginess concentration probability threshold from a preset set of fogginess concentration probability thresholds according to the object recognition result; wherein, the fogginess concentration probability threshold of the target image is less than that of the non-target image; in the target image, the first type of image includes a non-fog image, and the second type of image includes a first-level foggy image and a second-level foggy image; in the non-target image, the first type of image includes a non-fog image and a first-level foggy image, and the second type of image includes a second-level foggy image.

8. The method according to any one of claims 2 to 4, characterized in that, The image optimization processing of the image to be processed according to the fogginess recognition result to obtain a target image includes: In the case where the fogginess recognition result is the second type of image, determine the corresponding image de-fogging algorithm from the image de-fogging algorithm library according to the fogginess recognition result and the fogginess concentration probability; Use the image de-fogging algorithm of the image to be processed to perform image de-fogging processing on the image to be processed to obtain the target image; the image de-fogging processing degree of the image to be processed with a high fogginess concentration probability is greater than that of the image to be processed with a low fogginess concentration probability.

9. The method according to claim 8, wherein The method further includes: Perform object recognition on the image to be processed, and obtain the object recognition result of the image to be processed; the object recognition result includes a target image and a non-target image, and the object recognition includes at least one of portrait recognition and day / night recognition; Correspondingly, the determining the corresponding image de-fogging algorithm from the image de-fogging algorithm library according to the fogginess recognition result and the fogginess concentration probability includes: Determine the corresponding image de-fogging algorithm from the image de-fogging algorithm library according to the object recognition result, the fogginess recognition result and the fogginess concentration probability.

10. An image processing apparatus, characterized in that, The device includes: An acquisition module, configured to acquire an image to be processed; A recognition module, configured to perform fogginess recognition on the image to be processed, and obtain the fogginess recognition result of the image to be processed; the fogginess recognition result includes a first type of image or a second type of image, and the fogginess concentration of the first type of image is less than that of the second type of image; A processing module, configured to perform image processing on the image to be processed according to the fogginess recognition result to obtain a target image; the image processing methods of the first type of image and the second type of image are different, and the image processing of the second type of image includes de-fogging processing.

11. An electronic device, comprising a memory and a processor, the memory storing a computer program, characterized in that, When the processor executes the computer program, the steps of the method according to any one of claims 1 to 9 are implemented.

12. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, the steps of the method according to any one of claims 1 to 9 are implemented.

13. A computer program product comprising a computer program, characterized in that, When the computer program is executed by the processor, the steps of the method according to any one of claims 1 to 9 are implemented.

Images