Image processing method, device and computer readable storage medium

Abstract

The application discloses an image processing method, device and computer readable storage medium. The image processing method comprises the following steps: determining a probability distribution corresponding to degradation information in a to-be-processed image, and eliminating the degradation information based on the probability distribution; after eliminating the degradation information in the to-be-processed image, determining task information and underlying information contained in the to-be-processed image according to image features in the to-be-processed image; and repairing the to-be-processed image according to the task information and the underlying information of the to-be-processed image. Through the above method steps, even if there are multiple different interference sources in the image, the degradation information in the image can be well eliminated, and a clear image can be restored.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of image processing, in particular to an image processing method, device and computer readable storage medium. BACKGROUND

[0002] In an automatic driving system, when a car encounters extreme weather during driving, the image clarity obtained by the automatic driving system is very low, which will affect the judgment made by the automatic driving system and the personal safety of the driver. In order to improve the safety of automatic driving, it is necessary to eliminate the influence of weather on the image, so that the computer vision system is more reliable.

[0003] At present, in the processing process of repairing bad weather images, most task models focus on local repair of images. When there are multiple different interference sources in the image, for example, an image affected by both rain and fog, these models often cannot well eliminate the degradation information in the image and restore a clear image.

[0004] The above content is only used to assist in understanding the technical solutions of the present application and does not represent the acknowledgement of the above content as prior art. SUMMARY

[0005] The main purpose of the present application is to provide an image processing method, device and computer readable storage medium, which aims to solve the technical problem that most task models currently focus on local repair of images in the processing process of repairing bad weather images, and often cannot well eliminate the degradation information in the image and restore a clear image when there are multiple different interference sources in the image.

[0006] To achieve the above purpose, the present application provides an image processing method, which comprises the following steps:

[0007] determining the probability distribution corresponding to the degradation information in the image to be processed, and eliminating the degradation information based on the probability distribution;

[0008] After eliminating the degradation information in the image to be processed, determining the task information and the underlying information contained in the image to be processed according to the image features in the image to be processed;

[0009] repairing the image to be processed according to the task information and the underlying information of the image to be processed.

[0010] Optionally, the step of determining the probability distribution corresponding to the degradation information in the image to be processed, and eliminating the degradation information based on the probability distribution comprises:

[0011] acquire the to-be-processed image, and simulate the corresponding probability distribution based on the degradation information in the to-be-processed image;

[0012] perform deconvolution processing on the to-be-processed image based on the simulated probability distribution, and eliminate the degradation information.

[0013] Optionally, before the step of determining the task information and the underlying information contained in the to-be-processed image according to the image features in the to-be-processed image after eliminating the degradation information in the to-be-processed image, the method further comprises:

[0014] determining the image patches corresponding to the to-be-processed image based on a patch embedding module;

[0015] performing down-sampling processing on the image patches corresponding to the to-be-processed image based on a conversion module;

[0016] determining the image features corresponding to the to-be-processed image according to the data obtained through the down-sampling processing.

[0017] Optionally, the step of determining the task information and the underlying information contained in the to-be-processed image according to the image features in the to-be-processed image after eliminating the degradation information in the to-be-processed image further comprises:

[0018] normalizing the feature sequence corresponding to the image features;

[0019] determining the underlying information based on a preset neural network according to the normalized feature sequence.

[0020] Optionally, the step of determining the task information and the underlying information contained in the to-be-processed image according to the image features in the to-be-processed image after eliminating the degradation information in the to-be-processed image comprises:

[0021] taking the feature sequence corresponding to the image features as an input parameter of a task conversion module;

[0022] performing decoding processing on the feature sequence based on the task conversion module to obtain task features corresponding to the image features;

[0023] determining the task information according to the task features.

[0024] Optionally, after the step of determining the task information according to the task features, the method further comprises:

[0025] taking the task information as guide information for next down-sampling processing;

[0026] Based on the guidance information, the conversion module supplements the detail information of the image features when performing the down-sampling processing on the to-be-processed image.

[0027] Optionally, the step of repairing the to-be-processed image according to the task information and the bottom information of the to-be-processed image comprises:

[0028] calculating the fusion proportion of the task information and the bottom information corresponding to the task information and the bottom information of the to-be-processed image respectively;

[0029] fusing the task information and the bottom information according to the fusion proportion of the task information and the bottom information corresponding to the task information and the bottom information respectively.

[0030] Optionally, after the step of fusing the task information and the bottom information according to the fusion proportion, the method further comprises:

[0031] calculating the loss parameter corresponding to the image based on the fused image;

[0032] adjusting the fusion proportion of the next fusion according to the loss parameter.

[0033] In addition, to achieve the above object, the present application further provides a device, which comprises a memory, a processor and an image processing program stored in the memory and executable on the processor, and the image processing program is configured to implement the steps of the image processing method as described above.

[0034] In addition, to achieve the above object, the present application further provides a computer readable storage medium, which stores an image processing program, and the image processing program is executed by a processor to implement the steps of the image processing method as described above.

[0035] The present application provides an image processing method, which firstly pre-processes the to-be-processed image, i.e. determines the probability distribution corresponding to the degradation information in the to-be-processed image, performs deconvolution processing on it to eliminate the degradation information in the to-be-processed image, then extracts the image features in the to-be-processed image to determine the task information and the bottom information contained in the current to-be-processed image, and finally calculates the fusion proportion of the task information and the bottom information respectively, and repairs the to-be-processed image according to the fusion proportion. Through the above method, even if there are multiple different interference sources in the image, the degradation information in the image can also be well eliminated to restore a clear image. BRIEF DESCRIPTION OF DRAWINGS

[0036] Figure 1 The flowchart of the first embodiment of the image processing method of the present application is shown in the figure;

[0037] Figure 2 The flowchart of the second embodiment of the image processing method of the present application is shown in the figure; Figure 1The detailed flowchart of step S20;

[0038] Figure 3 For Figure 1 The detailed flowchart of another refinement of step S20;

[0039] Figure 4 For Figure 1 The detailed flowchart of step S30;

[0040] Figure 5 The flowchart of the second embodiment of the image processing method of the present application;

[0041] Figure 6 The model reference diagram in the first embodiment;

[0042] Figure 7 The effect diagram of the super-Laplacian preprocessing in the first embodiment;

[0043] Figure 8 The execution flowchart of the task conversion module in the first embodiment;

[0044] Figure 9 The terminal structure diagram of the hardware running environment involved in the embodiment scheme of the present application.

[0045] The implementation, functional features and advantages of the present application will be further described with reference to the accompanying drawings in conjunction with the embodiments. DETAILED DESCRIPTION

[0046] It should be understood that the specific embodiments described herein are merely intended to explain the present application and not to limit the present application.

[0047] How to remove bad weather conditions such as rain, snow, fog, etc. from images has always been an important problem in many fields. It is an important problem in many fields. Extreme weather can greatly reduce the ability of computer vision algorithms to extract information from images. For example, in an automatic driving system, when the car encounters extreme weather during driving, the image clarity obtained by the automatic driving system is very low, which will greatly affect the judgment made by the automatic driving system and affect the personal safety of the driver. Therefore, we must remove the influence of weather on images to make the computer vision system more reliable. Therefore, how to simultaneously restore multiple weather-degraded images is the key research content in the above-mentioned fields.

[0048] The current field still has problems: at present, most of the image processing in bad weather repair is modeling for a single task, but in real life we will face various weather conditions, so our device needs to train multiple networks at the same time to meet various weather conditions, but this will make our model very complex. In addition, most of the current multi-task models focus on local repair areas, and when there is a large area of degradation information such as fog that changes little, most models cannot achieve good repair.

[0049] To solve the above technical problems, the present application provides an image processing method, device and computer readable storage medium. The repair of multiple weather degradation tasks can be processed at the same time. First, the image to be processed will pass through the super Laplace preprocessing module, and the degradation information existing in the image to be processed will be preliminarily removed through the module. After eliminating the degradation information in the image to be processed, the image features corresponding to different weather types in the image to be processed are extracted through the processing of the patch embedding module and the conversion module. The image features can determine the bottom information and task information in the image to be processed. Different information will be extracted according to different weather degradation types to realize multi-degradation type restoration. Finally, the decoded image will pass through adaptive hybrid upsampling, calculate the fusion proportion of the task information and the bottom information, and selectively fuse the bottom information and the task information according to the fusion proportion, so as to restore a clear and real image.

[0050] The embodiment of the present application provides an image processing method, referring to Figure 1 , Figure 1 The flowchart of the first embodiment of the image processing method of the present application is shown in the figure.

[0051] In the embodiment, the image processing method comprises:

[0052] Step S10, determine the probability distribution corresponding to the degradation information in the image to be processed, and eliminate the degradation information based on the probability distribution.

[0053] In the imaging process, the image quality is reduced due to the influence of different factors, which is called image degradation. In the embodiment, the image degradation caused by weather factors is mainly solved. In some bad weather conditions (such as rain, snow, fog, etc.), the image will become blurred and unclear, and the factors such as large area of fog and rain that cause the image to be unclear are called degradation information in the image to be processed. In the past processing method of eliminating degradation information, most of them are based on known degradation information modeling, for example, assuming that the distribution information of the image conforms to Gaussian distribution, the image can be quickly restored in the frequency domain. But in the real world, the edge information of the image does not satisfy the Gaussian distribution, so the image effect of using this method for image preprocessing recovery is not good.

[0054] Optionally, in order to better eliminate large-area degradation information in the image, we here adopt the super-Laplacian deconvolution method, model based on this method, and apply it to the preprocessing of weather degradation image (to-be-processed image), thereby effectively eliminating the degradation information in the to-be-processed image. It should be noted that, compared with the assumption that the image conforms to the Gaussian distribution, the edge information distribution in the image in the real world is more in line with the super-Laplacian distribution, so we can preliminarily assume that the degradation information in the image conforms to the super-Laplacian distribution, simulate the corresponding super-Laplacian distribution by means of a mathematical model, and then combine the super-Laplacian distribution to perform deconvolution processing on the to-be-processed image, thereby achieving the purpose of preliminarily eliminating large-area degradation information in the to-be-processed image.

[0055] Optionally, the target function model can be processed by means of an algorithm to determine the data fidelity term of the super-Laplacian distribution and calculate the corresponding target function. The input parameters of the target function are determined, the initial inner loop iteration number is set, and the degradation information is eliminated by cyclic iteration for the to-be-processed image.

[0056] Referring to Figure 7 We input three pictures, which are three scenes in heavy rain weather, images after super-Laplacian preprocessing, and large-area degradation information such as fog and heavy rain is preliminarily filtered out.

[0057] In this embodiment, by using the super-Laplacian deconvolution scoring method, the large-area degradation information in the to-be-processed image can be preliminarily eliminated, so that the image features can be clearly extracted and the bottom information and task information of the image can be determined in the subsequent process.

[0058] Step S20, after eliminating the degradation information in the to-be-processed image, determining the task information and bottom information contained in the to-be-processed image according to the image features in the to-be-processed image.

[0059] In this embodiment, by means of the image features in the to-be-processed image, the task information and bottom information contained in the to-be-processed image can be further determined. Optionally, the network model can be used here to extract the image features, and the network model is as follows: Figure 6In the model, the image to be processed passes through the super Laplace preprocessing module to preliminarily eliminate the degradation information, and the preprocessed image is input into a plurality of patch embedding modules and a task patch embedding module for image feature extraction. The extracted image features are input into a plurality of task conversion modules to generate task features containing a large amount of task information. According to the task features, the corresponding task information can be determined, and the image features can determine the underlying information of the image to be processed. The task feature can be understood as the type of task that needs to be repaired in the image to be processed. For example, when an image is taken in a heavy rain scene, the task feature obtained from the image is the current weather feature. The underlying information can be understood as the feature information originally contained in the image, such as contour, edge, color, texture, and shape features.

[0060] It should be noted that the extraction of image features is multiple. As shown in Figure 6 The encoder model includes a plurality of patch embedding modules and conversion modules, and the task patch embedding module is composed of a patch embedding module and a task conversion module, and the task embedding patch module acts on each stage of the encoder. The patch embedding module will block the image to be processed. These small images processed by blocking are called patches. Further, the patches are further blocked into sub-patches with half the height and width of the original patches. This is more conducive to the extraction of image detail features and also helps to remove small degradation when operating on smaller patches. The sub-patches created from the original patches are input into the task embedding module, and through the task conversion module in the task embedding module, the task information corresponding to the image features of the image to be processed can be extracted. The task information is supplemented into the patch embedding module for the next feature extraction, which can guide the supplementation of different weather types of detail information in the next feature extraction. In this way, the next feature extraction can more accurately extract the weather type detail information in the image with the help of the guidance of the task information, and also can extract multiple weather types. We add task feature information in the patch embedding module to guide different weather types to supplement different types of detail information, and each time the feature extraction is performed, the feature extraction becomes more accurate.

[0061] Optionally, with reference to Figure 2 The step of determining the task information and the underlying information contained in the image to be processed according to the image features in the image to be processed comprises:

[0062] Step S21, normalizing the feature sequence corresponding to the image features.

[0063] Step S22, determining the underlying information based on a preset neural network according to the normalized feature sequence.

[0064] In this embodiment, the bottom layer information is extracted by means of the conversion module in the model, and the detailed division of the conversion module on the right side in FIG. 8 is as follows. Figure 6 First, the feature sequence corresponding to the extracted image feature is normalized, the attention weight of each position of each feature sequence in the encoding process is directly calculated by combining the multi-head attention mechanism, then the hidden vector representation of the feature sequence is calculated in the form of weight sum, and then the second normalization processing is performed to keep the weight value in the range of 0 to 1, and the feedforward neural network is combined for mapping, and finally the bottom layer information of the image to be processed is obtained.

[0065] Optionally, referring to Figure 3 , the step of determining the task information and the bottom layer information contained in the image to be processed according to the image feature in the image to be processed further includes:

[0066] Step S23, taking the feature sequence corresponding to the image feature as an input parameter of the task conversion module.

[0067] Step S24, decoding the feature sequence based on the task conversion module to obtain the task feature corresponding to the image feature.

[0068] Step S25, determining the task information according to the task feature.

[0069] In this embodiment, the task feature is still determined according to the image feature, please refer to Figure 6 , the image feature extracted based on the patch embedding module and the conversion module can be used as the input of the task conversion module in the decoder, that is, the feature sequence corresponding to the image feature is taken as the input parameter of the task conversion module, and then decoding processing is performed to obtain the corresponding task feature, and these task features contain a large amount of task information. Optionally, referring to Figure 8 , the specific flow chart of the task conversion module is as follows, which is changed from the traditional self-attention module. Different from the traditional self-attention module, q of the module comes from the externally introduced learnable task type sequence, k and v come from the output of the encoder. Q can be understood as a query vector, k is a searched vector, and v is a content vector. Similarly, by inputting into the feedforward neural network after normalization through the head attention mechanism, the task feature containing the task information can be obtained. In addition, the task feature sequence is also trained together with the network, so as to guide the network to have the function of clearly restoring the image.

[0070] In the embodiment, the low-level features of the to-be-processed image are extracted by using the patch embedding module, and the details of the feature extraction are supplemented by the task patch embedding module, so that the extracted features are more accurate, and there is no need to establish multiple models for different weather types as in the prior art. After determining the bottom information and the task information corresponding to the image features, the next step of information fusion can be carried out to facilitate the repair of the to-be-processed image.

[0071] Step S30, repairing the to-be-processed image according to the task information and the bottom information of the to-be-processed image.

[0072] In the embodiment, after the bottom information and the task information are determined, the respective fusion proportions are calculated, and then the model selectively fuses the bottom information and the task information. Alternatively, referring to Figure 4 , the step of repairing the to-be-processed image according to the task information and the bottom information of the to-be-processed image comprises:

[0073] Step S31, calculating the respective fusion proportions of the task information and the bottom information of the to-be-processed image.

[0074] Step S32, fusing the task information and the bottom information according to the respective fusion proportions of the task information and the bottom information.

[0075] In the embodiment, the fusion proportions of the bottom information and the task information are calculated by means of adaptive upsampling, referring to the detailed division of adaptive upsampling on the right uppermost side of Figure 6 , the multilayer perception machine can solve the binary classification problem, and the fusion proportions of the bottom information and the task information can be calculated by the multilayer perception machine, and then the two are fused by deep convolution. Through upsampling, new elements are inserted between the pixel points on the basis of the original to-be-processed image pixels by using a suitable interpolation algorithm. Alternatively, the upsampling algorithm is as follows:

[0076] f ↑i+1 =Mix(f ↓m-i ,f ↑i )=σ(θ i )*f ↑i +(1-σ(θ i )*f ↓m-i ),

[0077] wherein f ↑i and f ↓i represent the up-sampling and down-sampling feature maps of the i(i∈1,2,...m) stage, wherein f ↑0 is a task feature map, and f ↓0The feature map after the super-Laplacian preprocessing, σ(θ i ) represents a learnable factor of the i-th stage, used to fuse the low-level features from the decoder and the task features, the value of which is determined by the θ operator on the parameter θ i

[0078] It can be understood that when information fusion is performed, the down-sampling of the first stage and the up-sampling of the last stage are fused, and the other stages are fused in turn. Information fusion is multiple, in order to ensure that the image restoration effect is better, we will calculate the image restoration loss after each fusion by means of the loss function, and adjust the fusion proportion of the next time by means of the loss parameter, which is equivalent to a correction process, which can make the image restoration more perfect. Optionally, the loss function is as follows:

[0079] The loss parameter includes smooth L1 loss and perceptual loss. The smooth L1 loss is defined as follows:

[0080]

[0081] Wherein is the image recovered by the network, and G is the real clear image. In addition, we also use the 3rd, 8th and 15th layers of VGG16 to extract features and calculate perceptual loss. The perceptual loss formula is as follows:

[0082]

[0083] The overall loss can be defined as follows:

[0084]

[0085] Where λ is a parameter used to balance the weight of the two losses, and the specific value can be adjusted according to the experimental results.

[0086] Optionally, the loss function can be adjusted according to the actual needs of image restoration.

[0087] In this embodiment, the image is repaired by adaptive up-sampling, and the fusion proportion is corrected by means of the loss parameter, so that each stage of fusion can be more perfect.

[0088] Optionally, referring to Figure 5 , the second embodiment of the image processing method of the present application further comprises the following steps before step S20:

[0089] Step S40, determining the image patch corresponding to the to-be-processed image based on the patch embedding module.

[0090] ​Step S50, based on the conversion module, the image patch corresponding to the image to be processed is down-sampled.

[0091] Step S60, according to the data obtained by the down-sampling processing, the image features corresponding to the image to be processed are determined.

[0092] In this embodiment, the extraction of image features is mainly involved, referring to Figure 6 , the extraction of image features is mainly performed by the encoder, which contains a plurality of patch embedding modules and conversion modules, and the task patch embedding module is composed of a patch embedding module and a task conversion module, and the task embedding patch module acts on each stage of the encoder. The patch embedding module will block the image to be processed, and these small images after blocking are called patches, and further, the patches are further blocked into sub-patches with half the height and width of the original patch, which is more conducive to the extraction of image detail features and also helps to remove small degradation when operating on smaller patches. The sub-patch created from the original patch is input to the task embedding module, and through the task conversion module in the task embedding module, the task information corresponding to the image features of the current image to be processed can be extracted, and the task information is supplemented to the patch embedding module for the next feature extraction, which can guide the next feature extraction to supplement the detail information of different weather types. In this way, the next feature extraction can extract the detail information of the weather type in the image more accurately with the help of the guidance of the task information, and also can extract multiple weather types. We add task feature information in the patch embedding module to guide different weather types to supplement different types of detail information, and each time the next feature extraction is performed, the feature extraction becomes more accurate. Here we use the conversion module to complete the down-sampling of the image to be processed, which is opposite to the up-sampling method. The up-sampling method inserts new elements between the pixel points by using the interpolation algorithm, while the down-sampling method extracts, and the down-sampling, also known as the down-sampling, can be simply understood as reducing the image and reducing the number of sampling points of the matrix.

[0093] In this embodiment, by performing multiple down-sampling, the extraction of image features in the current image to be processed can be more accurate.

[0094] Referring to Figure 9 , Figure 9 is the image processing device structure schematic diagram of the hardware running environment involved in the embodiment of the present application.

[0095] As Figure 9As shown in the figure, the image processing device can include a processor 1001, such as a central processing unit (CPU), a communication bus 1002, a user interface 1003, a network interface 1004, and a memory 1005. The communication bus 1002 is used to realize the connection and communication between the components. The user interface 1003 can include a display, an input unit such as a keyboard, and can also include a standard wired interface, a wireless interface. The network interface 1004 can optionally include a standard wired interface, a wireless interface (such as a wireless fidelity (WIreless-FIdelity, WI-FI) interface). The memory 1005 can be a high-speed random access memory (RAM) memory, or a stable non-volatile memory (NVM) such as a disk memory. The memory 1005 can also be a storage device independent of the aforementioned processor 1001.

[0096] Those skilled in the art can understand that Figure 9 The structure shown in the figure does not constitute a limitation on the image processing device, and can include more or fewer components than the figure, or combine certain components, or different component arrangements.

[0097] As Figure 9 As shown, the memory 1005 as a storage medium can include an operating system, a data storage module, a network communication module, a user interface module, and an image processing program.

[0098] In Figure 9 In the image processing device shown in the figure, the network interface 1004 is mainly used for data communication with other devices; the user interface 1003 is mainly used for data interaction with the user; the processor 1001 and the memory 1005 in the image processing device of the present application can be arranged in the image processing device, and the image processing device calls the image processing program stored in the memory 1005 through the processor 1001, and performs the following steps:

[0099] Determine the probability distribution corresponding to the degradation information in the to-be-processed image, and eliminate the degradation information based on the probability distribution;

[0100] After eliminating the degradation information in the to-be-processed image, determine the task information and the underlying information contained in the to-be-processed image according to the image features in the to-be-processed image;

[0101] According to the task information and the underlying information of the to-be-processed image, repair the to-be-processed image.

[0102] Further, the image processing device further performs the following steps by invoking the image processing program stored in the memory 1005 through the processor 1001:

[0103] Obtaining the degradation information in the to-be-processed image, and simulating the corresponding probability distribution based on the degradation information in the to-be-processed image;

[0104] Based on the simulated probability distribution, the to-be-processed image is deconvolved to eliminate the degradation information.

[0105] Further, the image processing device further performs the following steps by invoking the image processing program stored in the memory 1005 through the processor 1001:

[0106] Based on the patch embedding module, determine the image patch corresponding to the to-be-processed image;

[0107] Based on the conversion module, the image patch corresponding to the to-be-processed image is down-sampled;

[0108] According to the data obtained by the down-sampling processing, determine the image feature corresponding to the to-be-processed image.

[0109] Further, the image processing device further performs the following steps by invoking the image processing program stored in the memory 1005 through the processor 1001:

[0110] The feature sequence corresponding to the image feature is normalized;

[0111] According to the feature sequence after the normalization processing, determine the bottom layer information based on the preset neural network.

[0112] Further, the image processing device further performs the following steps by invoking the image processing program stored in the memory 1005 through the processor 1001:

[0113] The feature sequence corresponding to the image feature is taken as the input parameter of the task conversion module;

[0114] Based on the task conversion module, the feature sequence is decoded to obtain the task feature corresponding to the image feature;

[0115] According to the task feature, determine the task information.

[0116] Further, the image processing device further performs the following steps by invoking the image processing program stored in the memory 1005 through the processor 1001:

[0117] The task information is taken as the guide information of the next down-sampling processing;

[0118] Based on the guidance information, the conversion module supplements the detail information of the image features when performing the down-sampling processing on the image to be processed.

[0119] Further, the image processing device further performs the following steps by invoking the image processing program stored in the memory 1005 through the processor 1001:

[0120] Calculate the fusion proportion corresponding to the task information and the bottom information of the image to be processed, respectively.

[0121] According to the fusion proportion corresponding to the task information and the bottom information, respectively, the task information and the bottom information are fused.

[0122] Further, the image processing device further performs the following steps by invoking the image processing program stored in the memory 1005 through the processor 1001:

[0123] Based on the fused image, calculate the loss parameter corresponding to the image;

[0124] According to the loss parameter, adjust the fusion proportion of the next fusion.

[0125] It should be noted that in this paper, the term "include", "contain" or any other variant thereof is intended to cover non-exclusive inclusion, so that the process, method, article or system including a series of elements not only includes those elements, but also includes other elements not explicitly listed, or includes elements inherent to such process, method, article or system. Without more limitations, the element defined by the statement "including a" does not exclude the presence of another identical element in the process, method, article or system including the element.

[0126] The above-mentioned embodiment number of the present application is only for description, not representing the pros and cons of the embodiments.

[0127] Through the description of the above embodiments, those skilled in the art can clearly understand that the above-mentioned embodiment method can be realized by software plus necessary general hardware platform, of course, it can also be realized by hardware, but in many cases, the former is a better embodiment. Based on such understanding, the technical solutions of the present application can be embodied in the form of software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) as described above, including a plurality of instructions to make a terminal device (which can be a mobile phone, computer, etc.) execute the method described in each embodiment of the present application.

[0128] The above merely describes the preferred embodiments of the present application, and is not intended to limit the patent scope of the present application, and any equivalent structure or equivalent process conversion, or direct or indirect application in other related technical fields, which are made by using the content of the present application specification and drawings, are also included in the patent protection scope of the present application.

Claims

1. An image processing method, characterized by, The image processing method comprises the following steps: determine the probability distribution corresponding to the degradation information in the image to be processed, and eliminate the degradation information based on the probability distribution; normalizing the feature sequence corresponding to the image features in the image to be processed; According to the normalized feature sequence, based on the preset neural network, determine the bottom layer information, the bottom layer information includes the contour, edge, color, texture and shape features of the image to be processed; The feature sequence corresponding to the image feature is used as the input parameter of the task conversion module; Based on the task conversion module, the feature sequence is decoded to obtain the task feature corresponding to the image feature; Wherein, the task conversion module introduces the learnable task type sequence from the outside world as the query vector, and takes the output of the encoder as the search vector and the content vector, and calculates the attention weight of the feature sequence through the multi-head self-attention mechanism, and inputs the normalized processing into the feedforward neural network for mapping, and outputs the task feature; According to the task feature, determine the task information, the task feature represents the task type that needs to be repaired in the image to be processed, and the task information is used to supplement the detail information of the task type; Through adaptive up-sampling, calculate the fusion proportion corresponding to the task information and the bottom layer information of the image to be processed respectively; According to the fusion proportion corresponding to the task information and the bottom layer information respectively, the task information and the bottom layer information are fused through deep convolution; Wherein, in the fusion process, interpolation algorithm is used to insert new elements between the pixel points of the image to be processed, and multi-stage fusion is carried out according to the corresponding fusion of the previous stage down-sampling and the next stage up-sampling; Based on the fused image, calculate the loss parameter corresponding to the image; According to the loss parameter, adjust the fusion proportion of the next fusion, and finally repair the image to be processed.

2. The image processing method of claim 1, wherein, The step of determining the probability distribution corresponding to the degradation information in the image to be processed, and eliminating the degradation information based on the probability distribution comprises: Obtain the degradation information in the image to be processed, and simulate the corresponding probability distribution based on the degradation information in the image to be processed; Based on the simulated probability distribution, the image to be processed is deconvolved to eliminate the degradation information.

3. The image processing method of claim 1, wherein, Before the step of normalizing the feature sequence corresponding to the image features in the image to be processed, it further comprises: Based on the patch embedding module, determine the image patch corresponding to the image to be processed; Based on the conversion module, the image patch corresponding to the image to be processed is down-sampled; According to the data obtained by the down-sampling processing, determine the image features corresponding to the image to be processed.

4. The image processing method of claim 1, wherein, After the step of determining the task information according to the task feature, it further comprises: The task information is used as the guide information for the next down-sampling processing; Based on the guide information, guide the conversion module to supplement the detail information of the image features when performing the down-sampling processing on the image to be processed.

5. An image processing apparatus characterized by comprising: The image processing device comprises a memory, a processor, and an image processing program stored in the memory and executable on the processor, and the image processing program is configured to implement the steps of the image processing method according to any one of claims 1 to 4.

6. A computer-readable storage medium, characterized in that, The storage medium stores an image processing program, and the image processing program, when executed by a processor, implements the steps of the image processing method according to any one of claims 1 to 4.

Images