Image restoration method and related equipment

Abstract

The invention provides an image restoration method and related equipment, and the method comprises the steps: collecting a real industrial severe environment image and a clear environment image, obtaining a synthesized severe environment image through the clear environment image, and inputting the synthesized severe environment image and the real industrial severe environment image into a constructed semi-supervised module for training data screening; a new data set is obtained and input into the constructed teacher-student model to be trained in three stages, and an industrial severe environment restoration model is obtained; inputting the obtained target industrial environment image into an industrial severe environment restoration model for restoration to obtain a clear image corresponding to the target industrial environment image; compared with the prior art, the method has the advantages that the training data is screened through the semi-supervised module, so that the performance and generalization of the model in different environments are improved; a teacher-student model is trained from three different stages, so that the model fully learns unique features and public features among various severe environments, and insufficient feature learning is prevented, thereby improving the robustness of industrial severe environment image restoration.

Description

Technical Field

[0001] The present invention relates to the technical field of computer vision and image processing, and in particular to an image restoration method and related equipment. Background Art

[0003] With the development of artificial intelligence, deep learning technology has gradually become the mainstream method for image restoration in industrial harsh environments. However, today's image restoration is only applicable to a certain type of typical industrial harsh environment. However, in the application of industrial vision systems, it is inevitable to face various types of harsh environments. Therefore, multiple sets of model parameters specific to harsh environments are required to handle various harsh environments, which brings additional computational and storage burdens. Therefore, it is crucial to develop a unified model that can solve various types of harsh environments with a set of network parameters.

[0004] Recently, some methods have been proposed to inpaint industrial harsh environment images using a set of parameters and to construct a unified model for image inpainting in industrial harsh environments. However, these methods also have their limitations. First, these methods rely on large-scale synthetic data for training, which affects their performance and generalization in real environments. Second, due to the domain differences of features between different harsh environments, these methods mainly focus on the unique features of different types of harsh environments, while ignoring all the common features between them. Finally, most of these methods are trained directly by original images and real labels, but the complexity of the task makes the model feature learning insufficient in the early stage of training, and learns suboptimal features, because it may focus on details that do not contribute to task generalization, which will hinder the model from learning more relevant high-level features.

[0005] Therefore, how to develop a unified image restoration model for harsh industrial environments so that it can fully explore the unique and common features of various types of harsh industrial environments and have good robustness and performance in real environments is a problem that needs to be solved urgently. Summary of the invention

[0006] The present invention provides an image restoration method and related equipment, the purpose of which is to improve the robustness of image restoration in industrial harsh environments.

[0007] In order to achieve the above object, the present invention provides an image restoration method, comprising:

[0008] Step 1, collecting real industrial harsh environment images and clear environment images, and using the clear environment images to obtain synthetic harsh environment images;

[0009] Step 2: Input the real industrial harsh environment images and the synthetic harsh environment images into the constructed semi-supervised module for training data screening to obtain a new data set;

[0010] Step 3, input the new data set into the constructed teacher-student model, and train the teacher-student model in three stages to obtain the industrial harsh environment restoration model;

[0011] Step 4, inputting the acquired target industrial environment image into the industrial harsh environment restoration model for restoration, and obtaining a clear image corresponding to the target industrial environment image;

[0012] The teacher-student model includes a first teacher network for repairing images, a second teacher network, a third teacher network, a student network for extracting image features, a knowledge interactor for feature interaction, a first common feature projector for projecting features into a common feature space, a second common feature projector, a third common feature projector, a fourth common feature projector, and a knowledge integrator for integrating features.

[0013] Furthermore, the clear environment image is used to obtain a synthetic harsh environment image, including:

[0014] Noise is added to the clear environment image to obtain a synthetic harsh environment image.

[0015] Specifically, the semi-supervised module is used to:

[0016] calculating a plurality of screening indicators for screening images;

[0017] Performing maximum-minimum normalization processing on each screening index respectively to obtain a normalized screening index;

[0018] After negating some of the normalized screening indicators, subtract the value before negation from 1 to obtain a new screening indicator;

[0019] The two-norm values ​​of all the normalized screening indicators that are not negated and all the new screening indicators are taken to obtain the final screening indicators;

[0020] The final screening indicators are sorted from large to small, and the real industrial harsh environment images ranked at the top are combined with the synthetic harsh environment images to form a new data set.

[0021] Specifically, the screening criteria include:

[0022] Image quality indicators used to evaluate the difficulty of image restoration in harsh industrial environments;

[0023] Information content indicators used to evaluate the amount of information contained in an image, including information entropy and total variation;

[0024] The restoration effect indicators used to evaluate the quality of image restoration include structural similarity and peak signal-to-noise ratio.

[0025] To further elaborate, the three stages include: knowledge accumulation stage, knowledge integration stage, and knowledge testing stage.

[0026] Further, the knowledge accumulation stage completes the unique feature accumulation of each harsh environment through the first teacher network, the second teacher network, the third teacher network, the student network, the knowledge interactor, the first common feature projector, the second common feature projector, the third common feature projector, and the fourth common feature projector;

[0027] The input end of the first teacher network, the input end of the second teacher network, the input end of the third teacher network, and the input end of the student network are all connected to the output end of the data set combination unit;

[0028] The first output end of the first teacher network is connected to the first input end of the first common feature projector, the first input end of the second common feature projector, the first input end of the third common feature projector, and the first input end of the fourth common feature projector, respectively, and the second output end of the first teacher network is connected to the first input end of the knowledge integration unit;

[0029] The first output end of the second teacher network is connected to the second input end of the first common feature projector, the second input end of the second common feature projector, the second input end of the third common feature projector, and the second input end of the fourth common feature projector, respectively, and the second output end of the second teacher network is connected to the second input end of the knowledge integration unit;

[0030] The first output end of the third teacher network is connected to the third input end of the first common feature projector, the third input end of the second common feature projector, the third input end of the third common feature projector, and the third input end of the fourth common feature projector, respectively, and the second output end of the third teacher network is connected to the third input end of the knowledge integration unit;

[0031] The first output terminal of the student network is connected to the input terminal of the knowledge interactor, and the second output terminal of the student network is connected to the fourth input terminal of the knowledge integration unit;

[0032] The output end of the knowledge interactor is connected to the fourth input end of the first common feature projector, the fourth input end of the second common feature projector, the fourth input end of the third common feature projector, and the fourth input end of the fourth common feature projector respectively.

[0033] Furthermore, the knowledge integration phase is completed using the knowledge integrator based on the knowledge accumulation phase;

[0034] The first input terminal of the knowledge integrator is connected to the second output terminal of the first teacher network;

[0035] The second input terminal of the knowledge integrator is connected to the second output terminal of the second teacher network;

[0036] The third input terminal of the knowledge integrator is connected to the second output terminal of the third teacher network;

[0037] The fourth input terminal of the knowledge integrator is connected to the second output terminal of the student network;

[0038] The output end of the knowledge integrator is the output end of the teacher-student model, which is used to output the knowledge integration results.

[0039] Furthermore, in the knowledge testing phase, feature extraction and testing are completed through the student network;

[0040] The output end of the student network is the output end of the teacher-student model, which is used to output the knowledge test results.

[0041] The present invention also provides a terminal device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the image repair method is implemented when the processor executes the computer program.

[0042] The present invention also provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the image restoration method is implemented.

[0043] The above scheme of the present invention has the following beneficial effects:

[0044] The present invention collects real industrial harsh environment images and clear environment images, and obtains synthetic harsh environment images using the clear environment images; inputs the real industrial harsh environment images and the synthetic harsh environment images into a constructed semi-supervised module for training data screening, obtains a new data set and inputs it into a constructed teacher-student model, trains the teacher-student model in three stages, and obtains an industrial harsh environment restoration model; inputs the acquired target industrial environment image into the industrial harsh environment restoration model for restoration, and obtains a clear image corresponding to the target industrial environment image; the teacher-student model includes a first teacher network for repairing the image, a second teacher network, a third teacher network, a student network for extracting image features, a knowledge interactor for feature interaction, a first common feature projector for projecting features into a common feature space, a second common feature projector, a third common feature projector, a fourth common feature projector, and a knowledge integrator for integrating features; compared with the prior art, the present invention screens training data through a semi-supervised module to improve the performance and generalization of the model in different environments; trains the teacher-student model from three different stages to allow the model to fully learn the unique features and common features between each harsh environment, prevents insufficient feature learning, and thus improves the robustness of industrial harsh environment image restoration.

[0045] Other beneficial effects of the present invention will be described in detail in the subsequent specific implementation section. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 A schematic diagram of a flow chart of an embodiment of the present invention;

[0047] Figure 2 This is a schematic diagram of the structure of the first teacher network in an embodiment of the present invention;

[0048] Figure 3 A schematic diagram of the connection between the teacher and student models in the knowledge accumulation stage of an embodiment of the present invention;

[0049] Figure 4 A schematic diagram of the connection between the teacher and student models in the knowledge integration stage of an embodiment of the present invention;

[0050] Figure 5 A schematic diagram of the connection between the teacher and student models in the knowledge testing phase of an embodiment of the present invention;

[0051] Figure 6 is a schematic structural diagram of a first common feature projector in an embodiment of the present invention;

[0052] Figure 7 A schematic diagram of the structure of a knowledge interactor in an embodiment of the present invention;

[0053] Figure 8 Schematic diagram of the structure of a terminal device in an embodiment of the present invention. DETAILED DESCRIPTION

[0054] In order to make the technical problems, technical solutions and advantages to be solved by the present invention clearer, the following will be described in detail with reference to the accompanying drawings and specific embodiments. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0055] In the description of the present invention, it should be noted that the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inner", "outer", etc., indicating the orientation or positional relationship, are based on the orientation or positional relationship shown in the drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as limiting the present invention. In addition, the terms "first", "second", and "third" are used for descriptive purposes only, and cannot be understood as indicating or implying relative importance.

[0056] In the description of the present invention, it should be noted that, unless otherwise clearly specified and limited, the terms "installation", "connection" and "connection" should be understood in a broad sense, for example, it can be a locking connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0057] In addition, the technical features involved in the different embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0058] In view of the existing problems, the present invention provides an image restoration method and related equipment.

[0059] like Figure 1 As shown, an embodiment of the present invention provides an image restoration method, comprising:

[0060] Step 1, collecting real industrial harsh environment images and clear environment images, and using the clear environment images to obtain synthetic harsh environment images;

[0061] Step 2: Input the real industrial harsh environment images and the synthetic harsh environment images into the constructed semi-supervised module for training data screening to obtain a new data set;

[0062] Step 3, input the new data set into the constructed teacher-student model, and train the teacher-student model in three stages to obtain the industrial harsh environment restoration model;

[0063] Step 4, inputting the acquired target industrial environment image into the industrial harsh environment restoration model for restoration, and obtaining a clear image corresponding to the target industrial environment image;

[0064] The teacher-student model includes a first teacher network for repairing images, a second teacher network, a third teacher network, a student network for extracting image features, a knowledge interactor for feature interaction, a first common feature projector for projecting features into a common feature space, a second common feature projector, a third common feature projector, a fourth common feature projector, and a knowledge integrator for integrating features.

[0065] Specifically, if Figure 2 As shown, the first teacher network has the same structure as the second teacher network, the third teacher network and the student network, and all consist of an encoder and a decoder.

[0066] The input end of the encoder is the input end of the network, and the first output end, the second output end, the third output end and the fourth output end of the encoder are respectively connected to the first to fourth input ends of the decoder.

[0067] The first input terminal, the second input terminal, the third input terminal and the fourth input terminal of the decoder are respectively connected to the first to fourth output terminals of the encoder, and the output terminal of the encoder is the output terminal of the network.

[0068] The encoder includes a first convolution module, a first down-sampling module, a second convolution module, a second down-sampling module, a third convolution module, a third down-sampling module, and a fourth convolution module.

[0069] The decoder is composed of a fifth convolution module, a first upsampling module, a sixth convolution module, a second upsampling module, a seventh convolution module, a third upsampling module, and an eighth convolution module.

[0070] The input end of the first convolution module is the input end of the decoder, the first output end of the first convolution module is connected to the input end of the first down-sampling module, and the second output end of the first convolution module is connected to the first input end of the eighth convolution module; the input end of the first down-sampling module is connected to the first output end of the first convolution module, and the output end of the first down-sampling module is connected to the input end of the second convolution module; the input end of the second convolution module is connected to the output end of the first down-sampling module, the first output end of the second convolution module is connected to the second down-sampling module, and the second output end of the second convolution module is connected to the first input end of the seventh convolution module; the input end of the second down-sampling module is connected to the first output end of the first convolution module, and the output end of the first down-sampling module is connected to the input end of the second convolution module. The input end of the first convolution module is connected to the first output end of the second convolution module, the output end of the second down-sampling module is connected to the input end of the third convolution module; the input end of the third convolution module is connected to the output end of the second down-sampling module, the first output end of the third convolution module is connected to the third down-sampling module, and the second output end of the third convolution module is connected to the first input end of the seventh convolution module; the input end of the third down-sampling module is connected to the first output end of the third convolution module, and the second output end of the third down-sampling module is connected to the input end of the sixth convolution module; the input end of the fourth convolution module is connected to the output end of the third down-sampling module, and the output end of the fourth convolution module is the encoding The output end of the decoder is connected to the fifth convolution module; the input end of the fifth convolution module is the input end of the decoder connected to the output end of the fourth convolution module, and the output end of the fifth convolution module is connected to the first up-sampling module; the input end of the first up-sampling module is connected to the output end of the fifth convolution module, and the output end of the first up-sampling module is connected to the first input end of the sixth convolution module; the first input end of the sixth convolution module is connected to the output end of the first up-sampling module, the second input end of the sixth convolution module is connected to the second output end of the third convolution module, and the output end of the sixth convolution module is connected to the second up-sampling module; the input end of the second up-sampling module is connected to the output end of the sixth convolution module, and the output end of the sixth convolution module is connected to the second up-sampling module. The output end of the second upsampling module is connected to the first input end of the seventh convolution module; the first input end of the seventh convolution module is connected to the output end of the third upsampling module, the second input end of the seventh convolution module is connected to the second output end of the second convolution module, and the output end of the seventh convolution module is connected to the third upsampling module; the input end of the third upsampling module is connected to the output end of the seventh convolution module, and the output end of the third upsampling module is connected to the first input end of the eighth convolution module; the first input end of the eighth convolution module is connected to the output end of the third upsampling module, the second input end of the eighth convolution module is connected to the second output end of the first convolution module, and the output end of the eighth convolution module is the decoder output end.

[0071] In an embodiment of the present invention, harsh environments include fog and haze, rainy days, and snowy days. Real industrial harsh environment images are images whose clarity is less than a preset threshold value, such as factory environment images and mountain environment images, and clear environment images are images whose clarity is greater than a preset threshold value, such as factory environment images and mountain environment images.

[0072] Specifically, the clear environment image is used to obtain the synthetic harsh environment image, including:

[0073] Noise is added to the clear environment image to obtain a synthetic harsh environment image.

[0074] In the embodiment of the present invention, the noise may be additive noise such as Gaussian noise, Rayleigh noise, exponential noise, etc., which is used to reduce the clarity of the clear environment image.

[0075] In the embodiment of the present invention, the clear environment image is used as the real marked data, and the synthetic harsh environment image is used as the data to be processed.

[0076] Most preferably, the semi-supervised module is used to:

[0077] calculating a plurality of screening indicators for screening images;

[0078] Perform maximum-minimum normalization processing on each screening index respectively to obtain a normalized screening index to eliminate the differences between its quantitative units;

[0079] After negating some of the normalized screening indicators, subtract the value before negation from 1 to obtain a new screening indicator;

[0080] The two-norm values ​​of all the normalized screening indicators that are not negated and all the new screening indicators are taken to obtain the final screening indicators;

[0081] The final screening indicators are sorted from large to small, and the real industrial harsh environment images ranked at the top are combined with the synthetic harsh environment images to form a new data set.

[0082] In the embodiment of the present invention, the semi-supervised module can be composed of an image screening unit for determining the screening index and a data combination unit for combining the screened images. Both the image screening unit and the data combination unit are soft modules. For example, in the specific structure, both realize data processing through mathematical models, and the connection relationship is as follows:

[0083] The input of the image screening unit is the input of the semi-supervised module;

[0084] The output end of the image screening unit is connected to the input end of the data set combination unit;

[0085] The input of the dataset combination unit is connected to the input of the semi-supervised module;

[0086] The output of the dataset combination unit is connected to the input of the teacher-student model.

[0087] Specifically, the formula for maximum-minimum normalization is as follows:

[0088]

[0089] Among them, the physical meaning of each parameter is x i Represents the original index to be normalized, x max and x min Represent the maximum and minimum values ​​of the original index, x' i Represents the normalized index.

[0090] Most preferably, the screening indicators include:

[0091] Image quality indicators used to evaluate the difficulty of image restoration in harsh industrial environments;

[0092] Information content indicators used to evaluate the amount of information contained in an image, including information entropy and total variation;

[0093] The restoration effect indicators used to evaluate the quality of image restoration include structural similarity and peak signal-to-noise ratio.

[0094] In the embodiment of the present invention, the image quality evaluation score corresponding to each image is calculated as the image quality index, and the image quality evaluation score corresponding to each image is calculated as follows:

[0095] The new data set is input into the teacher-student model for repeated training, multiple image blocks are extracted and normalized, and the expression is:

[0096]

[0097] Where x represents the pixel value in the image block, μ x and σ x are the mean and standard deviation of the image block, ∈; is a small constant used to avoid division by zero.

[0098] Extract the statistical features of each image block to form a feature vector:

[0099] f(x')=[μ' x ,σ' x ,cov(x')]

[0100] The Mahalanobis distance between the feature vector of each image block and the trained Gaussian model is calculated by the following formula:

[0101]

[0102] The Mahalanobis distances of all image blocks are averaged to obtain the overall NIQE value of the image, which is used as the image quality indicator. The calculation formula for the overall NIQE value of the image is:

[0103]

[0104] Among them, D(x i ') represents the Mahalanobis distance between the feature vector of the i-th image block x' and the trained Gaussian model, and n represents the total number of image blocks.

[0105] In the embodiment of the present invention, the information entropy and total variation corresponding to each image are calculated as information content indicators. The information entropy is calculated as follows:

[0106] For a given grayscale image, we first need to calculate the frequency of occurrence of each grayscale level (usually an integer between 0 and 255), which can be done by calculating the grayscale histogram:

[0107]

[0108] Calculate the self-information of each gray level. The self-information is used to measure the amount of information brought by a certain gray level:

[0109] I(i)=-log2(p i )

[0110] Calculate the information entropy of the image: The information entropy of the image is the expected value of the self-information of all gray levels, that is, the weighted average:

[0111]

[0112] Among them, the physical meaning of each parameter is: L is the number of gray levels, p i is the frequency of gray level i;

[0113] The total variation is calculated as follows:

[0114] First, we need to calculate the gradient of the image, that is, the rate of change of the image in the horizontal and vertical directions:

[0115]

[0116] Calculate the total variation: The total variation of an image is the sum of the modulus values ​​of the image gradient. For image I, its total variation TV(I) can be expressed as:

[0117]

[0118] The physical meaning of each parameter in the above expression is that I(i,j) represents the image of the total variation to be calculated, It is the modulus of the image gradient at position (i, j), that is, the rate of change of the image at that position.

[0119] In the embodiment of the present invention, the structural similarity and peak signal-to-noise ratio of each image pair are calculated as the restoration effect index, and the calculation method of the structural similarity is as follows:

[0120] The first component of image structure similarity is brightness comparison, which measures the similarity of brightness by calculating the average brightness of the two images. Suppose the grayscale values ​​of the two images are x and y, respectively, and the corresponding average brightness is:

[0121]

[0122] Among them, μ x and μ y Represent the average brightness of images x and y respectively, and N is the number of pixels in the image;

[0123] The brightness comparison function is:

[0124]

[0125] The second component of image structural similarity is contrast comparison, which measures the similarity of contrast by calculating the standard deviation of the images. The standard deviations of images x and y are:

[0126]

[0127] The contrast comparison function is:

[0128]

[0129] The third component of image structural similarity is structural comparison, which measures the structural similarity by calculating the covariance of images:

[0130]

[0131] Finally, the overall similarity of two images is measured by the product of the three components of brightness, contrast, and structure:

[0132] SSIM(x,y)=[l(x,y)] α [c(x,y)] β ·[s(x,y)} γ

[0133] Among them, α, β and γ are weight parameters used to adjust each component;

[0134] The peak signal-to-noise ratio is calculated as follows:

[0135] First, we need to calculate the mean square error (MSE) between the original image and the restored image:

[0136]

[0137] Where I(i,j) and K(i,j) represent the pixel values ​​of the original image and the reconstructed image at position (i,j), respectively;

[0138] After calculating the MSE, the peak signal-to-noise ratio PSNR can be further calculated:

[0139]

[0140] Among them, MAx represents the maximum pixel value of the image, and MSE represents the mean square error between the original image and the restored image.

[0141] Specifically, the three stages include: knowledge accumulation stage, knowledge integration stage, and knowledge testing stage.

[0142] It should be noted that the knowledge accumulation stage is similar to the initial stage of knowledge accumulation in human learning in real life. At this time, students do not have enough knowledge and need to absorb the knowledge of each teacher individually. In this stage, there are multiple well-trained teacher networks responsible for image restoration in different types of harsh environments and a student network. The student network is trained with all different teacher networks at the same time. Through learning, the knowledge of the relevant harsh environment type of each teacher network is transferred to itself, and the unique characteristics of each harsh environment are initially learned, so as to realize comprehensive image restoration in industrial harsh environments.

[0143] In the knowledge integration stage, the student network has acquired certain knowledge through the knowledge accumulation stage and has certain knowledge integration capabilities. It can extract the common features of different types of harsh environments, similar to the process of human learning. In this stage, when the student network is trained with different teacher networks and learns knowledge, it not only learns the knowledge of the corresponding teacher network, but also learns the knowledge from other teacher networks, integrates all the teacher network knowledge together to learn together, and jointly discovers the similar features of all harsh environments.

[0144] The knowledge testing phase mainly involves self-study and self-testing of the student network, and training without the guidance of the teacher network. By setting higher requirements, the robustness and discrimination ability of the network are improved, and satisfactory results are achieved in different types of harsh environments.

[0145] Most preferably, Figure 3 As shown, the knowledge accumulation stage completes the unique feature accumulation of each harsh environment through the first teacher network, the second teacher network, the third teacher network, the student network, the knowledge interactor, the first public feature projector, the second public feature projector, the third public feature projector, and the fourth public feature projector;

[0146] The input end of the first teacher network, the input end of the second teacher network, the input end of the third teacher network, and the input end of the student network are all connected to the output end of the data set combination unit;

[0147] The first output end of the first teacher network is connected to the first input end of the first common feature projector, the first input end of the second common feature projector, the first input end of the third common feature projector, and the first input end of the fourth common feature projector, respectively, and the second output end of the first teacher network is connected to the first input end of the knowledge integration unit;

[0148] The first output end of the second teacher network is connected to the second input end of the first common feature projector, the second input end of the second common feature projector, the second input end of the third common feature projector, and the second input end of the fourth common feature projector, respectively, and the second output end of the second teacher network is connected to the second input end of the knowledge integration unit;

[0149] The first output end of the third teacher network is connected to the third input end of the first common feature projector, the third input end of the second common feature projector, the third input end of the third common feature projector, and the third input end of the fourth common feature projector, respectively, and the second output end of the third teacher network is connected to the third input end of the knowledge integration unit;

[0150] The first output terminal of the student network is connected to the input terminal of the knowledge interactor, and the second output terminal of the student network is connected to the fourth input terminal of the knowledge integration unit;

[0151] The output end of the knowledge interactor is connected to the fourth input end of the first common feature projector, the fourth input end of the second common feature projector, the fourth input end of the third common feature projector, and the fourth input end of the fourth common feature projector respectively.

[0152] It should be noted that since the student network is only in the early stages of training at this stage and does not have sufficient knowledge, the loss of the student model at this stage is mainly based on the prediction results of the teacher model.

[0153] Most preferably, Figure 4 As shown, the knowledge integration stage uses the knowledge integrator to integrate similar features of all harsh environments based on the knowledge accumulation stage, where the knowledge integrator corresponds to the teacher knowledge integrator in the figure;

[0154] The first input terminal of the knowledge integrator is connected to the second output terminal of the first teacher network;

[0155] The second input terminal of the knowledge integrator is connected to the second output terminal of the second teacher network;

[0156] The third input terminal of the knowledge integrator is connected to the second output terminal of the third teacher network;

[0157] The fourth input terminal of the knowledge integrator is connected to the second output terminal of the student network;

[0158] The output end of the knowledge integrator is the output end of the teacher-student model, which is used to output the knowledge integration results.

[0159] It should be noted that the loss in this stage consists of three parts: the loss between the feature projections of the teacher network and the student network, the loss between the final restoration results of the teacher network and the student network, and the loss between the restoration results of the student network and the true restoration results. The difference is that the reply result of the teacher network is the result after integration.

[0160] Most preferably, Figure 5 As shown, in the knowledge testing phase, feature extraction and testing are completed through the student network;

[0161] The output end of the student network is the output end of the teacher-student model, which is used to output the knowledge test results.

[0162] It should be noted that the loss at this stage is composed solely of the loss between the restoration result of the student network and the true restoration result.

[0163] In an embodiment of the present invention, the common feature projector is used to project teacher features and student features from teacher and student networks into a common feature space, and is a learnable module that enables the network to determine the feature space most suitable for common feature learning.

[0164] like Figure 6 As shown in the figure, the first common feature projector, the second common feature projector, the third common feature projector, and the fourth common feature projector are all composed of several convolution blocks with a stride of 1 and a ReLU activation function; in the common feature space, the projection feature error is used to constrain the feature learning process to make the projected features closer, and the L1 loss is used to calculate the distance between two projected features; in addition, pyramid pooling is used for the projected features to expand the context information at different levels.

[0165] In addition, in order to robustly learn knowledge from multiple teacher networks, bidirectional feature matching is used to constrain the learned features; the projected features of the teacher network are projected back to the original input space through the inverse progressive feature projector; then, the difference between them and the original features is calculated through the loss of projected feature verification. Specifically, Figure 7 As shown, the knowledge interactor is used to first transform the input encoder feature F en and decoder features F de By combining convolution blocks, α and β are transformed into corresponding hidden states and combined with F en and F de Interact and obtain interactive features and After that, the interaction features are passed through two other combined convolution blocks γ and θ respectively, and then added or subtracted to the interaction features to obtain the updated features F' en and F' de :

[0166]

[0167] In the embodiment of the present invention, the final segmentation result image and the binary label of the corresponding image are input into the loss function to calculate the loss of the response. The loss calculation formula is as follows:

[0168] L=βL MAE +(1-β)L CP ,β∈[0,1]

[0169] Among them, L MAE Represents the absolute error loss, L CP represents the projection loss, β represents the weight;

[0170] The calculation formula for cross entropy loss is:

[0171]

[0172] Where N represents the number of samples, Represents the actual image of the sample, y i Represents the model prediction sample;

[0173] The calculation formula of projection loss is as follows:

[0174]

[0175] Where L represents the total number of encoders. Denotes the encoder L in the corresponding teacher network T i The feature map in represents the feature map of encoder L in the student network, and f(·) represents the projection operation.

[0176] In the embodiment of the present invention, after calculating the loss, back propagation is required to further optimize the parameters. The gradient of the loss function to the parameters in the network is calculated by the chain rule. The Nadam optimizer is used to update each parameter using the gradient in the direction of loss reduction. The Nadam parameter update formula is:

[0177]

[0178] m t =β1m t-1 +(1-β1)g t

[0179]

[0180] Where t represents the time step, represents the gradient at time t, θ t-1 represents the parameter value at time t-1, α represents the learning rate, β1, β2, ε, w, ∈ are constants, respectively It is the t-th power of β1 and β2, and the model with the best performance on the validation set is finally saved.

[0181] Specifically, step 4 includes:

[0182] The acquired target industrial environment image is input into the industrial harsh environment restoration model, and the unique features of the image are extracted through the student network to obtain the student restoration results. After the extracted unique features are interacted through the knowledge interactor, the interacted features are projected into the common feature space through the first common feature projector, the second common feature projector, the third common feature projector, and the fourth common feature projector to obtain the similar features of the image. The similar features are input into the first teacher model, the second teacher model, and the third teacher model to repair the image to obtain the teacher restoration results. The student restoration results and the teacher restoration results are input into the knowledge integrator for integration to obtain a clear image corresponding to the target industrial environment image.

[0183] In the embodiment of the present invention, the target industrial environment image is an industrial environment image collected in any environment.

[0184] The embodiment of the present invention compares the effects of the image restoration method provided by the present application with other commonly used methods on the collected dehazing, deraining and desnowing data sets. The methods used for comparison are divided into two categories. The first category is a single method specifically for a certain specific harsh environment, and the second category is a unified model.

[0185] For a single method, in addition to comparing its recovery effect on the corresponding data set, the embodiment of the present invention also uses a mixed data set to train and verify its effect; for a unified model, the mixed data set is directly used for training and performance verification. The results are shown in the following table:

[0186] Table 1

[0187]

[0188] It should be noted that the numbers in Table 1 above are in the form of X / Y, where X represents the structural similarity index of the corresponding method, and Y represents the maximum signal-to-noise ratio index; the embodiment of the present invention uses these two indicators to judge the removal effect.

[0189] It can be seen from Table 1 that the image restoration method under harsh industrial environments provided by the embodiment of the present invention exceeds other comparison methods in various indicators and has better performance than other methods. The experimental results prove that the embodiment of the present invention has good robustness in image restoration tasks under harsh industrial environments.

[0190] The embodiment of the present invention collects real industrial harsh environment images and clear environment images, and obtains synthetic harsh environment images using the clear environment images; inputs the real industrial harsh environment images and the synthetic harsh environment images into the constructed semi-supervised module for training data screening, obtains a new data set and inputs it into the constructed teacher-student model, trains the teacher-student model in three stages, and obtains an industrial harsh environment restoration model; inputs the acquired target industrial environment image into the industrial harsh environment restoration model for restoration, and obtains a clear image corresponding to the target industrial environment image; the teacher-student model includes a first teacher network for repairing the image, a second teacher network, a third teacher network, a student network for extracting image features, a knowledge interactor for feature interaction, a first common feature projector for projecting features into a common feature space, a second common feature projector, a third common feature projector, a fourth common feature projector, and a knowledge integrator for integrating features; compared with the prior art, the embodiment of the present invention screens training data through a semi-supervised module to improve the performance and generalization of the model in different environments; trains the teacher-student model from three different stages to allow the model to fully learn the unique features and common features between each harsh environment, prevents insufficient feature learning, and thus improves the robustness of industrial harsh environment image restoration.

[0191] An embodiment of the present invention further provides a terminal device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the image restoration method is implemented when the processor executes the computer program.

[0192] like Figure 8 As shown, the terminal device D10 of this embodiment includes: at least one processor D100 ( Figure 8 Only one processor is shown in the figure), a memory D101, and a computer program D102 stored in the memory D101 and executable on the at least one processor D100, the processor D100 implements the steps of the above-mentioned method for constructing the uncoupled blasting load model when executing the computer program D102. Alternatively, the processor D100 implements the functions of each module / unit in the above-mentioned device embodiments when executing the computer program D102.

[0193] The terminal device D10 may be a computing device such as a desktop computer, a notebook, a PDA, a server, a server cluster, a cloud server, etc. The terminal device may include, but is not limited to, a processor D100 and a memory D101. Those skilled in the art will appreciate that Figure 8 This is only an example of the terminal device D10 and does not constitute a limitation on the terminal device D10. The terminal device D10 may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, it may also include input and output devices, network access devices, etc.

[0194] The processor D100 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0195] In some embodiments, the memory D101 may be an internal storage unit of the terminal device D10, such as a hard disk or memory of the terminal device D10. In other embodiments, the memory D101 may also be an external storage device of the terminal device D10, such as a plug-in hard disk, a smart memory card (SMC, SmartMedia Card), a secure digital (SD, Secure Digital) card, a flash card (Flash Card), etc. equipped on the terminal device D10. Further, the memory D101 may also include both an internal storage unit of the terminal device D10 and an external storage device. The memory D101 is used to store an operating system, an application program, a boot loader (BootLoader), data and other programs, such as the program code of the computer program, etc. The memory D101 may also be used to temporarily store data that has been output or is to be output.

[0196] It should be noted that the information interaction, execution process, etc. between the above-mentioned devices / units are based on the same concept as the method embodiment of the present application. Their specific functions and technical effects can be found in the method embodiment part and will not be repeated here.

[0197] The technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.

[0198] An embodiment of the present invention further provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, an image restoration method is implemented.

[0199] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the process in the above-mentioned embodiment method, which can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may at least include: any entity or device that can carry the computer program code to the construction device / terminal device, recording medium, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electric carrier signal, telecommunication signal and software distribution medium. For example, a USB flash drive, a mobile hard disk, a disk or an optical disk.

[0200] The above is a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. An image restoration method, characterized in that: include: Step 1, collecting a real industrial harsh environment image and a clear environment image, and using the clear environment image to obtain a synthetic harsh environment image; Step 2, inputting the real industrial harsh environment images and the synthetic harsh environment images into the constructed semi-supervised module to screen the training data to obtain a new data set; Step 3, inputting the new data set into the constructed teacher-student model, and training the teacher-student model in three stages to obtain an industrial harsh environment restoration model; Step 4, inputting the acquired target industrial environment image into the industrial harsh environment restoration model for restoration, to obtain a clear image corresponding to the target industrial environment image; The teacher-student model includes a first teacher network for repairing images, a second teacher network, a third teacher network, a student network for extracting image features, a knowledge interactor for feature interaction, a first common feature projector for projecting features into a common feature space, a second common feature projector, a third common feature projector, a fourth common feature projector, and a knowledge integrator for integrating features.

2. The image restoration method according to claim 1, characterized in that: The method of obtaining a synthetic harsh environment image by using the clear environment image comprises: Noise is added to the clear environment image to obtain a synthetic harsh environment image.

3. The image restoration method according to claim 2, characterized in that: The semi-supervised module is used to: calculating a plurality of screening indicators for screening images; Performing maximum-minimum normalization processing on each screening index respectively to obtain a normalized screening index; After negating some of the normalized screening indicators, subtract the value before negation from 1 to obtain a new screening indicator; The two-norm values ​​of all the normalized screening indicators that are not negated and all the new screening indicators are taken to obtain the final screening indicators; The final screening indicators are sorted from large to small, and the real industrial harsh environment images ranked at the top are combined with the synthetic harsh environment images to form a new data set.

4. The image restoration method according to claim 3, characterized in that: The screening indicators include: Image quality indicators used to evaluate the difficulty of image restoration in harsh industrial environments; An information content index for evaluating the amount of information contained in an image, wherein the information content index includes information entropy and total variation; The restoration effect index used to evaluate the quality of image restoration includes structural similarity and peak signal-to-noise ratio.

5. The image restoration method according to claim 4, characterized in that: The three stages include: knowledge accumulation stage, knowledge integration stage, and knowledge testing stage.

6. The image restoration method according to claim 5, characterized in that: The knowledge accumulation stage completes the unique feature accumulation of each harsh environment through the first teacher network, the second teacher network, the third teacher network, the student network, the knowledge interactor, the first common feature projector, the second common feature projector, the third common feature projector, and the fourth common feature projector; The input end of the first teacher network, the input end of the second teacher network, the input end of the third teacher network, and the input end of the student network are all connected to the output end of the data set combination unit; The first output end of the first teacher network is connected to the first input end of the first common feature projector, the first input end of the second common feature projector, the first input end of the third common feature projector, and the first input end of the fourth common feature projector, respectively, and the second output end of the first teacher network is connected to the first input end of the knowledge integration unit; The first output end of the second teacher network is connected to the second input end of the first common feature projector, the second input end of the second common feature projector, the second input end of the third common feature projector, and the second input end of the fourth common feature projector, respectively, and the second output end of the second teacher network is connected to the second input end of the knowledge integration unit; The first output end of the third teacher network is connected to the third input end of the first common feature projector, the third input end of the second common feature projector, the third input end of the third common feature projector, and the third input end of the fourth common feature projector, respectively, and the second output end of the third teacher network is connected to the third input end of the knowledge integration unit; The first output terminal of the student network is connected to the input terminal of the knowledge interactor, and the second output terminal of the student network is connected to the fourth input terminal of the knowledge integration unit; The output end of the knowledge interactor is respectively connected to the fourth input end of the first common feature projector, the fourth input end of the second common feature projector, the fourth input end of the third common feature projector, and the fourth input end of the fourth common feature projector.

7. The image restoration method according to claim 6, characterized in that: The knowledge integration stage integrates similar features of all harsh environments using a knowledge integrator based on the knowledge accumulation stage; The first input terminal of the knowledge integrator is connected to the second output terminal of the first teacher network; The second input terminal of the knowledge integrator is connected to the second output terminal of the second teacher network; The third input terminal of the knowledge integrator is connected to the second output terminal of the third teacher network; The fourth input terminal of the knowledge integrator is connected to the second output terminal of the student network; The output end of the knowledge integrator is the output end of the teacher-student model, which is used to output the knowledge integration result.

8. The image restoration method according to claim 7, characterized in that: In the knowledge testing stage, feature extraction and testing are completed through the student network; The output end of the student network is the output end of the teacher-student model, which is used to output the knowledge test result.

9. A terminal device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the image restoration method according to any one of claims 1 to 8 is implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the image restoration method according to any one of claims 1 to 8 is implemented.

Images