Image restoration method and related equipment

Abstract

The present invention provides an image restoration method and related equipment, which collects real industrial harsh environment images and clear environment images, and uses the clear environment images to obtain synthetic harsh environment images and the real industrial harsh environment images as input to construct a semi-supervised module for screening training data, obtains a new data set and inputs it into the constructed teacher-student model for training in three stages to obtain an industrial harsh environment restoration model; inputs the acquired target industrial environment image into the industrial harsh environment restoration model for restoration to obtain a clear image corresponding to the target industrial environment image; compared with the existing technology, the present invention screens training data through a semi-supervised module to improve the performance and generalization of the model in different environments; trains the teacher-student model from three different stages to allow the model to fully learn the unique features and common features between each harsh environment, prevents insufficient feature learning, and thus improves the robustness of industrial harsh environment image restoration.

Description

Technical Field

[0001] The present invention relates to the technical field of computer vision and image processing, and in particular to an image restoration method and related equipment. Background Art

[0002] With the development of artificial intelligence, deep learning technology has gradually become the mainstream method for image restoration in harsh industrial environments. However, current image restoration methods are only applicable to a specific type of typical industrial harsh environment. However, in the application of industrial vision systems, encountering a variety of different types of harsh environments is inevitable. Therefore, multiple sets of harsh environment-specific model parameters are required to handle various harsh environments, which imposes additional computational and storage burdens. Therefore, it is crucial to develop a unified model that can address various types of harsh environments with a single set of network parameters.

[0003] Recently, some methods have been used to repair images in industrial harsh environments using a set of parameters, and a unified model for image restoration in industrial harsh environments has been constructed. However, these methods also have their limitations. First, these methods rely on large-scale synthetic data for training, which affects their performance and generalization in real environments. Second, due to the domain differences of features between different harsh environments, these methods mainly focus on the unique features of different types of harsh environments, while ignoring all the common features between them. Finally, most of these methods are trained directly with original images and real labels, but the complexity of the task makes the model feature learning insufficient in the early stages of training, and learns suboptimal features because it may focus on details that do not contribute to task generalization, which will hinder the model from learning more relevant high-level features.

[0004] Therefore, how to develop a unified image restoration model for harsh industrial environments so that it can fully exploit the unique and common features of various types of harsh industrial environments and have good robustness and performance in real environments is an urgent problem that needs to be solved. Summary of the Invention

[0005] The present invention provides an image restoration method and related equipment, the purpose of which is to improve the robustness of image restoration in harsh industrial environments.

[0006] In order to achieve the above object, the present invention provides an image restoration method, comprising:

[0007] Step 1: collect real industrial harsh environment images and clear environment images, and use the clear environment images to obtain a synthetic harsh environment image;

[0008] Step 2: Input the real industrial harsh environment images and synthetic harsh environment images into the constructed semi-supervised module for training data screening to obtain a new dataset;

[0009] Step 3: Input the new data set into the constructed teacher-student model and train the teacher-student model in three stages to obtain the industrial harsh environment remediation model;

[0010] Step 4: Input the acquired target industrial environment image into the industrial harsh environment restoration model for restoration, and obtain a clear image corresponding to the target industrial environment image;

[0011] The teacher-student model includes a first teacher network for repairing images, a second teacher network, a third teacher network, a student network for extracting image features, a knowledge interactor for feature interaction, a first common feature projector for projecting features into a common feature space, a second common feature projector, a third common feature projector, a fourth common feature projector, and a knowledge integrator for integrating features.

[0012] Furthermore, the clear environment image is used to obtain a synthetic harsh environment image, including:

[0013] Noise is added to the clear environment image to obtain a synthetic harsh environment image.

[0014] More specifically, the semi-supervised module is used to:

[0015] calculating a plurality of screening indices for screening images;

[0016] Perform maximum-minimum normalization processing on each screening index respectively to obtain the normalized screening index;

[0017] After negating some of the normalized screening indicators, subtract the value before negation from 1 to obtain a new screening indicator;

[0018] The two-norm values ​​of all the normalized screening indicators that are not negated and all the new screening indicators are taken to obtain the final screening indicators;

[0019] The final screening indicators are sorted from large to small, and the real industrial harsh environment images ranked at the top are combined with the synthetic harsh environment images to form a new dataset.

[0020] Specifically, the screening criteria include:

[0021] Image quality metrics used to evaluate the difficulty of image restoration in harsh industrial environments;

[0022] Information content indicators used to evaluate the amount of information contained in an image, including information entropy and total variation;

[0023] Restoration effect indicators used to evaluate the quality of image restoration, including structural similarity and peak signal-to-noise ratio.

[0024] To further elaborate, the three stages include: knowledge accumulation stage, knowledge integration stage, and knowledge testing stage.

[0025] Furthermore, the knowledge accumulation stage completes the unique feature accumulation of each harsh environment through the first teacher network, the second teacher network, the third teacher network, the student network, the knowledge interactor, the first public feature projector, the second public feature projector, the third public feature projector, and the fourth public feature projector;

[0026] The input end of the first teacher network, the input end of the second teacher network, the input end of the third teacher network, and the input end of the student network are all connected to the output end of the data set combination unit;

[0027] The first output end of the first teacher network is connected to the first input end of the first common feature projector, the first input end of the second common feature projector, the first input end of the third common feature projector, and the first input end of the fourth common feature projector respectively, and the second output end of the first teacher network is connected to the first input end of the knowledge integration unit;

[0028] The first output end of the second teacher network is connected to the second input end of the first common feature projector, the second input end of the second common feature projector, the second input end of the third common feature projector, and the second input end of the fourth common feature projector respectively, and the second output end of the second teacher network is connected to the second input end of the knowledge integration unit;

[0029] The first output end of the third teacher network is connected to the third input end of the first common feature projector, the third input end of the second common feature projector, the third input end of the third common feature projector, and the third input end of the fourth common feature projector, respectively; the second output end of the third teacher network is connected to the third input end of the knowledge integration unit;

[0030] The first output terminal of the student network is connected to the input terminal of the knowledge interactor, and the second output terminal of the student network is connected to the fourth input terminal of the knowledge integration unit;

[0031] The output end of the knowledge interactor is connected to the fourth input end of the first common feature projector, the fourth input end of the second common feature projector, the fourth input end of the third common feature projector, and the fourth input end of the fourth common feature projector respectively.

[0032] Furthermore, the knowledge integration phase is completed using the knowledge integrator based on the knowledge accumulation phase;

[0033] The first input terminal of the knowledge integrator is connected to the second output terminal of the first teacher network;

[0034] The second input terminal of the knowledge integrator is connected to the second output terminal of the second teacher network;

[0035] The third input terminal of the knowledge integrator is connected to the second output terminal of the third teacher network;

[0036] The fourth input terminal of the knowledge integrator is connected to the second output terminal of the student network;

[0037] The output end of the knowledge integrator is the output end of the teacher-student model, which is used to output the knowledge integration results.

[0038] Furthermore, in the knowledge testing phase, feature extraction and testing are completed through the student network;

[0039] The output end of the student network is the output end of the teacher-student model, which is used to output the knowledge test results.

[0040] The present invention also provides a terminal device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the image restoration method is implemented when the processor executes the computer program.

[0041] The present invention also provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the image restoration method is implemented.

[0042] The above solution of the present invention has the following beneficial effects:

[0043] The present invention collects real industrial harsh environment images and clear environment images, and uses the clear environment images to obtain synthetic harsh environment images; the real industrial harsh environment images and the synthetic harsh environment images are input into a constructed semi-supervised module for training data screening to obtain a new data set and input into the constructed teacher-student model, and the teacher-student model is trained in three stages to obtain an industrial harsh environment restoration model; the acquired target industrial environment image is input into the industrial harsh environment restoration model for restoration to obtain a clear image corresponding to the target industrial environment image; the teacher-student model includes a first teacher network for image restoration, a second teacher network, a third teacher network, a student network for extracting image features, a knowledge interactor for feature interaction, a first common feature projector for projecting features into a common feature space, a second common feature projector, a third common feature projector, a fourth common feature projector, and a knowledge integrator for integrating features; compared with the prior art, the present invention screens training data through a semi-supervised module to improve the performance and generalization of the model in different environments; the teacher-student model is trained from three different stages to allow the model to fully learn the unique features and common features between each harsh environment, prevent insufficient feature learning, and thus improve the robustness of industrial harsh environment image restoration.

[0044] Other beneficial effects of the present invention will be described in detail in the subsequent specific implementation section. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 A schematic diagram of a flow chart of an embodiment of the present invention;

[0046] Figure 2 Schematic diagram of the structure of the first teacher network in an embodiment of the present invention;

[0047] Figure 3 A schematic diagram of the connection between the teacher and student models in the knowledge accumulation stage of an embodiment of the present invention;

[0048] Figure 4 A schematic diagram of the connection between the teacher and student models in the knowledge integration stage of an embodiment of the present invention;

[0049] Figure 5 A schematic diagram of the connection between the teacher and student models in the knowledge testing phase of an embodiment of the present invention;

[0050] Figure 6 Schematic diagram of the structure of the first common feature projector in an embodiment of the present invention;

[0051] Figure 7 Schematic diagram of the structure of the knowledge interactor in an embodiment of the present invention;

[0052] Figure 8 Schematic diagram of the structure of the terminal device in an embodiment of the present invention. DETAILED DESCRIPTION

[0053] To make the technical problems, technical solutions, and advantages to be solved by the present invention more clear, the following is a detailed description with reference to the accompanying drawings and specific embodiments. It is obvious that the embodiments described are only some of the embodiments of the present invention, not all of them. All other embodiments derived by persons of ordinary skill in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0054] In the description of the present invention, it should be noted that the terms "center," "upper," "lower," "left," "right," "vertical," "horizontal," "inner," and "outer," etc., indicating orientations or positional relationships, are based on the orientations or positional relationships shown in the accompanying drawings and are intended solely to facilitate and simplify the description of the present invention. They are not intended to indicate or imply that the devices or components referred to must have, be constructed, or operate in a specific orientation, and therefore should not be construed as limitations on the present invention. Furthermore, the terms "first," "second," and "third" are used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0055] In the description of the present invention, it should be noted that, unless otherwise expressly specified or limited, the terms "mounted," "connected," and "connected" should be understood broadly. For example, they may refer to a locking connection, a detachable connection, or an integral connection; they may refer to a mechanical connection or an electrical connection; they may refer to a direct connection or an indirect connection through an intermediate medium; and they may refer to internal communication between two components. Those skilled in the art will understand the specific meanings of the above terms in the present invention based on the specific circumstances.

[0056] In addition, the technical features involved in the different embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0057] In view of the existing problems, the present invention provides an image restoration method and related equipment.

[0058] like Figure 1 As shown, an embodiment of the present invention provides an image restoration method, comprising:

[0059] Step 1: collect real industrial harsh environment images and clear environment images, and use the clear environment images to obtain a synthetic harsh environment image;

[0060] Step 2: Input the real industrial harsh environment images and synthetic harsh environment images into the constructed semi-supervised module for training data screening to obtain a new dataset;

[0061] Step 3: Input the new data set into the constructed teacher-student model and train the teacher-student model in three stages to obtain the industrial harsh environment remediation model;

[0062] Step 4: Input the acquired target industrial environment image into the industrial harsh environment restoration model for restoration, and obtain a clear image corresponding to the target industrial environment image;

[0063] The teacher-student model includes a first teacher network for repairing images, a second teacher network, a third teacher network, a student network for extracting image features, a knowledge interactor for feature interaction, a first common feature projector for projecting features into a common feature space, a second common feature projector, a third common feature projector, a fourth common feature projector, and a knowledge integrator for integrating features.

[0064] Specifically, if Figure 2 As shown in the figure, the first teacher network has the same structure as the second teacher network, the third teacher network and the student network, and all consist of an encoder and a decoder.

[0065] The input end of the encoder is the input end of the network, and the first output end, the second output end, the third output end and the fourth output end of the encoder are respectively connected to the first to fourth input ends of the decoder.

[0066] The first input terminal, the second input terminal, the third input terminal and the fourth input terminal of the decoder are respectively connected to the first to fourth output terminals of the encoder, and the output terminal of the encoder is the output terminal of the network.

[0067] The encoder includes a first convolution module, a first downsampling module, a second convolution module, a second downsampling module, a third convolution module, a third downsampling module, and a fourth convolution module.

[0068] The decoder consists of a fifth convolution module, a first upsampling module, a sixth convolution module, a second upsampling module, a seventh convolution module, a third upsampling module, and an eighth convolution module.

[0069] The input end of the first convolution module is the input end of the decoder, the first output end of the first convolution module is connected to the input end of the first down-sampling module, and the second output end of the first convolution module is connected to the first input end of the eighth convolution module; the input end of the first down-sampling module is connected to the first output end of the first convolution module, and the output end of the first down-sampling module is connected to the input end of the second convolution module; the input end of the second convolution module is connected to the output end of the first down-sampling module, the first output end of the second convolution module is connected to the second down-sampling module, and the second output end of the second convolution module is connected to the first input end of the seventh convolution module; the input end of the second down-sampling module is connected to the output end of the first convolution module, the first output end of the second convolution module is connected to the second down-sampling module, and the second output end of the second convolution module is connected to the first input end of the seventh convolution module; the input end of the second down-sampling module is connected to the first output end of the first convolution module The input end is connected to the first output end of the second convolution module, the output end of the second down-sampling module is connected to the input end of the third convolution module; the input end of the third convolution module is connected to the output end of the second down-sampling module, the first output end of the third convolution module is connected to the third down-sampling module, and the second output end of the third convolution module is connected to the first input end of the seventh convolution module; the input end of the third down-sampling module is connected to the first output end of the third convolution module, and the second output end of the third down-sampling module is connected to the input end of the sixth convolution module; the input end of the fourth convolution module is connected to the output end of the third down-sampling module, and the output end of the fourth convolution module is the encoding end. The output end of the decoder is connected to the fifth convolution module; the input end of the fifth convolution module is connected to the input end of the decoder and the output end of the fourth convolution module, and the output end of the fifth convolution module is connected to the first upsampling module; the input end of the first upsampling module is connected to the output end of the fifth convolution module, and the output end of the first upsampling module is connected to the first input end of the sixth convolution module; the first input end of the sixth convolution module is connected to the output end of the first upsampling module, the second input end of the sixth convolution module is connected to the second output end of the third convolution module, and the output end of the sixth convolution module is connected to the second upsampling module; the input end of the second upsampling module is connected to the output end of the sixth convolution module, and the output end of the sixth convolution module is connected to the second upsampling module. The output end of the second upsampling module is connected to the first input end of the seventh convolution module; the first input end of the seventh convolution module is connected to the output end of the third upsampling module, the second input end of the seventh convolution module is connected to the second output end of the second convolution module, and the output end of the seventh convolution module is connected to the third upsampling module; the input end of the third upsampling module is connected to the output end of the seventh convolution module, and the output end of the third upsampling module is connected to the first input end of the eighth convolution module; the first input end of the eighth convolution module is connected to the output end of the third upsampling module, the second input end of the eighth convolution module is connected to the second output end of the first convolution module, and the output end of the eighth convolution module is the decoder output end.

[0070] In an embodiment of the present invention, harsh environments include fog and haze, rainy days, and snowy days. Real industrial harsh environment images are images with a clarity less than a preset threshold value collected in fog and haze, rainy days, and snowy days, such as factory environment images and mountain environment images. Clear environment images are images with a clarity greater than a preset threshold value, such as factory environment images and mountain environment images.

[0071] Specifically, the clear environment image is used to obtain the synthetic harsh environment image, including:

[0072] Noise is added to the clear environment image to obtain a synthetic harsh environment image.

[0073] In the embodiment of the present invention, the noise may be additive noise such as Gaussian noise, Rayleigh noise, exponential noise, etc., which is used to reduce the clarity of the clear environment image.

[0074] In the embodiment of the present invention, the clear environment image is used as the real marked data, and the synthetic harsh environment image is used as the data to be processed.

[0075] Most preferably, the semi-supervised module is used to:

[0076] calculating a plurality of screening indices for screening images;

[0077] Perform maximum-minimum normalization on each screening index to obtain a normalized screening index to eliminate the differences between its quantitative units;

[0078] After negating some of the normalized screening indicators, subtract the value before negation from 1 to obtain a new screening indicator;

[0079] The two-norm values ​​of all the normalized screening indicators that are not negated and all the new screening indicators are taken to obtain the final screening indicators;

[0080] The final screening indicators are sorted from large to small, and the real industrial harsh environment images ranked at the top are combined with the synthetic harsh environment images to form a new dataset.

[0081] In an embodiment of the present invention, the semi-supervised module may be composed of an image screening unit for determining a screening index and a data combination unit for combining the screened images. Both the image screening unit and the data combination unit are soft modules. For example, in specific structures, both implement data processing through mathematical models, and their connection relationship is as follows:

[0082] The input of the image screening unit is the input of the semi-supervised module;

[0083] The output end of the image screening unit is connected to the input end of the data set combination unit;

[0084] The input of the dataset combination unit is connected to the input of the semi-supervised module;

[0085] The output of the dataset combination unit is connected to the input of the teacher-student model.

[0086] Specifically, the formula for maximum-minimum normalization is as follows:

[0087]

[0088] Among them, the physical meaning of each parameter is x i Represents the original index to be normalized, x max and x min Represent the maximum and minimum values ​​of the original index, x' i Represents the normalized index.

[0089] Most preferably, the screening indicators include:

[0090] Image quality metrics used to evaluate the difficulty of image restoration in harsh industrial environments;

[0091] Information content indicators used to evaluate the amount of information contained in an image, including information entropy and total variation;

[0092] Restoration effect indicators used to evaluate the quality of image restoration, including structural similarity and peak signal-to-noise ratio.

[0093] In the embodiment of the present invention, the image quality assessment score corresponding to each image is calculated as the image quality indicator. The image quality assessment score corresponding to each image is calculated as follows:

[0094] The new dataset is input into the teacher-student model for repeated training. Multiple image blocks are extracted and normalized. The expression is:

[0095]

[0096] Where x represents the pixel value in the image block, μ x and σ x are the mean and standard deviation of the image block, ∈; is a small constant used to avoid division by zero.

[0097] Extract the statistical features of each image block and form a feature vector:

[0098] f(x')=[μ' x ,σ' x ,cov(x')]

[0099] The Mahalanobis distance between the feature vector of each image block and the trained Gaussian model is calculated by the following formula:

[0100]

[0101] The Mahalanobis distances of all image blocks are averaged to obtain the overall NIQE value of the image. The overall NIQE value of the image is used as the image quality indicator. The calculation formula for the overall NIQE value of the image is:

[0102]

[0103] Among them, D(x i ') represents the Mahalanobis distance between the feature vector of the i-th image block x' and the trained Gaussian model, and n represents the total number of image blocks.

[0104] In the embodiment of the present invention, the information entropy and total variation corresponding to each image are calculated as information content indicators. The information entropy is calculated as follows:

[0105] For a given grayscale image, we first need to calculate the frequency of each grayscale level (usually an integer between 0 and 255), which can be done by calculating the grayscale histogram:

[0106]

[0107] Calculate the self-information of each gray level. The self-information is used to measure the amount of information brought by the appearance of a gray level:

[0108] I(i)=-log2(p i )

[0109] Calculate the information entropy of an image: The information entropy of an image is the expected value of the self-information of all gray levels, that is, the weighted average:

[0110]

[0111] Among them, the physical meaning of each parameter is: L is the number of gray levels, p i is the frequency of gray level i;

[0112] The total variation is calculated as follows:

[0113] First, we need to calculate the gradient of the image, that is, the rate of change of the image in the horizontal and vertical directions:

[0114]

[0115] Calculate the total variation: The total variation of the image is the sum of the modulus of the image gradient. For image I, its total variation TV(I) can be expressed as:

[0116]

[0117] The physical meaning of each parameter in the above expression is that I(i,j) represents the image of the total variation to be calculated, It is the modulus of the image gradient at position (i, j), that is, the rate of change of the image at that position.

[0118] In the embodiment of the present invention, the structural similarity and peak signal-to-noise ratio of each image pair are calculated as the restoration effect index. The structural similarity is calculated as follows:

[0119] The first component of image structure similarity is brightness comparison, which measures the similarity of brightness by calculating the average brightness of the two images. Suppose the grayscale values ​​of the two images are x and y respectively, and the corresponding average brightness is:

[0120]

[0121] Among them, μ x and μ y Represents the average brightness of image x and y respectively, and N is the number of pixels in the image;

[0122] The brightness comparison function is:

[0123]

[0124] The second component of image structural similarity is contrast comparison, which measures the contrast similarity by calculating the standard deviation of the images. The standard deviations of images x and y are:

[0125]

[0126] The contrast comparison function is:

[0127]

[0128] The third component of image structural similarity is structural comparison, which measures the structural similarity by calculating the covariance of images:

[0129]

[0130] Finally, the overall similarity between two images is measured by the product of the three components of brightness, contrast and structure:

[0131] SSIM(x,y)=[l(x,y)] α [c(x,y)] β [s(x,y)} γ

[0132] Among them, α, β and γ are weight parameters used to adjust each component;

[0133] The peak signal-to-noise ratio is calculated as follows:

[0134] First, we need to calculate the mean square error (MSE) between the original image and the restored image:

[0135]

[0136] Where I(i, j) and K(i, j) represent the pixel values ​​of the original image and the reconstructed image at position (i, j), respectively;

[0137] After calculating the MSE, we can further calculate the peak signal-to-noise ratio PSNR:

[0138]

[0139] Among them, MAx represents the maximum pixel value of the image, and MSE represents the mean square error between the original image and the restored image.

[0140] Specifically, the three stages include: knowledge accumulation stage, knowledge integration stage, and knowledge testing stage.

[0141] It should be noted that the knowledge accumulation stage is similar to the initial stage of knowledge accumulation in real-life human learning. At this stage, the student does not have enough knowledge and needs to absorb the knowledge of each teacher individually. In this stage, there are multiple well-trained teacher networks responsible for image restoration in different types of harsh environments and a student network. The student network is trained simultaneously with all different teacher networks. Through learning, it transfers the knowledge of each teacher network related to the type of harsh environment to itself, preliminarily learns the unique characteristics of each harsh environment, and realizes comprehensive image restoration in industrial harsh environments.

[0142] In the knowledge integration stage, the student network has already acquired a certain amount of knowledge through the knowledge accumulation stage and has a certain knowledge integration ability. It can extract the common characteristics of different types of harsh environments, similar to the process of human learning. In this stage, when the student network is trained with different teacher networks, it not only learns the knowledge of the corresponding teacher network, but also learns the knowledge from other teacher networks. It integrates the knowledge of all teacher networks together to learn and discover the similar characteristics of all harsh environments.

[0143] The knowledge testing phase mainly involves self-study and self-testing of the student network, and training without the guidance of the teacher network. By setting higher requirements, the robustness and discrimination ability of the network are improved, and satisfactory results are achieved in different types of harsh environments.

[0144] Most preferably, Figure 3 As shown, the knowledge accumulation stage completes the unique feature accumulation of each harsh environment through the first teacher network, the second teacher network, the third teacher network, the student network, the knowledge interactor, the first public feature projector, the second public feature projector, the third public feature projector, and the fourth public feature projector;

[0145] The input end of the first teacher network, the input end of the second teacher network, the input end of the third teacher network, and the input end of the student network are all connected to the output end of the data set combination unit;

[0146] The first output end of the first teacher network is connected to the first input end of the first common feature projector, the first input end of the second common feature projector, the first input end of the third common feature projector, and the first input end of the fourth common feature projector respectively, and the second output end of the first teacher network is connected to the first input end of the knowledge integration unit;

[0147] The first output end of the second teacher network is connected to the second input end of the first common feature projector, the second input end of the second common feature projector, the second input end of the third common feature projector, and the second input end of the fourth common feature projector respectively, and the second output end of the second teacher network is connected to the second input end of the knowledge integration unit;

[0148] The first output end of the third teacher network is connected to the third input end of the first common feature projector, the third input end of the second common feature projector, the third input end of the third common feature projector, and the third input end of the fourth common feature projector, respectively; the second output end of the third teacher network is connected to the third input end of the knowledge integration unit;

[0149] The first output terminal of the student network is connected to the input terminal of the knowledge interactor, and the second output terminal of the student network is connected to the fourth input terminal of the knowledge integration unit;

[0150] The output end of the knowledge interactor is connected to the fourth input end of the first common feature projector, the fourth input end of the second common feature projector, the fourth input end of the third common feature projector, and the fourth input end of the fourth common feature projector respectively.

[0151] It should be noted that since the student network is only in the early stages of training at this stage and does not have sufficient knowledge, the loss of the student model at this stage is mainly based on the prediction results of the teacher model.

[0152] Most preferably, Figure 4 As shown, the knowledge integration stage uses the knowledge integrator to integrate the similar features of all harsh environments based on the knowledge accumulation stage, where the knowledge integrator corresponds to the teacher knowledge integrator in the figure;

[0153] The first input terminal of the knowledge integrator is connected to the second output terminal of the first teacher network;

[0154] The second input terminal of the knowledge integrator is connected to the second output terminal of the second teacher network;

[0155] The third input terminal of the knowledge integrator is connected to the second output terminal of the third teacher network;

[0156] The fourth input terminal of the knowledge integrator is connected to the second output terminal of the student network;

[0157] The output end of the knowledge integrator is the output end of the teacher-student model, which is used to output the knowledge integration results.

[0158] It should be noted that the loss in this stage consists of three parts: the loss between the feature projections of the teacher network and the student network, the loss between the final restoration results of the teacher network and the student network, and the loss between the restoration results of the student network and the true restoration results. The difference is that the reply result of the teacher network is the result after integration.

[0159] Most preferably, Figure 5 As shown, in the knowledge testing phase, feature extraction and testing are completed through the student network;

[0160] The output end of the student network is the output end of the teacher-student model, which is used to output the knowledge test results.

[0161] It should be noted that the loss at this stage is composed solely of the loss between the restoration results of the student network and the true restoration results.

[0162] In an embodiment of the present invention, the common feature projector is used to project teacher features and student features from the teacher and student networks into a common feature space. It is a learnable module that enables the network to determine the feature space most suitable for common feature learning.

[0163] like Figure 6 As shown in the figure, the first common feature projector, the second common feature projector, the third common feature projector, and the fourth common feature projector are all composed of several convolution blocks with a stride of 1 and a ReLU activation function; in the common feature space, the projection feature error is used to constrain the feature learning process to make the projected features closer, and the L1 loss is used to calculate the distance between the two projected features; in addition, pyramid pooling is used for the projected features to expand the context information at different levels.

[0164] In addition, in order to robustly learn knowledge from multiple teacher networks, bidirectional feature matching is used to constrain the learned features; the projected features of the teacher network are projected back to the original input space through the inverse progressive feature projector; then, the difference between them and the original features is calculated through the loss of projected feature verification. Specifically, Figure 7 As shown, the knowledge interactor is used to first transform the input encoder features F en and decoder features F de By combining convolution blocks, α and β are transformed into corresponding hidden states and combined with F en and F de Interact and obtain interactive features and After that, the interaction features are passed through two other combined convolution blocks γ and θ, and then added or subtracted to the interaction features to obtain the updated features F' en and F' de :

[0165]

[0166] In this embodiment of the present invention, the final segmentation result image and the binary label of the corresponding image are input into the loss function to calculate the loss of the response. The loss calculation formula is as follows:

[0167] L=βL MAE +(1-β)L CP ,β∈[0,1]

[0168] Among them, L MAE Denotes the absolute error loss, L CP represents the projection loss, and β represents the weight;

[0169] The formula for calculating cross entropy loss is:

[0170]

[0171] Where N represents the number of samples, Represents the actual image of the sample, y i Represents the model prediction sample;

[0172] The calculation formula of projection loss is as follows:

[0173]

[0174] Where L represents the total number of encoders, Denotes the encoder L in the corresponding teacher network T i The feature map in represents the feature map of encoder L in the student network, and f(·) represents the projection operation.

[0175] In the embodiment of the present invention, after calculating the loss, back propagation is required to further optimize the parameters. The gradient of the loss function with respect to the parameters in the network is calculated by the chain rule. The Nadam optimizer is used to update each parameter using the gradient in the direction of loss reduction. The Nadam parameter update formula is:

[0176]

[0177] m t =β1m t-1 +(1-β1)g t

[0178]

[0179] Where t represents the time step, represents the gradient at time t, θ t-1 represents the parameter value at time t-1, α represents the learning rate, β1, β2, ε, w, ∈ are constants, respectively It is the t-th power of β1 and β2, and the model with the best performance on the validation set is finally saved.

[0180] Specifically, step 4 includes:

[0181] The acquired target industrial environment image is input into the industrial harsh environment restoration model, and the unique features of the image are extracted through the student network to obtain the student restoration results. After the extracted unique features are interacted through the knowledge interactor, the interacted features are projected into the common feature space through the first common feature projector, the second common feature projector, the third common feature projector, and the fourth common feature projector to obtain the similar features of the image. The similar features are input into the first teacher model, the second teacher model, and the third teacher model to repair the image to obtain the teacher restoration results. The student restoration results and the teacher restoration results are input into the knowledge integrator for integration to obtain a clear image corresponding to the target industrial environment image.

[0182] In the embodiment of the present invention, the target industrial environment image is an industrial environment image collected in any environment.

[0183] The embodiment of the present invention compares the effects of the image restoration method provided by this application with other commonly used methods on the collected defogging, deraining, and desnowing datasets. The methods used for comparison are divided into two categories. The first category is a single method specifically for a specific harsh environment, and the second category is a unified model.

[0184] For a single method, in addition to comparing its recovery effect on the corresponding dataset, the embodiment of the present invention also uses a mixed dataset to train and verify its effect; for a unified model, the mixed dataset is directly used for training and performance verification. The results are shown in the following table:

[0185] Table 1

[0186]

[0187] It should be noted that the numbers in Table 1 above are in the form of X / Y, where X represents the structural similarity index of the corresponding method and Y represents the maximum signal-to-noise ratio index; the embodiment of the present invention uses these two indicators to judge the removal effect.

[0188] As can be seen from Table 1, the image restoration method for harsh industrial environments provided by the embodiment of the present invention surpasses other comparison methods in all indicators and has better performance than other methods. The experimental results prove that the embodiment of the present invention has good robustness in image restoration tasks in harsh industrial environments.

[0189] The embodiment of the present invention collects real industrial harsh environment images and clear environment images, and uses the clear environment images to obtain synthetic harsh environment images; the real industrial harsh environment images and synthetic harsh environment images are input into a constructed semi-supervised module for training data screening to obtain a new data set and input into the constructed teacher-student model, and the teacher-student model is trained in three stages to obtain an industrial harsh environment restoration model; the acquired target industrial environment image is input into the industrial harsh environment restoration model for restoration to obtain a clear image corresponding to the target industrial environment image; the teacher-student model includes a first teacher network for image restoration, a second teacher network, a third teacher network, a student network for extracting image features, a knowledge interactor for feature interaction, a first common feature projector for projecting features into a common feature space, a second common feature projector, a third common feature projector, a fourth common feature projector, and a knowledge integrator for integrating features; compared with the prior art, the embodiment of the present invention screens training data through a semi-supervised module to improve the performance and generalization of the model in different environments; the teacher-student model is trained from three different stages to allow the model to fully learn the unique features and common features between each harsh environment, prevent insufficient feature learning, and thus improve the robustness of industrial harsh environment image restoration.

[0190] An embodiment of the present invention further provides a terminal device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the image restoration method is implemented when the processor executes the computer program.

[0191] like Figure 8 As shown, the terminal device D10 of this embodiment includes: at least one processor D100 ( Figure 8 Only one processor is shown in the figure), a memory D101, and a computer program D102 stored in the memory D101 and executable on the at least one processor D100. When the processor D100 executes the computer program D102, the steps of the above-described method for constructing an uncoupled blasting load model are implemented. Alternatively, when the processor D100 executes the computer program D102, the functions of the modules / units in the above-described device embodiments are implemented.

[0192] The terminal device D10 can be a computing device such as a desktop computer, a notebook, a PDA, a server, a server cluster, a cloud server, etc. The terminal device may include, but is not limited to, a processor D100 and a memory D101. It will be understood by those skilled in the art that Figure 8 This is merely an example of the terminal device D10 and does not constitute a limitation on the terminal device D10 . The terminal device D10 may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, the terminal device D10 may also include input and output devices, network access devices, etc.

[0193] The processor D100 may be a central processing unit (CPU), or other general-purpose processors, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. A general-purpose processor may be a microprocessor or any conventional processor.

[0194] In some embodiments, the memory D101 may be an internal storage unit of the terminal device D10, such as a hard disk or memory of the terminal device D10. In other embodiments, the memory D101 may also be an external storage device of the terminal device D10, such as a plug-in hard disk, a smart memory card (SMC, SmartMedia Card), a secure digital (SD, Secure Digital) card, a flash card, etc. equipped on the terminal device D10. Furthermore, the memory D101 may also include both an internal storage unit of the terminal device D10 and an external storage device. The memory D101 is used to store an operating system, an application program, a boot loader (BootLoader), data, and other programs, such as the program code of the computer program. The memory D101 may also be used to temporarily store data that has been output or is to be output.

[0195] It should be noted that the information interaction, execution process, etc. between the above-mentioned devices / units are based on the same concept as the method embodiment of this application. Their specific functions and technical effects can be found in the method embodiment section and will not be repeated here.

[0196] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated into one processing unit, or each unit can exist physically alone, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0197] An embodiment of the present invention further provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the image restoration method is implemented.

[0198] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the process in the above-mentioned embodiment method, which can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a computer-readable storage medium. When the computer program is executed by the processor, the steps of the above-mentioned various method embodiments can be implemented. Wherein, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include at least: any entity or device capable of carrying the computer program code to a construction device / terminal device, a recording medium, a computer memory, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), an electric carrier signal, a telecommunication signal and a software distribution medium. For example, a USB flash drive, a mobile hard disk, a magnetic disk or an optical disk.

[0199] The above is a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should also be regarded as within the scope of protection of the present invention.

Claims

1. An image restoration method, characterized in that: include: Step 1: collecting a real industrial harsh environment image and a clear environment image, and using the clear environment image to obtain a synthetic harsh environment image; Step 2: Input the real industrial harsh environment images and the synthetic harsh environment images into the constructed semi-supervised module to perform training data screening to obtain a new data set; Step 3: Input the new data set into the constructed teacher-student model and train the teacher-student model in three stages to obtain an industrial harsh environment remediation model, wherein the three stages include: knowledge accumulation stage, knowledge integration stage, and knowledge testing stage; The knowledge accumulation stage completes the unique feature accumulation of each harsh environment through the first teacher network, the second teacher network, the third teacher network, the student network, the knowledge interactor, the first public feature projector, the second public feature projector, the third public feature projector, and the fourth public feature projector; The input end of the first teacher network, the input end of the second teacher network, the input end of the third teacher network, and the input end of the student network are all connected to the output end of the data set combination unit; The first output end of the first teacher network is connected to the first input end of the first common feature projector, the first input end of the second common feature projector, the first input end of the third common feature projector, and the first input end of the fourth common feature projector respectively, and the second output end of the first teacher network is connected to the first input end of the knowledge integration unit; The first output end of the second teacher network is connected to the second input end of the first common feature projector, the second input end of the second common feature projector, the second input end of the third common feature projector, and the second input end of the fourth common feature projector respectively, and the second output end of the second teacher network is connected to the second input end of the knowledge integration unit; The first output end of the third teacher network is connected to the third input end of the first common feature projector, the third input end of the second common feature projector, the third input end of the third common feature projector, and the third input end of the fourth common feature projector respectively, and the second output end of the third teacher network is connected to the third input end of the knowledge integration unit; The first output terminal of the student network is connected to the input terminal of the knowledge interactor, and the second output terminal of the student network is connected to the fourth input terminal of the knowledge integration unit; The output end of the knowledge interactor is connected to the fourth input end of the first common feature projector, the fourth input end of the second common feature projector, the fourth input end of the third common feature projector, and the fourth input end of the fourth common feature projector respectively; The knowledge integration stage integrates similar features of all harsh environments using a knowledge integrator based on the knowledge accumulation stage; The first input terminal of the knowledge integrator is connected to the second output terminal of the first teacher network; The second input terminal of the knowledge integrator is connected to the second output terminal of the second teacher network; The third input terminal of the knowledge integrator is connected to the second output terminal of the third teacher network; The fourth input terminal of the knowledge integrator is connected to the second output terminal of the student network; The output end of the knowledge integrator is the output end of the teacher-student model, which is used to output the knowledge integration result; Step 4: input the acquired target industrial environment image into the industrial harsh environment restoration model for restoration, to obtain a clear image corresponding to the target industrial environment image; The teacher-student model includes a first teacher network for repairing images, a second teacher network, a third teacher network, a student network for extracting image features, a knowledge interactor for feature interaction, a first common feature projector for projecting features into a common feature space, a second common feature projector, a third common feature projector, a fourth common feature projector, and a knowledge integrator for integrating features.

2. The image restoration method according to claim 1, wherein: The method of obtaining a synthetic harsh environment image by using the clear environment image includes: Noise is added to the clear environment image to obtain a synthetic harsh environment image.

3. The image restoration method according to claim 2, wherein: The semi-supervised module is used to: calculating a plurality of screening indices for screening images; Perform maximum-minimum normalization processing on each screening index to obtain the normalized screening index; After negating some of the normalized screening indicators, subtract the value before negation from 1 to obtain a new screening indicator; Perform the two-norm value calculation on all the normalized screening indicators that are not negated and all the new screening indicators to obtain the final screening indicators; The final screening indicators are sorted from large to small, and the real industrial harsh environment images ranked at the top are combined with the synthetic harsh environment images to form a new dataset.

4. The image restoration method according to claim 3, wherein: The screening indicators include: Image quality metrics used to evaluate the difficulty of image restoration in harsh industrial environments; Information content indicators for evaluating the amount of information contained in an image, including information entropy and total variation; Restoration effect indicators used to evaluate the quality of image restoration, including structural similarity and peak signal-to-noise ratio.

5. The image restoration method according to claim 4, wherein: In the knowledge testing phase, feature extraction and testing are completed through the student network; The output end of the student network is the output end of the teacher-student model, which is used to output the knowledge test results.

6. A terminal device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the image restoration method according to any one of claims 1 to 5 is implemented.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the image restoration method according to any one of claims 1 to 5 is implemented.

Images