Representation learning and automatic restoration method for mixed distortion of video images in complex environments

Abstract

The present invention discloses a representation learning method for mixed distortion of video images in complex environments. Data sets with the same and different types of mixed distortion are established from videos. A method for learning distortion representation by contrastive learning is adopted, thereby realizing automatic modeling of mixed distortion directly from distorted video images. On this basis, an automatic restoration method for mixed distortion of video images in complex environments is constructed to realize automatic restoration of mixed distortion.

Description

Technical Field

[0001] The present invention relates to the field of image processing, and in particular to a representation learning and automatic restoration method for mixed distortion of video images in complex environments. Background Art

[0002] With the development of optical imaging technology and the improvement of intelligent video processing, video images have been widely used in industry and life. However, the video shooting environment is complex, with insufficient or uneven lighting, interference information such as high concentrations of dust and water mist, and electromagnetic interference, so there are different types of distortion in the video image. These distortions are mixed, mainly including uneven illumination or color distribution, noise, interference signals, and blur. Poor video image quality in certain environments has become a common problem, seriously hindering the application of video technology and its systems.

[0003] Traditional video image distortion processing algorithms are based on the filter concept and operate in the spatial domain or frequency domain, such as Gaussian filtering, bilateral filtering, Wiener filtering, least squares filtering, etc. However, these algorithms rely on fixed formulas, and a single filter is often only suitable for specific distortion types and lacks the robustness and ability to adaptively process distortion.

[0004] Deep learning video distortion correction algorithms mainly include supervised and unsupervised solutions. Supervised solutions require paired distorted images and standard images, but such image pairs do not exist in reality. Unsupervised algorithms do not require paired image pairs, but often assume that the distortion in the image is of a certain type. When it is inconsistent with the actual situation, the restoration quality is significantly reduced.

[0005] In summary, the existing video image correction methods lack an effective method to directly learn mixed distortion from distorted videos, making it difficult to correct multiple mixed distortions in complex environment video images, limiting the quality of video images and the development of related applications. Therefore, it is necessary to establish a method that does not require artificial assumptions about the distortion type and directly learns mixed distortion from distorted videos; on this basis, an automatic restoration method for mixed distortion is established based on unpaired distorted images and standard images. Summary of the invention

[0006] The present invention provides a representation learning and automatic recovery method for mixed distortion of video images in complex environments, which is used to solve the technical problem that the prior art lacks an effective method for directly learning mixed distortion from distorted videos and is difficult to correct multiple mixed distortions of video images in complex environments.

[0007] In order to solve the above technical problems, the technical solution proposed by the present invention is:

[0008] A representation learning method for mixed distortion of video images in complex environments, comprising the following steps:

[0009] Each frame of the decomposed video image in a complex environment is divided into multiple image blocks, and the image blocks are classified according to the temporal and spatial distances of each image block. The distortion categories of any two image blocks whose temporal and spatial distances are less than or equal to the temporal and spatial threshold are classified into the same category, and any two image blocks whose temporal and spatial distances are greater than the temporal and spatial threshold are classified into different categories.

[0010] Based on the divided categories, a set of homogeneous image pairs and a set of heterogeneous image pairs are constructed, wherein the homogeneous image object set includes multiple pairs of image pairs with the same distortion category, and the heterogeneous image pair set includes multiple pairs of image pairs with different categories;

[0011] A mixed distortion representation model is constructed with a contrastive learning network as a framework, an image block as an input, and a distortion representation vector of the image block as an output, and the mixed distortion representation model is trained with the similar image pair set and the heterogeneous image pair set, and the trained mixed distortion representation model is used to extract the distortion representation vector of the target image block.

[0012] Preferably, the cost function of the mixed distortion representation model is defined as maximizing the similarity of representation vectors of distorted similar image pairs and minimizing the similarity of representation vectors of heterogeneous image pairs; the mixed distortion representation model solves the gradient of the network weights through the cost function and iteratively updates the distortion representation learning model through back propagation; wherein the cost function is as follows:

[0013]

[0014] Among them, l + and l ++ is the representation vector of the same type of image pair, l - and l -- is the representation vector of the heterogeneous image pair, μ is a constant, and K is the number of homogeneous image pairs and heterogeneous image pairs. The mixed distortion representation model includes a feature extractor and a multi-layer perceptron, wherein the feature extractor is used to extract a high-dimensional distortion feature matrix of an input image block, and the multi-layer perceptron is used to perform dimensionality reduction processing on the high-dimensional distortion feature matrix to obtain a distortion representation vector.

[0015] Preferably, each frame of the image is segmented by random segmentation, and the number of image pairs in the same type of image pair set and the number of image pairs in the different type of image pair set are kept balanced;

[0016] The similar image pair set includes a similar image training set and a similar image test set, and the heterogeneous image pair set includes a heterogeneous image training set and a heterogeneous image test set; the image frames where the image blocks in the similar image training set and the similar image test set are located do not overlap in time, and the image frames where the image blocks in the heterogeneous image training set and the heterogeneous image test set are located do not overlap in time; the similar image training set and the heterogeneous image training set are used to iteratively train the mixed distortion representation model, and after the training is completed, the similar image test set and the heterogeneous image test set are used to test the trained mixed distortion representation model, and the test accuracy is counted until the test accuracy reaches a preset accuracy threshold, and the training is completed.

[0017] Preferably, the spatiotemporal threshold includes a time threshold and a distance threshold; the distortion categories of any two image blocks whose spatiotemporal distance is less than or equal to the spatiotemporal distance threshold are classified into the same category, and any two image blocks whose spatiotemporal distance is greater than the spatiotemporal distance threshold are classified into different categories, specifically:

[0018] For any pair of image patches Among them, t1 and t2 are the time sequence of the image frame where the image block is located; i and j are the spatial coordinates of the image block in the image frame:

[0019] If the time interval between t1 and t2 is less than or equal to the preset time threshold, and the distance between i and j is less than or equal to the preset distance threshold, then determine The distortion type is the same as

[0020] If the time interval between t1 and t2 is greater than the preset time threshold, or the distance between i and j is greater than the preset distance threshold, then determine The distortion types are different;

[0021] The time threshold is determined by the motion characteristics of the photographer and the object and the imaging conditions of the shooting environment. When the photographer is stationary and the imaging conditions of the shooting environment remain unchanged, the slower the motion speed of the object, the longer the time threshold, and the faster the motion speed of the object, the shorter the time threshold. The distance threshold is determined based on whether there is a local distortion difference in the image. When there is uneven illumination or uneven blur distortion, the smaller the distance threshold, the image blocks must be adjacent. If not, the larger the distance threshold, the m image blocks can be separated, where m≥1.

[0022] A method for automatically restoring mixed distortion of video images in a complex environment comprises the following steps:

[0023] The mixed distortion representation learning method of the complex environment video image is used to extract the distortion representation vector of each target image block in the target video frame decomposed in the target video;

[0024] Distortion restoration is performed on each target image block based on the distortion representation vector of each target image block, and the restored target image blocks are merged according to their spatiotemporal positions in the target video to obtain a restored video image.

[0025] Preferably, performing distortion restoration on each target image block based on the distortion representation vector of each target image block comprises the following steps:

[0026] For any distortion, the vector is q i For the target image block i, the following steps are performed:

[0027] Use the first generator to transform the target image block i into a mixed distorted image feature vector p i And the image feature vector r without mixing distortion i , p i -r i and q i The first discriminator is input to identify the authenticity, and the identification result of the first discriminator is fed back to the first generator, so that the first generator gradually generates the image feature vector r with the mixed distortion removed. i ;

[0028] The corrected image feature vector r i and the standard color distribution encoding vector c in the pre-stored memory bank are input into the domain adaptation module for domain adaptation alignment operation to minimize r i The distribution difference between the standard color distribution encoding vector c makes the image feature vector r i The color distribution characteristic components in are corrected, and the output vector r' i ;

[0029] The vector r' i The image block i' is input into the second generator, which restores it to a standardized image block i'; the standardized image block i' and the target image block i are simultaneously input into the second discriminator, which authenticates the authenticity of the image block; the authentication result of the second discriminator is fed back to the second generator, so that the second generator gradually optimizes the generated standardized image block i' to obtain the restored target image block.

[0030] Preferably, the training of the first generator and the first discriminator is guided by a first objective function, wherein the first objective function is as follows:

[0031] L1=E[log(D1(p i -r i ))]+E[log(1-D1(q i ))]

[0032] Among them, L1 is the first objective function value, D1 is the output value of the first discriminator, and E is the expected function.

[0033] The training of the second generator and the second discriminator is guided by a second objective function, wherein the second objective function is as follows:

[0034]

[0035] L2 is the second objective function value, D2 is the output value of the second discriminator, n is the number of distorted image blocks, Φ is the mapping function, H is the regenerated Hilbert space, and c is the standard color distribution encoding vector.

[0036] Preferably, the standard color distribution coding vector c is obtained by the following steps:

[0037] Select a group of standard image blocks with standard color distribution, use the color distribution encoder to encode each image block in the group, and obtain the color distribution encoding vector c0, c1, ..., c x , where c0, c1, …, c x are the color distribution coding vectors of the 1st, 2nd, ..., x+1th image blocks respectively; after weighted averaging the color distribution coding vectors of this group, a standard color distribution coding vector c is obtained, and c is stored in the memory bank.

[0038] Preferably, the method further comprises the following steps:

[0039] Sending each target image block before and after restoration into a color distribution encoder to retrain the color distribution encoder, and using a third objective function to guide the retraining of the color distribution encoder to update the standard distribution c of the memory bank;

[0040] Among them, the third objective function is:

[0041]

[0042] Where s is the standard image block, i' is the corrected output image block, n is the number of image blocks, and Φ2 is the color distribution encoder.

[0043] Preferably, the first generator, the first discriminator, the second generator, the second discriminator, the domain adaptation module and the color distribution encoder are trained synchronously, and a comprehensive objective function is constructed to guide the synchronous training to achieve global optimal training, wherein the comprehensive objective function takes the weighted sum minimization of the first objective function value, the second objective function value and the third objective function value as the optimization target.

[0044] The present invention has the following beneficial effects:

[0045] 1. The representation learning method of mixed distortion of video images in complex environments in the present invention establishes data sets with the same and different types of mixed distortion from videos, and adopts a method of learning distortion representation by contrastive learning, thereby realizing automatic modeling of mixed distortion directly from distorted video images, and on this basis constructs an automatic restoration method for mixed distortion of video images in complex environments to realize automatic restoration of mixed distortion.

[0046] 2. In the preferred embodiment, based on the representation learning method of mixed distortion of video images in complex environments, the present invention constructs a mixed distortion automatic correction model including four modules: mixed distortion representation learning, color coding, domain adaptation, and dual generative adversarial network. The model can effectively correct mixed distortion and improve the quality of video images.

[0047] In addition to the above-described purposes, features and advantages, the present invention has other purposes, features and advantages. The present invention will be further described in detail below with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] The drawings constituting a part of this application are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0049] Figure 1 A flowchart of an automatic restoration method for mixed distortion of video images in a complex environment provided by an embodiment of the present invention;

[0050] Figure 2 A schematic diagram of mixed distortion classification provided by an embodiment of the present invention;

[0051] Figure 3 A simplified structural diagram of a mixed distortion representation model provided by an embodiment of the present invention;

[0052] Figure 4 A first structural diagram of a hybrid distortion restoration model provided by an embodiment of the present invention;

[0053] Figure 5 A second structural diagram of the hybrid distortion restoration model provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0054] The embodiments of the present invention are described in detail below with reference to the accompanying drawings, but the present invention can be implemented in many different ways as defined and covered by the claims.

[0055] Embodiment 1:

[0056] This embodiment discloses a representation learning method for mixed distortion of video images in a complex environment, including the following steps:

[0057] Each frame of the decomposed video image in a complex environment is divided into multiple image blocks, and the image blocks are classified according to the spatiotemporal distance. Any two image blocks whose spatiotemporal distance is less than or equal to the spatiotemporal threshold are classified into the same category, and any two image blocks whose spatiotemporal distance is greater than the spatiotemporal threshold are classified into different categories.

[0058] Based on the divided categories, a set of homogeneous image pairs and a set of heterogeneous image pairs are constructed, wherein the homogeneous image object set includes multiple pairs of image pairs with the same distortion category, and the heterogeneous image pair set includes multiple pairs of image pairs with different categories;

[0059] A mixed distortion representation model is constructed with a contrastive learning network as a framework, an image block as an input, and a distortion representation vector of the image block as an output, and the mixed distortion representation model is trained with the similar image pair set and the heterogeneous image pair set, and the trained mixed distortion representation model is used to extract the distortion representation vector of the target image block.

[0060] In addition, in this embodiment, a method for automatically restoring mixed distortion of video images in a complex environment is also disclosed, comprising the following steps:

[0061] The mixed distortion representation learning method of the complex environment video image is used to extract the distortion representation vector of each target image block in the target video frame decomposed in the target video;

[0062] Distortion restoration is performed on each target image block based on the distortion representation vector of each target image block, and the restored target image blocks are merged according to their spatiotemporal positions in the target video to obtain a restored video image.

[0063] The representation learning method of mixed distortion of video images in complex environments in the present invention establishes data sets with the same and different types of mixed distortion from videos, and adopts a method of learning distortion representation by contrastive learning, thereby realizing automatic modeling of mixed distortion directly from distorted video images, and on this basis constructs an automatic restoration method of mixed distortion of video images in complex environments to realize automatic restoration of mixed distortion.

[0064] Embodiment 2:

[0065] Embodiment 2 is a preferred embodiment of Embodiment 1. The difference between Embodiment 2 and Embodiment 1 is that the specific steps of the method for automatically restoring mixed distortion of video images in a complex environment are introduced in detail:

[0066] like Figure 1 As shown, this embodiment discloses a method for automatically restoring mixed distortion of video images in a complex environment, including the following steps:

[0067] S1. Separate the mixed distorted video image into separate image frames, and construct a data set of the mixed distorted representation model based on the image frames:

[0068] S11. Shoot a video image from a complex environment, assuming that there is mixed distortion. Decompose the video image into a single image frame, and then decompose the image frame into a number of non-overlapping image blocks to create a data set of image blocks.

[0069] In this embodiment, the segmented image blocks can be randomly segmented or segmented according to other standard segmentation methods, such as average segmentation. The captured video should be a video captured in an actual environment, and the video covers all image frames with mixed distortion, and the video image is extracted at a certain interval, such as 10 frames.

[0070] S12, classifying the image blocks according to their temporal and spatial distances:

[0071] like Figure 2 As shown, for any pair of image blocks t1 and t2 are the time sequence of the image frame where the image block is located; i and j are the spatial coordinates of the image block in the image frame. Set the temporal and spatial distance threshold. If the time interval between t1 and t2 is less than or equal to the preset time threshold, and the spatial distance between i and j is less than or equal to the preset distance threshold, then it is considered If the time interval between t1 and t2 is greater than the preset time threshold, or the spatial distance between i and j is greater than the preset distance threshold, then it is considered The distortion type is different.

[0072] The time threshold setting depends on whether the video is motion or still. If the video is still, the time thresholds of t1 and t2 can be set larger, such as 10 seconds; if the video is motion, the time thresholds of t1 and t2 can be set smaller, such as 100 milliseconds.

[0073] The distance threshold between i and j is set based on whether there is a local distortion difference in the image, such as uneven illumination or uneven blur. If so, the distance threshold between i and j is smaller, such as 1 (adjacent image blocks). If not, the distance threshold between i and j is larger, such as m (adjacent m image blocks, m ≥ 1).

[0074] S13, establish a dataset P of the same distortion type for image pairs with the same distortion type. Establish a dataset D of different distortion types for image pairs with different distortion types. Divide the two datasets into a training set and a test set respectively: {P train ,P test} and {D train ,D test}.

[0075] The dataset includes several image pairs, each of which includes 2 image blocks, which are divided into two types: image pairs with the same distortion type and image pairs with different distortion types. Based on the distortion category, they can be divided into the same type of image pair set {P train ,P test} and heterogeneous image pairs {D train ,D test}, wherein the homogeneous image object set includes multiple pairs of image pairs with the same distortion category, and the heterogeneous image pair set includes multiple pairs of image pairs with different distortion categories;

[0076] In this embodiment, the image frames where the image blocks are located in the training set and the test set do not overlap in time; the number of image pairs in the data sets with the same distortion and different distortions is balanced;

[0077] S2. Construct a mixed distortion representation model, and train the mixed distortion representation model with the data in the data set to extract a mixed distortion representation vector.

[0078] Mixed distortion includes color distribution distortion and non-color distribution distortion. Non-color distribution distortion includes noise, blur, dust or water mist interference, etc. Color distribution distortion refers to the difference in color or gray distribution, contrast, and uneven illumination between the distorted image and the standard image.

[0079] The mixed distortion representation model automatically learns to extract non-color distribution distortion representations from image frames / image patches, such as blur, noise, dust, water mist interference, etc., but does not contain distortions that significantly change the image color distribution.

[0080] The mixed distortion representation model is based on spatial and temporal dimensions. Its characteristics are to extract image patches with the same and different distortion types from the video using image frames in the temporal dimension and spatial dimensions of the same image frame. Mixed distortion is automatically learned through contrastive learning without manual annotation or pairing of standard and distorted image patches.

[0081] The mixed distortion in this embodiment means that the learning model divides the same image frame into several image blocks, and the mixed distortion is the same for the adjacent or spatially close image blocks and the same image blocks in the image frames that are close in time. The mixed distortion is different for the image blocks in the same image frame that are spatially far away and the same image blocks in the image frames that were taken far away.

[0082] like Figure 3 As shown in Figure 1, the mixed distortion representation model includes a feature extraction module and a multi-layer perceptron module, which outputs a distortion representation vector. The training of the representation model uses the dataset P train and D train , the optimization goal is P trainThe representation vectors of each image patch pair are similar, and D train The representation vector of each image block pair is different. That is, the training goal is to minimize the similarity of the distortion representation vectors of image blocks with different distortion types and to maximize the similarity of the distortion representation vectors of image blocks with the same distortion type.

[0083] Among them, the feature extractor is implemented based on convolutional neural networks, such as Resnet, Efficient, GoogleNet, VGG and other network structures. The multi-layer perceptron is a fully connected neural network structure that converts features into vectors; it can also be replaced by other methods that reduce feature maps to vectors, such as global average pooling and global maximum pooling.

[0084] Among them, the training strategy of the mixed distortion representation model is as follows:

[0085] Let the representation vector of the currently processed image block be x, and the representation vector of another image block with the same distortion be x + , and the representation vector of another image block with different distortion is x - The training cost function is defined as maximizing the distortion representation vector x + and x, while minimizing the representation vectors x and x - The gradient of the network weights is solved through the cost function, and the mixed distortion representation model is iteratively updated through back propagation.

[0086] After training, use P test and D test Test the mixed distortion representation model. When training the mixed distortion restoration model, the distortion representation model keeps the weights unchanged to simplify the training process.

[0087] S3. Build a mixed distortion restoration model based on the trained mixed distortion representation model, and use the built mixed distortion restoration model to restore the image frame:

[0088] like Figure 4 As shown in FIG, the mixed distortion restoration model includes a mixed distortion representation learning model module, a color distribution domain adaptation module, and a dual generative adversarial network module. Specifically, Figure 5 As shown, the color distribution domain adaptation module includes a color distribution encoder, a memory bank, and a domain adaptation module; the dual generative adversarial network module includes two generators and two discriminators.

[0089] The color distribution encoder is composed of a multi-layer convolutional neural network and a multi-layer perceptron, which is used to select a set of standard image blocks with color distribution standards, and use the color distribution encoder to encode the image blocks to obtain a set of color distribution encoding vectors c0, c1, ..., c x, after weighted averaging this set of color distribution coding vectors, a standard color distribution coding vector c is obtained, and c is stored in the memory bank. The standard image block and the distorted image block do not need to be paired, and can come from different video images.

[0090] In this embodiment, the memory bank is a storage unit for storing the standard color distribution coding vector c.

[0091] The standard image block refers to an image block obtained by segmenting an image frame extracted from a video image taken under normal imaging conditions; normal imaging conditions refer to the absence of mixed distortion, or the current distortion is not of concern to the video, such as low noise and does not affect the task of use. The image frame and the image block are randomly extracted.

[0092] S31, input any distorted image block into the distortion learning model, and extract the distorted representation vector q i At the same time, the first generator (hereinafter referred to as generator 1) is used to convert the distorted image block into an image feature vector q including mixed distortion i And the image feature vector r without mixing distortion i . i -r i and q i The image is input into the first discriminator (hereinafter referred to as discriminator 1), which identifies the authenticity and feeds the identification result of the discriminator back to generator 1, so that the generator gradually generates the image feature vector r with the mixed distortion removed. i .

[0093] Among them, the generator 1 and the discriminator 1 can be composed of multi-layer convolutional neural networks.

[0094] S32, the vector r after the distortion correction i The standard color distribution encoding vector c in the memory bank is input into the domain adaptation module for domain adaptation alignment operation to minimize r i The difference between the distribution of r and c features makes r i The color distribution characteristic components in are corrected, and the output vector r' i .

[0095] Among them, the domain adaptation operation can be achieved through methods such as the maximum mean difference method and the feature adversarial method.

[0096] S33, vector r' i Input into the second generator (hereinafter referred to as generator 2), which restores it to a standardized image block i'; i' and i are simultaneously input into the second discriminator (hereinafter referred to as discriminator 2), and discriminator 2 identifies the authenticity. The objective function of this stage is to make discriminator 2 believe that the standardized image block i' with the mixed distortion removed is similar to the content of the input image i.

[0097] Among them, the generator 2 can be composed of multiple layers of convolutional layers and upsampling layers; the discriminator 2 is composed of networks such as Densenet and Efficient.

[0098] S34, after obtaining the corrected output image block i', use the color distribution encoder to encode it again, and feed the encoded color distribution encoding vector back to the memory bank, and iteratively update the vector c stored therein.

[0099] Among them, the training strategy of the mixed distortion restoration model is as follows:

[0100] The mixed distortion representation learning model is first trained separately and then used as a fixed untrained module of the mixed distortion restoration model;

[0101] Randomly take out the distorted image block from the training set and send it to the distortion representation model to extract the distortion representation vector; Generator 1 obtains the mixed distorted image feature vector p i And the image feature vector r without mixing distortion i ; Discriminator 1 will p i -r i and q i Identification requires that the output of generator 1 is p i -r i and q i The difference is small, which is used as the constraint condition of generator 1 and defined as objective function 1.

[0102] The randomly selected standard image block enters the color distribution encoder and updates the standard distribution c of the memory bank. i Perform domain adaptation operation with c to get r' i The restored image block is generated by the generator 2. The discriminator 2 discriminates the difference between the distorted image block and the restored image block as the objective function 2.

[0103] Based on objective function 2, the training of discriminator 2, generator 2, generator 1 and distribution domain adaptation module is guided.

[0104] Based on objective function 1, the training of generator 1 and discriminator 1 is guided.

[0105] Objective function 1 can be defined as shown in formula 1; objective function 2 can be defined as shown in formula 2:

[0106] L1=E[log(D1(p i -r i ))]+E[log(1-D1(q i ))] Formula 1

[0107]

[0108] Where D1 is discriminator 1, D2 is discriminator 2, n is the number of distorted image blocks, Φ is the mapping function, H is the regenerated Hilbert space, and c is the standard color distribution encoding vector.

[0109] The restored image block and the standard image block are sent to the color distribution encoder, and the color distribution difference between the two is minimized as the objective function 3.

[0110] Based on objective function 3, the training of the color distribution encoder is guided.

[0111] Preferably, the objective function 3 can be defined as:

[0112]

[0113] Where s is the standard image block, i' is the corrected output image block, n is the number of image blocks, and Φ2 is the color distribution encoder.

[0114] When training the hybrid distortion restoration model, the weighted sum of the three objective functions is minimized to achieve the optimization of the hybrid model. The hybrid distortion restoration model is trained with the standard image and the distorted image, and the two do not need to be paired. The hybrid distortion representation learning model extracts the hybrid distortion representation vector of the distorted image; the color distribution domain adaptation module corrects the color distribution difference of the distorted image; the generator 2 generates a restored quasi-image, and the discriminator 2 analyzes the restoration quality of the image.

[0115] In this embodiment, generator 1 corrects the mixed distortion of non-color distribution; then a domain adaptation operation is performed to correct the distortion of color distribution; generator 2 is used to generate a standard image, and the corrected standard image is similar to the original distorted image and is identified as true by discriminator 2.

[0116] S4, merging the restored image frames together according to the time sequence relationship to obtain a restored video image;

[0117] After completing the training of the mixed distortion restoration model, the distorted image blocks are input into generator 1 and then into the domain adaptation module to complete the automatic correction of the mixed distortion. The restored image blocks are generated using generator 2. The image blocks from the same frame are restored and spliced ​​to obtain a restored single image. The single images are connected in time sequence to obtain a restored video.

[0118] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A representation learning method for mixed distortion of video images in complex environments, characterized by: The following steps are involved: Each frame of the decomposed video image in a complex environment is divided into multiple image blocks, and the image blocks are classified according to the temporal and spatial distances of each image block. The distortion categories of any two image blocks whose temporal and spatial distances are less than or equal to the temporal and spatial threshold are classified into the same category, and any two image blocks whose temporal and spatial distances are greater than the temporal and spatial threshold are classified into different categories. Based on the divided categories, a set of homogeneous image pairs and a set of heterogeneous image pairs are constructed, wherein the homogeneous image object set includes multiple pairs of image pairs with the same distortion category, and the heterogeneous image pair set includes multiple pairs of image pairs with different categories; A mixed distortion representation model is constructed with a contrastive learning network as a framework, an image block as an input, and a distortion representation vector of the image block as an output, and the mixed distortion representation model is trained with the similar image pair set and the heterogeneous image pair set, and the trained mixed distortion representation model is used to extract the distortion representation vector of the target image block.

2. The representation learning method of mixed distortion of video images in complex environments according to claim 1 is characterized in that: The cost function of the mixed distortion representation model is defined as maximizing the similarity of the representation vectors of the distorted similar image pairs, while minimizing the similarity of the representation vectors of the heterogeneous image pairs; the mixed distortion representation model solves the gradient of the network weights through the cost function, and iteratively updates the distortion representation learning model through back propagation; wherein the cost function is as follows: Among them, l + and l ++ is the representation vector of the same type of image pair, l - and l -- is the representation vector of heterogeneous image pairs, μ is a constant, and K is the number of homogeneous image pairs and heterogeneous image pairs; The mixed distortion representation model includes a feature extractor and a multi-layer perceptron. The feature extractor is used to extract a high-dimensional distortion feature matrix of an input image block, and the multi-layer perceptron is used to perform dimensionality reduction processing on the high-dimensional distortion feature matrix to obtain a distortion representation vector.

3. The representation learning method of mixed distortion of video images in complex environments according to claim 1 is characterized in that: Each frame of the image is segmented randomly, and the number of image pairs in the same image pair set and the number of image pairs in the different image pair sets are balanced; The similar image pair set includes a similar image training set and a similar image test set, and the heterogeneous image pair set includes a heterogeneous image training set and a heterogeneous image test set; the image frames where the image blocks in the similar image training set and the similar image test set are located do not overlap in time, and the image frames where the image blocks in the heterogeneous image training set and the heterogeneous image test set are located do not overlap in time; the similar image training set and the heterogeneous image training set are used to iteratively train the mixed distortion representation model, and after the training is completed, the similar image test set and the heterogeneous image test set are used to test the trained mixed distortion representation model, and the test accuracy is counted until the test accuracy reaches a preset accuracy threshold, and the training is completed.

4. The representation learning method of mixed distortion of video images in complex environments according to claim 2 is characterized in that: The spatiotemporal threshold includes a time threshold and a distance threshold; the distortion categories of any two image blocks whose spatiotemporal distance is less than or equal to the spatiotemporal distance threshold are classified into the same category, and any two image blocks whose spatiotemporal distance is greater than the spatiotemporal distance threshold are classified into different categories, specifically: For any pair of image patches Among them, t1 and t2 are the time sequence of the image frame where the image block is located; i and j are the spatial coordinates of the image block in the image frame: If the time interval between t1 and t2 is less than or equal to the preset time threshold, and the spatial distance between i and j is less than or equal to the preset distance threshold, then determine The distortion type is the same as If the time interval between t1 and t2 is greater than the preset time threshold, or the spatial distance between i and j is greater than the preset distance threshold, then the judgment The distortion types are different; The time threshold is determined by the motion characteristics of the photographer and the object and the imaging conditions of the shooting environment. When the photographer is stationary and the imaging conditions of the shooting environment remain unchanged, the slower the motion speed of the object, the longer the time threshold, and the faster the motion speed of the object, the shorter the time threshold. The distance threshold is determined based on whether there is a local distortion difference in the image. When there is uneven illumination or uneven blur distortion, the smaller the distance threshold, the image blocks must be adjacent. If not, the larger the distance threshold, the m image blocks can be separated, where m≥1.

5. A method for automatically restoring mixed distortion of video images in complex environments, characterized in that: The following steps are involved: The method for representing mixed distortion of video images in a complex environment as described in any one of claims 1 to 4 is used to extract the distortion representation vector of each target image block in the target video frame decomposed in the target video; Distortion restoration is performed on each target image block based on the distortion representation vector of each target image block, and the restored target image blocks are merged according to their spatiotemporal positions in the target video to obtain a restored video image.

6. The automatic restoration method of mixed distortion of video images in complex environments according to claim 5 is characterized in that: Distortion restoration is performed on each target image block based on the distortion representation vector of each target image block, including the following steps: For any distortion, the vector is q i For the target image block i, the following steps are performed: Use the first generator to transform the target image block i into an image feature vector p including mixed distortion i And the image feature vector r without mixing distortion i , p i -r i and q i The first discriminator is input to identify the authenticity, and the identification result of the first discriminator is fed back to the first generator, so that the first generator gradually generates the image feature vector r with the mixed distortion removed. i ; The corrected image feature vector r i and the standard color distribution encoding vector c in the pre-stored memory bank are input into the domain adaptation module for domain adaptation alignment operation to minimize r i The distribution difference between the standard color distribution encoding vector c makes the image feature vector r i The color distribution characteristic components in are corrected, and the output vector r' i ; The vector r' i The image block i' is input into the second generator, which restores it to a standardized image block i'; the standardized image block i' and the target image block i are simultaneously input into the second discriminator, which authenticates the authenticity, and feeds the authentication result of the second discriminator back to the second generator, so that the second generator gradually optimizes and generates the standardized image block i', and obtains the restored target image block.

7. The automatic restoration method of mixed distortion of video images in complex environments according to claim 6 is characterized in that: The training of the first generator and the first discriminator is guided by a first objective function, wherein the first objective function is as follows: L1=E[log(D1(p i -r i ))]+E[log(1-D1(q i ))] Wherein, L1 is the first objective function value, D1 is the output value of the first discriminator, and E is the expected function; The training of the second generator and the second discriminator is guided by a second objective function, wherein the second objective function is as follows: L2 is the second objective function value, D2 is the output value of the second discriminator, n is the number of distorted image blocks, Φ is the mapping function, H is the regenerated Hilbert space, and c is the standard color distribution encoding vector.

8. The automatic restoration method of mixed distortion of video images in complex environments according to claim 7 is characterized in that: The standard color distribution coding vector c is obtained by the following steps: Select a group of standard image blocks with standard color distribution, use the color distribution encoder to encode each image block in the group, and obtain the color distribution encoding vector c0, c1, ..., c x , where c0, c1, …, c x are the color distribution coding vectors of the 1st, 2nd, ..., x+1th image blocks respectively; after weighted averaging the color distribution coding vectors of this group, a standard color distribution coding vector c is obtained, and c is stored in the memory bank.

9. The automatic restoration method of mixed distortion of video images in complex environments according to claim 8, characterized in that: The following steps are also included: Sending each target image block before and after restoration into a color distribution encoder to retrain the color distribution encoder, and using a third objective function to guide the retraining of the color distribution encoder to update the standard distribution c of the memory bank; Among them, the third objective function is: Where s is the standard image block, i' is the corrected output image block, n is the number of image blocks, and Φ2 is the color distribution encoder.

10. The automatic restoration method of mixed distortion of video images in complex environments according to claim 9, characterized in that: The first generator, the first discriminator, the second generator, the second discriminator, the domain adaptation module and the color distribution encoder are synchronously trained, and a comprehensive objective function is constructed to guide the synchronous training to achieve global optimal training, wherein the comprehensive objective function takes the weighted sum minimization of the first objective function value, the second objective function value and the third objective function value as the optimization target.

Images