A cross-modal image restoration method and device based on attention mechanism

Through the cross-modal image repair method based on attention mechanism, the AGVI model combined with tactile signals can accurately locate and repair the image defect area, solving the problems of distortion and defects during image transmission, achieving high-quality image repair and improving Accuracy of user experience and image repair.

CN114677311BActive Publication Date: 2024-11-12NANJING UNIV OF POSTS & TELECOMM
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202210205553.5
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-03-03
Publication Date
2024-11-12
Estimated Expiration
2042-03-03

AI Technical Summary

Technical Problem

In the prior art, due to image distortion and defects caused by noise and transmission losses during image transmission, it is difficult to achieve high-quality multimodal signal transmission, especially in application scenarios such as virtual games and distance education, and the user experience is affected.

Method used

Using a cross-modal image repair method based on attention mechanism, the AGVI model combines tactile signals to accurately locate the image defect area, and using the learned feature extraction, diverting feature attention, related embedding learning and cross-modal image repair modules to achieve high Quality image repair to improve the accuracy and completeness of image repair.

Benefits of technology

Without increasing the complexity of the model, high-quality and fine-grained image repair is achieved, which improves the terminal business level and user immersive experience, and ensures the accuracy and detailed capture capabilities of image repair.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114677311B_ABST
    Figure CN114677311B_ABST
Patent Text Reader

Abstract

The present invention discloses a cross-modal image restoration method and device based on an attention mechanism, the method comprising: selecting a multimodal data set, including defective image data, real image data and tactile signals, and dividing the data set into a training set and a test set; designing a cross-modal image restoration AGVI model based on an attention mechanism, the model comprising four modules: learnable feature extraction, feature attention transfer, related embedding learning and cross-modal image restoration; using the training set to train the cross-modal image restoration AGVI model to obtain the optimal cross-modal image restoration AGVI model structure and network parameters; using the tactile signals and defective image data in the test set, based on the optimal cross-modal image restoration AGVI model, cross-modal restoration of the image. The present invention introduces an attention mechanism to accurately locate the defective area of ​​the image, and uses the key information in the tactile signal to repair the prediction and fill the area, thereby achieving high-quality and fine-grained restoration of the image.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to a cross-modal image restoration method and device based on an attention mechanism, and belongs to the technical field of image restoration. Background Art

[0002] In recent years, with the rapid development of wireless communication and multimedia technology, multimedia audio-visual services such as ultra-high-definition video and live webcast have basically met the audio-visual needs of users. In order to further pursue more extreme interactive feelings and scene experiences, more and more users are exploring sensory experiences other than audio-visual, such as touch, smell, taste, etc. With the rapid development of tactile Internet under 5G communication, the integration of emerging tactile services, namely multimodal services, into traditional audio-visual services will provide users with immersive experience in various application scenarios such as virtual games, distance education, rehabilitation medicine, etc. In order to support such multimodal services composed of audio, video, and touch, it is crucial to achieve efficient transmission of heterogeneous code streams. However, in actual transmission, due to the influence of factors such as noise and transmission loss, the receiving end often has serious problems of image data distortion or loss, and the quality of multimodal signals is difficult to guarantee.

[0003] At present, the research on image distortion and partial defect problems during transmission is mainly considered from the perspective of image restoration. The mainstream image restoration methods mainly include: 1) based on signal processing; 2) based on loss function; 3) based on deep model.

[0004] First, typical signal processing techniques, such as low-rank matrix and sparse representation, are used for image restoration. The main idea is to represent the original data matrix as a low-rank part or a sparse part, and analyze and apply it. For example, researchers proposed a sparse optimization method called "l 0 "TV-PADMM", this method transforms image distortion into an equivalent biconvex mathematical programming (MPEC) problem with equilibrium constraints, and uses sparse representation to find specific signals with similar structures to estimate the defective content of the image. In order to further utilize the local and non-local characteristics of the image, some scholars applied transform learning to adaptive sparse representation and proposed an image restoration scheme based on joint low-rank regularized transform learning. This type of method is mainly based on signal processing technology for image restoration, with simple principles and easy operation, but the image restoration effect is often poor and the quality is low.

[0005] Secondly, the image restoration process is standardized by strengthening the constraints of the loss function. For example, some scholars make full use of the prior knowledge of visual images to construct a hybrid regularized loss function, and through the regularization terms in the spatial and spectral domains, find the error fitting of all image frames to achieve high-quality image restoration. In order to further enhance the controllability of image denoising, some scholars have introduced diverse objective functions to associate input noise with image semantics, minimize the restoration distance between images, and allow users to manipulate the output image by manually adjusting the noise. This type of method strengthens the image restoration process by constraining the loss function, which is beneficial to improving the quality of the restored image, but there are still defects in capturing the details of image restoration.

[0006] Finally, the superior performance of deep learning is used to achieve accurate image restoration. With the rapid development of artificial intelligence technology, deep learning has become the mainstream technology for current image restoration due to its powerful learning ability. For example, researchers designed a quality enhancement network based on a deep model that uses a residual network and recursive learning to significantly reduce image artifacts with similar frequency characteristics and remove distortion and blur caused by noise. In addition, some people model the denoising problem as a function optimization process with design costs. Based on a large number of defective images and a small number of labeled training samples, the model parameters are estimated through semi-supervised learning, and the noise is accurately and effectively removed to achieve high-quality image restoration. This type of method often achieves good restoration results, but faces serious problems such as high model complexity and large amount of calculation. Summary of the invention

[0007] The technical problem to be solved by the present invention is to overcome the deficiencies of the prior art and provide a cross-modal image restoration method and device based on an attention mechanism. The present invention uses tactile signals to cross-modally restore visual images; specifically, the present invention first selects a standard multi-modal data set model for training and testing, and establishes a cross-modal image restoration AGVI (Attention-Guided Cross-Modal Visual Image Inpainting) model based on tactile signals to achieve cross-modal image restoration. The present invention can accurately locate the image defect area, and restore high-quality, fine-grained images with low model complexity, thereby improving the image restoration quality and ensuring the terminal service level and user immersive experience.

[0008] The present invention specifically adopts the following technical solutions to solve the above technical problems:

[0009] A cross-modal image restoration method based on an attention mechanism comprises the following steps:

[0010] Step 1: selecting a multimodal dataset, wherein the multimodal dataset includes defective image data, real image data, and tactile signals, and dividing the multimodal dataset into a training set and a test set;

[0011] Step 2: Design a cross-modal image restoration AGVI model based on attention mechanism, which includes:

[0012] A learnable feature extraction module is used to extract features of tactile signals, defective image data, and real image data, and participate in subsequent end-to-end model training;

[0013] The transfer feature attention module is used to introduce the attention mechanism, locate the defective area of ​​the image, and obtain the transfer features that represent the defective area;

[0014] The correlation embedding learning module is used to construct the correlation embedding learning space in combination with the real label information. While minimizing the semantic association objective function to complete the semantic feature learning task, the classification metric objective function based on cross entropy is used to minimize the difference between the predicted label and the real label, and the total objective function of the semantic similarity learning between different modalities in the final correlation embedding learning stage is obtained to mine the tactile features that are most relevant to the image defect area.

[0015] The cross-modal image restoration module is used to combine the inter-pixel perception constraint loss function, the appearance constraint loss function and the adversarial loss function, and use the mined tactile features most relevant to the image defect area to perform cross-modal restoration on the defective image data;

[0016] Step 3: Use the training set to train the cross-modal image restoration AGVI model to obtain the optimal cross-modal image restoration AGVI model structure and network parameters;

[0017] Step 4: Use the tactile signals and defective image data in the test set to perform cross-modal image restoration based on the optimal cross-modal image restoration AGVI model.

[0018] Furthermore, as a preferred technical solution of the present invention, the step 1 of selecting a multimodal data set specifically includes:

[0019] Step (1-1), selecting three different modal data, namely, defective image data I, real image data R and tactile signal H, to form a multimodal data set D; wherein the real image data is the original color image signal, the tactile signal is the tactile power spectrum density obtained by preprocessing the original tactile signal, and the defective image data is an image with a defect rate of λ obtained by defect preprocessing the real image data, and the value range of λ is between 0 and 1;

[0020] Step (1-2): for data of different modes in the multimodal dataset D, their true label information Y is counted, that is, one-hot encoding is used to label each data with the category label to which the content information represented belongs;

[0021] Step (1-3), randomly select a proportion of data α from the multimodal dataset D as the training set D tr , the remaining 1-α proportion of data is used as the test set D te , the value range of α is between 0 and 1.

[0022] Further, as a preferred technical solution of the present invention, the feature extraction module in step 2 can be learned to extract features of tactile signals, defective image data and real image data, specifically including:

[0023] For the tactile signal H, the gated recurrent unit GRU and a 3-layer fully connected network are used as the tactile mapping network to obtain the tactile feature h and the tactile feature prediction label y. (h) ;

[0024] For both defective image data I and real image data R, an image mapping network composed of a deep convolutional neural network is used to extract shallow defective image features i and real image features r; the specific process is as follows:

[0025]

[0026]

[0027]

[0028] Among them, h, i and r are tactile features, defective image features and real image features respectively, and the dimensions of h, i and r are and θ h and θ i They are tactile mapping network F h (H; θ h ) and the image mapping network F i (I / R;θ i ) parameter set.

[0029] Further, as a preferred technical solution of the present invention, the attention transfer module in step 2 obtains the transfer features representing the defect area, specifically including:

[0030] Step A: For the defective image feature i and the real image feature r, each feature value is defined as a feature unit, that is, and i k and r l They represent the kth eigenvalue of the defective image feature i and the lth eigenvalue of the real image feature r, respectively. and denote the dimensions of the defective image feature i and the true image feature r respectively; then, the normalized inner product is used to calculate the dimension of each i in the defective image feature i. k And each r in the real image feature r l The cosine similarity between all feature units of the two features is calculated and expressed as follows:

[0031]

[0032] Among them, c k,l represents the cosine similarity matrix, ||·|| represents the modulo operation, and <·,·> represents the normalized inner product operation;

[0033] Step B: Transfer the most relevant part of each feature unit representing the defective area in the defective image features from the real image features, that is, the cosine similarity matrix c k,l Take the maximum value of each column of , and this process is expressed as:

[0034]

[0035] Among them, a k is the attention transfer index, which represents the feature metric most relevant to the k-th position of the defective image feature i in the real image feature r;

[0036] Step C: Based on the attention transfer index, a feature selection operation is performed on the real image feature r, in order to obtain the transfer characteristic t representing the image defect area from the real image feature. The process is expressed as:

[0037] t k =ra k ,

[0038] Among them, t k Indicates the selection of the ath feature in the real image r k The feature value of the kth position is transferred to obtain the feature value of the kth position in the transferred feature t, and then it is classified through the sigmoid layer to obtain the predicted label y of the transferred feature (t) .

[0039] Further, as a preferred technical solution of the present invention, the relevant embedding learning module in step 2 mines the tactile features most relevant to the image defect area among the tactile features, specifically including:

[0040] Step A: Use the real label information Y = {y}, y∈{1,2,…,C} to construct the relevant embedding learning space and complete the semantic feature learning task. This process is mainly achieved by minimizing the semantic association objective function L rem accomplish:

[0041]

[0042]

[0043] δ pq =h p *t q ,

[0044] Among them, y represents the true label, C represents the total number of training data categories, N represents the total number of training data, and s pq is the category association factor, δ pq is the characteristic correlation factor, and are the predicted labels of the pth tactile feature and the qth transfer feature, respectively, (·) T represents the transpose operation, h p and t q denote the pth tactile feature and the qth transfer feature respectively; the semantic association objective function ensures that in the relevant embedding learning space, the transfer features with the same semantics can assist the tactile sense in semantic feature learning, that is, to extract the tactile features with the highest correlation with the image defect area from the tactile features;

[0045] Step B: In order to further enhance the feature discrimination capability, a cross-entropy-based loss function is used to minimize the difference between the predicted label and the true label to complete the semantic classification. This process is expressed as:

[0046]

[0047]

[0048] Among them, L 1 and L 2 Denote the classification metric objective functions of transfer features and tactile features, y p and denote the true label and predicted label of the pth tactile feature, y q and Represent the true label and predicted label of the qth transfer feature respectively;

[0049] Step C: Combining the above steps A and B, the overall objective function of semantic similarity learning between different modalities in the relevant embedding learning stage is obtained, which is expressed as:

[0050] L sim =L rem +α 1 L 1 +α 2 L 2 ,

[0051] Among them, Lsim is the final semantic similarity loss function, α 1 and α 2 is a hyperparameter, L rem Used to indicate the semantic association between two modalities.

[0052] Further, as a preferred technical solution of the present invention, the cross-modal image restoration module in step 2 uses the mined tactile features most relevant to the image defective area to perform cross-modal restoration on the defective image data, specifically including:

[0053] Step A: Use the decoder network to implement cross-modal image restoration; specifically, add the tactile feature h and the defective image feature i to obtain the restored image feature Then the image features will be repaired Input decoder De to get the repaired image Right now Using real image data, the restoration process is constrained from two perspectives: appearance features and perceptual characteristics, so that the restored image As similar as possible to the real image R, this process is expressed as:

[0054]

[0055]

[0056]

[0057] Among them, θ de is the network parameter set of the decoder De, L a and L p They are appearance constraint loss function and perception constraint loss function respectively. is the inpainted image distribution function expectations, It is a feature extraction network similar to VGG,||·|| 1 To take the L1 norm operation;

[0058] Step B: constrain the distribution structure of the restored image in step A above; specifically, use the adversarial loss function to learn the distribution of real image data, and define the adversarial loss function L adv It is expressed as:

[0059]

[0060] Among them, L adv is the adversarial loss function of the discriminator D, θ d is the network parameter set of the discriminator D, and They are the true image distribution function Pdata(R) And the inpainted image distribution function The expectation of D(R;θ d )and are the probabilities that the discriminator D identifies the real image and the restored image as true;

[0061] Step C: Combining the appearance constraint loss function L in step A and step B a , Perceptual Constraint Loss Function L p And the adversarial loss function L adv , the loss function that the decoder De finally needs to minimize is:

[0062] L imp =L a +β 1 L p +β 2 L adv ,

[0063] Among them, L imp is the final restoration loss function, β 1 and β 2 are all hyperparameters.

[0064] Further, as a preferred technical solution of the present invention, the step 3 uses the training set to train the cross-modal image restoration AGVI model, specifically including:

[0065] Step (3-1), based on the repaired image Combine the training set divided in step 1 and the true label information in steps (2-3) into a standardized input training dataset D tr :

[0066] D tr ={(I n ,R n ,H n ,y n ),n=1,2,…,N},

[0067] Among them, y n is the defective image data I of the nth group participating in the training n , real image data R n and tactile signal H n The true label it belongs to, N is the total capacity of training data;

[0068] Step (3-2), initialize the network parameter set of the AGVI model, which includes θ h ,θ i ,θ de ,θ d , these parameters are initialized to the standard normal distribution; where θh and θ i are the parameter sets of the tactile mapping network and the image mapping network respectively; θ de is the network parameter set of the decoder De, θ d is the network parameter set of the discriminator D;

[0069] Step (3-3), set the total number of iterations to G, and use g to record the specific number of iterations;

[0070] Step (3-4) uses the stochastic gradient descent method to train the cross-modal image restoration AGVI model. The specific process is as follows:

[0071] Step 1: Set the hyperparameter α 1 ,α 2 ,β 1 ,β 2 , the learning rates μ for the tactile mapping network and the image mapping network 1 , the learning rate μ of the decoder and discriminator 2 ;

[0072] Step 2: Calculate the output of each network in the cross-modal image restoration AGVI model:

[0073] h=F h (H; θ h );i=F i (I; θ i ); r = F i (R; θ i ),

[0074]

[0075] Step 3: Start iteration; based on the gradient descent method and Adam optimizer, update the parameter set of each network from the negative gradient direction of the target:

[0076]

[0077]

[0078]

[0079]

[0080] Among them, L sim (*) is the final semantic similarity loss function L sim , L imp (*) is the final restoration loss function L imp ; and The tactile mapping network F after the g+1th and gth iterations respectivelyh , image mapping network F i , the network parameter set of the decoder network De and the discriminator network D; is the derivative;

[0081] Step ④: If g<G, jump to step ③ in step (3-4), g=g+1, and continue to the next iteration; otherwise, terminate the iteration;

[0082] Step (3-5), after G rounds of iterations, finally output the optimal cross-modal image restoration AGVI model structure and network parameters.

[0083] Further, as a preferred technical solution of the present invention, the cross-modal image restoration is performed based on the optimal cross-modal image restoration AGVI model in step 4, specifically including:

[0084] Step (4-1), the test set D divided in step 1 te for:

[0085] D te ={(I' j ,H' j ),j=1,2,…,F},

[0086] Among them, I' j and H' j is the jth pair of defective image data and tactile signal, I' j and H' j For model testing, F is the total amount of test data;

[0087] Step (4-2), test set D te The data in are input in pairs into the optimal cross-modal image restoration AGVI model trained in step 3, and the output is the restored image.

[0088] The present invention also proposes a cross-modal image restoration device based on an attention mechanism, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program implements the cross-modal image restoration method based on an attention mechanism when loaded into the processor.

[0089] The present invention adopts the above technical solution to produce the following technical effects:

[0090] (1) In order to accurately locate the defective area of ​​the image, the present invention introduces an attention mechanism to allocate weights, focusing on the defective area of ​​the image to fully obtain the transfer features that characterize the defective area, thereby improving the accuracy and completeness of the model restoration.

[0091] (2) The model of the present invention introduces common semantics and label information when measuring the similarity of different modalities. Through the dual constraints of semantic association and category measurement objective function, it continuously narrows the gap between transfer features and tactile features, and extracts some tactile features with the highest correlation with the image defective area for cross-modal repair of defective image data.

[0092] (3) The present invention uses a decoder network structure, under the constraints of semantics and supervision information, through the perceptual constraint loss function and appearance constraint loss function between pixels, and the adversarial loss function based on image distribution, to achieve high-quality, fine-grained comprehensive restoration of defective images. At the same time, without increasing the complexity of the model, the image quality is improved from the semantic and distribution perspectives. BRIEF DESCRIPTION OF THE DRAWINGS

[0093] Figure 1 It is a schematic diagram of the process of the present invention.

[0094] Figure 2 Schematic diagram of the structure of the cross-modal image restoration model based on the attention mechanism of the present invention.

[0095] Figure 3 It is a structural framework diagram of the device of the present invention.

[0096] Figure 4 The figure is a comparison result diagram between the method of the present invention and the existing method. DETAILED DESCRIPTION

[0097] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0098] It is necessary to develop an efficient and accurate cross-modal image restoration method that can accurately locate the defective area and mine the effective information in the tactile signal to achieve high-quality image data restoration. In recent years, the encoder-decoder network has achieved good success in the field of image restoration, and the attention mechanism also provides a simple and efficient way to accurately locate the defective area of ​​the image. Therefore, the present invention proposes a cross-modal image restoration method based on the attention mechanism. Based on the selective attention ability of the attention mechanism, it can focus on and accurately locate the defective area of ​​the image and obtain the transfer features of the area; the relevant embedding learning module combines the real label information to construct the relevant embedding learning space, and while minimizing the semantic association objective function to complete the semantic feature learning task, it adopts the classification measurement objective function based on cross entropy to minimize the difference between the predicted label and the real label, and obtains the total objective function of semantic similarity learning between different modalities in the final relevant embedding learning stage, and mines the tactile features that are most relevant to the image defective area in the tactile features; the generative model based on the autoencoder uses the mined tactile features that are most relevant to the image defective area under the constraints of the perceptual constraint loss function, appearance constraint loss function and adversarial loss function that fit the difference between pixels and distribution differences to realize cross-modal repair of defective image data, and improves the image quality in terms of semantics and distribution without increasing the complexity of the model.

[0099] Specifically, Figure 1 As shown, the present invention relates to a cross-modal image restoration method based on an attention mechanism, and the method specifically comprises the following steps:

[0100] Step 1: Select a multimodal dataset, which includes defective image data, real image data, and tactile signals, and divide the multimodal dataset into a training set and a test set, as follows:

[0101] Step (1-1), select three different modal data, namely, defective image data I, real image data R and tactile signal H, to form a multimodal data set D; wherein the real image data is the original color image signal, the tactile signal is the tactile power spectrum density obtained by preprocessing the original tactile signal, and the defective image data is the image with a defect rate of λ obtained by defect preprocessing the real image data, where λ is 40%.

[0102] Step (1-2): For data of different modes in the multimodal dataset D, their true label information Y is counted, that is, one-hot encoding is used to label each data with the category label to which the content information represented belongs.

[0103] Step (1-3), randomly select a proportion of data α from the multimodal dataset D as the training set D tr , the remaining 1-α proportion of data is used as the test set D te , here we take α=0.8.

[0104] Step 2: Design a cross-modal image restoration AGVI model based on attention mechanism, such as Figure 2 As shown in FIG, the model includes four modules: a learnable feature extraction module, a transfer feature attention module, a related embedding learning module, and a cross-modal image restoration module. The functions of these modules are: first, extract the underlying features of real image data, defective image data, and tactile signals; second, according to the selective attention ability of the attention mechanism, focus on locating the defective area of ​​the image and obtain the transfer features of the area; then, introduce similarity measurement and supervision information, and construct a related embedding learning space in combination with the real label information under the constraints of the semantic association and category measurement loss functions. While minimizing the semantic association objective function to complete the semantic feature learning task, the cross-entropy-based classification measurement objective function is used to minimize the difference between the predicted label and the real label, and the total objective function of the semantic similarity learning between different modalities in the final related embedding learning stage is obtained, and the tactile feature part that is most relevant to the image defective area in the tactile feature is mined as much as possible; finally, the tactile features that are most relevant to the image defective area are used to perform cross-modal restoration of the defective image data by combining the inter-pixel perception constraint loss function, the appearance constraint loss function, and the adversarial loss function, to ensure the quality of the terminal video signal, as follows:

[0105] Step (2-1), a learnable feature extraction module is used to extract features of tactile signals, defective image data and real image data, and participate in subsequent end-to-end model training. The specific implementation process is as follows:

[0106] For the tactile signal H, a gated recurrent unit (GRU) and a 3-layer fully connected network are used as the tactile mapping network to extract the tactile feature h. The GRU network consists of a reset gate and an update gate. The number of units is set to 128, and the output dimensions of the fully connected layer are 1024, 128, and 8. A 128-dimensional tactile feature h is output. The last layer of the fully connected layer is a sigmoid layer, which is used to output the tactile feature prediction label y. (h) .

[0107] For the defective image data I and real image data R of size 128x128, in order to ensure the consistency of the distribution of the defective image and the real image features, a deep convolutional neural network is used as the image mapping network to extract hierarchical features. The network contains 3 convolutional layers and 3 fully connected layers. The number of convolution kernels is 256, 128, and 64 respectively, the convolution kernel size is 4x4, the output dimensions of the fully connected layer are 1024 and 128, and the last fully connected layer outputs 128-dimensional defective image features i and real image features r. The specific process is as follows:

[0108]

[0109]

[0110]

[0111] In the above formula, h, i and r are tactile features, defective image features and real image features respectively, and the dimensions of h, i and r are and θ h and θ i They are tactile mapping network F h (H; θ h ) and the image mapping network F i (I / R;θ i ) parameter set.

[0112] Step (2-2), the transfer feature attention module is used to introduce the attention mechanism, accurately locate the image defect area, and obtain the transfer features that characterize the defect area. The specific implementation process is as follows:

[0113] Step A: For the defective image feature i and the real image feature r, each feature value is defined as a feature unit, that is, and i k and r l They represent the kth eigenvalue of the defective image feature i and the lth eigenvalue of the real image feature r, respectively. and denote the dimensions of the defective image feature i and the true image feature r respectively. Then, the normalized inner product is used to calculate the dimension of each i in the defective image feature i. k And each r in the real image feature r l The cosine similarity between all feature units of the two features is calculated and expressed as follows:

[0114]

[0115] Among them, c k,l represents the cosine similarity matrix, ||·|| represents the modulo operation, and <·,·> represents the normalized inner product operation.

[0116] Step B: Transfer the most relevant part of each feature unit representing the defective area in the defective image features from the real image features, that is, the cosine similarity matrix c k,l Take the maximum value of each column of , and this process is expressed as:

[0117]

[0118] Among them, a kis the attention transfer index, which represents the feature metric that is most correlated with the k-th position of the defective image feature i in the real image feature r.

[0119] Step C: Based on the attention transfer index, a feature selection operation is performed on the real image feature r, in order to obtain the transfer characteristic t representing the image defect area from the real image feature. The process is expressed as:

[0120] t k =ra k ,

[0121] Among them, t k Indicates the selection of the ath feature in the real image r k The feature value of the kth position is transferred to obtain the feature value of the kth position in the transfer feature t. Finally, in order to enhance the distinguishability of the transfer feature, it is classified through the sigmoid layer to obtain the predicted label y of the transfer feature (t) .

[0122] Step (2-3), the related embedding learning module is used to construct the related embedding learning space in combination with the real label information. While minimizing the semantic association objective function to complete the semantic feature learning task, the classification metric objective function based on cross entropy is used to minimize the difference between the predicted label and the real label, and the total objective function of the semantic similarity learning between different modalities in the final related embedding learning stage is obtained, and the tactile features that are most relevant to the image defect area are mined. The specific implementation process is as follows:

[0123] Step A: Use the category label information Y = {y}, y∈{1,2,…,C} to construct the relevant embedding learning space and complete the semantic feature learning task. This process is mainly achieved by minimizing the semantic association objective function L rem accomplish:

[0124]

[0125]

[0126] δ pq =h p *t q ,

[0127] Among them, y represents the true label, C represents the total number of training data categories, N represents the total number of training data, and s pq is the category association factor, δ pq is the characteristic correlation factor, and are the predicted labels of the pth tactile feature and the qth transfer feature, respectively, (·) T represents the transpose operation, and h p and tq denote the pth tactile feature and the qth transfer feature respectively. By observing the semantic association function L rem It can be found that when s pq =1,δ pq The larger the L rem The smaller the semantic association objective function L rem It ensures that in the relevant embedding learning space, the transferred features with the same semantics can assist the tactile sense in semantic feature learning, that is, the tactile features with the highest correlation with the image defect area are extracted from the tactile features extracted by the learnable feature extraction module.

[0128] Step B: In order to further enhance the feature discrimination capability, a cross-entropy-based loss function is used to minimize the difference between the predicted label and the true label to complete the semantic classification. This process is expressed as:

[0129]

[0130]

[0131] Among them, L 1 and L 2 Denote the classification metric objective functions of the transfer feature and tactile feature respectively. p and denote the true label and predicted label of the pth tactile feature, y q and denote the true label and predicted label of the qth transfer feature respectively.

[0132] Step C: Combining the loss functions of steps A and B, we get the overall objective function of semantic similarity learning between different modalities in the relevant embedding learning stage, which can be expressed as:

[0133] L sim =L rem +α 1 L 1 +α 2 L 2 ,

[0134] Among them, L sim is the final semantic similarity loss function, α 1 and α 2 is a hyperparameter, L rem Used to indicate the semantic association between two modalities.

[0135] (2-4) The cross-modal image restoration module is used to combine the inter-pixel perception constraint loss function, the appearance constraint loss function and the adversarial loss function, and use the mined tactile features that are most relevant to the image defect area to perform cross-modal restoration on the defective image data. The specific implementation process is as follows:

[0136] Step A: Use the decoder network to implement cross-modal image restoration; specifically, add the tactile feature h and the defective image feature i to obtain the restored image feature Then the image features will be repaired Input decoder De to get the repaired image Right now Using real image data, the restoration process is constrained from two perspectives: appearance features and perceptual characteristics, so that the restored image As similar as possible to the real image R, this process is expressed as:

[0137]

[0138]

[0139]

[0140] Among them, θ de is the network parameter set of the decoder De, L a and L p They are appearance constraint loss function and perception constraint loss function respectively. is the inpainted image distribution function expectations. It is a feature extraction network similar to VGG,||·|| 1 To take the L1 norm operation.

[0141] In this module, the decoder De contains 2 fully connected layers and 4 deconvolution layers. The dimensions of the fully connected layers are 128 and 1024, respectively, and the number of deconvolutions is 64, 128, 256, and 512, and the output is a 128x128 inpainted image. The discriminator D contains 3 convolutional layers and 3 fully connected layers. The output dimensions of the convolutional layers are 512, 256, and 128, the convolution kernel size is 5x5, and the dimensions of the fully connected layers are 1024, 128, and 1. Finally, it outputs a number in the range of (0, 1) to represent the probability that the input image is a real image.

[0142] Step B: After cross-modal image restoration, the distribution structure of the restored image in step A is further constrained; specifically, the adversarial loss function is used to learn the distribution of real image data, and the adversarial loss function L is defined as adv It can be expressed as:

[0143]

[0144] Among them, L adv is the adversarial loss function of the discriminator D, θ d is the network parameter set of the discriminator D, and They are the true image distribution function P data(R) And the inpainted image distribution function The expectation of D(R;θ d )and are the probabilities that the discriminator D identifies the real image and the restored image as true.

[0145] Step C: Combining the appearance constraint loss function L in step A and step B a , Perceptual Constraint Loss Function L p And the adversarial loss function L adv , the loss function that the decoder De finally needs to minimize is:

[0146] L imp =L a +β 1 L p +β 2 L adv ,

[0147] Among them, L imp is the final restoration loss function, β 1 and β 2 are all hyperparameters.

[0148] Step 3: Use the training set to train the cross-modal image restoration AGVI model to obtain the optimal cross-modal image restoration AGVI model structure and network parameters, as follows:

[0149] Step (3-1), based on the repaired image Combine the training set divided in step 1 and the true label information in steps (2-3) into a standardized input training dataset D tr :

[0150] D tr ={(I n ,R n ,H n ,y n ),n=1,2,…,N},

[0151] Among them, y n is the defective image data I of the nth group participating in the training n , real image data R n and tactile signal H n The true label belongs to, and N is the total capacity of training data.

[0152] Step (3-2), initialize the network parameter set of the AGVI model, which includes θ h ,θ i ,θ de,θ d , these parameters are initialized to the standard normal distribution. Among them, θ h and θ i are the parameter sets of the tactile mapping network and the image mapping network respectively; θ de is the network parameter set of the decoder De, θ d is the network parameter set of the discriminator D.

[0153] Step (3-3), set the total number of iterations to G = 400, and use g to record the specific number of iterations.

[0154] Step (3-4), use the stochastic gradient descent method to train the AGVI model; the specific process is as follows:

[0155] Step 1: Set the hyperparameter α 1 =10 -3 , α 2 =10 -4 , β 1 =10 -4 , β 2 =10 -5 , the learning rates μ for the tactile mapping network and the image mapping network 1 =0.0005, the learning rate of the decoder discriminator μ 2 =0.0003;

[0156] Step 2: Calculate the output of each network in the AGVI model:

[0157] h=F h (H; θ h );i=F i (I; θ i ); r = F i (R; θ i ),

[0158]

[0159] Step 3: Start iteration; based on the gradient descent method and Adam optimizer, update the parameter set of each network from the negative gradient direction of the target:

[0160]

[0161]

[0162]

[0163]

[0164] Among them, L sim (*) is the final semantic similarity loss function Lsim , L imp (*) is the final restoration loss function L imp ; and The tactile mapping network F after the g+1th and gth iterations respectively h , image mapping network F i , the network parameter set of the decoder network De and the discriminator network D; is the derivative.

[0165] Step ④: If g<G, jump to step ③ in step (3-4), increase the number of iterations by 1 (g=g+1), and continue to the next iteration; otherwise, terminate the iteration.

[0166] After G rounds of iterations in steps (3-5), the optimal AGVI model structure and network parameters are finally output.

[0167] Step 4: Using the tactile signals and defective image data in the test set, cross-modal image restoration is performed based on the optimal cross-modal image restoration AGVI model, as follows:

[0168] Step (4-1), the test set D divided in step 1 te for:

[0169] D te ={(I' j ,H' j ),j=1,2,…,F},

[0170] Among them, I' j and H' j is the jth pair of defective image data and tactile signal, I' j and H' j For model testing, F is the total amount of test data;

[0171] (4-2), the test set D te The data in are input in pairs into the optimal cross-modal image restoration AGVI model trained in step 3, and the output is the restored image.

[0172] like Figure 3 As shown, the present invention also relates to a cross-modal image restoration device based on an attention mechanism, the device specifically comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor; wherein:

[0173] 1. Use a memory to store at least one program.

[0174] 2. Use a processor to load at least one program to execute a cross-modal image restoration method based on an attention mechanism described in this embodiment to achieve high-quality and fine-grained restoration of defective images.

[0175] Performance evaluation:

[0176] The present invention conducts experiments according to the above process, and selects the LMT material surface standard data set as the experimental data set. The data set is from the document "Multimodal Feature-based Surface Material Classification" (authors are Matti Strese, Clemens Schuwerk, Albert Iepure, and Eckehard Steinbach, published in the IEEE TRANSACTIONS ON HAPTICS journal in April 2017). It contains material information of three instances of image, sound and tactile acceleration. 80% of the data (tactile, image) are selected from each category as the training set, and the remaining 20% ​​are used as the test set.

[0177] Existing method 1: The document "PatchMatch: A randomized correspondence algorithm for structural image editing" (authors Connelly Barnes, Eli Shechtman, Adam Finkelstein, Dan B Goldman, etc.) proposes a typical block matching image restoration algorithm, which performs image restoration by finding similar blocks in the intact area and copying them to the defective area. This is an intra-modal restoration method that mainly performs restoration based on the information of the image itself.

[0178] Existing method 2: "Context Encoders: Feature Learning by Inpainting" (Author: Deepak Pathak, Philipp Jeff Donahue and Trevor Darrell proposed an unsupervised image feature learning algorithm based on contextual pixels, which uses the codec as the basic structure and uses the pixel information of the image itself to repair and fill in the content of the defective area.

[0179] Existing method three: The paper "Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks" (authors Radford Alec, Metz Luke, Chintala Soumith) proposed an image restoration model based on deep convolutional generative adversarial networks, which includes two models: a generative model G used to capture data distribution, and a discriminative model D used to estimate the probability that the sample comes from training data rather than G. Image restoration is achieved based on an adversarial game.

[0180] Existing method 4: The document "Text-Guided Neural Image Inpainting" (authors Lisai Zhang, Qingcai Chen, Baotian Hu, Shuoran Jiang) proposed an image restoration algorithm based on text guidance. It takes descriptive text as a condition, compares the semantic content of the given text and the remaining image through the attention mechanism, and finds the semantic content to fill the missing part for image restoration.

[0181] Baseline method 1: Remove the attention mechanism module and only use tactile features to fuse with the defective image features to achieve image restoration. This method can be used to verify the importance of the attention mechanism module.

[0182] The present invention: the method of this embodiment.

[0183] The experiment mainly evaluates the quality of the restored image from two perspectives: subjective qualitative evaluation and objective quantitative evaluation.

[0184] First, in terms of subjective qualitative evaluation, Figure 4 The image restoration effect diagrams of the comparative scheme and the AGVI model of the present invention are shown respectively. The defect rate of this experiment is 40%, and the size of the defect area is about 51*51. Among them, each row of images from left to right represents the original image, the defective image, the existing method 1, the existing method 2, the existing method 3, the existing method 4, the benchmark method 1 and the AGVI model of the present invention. Figure 4It can be seen that compared with the deep image restoration method, the restoration result of the existing method one has poor visual perception, the texture cannot be clearly distinguished, and there are obvious image blurring and missing details. The existing method two based on the autoencoder only has accurate and reasonable color restoration for the damaged area, while the structure and texture details are still very blurred. Compared with the existing methods one and two, the restoration results of the existing method three show clearer texture and structural features, but there are obvious blurring and artifacts at the edges. In the restoration results of the AGVI model of the method of the present invention, not only is the semantic information of the repaired area more consistent with the overall image, but the artifacts and blurring at the edges disappear, and the restoration effect is more realistic. At the same time, compared with the benchmark method one, the designed model has more complete texture and structural information, and the image restoration performance is the best.

[0185] Table 1 Evaluation results of the present invention are shown

[0186] method MSE SSIM Existing Literature 130.230 0.587 Existing Literature II 125.622 0.616 Existing Literature III 126.145 0.606 Existing Literature 4 118.547 0.657 Benchmark Method 1 115.369 0.623 AGVI 100.750 0.845

[0187] Secondly, in terms of objective quantitative evaluation, the mean square error (MSE) and structure similarity index (SSIM) commonly used in image quality evaluation indicators are used. The smaller the MAE and the smaller the SSIM, the better the image quality of cross-modal repair. Table I shows the MSE scores and SSIM scores of the AGVI model of the present invention and other comparison models, and evaluates the performance of the AGVI model from the two perspectives of image perception and structural comparison. As can be seen from the table, the AGVI model of the present invention has the highest MSE score and the highest SSIM score. Existing method one, existing method two and existing method three mainly perform intra-modal repair based on the information of the defective image itself, but when the image is severely defective, the repair effect is poor, and its MSE score reaches about 130, while the SSIM value is only about 0.6. Existing method four, benchmark method one and the AGVI model of the present invention verify the content repair ability of non-image modality information for defective areas, and the significant improvement of the repair effect by the attention mechanism. In particular, the AGVI model designed by the method of the present invention combines the repair ability of the autoencoder with the selective attention ability of the attention mechanism, and the repair result has the best visual quality.

[0188] The above description is only a specific implementation of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by a technician familiar with the technical field within the technical scope disclosed by the present invention should be covered by the protection scope of the present invention.

Claims

1. A cross-modal image restoration method based on attention mechanism, characterized in that: The following steps are involved: Step 1: selecting a multimodal dataset, wherein the multimodal dataset includes defective image data, real image data, and tactile signals, and dividing the multimodal dataset into a training set and a test set; Step 2: Design a cross-modal image restoration AGVI model based on attention mechanism, which includes: A learnable feature extraction module is used to extract features of tactile signals, defective image data, and real image data, and participate in subsequent end-to-end model training; The transfer feature attention module is used to introduce the attention mechanism, locate the defective area of ​​the image, and obtain the transfer features that represent the defective area; The correlation embedding learning module is used to construct the correlation embedding learning space in combination with the real label information. While minimizing the semantic association objective function to complete the semantic feature learning task, the classification metric objective function based on cross entropy is used to minimize the difference between the predicted label and the real label, and the total objective function of the semantic similarity learning between different modalities in the final correlation embedding learning stage is obtained, and the tactile features most relevant to the image defect area among the extracted tactile features are mined; The cross-modal image restoration module is used to combine the inter-pixel perception constraint loss function, the appearance constraint loss function and the adversarial loss function, and use the mined tactile features most relevant to the image defect area to perform cross-modal restoration on the defective image data; Step 3: Use the training set to train the cross-modal image restoration AGVI model to obtain the optimal cross-modal image restoration AGVI model structure and network parameters; Step 4: Using the tactile signals and defective image data in the test set, cross-modal image restoration is performed based on the optimal cross-modal image restoration AGVI model; In the above step 3, the cross-modal image restoration AGVI model is trained using the training set, specifically including: Step (3-1), based on the repaired image Combine the training set divided in step 1 and the true label information in steps (2-3) into a standardized input training dataset D tr : D tr ={(I n ,R n ,H n ,y n ),n=1,2,…,N}, Among them, y n is the defective image data I of the nth group participating in the training n , real image data R n and tactile signal H n The true label it belongs to, N is the total capacity of training data; Step (3-2), initialize the network parameter set of the AGVI model, which includes θ h ,θ i ,θ de ,θ d , these parameters are initialized to the standard normal distribution; where θ h and θ i are the parameter sets of the tactile mapping network and the image mapping network respectively; θ de is the network parameter set of the decoder De, θ d is the network parameter set of the discriminator D; Step (3-3), set the total number of iterations to G, and use g to record the specific number of iterations; Step (3-4) uses the stochastic gradient descent method to train the cross-modal image restoration AGVI model. The specific process is as follows: Step ①, set the hyperparameters α1, α2, β1, β2, the learning rate μ1 of the tactile mapping network and the image mapping network, and the learning rate μ2 of the decoder and the discriminator; Step 2: Calculate the output of each network in the cross-modal image restoration AGVI model: h=F h (H;θ h );i=F i (I;θ i );r=F i (R;θ i ), Step 3: Start iteration; based on the gradient descent method and Adam optimizer, update the parameter set of each network from the negative gradient direction of the target: Among them, L sim (*) is the final semantic similarity loss function L sim , L imp (*) is the final restoration loss function L imp ; and The tactile mapping network F after the g+1th and gth iterations respectively h , image mapping network F i , the network parameter set of the decoder network De and the discriminator network D; is the derivative; Step ④: If g<G, jump to step ③ in step (3-4), g=g+1, and continue to the next iteration; otherwise, terminate the iteration; Step (3-5), after G rounds of iterations, finally output the optimal cross-modal image restoration AGVI model structure and network parameters.

2. The cross-modal image restoration method based on the attention mechanism according to claim 1, characterized in that: The step 1 of selecting a multimodal dataset specifically includes: Step (1-1), selecting three different modal data, namely, defective image data I, real image data R and tactile signal H, to form a multimodal data set D; wherein the real image data is the original color image signal, the tactile signal is the tactile power spectrum density obtained by preprocessing the original tactile signal, and the defective image data is an image with a defect rate of λ obtained by defect preprocessing the real image data, and the value range of λ is between 0 and 1; Step (1-2): for data of different modes in the multimodal dataset D, their true label information Y is counted, that is, one-hot encoding is used to label each data with the category label to which the content information represented belongs; Step (1-3), randomly select a proportion of data α from the multimodal dataset D as the training set D tr , the remaining 1-α proportion of data is used as the test set D te , the value range of α is between 0 and 1.

3. The cross-modal image restoration method based on the attention mechanism according to claim 1, characterized in that: In step 2, the feature extraction module can be learned to extract features of the tactile signal, the defective image data and the real image data, specifically including: For the tactile signal H, the gated recurrent unit GRU and a 3-layer fully connected network are used as the tactile mapping network to obtain the tactile feature h and the tactile feature prediction label y. (h) ; For both the defective image data I and the real image data R, an image mapping network composed of a deep convolutional neural network is used to extract shallow defective image features i and real image features r. The specific process is as follows: Among them, h, i and r are tactile features, defective image features and real image features respectively, and the dimensions of h, i and r are and θ h and θ i They are tactile mapping network F h (H; θ h ) and the image mapping network F i (I / R;θ i ) parameter set.

4. The cross-modal image restoration method based on the attention mechanism according to claim 1, characterized in that: In step 2, the attention transfer module obtains the transfer features representing the defect area, specifically including: Step A: For the defective image feature i and the real image feature r, each feature value is defined as a feature unit, that is, and i k and r l They represent the kth eigenvalue of the defective image feature i and the lth eigenvalue of the real image feature r, respectively. and denote the dimensions of the defective image feature i and the true image feature r respectively; then, the normalized inner product is used to calculate the dimension of each i in the defective image feature i. k And each r in the real image feature r l The cosine similarity between all feature units of the two features is calculated and expressed as follows: Among them, c k,l represents the cosine similarity matrix, ||·|| represents the modulo operation, and <·,·> represents the normalized inner product operation; Step B: Transfer the most relevant part of each feature unit representing the defective area in the defective image features from the real image features, that is, the cosine similarity matrix c k,l Take the maximum value of each column of , and this process is expressed as: Among them, a k is the attention transfer index, which represents the feature metric most relevant to the k-th position of the defective image feature i in the real image feature r; Step C: Based on the attention transfer index, a feature selection operation is performed on the real image feature r, in order to obtain the transfer characteristic t representing the image defect area from the real image feature. The process is expressed as: t k =ra k , Among them, t k Indicates the selection of the ath feature in the real image r k The feature value of the kth position is transferred to obtain the feature value of the kth position in the transferred feature t, and then it is classified through the sigmoid layer to obtain the predicted label y of the transferred feature (t) .

5. The cross-modal image restoration method based on the attention mechanism according to claim 1, characterized in that: The relevant embedding learning module in step 2 mines the tactile features that are most relevant to the image defect area among the tactile features, specifically including: Step A: Use the real label information Y = {y}, y∈{1,2,…,C} to construct the relevant embedding learning space and complete the semantic feature learning task. This process is mainly achieved by minimizing the semantic association objective function L rem accomplish: d pq =h p *t q , Among them, y represents the true label, C represents the total number of training data categories, N represents the total number of training data, and s pq is the category association factor, δ pq is the characteristic correlation factor, and are the predicted labels of the pth tactile feature and the qth transfer feature, respectively, (·) T represents the transpose operation, h p and t q denote the pth tactile feature and the qth transfer feature respectively; the semantic association objective function ensures that in the relevant embedding learning space, the transfer features with the same semantics can assist the tactile sense in semantic feature learning, that is, to extract the tactile features with the highest correlation with the image defect area from the tactile features; Step B: In order to further enhance the feature discrimination capability, a cross-entropy-based loss function is used to minimize the difference between the predicted label and the true label to complete the semantic classification. This process is expressed as: Among them, L1 and L2 represent the classification metric objective functions of transfer features and tactile features respectively, and y p and denote the true label and predicted label of the pth tactile feature, y q and Represent the true label and predicted label of the qth transfer feature respectively; Step C: Combining the above steps A and B, the overall objective function of semantic similarity learning between different modalities in the relevant embedding learning stage is obtained, which is expressed as: L sim =L rem +α1L1+α2L2, Among them, L sim is the final semantic similarity loss function, α1 and α2 are hyperparameters, L rem Used to indicate the semantic association between two modalities.

6. The cross-modal image restoration method based on the attention mechanism according to claim 1, characterized in that: In step 2, the cross-modal image restoration module uses the mined tactile features most relevant to the image defective area to perform cross-modal restoration on the defective image data, specifically including: Step A: Use the decoder network to implement cross-modal image restoration; specifically, add the tactile feature h and the defective image feature i to obtain the restored image feature Then the image features will be repaired Input decoder De to get the repaired image Right now Using real image data, the restoration process is constrained from two perspectives: appearance features and perceptual characteristics, so that the restored image As similar as possible to the real image R, this process is expressed as: Among them, θ de is the network parameter set of the decoder De, L a and L p They are appearance constraint loss function and perception constraint loss function respectively. is the inpainted image distribution function expectations, It is a feature extraction network similar to VGG, and ||·||1 is the L1 norm operation; Step B: constrain the distribution structure of the restored image in step A above; specifically, use the adversarial loss function to learn the distribution of real image data, and define the adversarial loss function L adv It is expressed as: Among them, L adv is the adversarial loss function of the discriminator D, θ d is the network parameter set of the discriminator D, and They are the true image distribution function P data(R) And the inpainted image distribution function The expectation of D(R;θ d )and are the probabilities that the discriminator D identifies the real image and the restored image as true; Step C: Combining the appearance constraint loss function L in step A and step B a , Perceptual Constraint Loss Function L p And the adversarial loss function L adv , the loss function that the decoder De finally needs to minimize is: L imp =L a +β1L p +β2L adv , Among them, L imp is the final repair loss function, β1 and β2 are both hyperparameters.

7. The cross-modal image restoration method based on the attention mechanism according to claim 1, characterized in that: In step 4, cross-modal image restoration is performed based on the optimal cross-modal image restoration AGVI model, specifically including: Step (4-1), the test set D divided in step 1 te for: D te ={(I' j ,H' j ),j=1,2,…,F}, Among them, I' j and H' j is the jth pair of defective image data and tactile signal, I' j and H' j For model testing, F is the total amount of test data; Step (4-2), test set D te The data in are input in pairs into the optimal cross-modal image restoration AGVI model trained in step 3, and the output is the restored image.

8. A cross-modal image restoration device based on an attention mechanism, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the computer program is loaded into a processor, it implements a cross-modal image restoration method based on an attention mechanism as described in any one of claims 1 to 7.