Image translation method based on hyper-vector and attention mechanism

Abstract

The application relates to an image translation method based on a hyper-dimensional vector and an attention mechanism, and belongs to the field of computer vision under the computer discipline. The implementation steps are as follows: firstly, a generator based on a double attention mechanism is used, and a feature discriminator based on patchGAN is used to form an adversarial image generation framework; a feature extractor based on VGG19 is used to extract feature vectors of source domain, target domain and conversion domain images, and a hyper-dimensional vector mapping module is used to map the feature vectors into a VSA hyperspace to obtain corresponding feature hyper-dimensional vectors; a mapping network based on a multilayer perceptron is constructed, and the mapping network and each module are combined to form an image translation model based on a hyper-dimensional vector and an attention mechanism, a model loss function is defined, and the model is trained; the trained model can be used to generate a translated target domain result image for an input source domain image; the method can significantly reduce the present semantic flip problem in an image translation task with complex features and reduce the probability of abnormal noise points.

Description

Technical Field

[0001] This invention relates to an image translation method based on hyperdimensional vectors and attention mechanisms, belonging to the field of computer vision within the discipline of computer science. Background Technology

[0002] Image translation has wide applications in computer vision, and improving the image quality and realism of translation results has always been a research focus in this field. In unpaired image translation tasks, when there is a large semantic gap between the source and target domains, many existing image translation methods may suffer from semantic flipping problems, mainly manifested as image artifacts, object or feature illusions.

[0003] Semantic flipping significantly impacts the quality and realism of image translation results. Current methods for addressing this problem mainly fall into three categories: first, pixel-level reconstruction based on cycle consistency methods, but this can lead to assignment transfer issues; second, improving robustness of transformations like flipping or rotation by imposing geometric consistency constraints on image translation, but this method cannot solve the semantic flipping problem of local features; and third, using contrastive learning methods guided by mutual information maximization theory, but this still does not fully resolve the semantic flipping issue. In addition, some researchers have attempted to introduce hyperdimensional vectors into image translation models to address the semantic flipping problem, but this method leads to decreased sharpness and poor performance in image translation tasks with no contours or complex features. Summary of the Invention

[0004] In view of this, the present invention provides an image translation method based on hyperdimensional vectors and attention mechanisms, the implementation of which is as follows:

[0005] Step 1: Establish the source and target domain datasets for the image translation method. The specific steps are as follows: Step 1.1: According to the requirements of the image translation task, divide the images in the dataset into source and target domains; name the source domain image as Source(x) and the target domain image as Target(y); Step 1.2: Divide the images in the source domain and the target domain into training and validation parts, respectively; Step 1.3: Convert the images in the training set to a uniform size; Step 1.4: Randomly crop the images in the training set to augment the dataset and prevent overfitting during training;

[0006] Step 2: Construct a generator G based on stacked ResNet blocks and initialize it with random weights;

[0007] Step 3: Add a "frequency channel attention mechanism" to the generator G based on stacked ResNet blocks to extract spectral features from different channels and use more frequency information during image translation. The specific implementation method is as follows: Step 3.1: Let the feature map output by the generator convolutional layer for the source domain image Source(x) be... Where C represents the number of frequency channels of the feature map, H represents the height, and W represents the width; feature map Divided into Where n is the number of frequency channel groups, and C is required to be divisible by n, and the number of channels in each group is [missing information]. , , ; Step 3.2, for each Perform a two-dimensional discrete cosine transform to obtain its corresponding multi-spectral components. ; for each The calculation method for performing the two-dimensional discrete cosine transform is shown in Equation 1. (1) in, For compressed dimensional vector, Indicates corresponding to Two-dimensional index of multi-spectral components, 2DDCT is a two-dimensional discrete cosine transform. This is the basic form of the two-dimensional discrete cosine transform, and its calculation method is shown in Equation 2. (2) The variables in Formulas 1 and 2 have the same meanings as in step 3.1; Step 3.3: Splicing multi-spectral components Obtain the overall multi-spectral vector As shown in Formula 3. (3) in To obtain the multi-spectral vector, Represents the compression function. For each The corresponding multi-spectral components, It is an operation that connects the different terms of a frequency channel together in exponentiation to form a tensor containing multiple frequency channels; Step 3.4: Multi-spectral vector Calculate the Sigmoid value to obtain the channel attention for the entire multi-spectrum, as shown in Equation 4. (4) in, It is a mapping function that maps any real number to a value in the range (0, 1). This represents the one-dimensional convolution mapping function of a fully connected layer. Attention weights for different spectral channels; Step 3.5, obtain Results and Feature Map Components Multiplication involves assigning different weight values ​​to each spectral channel to obtain a feature map weighted by the frequency channel attention mechanism, which is then used for training subsequent layers of the model.

[0008] Step 4: Add a "self-attention mechanism" to the generator G based on stacked ResNet blocks, so that the image translation process can utilize the correlation information between different parts of the input image. The specific implementation method is as follows: Step 4.1: Let the feature map output by the intermediate layer of the generator for the source domain image Source(x) be... Where C is the number of channels, H represents the height, W represents the width, and N = W × H is the number of feature locations in the feature map; Step 4.2: Obtain feature maps The degree of attention paid by the j-th region to the i-th position is calculated as shown in Formula 5. (5) in, Representation of feature map The degree of attention paid by the j-th region to the i-th position, Let be the similarity score between the j-th region and the i-th location, calculated as shown in Formula 6. = (6) in, The query vector is obtained by performing a convolution operation on the feature map. The feature map is convolved to obtain the key vector. and It is to proceed from C-dimensional to The weight matrix of a dimensional convolution; It represents the dimensions of the query vector and the key vector; Step 4.3: Calculate the feature map The self-attention output of the j-th region on the entire feature map is calculated as shown in Equation 7. (7) in, The output of Formula 5 is the degree of attention paid by the j-th region of the feature map to the i-th position; The feature map is convolved to obtain the value vector. It is to proceed from C-dimensional to The weight matrix of a dimensional convolution; Will Vectors are performed by Convolution operations from 3D to CD dimensions, Is to be carried out by The weight matrix for a C-dimensional convolution; Step 4.4: Concatenate the self-attention outputs of all regions in the feature map to obtain the self-attention output for the entire feature map, as shown in Equation 8. (8) Among them, self-attention output Dimension and Feature Map correspond; Step 4.5: For the i-th position, determine the degree of self-attention it receives. Multiply by the scale parameter γ and combine with the feature The summation achieves the fusion of the original feature at position i with the feature output by self-attention. As shown in Formula 9, (9) Where γ is a learnable scalar, initialized to 0; during model training, γ learns key regions in the local neighborhood and continuously increases, assigns more weights to non-local regions, learns appropriate parameters to obtain feature maps processed by the self-attention mechanism, and enters subsequent layers for training. Thus far, through steps two, three, and four, a generator based on a dual attention mechanism has been successfully developed. "Construction complete;" Step 5: Construct a "feature discriminator based on patchGAN" "; and the "generator based on dual attention mechanism" implemented in steps two, three, and four Together, they form the basic adversarial image generation framework; the role of the adversarial image generation framework is to generate a corresponding "transformed domain image" for the source domain image Source(x), and name the transformed domain image G(x);

[0009] Step 6: Using VGG19 pre-trained on ImageNet as the underlying network, construct a "feature extractor based on VGG19 network"; use this feature extractor to extract features from the source domain image Source(x), the target domain image Target(y), and the transformed domain image G(x); Flatten the features extracted by the feature extractor and concatenate them along the dimension, to finally obtain a feature vector with a dimension of m; wherein the feature vector corresponding to the feature of the source domain image Source(x) is named , the feature vector corresponding to the feature of the target domain image Target(y) is named , the feature vector corresponding to the feature of the transformed domain image G(x) is named ;

[0010] Step 7: Construct a "LSH algorithm-based hyperdimensional vector mapping module", which maps the source domain image feature vector , the target domain image feature vector and the transformed domain image feature vector into the VSA hyperspace to obtain corresponding hyperdimensional vectors, and the specific implementation method is as follows: Use the Locality Sensitive Hashing (LSH) algorithm to convert the feature vector , , perform dimensionality reduction mapping to a random hyperspace , where n<<m, and encode the image feature vector , , into hyperdimensional vectors in the random hyperspace; The source domain feature hyperdimensional vector corresponding to the source domain image feature vector is named , the target domain feature hyperdimensional vector corresponding to the target domain image feature vector is named , the transformed domain feature hyperdimensional vector corresponding to the transformed domain image feature vector is named ; Taking the hyperdimensional vector as an example, in the random hyperspace corresponding to the source domain image, the hyperdimensional vector is decomposed into a combination result of binding operations between all objects in the source domain image and binding operations between the objects and object attributes, as shown in Formula 10, (10) where c and p are hyperdimensional vectors corresponding to the objects in the source domain image, and are hyperdimensional vectors of the attributes of the objects corresponding to c and p; " The "+" sign indicates a binding operation between hyperdimensional vectors, which is the binding operation between objects and their attributes in an image; the "+" sign indicates a bundling operation between hyperdimensional vectors, which is the addition of objects and their attributes to an image; the result of all the bundling operations between objects, as well as the binding operations between objects and their attributes, is the hyperdimensional vector corresponding to the entire image. ;

[0011] Step 8: Construct a "mapping network F based on a multilayer perceptron (MLP)"; use this mapping network F to construct a hyperdimensional vector of the source features. To target feature hyperdimensional vector mapping ; Thus far, the "generator based on dual attention mechanism" constructed in steps two through four has been completed. Step 5 constructs a "feature discriminator based on patchGAN". The “VGG19-based feature extractor” constructed in step six, the “LSH-based hyperdimensional vector mapping module” constructed in step seven, and the “multilayer perceptron (MLP)-based mapping network F” constructed in step eight together form the “image translation model based on hyperdimensional vectors and attention mechanism”, which is named the att_VSAIT model.

[0012] Step 9: Define the loss function for the att_VSAIT model. The loss function consists of two parts: a generative adversarial vector loss based on an attention mechanism and a recurrent loss based on a vector symbolic architecture. The specific definitions of these two parts are as follows: Step 9.1: Define the generative adversarial vector loss based on the attention mechanism. Its calculation method is shown in Equation 11. (11) in, It is training , Hyperdimensional vector adversarial loss in the F process; Discriminator hyperdimensional vectors of the target domain image The natural logarithm of the discrimination probability output; Follow generator The loss is due to the discriminator. For generator The resulting hyperdimensional vector of the transformed domain image The probability output is the natural logarithm of the complement of the set; Focus on mapping networks The loss, this part is caused by the discriminator Translational hyperdimensional vectors affected by the mapping network F The probability of outputting the natural logarithm of the complement; The calculation method is shown in Formula 12. (12) in, Represents a translation hyperdimensional vector that maps from the source domain to the target domain; It is a hyperdimensional vector of the source domain image. For source domain image hyperdimensional vector The hyperdimensional vector mapping from the source domain to the target domain obtained through the mapping network F, i.e. ; "" indicates the binding operation between hyperdimensional vectors, that is, the binding operation between objects in an image and their attributes; Step 9.2: Define the cyclic loss of the vector symbolic architecture, and calculate it as shown in Formula 13. (13) in Through source domain image train Cyclic loss during time, This represents the source domain image hypervector corresponding to the image patch at index i. The translational hyperdimensional vector that maps the features of the transformed domain image to the source domain. The cosine distance between them Is using and The transformed domain image feature map obtained by performing hyperdimensional vector binding operation is mapped to the source domain translation hyperdimensional vector. The calculation method is shown in Equation 14. (14) The cyclic loss of the vector symbolic architecture is minimized. and The mean cosine distance between them ensures that the superdimensional vector obtained when mapping the superdimensional vector of the transformed domain image back to the source domain is the same as the superdimensional vector obtained in the source domain. Hyperdimensional vectors of the source domain image They have the greatest similarity, that is This ensures that the transformation domain image G(x) in the image translation process can recover the features most similar to the source domain image X, thereby preserving the source content and reducing semantic flipping. Step 9.3: Set the proportional coefficient for the two parts of the loss. This forms the total loss of the entire att_VSAIT model, as shown in Equation 15. (15) in ;

[0013] Step 10: Set the model training epoch parameters, Adam optimizer, and StepLR scheduler for the generator. The mapping network F is trained to iteratively optimize the mapping from the source feature hyperdimensional vector to the target feature hyperdimensional vector. The model is validated using the validation dataset, and the trained att_VSAIT model is obtained after validation. The trained att_VSAIT model includes the network architecture and the optimal configuration and weight settings of each component.

[0014] Step 11: After the att_VSAIT model is trained, use it directly for image translation during testing. Load the trained att_VSAIT model, input the source domain image to be translated, and obtain the translated target domain result image. The specific steps are as follows: Step 11.1: Extract features from the source domain image Source(x) using a feature extractor. ， The source domain image feature vector is obtained by planarization and dimensional connectivity. ; Step 11.2: Using the LSH algorithm-based hyperdimensional vector mapping module ， source domain image feature vector Mapping to the VSA hyperspace yields the source domain image hyperdimensional vector. ; Step 11.3: Obtain the hyperdimensional vector of the source domain image through vector mapping of the mapping network F. The mapping is In the already trained att_VSAIT model, the mapping That is, the optimal hyperdimensional vector mapping from the source domain to the target domain. ; Step 11.4: Convert the "source domain image hyperdimensional vector" "and "Optimal hyperdimensional vector mapping from source domain to target domain" "Perform binding operations between hyperdimensional vectors" To obtain the optimal translation hyperdimensional vector mapping from the source domain to the target domain, i.e. Translation of hyperdimensional vectors This involves unbinding attributes from the source domain image and binding them to attributes from the target domain image to obtain a hyperdimensional vector. This vector is the hyperdimensional vector that most closely approximates the "translation result image that the image translation model hopes to obtain." ; Step 11.5, via generator The forward propagation generates the translated target domain result image; At this point, the "Image Translation Method Based on Hyperdimensional Vectors and Attention Mechanism" has been completed.

[0015] Beneficial effects: The method proposed in this invention minimizes the translation hyperdimensional vector from the transform domain image feature map to the source domain during the att_VSAIT model training process. and source domain image hyperdimensional vector The mean cosine distance between them ensures that the hyperdimensional vector obtained when mapping the hyperdimensional vector of the transformation domain image back to the source domain is the same as that in the image translation process. Hyperdimensional vectors of the source domain image The method maximizes the similarity between the source and source domain images, ensuring that the translated image G(x) recovers the features most similar to the source domain image X, thus preserving the source content and reducing semantic flipping. This method reduces the probability of anomalous noise and improves image clarity while mitigating semantic flipping, and performs exceptionally well in image translation tasks with complex features, particularly showing significant improvement in the detail representation of near-field objects. Attached Figure Description

[0016] Figure 1 shows the architecture diagram of the att_VSAIT model proposed in this invention. Figure 2 is a flowchart of the training process of the image translation method proposed in this invention. Figure 3 is a flowchart of the testing process for the image translation method proposed in this invention. Figure 4 shows a comparison of the translation results of GTA→Cityscapes images in an embodiment of the present invention. Detailed Implementation

[0017] The present invention will now be described in detail with reference to the accompanying drawings and embodiments; wherein, the training process of the image translation method proposed in the present invention includes steps one to ten; the testing process of the image translation method proposed in the present invention is as described in step eleven;

[0018] Step 1: Establish the source and target domain datasets for the image translation method. The specific steps in this embodiment are as follows: Step 1.1: According to the requirements of the GTA→Cityscapes image translation task, set 20,000 GTA dataset images as the source domain and name it Source(x); set 5,000 Cityscapes dataset images as the target domain and name it Target(y); where the GTA dataset and Cityscapes dataset are public datasets; Step 1.2: Divide the images in the source domain and target domain into training and validation parts, respectively, and save them in the / source / train, / source / val, / target / train, and / target / val folders; Step 1.3: For GTA→Cityscapes, convert the images in the training set from 1024×2048 to a uniform 256×512 size; Step 1.4: Images in the GTA→Cityscapes training set are randomly cropped from 256×512 to 256×256 to augment the dataset and prevent overfitting during training.

[0019] Step 2: Construct a "generator G based on stacked ResNet blocks". In this embodiment, some parameters of the generator G are set as follows: the number of input and output image channels is 3, the number of residual block channels in ResNet is 64, the number of ResNet blocks in the generator is 9, of which the number of encoder ResNet blocks is 5 and the number of decoder ResNet blocks is 4.

[0020] Step 3: Add a "frequency channel attention mechanism" to the generator G based on stacked ResNet blocks to extract spectral features from different channels and use more frequency information during image translation. The specific implementation method is as follows: Step 3.1: For the source domain image Source(x) of the GTA dataset in the GTA→Cityscapes task, the feature map output by the generator convolutional layer is as follows: In this embodiment, the number of frequency channels C of the feature map is 64, H represents the height of 256, and W represents the width of 256. In this embodiment, the feature map Divided into The number of frequency channel groups, n, is 8, and the number of channels in each group after grouping is... It is 8. , ; Step 3.2, for each Perform a two-dimensional discrete cosine transform to obtain its corresponding multi-spectral components. ; for each The calculation method for performing the two-dimensional discrete cosine transform is shown in Equation 16. (16) in, For compressed dimensional vector, Indicates corresponding to Two-dimensional index of multi-spectral components, 2DDCT is a two-dimensional discrete cosine transform. This is the fundamental form of the two-dimensional discrete cosine transform, and its calculation method is shown in Equation 17. (17) The variables in Formulas 1 and 2 have the same meanings as in step 3.1; Step 3.3: Splicing multi-spectral components Obtain the overall multi-spectral vector As shown in Formula 18, (18) in To obtain the multi-spectral vector, Represents the compression function. For each The corresponding multi-spectral components, It is an operation that connects the different terms of a frequency channel together in exponentiation to form a tensor containing multiple frequency channels; Step 3.4: Multi-spectral vector Calculate the Sigmoid value to obtain the channel attention across the entire multispectral range. As shown in Formula 19, (19) in, It is a mapping function that maps any real number to a value in the range (0, 1). This represents the one-dimensional convolution mapping function of a fully connected layer. Attention weights for different spectral channels; Step 3.5, obtain Results and Feature Map Components Multiplication involves assigning different weight values ​​to each spectral channel to obtain a feature map weighted by the frequency channel attention mechanism, which is then used for training subsequent layers of the model.

[0021] Step 4: Add a "self-attention mechanism" to the generator G based on stacked ResNet blocks, so that the image translation process can utilize the correlation information between different parts of the input image. The specific implementation method is as follows: Step 4.1: Let the feature map output by the intermediate layer of the generator be for the source domain image Source(x). In this embodiment, the number of intermediate layer channels C is 64, the height H is 256, the width W is 256, and the number of feature positions N=W×H of the feature map is 65536; Step 4.2: Obtain the attention level of the j-th region to the i-th position in the feature map. The calculation method is shown in Formula 20. (20) in, Representation of feature map The degree of attention paid by the j-th region to the i-th position, Let be the similarity score between the j-th region and the i-th position, calculated as shown in Formula 21. = (twenty one) in, The query vector is obtained by performing a convolution operation on the feature map. The feature map is convolved to obtain the key vector. and It is to proceed from C-dimensional to The weight matrix of a dimensional convolution; It represents the dimensions of the query vector and the key vector; In this embodiment, to improve memory efficiency, GTA→Cityscapes is configured... = C, that is =8; and It is a process from 64 dimensions to The weight matrix of the dimensional convolution; the query vector and key vector are 8-dimensional; Step 4.3: Calculate the self-attention output of the j-th region of the feature map on the entire feature map. The calculation method is shown in Formula 22. (twenty two) in, The output of Formula 5 represents the degree of attention paid by the j-th region of the feature map to the i-th position; in this embodiment, The feature map is convolved to obtain the value vector. It is the weight matrix for performing convolutions from 64 dimensions to 8 dimensions; Will Perform convolution operations on vectors from 8 dimensions to 64 dimensions. It is the weight matrix for performing convolutions from 8 dimensions to 64 dimensions; Step 4.4: Concatenate the self-attention outputs of all regions in the feature map to obtain the self-attention output for the entire feature map. As shown in Formula 23, (twenty three) Among them, self-attention output Dimension and Feature Map correspond; Step 4.5: For the i-th position, determine the degree of self-attention it receives. Multiply by a scale parameter γ and with the feature The features at position i are added together to obtain the result of fusing the original features at position i with the features output by self-attention. The learnable scalar γ is initialized to 0 and will be continuously increased during model training to obtain the optimal value, thus assigning appropriate weights to non-local regions. The feature map obtained after the self-attention mechanism is then used for subsequent layer training; Thus, through steps two, three, and four, a "generator based on a dual attention mechanism" is successfully developed. "Construction complete;"

[0022] Step 5: Construct a "feature discriminator based on patchGAN" In this embodiment, the discriminator convolution kernel size is 1, the stride is 1, and a three-layer fully convolutional network with a 1×1 convolutional filter is used, with the number of channels in the convolutional layers being 1024, 1024, and 1, respectively. The generator based on dual attention mechanism implemented in steps two, three, and four Together, they form the basic adversarial image generation framework; the role of the adversarial image generation framework is to generate a corresponding "transformed domain image" for the source domain image Source(x), and name the transformed domain image G(x);

[0023] Step 6: Using VGG19 pre-trained on ImageNet as the underlying network, construct a "VGG19-based feature extractor"; use this feature extractor to extract features from the source domain image Source(x), the target domain image Target(y), and the transformed domain image G(x); In this embodiment, for GTA→Cityscapes, the outputs of convolutional layers 2, 7, 12, 21, and 30 of VGG19 are selected to extract features from the source domain image Source(x), target domain image Target(y), and transform domain image G(x). The feature extractor extracts features of sizes 128×256×128, 64×128×256, 32×64×512, 16×32×512, and 8×16×512, which are then planarized to have dimensions of 4,194,304, 2,097,152, 1,048,576, 262,144, and 65,536, respectively, resulting in a feature vector with a dimension of 7,667,712. The feature vectors corresponding to the features of the source domain image Source(x) are named The feature vector corresponding to the features of the target domain image Target(y) is named The eigenvectors corresponding to the features of the transformed domain image G(x) are named ;

[0024] Step 7: Construct a "hyperdimensional vector mapping module based on LSH algorithm" to map the features of the source domain image Source(x), target domain image Target(y), and transformed domain image G(x) into the VSA hyperspace to obtain the corresponding hyperdimensional vectors. , , The specific implementation method is as follows: Use the Locality Sensitive Hash (LSH) algorithm to extract feature vectors , , Dimensionality reduction mapping to random hyperspace In this embodiment, n=4096, and the image feature vector , , Encoded as a hyperdimensional vector in a random hyperspace; Source domain image feature vector The corresponding hyperdimensional vector is named Target domain image feature vector The corresponding hyperdimensional vector is named Transform domain image feature vector The hyperdimensional vector is named ;

[0025] Step 8: Construct a "mapping network F based on multilayer perceptron (MLP)" and use this mapping network F to construct a hyperdimensional vector of the source features. To target feature hyperdimensional vector mapping In this embodiment, the mapping network F uses a two-layer fully convolutional network with a learning rate of 0.0001. Thus far, the "generator based on dual attention mechanism" constructed in steps two through four has been completed. Step 5 constructs a "feature discriminator based on patchGAN". The “VGG19-based feature extractor” constructed in step six, the “LSH-based hyperdimensional vector mapping module” constructed in step seven, and the “multilayer perceptron (MLP)-based mapping network F” constructed in step eight together form the “image translation model based on hyperdimensional vectors and attention mechanism”, which is named the att_VSAIT model.

[0026] Step 9: Define the loss function for the entire att_VSAIT model. The loss of this model consists of two parts: GAN_Loss, a hyperdimensional vector adversarial loss based on the attention mechanism, and att_VSA_Loss, a cyclic loss based on the vector symbolic architecture. Step 9.1: Define the hyperdimensional vector adversarial loss based on the attention mechanism. Its calculation method is shown in Equation 24. (twenty four) in, It is training , Hyperdimensional vector adversarial loss in the F process; Discriminator hyperdimensional vectors of the target domain image The natural logarithm of the discrimination probability output; Follow generator The loss is due to the discriminator. For generator The resulting hyperdimensional vector of the transformed domain image The probability output is the natural logarithm of the complement of the set; Step 9.2: Define the cyclic loss of the vector symbolic architecture, and calculate it as shown in Formula 25. (25) in Through source domain image train Cyclic loss during time, Is using and The transformed domain image features obtained by performing hyperdimensional vector binding operations are mapped to the source domain translation hyperdimensional vectors. Step 9.3: Set the proportional coefficient for the two parts of the loss. This forms the total loss of the entire att_VSAIT model, as shown in Equation 26. (26) In this embodiment, a proportional coefficient is set for the two-part loss of the GTA→Cityscapes task. The value is 10, which forms the total loss of the entire att_VSAIT model;

[0027] Step 10: Set the model training epoch parameters, Adam optimizer, and StepLR scheduler for the generator. Discriminator The mapping network F is trained to iteratively optimize the mapping from the source feature hyperdimensional vector to the target feature hyperdimensional vector. The model is validated using the validation dataset, and the trained att_VSAIT model is obtained after validation. The trained att_VSAIT model includes the network architecture and the optimal configuration and weight settings of each component. In this embodiment, for GTA→Cityscapes, the model training epoch parameter is set to 20; the Adam optimizer is set, and the optimizer is configured... The parameters are = 0, = 0.9, decay weight is 0.0001, the learning rate of the Adam optimizer added to the generator is set to 0.0001, and the learning rate of the Adam optimizer added to the discriminator is set to 0.0004; set the StepLR scheduler, set the learning rate decay factor to 0.5, and the number of steps to perform learning rate decay is 100,000; configure the environment required by att_VSAIT according to the contents of the requirements.txt folder, and enter the command python train.py --name="vsait" to configure the generator. Discriminator The model is trained with the mapping network F; the model is validated using the validation dataset, and the trained att_VSAIT model containing the network structure and weights is obtained after validation.

[0028] Step 11: After the att_VSAIT model is trained, use the model directly for image translation during testing; load the trained att_VSAIT model, input the source domain image to be translated, and obtain the translated target domain result image; In this embodiment, the trained att_VSAIT model is recorded as a .ckpt file and saved in the . / checkpoints / vsait / version_* folder, where * represents the epoch value of the current model training. This model can be directly used for image translation during testing. The command `python test.py --name="vsait_adapt"--checkpoint=". / checkpoints / vsait / version_0 / checkpoints / epoch ={i}-step ={j}.ckpt"` loads the trained att_VSAIT model. The source domain image to be translated is then input, and the translated target domain result image is obtained. The specific steps are as follows: Step 11.1: Extract features from the source domain image Source(x) using a feature extractor, and then planarize and connect along the dimension to obtain the source domain image feature vector. ; Step 11.2: Using the LSH-based hyperdimensional vector mapping module, the source domain image feature vector is transformed... Mapping to the VSA hyperspace yields the source domain image hyperdimensional vector. ; Step 11.3: Obtain the hyperdimensional vector of the source domain image through vector mapping of the mapping network F. The mapping is In the already trained att_VSAIT model, the mapping That is, the optimal hyperdimensional vector mapping from the source domain to the target domain. ; Step 11.4: Convert the "source domain image hyperdimensional vector" "and "Optimal hyperdimensional vector mapping from source domain to target domain" "Perform binding operations between hyperdimensional vectors" To obtain the optimal translation hyperdimensional vector mapping from the source domain to the target domain, i.e. Translation of hyperdimensional vectors This involves unbinding attributes from the source domain image and binding them to attributes from the target domain image to obtain a hyperdimensional vector. This vector is the hyperdimensional vector that most closely approximates the "translation result image that the image translation model hopes to obtain." ; Step 11.5, via generator The forward propagation generates the translated target domain result image and saves it in the . / checkpoints / vsait_adapt / images / folder; at this point, the "Image Translation Method Based on Hyperdimensional Vector and Attention Mechanism" has been completed.

[0029] Appendix Figure 4 The translation results of GTA→Cityscapes images in this embodiment are compared and shown. Sub-figure (1) is the source domain image, sub-figure (2) is the target domain image, sub-figure (3) is the image translation result generated using the classic model, and the semantic flip problem is marked by a box. Sub-figure (4) is the image translation result using the method proposed in this invention, and it can be seen that the semantic flip problem is avoided.

[0030] The translation model guided by this method shows improvements in all evaluation metrics on the GTA→Cityscapes dataset, including initial kernel perceptual distance (KID), pixel accuracy, class accuracy, and mean intersection over union (mIoU), as shown in Table 1. Table 1. Details of each indicator VSAIT 8.74 76.48 45.33 30.89 att_VSAIT 6.19 79.12 48.45 33.74

[0031] The att_VSAIT model, in the GTA→Cityscapes task, achieves the best performance in terms of pixel accuracy, class accuracy, and mean IoU (mIoU) compared to other classic models, as shown in Table 2. Table 2. Evaluation Indicators of the Model GcGAN 65.62 32.38 22.64 DRIT 64.28 32.17 20.99 CUT 64.59 32.19 20.35 SRUNIT 67.21 32.97 22.69 VSAIT 76.48 45.33 30.89 att_VSAIT 79.12 48.45 33.74

[0032] The att_VSAIT model improved by 3.45%, 6.88%, and 9.23% compared to VSAIT across these three metrics; In addition, the att_VSAIT model reduced the training time by an average of 10.96% and the translation time by an average of 19.18% compared to the VSAIT model across multiple image translation tasks.

Claims

1. An image translation method based on hyperdimensional vectors and attention mechanisms, characterized in that, Includes the following steps: Step 1: Establish the source and target domain datasets for the image translation method. The specific steps are as follows: Step 1.1: According to the requirements of the image translation task, divide the images in the dataset into source and target domains; name the source domain image as Source(x) and the target domain image as Target(y); Step 1.2: Divide the images in the source domain and the target domain into training and validation parts, respectively; Step 1.3: Convert the images in the training set to a uniform size; Step 1.4: Randomly crop the images in the training set to augment the dataset and prevent overfitting during training; Step 2: Construct a generator G based on stacked ResNet blocks and initialize the weights to random weights; Step 3: Add a "frequency channel attention mechanism" to the generator G based on stacked ResNet blocks to extract spectral features from different channels and use more frequency information during image translation. The specific implementation method is as follows: Step 3.1: Let the feature map output by the generator convolutional layer for the source domain image Source(x) be... X∈R C×H×W ,in C This represents the number of frequency channels in the feature map. H Indicates altitude, W Indicates width; feature map X Divided into [ X 0 , X 1 , ……, X n-1 ],in n The required number of frequency channel groups C Can be n Divisible by integer, number of channels in each group after grouping C'=C / n, X i ∈R C'×H×W , i ∈{0, 1, ..., n-1}; Step 3.2, for each X i Perform a two-dimensional discrete cosine transform to obtain its corresponding multi-spectral components. Freq i ; for each X i The calculation method for performing the two-dimensional discrete cosine transform is shown in Equation 1. (1) in, Freq i ∈R C' For compressed C' dimensional vector, [ u i , v i ] indicates corresponding to X i Two-dimensional index of multi-spectral components, 2DDCT is a two-dimensional discrete cosine transform. This is the basic form of the two-dimensional discrete cosine transform, and its calculation method is shown in Equation 2. （2) The variables in Formulas 1 and 2 have the same meanings as in step 3.1; Step 3.3: Splicing multi-spectral components Freq i Obtain the overall multi-spectral vector Freq As shown in Formula 3. (3) in Freq ∈R C To obtain the multi-spectral vector, Compress(X) Represents the compression function. Freq i For each X i The corresponding multi-spectral components, cat() It is an operation that connects the different terms of a frequency channel together in exponentiation to form a tensor containing multiple frequency channels; Step 3.4: Multi-spectral vector Freq Calculate the Sigmoid value to obtain the channel attention for the entire multi-spectrum, as shown in Equation 4. (4) in, sigmoid() It is a mapping function that maps any real number to a value in the range (0, 1). fc() This represents the one-dimensional convolution mapping function of a fully connected layer. ms_att Attention weights for different spectral channels; Step 3.5, obtain ms_att Results and Feature Map Components X i Multiplication involves assigning different weight values ​​to each spectral channel to obtain a feature map weighted by the frequency channel attention mechanism, which is then used for training subsequent layers of the model. Step 4: Add a "self-attention mechanism" to the generator G based on stacked ResNet blocks, so that the image translation process can utilize the correlation information between different parts of the input image. The specific implementation method is as follows: Step 4.1: Let the feature map output by the intermediate layer of the generator for the source domain image Source(x) be x. ∈R CxN ,in C For the number of channels, H Indicates altitude, W Indicates width, N=W×H This represents the number of feature locations in the feature map. Step 4.2: Obtain feature maps x The j The region for the first i The level of attention given to each position is calculated using Formula 5. (5) in, β j,i Representation of feature map x The j The region for the first i The level of attention given to each position s ij Let be the similarity score between the j-th region and the i-th location, calculated as shown in Formula 6. = (6) in, f(x) i ) = W f x i The query vector is obtained by performing a convolution operation on the feature map. g( x j ) = W g x j The feature map is convolved to obtain the key vector. and Is to be carried out by C Via The weight matrix of a dimensional convolution; It represents the dimensions of the query vector and the key vector; Step 4.3: Calculate the feature map x The j The self-attention output of each region on the entire feature map is calculated as shown in Equation 7. (7) in, β j,i The output of Formula 5 is the degree of attention paid by the j-th region of the feature map to the i-th position; h( x i ) = W h x i The feature map is convolved to obtain the value vector. It is to proceed from C-dimensional to The weight matrix of a dimensional convolution; Will Vectors are performed by Convolution operations from 3D to CD dimensions, Is to be carried out by The weight matrix for a C-dimensional convolution; Step 4.4: Concatenate the self-attention outputs of all regions in the feature map to obtain the self-attention output for the entire feature map, as shown in Equation 8. (8) Among them, self-attention output o∈R CxN Dimension and Feature Map x∈R CxN correspond; Step 4.5: For the i-th position, determine the degree of self-attention it receives. o i Multiply by the scale parameter γ and combine with the feature x i The summation achieves the fusion of the original feature at position i with the feature output by self-attention. y i As shown in Formula 9, (9) in γ It is a learnable scalar, initialized to 0; during model training, γ The system learns key regions in the local neighborhood and adds them continuously, assigns more weights to non-local regions, learns appropriate parameters to obtain feature maps with self-attention mechanism, and then enters subsequent layers for training. Thus far, through steps two, three, and four, the "generator based on dual attention mechanism" has been successfully developed. G att "Construction complete;" Step 5: Construct a "feature discriminator based on patchGAN" D Y "; and the generator based on dual attention mechanism implemented in steps two, three, and four" G att Together, they form the basic adversarial image generation framework; the role of the adversarial image generation framework is to generate a corresponding "transformed domain image" for the source domain image Source(x), and name the transformed domain image G(x); Step 6: Using VGG19 pre-trained on ImageNet as the underlying network, construct a "VGG19-based feature extractor"; use this feature extractor to extract features from the source domain image Source(x), the target domain image Target(y), and the transformed domain image G(x); The features extracted by the feature extractor are planarized and concatenated along the dimension to obtain a feature vector of dimension m; the feature vector corresponding to the features of the source domain image Source(x) is named f X The feature vector corresponding to the features of the target domain image Target(y) is named f Y The eigenvectors corresponding to the features of the transformed domain image G(x) are named f G(x) ; Step 7: Construct a "hyperdimensional vector mapping module based on LSH algorithm" to map the source domain image feature vectors. f X Target domain image feature vector f Y and transform domain image feature vector f G(x) Mapping to the VSA hyperspace yields the corresponding hyperdimensional vector, as detailed below: Using the Locality Sensitive Hashing LSH algorithm to map the feature vectors f X , f Y , f G(x) to a random hyperspace by dimensionality reduction , where n<<m, and encoding the image feature vectors f X , f Y , f G(x) into hyperdimensional vectors in the random hyperspace; Source domain image feature vector f X The corresponding source domain feature hyperdimensional vector is named v X Target domain image feature vector f Y The corresponding target domain feature hyperdimensional vector is named v Y Transform domain image feature vector f G(x) The corresponding transformation domain feature hyperdimensional vector is named v G(x) ; With hyperdimensional vectors v x For example, in the random hyperspace corresponding to the source domain image, the hyperdimensional vector v X The decomposition consists of the bundling operations between all objects in the source domain image, and the combined result of the binding operations between objects and object attributes, as shown in Equation 10. (10) in, c , p It is the hyperdimensional vector corresponding to the object in the source domain image. c src and p src yes c , p A hyperdimensional vector corresponding to the properties of an object; The "" symbol represents the binding operation between hyperdimensional vectors, i.e., the binding operation between objects in an image and their attributes; the "+" symbol represents the bundling operation between hyperdimensional vectors, i.e., adding objects and their attributes to an image; the result of all the bundling operations between objects, as well as the binding operations between objects and their attributes, is the hyperdimensional vector corresponding to the entire image. v x ; Step 8: Construct a "mapping network F based on a multilayer perceptron (MLP)"; use this mapping network F to construct a hyperdimensional vector of source domain features. v X To target domain feature hyperdimensional vector v Y mapping ; Thus far, the "generator based on dual attention mechanism" constructed in steps two through four has been completed. G att Step 5 constructs a "feature discriminator based on patchGAN". D Y The "VGG19-based feature extractor" constructed in step six, the "LSH-based hyperdimensional vector mapping module" constructed in step seven, and the "multilayer perceptron (MLP)-based mapping network F" constructed in step eight together form the "image translation model based on hyperdimensional vectors and attention mechanism", which is named the att_VSAIT model. Step 9: Define the loss function for the att_VSAIT model. The loss function consists of two parts: a generative adversarial vector loss based on an attention mechanism and a recurrent loss based on a vector symbolic architecture. The specific definitions of these two parts are as follows: Step 9.1: Define the generative adversarial vector loss based on the attention mechanism. Its calculation method is shown in Equation 11. (11) in, L GAN (G att , D Y (, F, X, Y) It is training G att , D Y and F Hyperdimensional vector adversarial loss during the process; log D Y (v Y ) Discriminator D Y hyperdimensional vectors of the target domain image v Y The natural logarithm of the discrimination probability output; log(1-D Y (v G(x) ) Follow generator G att The loss is due to the discriminator. D Y For generator G att The resulting hyperdimensional vector of the transformed domain image v G(x) The probability output is the natural logarithm of the complement of the set; Focus on mapping networks F The loss, this part is caused by the discriminator D Y Translational hyperdimensional vectors affected by the mapping network F The probability of outputting the natural logarithm of the complement; The calculation method is shown in Formula 12. (12) in, Represents a translation hyperdimensional vector that maps from the source domain to the target domain; v X It is a hyperdimensional vector of the source domain image. For source domain image hyperdimensional vector v X The hyperdimensional vector mapping from the source domain to the target domain obtained through the mapping network F, i.e. F(v x ) ;" "" indicates the binding operation between hyperdimensional vectors, that is, the binding operation between objects in an image and their attributes; Step 9.2: Define the cyclic loss of the vector symbolic architecture, and calculate it as shown in Formula 13. (13) in Through source domain image X train G att Cyclic loss during time, This represents the source domain image hypervector corresponding to the image patch at index i. The translational hyperdimensional vector that maps the features of the transformed domain image to the source domain. The cosine distance between them v G(x)->x Is using and v G(x) The transformed domain image feature map obtained by performing hyperdimensional vector binding operation is mapped to the source domain translation hyperdimensional vector. The calculation method is shown in Equation 14. (14) The cyclic loss of the vector symbolic architecture is minimized. v G(x)->x and v X The mean cosine distance between them ensures that the superdimensional vector obtained when mapping the superdimensional vector of the transformed domain image back to the source domain is the same as the superdimensional vector obtained in the source domain. v G(x)->x Hyperdimensional vectors of the source domain image v X They have the greatest similarity, that is v G(x)->x ≈ v X This ensures that the transformation domain image G(x) in the image translation process can recover the features most similar to the source domain image X, thereby preserving the source content and reducing semantic flipping. Step 9.3: Set the scaling factor λ for the two parts of the loss to form the total loss of the entire att_VSAIT model, as shown in Equation 15. (15) Where λ≥5; Step 10: Set the model training epoch parameters, Adam optimizer, and StepLR scheduler for the generator. G att Discriminator D Y The mapping network F is trained to iteratively optimize the mapping from the source feature hyperdimensional vector to the target feature hyperdimensional vector. The model is validated using the validation dataset, and the trained att_VSAIT model is obtained after validation. The trained att_VSAIT model includes the network architecture and the optimal configuration and weight settings of each component. Step 11: After the att_VSAIT model is trained, use it directly for image translation during testing. Load the trained att_VSAIT model, input the source domain image to be translated, and obtain the translated target domain result image. The specific steps are as follows: Step 11.1: Extract features from the source domain image Source(x) using a feature extractor, and then planarize and connect along the dimension to obtain the source domain image feature vector. f X ; Step 11.2: Using the LSH-based hyperdimensional vector mapping module, the source domain image feature vector is transformed... f X Mapping to the VSA hyperspace yields the source domain image hyperdimensional vector. v x ; Step 11.3: Obtain the hyperdimensional vector of the source domain image through vector mapping of the mapping network F. v X The mapping is F(v X ) In the already trained att_VSAIT model, the mapping F(v X ) That is, the optimal hyperdimensional vector mapping from the source domain to the target domain. ; Step 11.4: Convert the "source domain image hypervector" v X "and "Optimal hyperdimensional vector mapping from source domain to target domain "Perform binding operations between hyperdimensional vectors" To obtain the optimal translation hyperdimensional vector mapping from the source domain to the target domain, i.e. Translation of hyperdimensional vectors This involves unbinding attributes from the source domain image and binding them to attributes from the target domain image to obtain a hyperdimensional vector. This vector is the hyperdimensional vector that most closely approximates the "translation result image that the image translation model hopes to obtain." ; Step 11.5, via generator G att The forward propagation generates the translated target domain result image.

Images