Digital Hunan embroidery image style migration method and device based on regional semantics

Abstract

The invention relates to a digital Hunan embroidery image style migration method and device based on regional semanteme, and the method comprises the steps: finding the semantic information of the closest semantic segmentation region from a digital Hunan embroidery image according to the semantic information of the semantic segmentation region in a content image, and achieving the regional style migration of different semantic segmentation regions. Local stylized features are obtained; global style migration is carried out according to the features from the content image and the features of the digital Hunan embroidery image to obtain global stylized features, and then the local stylized features and the global stylized features are fused to obtain overall stylized features; and finally, generating a Hunan embroidery style image with more specific style features and texture details of the Hunan embroidery according to the overall stylization features, thereby improving the finally generated visual effect.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of image processing technology, and in particular to a method and device for style transfer of digital Hunan embroidery images based on regional semantics. Background Art

[0002] Hunan embroidery, as one of the four famous embroidery styles in China, is famous for its exquisite craftsmanship and unique artistic style.

[0003] Style transfer is a popular image generation technology. It transfers the artistic style of an artistic image to another content image, aiming to use the artistic style of the artistic image to re-present the objects in the content image, thereby generating a new artistic image.

[0004] At present, the main method is to extract features from deep convolutional networks pre-trained with the ImageNet dataset, and to express the style by calculating high-order statistics such as the Gram matrix, mean variance, etc. on these features, laying the foundation for implementing style transfer methods based on deep convolutional networks. However, this type of method has limited effect on the grasp of local style characteristics, especially the migration effect of different needlework methods like Hunan embroidery. At the same time, the texture, color, needlework and other style characteristics of different semantic objects in artistic images are not the same. If the style of the image is transferred only from the perspective of the overall style characteristics, without considering the semantic relationship between the Hunan embroidery style image and the objects in the content image, the final generated visual effect will also be greatly affected. Summary of the invention

[0005] The following is a summary of the subject matter described in detail herein. This summary is not intended to limit the scope of the claims.

[0006] The present application aims to at least solve the technical problems existing in the prior art. To this end, the present application proposes a style transfer method and device for digital Xiang embroidery images based on regional semantics, wherein the method finds the semantic information of the closest semantic segmentation region from the digital Xiang embroidery image based on the semantic information of the semantic segmentation region in the content image, implements regional style transfer for different semantic segmentation regions, obtains local stylized features, performs global style transfer based on the features from the content image and the features of the digital Xiang embroidery image, obtains global stylized features, and enables the style transfer model to generate a Xiang embroidery style image with more style characteristics and texture details unique to Xiang embroidery through the overall stylized features that fuse the local stylized features and the global stylized features.

[0007] A first aspect of an embodiment of the present application provides a method for style transfer of a digital Hunan embroidery image based on regional semantics, the method comprising:

[0008] Acquire a target content image and a target digital Xiang embroidery image;

[0009] The target content image and the target digital Xiang embroidery image are input into a preset style transfer model, and the style transfer model outputs a Xiang embroidery style image corresponding to the target content image; the training process of the style transfer model includes:

[0010] Extracting a plurality of first semantic segmentation regions from a first image and extracting a plurality of second semantic segmentation regions from a second image; the first image is a content image for training, and the second image is a digital Xiang embroidery image for training;

[0011] Perform semantic matching on the plurality of the first semantic segmentation regions and the plurality of the second semantic segmentation regions to obtain a plurality of semantic matching pairs; any one of the semantic matching pairs includes: one of the first semantic segmentation regions and the second semantic segmentation region closest thereto;

[0012] extracting a first image feature of the first image and extracting a second image feature of the second image;

[0013] Performing global style migration according to the first image feature and the second image feature to obtain a first stylized feature;

[0014] Calculate a first pixel dot product result of a first semantic segmentation region mask corresponding to a first semantic segmentation region in each of the semantic matching pairs and the first image feature, and a second pixel dot product result of a second semantic segmentation region mask corresponding to a second semantic segmentation region and the second image feature, and perform regional style migration according to the first pixel dot product result and the second pixel dot product result to obtain a second stylized feature corresponding to each of the semantic matching pairs;

[0015] fusing the first stylized feature and all the second stylized features to obtain an overall stylized feature;

[0016] A Hunan embroidery style image corresponding to the first image is generated according to the overall stylized features.

[0017] This embodiment has at least the following beneficial effects:

[0018] The method provides a style transfer model, through which the content image and the digital Xiang embroidery image can be semantically segmented first, and the semantic segmentation area of ​​the content image and the semantic segmentation area of ​​the digital Xiang embroidery image can be extracted respectively; then the semantic segmentation area of ​​the content image and the semantic segmentation area of ​​the digital Xiang embroidery image are semantically matched, and the semantic segmentation area of ​​the digital Xiang embroidery image that is most similar to the semantic segmentation area of ​​the content image is found; secondly, on the one hand, global style transfer can be performed according to the respective image features of the content image and the digital Xiang embroidery image, and on the other hand, local regional style transfer can be performed according to the semantic matching pair, the content image, and the digital Xiang embroidery image; finally, the stylized features obtained by the global style transfer and the stylized features of the regional style transfer are fused, and a Xiang embroidery style image corresponding to the content image is generated according to the fusion result. This method finds the semantic information (such as texture, stitch and color) of the closest semantic segmentation area from the digital Hunan embroidery image based on the semantic information of the semantic segmentation area in the content image, realizes regional style transfer for different semantic segmentation areas, and obtains local stylized features; performs global style transfer based on the features from the content image and the features of the digital Hunan embroidery image, and obtains global stylized features, so that the style transfer model generates a Hunan embroidery style image with more style characteristics and texture details unique to Hunan embroidery through the overall stylized features that integrate the local stylized features and the global stylized features.

[0019] In some embodiments, the extracting a first image feature of the first image and extracting a second image feature of the second image include:

[0020] Inputting the first image into a lightweight reversible network to extract first image features of the first image through a forward process of the lightweight reversible network;

[0021] Inputting the second image into the lightweight reversible network to extract second image features of the second image through a forward process of the lightweight reversible network;

[0022] The lightweight reversible network includes a plurality of stacked reversible units, and the forward process of the lightweight reversible network includes:

[0023] Y k+1 =Concat(Y k+1 [c+1:C],Y k+1 [1:c]);

[0024] Y k+1 [c+1:C]=Y k [c+1:C]+H(Y k [1:c]);

[0025] Y k+1 [1:c]=Y k[1:c];

[0026] Among them, Y k+1 is the output of the k+1th reversible unit, Concat is the channel concatenation function, [1:c] is the 1st to cth layer channel of any reversible unit, [c+1:C] is the cth to Cth layer channel of any reversible unit, Y k+1 [c+1:C] is the output of the k+1th reversible unit at [1:c], Y k+1 [1:c] is the output of the k+1th reversible unit at [c+1:C], and H is the mapping function.

[0027] In some embodiments, generating a Hunan embroidery style image corresponding to the first image according to the overall stylized features includes:

[0028] Inputting the overall stylized features into the lightweight reversible network to generate a Hunan embroidery style image corresponding to the first image through a reverse process of the lightweight reversible network;

[0029] The reverse process of the lightweight reversible network includes:

[0030] Y k =Concat(Y k [c+1:C],Y k [1:c]);

[0031] Y k [c+1:C]=Y k+1 [1:c]-H(Y k+1 [1:c]);

[0032] Y k [1:c]=Y k+1 [1:c];

[0033] Among them, the last reversible unit of the reverse process outputs the Hunan embroidery style image corresponding to the first image.

[0034] In some embodiments, the loss function of the style transfer model is a content loss function Style loss function First consistency loss function and the second consistency loss function The weighted sum of

[0035]

[0036] Where N is the total number of lightweight reversible networks, For a lightweight reversible network, from the first image I c Corresponding Xiang embroidery style image I csThe i-th layer features extracted from For a lightweight reversible network, from the first image I c The i-th layer features extracted from For a lightweight reversible network, from the second image I s The i-th layer features extracted from Features The mean of Features The mean of Features The variance of Features is the variance, ||·|| is the norm, ||·|| 2 is the 2-norm, I cc The second image I s and the first image I c The first image I is used c As the input generated Xiang embroidery style image, I ss The second image I s and the first image I c The second image I is used s The Xiang embroidery style image generated as input, For lightweight reversible networks from I cc The i-th layer features extracted from For lightweight reversible networks from I ss The i-th layer features extracted from .

[0037] In some embodiments, extracting a plurality of first semantic segmentation regions of the first image and a plurality of second semantic segmentation regions of the second image includes:

[0038] Extracting a plurality of first semantic segmentation region masks of the first image according to the first semantic segmentation network;

[0039] Extracting a plurality of second semantic segmentation region masks of the second image through a second semantic segmentation network; the second semantic segmentation network is a deep learning SAM network;

[0040] Performing pixel-by-pixel multiplication of the first semantic segmentation region masks with the first image to obtain a plurality of first semantic segmentation regions;

[0041] Perform pixel-by-pixel multiplication of the second image with the plurality of second semantic segmentation region masks to obtain a plurality of second semantic segmentation regions.

[0042] In some embodiments, the process of calculating the first stylized feature includes:

[0043]

[0044] Where AdaIN is the AdaIN migration function, F c is the first image feature, F s is the second image feature, It is the first stylized feature.

[0045] In some embodiments, the process of calculating the second stylized feature includes:

[0046]

[0047] Among them, AdaIN is the AdaIN migration function, Dawn(m cmatch(j) )⊙F c Dawn cmatch(j) ) and F c The first pixel multiplication result, Dawn(m cmatch(j) ) is m cmatch(j) Downsampling, m cmatch(j) is the first semantic segmentation region mask j, F c is the first image feature, Dawn(m sj )⊙F s Dawn(m sj ) and F s The second pixel dot multiplication result, Dawn(m sj ) is m sj Downsampling, m sj is the second semantic segmentation region mask j, F s is the second image feature, m cmatch(j ) and the second semantic segmentation region corresponding to msj form a semantic matching pair j; is the second stylized feature corresponding to the semantic matching pair j.

[0048] In some embodiments, the semantically matching the plurality of first semantic segmentation regions with the plurality of second semantic segmentation regions to obtain a plurality of semantic matching pairs includes:

[0049] Inputting the first semantic labels corresponding to the plurality of second semantic segmentation regions and the plurality of first semantic segmentation regions into the multimodal CLIP network to obtain a plurality of matching results output by the multimodal CLIP network, wherein the process of generating any matching result by the multimodal CLIP network includes:

[0050] C match(j)L = argmax CLIP(C iL , S j );

[0051] Among them, C iL is the first semantic label corresponding to the first semantic segmentation region i, Sj is the second semantic segmentation region j, C match(j)L C iL and S j The matching result of , argmax is the maximum value function, CLIP() is the multimodal CLIP network function;

[0052] The first semantic label in each of the matching results is replaced with the corresponding first semantic segmentation region to obtain a plurality of semantic matching pairs.

[0053] In some embodiments, the fusing the first stylized feature and all the second stylized features to obtain an overall stylized feature includes:

[0054]

[0055] in, is the first stylized feature, To fuse all the features after the second stylized features, Concat is the channel concatenation function, avg is the average pooling function, and CONVS is the convolution function.

[0056] A second aspect of the embodiments of the present application provides a style migration device for digital Hunan embroidery images based on regional semantics, the device comprising:

[0057] A data acquisition module is used to acquire a target content image and a target digital Xiang embroidery image;

[0058] The style transfer module is used to input the target content image and the target digital Hunan embroidery image into a preset style transfer model, and obtain the Hunan embroidery style image corresponding to the target content image output by the style transfer model; the training process of the style transfer model includes:

[0059] Extracting a plurality of first semantic segmentation regions from a first image and extracting a plurality of second semantic segmentation regions from a second image; the first image is a content image for training, and the second image is a digital Xiang embroidery image for training;

[0060] Perform semantic matching on the plurality of the first semantic segmentation regions and the plurality of the second semantic segmentation regions to obtain a plurality of semantic matching pairs; any one of the semantic matching pairs includes: one of the first semantic segmentation regions and the second semantic segmentation region closest thereto;

[0061] extracting a first image feature of the first image and extracting a second image feature of the second image;

[0062] Performing global style migration according to the first image feature and the second image feature to obtain a first stylized feature;

[0063] Calculate a first pixel dot product result of a first semantic segmentation region mask corresponding to a first semantic segmentation region in each of the semantic matching pairs and the first image feature, and a second pixel dot product result of a second semantic segmentation region mask corresponding to a second semantic segmentation region and the second image feature, and perform regional style migration according to the first pixel dot product result and the second pixel dot product result to obtain a second stylized feature corresponding to each of the semantic matching pairs;

[0064] fusing the first stylized feature and all the second stylized features to obtain an overall stylized feature;

[0065] A Hunan embroidery style image corresponding to the first image is generated according to the overall stylized features. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0067] Figure 1 It is a flow chart of the style transfer method of digital Xiang embroidery images based on regional semantics provided by this application;

[0068] Figure 2 is a flowchart of the training style transfer model provided in this application;

[0069] Figure 3 It is a schematic diagram of the content leakage of VGG encoding provided by this application;

[0070] Figure 4 is a schematic diagram of the deviation of style conversion provided by this application;

[0071] Figure 5 This is a flowchart of another method for style transfer of digital Hunan embroidery images based on regional semantics provided by this application;

[0072] Figure 6 It is a structural diagram of a style transfer device for digital Hunan embroidery images based on regional semantics provided by the present application;

[0073] Figure 7 It is a structural diagram of the electronic device provided in this application. DETAILED DESCRIPTION

[0074] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0075] It should be noted that, although the functional modules are divided in the device schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first", "second", etc. in the specification, claims and the above drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0076] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0077] like Figure 1 and Figure 2 An embodiment of the present application provides a style transfer method for digital Hunan embroidery images based on regional semantics, the method comprising:

[0078] Step S110, obtaining a target content image and a target digital Xiang embroidery image;

[0079] Step S120, inputting the target content image and the target digital Hunan embroidery image into a preset style transfer model, and obtaining a Hunan embroidery style image corresponding to the target content image output by the style transfer model.

[0080] It should be noted that the above steps S110 and S120 are the process of using the style transfer model.

[0081] Among them, the training process of the style transfer model includes:

[0082] Step S1210, extracting a plurality of first semantic segmentation regions of a first image and extracting a plurality of second semantic segmentation regions of a second image; the first image is a content image used for training, and the second image is a digital Xiang embroidery image used for training;

[0083] Step S1220, semantically matching the plurality of first semantic segmentation regions with the plurality of second semantic segmentation regions to obtain a plurality of semantic matching pairs; any semantic matching pair includes: a first semantic segmentation region and the second semantic segmentation region closest thereto;

[0084] Step S1230, extracting a first image feature of the first image and extracting a second image feature of the second image;

[0085] Step S1240, performing global style transfer according to the first image feature and the second image feature to obtain a first stylized feature;

[0086] Step S1250, calculating a first pixel dot product result of a first semantic segmentation region mask and a first image feature corresponding to a first semantic segmentation region in each semantic matching pair, and a second pixel dot product result of a second semantic segmentation region mask and a second image feature corresponding to a second semantic segmentation region, and performing regional style transfer according to the first pixel dot product result and the second pixel dot product result to obtain a second stylized feature corresponding to each semantic matching pair;

[0087] Step S1260, fusing the first stylized feature and all the second stylized features to obtain an overall stylized feature;

[0088] Step S1270: Generate a Hunan embroidery style image corresponding to the first image according to the overall stylized features.

[0089] In step S1210, a plurality of first semantic segmentation regions of the first image and a plurality of second semantic segmentation regions of the second image may be extracted through a conventional semantic segmentation network (such as FCN, UNet, SegNet).

[0090] In some embodiments, the step of extracting a plurality of first semantically segmented regions of the first image and a plurality of second semantically segmented regions of the second image comprises:

[0091] Step S1211: extract multiple first semantic segmentation region masks of the first image according to the first semantic segmentation network.

[0092] Step S1212, extracting multiple second semantic segmentation region masks of the second image through a second semantic segmentation network; the second semantic segmentation network is a SAM network.

[0093] Step S1213: perform pixel-by-pixel dot multiplication on the first image and the plurality of first semantic segmentation region masks to obtain a plurality of first semantic segmentation regions.

[0094] Step S1214: perform pixel-by-pixel dot multiplication on the second image and the plurality of second semantic segmentation region masks to obtain a plurality of second semantic segmentation regions.

[0095] Because the first image is a content image, in this embodiment, a conventional semantic segmentation network can be used as the first semantic segmentation network. When the first semantic segmentation network outputs a semantic segmentation region mask, it also outputs a semantic label corresponding to the region.

[0096] Since conventional semantic segmentation networks are only trained on natural images, they rarely use digital Xiang embroidery images during training. If the semantic segmentation labels for Xiang embroidery images are remade, it is not only time-consuming and laborious, but also the digital Xiang embroidery images that can be collected are very limited compared to the hundreds of thousands of images required for semantic segmentation model training. Moreover, the training labels of the semantic segmentation network are fixed, and only these labels can be used for prediction. This will greatly affect the accuracy of the style area, which is not only reflected in whether the segmented area is complete and accurate, but also in the judgment of the semantic information in the area. Therefore, this embodiment uses a large visual model (SegmentAnything, SAM) that has been trained with massive data and has a strong zero-sample segmentation capability to automatically segment the digital Xiang embroidery image (i.e., the second image) to obtain different semantic areas.

[0097] Through this step, a first semantic segmentation region of the first image and its semantic label, and a second semantic segmentation region of the second image can be obtained.

[0098] In step S1220, the purpose of semantic matching is to find each first semantic segmentation region and the second semantic segmentation region closest to it. In the Hunan embroidery image, the style features such as needlework, color, texture, etc. in different semantic regions are varied, each with its own particularity, and are often not the same. Therefore, in the process of style transfer, this embodiment first needs to find the content semantic region closest to its semantics from the second semantic region of the second image according to the degree of semantic similarity, and then perform regionalized style transfer.

[0099] In some embodiments, semantic matching is performed on the plurality of first semantic segmentation regions and the plurality of second semantic segmentation regions to obtain a plurality of semantic matching pairs, including:

[0100] Step S1221, inputting the first semantic labels corresponding to the plurality of second semantic segmentation regions and the plurality of first semantic segmentation regions into the CLIP network, and obtaining a plurality of matching results output by the CLIP network, wherein the process of the CLIP network generating any matching result includes:

[0101] C match(j)L = argmax CLIP(C iL , S j );

[0102] Among them, C iL is the first semantic label corresponding to the first semantic segmentation region i, S j is the second semantic segmentation region j, C match(j)L C iL and S j The matching result, CLIP is the function corresponding to the CLIP network;

[0103] Step S1222: Replace the first semantic label in each matching result with the corresponding first semantic segmentation region to obtain a plurality of semantic matching pairs.

[0104] In order to match semantic regions, this embodiment uses the CLIP network with cross-modal understanding capabilities. CLIP (Contrastive Language-Image Pre-training) is an advanced multimodal pre-training large model proposed by OpenAI. It is trained by a large number of image-text pairs collected from the Internet, and learns rich visual and language feature representations, so that it can understand the content of the image and the text description related to it. The core idea of ​​the CLIP model is to use contrastive learning to train a network that can map images and texts to a common feature space. In this feature space, the embedding vectors of semantically related images and texts will be close to each other, while those that are not related will be far away from each other. Since CLIP has learned rich visual and language representations, it can achieve zero-sample learning on different image classification tasks without additional training. This means that CLIP can directly classify unseen categories. Therefore, the matching process can be converted to classifying each region using CLIP.

[0105] In step S1230, first image features of the first image are extracted and second image features of the second image are extracted.

[0106] In some embodiments, the image features may be extracted or the image may be decoded based on the image features through a common encoder-decoder structure, which will not be described in detail here.

[0107] In some embodiments, extracting a first image feature of the first image and extracting a second image feature of the second image include:

[0108] Step S1231: input the first image into a lightweight reversible network to extract first image features of the first image through a forward process of the lightweight reversible network.

[0109] Step S1232, inputting the second image into the lightweight reversible network to extract second image features of the second image through a forward process of the lightweight reversible network;

[0110] The lightweight reversible network includes multiple stacked reversible units. The forward process of the lightweight reversible network includes:

[0111] Y k+1 =Concat(Y k+1 [c+1:C],Y k+1 [1:c]);

[0112] Y k+1 [c+1:C]=Yk [c+1:C]+H(Y k [1:c]);

[0113] Y k+1 [1:c]=Y k [1:c];

[0114] Among them, Y k+1 is the output of the k+1th reversible unit, Concat is the channel concatenation function, [1:c] is the 1st to cth layer channel of any reversible unit, [c+1:C] is the cth to Cth layer channel of any reversible unit, Y k+1 [c+1:C] is the output of the k+1th reversible unit at [1:c], Y k+1 [1:c] is the output of the k+1th reversible unit at [c+1:C], and H is the mapping function.

[0115] The content leakage problem may occur in the style transfer method based on the common encoder-decoder structure. The reasons include:

[0116] The encoder and decoder architecture can use the encoder to encode the image into features, and then use the decoder to decode the features into an image. The image obtained after decoding is very similar to the original image. However, if the decoded image is input into the encoder and decoder again, after multiple rounds of iterations, the final image will be disorganized. The effect will be more obvious as the number of rounds increases. At the same time, there are deviations in the decoding process. The two work together to cause content leakage problems. Even if a first image (i.e., content image) and a second image (i.e., style image) are used to generate a Hunan embroidery style image through a style transfer model, if the Hunan embroidery style image is used as the first image again, and the second image continues to be used as the second image for multiple rounds of style transfer operations, it is found that the original content of the first image is missing in the final Hunan embroidery style image, and the content of the second image is introduced instead. For details, see Figure 3 and Figure 4 . Figure 3 The top and bottom rows show the process of content leakage of two images from left to right.

[0117] In order to solve the existing content leakage, while taking into account the feature extraction capability and computational efficiency, the present embodiment provides a lightweight reversible network, which extracts content features in the first image and Hunan embroidery style features in the second image through the forward process of the lightweight reversible network, thereby extracting pure content features (i.e., first image features) and style features (i.e., second image features), maintaining the integrity of the features during the forward propagation process, and preventing content leakage problems, i.e., performing lossless style transfer.

[0118] In step S1240, global style transfer is performed based on the first image feature and the second image feature to obtain a first stylized feature.

[0119] In some embodiments, a biased transfer network may be used to perform global or local style transfer.

[0120] In some embodiments, the process of calculating the first stylized feature includes:

[0121]

[0122] Where AdaIN is the AdaIN migration function, F c is the first image feature, F s is the second image feature, It is the first stylized feature.

[0123] Although a biased transfer network can be used to perform global or local style transfer, the biased transfer network may cause content leakage. Therefore, the unbiased AdaIN transfer algorithm of this embodiment performs global style transfer to achieve lossless style transfer.

[0124] In step S1250, a second stylized feature corresponding to each semantic matching pair is calculated. In some embodiments, the process of calculating the second stylized feature includes:

[0125]

[0126] Among them, AdaIN is the AdaIN migration function, Dawn(m cmatch(j) )⊙F c Dawn cmatch(j) ) and F c The first pixel multiplication result, Dawn(m cmatch(j) ) is m cmatch(j) Downsampling, m cmatch(j) is the first semantic segmentation region mask j, F c is the first image feature, Dawn(m sj )⊙F s Dawn(m sj ) and F s The second pixel dot multiplication result, Dawn(m sj ) is m sj Downsampling, m sj is the second semantic segmentation region mask j, F s is the second image feature, m cmatch(j) The corresponding first semantic segmentation area and m sj The corresponding second semantic segmentation regions form a pair of semantic matching pairs j; is the second stylized feature corresponding to the semantic matching pair j.

[0127] The unbiased AdaIN migration algorithm of this embodiment performs local style migration and can achieve lossless style migration.

[0128] Because the texture, color, stitch and other style characteristics of different semantic objects in the second image (i.e., the digital Xiang embroidery image) are different, if the image style is transferred only from the perspective of global style characteristics without considering the semantic relationship between the objects in the second image and the first image, the final visual effect will be greatly affected.

[0129] In the above steps, the global first stylized features are extracted. Based on the above embodiments, the present embodiment extracts the local second stylized features corresponding to each semantic matching pair. The global first stylized features and the local second stylized features are used together in the image generation process, which can not only perform style transfer from an overall perspective, but also limit the transfer to the semantic matching area, especially transfer the style features such as needlework, color, texture, etc. in different semantic areas in the second image, so that the generated Hunan embroidery style image has more unique style characteristics and texture details of Hunan embroidery.

[0130] In step S1260, the first stylized feature and all the second stylized features are fused to obtain an overall stylized feature.

[0131] In some embodiments, the first stylized feature and all the second stylized features are fused to obtain an overall stylized feature, including:

[0132]

[0133] in, is the first stylized feature, To fuse all the features after the second stylized features, Concat is the channel concatenation function, avg is the average pooling function, and CONVS is the convolution function.

[0134] In this embodiment, an adaptive fusion process is designed to adaptively fuse the local second stylized feature and the global first stylized feature. The adaptive fusion process includes: firstly, the global first stylized feature and the second stylized feature of local region The concatenation is performed in the channel dimension, followed by an average pooling operation avg(.) and three connected convolution operations CONVS(·) for coordination and fusion.

[0135] In step S1270, a Hunan embroidery style image corresponding to the first image is generated according to the overall stylized features.

[0136] In some embodiments, this can be implemented by a common encoder-decoder structure.

[0137] In some embodiments, generating a Hunan embroidery style image corresponding to the first image according to the overall stylized features includes:

[0138] Inputting the overall stylized features into a lightweight reversible network to generate a Hunan embroidery style image corresponding to the first image through a reverse process of the lightweight reversible network;

[0139] The reverse process of the lightweight reversible network includes:

[0140] Y k =Concat(Y k [c+1:C],Y k [1:c]);

[0141] Y k [c+1:C]=Y k+1 [1:c]-H(Y k+1 [1:c]);

[0142] Y k [1:c]=Y k+1 [1:c];

[0143] Among them, the last reversible unit of the reverse process outputs the Hunan embroidery style image corresponding to the first image.

[0144] This embodiment not only uses the forward process of a lightweight reversible network for feature extraction, but also uses the reverse process of a lightweight reversible network for image reconstruction. The fully reversible design reduces the content and style leakage problems in the entire style transfer process.

[0145] After local style transfer, the style characteristics of each region will be more obvious. When there is a large difference in the semantics of two adjacent regions, the stitching, color, etc. of the adjacent regions will not be very coherent. Therefore, the local style transfer result (i.e., the second stylized feature) will produce visually incoherent edges between adjacent regions containing different object categories, affecting the overall visual effect. In order to coordinate the consistency of the styles of each region, in some embodiments, the loss function of the style transfer model is the content loss function Style loss function First consistency loss function and the second consistency loss function The weighted sum of

[0146]

[0147]

[0148] Where N is the total number of lightweight reversible networks, For a lightweight reversible network, from the first image I c Corresponding Xiang embroidery style image I cs The i-th layer features extracted from For a lightweight reversible network, from the first image I c The i-th layer features extracted from For a lightweight reversible network, from the second image I s The i-th layer features extracted from Features The mean of Features The mean of Features The variance of Features is the variance, ||·|| is the norm, ||·|| 2 is the 2-norm, I cc The second image I s and the first image I c The first image I is used c As the input generated Xiang embroidery style image, I ss The second image I s and the first image I c The second image I is used s The Xiang embroidery style image generated as input, For lightweight reversible networks from I cc The i-th layer features extracted from For lightweight reversible networks from I ss The i-th layer features extracted from .

[0149] Setting the content loss function The purpose is to preserve the content of the first image in the generated Xiang embroidery style image and use content loss to shorten the distance between the generated Xiang embroidery style image and the second image features.

[0150] Setting the style loss function The purpose is to enable the style transfer model to learn the ability to transfer style.

[0151] Set the first consistency loss function and the second consistency loss function The purpose is to enhance the model's ability to extract features so that the content and style features extracted by the model from the same image can be used to reconstruct the original image. At this time, the generated Xiang embroidery style image should be as consistent as possible with the original image (the first image).

[0152] like Figure 5An embodiment of the present application provides a method for style transfer of digital Hunan embroidery images based on regional semantics, and the method comprises the following steps:

[0153] Step S910: construct a style transfer model. The style transfer model includes: a first semantic segmentation network, a second semantic segmentation network, a CLIP network, a transfer network, an adaptive fusion network, and a lightweight reversible network.

[0154] Step S920, extracting the first image I c The first semantic segmentation region and the second image I s The first image is a content image used for training, and the second image is a digital Xiang embroidery image used for training.

[0155] The first image I c Input into the first semantic segmentation network to obtain the semantic region mask and the label corresponding to each region:

[0156] m c ={m c1 , m c2 , ..., m ci , ..., m cn};

[0157] CL={C 1L , C 2L , ..., C iL , ..., C nL};

[0158] Among them, cn is the total segmentation area, m ci represents the mask of the i-th first semantic segmentation region, and iL represents the semantic category corresponding to the i-th first semantic segmentation region.

[0159] The second image I s Input into the second semantic segmentation network (ie SAM) to obtain the second semantic segmentation region mask:

[0160] m s ={m s1 , m s2 , ..., m sj , ..., m so};

[0161] so is the total number of partitioned regions of SAM.

[0162] After obtaining the masks of each region, the mask and the image are multiplied pixel by pixel to obtain the segmented region corresponding to the image.

[0163] The first semantic segmentation region set can be expressed as:

[0164] C={C 1 , C 2 , ..., C i , ..., C n};

[0165] Where n is the total number of segmented regions. i The formula is:

[0166] C i =m ci ⊙I c ;

[0167] The second semantic segmentation region set can be expressed as:

[0168] S={S 1 , S 2 , …, S j , …, S m};

[0169] Where m is the total number of segmented areas. The formula is:

[0170] S j =m sj ⊙I s ;

[0171] Step S930, input the second semantic segmentation region set S obtained by the second semantic segmentation network and all the semantic labels CL of the first semantic segmentation region set C into the CLIP network, and obtain the semantic label with the highest probability output by the CLIP network, which is the final matching result.

[0172] The specific formula is:

[0173]

[0174] So far, we have found the second semantic segmentation area S j The label C corresponding to the first semantic segmentation region that is semantically closest match(j)L , so we can get C match(j)L The corresponding first semantic segmentation area C match(j) , and finally we can get the semantic matching pair P j =(C match(j) , S j ).

[0175] For each second semantic segmentation region S in the second semantic segmentation region set S j Repeat the above process to obtain the set P of all semantic matching pairs.

[0176] Step S940: The first image I c and the second image Is Input into a lightweight reversible network.

[0177] The lightweight reversible network consists of designed reversible units, which can losslessly extract image features through the radial changes of the forward process of the reversible units.

[0178] Specific operations include:

[0179] Y k+1 [c+1:C]=Y k [c+1:C]+H(Y k [1:c]);

[0180] Y k+1 [1:c]=Y k [1:c];

[0181] Y k+1 =Concat(Y k+1 [c+1:C],Y k+1 [1:c]);

[0182] The explanation of the relevant characters of the formula has been introduced in the above-mentioned implementation mode and will not be described in detail here;

[0183] The extraction ability of reversible units is largely related to the extraction ability of H. If the H function is too simple, it will affect the feature extraction ability of the model, and the complex mapping function will also increase the computational complexity of the model. Therefore, while considering both the extraction ability and the computational complexity, the component module of the lightweight network SqueezeNet is used as H.

[0184] During each forward propagation, the features of the [1:c] channel are not extracted, which will seriously affect the model's extraction ability. Therefore, before each feature is input into the lightweight reversible network, all channels will be randomly shuffled, and the value of c is set to a different number for each module, further improving the lightweight reversible network's ability to learn information from different channels, thereby improving the network's extraction ability. Subsequently, all the features of [1:C] are concatenated together and sent to a reversible unit. The extraction process of the lightweight reversible network can be expressed as:

[0185] Repeat the above process until the last reversible unit outputs the first image feature F of the required first image c and the second image feature F of the second image s .

[0186] Step S930: Use the migration network to perform style migration between semantic regions and global style migration.

[0187] The biased transfer network will lead to content leakage problems, because the unbiased AdaIN transfer network is used for style transfer during the transfer process. Therefore, let the first stylized feature be The calculation formula can be expressed as:

[0188]

[0189] For the semantic matching pair P j =(C match(j) , S j ), the second stylized feature of its region It can be expressed as:

[0190]

[0191] Repeat the above operation to get the second stylized feature.

[0192] Step S960: Adaptively fuse the first stylized feature and all second stylized features Right now

[0193]

[0194] Step S970: transform the overall stylized feature F cs In the reverse process of the lightweight reversible network fed into it.

[0195] This process is the inverse transformation of the affine transformation in the forward propagation process, and the entire process is completely reversible.

[0196] Its operation can be expressed as:

[0197] Y k [c+1:C]=Y k+1 [1:c]-H(Y k+1 [1:c]);

[0198] Y k [1:c]=Y k+1 [1:c];

[0199] Y k =Concat(Y k [c+1:C],Y k [1:c]);

[0200] In the lightweight reversible units of the model stack, the above affine transformation inverse process is repeated until the last reversible unit can output the final Xiang embroidery style image I cs .

[0201] During the training process, it is necessary to calculate the gradient of the loss function and use the back-propagation algorithm to update the parameters in the network.

[0202]

[0203] To enable the model to learn the ability to transfer style, style loss is used:

[0204]

[0205] In order to enhance the model's ability to extract features, consistency loss is used on the first image and the second image respectively, as follows:

[0206]

[0207] Step S980, model application.

[0208] For example, the target content image and the target digital Hunan embroidery image are input into a preset style transfer model, and the Hunan embroidery style image corresponding to the target content image output by the style transfer model is obtained.

[0209] This method first obtains the semantic segmentation region of the content image with semantic labels, and uses SAM with strong zero-sample segmentation ability to segment the digital Xiang embroidery image to obtain the semantic segmentation region of the unlabeled digital Xiang embroidery image. Subsequently, the CLIP model is used to match the semantic segmentation region of the content image with the semantic segmentation region of the digital Xiang embroidery image based on the cosine distance between the features to obtain a region-based semantic matching pair. At the same time, a lightweight reversible network is used to extract the features of the content image and the features of the digital Xiang embroidery image using the forward process of the reversible network. Subsequently, guided by the mask image, style transfer and fusion are performed between the semantic segmentation region of the matching content image and the semantic segmentation region of the digital Xiang embroidery image in the semantic matching pair to obtain local stylized features. In this way, based on the semantic information of the semantic segmentation region of the content image, the most suitable needle method, color, etc. are found from the digital Xiang embroidery image to achieve style transfer for different regions, which is also very suitable for the creation process of Xiang embroidery. Next, the features of the extracted content image and the features of the digital Xiang embroidery image are used again for global style transfer, and the global stylized features are obtained. Finally, the global stylized features are adaptively fused with the local stylized features to coordinate the consistency of the styles of each region. Finally, the overall stylized features are fused and converted into the final Xiang embroidery style image through the inverse process in the lightweight reversible network that is completely reversible with the forward process, thereby losslessly realizing style transfer in conjunction with the forward process of the reversible network and solving the content leakage problem.

[0210] like Figure 6As shown, an embodiment of the present application provides a style transfer device for digital Hunan embroidery images based on regional semantics, the device comprising:

[0211] The data acquisition module 1100 is used to acquire the target content image and the target digital Xiang embroidery image;

[0212] The style transfer module 1200 is used to input the target content image and the target digital Xiang embroidery image into a preset style transfer model to obtain a Xiang embroidery style image corresponding to the target content image output by the style transfer model; the training process of the style transfer model includes:

[0213] Extracting a plurality of first semantic segmentation regions from a first image and extracting a plurality of second semantic segmentation regions from a second image; the first image is a content image for training, and the second image is a digital Xiang embroidery image for training;

[0214] Semantically matching the plurality of first semantic segmentation regions with the plurality of second semantic segmentation regions to obtain a plurality of semantic matching pairs; any semantic matching pair includes: a first semantic segmentation region and the second semantic segmentation region closest thereto;

[0215] extracting a first image feature of the first image and extracting a second image feature of the second image;

[0216] Performing global style transfer according to the first image feature and the second image feature to obtain a first stylized feature;

[0217] Calculate a first pixel dot product result of a first semantic segmentation region mask and a first image feature corresponding to a first semantic segmentation region in each semantic matching pair, and a second pixel dot product result of a second semantic segmentation region mask and a second image feature corresponding to a second semantic segmentation region, and perform regional style migration according to the first pixel dot product result and the second pixel dot product result to obtain a second stylized feature corresponding to each semantic matching pair;

[0218] Combining the first stylized feature with all the second stylized features to obtain an overall stylized feature;

[0219] A Hunan embroidery style image corresponding to the first image is generated according to the overall stylized features.

[0220] It should be noted that the present device embodiment and the above method embodiment are based on the same inventive concept, so the content of the above method embodiment is also applicable to the present device embodiment and will not be repeated here.

[0221] like Figure 7 The embodiment of the present application further provides an electronic device, the electronic device comprising:

[0222] at least one memory;

[0223] at least one processor;

[0224] at least one program;

[0225] The program is stored in the memory, and the processor executes at least one program to implement the above-mentioned style transfer method of digital Hunan embroidery images based on regional semantics implemented in the present disclosure.

[0226] The electronic device may be any intelligent terminal including a mobile phone, a tablet computer, a personal digital assistant (PDA), a vehicle-mounted computer, etc.

[0227] The electronic device according to the embodiment of the present application is described in detail below.

[0228] The processor 1600 may be implemented by a general-purpose central processing unit (CPU), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present invention.

[0229] The memory 1700 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 1700 can store an operating system and other application programs. When the technical solution provided in the embodiment of this specification is implemented by software or firmware, the relevant program code is stored in the memory 1700, and the processor 1600 calls and executes the style transfer method of digital Hunan embroidery images based on regional semantics in the embodiment of the present invention.

[0230] Input / output interface 1800, used to implement information input and output;

[0231] The communication interface 1900 is used to realize the communication interaction between the device and other devices. The communication can be realized through a wired manner (such as USB, network cable, etc.) or a wireless manner (such as mobile network, WIFI, Bluetooth, etc.);

[0232] Bus 2000 , which transmits information between various components of the device (e.g., processor 1600 , memory 1700 , input / output interface 1800 , and communication interface 1900 );

[0233] The processor 1600 , the memory 1700 , the input / output interface 1800 , and the communication interface 1900 are connected to each other in communication within the device via the bus 2000 .

[0234] An embodiment of the present invention further provides a storage medium, which is a computer-readable storage medium. The computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are used to enable a computer to execute the above-mentioned style transfer method of digital Hunan embroidery images based on regional semantics.

[0235] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0236] The embodiments described in the present invention are intended to more clearly illustrate the technical solutions of the embodiments of the present invention and do not constitute a limitation on the technical solutions provided by the embodiments of the present invention. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of the present invention are also applicable to similar technical problems.

[0237] Those skilled in the art will appreciate that the technical solutions shown in the figures do not limit the embodiments of the present invention and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0238] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0239] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0240] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0241] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0242] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0243] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0244] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions to enable an electronic device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of each embodiment method of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, ROM), random access memory (Random Access Memory, RAM), disk or optical disk and other media that can store programs.

[0245] The above is a specific description of the preferred implementation of the embodiments of the present application, but the embodiments of the present application are not limited to the above-mentioned implementation methods. Technical personnel familiar with the field can also make various equivalent modifications or substitutions without violating the spirit of the embodiments of the present application. These equivalent modifications or substitutions are all included in the scope defined by the claims of the embodiments of the present application.

Claims

1. A style transfer method for digital Hunan embroidery images based on regional semantics, characterized in that: The method comprises: Acquire a target content image and a target digital Xiang embroidery image; The target content image and the target digital Xiang embroidery image are input into a preset style transfer model, and the style transfer model outputs a Xiang embroidery style image corresponding to the target content image; the training process of the style transfer model includes: Extracting a plurality of first semantic segmentation regions from a first image and extracting a plurality of second semantic segmentation regions from a second image; the first image is a content image for training, and the second image is a digital Xiang embroidery image for training; Perform semantic matching on the plurality of the first semantic segmentation regions and the plurality of the second semantic segmentation regions to obtain a plurality of semantic matching pairs; any one of the semantic matching pairs includes: one of the first semantic segmentation regions and the second semantic segmentation region closest thereto; extracting a first image feature of the first image and extracting a second image feature of the second image; Performing global style migration according to the first image feature and the second image feature to obtain a first stylized feature; Calculate a first pixel dot product result of a first semantic segmentation region mask corresponding to a first semantic segmentation region in each of the semantic matching pairs and the first image feature, and a second pixel dot product result of a second semantic segmentation region mask corresponding to a second semantic segmentation region and the second image feature, and perform regional style migration according to the first pixel dot product result and the second pixel dot product result to obtain a second stylized feature corresponding to each of the semantic matching pairs; fusing the first stylized feature and all the second stylized features to obtain an overall stylized feature; A Hunan embroidery style image corresponding to the first image is generated according to the overall stylized features.

2. The style transfer method of digital Hunan embroidery images based on regional semantics according to claim 1 is characterized in that: The extracting a first image feature of the first image and extracting a second image feature of the second image comprises: Inputting the first image into a lightweight reversible network to extract first image features of the first image through a forward process of the lightweight reversible network; Inputting the second image into the lightweight reversible network to extract second image features of the second image through a forward process of the lightweight reversible network; The lightweight reversible network includes a plurality of stacked reversible units, and the forward process of the lightweight reversible network includes: ； ； ； in, For the The output of the reversible unit, is the channel splicing function, is the 1st to cth layer channel of any reversible unit, For any reversible unit, For the Reversible unit in The output, For the Reversible unit in The output, is the mapping function.

3. The style transfer method of digital Hunan embroidery images based on regional semantics according to claim 2 is characterized in that: Generating a Hunan embroidery style image corresponding to the first image according to the overall stylized features includes: Inputting the overall stylized features into the lightweight reversible network to generate a Hunan embroidery style image corresponding to the first image through a reverse process of the lightweight reversible network; The reverse process of the lightweight reversible network includes: ； ； ； Among them, the last reversible unit of the reverse process outputs the Hunan embroidery style image corresponding to the first image.

4. The style transfer method of digital Hunan embroidery images based on regional semantics according to claim 3 is characterized in that: The loss function of the style transfer model is the content loss function , style loss function , the first consistency loss function and the second consistency loss function The weighted sum of ； ； ； ； in, is the total number of lightweight reversible networks, For a lightweight reversible network from the first image Corresponding Xiang embroidery style images Extracted from Layer features, For a lightweight reversible network from the first image Extracted from Layer features, For a lightweight reversible network from the second image Extracted from Layer features, Features The mean of Features The mean of Features The variance of Features The variance of is the norm, is the 2-norm, For the second image and the first image Use the first image The Xiang embroidery style image generated as input, For the second image and the first image Use the second image The Xiang embroidery style image generated as input, For lightweight reversible networks Extracted from Layer features, For lightweight reversible networks Extracted from Layer features.

5. The style transfer method of digital Hunan embroidery images based on regional semantics according to claim 1 is characterized in that: The step of extracting a plurality of first semantic segmentation regions of the first image and a plurality of second semantic segmentation regions of the second image comprises: Extracting a plurality of first semantic segmentation region masks of the first image according to the first semantic segmentation network; Extracting a plurality of second semantic segmentation region masks of the second image through a second semantic segmentation network; the second semantic segmentation network is a deep learning SAM network; Performing pixel-by-pixel multiplication of the first semantic segmentation region masks with the first image to obtain a plurality of first semantic segmentation regions; Perform pixel-by-pixel multiplication of the second image with the plurality of second semantic segmentation region masks to obtain a plurality of second semantic segmentation regions.

6. The style transfer method of digital Hunan embroidery images based on regional semantics according to claim 5 is characterized in that: The process of calculating the first stylized feature includes: ； in, is the AdaIN migration function, is the first image feature, is the second image feature, It is the first stylized feature.

7. The method for style transfer of digital Hunan embroidery images based on regional semantics according to claim 5 is characterized in that: The process of calculating the second stylized feature includes: ； in, is the AdaIN migration function, for and The first pixel multiplication result is for Downsampling of is the first semantic segmentation region mask , is the first image feature, for and The second pixel dot multiplication result is for Downsampling of is the second semantic segmentation region mask , is the second image feature, The corresponding first semantic segmentation area and The corresponding second semantic segmentation regions form a pair of semantic matching pairs ; is a semantic matching pair The corresponding second stylized feature.

8. The method for style transfer of digital Hunan embroidery images based on regional semantics according to claim 5 is characterized in that: The performing semantic matching on the plurality of the first semantic segmentation regions and the plurality of the second semantic segmentation regions to obtain a plurality of semantic matching pairs includes: Inputting the first semantic labels corresponding to the plurality of second semantic segmentation regions and the plurality of first semantic segmentation regions into the multimodal CLIP network to obtain a plurality of matching results output by the multimodal CLIP network, wherein the process of generating any matching result by the multimodal CLIP network includes: ； in, is the first semantic segmentation region The corresponding first semantic label, is the second semantic segmentation area , for and The matching results of To obtain the maximum value function, It is a multimodal CLIP network function; The first semantic label in each of the matching results is replaced with the corresponding first semantic segmentation region to obtain a plurality of semantic matching pairs.

9. The method for style transfer of digital Hunan embroidery images based on regional semantics according to claim 5, characterized in that: The fusing of the first stylized feature and all the second stylized features to obtain an overall stylized feature includes: ； in, is the first stylized feature, To fuse all the features after the second stylized features, is the channel concatenation function, is the average pooling function, is the convolution function.

10. A style transfer device for digital Hunan embroidery images based on regional semantics, characterized in that: The device comprises: A data acquisition module is used to acquire a target content image and a target digital Xiang embroidery image; The style transfer module is used to input the target content image and the target digital Hunan embroidery image into a preset style transfer model, and obtain the Hunan embroidery style image corresponding to the target content image output by the style transfer model; the training process of the style transfer model includes: Extracting a plurality of first semantic segmentation regions from a first image and extracting a plurality of second semantic segmentation regions from a second image; the first image is a content image for training, and the second image is a digital Xiang embroidery image for training; Perform semantic matching on the plurality of the first semantic segmentation regions and the plurality of the second semantic segmentation regions to obtain a plurality of semantic matching pairs; any one of the semantic matching pairs includes: one of the first semantic segmentation regions and the second semantic segmentation region closest thereto; extracting a first image feature of the first image and extracting a second image feature of the second image; Performing global style migration according to the first image feature and the second image feature to obtain a first stylized feature; Calculate a first pixel dot product result of a first semantic segmentation region mask corresponding to a first semantic segmentation region in each of the semantic matching pairs and the first image feature, and a second pixel dot product result of a second semantic segmentation region mask corresponding to a second semantic segmentation region and the second image feature, and perform regional style migration according to the first pixel dot product result and the second pixel dot product result to obtain a second stylized feature corresponding to each of the semantic matching pairs; fusing the first stylized feature and all the second stylized features to obtain an overall stylized feature; A Hunan embroidery style image corresponding to the first image is generated according to the overall stylized features.

Images