Image processing method, device, equipment, storage medium and program product

Abstract

The present application provides an image processing method, apparatus, device, storage medium, and program product. The method includes: obtaining an image to be processed and semantic mask information of the image to be processed; combining the semantic mask information of the image to be processed with the image to be processed to obtain a target semantic mask instance; obtaining instance semantic information in the target semantic mask instance based on a preset CLIP network and the target semantic mask instance; performing a similarity comparison between the instance semantic information and the comparison instance semantic information to determine the target mask instance in the instance semantic information; and determining the target image based on the target mask instance and the image to be processed. The method of the present application improves the image quality of the original image and the image processing efficiency.

Description

Technical Field

[0001] The present application relates to the field of image processing, and in particular to an image processing method, apparatus, device, storage medium, and program product. Background Art

[0002] Image enhancement is a method of image processing. Image enhancement processes the original image through a series of operations to improve its visual effects or make it more suitable for specific application requirements.

[0003] Existing technologies typically use spatial domain or transform domain methods for image enhancement. Spatial domain image enhancement methods include grayscale transformation, histogram adjustment, and filtering. Transform domain image enhancement methods modify the image's transform coefficients within a certain transform domain and then obtain the enhanced image through an inverse transform. Common transform domains used in these methods include the frequency domain and the wavelet domain.

[0004] However, when performing image enhancement using the existing technology, the original image quality is not high, resulting in poor image enhancement effect. Summary of the Invention

[0005] The present application provides an image processing method, apparatus, device, storage medium and program product to solve the problem of poor image enhancement effect caused by low original image quality.

[0006] In a first aspect, the present application provides an image processing method, comprising:

[0007] Obtaining an image to be processed and semantic mask information of the image to be processed;

[0008] Combining the semantic mask information of the image to be processed with the image to be processed to obtain a target semantic mask instance;

[0009] According to the preset CLIP network and the target semantic mask instance, the instance semantic information in the target semantic mask instance is obtained;

[0010] Performing a similarity comparison between the instance semantic information and the comparison instance semantic information to determine a target mask instance in the instance semantic information, wherein the comparison instance semantic information is obtained based on the CLIP network and the comparison semantic mask instance, and the comparison semantic mask instance is obtained based on the semantic mask information of the comparison image and the comparison image, wherein when the image to be processed is a style image, the comparison image is a content image, and when the image to be processed is a content image, the comparison image is a style image;

[0011] According to the target mask instance and the image to be processed, the target image is obtained.

[0012] In this application, a similarity comparison process is performed on the instance semantic information and the comparison instance semantic information to determine the target mask instance in the instance semantic information, including:

[0013] Determine an instance semantic vector group according to a target semantic mask instance of the instance semantic information, wherein the instance semantic vector group includes an instance query vector and an instance key vector;

[0014] Determining a comparison instance semantic vector group according to a target comparison semantic mask instance of the comparison instance semantic information, wherein the comparison instance semantic vector group includes a comparison instance query vector and a comparison instance key vector;

[0015] Determining, based on the instance semantic vector group and the comparison instance semantic vector group, a similarity between a first target vector in the instance semantic vector group and a second target vector in the comparison instance semantic vector group, wherein when the first target vector is an instance query vector, the second target vector is a comparison instance key vector; and when the first target vector is an instance key vector, the second target vector is a comparison instance query vector;

[0016] Determining mask similarity based on a similarity between a first target vector in the instance semantic vector group and a second target vector in the comparison instance semantic vector group;

[0017] According to the preset similarity requirement and mask similarity, the target mask instance in the instance semantic information is determined.

[0018] In the present application, when the target image is a content image, after obtaining the target image according to the target mask instance and the image to be processed, the method further includes:

[0019] The target image is input into the content encoder network and the preset HRNet network respectively to obtain the initial multi-scale content features and multi-scale semantic features;

[0020] Inputting the initial multi-scale content features, the multi-scale semantic features, and the initial multi-scale style features into the instance normalization module of the decoder network to obtain a target instance, wherein the initial multi-scale style features are obtained based on the target comparison image and the style encoder network, the target comparison image is determined based on the target comparison mask instance in the comparison instance semantic information and the comparison image, and the target comparison mask instance is a comparison mask instance determined after performing similarity comparison processing on the comparison instance semantic information and the instance semantic information;

[0021] The target instance is input into the upsampling layer of the decoder network to obtain the target transformed image.

[0022] In this application, the initial multi-scale content features, multi-scale semantic features, and initial multi-scale style features are input into the instance normalization module of the decoder network to obtain the target instance, including:

[0023] The multi-scale semantic features and the initial multi-scale style features are input into the semantic enhancement style network of the instance normalization module to obtain the semantically enhanced multi-scale style features.

[0024] Input the initial multi-scale content features and the initial multi-scale style features into the content-enhanced style network of the instance normalization module to obtain content-enhanced multi-scale style features;

[0025] Obtain target multi-scale style features based on semantically enhanced multi-scale style features and content-enhanced multi-scale style features;

[0026] According to the initial multi-scale content features and the target multi-scale style features, the target instance is obtained.

[0027] In this application, the target multi-scale style features are obtained based on the semantically enhanced multi-scale style features and the content enhanced multi-scale style features, including:

[0028] Determine style weights based on content-enhanced style networks and semantic-enhanced style networks;

[0029] The target multi-scale style feature is determined according to the style weight, the semantically enhanced multi-scale style feature and the content enhanced multi-scale style feature.

[0030] In this application, a target instance is obtained based on the initial multi-scale content features and the target multi-scale style features, including:

[0031] Input the initial multi-scale content features into the content normalization network of the instance normalization module for feature extraction to obtain the target multi-scale content features;

[0032] A target instance is obtained according to the target multi-scale content features and the target multi-scale style features.

[0033] In a second aspect, the present application provides an image processing device, comprising:

[0034] An acquisition module, used to acquire an image to be processed and a semantic mask of the image to be processed;

[0035] A calculation module is used to combine the semantic mask information of the image to be processed with the image to be processed to obtain a target semantic mask instance;

[0036] A deep learning module is used to obtain instance semantic information in the target semantic mask instance based on a preset CLIP network and the target semantic mask instance;

[0037] a comparison module for performing a similarity comparison between the instance semantic information and the comparison instance semantic information, and determining a target mask instance in the instance semantic information, wherein the comparison instance semantic information is obtained based on the CLIP network and the comparison semantic mask instance, and the comparison semantic mask instance is obtained based on the semantic mask information of the comparison image and the comparison image, wherein when the image to be processed is a style image, the comparison image is a content image, and when the image to be processed is a content image, the comparison image is a style image;

[0038] The execution module is used to obtain according to the target mask instance and the image to be processed.

[0039] In a third aspect, the present application provides an electronic device, comprising: a processor, and a memory communicatively connected to the processor;

[0040] Memory stores computer-executable instructions;

[0041] The processor executes the computer-executable instructions stored in the memory to implement the method of the present application.

[0042] In a fourth aspect, the present application provides a computer-readable storage medium, comprising: computer-readable storage medium storing computer execution instructions, which are used to implement the method of the present application when executed by a processor.

[0043] In a fifth aspect, an embodiment of the present application provides a computer program product, including a computer program, which implements the method of the present application when executed by a processor.

[0044] The present application provides an image processing method, apparatus, device, storage medium and program product, which obtains an image to be processed and semantic mask information of the image to be processed; combines the semantic mask information of the image to be processed with the image to be processed to obtain a target semantic mask instance; obtains instance semantic information in the target semantic mask instance based on a preset CLIP network and the target semantic mask instance; performs similarity comparison processing on the instance semantic information and the comparison instance semantic information to determine the target mask instance in the instance semantic information, the comparison instance semantic information is obtained based on the CLIP network and the comparison semantic mask instance, and the comparison semantic mask instance is based on the target semantic mask instance. According to the semantic mask information of the comparison image and the comparison image, when the image to be processed is a style image, the comparison image is a content image, and when the image to be processed is a content image, the comparison image is a style image; according to the target mask instance and the image to be processed, a means of obtaining the target image is obtained by superimposing the pixel-level features of the image to be processed with the original image to be processed to obtain a target semantic mask instance, extracting the instance semantic information of the target semantic mask instance, and performing similarity comparison processing with the features in the instance semantic information of the comparison image. According to the comparison result, a target image whose image quality and image features meet the requirements is obtained, thereby improving the image quality of the target image. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0046] Figure 1 A schematic diagram of a scene of an image processing method provided in an embodiment of the present application;

[0047] Figure 2 A flowchart of an image processing method provided in an embodiment of the present application;

[0048] Figure 3 A flowchart of another image processing method provided in an embodiment of the present application;

[0049] Figure 4 A network structure diagram of an example normalization module provided in an embodiment of the present application;

[0050] Figure 5 A flowchart of another image processing method provided in an embodiment of the present application;

[0051] Figure 6 A schematic structural diagram of an image processing device provided in an embodiment of the present application;

[0052] Figure 7 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application.

[0053] The above drawings illustrate specific embodiments of the present application, which will be described in more detail below. These drawings and the textual description are not intended to limit the scope of the present application in any way, but rather to illustrate the concepts of the present application to those skilled in the art by reference to specific embodiments. DETAILED DESCRIPTION

[0054] Exemplary embodiments will be described in detail herein, with examples illustrated in the accompanying drawings. In the following description, when referring to the drawings, identical numerals in different figures represent identical or similar elements, unless otherwise indicated. The embodiments described in the following exemplary embodiments are not intended to represent all embodiments consistent with the present application. Rather, they are merely examples of apparatus and methods consistent with certain aspects of the present application, as detailed in the appended claims.

[0055] Existing image preprocessing techniques, such as image enhancement and style transfer, typically involve image resizing, noise removal, pixel value normalization, and outlier identification and removal. Alternatively, images may be grayscaled, histogram equalized, sharpened, or filtered. However, existing image preprocessing techniques fail to improve image quality, resulting in poor performance in subsequent image operations.

[0056] In order to solve the problem that the existing technology cannot improve the image quality after image preprocessing, resulting in poor results of subsequent image operations, the present application performs pixel-level feature extraction on the image to be processed, and superimposes it with the original image to obtain the target semantic mask instance of the image, extracts the instance semantic information of the target semantic mask instance, and performs similarity comparison processing with the instance semantic information of the comparison image. Based on the similarity comparison results, the target image is generated, thereby improving the image quality of the target image and the subsequent processing efficiency.

[0057] The present application provides an image processing method, which aims to solve the above technical problems in the prior art.

[0058] The following specific embodiments describe in detail the technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0059] Figure 1 A schematic diagram of a scene of an image processing method provided in an embodiment of the present application. Figure 1 As shown, the execution subject of the method can be an image processing system, which includes: an image execution module, an image encoding module, an image decoding module, and a discriminator module.

[0060] The image execution module may refer to a unit that performs a series of processing on the image to improve the performance and accuracy of the model before inputting the image into the model for training or inference. In this application, the image execution module is intended to extract pixel-level semantic features of the image, generate mask instances to enhance the original input image, and obtain the target image.

[0061] The image encoding module can refer to a module used to extract features from an image. In this application, the image encoding module includes a content encoding network and a style encoding network. The content encoding network and the style encoding network are two identical network structures designed to extract content or style feature representations at different levels. The module consists of multiple Swin Transformer structural blocks and downsampling operations. Each layer produces a feature image of a specific size. The width and height are halved compared to the previous layer, but the channels are expanded. This contains high-level features of the content image or style reference image.

[0062] The image decoding module may refer to a module that generates a target image based on input features. In this application, the image decoding module and the image encoding module have completely symmetrical or opposite structures, aiming to consistently match the content or style feature representations at different levels. The module contains multiple Swin Transformer structural blocks and upsampling operations. Each layer will obtain a feature image of a specific size, with the width and height doubled compared to the previous layer but the number of channels reduced. The entire generation process is carried out in a coarse-to-fine manner, and the content or style features are injected layer by layer and effectively fused.

[0063] The discriminator module may refer to a module that judges the effect of the generated image, and the discriminator also acts as a feature extractor to optimize the feature matching distance. In an embodiment of the present application, the discriminator module includes three conventional discriminators with the same architecture to distinguish images of different scales, and its structure includes multiple residual convolution blocks and downsampling operations. The discriminator trained at the coarsest scale has the largest receptive field and captures the global information of the image. The discriminator trained at the finest scale has the smallest receptive field, which enables the generated target image to produce better details, further improving the robustness of the model in different conversion tasks.

[0064] In the embodiment of the present application, the image encoding module inputs the image to be processed into the pre-trained HRNet network for semantic segmentation to obtain a predicted semantic mask, enhances the input image according to the predicted semantic mask, inputs the enhanced image into the pre-trained CLIP network to obtain instance semantic information, performs similarity matching on the instance semantic information of the image to be processed and the comparison instance semantic information of the comparison image, determines the target mask instance, determines the target image according to the target mask instance and the image to be processed, and sends the target image to the image encoding module. When the target image is a content image and the comparison image is a style image, the image encoding module obtains initial multi-scale content features and multi-scale semantic features according to the content image, obtains initial multi-scale style features according to the style image, and sends the initial multi-scale content features, multi-scale semantic features, and initial multi-scale style features to the image decoding module. The image decoding module determines the target conversion image according to the initial multi-scale content features, multi-scale semantic features, and initial multi-scale style features. The discriminator module judges the authenticity of the target conversion image and outputs the target conversion image when the preset number of iterations is reached.

[0065] The execution subject of an image processing method provided in an embodiment of the present application can also be a server. Among them, the server can be a computer, tablet or other device. This embodiment does not impose any special restrictions on the implementation method of the execution subject, as long as the execution subject can obtain the image to be processed and the semantic mask information of the image to be processed; the semantic mask information of the image to be processed and the image to be processed are combined and processed to obtain a target semantic mask instance; according to the preset CLIP network and the target semantic mask instance, the instance semantic information in the target semantic mask instance is obtained; the instance semantic information and the comparison instance semantic information are compared for similarity to determine the target mask instance in the instance semantic information, the comparison instance semantic information is obtained according to the CLIP network and the comparison semantic mask instance, and the comparison semantic mask instance is obtained according to the semantic mask information of the comparison image and the comparison image, wherein, when the image to be processed is a style image, the comparison image is a content image, and when the image to be processed is a content image, the comparison image is a style image; according to the target mask instance and the image to be processed, the target image can be obtained.

[0066] Image processing refers to the process of analyzing and modifying image data to improve image quality, extract features, enhance information, or transform image representations to meet specific application requirements. Image processing technology is widely used in a variety of fields, including medical imaging, satellite imaging, video surveillance, photography, machine vision, and art design.

[0067] Figure 2 This is a flow chart of an image processing method provided in an embodiment of the present application. The execution subject of this method can be a server or other server, and this embodiment does not make any special restrictions here, such as Figure 2 As shown, the method may include:

[0068] S201: Acquire an image to be processed and semantic mask information of the image to be processed.

[0069] The image to be processed may refer to an original image that needs to be enhanced or transformed in an image processing task.

[0070] Semantic mask information can refer to a binary matrix used to represent the semantic category to which each pixel in an image belongs. In semantic segmentation tasks, the model outputs a mask of the same size as the input image, where each pixel value corresponds to a specific semantic category label. A mask can refer to an image or data structure that indicates the presence or attributes of certain regions in an image.

[0071] The method for obtaining the semantic mask information of the image to be processed may be: inputting the image to be processed into a pre-trained HRNet network for semantic segmentation, and the model outputs the semantic mask information of the image to be processed.

[0072] The High-Resolution Network (HRNet) is a deep neural network architecture used for computer vision tasks, primarily for tasks requiring high-resolution feature representation, such as image segmentation, pose estimation, and object detection. HRNet maintains high-resolution feature maps throughout the network, improving accuracy and detail preservation. In semantic segmentation tasks, HRNet is capable of producing high-quality segmentation results.

[0073] S202 : Combining the semantic mask information of the image to be processed with the image to be processed to obtain a target semantic mask instance.

[0074] The target semantic mask instance may refer to a specific object or instance separated from the entire semantic mask and corresponding to the image, specifically referring to a single instance or object in a certain semantic category.

[0075] The semantic mask information is superimposed on the image to be processed, and the category label information in the semantic mask information is mapped to each pixel of the image to be processed, thereby obtaining a target semantic mask instance.

[0076] S203 : Obtain instance semantic information in the target semantic mask instance according to the preset CLIP network and the target semantic mask instance.

[0077] Among them, the CLIP network (Contrastive Language-Image Pre-training) can refer to a model that maps images and text into the same embedding space to learn shared representations between them. The core innovation of CLIP lies in mapping images and text into a unified vector space and pre-training them through contrastive learning, so that the model can directly calculate the similarity between images and text in the vector space without the need for additional intermediate representations.

[0078] The target semantic mask instance of the image to be processed is input into the pre-trained CLIP network, and after feature extraction and normalization processing by the image encoder of the CLIP network, a deep semantic representation of the image to be processed, namely the instance semantic information in the target semantic mask instance, is obtained.

[0079] S204. Perform a similarity comparison process on the instance semantic information and the comparison instance semantic information to determine a target mask instance in the instance semantic information. The comparison instance semantic information is obtained based on the CLIP network and the comparison semantic mask instance. The comparison semantic mask instance is obtained based on the semantic mask information of the comparison image and the comparison image. When the image to be processed is a style image, the comparison image is a content image. When the image to be processed is a content image, the comparison image is a style image.

[0080] The semantic information of the comparison instance can be a deep semantic representation of the comparison image. The comparison image and the image to be processed are of different types, and the comparison image can be a style image or a content image. For example, when the image to be processed is a content image, the comparison image is a style image; when the image to be processed is a style image, the comparison image is a content image.

[0081] Among them, the method for determining the semantic information of the comparison instance can be to input the comparison image into the pre-trained HRNet network to obtain the semantic mask information of the comparison image, superimpose the semantic mask information of the comparison image and the comparison image to obtain a comparison semantic mask instance, and input the comparison semantic mask instance into the pre-trained CLIP network to obtain a deep semantic representation of the comparison image, that is, the process of comparing the semantic information of the instance.

[0082] Similarity can be a measure of the intercorrelation of features between object instances represented by different masks. Instances with intercorrelated features are sorted based on similarity. Among the mask instances in the image to be processed, the mask instance that best matches the semantic features of the comparison mask instance is selected as the target instance and used to enhance the image to be processed, thereby improving the image quality of the image to be processed.

[0083] In the embodiment of the present application, performing a similarity comparison process on the instance semantic information and the comparison instance semantic information to determine the target mask instance in the instance semantic information includes:

[0084] Determine an instance semantic vector group according to a target semantic mask instance of the instance semantic information, wherein the instance semantic vector group includes an instance query vector and an instance key vector;

[0085] Determining a comparison instance semantic vector group according to a target comparison semantic mask instance of the comparison instance semantic information, wherein the comparison instance semantic vector group includes a comparison instance query vector and a comparison instance key vector;

[0086] Determining, based on the instance semantic vector group and the comparison instance semantic vector group, a similarity between a first target vector in the instance semantic vector group and a second target vector in the comparison instance semantic vector group, wherein when the first target vector is an instance query vector, the second target vector is a comparison instance key vector; and when the first target vector is an instance key vector, the second target vector is a comparison instance query vector;

[0087] Determining mask similarity based on a similarity between a first target vector in the instance semantic vector group and a second target vector in the comparison instance semantic vector group;

[0088] According to the preset similarity requirement and mask similarity, the target mask instance in the instance semantic information is determined.

[0089] The instance semantic vector group includes an instance query vector (Query Vector), an instance key vector (KeyVector), and an instance value vector (Value Vector).

[0090] The comparison instance semantic vector group includes a comparison instance query vector, a comparison instance key vector, and a comparison instance value vector.

[0091] The preset similarity requirement may be to select N groups of mask instances with the highest correlation, where N is an integer. For example, when the value of N is 3, the three groups of mask instances with the highest similarity are selected for subsequent image processing.

[0092] Among them, the target semantic mask instance and the target comparison semantic mask instance are respectively converted into vector form by using the regional pooling operation, and then the self-attention mechanism is applied to generate the instance semantic vector group and the comparison instance semantic vector group. When the image to be processed is the content image and the comparison image is the style image, this image preprocessing operation is used to enhance the image content features of the image to be processed. The dot product operation is used to calculate the similarity matrix between the instance query vector in the instance semantic vector group and the comparison key vector in the comparison instance semantic vector group. The similarity matrix is ​​softmax normalized and then multiplied with the comparison value vector in the comparison instance semantic vector group to obtain the similarity between the instance query vector in the instance semantic vector group and the comparison key vector in the comparison instance semantic vector group. The mask similarity is further obtained, and the target mask instance is determined according to the preset similarity requirements.

[0093] Among them, the region pooling operation can refer to an operation used to reduce the size and computational complexity of the feature map while retaining key information and improving the efficiency and accuracy of the model.

[0094] The content image refers to an image with clear content or theme, which contains information about the specific object, shape, color, etc. In image style transfer, the purpose of the content image is to provide the image content that needs to be converted.

[0095] A style image is an image that provides a style or form of expression, typically including features such as lines, color schemes, and textures. The purpose of a style image is to apply its style or form of expression to the content image, thereby creating an image with a new style.

[0096] When the image to be processed is a style image and the comparison image is a content image, and this image preprocessing operation is used to enhance the image style features of the style image, then the dot product operation is used to calculate the similarity matrix between the comparison instance query vector in the comparison instance semantic vector group and the key vector in the instance semantic vector group. The similarity matrix is ​​softmax normalized and then multiplied with the value vector in the instance semantic vector group to obtain the similarity between the comparison instance query vector in the comparison instance semantic vector group and the key vector in the instance semantic vector group. The mask similarity is further obtained, and the target mask instance is determined according to the preset similarity requirements.

[0097] S205: Determine a target image according to the target mask instance and the image to be processed.

[0098] The target image may refer to an image to be processed that has been enhanced by the mask instance.

[0099] For example, when N = 3, the target mask instances are the three groups of mask instances with the highest similarity. The images to be processed are enhanced based on the three groups of mask instances to obtain three target images. When N = 1, the target mask instance is the one group of mask instances with the highest similarity. The images to be processed are enhanced based on the above mask instances to obtain one target image. The image processing method provided in the embodiment of the present application performs deep semantic extraction on the mask instances of the image to be processed and measures feature cross-correlation with the comparison mask instances of the comparison image to determine the mask instances of the image to be processed that meet the semantic feature similarity requirements. Feature enhancement is then performed on the image to be processed based on the instances that meet the requirements to obtain the target image, which is beneficial for improving the efficiency of image processing such as image enhancement.

[0100] Figure 3 This is a flow chart of another image processing method provided in an embodiment of the present application. Figure 3 As shown, the method includes:

[0101] S301: Input the target image into the content encoder network and the preset HRNet network respectively to obtain initial multi-scale content features and multi-scale semantic features.

[0102] The content encoder network may refer to a network for extracting intrinsic feature representations of input data (such as images).

[0103] Content features can be used to describe the appearance and structure of an image, that is, its visual composition. They do not involve deep semantic understanding and focus on the physical properties and visual patterns of the image, such as texture, color, shape, and layout. Content features describe the "what" of the image, that is, the observable elements and structures in the image.

[0104] Semantic features can refer to high-level descriptions of objects and scenes in an image. They are related to the actual meaning or concept of the objects in the image and convey information about the "what" and "why" of the image. Semantic features are used to identify entities in an image and the relationships between them, such as "a cat is sitting on a mat."

[0105] S302: Input the initial multi-scale content features, the multi-scale semantic features, and the initial multi-scale style features into the instance normalization module of the decoder network to obtain a target instance, wherein the first multi-scale style features are obtained based on the target comparison image and the style encoder network, the target comparison image is determined based on the target comparison mask instance in the comparison instance semantic information and the comparison image, and the target comparison mask instance is a comparison mask instance determined after performing similarity comparison processing on the comparison instance semantics and the instance semantic information.

[0106] Among them, style features can refer to the unique style and characteristics presented by the image in visual expression. Style features are mainly composed of factors such as image expression techniques, color application, texture processing and overall atmosphere.

[0107] Among them, the style encoder network can refer to the part used to extract and encode image style features. In the style transfer task, the role of the style encoder is to capture the style information of the input image and encode it into a style vector or feature representation. Among them, in the embodiment of the present application, the structure of the content encoder network and the style encoder network are exactly the same. The structure of the encoder network can include multiple Swin Transformer structural blocks and downsampling layers. Each layer will obtain a feature image of a specific size. The width and height are halved compared to the previous layer, but the channels are expanded, and it contains high-level features of the content image or style reference image.

[0108] The Swin Transformer architecture consists of a sequential first-stage layer normalization operation, a two-stage self-attention mechanism, a second-stage layer normalization operation, and finally a fully connected operation. The two-stage self-attention mechanism includes parallel multi-head self-attention mechanisms without a sliding window and with a sliding window, enabling a more flexible global receptive field.

[0109] Among them, the decoder network can refer to the network part used to reconstruct the original data from the encoded feature representation. In the autoencoder structure, the decoder is a key component that maps the potential features extracted by the encoder back to the data space. Among them, in an embodiment of the present application, the normalization part of the decoder network is completely symmetrical with the encoder network, aiming to consistently match the content or style feature representations of different levels. The module contains multiple Swin Transformer structural blocks and upsampling operations. Each layer will obtain a feature image of a specific size, and the width and height dimensions are doubled compared to the previous layer, but the number of channels is reduced.

[0110] The instance normalization module may refer to a module that performs comprehensive processing on input features and outputs instances. In the embodiment of the present application, the instance normalization module includes a semantic enhancement style network, a content enhancement style network, and a content normalization network. The semantic enhancement style network may refer to a convolutional neural network that semantically enhances style features, the content enhancement style network may refer to a convolutional neural network that content-enhances style features, and the content normalization network may refer to a convolutional neural network that normalizes multi-scale content features.

[0111] Among them, the method for obtaining the first multi-scale style feature in the embodiment of the present application can be: according to the similarity comparison processing between the image to be processed and the comparison image, determine the target mask instance and the target comparison mask instance group that meet the similarity requirements, enhance the comparison image according to the target comparison mask instance to obtain the target comparison image, input the target comparison image into the style encoder network for feature extraction, and obtain the first multi-scale style feature.

[0112] In this embodiment of the present application, the initial multi-scale content features, the multi-scale semantic features, and the initial multi-scale style features are input into the instance normalization module of the decoder network to obtain the target instance, including:

[0113] The multi-scale semantic features and the initial multi-scale style features are input into the semantic enhancement style network of the instance normalization module to obtain the semantically enhanced multi-scale style features.

[0114] Input the initial multi-scale content features and the initial multi-scale style features into the content-enhanced style network of the instance normalization module to obtain content-enhanced multi-scale style features;

[0115] Obtain target multi-scale style features based on semantically enhanced multi-scale style features and content-enhanced multi-scale style features;

[0116] According to the initial multi-scale content features and the target multi-scale style features, the target instance is obtained.

[0117] In the embodiment of the present application, the target multi-scale style features are obtained based on the semantically enhanced multi-scale style features and the content enhanced multi-scale style features, including:

[0118] determining a style weight, wherein the style weight is determined based on a content-enhanced style network and a semantic-enhanced style network;

[0119] The target multi-scale style feature is determined according to the style weight, the semantically enhanced multi-scale style feature and the content enhanced multi-scale style feature.

[0120] The style weight may refer to the proportion of semantically enhanced multi-scale style features and the proportion of content-enhanced multi-scale style features in the target multi-scale style features.

[0121] Among them, when image style transfer is performed for the first time, initial style weights are set for the content enhancement style network and the semantic enhancement style network respectively. According to the initial style weights, the multi-scale style features of semantic enhancement and content enhancement are weighted and summed to obtain the target multi-scale style features.

[0122] In the embodiment of the present application, obtaining a target instance according to the initial multi-scale content features and the target multi-scale style features includes:

[0123] Input the initial multi-scale content features into the content normalization network of the instance normalization module for feature extraction to obtain the target multi-scale content features;

[0124] A target instance is obtained according to the target multi-scale content features and the target multi-scale style features.

[0125] The target comparison image may refer to an image that is determined based on the mask similarity and has the highest similarity to the target image.

[0126] Among them, when the image to be processed is a content image, the target image is the content image, and the target comparison image is the style image. The content image is input into the content encoder network for feature extraction to obtain initial multi-scale content features. The content image is input into the preset HRNet network for semantic feature extraction to obtain multi-scale semantic features. The style image is input into the style encoder network for feature extraction to obtain initial multi-scale style features. The initial multi-scale content features, multi-scale semantic features, and initial multi-scale style features are input into the instance normalization module of the decoder network for image style conversion to obtain the target instance.

[0127] Among them, the target multi-scale content features and the target multi-scale style features are input into the AdaIN layer to obtain the target instance, and the target instance is input into the upsampling layer of the decoding network to obtain the target conversion image.

[0128] Among them, the AdaIN layer, namely the adaptive instance normalization layer, aligns the features of the content image with the features of the style image at a statistical level, thereby generating a content image with the style of the style image.

[0129] S303: Input the target instance into the upsampling layer of the decoder network to obtain a target transformed image.

[0130] The target conversion image may refer to an image that is consistent in content with the target image and consistent in style with the target comparison image.

[0131] After obtaining the target conversion image, the method further includes:

[0132] According to the target conversion image and the target comparison image, the feature matching distance is determined. According to the target conversion image and the target image, the perceptual distance is determined. The feature matching distance and the perceptual distance are back-propagated. The content encoder network, the style encoder network and the decoder network are updated. According to the updated content encoder network, the style encoder network, the decoder network, the target image and the target comparison image, the first target conversion image is determined. According to the first target conversion image, the target comparison image and the target image, the feature matching distance and the perceptual distance are updated. The updated feature matching distance and the perceptual distance are back-propagated. The content encoder network, the style encoder network and the decoder network are updated again. The style conversion of the target image and the target comparison image is performed again.

[0133] When the preset number of conversion training times is reached, the target conversion result is obtained. The discriminator determines the adversarial loss based on the target conversion result. According to the adversarial loss, the parameters of the content encoder network, the parameters of the style encoder network and the parameters of the decoder network are updated, and the true or false judgment result of the target conversion result is output at the same time. The style conversion is performed again in the network with updated parameters to obtain an updated conversion result. According to the updated conversion result, the adversarial loss is recalculated, the network is updated again and the true or false judgment result is output. When the preset number of discriminator training times is reached, the style conversion image and the image true or false judgment result are obtained.

[0134] Figure 4 This is a network structure diagram of the example normalization module provided in the embodiment of the present application, which represents Figure 3 Step S302 is a process of inputting the initial multi-scale content features, the multi-scale semantic features, and the initial multi-scale style features into the instance normalization module of the decoder network to obtain the target instance.

[0135] Figure 5 This is a flow chart of another image processing method provided in an embodiment of the present application. Figure 5 As shown, the method includes:

[0136] S501: Input the content image and the style image into the preset HRNet network for semantic segmentation to obtain a content mask and a style mask.

[0137] S502: Obtain content instance voice information and style instance semantic information according to the content mask, the style mask, and the preset CLIP network.

[0138] S503: Determine a target content instance and a target style instance based on the similarity between the voice information of the content instance and the semantic information of the style instance.

[0139] S504: Determine a target content image based on the target content instance and the content image, and determine a target style image based on the target style instance and the style image.

[0140] S505: Input the target content image and the target style image into the encoder network for feature extraction to obtain initial multi-scale content features and initial multi-scale style features, and input the target content image into the preset HRNet network to obtain multi-scale semantic features.

[0141] S506 : Obtain a target style-transferred image based on the initial multi-scale content features, the initial multi-scale style features, the multi-scale semantic features, and the style transfer model.

[0142] The style transfer model consists of a content encoder, a style encoder, and a decoder. The content encoder has the same structure as the style encoder to generate features at the same level. The decoder includes a semantic enhancement network, a content enhancement network, and a content normalization network. The semantic enhancement network and the content enhancement network are used to perform semantic and content enhancement on the initial multi-scale style features. Together, they train a style weight, then perform a weighted summation of the semantically enhanced and content-enhanced style features to determine the target multi-scale style features. The content normalization network is used to normalize the initial multi-scale content features extracted by the content encoder to obtain the target multi-scale content features.

[0143] Among them, the target multi-scale style features and target multi-scale content features are input into the AdaIN layer and the upsampling layer to obtain the target style transfer image.

[0144] According to the target style transfer image, content image, and style image, the perceptual distance and feature matching distance are determined. According to the perceptual distance and feature matching distance, the parameters of the style transfer model are updated, and the content image and style image are input into the updated style transfer model for style transfer. When the preset number of conversions is reached, the discriminator calculates the adversarial loss based on the latest style transfer image, content image, and style image to determine the authenticity of the latest style transfer image, and at the same time updates its own parameters to enhance the discrimination ability. When the preset number of discrimination iterations is reached, the latest style transfer image and the judgment result are output.

[0145] Figure 6 A schematic diagram of the structure of an image processing device provided in an embodiment of the present application is shown in FIG. Figure 6 As shown, the image processing device 60 includes: an acquisition module 601, a calculation module 602, a deep learning module 603, a comparison module 604, and an execution module 605.

[0146] An acquisition module 601 is configured to acquire an image to be processed and a semantic mask of the image to be processed;

[0147] An execution module 602 is configured to combine the semantic mask information of the image to be processed with the image to be processed to obtain a target semantic mask instance;

[0148] A deep learning module 603 is configured to obtain instance semantic information in a target semantic mask instance based on a preset CLIP network and a target semantic mask instance;

[0149] a comparison module 604 for performing a similarity comparison process on the instance semantic information and the comparison instance semantic information, and determining a target mask instance in the instance semantic information, wherein the comparison instance semantic information is obtained based on the CLIP network and the comparison semantic mask instance, and the comparison semantic mask instance is obtained based on the semantic mask information of the comparison image and the comparison image, wherein when the image to be processed is a style image, the comparison image is a content image, and when the image to be processed is a content image, the comparison image is a style image;

[0150] The execution module 605 is configured to obtain a target image according to the target mask instance and the image to be processed.

[0151] In the embodiment of the present application, the calculation module 602 may also be specifically configured to:

[0152] Determine an instance semantic vector group according to a target semantic mask instance of the instance semantic information, wherein the instance semantic vector group includes an instance query vector and an instance key vector;

[0153] Determining a comparison instance semantic vector group according to a target comparison semantic mask instance of the comparison instance semantic information, wherein the comparison instance semantic vector group includes a comparison instance query vector and a comparison instance key vector;

[0154] Determining, based on the instance semantic vector group and the comparison instance semantic vector group, a similarity between a first target vector in the instance semantic vector group and a second target vector in the comparison instance semantic vector group, wherein when the first target vector is an instance query vector, the second target vector is a comparison instance key vector; and when the first target vector is an instance key vector, the second target vector is a comparison instance query vector;

[0155] Determining mask similarity based on a similarity between a first target vector in the instance semantic vector group and a second target vector in the comparison instance semantic vector group;

[0156] According to the preset similarity requirement and mask similarity, the target mask instance in the instance semantic information is determined.

[0157] In the embodiment of the present application, the execution module 605 may also be specifically configured to:

[0158] The target image is input into the content encoder network and the preset HRNet network respectively to obtain the initial multi-scale content features and multi-scale semantic features;

[0159] Inputting the initial multi-scale content features, the multi-scale semantic features, and the initial multi-scale style features into the instance normalization module of the decoder network to obtain a target instance, wherein the initial multi-scale style features are obtained based on the target comparison image and the style encoder network, the target comparison image is determined based on the target comparison mask instance in the comparison instance semantic information and the comparison image, and the target comparison mask instance is a comparison mask instance determined after performing similarity comparison processing on the comparison instance semantic information and the instance semantic information;

[0160] The target instance is input into the upsampling layer of the decoder network to obtain the target transformed image.

[0161] In the embodiment of the present application, the execution module 605 may also be specifically configured to:

[0162] The multi-scale semantic features and the initial multi-scale style features are input into the semantic enhancement style network of the instance normalization module to obtain the semantically enhanced multi-scale style features.

[0163] Input the initial multi-scale content features and the initial multi-scale style features into the content-enhanced style network of the instance normalization module to obtain content-enhanced multi-scale style features;

[0164] Obtain target multi-scale style features based on semantically enhanced multi-scale style features and content-enhanced multi-scale style features;

[0165] According to the initial multi-scale content features and the target multi-scale style features, the target instance is obtained.

[0166] In the embodiment of the present application, the execution module 605 may also be specifically configured to:

[0167] Determine style weights based on content-enhanced style networks and semantic-enhanced style networks;

[0168] The target multi-scale style feature is determined according to the style weight, the semantically enhanced multi-scale style feature and the content enhanced multi-scale style feature.

[0169] In the embodiment of the present application, the execution module 605 may also be specifically configured to:

[0170] Input the initial multi-scale content features into the content normalization network of the instance normalization module for feature extraction to obtain the target multi-scale content features;

[0171] A target instance is obtained according to the target multi-scale content features and the target multi-scale style features.

[0172] Figure 7 This is a schematic diagram of the structure of the electronic device provided in the embodiment of the present application. Figure 7 As shown, the electronic device 70 includes:

[0173] The electronic device 70 may include one or more processors 701 , one or more computer-readable storage media memories 702 , a communication component 703 , and other components. The processor 701 , the memory 702 , and the communication component 703 are connected via a bus 704 .

[0174] In a specific implementation process, at least one processor 701 executes the computer-executable instructions stored in the memory 702, so that the at least one processor 701 performs the above-mentioned image processing method.

[0175] The specific implementation process of the processor 701 can be found in the above method embodiment. Its implementation principle and technical effects are similar and will not be repeated here in this embodiment.

[0176] In the above Figure 7In the illustrated embodiment, it should be understood that the processor may be a central processing unit (CPU), other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASICs), etc. A general-purpose processor may be a microprocessor or any conventional processor. The steps of the method disclosed in the present invention may be directly executed by a hardware processor or by a combination of hardware and software modules within the processor.

[0177] The memory may include a high-speed memory (Random Access Memory, RAM), and may also include a non-volatile memory (NVM), such as at least one disk memory.

[0178] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus. Buses can be classified into address buses, data buses, and control buses. For ease of illustration, the buses in the drawings of this application are not limited to just one bus or just one type of bus.

[0179] The present application also provides a computer program product, including a computer program, which implements the above method when executed by a processor.

[0180] The present application also provides a computer-readable storage medium, in which computer-executable instructions are stored. When a processor executes the computer-executable instructions, the above method is implemented.

[0181] The above-mentioned readable storage medium can be implemented by any type of volatile or non-volatile memory device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk. The readable storage medium can be any available medium that can be accessed by a general-purpose or special-purpose computer.

[0182] An exemplary readable storage medium is coupled to a processor so that the processor can read information from the readable storage medium and write information to the readable storage medium. Of course, the readable storage medium can also be an integral part of the processor. The processor and the readable storage medium can be located in an application specific integrated circuit (ASIC). Of course, the processor and the readable storage medium can also exist in the device as discrete components.

[0183] The division of units is merely a logical functional division; actual implementations may employ alternative divisions, such as combining or integrating multiple units or components into another system, or omitting or disabling certain features. Furthermore, any direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection between devices or units, either through an interface, electrical, mechanical, or other means.

[0184] Units described as separate components may or may not be physically separate, and components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0185] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0186] If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the various embodiments of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, and other media that can store program code.

[0187] Those skilled in the art will appreciate that all or part of the steps in the above-described method embodiments can be implemented using hardware associated with program instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0188] Finally, it should be noted that those skilled in the art will readily identify other embodiments of the present invention after considering the specification and practicing the invention disclosed herein. The present invention is intended to cover any variations, uses, or adaptations of the present invention that follow the general principles of the present invention and include common knowledge or customary techniques in the art not disclosed herein. The present invention is not limited to the precise structure described above and illustrated in the accompanying drawings, and various modifications and variations may be made without departing from the scope thereof. The scope of the present invention is limited solely by the appended claims.

Claims

1. An image processing method, characterized in that: include: Acquire an image to be processed and semantic mask information of the image to be processed; Combining the semantic mask information of the image to be processed with the image to be processed to obtain a target semantic mask instance; Obtaining instance semantic information in the target semantic mask instance according to a preset CLIP network and the target semantic mask instance; performing a similarity comparison process on the instance semantic information and the comparison instance semantic information to determine a target mask instance in the instance semantic information, wherein the comparison instance semantic information is obtained based on the CLIP network and the comparison semantic mask instance, and the comparison semantic mask instance is obtained based on the semantic mask information of the comparison image and the comparison image, wherein when the image to be processed is a style image, the comparison image is a content image, and when the image to be processed is a content image, the comparison image is a style image; Obtaining a target image according to the target mask instance and the image to be processed; The instance semantic information in the target semantic mask instance is a deep semantic representation of the image to be processed, and obtaining the instance semantic information in the target semantic mask instance according to the preset CLIP network and the target semantic mask instance includes: The target semantic mask instance is input into the CLIP network, and the image encoder of the CLIP network performs feature extraction and normalization processing to obtain a deep semantic representation of the image to be processed.

2. The method according to claim 1, characterized in that The performing similarity comparison processing on the instance semantic information and the comparison instance semantic information to determine the target mask instance in the instance semantic information includes: Determining an instance semantic vector group according to a target semantic mask instance of the instance semantic information, wherein the instance semantic vector group includes an instance query vector and an instance key vector; Determining a comparison instance semantic vector group according to a target comparison semantic mask instance of the comparison instance semantic information, wherein the comparison instance semantic vector group includes a comparison instance query vector and a comparison instance key vector; Determining, based on the instance semantic vector group and the comparison instance semantic vector group, a similarity between a first target vector in the instance semantic vector group and a second target vector in the comparison instance semantic vector group, wherein when the first target vector is an instance query vector, the second target vector is a comparison instance key vector; and when the first target vector is an instance key vector, the second target vector is a comparison instance query vector; Determining mask similarity based on a similarity between a first target vector in the instance semantic vector group and a second target vector in the comparison instance semantic vector group; According to a preset similarity requirement and mask similarity, a target mask instance in the instance semantic information is determined.

3. The method according to claim 1, characterized in that When the target image is a content image, after obtaining the target image according to the target mask instance and the image to be processed, the method further includes: Inputting the target image into the content encoder network and the preset HRNet network respectively to obtain initial multi-scale content features and multi-scale semantic features; Inputting the initial multi-scale content features, the multi-scale semantic features, and the initial multi-scale style features into an instance normalization module of a decoder network to obtain a target instance, wherein the initial multi-scale style features are obtained based on a target comparison image and a style encoder network, the target comparison image is determined based on a target comparison mask instance in the comparison instance semantic information and the comparison image, and the target comparison mask instance is a comparison mask instance determined after performing a similarity comparison process on the comparison instance semantic information and the instance semantic information; The target instance is input into the upsampling layer of the decoder network to obtain a target transformed image.

4. The method according to claim 3, characterized in that Inputting the initial multi-scale content features, the multi-scale semantic features, and the initial multi-scale style features into an instance normalization module of a decoder network to obtain a target instance includes: Inputting the multi-scale semantic features and the initial multi-scale style features into the semantic enhancement style network of the instance normalization module to obtain the semantic enhancement multi-scale style features; Inputting the initial multi-scale content features and the initial multi-scale style features into the content-enhanced style network of the instance normalization module to obtain content-enhanced multi-scale style features; Obtaining a target multi-scale style feature based on the semantically enhanced multi-scale style feature and the content-enhanced multi-scale style feature; A target instance is obtained according to the initial multi-scale content features and the target multi-scale style features.

5. The method according to claim 4, characterized in that Obtaining a target multi-scale style feature based on the semantically enhanced multi-scale style feature and the content-enhanced multi-scale style feature includes: determining a style weight according to the content-enhanced style network and the semantic-enhanced style network; A target multi-scale style feature is determined according to the style weight, the semantically enhanced multi-scale style feature, and the content-enhanced multi-scale style feature.

6. The method according to claim 4, characterized in that Obtaining a target instance according to the initial multi-scale content feature and the target multi-scale style feature includes: Inputting the initial multi-scale content features into the content normalization network of the instance normalization module for feature extraction to obtain target multi-scale content features; A target instance is obtained according to the target multi-scale content feature and the target multi-scale style feature.

7. An image processing device, characterized in that: include: An acquisition module, configured to acquire an image to be processed and a semantic mask of the image to be processed; a calculation module, configured to combine the semantic mask information of the image to be processed with the image to be processed to obtain a target semantic mask instance; A deep learning module, configured to obtain instance semantic information in the target semantic mask instance based on a preset CLIP network and the target semantic mask instance; a comparison module, configured to perform a similarity comparison process on the instance semantic information and the comparison instance semantic information, and determine a target mask instance in the instance semantic information, wherein the comparison instance semantic information is obtained based on the CLIP network and the comparison semantic mask instance, and the comparison semantic mask instance is obtained based on the semantic mask information of the comparison image and the comparison image, wherein when the image to be processed is a style image, the comparison image is a content image, and when the image to be processed is a content image, the comparison image is a style image; an execution module, configured to obtain a target image according to the target mask instance and the image to be processed; The instance semantic information in the target semantic mask instance is a deep semantic representation of the image to be processed. The deep learning module is specifically used to: input the target semantic mask instance into the CLIP network, and obtain the deep semantic representation of the image to be processed through feature extraction and normalization processing of the image encoder of the CLIP network.

8. An electronic device, characterized in that: include: a processor, and a memory communicatively connected to the processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory to implement the method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer-executable instructions, which are used to implement the method according to any one of claims 1 to 6 when executed by a processor.

10. A computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images