A semantically guided content feature transfer style transfer method and system

Abstract

The present invention discloses a semantically guided content feature transfer style transfer method and system, which belongs to the field of deep learning style transfer. In order to achieve complete and consistent style transfer of content features, the present invention proposes a content calibration module, which includes a feature optimization unit and an attribute reasoning unit. The feature optimization unit uses the network's deep extraction capability to keep the multi-channel content features intact, and the attribute reasoning unit ignores the position space and reclassifies the content semantics with the help of attention grouping interaction to provide assistance in finding suitable content expression. The content attributes extracted from the original content features are re-assigned to the deep content features, the content feature mapping deviation is calibrated, the content feature noise is reduced, and the bottom-up content consistency is implemented. The present invention is applicable to fields such as autonomous driving and security monitoring.

Description

Technical Field

[0001] The present invention relates to the technical field of deep learning style transfer, and in particular to a semantically guided content feature transfer style transfer method and system. Background Art

[0002] With the rapid development of autonomous driving and industrial and service robot applications, style transfer technology, which is indispensable for autonomous driving and path planning perception systems, has become one of the current research hotspots. In terms of hardware, autonomous driving systems mostly rely on radars, infrared cameras and other equipment to enhance the system's ability to perceive the surrounding environment, but their cost is relatively high, and the positioning and prediction of small targets and high-speed moving targets are inaccurate; in terms of software, most existing style transfer methods improve their performance by deepening and widening the network or improving the loss function, but it is very easy to produce content mapping deviations during the training process, making it difficult to achieve complete and consistent feature transfer, affecting the driving safety of the autonomous driving system. Existing style transfer methods can be divided into two categories: those based on convolutional neural networks and those based on generative adversarial networks:

[0003] The style transfer method based on convolutional neural network generally uses a classification neural network to convert the content image into a predetermined style image by optimizing or training an image transfer neural network. Specifically, the method is named as an image style transfer method combining deep learning and depth perception. In the invention patent with publication number CN107705242B, the depth perception network is added to the target loss function to estimate the depth of field of the original image and the generated style image. In the process of style transfer, the generated image not only integrates the corresponding style and content, but also maintains the far and near structural information of the original image. In the invention patent application named as an image transfer method based on mean standard deviation, the feature space is constructed to store the feature information of different filters, so as to better obtain multi-scale and stable features. It does not require training on real data and can flexibly perform style transformation. The invention patent application entitled "A style transfer method, device and related components based on feature fusion" with publication number CN113808011A extracts features from content and style images through a pre-trained content and style encoder, and then fuses and outputs the content and style features with a trained content and style decoder to obtain a target style transfer image. The above-mentioned style transfer method based on convolutional neural network focuses on using VGG and other functional layers to deepen or widen the network to extract content and style features in the image. The resulting style transfer effect has overlaps in detail texture and color filling, and cannot be well applied to style transfer in real-world scenarios such as autonomous driving and mobile robots.

[0004] Style transfer methods based on generative adversarial networks have accelerated the progress in the field of style transfer. They are generally based on an encoder-decoder structure, using an encoder to simultaneously extract content features and style features, and directly input the two features into a decoder for decoding. At the same time, relevant loss functions are designed from the perspectives of color, content, smoothness, etc. to supervise the network to obtain stylized results. In the invention patent application entitled "A cross-domain variational adversarial autoencoder method", the encoder is used to decouple the content encoding and style encoding of the cross-domain input image, and the adversarial operation and variational operation are used to fit the content and style encoding of the image respectively, and the content and style encoding of different domains are cross-coded to achieve a one-to-many transformation of the cross-domain image. In the invention patent application entitled "A method for multi-style conversion of images based on latent variable feature generation", the style encoding generator is designed to fit the style encoding of the image based on a multimodal unsupervised image conversion network, and a jump connection is introduced between the content encoding and the style encoding, and an attention mechanism is introduced in the style encoding, which improves the quality and diversity of multi-style conversion of images. The invention patent application entitled "A Multi-channel Parallel Image Content Feature Optimization Style Transfer Method" with publication number CN113284042A uses a multi-channel parallel method to separate the image content features of a single feature channel and multiple feature channels, which can improve the ability to separate and extract small targets and blurred targets and the ability to transfer image detail texture information. However, most of the above-mentioned style transfer methods are based on the network's own performance in a closed environment, combined with contextual information for reasoning. The training process is difficult to avoid the influence of various confusing factors such as different target attributes, resulting in deviations between the actual network output and the theoretical output. Therefore, how to effectively utilize the deep features extracted from the image, ensure the consistency of the image content before and after style transfer, and better apply it to traffic scenes has become a problem that needs to be solved. Summary of the invention

[0005] The purpose of the present invention is to provide a semantically guided content feature transfer style transfer method and system, which divides the noisy attributes generated by content features into multiple groups through channel attention, performs random information exchange between different groups and the same group, weakens the noise, and fuses and corrects the remaining content features channel by channel, assigns correct content attribute labels to the content features in the transfer process, guides the generation of content features, reduces the mapping deviation in the feature transfer process, and effectively realizes the style transfer of image content consistency.

[0006] To achieve the above object, the technical solution of the present invention is: a semantically guided content feature transfer style migration method, comprising:

[0007] Prepare a dataset for training the style transfer network;

[0008] Get the source domain input image with feature channel c and the target domain input image Performing double downsampling operations on each of them, wherein the double downsampling operations include convolution operations and nonlinear activation function processing;

[0009] Downsampling result of the target domain input image Use global average pooling and full connection function processing to obtain the style feature vector

[0010] Downsampling result of source domain input image Use multi-layer residual unit processing to obtain a four-dimensional feature vector

[0011] The four-dimensional feature vector After global maximum pooling, full connection function processing, deep convolutional neural network, information exchange, and point convolutional neural network processing, a four-dimensional feature vector is obtained. At the same time, multiple layers of residual units are used to deepen the processing of the four-dimensional feature vector Get the four-dimensional feature vector

[0012] The four-dimensional feature vector and the four-dimensional eigenvector Multiply to generate a four-dimensional content feature vector Realize the redistribution of target attributes in content features and correct feature transfer deviation;

[0013] The style feature vector and the four-dimensional content feature vector Add and fuse to get the four-dimensional feature vector Then upsample and output the style transfer result Y c×2h×2w .

[0014] Furthermore, for the source domain input image and the target domain input image Perform double downsampling operation, specifically:

[0015] Use the convolution kernel M c×3×3 Extract the source domain input image Content features in and the target domain input image Style features in The formula is:

[0016]

[0017]

[0018] in For the convolution process, each matrix represents a 3×3 feature block;

[0019] The output feature vector and Using nonlinear activation function processing, when the eigenvalue of the activation processing is less than or equal to 0, the output value of the activation function is 0, as shown in formula (3); conversely, when the output value of the activation function is the same as the input value, as shown in formula (4):

[0020]

[0021]

[0022] Among them, the function A(·) is the activation function.

[0023] Furthermore, the downsampling result of the target domain input image is Use global average pooling and full connection function processing, specifically:

[0024] Use global average pooling to average the features of each unit and obtain the feature vector of each unit The formula is as follows:

[0025]

[0026] Among them, P average (·) is the global average pooling function, M c×2×2 Perform pixel-by-pixel operations on the convolution kernel features with filter k=2, select the average value and output it;

[0027] For the feature vector Use the fully connected function to process each feature channel one by one and output the feature vector The formula is as follows:

[0028]

[0029] Among them, C fully (·) is the fully connected function, and M is selected c×1×1 That is, the convolution kernel of the filter k=1 is operated;

[0030] For the feature vector Perform style cosine normalization to obtain a four-dimensional style feature vector The formula is:

[0031]

[0032] Among them, cos IN(·) is the style cosine normalization process function, μ(x) and μ(y) are the means of the length and width dimensions of the feature vector, σ(x) and σ(y) are the standard deviations of the length and width dimensions of the four-dimensional feature vector, respectively.

[0033] Furthermore, the downsampling result of the source domain input image is Using multi-layer residual unit processing, the formula is:

[0034]

[0035]

[0036] Among them, F(·) is the single-layer residual unit process function, and ω3 is the weight matrix.

[0037] Furthermore, the four-dimensional feature vector After global maximum pooling, full connection function processing, deep convolutional neural network, information exchange, and point convolutional neural network processing, a four-dimensional feature vector is obtained. Specifically:

[0038] Use global maximum pooling to process four-dimensional feature vectors And get the feature vector The formula is:

[0039]

[0040] Among them, P max (·) is the global maximum pooling function, M c×2×2 Perform pixel-by-pixel operations on the convolution kernel features with filter k=2, select the maximum value and output it;

[0041] Use the fully connected function to process the feature vector And get the feature vector The formula is:

[0042]

[0043] Among them, C fully (·) is the fully connected function, and M is selected c×1×1 That is, the convolution kernel of the filter k=1 is operated;

[0044] Use deep convolutional neural network to transform the feature vector According to the characteristic channel, it is evenly divided into p branches (p≤c), and the characteristic components of each characteristic channel are obtained The formula is as follows:

[0045]

[0046] Among them, F deep(·) is the process function of the deep convolutional neural network;

[0047] Each branch is divided into q groups for random exchange of features, disrupting the inherent order of information between different channels, reclassifying and combining the feature information to obtain the feature components The formula is as follows:

[0048]

[0049] Among them, Shuffle(·) is the information exchange function;

[0050] Use point convolutional neural network to classify feature components Merge to get a four-dimensional feature vector The point convolutional neural network randomly deletes some neurons during the merging process. The formula is as follows:

[0051]

[0052] Among them, D ran is the random deletion function, m is the proportion of randomly deleted neurons;

[0053]

[0054] Among them, F poi (·) is the point convolutional neural network process function, using M c×1×1 The point convolution of the form performs a point convolution operation on the feature vector.

[0055] Furthermore, a multi-layer residual unit is used to further process the feature vector Get the four-dimensional feature vector Specifically:

[0056]

[0057] in, is the weight matrix.

[0058] Furthermore, the feature vector and the four-dimensional eigenvector Multiply to generate a four-dimensional content feature vector Specifically:

[0059]

[0060] in, and is the weight matrix, and × represents the multiplication of the feature matrix.

[0061] The present invention also provides a semantically guided content feature transfer style migration system, including an encoding module, a content calibration module and a decoding module;

[0062] The encoding module includes a content encoding module and a style encoding module; the content encoding module inputs the source domain into the image As input, it performs a double downsampling operation on it and outputs a four-dimensional feature vector The style encoding module inputs the target domain into the image As input, double downsampling, global average pooling, full connection function, and style cosine normalization are used in sequence to output a four-dimensional feature vector

[0063] The content calibration module includes a feature optimization unit and an attribute reasoning unit; the feature optimization unit downsamples the source domain input image Use multi-layer residual unit processing to obtain a four-dimensional feature vector The attribute reasoning unit performs a four-dimensional feature vector After global maximum pooling, full connection function processing, deep convolutional neural network, information exchange, and point convolutional neural network processing, a four-dimensional feature vector is obtained. At the same time, multi-layer residual units are used to deepen the processing of feature vectors Get the four-dimensional feature vector The four-dimensional feature vector With the four-dimensional eigenvector Multiply ω1 and ω2 at a fixed ratio and output a four-dimensional feature vector

[0064] The decoding module converts the four-dimensional feature vector and the four-dimensional eigenvector Add and fuse to get the four-dimensional feature vector Then upsample and output the style transfer result Y c×2h×2w .

[0065] Further, the content calibration module is expressed as:

[0066]

[0067] Among them, F opt (x) is the feature optimization unit learning process function, F re (x) is the learning process function of the attribute reasoning unit.

[0068] By adopting the above technical solution, the present invention can achieve the following beneficial effects: it can be applied to real-life scenarios such as autonomous driving, industrial and service robots, realize style changes in any weather and environmental scenes, and provide assistance for accurate identification of small and fuzzy targets. The following is a list of the beneficial effects of the present invention:

[0069] (1) Applicable to small target features

[0070] The attribute reasoning unit of the present invention can separate potential instance target attributes from image content features, fully tap deep feature information, and accurately and clearly identify and extract content feature information containing small target images in an unsupervised manner.

[0071] (2) Applicable to high-speed moving target characteristics

[0072] The present invention uses a feature optimization unit and an attribute reasoning unit to respectively extract content features and target attributes in an input image, and corrects feature transfer deviations in the feature optimization unit according to the extracted target attributes, thereby effectively improving blurring caused by high-speed motion of the target and realizing the extraction of high-speed motion targets.

[0073] (3) Applicable to public security monitoring systems

[0074] The present invention can meet the needs of effective multi-scale feature extraction and style transformation in any complex scenes (such as underground parking lots, fire passages, and traffic scenes) captured by security surveillance cameras and Sky Eye systems around the clock, providing favorable conditions for the next step of detection and identification, improving the working efficiency of public systems, and providing guarantees for improving production and living efficiency and maintaining public safety.

[0075] (4) Applicable to autonomous driving technology

[0076] The present invention is a computer vision environment perception technology, which is applicable to the field of autonomous driving. It can extract the target features and positions of pedestrians, vehicles, buildings, traffic signs, etc. around the driving environment, provide comprehensive feature information for the style migration model, and provide strong protection for driving safety.

[0077] (5) Applicable to situations with unclear vision

[0078] The present invention is applicable to style transfer situations in different complex scenes. It restores the unclear target features under infrared and visible light conditions obtained by camera lenses with different exposure and clarity, and performs style transfer after improving the image clarity. BRIEF DESCRIPTION OF THE DRAWINGS

[0079] Figure 1 It is a flowchart of the semantic-guided content feature transfer style transfer method;

[0080] Figure 2 is a schematic diagram of the content calibration module structure;

[0081] Figure 3 This is a schematic diagram of the security monitoring style migration in Example 1;

[0082] Figure 4is a schematic diagram of autonomous driving style migration in Example 2;

[0083] Figure 5 This is a schematic diagram of the style transfer of the visually blurred scene in Example 3. DETAILED DESCRIPTION

[0084] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application, that is, the described embodiments are only part of the embodiments of the present application, rather than all of the embodiments. The components of the embodiments of the present application described and shown in the drawings here can be arranged and designed in various different configurations.

[0085] The present invention proposes a semantically guided content feature transfer style transfer method and system. First, the network is deepened to enhance the ability of deep feature extraction, ensure the integrity of image content features during style transfer, and reduce feature redundancy; second, the content features are compressed into a one-dimensional semantic information expression, and information is exchanged in batches by channel to enhance the correlation between different channel features, and a correct content label is assigned to each content feature, so as to improve the phenomenon of mismatch between target attributes and content feature expressions during feature transfer and ensure the consistency of content feature transfer. Figure 1 The specific migration method implementation steps are as follows:

[0086] Step 1: Prepare a dataset for training the style transfer network, where the dataset size may be 2h×2w;

[0087] Step 2: Get the source domain input image with feature channel c and the target domain input image A double downsampling operation is performed on each of them, and the double downsampling operation includes two parts: a convolution operation and a nonlinear activation function processing, specifically:

[0088] (1) Use convolution kernel M with step size s=2 and filter k=3 c×3×3 Extract the source domain input image Content features in and the target domain input image Style features in The formula is:

[0089]

[0090]

[0091] in For the convolution process, each matrix represents a 3×3 feature block;

[0092] (2) Output the feature vector and Using nonlinear activation function processing, when the eigenvalue of the activation processing is less than or equal to 0, the output value of the activation function is 0, as shown in formula (3); conversely, when the output value of the activation function is the same as the input value, as shown in formula (4):

[0093]

[0094]

[0095] Among them, function A(·) is the activation function. Using the activation function to perform nonlinear processing on the feature vector can improve the effectiveness of the feature vector and reduce feature redundancy, which helps to achieve consistent style transfer of image content features.

[0096] Step 3: To reduce the influence of feature position on style classification, the feature vector extracted in step 2 is Use global average pooling and full connection function processing to output feature vector Specifically:

[0097] (1) Use global average pooling to average the features of each unit and obtain the feature vector of each unit The formula is as follows:

[0098]

[0099] Among them, P average (·) is the global average pooling function, M c×2×2 Perform pixel-by-pixel operations on the convolution kernel with k=2, select the average value and output it;

[0100] (2) For the eigenvector Use the fully connected function to process each feature channel one by one, reduce the influence of pixel and feature position on feature classification, and output the feature vector The formula is as follows:

[0101]

[0102] Among them, C fully (·) is the fully connected function, and M is selected c×1×1 That is, the convolution kernel with k=1 is used for the operation.

[0103] (3) To change the distribution of style feature data and achieve accurate style feature transfer, the present invention performs Perform style cosine normalization to suppress feature information irrelevant to the style and output a four-dimensional feature vector To prepare for integration with content features, the formula is:

[0104]

[0105] Among them, cos IN (·) is the style cosine normalization process function, μ(x) and μ(y) are the means of the length and width dimensions of the feature vector, σ(x) and σ(y) are the standard deviations of the length and width dimensions of the four-dimensional feature vector, respectively.

[0106] Step 4: The input of the feature optimization unit is the downsampling result of the source domain input image Use multiple layers (preferably 8 layers) of residual units to process the feature vector of the downsampled output to reduce feature redundancy and ensure the integrity of the image content during the style transfer process. The formula is:

[0107]

[0108]

[0109] Among them, F(·) is the single-layer residual unit process function, ω3 is the weight matrix;

[0110] Step 5: The attribute reasoning unit input is a four-dimensional feature vector It goes through global maximum pooling, fully connected function processing, deep convolutional neural network, information exchange, and point convolutional neural network processing in turn, and outputs a four-dimensional feature vector In the process of network training, the reflection mechanism is introduced to re-assign the correct content attribute labels to the content features. At the same time, a multi-layer residual unit is used to deepen the processing of the four-dimensional feature vector Output four-dimensional feature vector Implement the consistent transfer of spatial structure information in features, specifically:

[0111] (1) To extract the hidden target attributes in the content features, the present invention uses global maximum pooling to process the four-dimensional feature vector And output Eliminate the impact of target attribute classification caused by target position, the formula is

[0112]

[0113] Among them, P max (·) is the global maximum pooling function, M c×2×2 The convolution kernel with k=2 operates pixel by pixel, selects the maximum value and outputs it;

[0114] (2) To enhance the network’s ability to learn the attributes of hidden content, a fully connected function is used Enhance the correlation between channels and output feature vector The formula is:

[0115]

[0116] Among them, C fully (·) is the fully connected function, and M is selected c×1×1 That is, the convolution kernel with k=1 is operated;

[0117] (3) Use a deep convolutional neural network to transform the feature vector According to the characteristic channel, it is evenly divided into p branches (p≤c) to obtain the characteristic components of each characteristic channel The formula is as follows:

[0118]

[0119] Among them, F deep (·) is the process function of the deep convolutional neural network;

[0120] (4) Each branch is divided into q groups for random feature exchange, disrupting the inherent order of information between different channels, reclassifying and combining the feature information, and outputting the feature components The formula is as follows:

[0121]

[0122] Among them, Shuffle(·) is an information exchange function, which divides the channel features on each branch into q groups, and randomly shuffles the order between each group and between different groups to seek new content and attribute matching relationships.

[0123] (5) Use the point convolutional neural network to merge the feature vectors after interaction and output the content reasoning unit results The feature reorganization and fusion between different feature channels provide greater possibilities for accurate transmission of content features. The point convolutional neural network will randomly delete some neurons during the processing process. The formula is as follows:

[0124]

[0125] Among them, D ran is a random deletion function, and m is the ratio of randomly deleted neurons. This operation can prevent the network from overfitting.

[0126]

[0127] Among them, F poi (·) is the point convolutional neural network process function, using M c×1×1 The point convolution of the form performs a point convolution operation on the feature vector.

[0128] (6) Using multi-layer residual units to process four-dimensional feature vectors And output the four-dimensional feature vector Ensure the spatial structure and semantic consistency during feature vector transmission, and lay the foundation for the effective interaction of feature components in different feature channels.

[0129]

[0130] in, is the weight matrix.

[0131] Step 6: Output the four-dimensional feature vector from step 5 and the four-dimensional eigenvector Multiply to generate a four-dimensional feature vector To achieve the redistribution of target attributes in content features and correct feature transfer deviation, the formula is:

[0132]

[0133] in, and is the weight matrix, and × represents the multiplication of the feature matrix.

[0134] Step 7: Output the style features from step 3 and the content features output from step 6 Add and fuse to get the four-dimensional feature vector Upsample and output the style transfer result Y in the decoder c×2h×2w .

[0135] This embodiment also provides a system for implementing the above method, which includes: an encoding module, a content calibration module and a decoding module; each part is described in detail below:

[0136] The encoding module includes a content encoding module and a style encoding module; the content encoding module inputs the source domain into the image As input, it performs a double downsampling operation on it and outputs a four-dimensional feature vector The style encoding module inputs the target domain into the image As input, global average pooling, full connection function, and style cosine normalization are used in sequence to output a four-dimensional feature vector Reduce the interference of spatial structure on color semantic information.

[0137] The content calibration module includes a feature optimization unit and an attribute reasoning unit; Figure 2As shown in the figure, the four-dimensional feature vector of multiple feature channels is input into the feature optimization unit, and more content structure and detailed texture information are learned while maintaining the complete transmission of image content features of multiple feature channels; the attribute reasoning unit extracts hidden attributes from the four-dimensional feature vector output by the residual unit, and re-assigns the correct feature expression to each content feature through multi-channel and batch training, thereby achieving accurate correspondence between one-dimensional semantics and two-dimensional pixels. The outputs of different units are multiplied in the form of channel attention to correct the content feature transmission deviation, achieve accurate classification of content structure, and implement consistency in image content feature transmission; the feature optimization unit and attribute reasoning unit in the content calibration module are described in detail below: the input of the attribute reasoning unit is a four-dimensional feature vector of size h×w containing n feature channels after being processed by multiple layers of residual units in the feature optimization unit Among them, the feature optimization unit learning process function is F opt (x); the learning process function of the attribute reasoning unit is F re (x); Assume that after the output features of the two units are fused, the input to the next stage is a four-dimensional feature vector containing n feature channels and a size of h×w The content calibration module is expressed as:

[0138]

[0139] The feature optimization unit uses a multi-layer residual unit to extract deep features with n feature channels. Reduce feature redundancy and maintain the texture details and contour feature information of the source domain input image, and use it as the input of the attribute reasoning unit and the next stage.

[0140] The attribute reasoning unit uses global average pooling to Perform dimensionality reduction and regularization processing to output a four-dimensional feature vector On this basis, a fully connected layer is used to weaken the influence of spatial information such as target position on content attribute classification, so that the network focuses on the association between different channel features, strengthens the expression of semantic features, and enhances the ability to extract deep features. In order to further enhance the association between different feature channels and fully extract the hidden attributes in the source domain input image, a deep convolution with a convolution kernel of 3×3 is used to It is divided into p branches, and the content feature information on each feature channel is extracted respectively. Different branches learn and supervise each other to achieve reference across feature channels. At the same time, the feature channels are divided into q groups in each branch, and the channel order is disrupted within each group and between different groups to increase the randomness of the hidden content attributes, improve the generalization ability of the network, and reallocate the classification attributes of the target to each content feature, reducing the feature transfer deviation. In order to obtain accurate content attributes, the features of each branch are filtered to obtain Use 1×1 point convolution to complete the integration of p branch feature information and output a four-dimensional feature vector Each content feature is given an enhanced feature expression to guide the accurate generation of content features. While processing the feature vector in a single feature channel, a multi-layer residual unit is used in the form of parallel branches to deal with the feature vector at the multi-feature channel level. Processing, output four-dimensional feature vector Ensure the integrity of feature vector transmission; and Multiply ω1 and ω2 at a fixed ratio and output a four-dimensional feature vector Hidden attributes and inherent content features are used for adversarial screening to fix the problem of incorrect content attribute assignment, reduce the content feature transfer deviation, and help achieve consistent and accurate style transfer.

[0141] The decoding module performs source domain and target domain feature vector operations: and Add and fuse to get the four-dimensional feature vector Then upsample and output the style transfer result Y c×2h×2w .

[0142] In this embodiment, the characteristic parameter constraint condition may be:

[0143] (1) Downsample the input RGB three-channel image with a size of 256×256, and reduce the input image size to 128×128, feature channels n∈{4,8,16,64,256,512}, and output a four-dimensional feature vector containing image content features, any of {1,128,128,4}, {1,128,128,8}, {1,128,128,16}, {1,128,128,64}, {1,128,128,256}, and {1,128,128,512}.

[0144] (2) The content calibration module selects four-dimensional feature vectors of different feature channels as input according to the content of the input image: when the input image contains small targets or blurred targets, the four-dimensional feature vector with feature channels of n = 256 is selected as the input of the content calibration module; when the input image does not contain small targets or blurred targets, the four-dimensional feature vector with feature channels of n = 8 is selected as the input of the content calibration module.

[0145] (3) The feature channel transmitted in the feature optimization unit is a four-dimensional feature vector of n∈{4,8,16,64,256,512}.

[0146] (4) A four-dimensional feature vector with feature channel n=1 is transmitted in the attribute reasoning unit.

[0147] Structural unit constraints:

[0148] (1) A four-layer residual unit is used in the feature optimization unit to extract deep content features.

[0149] (2) The attribute reasoning unit contains p branches, and p∈{0,1,2,3,4}. When p=0, the content calibration module only contains the feature optimization unit.

[0150] (3) Each branch in the attribute reasoning unit contains q groups, where q = {q|10≤q≤512,q∈Z +}.

[0151] (4) The attribute reasoning unit selects different numbers of groups according to the complexity of the input image content: when the input image contains small or blurred targets, select q = {q|128≤q≤512,q∈Z +}; when there is no small target or blurred target in the input image, select q = {q|10≤q≤128,q∈Z +}The number of groups.

[0152] Example 1: Security monitoring style transfer

[0153] This example is used to monitor unmanned places such as schools and crossroads. The application of this invention in outdoor security monitoring can effectively improve the recognition ability of targets under complex lighting conditions. Figure 3 shown.

[0154] Example 2: Autonomous driving style transfer

[0155] This example is about the style transfer of the autonomous driving system. The present invention is applied to the vehicle-mounted camera to perceive the driving environment, provide assistance to the driver, reduce the traffic accident rate, and improve the vehicle's safe driving ability. The autonomous driving style transfer is as follows Figure 4 shown.

[0156] Example 3: Style transfer in visually blurred scenes

[0157] This example aims at style transfer of visually blurred scenes caused by uneven lighting or natural weather. It can improve the image quality of style transfer in complex scenes and prepare for the next step of target detection or image segmentation. Figure 5 shown.

[0158] Those skilled in the art will appreciate that the embodiments described herein are intended to help readers understand the principles of the present invention, and should be understood that the scope of protection of the invention is not limited to such specific statements and embodiments. Any possible equivalent substitutions or changes made based on the above description are considered to fall within the scope of protection of the claims of the present invention.

Claims

1. A semantically guided content feature transfer style transfer method, characterized in that: include: Prepare a dataset for training the style transfer network, the dataset size is 2h×2w; Get the source domain input image with feature channel c and the target domain input image Performing double downsampling operations on each of them, wherein the double downsampling operations include convolution operations and nonlinear activation function processing; Downsampling result of the target domain input image Use global average pooling and full connection function processing to obtain a four-dimensional feature vector Downsampling result of source domain input image Use multi-layer residual unit processing to obtain a four-dimensional feature vector The four-dimensional feature vector After global maximum pooling, full connection function processing, deep convolutional neural network, information exchange, and point convolutional neural network processing, a four-dimensional feature vector is obtained. At the same time, multiple layers of residual units are used to deepen the processing of the four-dimensional feature vector Get the four-dimensional feature vector The four-dimensional feature vector and the four-dimensional eigenvector Multiply to generate a four-dimensional content feature vector Y1 c ×h×w ,Realize the redistribution of target attributes in content features and correct feature transfer deviation; The four-dimensional feature vector and the four-dimensional content feature vector Y1 c×h×w Add and fuse to get the four-dimensional feature vector Then upsample and output the style transfer result Y c×2h×2w ; Downsampling result of source domain input image Using multi-layer residual unit processing, the formula is: Among them, F i is the process function of a single-layer residual unit, ω3 is the weight matrix; The four-dimensional feature vector and the four-dimensional eigenvector Multiply to generate a four-dimensional content feature vector Y1 c ×h×w , specifically: in, and is the weight matrix, and × represents the multiplication of the feature matrix.

2. According to claim 1, a semantically guided content feature transfer style transfer method is characterized in that: Input image to the source domain and the target domain input image Perform double downsampling operation, specifically: Use the convolution kernel M c×3×3 Extract the source domain input image Content features in and the target domain input image Style features in The formula is: in For the convolution process, each matrix represents a 3×3 feature block; The output content vector and style features Using nonlinear activation function processing, when the eigenvalue of the activation processing is less than or equal to 0, the output value of the activation function is 0, as shown in formula (3); conversely, when the output value of the activation function is the same as the input value, as shown in formula (4): Among them, the function A(·) is the activation function.

3. According to claim 2, a semantically guided content feature transfer style transfer method is characterized in that: Downsampling result of the target domain input image Use global average pooling and full connection function processing, specifically: Use global average pooling to average the features of each unit and obtain the feature vector of each unit The formula is as follows: Among them, P average (·) is the global average pooling function, M c×2×2 Perform pixel-by-pixel operations on the convolution kernel features with filter k=2, select the average value and output it; For the feature vector Use the fully connected function to process each feature channel one by one and output the feature vector The formula is as follows: Among them, C fully (·) is the fully connected function, and M is selected c×1×1 That is, the convolution kernel of the filter k=1 is operated; For the feature vector Perform style cosine normalization to obtain a four-dimensional feature vector The formula is: Among them, cos IN (·) is the style cosine normalization process function, μ(x) and μ(y) are the means of the length and width dimensions of the feature vector, σ(x) and σ(y) are the standard deviations of the length and width dimensions of the four-dimensional feature vector, respectively.

4. According to claim 3, a semantically guided content feature transfer style transfer method is characterized in that: The four-dimensional feature vector After global maximum pooling, full connection function processing, deep convolutional neural network, information exchange, and point convolutional neural network processing, a four-dimensional feature vector is obtained. Specifically: Use global maximum pooling to process four-dimensional feature vectors And get the feature vector The formula is: Among them, P max (·) is the global maximum pooling function, M c×2×2 Perform pixel-by-pixel operations on the convolution kernel features with filter k=2, select the maximum value and output it; Use the fully connected function to process the feature vector And get the feature vector The formula is: Among them, C fully (·) is the fully connected function, and M is selected c×1×1 That is, the convolution kernel of the filter k=1 is operated; Use deep convolutional neural network to transform the feature vector According to the characteristic channel, it is evenly divided into p branches, p≤c, and the characteristic components of each characteristic channel are obtained. The formula is as follows: Among them, F deep (·) is the process function of the deep convolutional neural network; Each branch is divided into q groups for random exchange of features, disrupting the inherent order of information between different channels, reclassifying and combining the feature information to obtain the feature components The formula is as follows: Among them, Shuffle(·) is the information exchange function; Use point convolutional neural network to classify feature components Merge to get a four-dimensional feature vector The point convolutional neural network randomly deletes some neurons during the merging process. The formula is as follows: Among them, D ran is the random deletion function, m is the proportion of randomly deleted neurons; Among them, Fpoi(·) is the point convolutional neural network process function, using M c×1×1 The point convolution of the form performs a point convolution operation on the feature vector.

5. According to claim 4, a semantically guided content feature transfer style transfer method is characterized in that: Deep processing of 4D feature vectors using multi-layer residual units Get the four-dimensional feature vector Specifically: in, is the weight matrix.

6. A semantically guided content feature transfer style migration system, used to implement the method according to any one of claims 1 to 5, characterized in that: It includes an encoding module, a content calibration module and a decoding module; The encoding module includes a content encoding module and a style encoding module; the content encoding module inputs the source domain into the image As input, perform a double downsampling operation on it to get the downsampling result The style encoding module inputs the target domain into the image As input, global average pooling, full connection function, and style cosine normalization are used in sequence to output a four-dimensional feature vector The content calibration module includes a feature optimization unit and an attribute reasoning unit; The downsampling result of the source domain input image by the feature optimization unit Use multi-layer residual unit processing to obtain a four-dimensional feature vector The attribute reasoning unit performs a four-dimensional feature vector After global maximum pooling, full connection function processing, deep convolutional neural network, information exchange, and point convolutional neural network processing, a four-dimensional feature vector is obtained. At the same time, multiple layers of residual units are used to deepen the processing of the four-dimensional feature vector Get the four-dimensional feature vector The four-dimensional feature vector With the four-dimensional eigenvector Multiply ω1 and ω2 at a fixed ratio and output the four-dimensional content feature vector Y1 c×h×w ; The decoding module converts the four-dimensional feature vector and the four-dimensional content feature vector Y1 c×h×w Add and fuse to get the four-dimensional feature vector Then upsample and output the style transfer result Y c×2h×2w .

7. According to claim 6, a semantically guided content feature transfer style transfer system is characterized in that: The content calibration module is expressed as: Among them, F opt (x) is the feature optimization unit learning process function, F re (x) is the learning process function of the attribute reasoning unit.

Images