A structure-guided style deviation correction style transfer method and system

Abstract

The present invention discloses a structure-guided style deviation correction style transfer method and system, which belongs to the field of deep learning style transfer. In order to achieve accurate mapping of style features, the present invention proposes a structure-guided module, which includes a feature optimization unit and a style correction unit. The feature optimization unit uses the network's deep extraction capability to ensure the complete transmission and accurate classification of single-feature channel style features. The style correction unit adds spatial structure position information to the style features in semantic form, and re-matches the style semantics with the content structure by means of attention grouping interaction, so as to provide assistance for the subsequent search for suitable style and content expression. The style features endowed with spatial structure information are multiplied with the style features processed by the feature optimization unit to calibrate the style feature mapping deviation. The present invention is applicable to the fields of autonomous driving, security monitoring, etc.

Description

Technical Field

[0001] The present invention relates to the technical field of deep learning style transfer, and in particular to a structure-guided style deviation correction style transfer method and system. Background Art

[0002] With the rapid development of autonomous driving and industrial and service robotics, style transfer technology, which is essential for autonomous driving and path planning perception systems, has become one of the current research hotspots. In terms of hardware, autonomous driving systems mostly rely on radars, infrared cameras and other equipment to enhance the system's ability to perceive the surrounding environment, but their cost is high, and the positioning and prediction of small targets and high-speed moving targets are inaccurate; in terms of software, most existing style transfer methods improve their performance by deepening and widening the network or improving the loss function, but it is difficult to ensure accurate style transformation during the training process, and it is easy to produce style mapping deviations, which affects the accuracy of the autonomous driving system in judging road information and induces traffic accidents.

[0003] Most existing style transfer methods are based on generative adversarial networks and combined with encoder-decoder structures. The encoder is used to extract content features and style features simultaneously, and the two features are directly input into the decoder for decoding. At the same time, relevant loss functions are designed from the perspectives of color, content, smoothness, etc. to supervise the network to obtain stylized results. Specifically, the goals of the style transfer network are divided into two categories: ensuring the consistency of content feature transmission and accurate transformation of style features.

[0004] In terms of consistent transfer of content features, the invention patent with publication number CN107705242B adds depth of field loss to the target loss function by integrating the depth perception network to estimate the depth of field of the original image and the generated style image. In the process of style transfer, the generated image not only integrates the corresponding style and content, but also maintains the near and far structural information of the original image. The invention patent application named Image Migration Method Based on Mean Standard Deviation and with publication number CN13837926A constructs a feature space to store the feature information of different filters, so as to better obtain multi-scale and stable features, without the need for training on real data, and can flexibly perform style transformation.

[0005] In terms of precise transformation of style features, the invention patent application entitled "A method for multi-style conversion of images based on latent variable feature generation" with publication number CN11099225A, based on a multimodal unsupervised image conversion network, designs a style code generator to fit the style code of the image, introduces jump connections between content code and style code, and introduces an attention mechanism in style code, which improves the quality and diversity of multi-style conversion of images. The article entitled "Training GAN to explain classifiers in style space" found that image classification models can rely on multiple different semantic attributes of images. It trains a generative model to determine the specific attribute generation of style features, and can complete the precise transmission of detailed texture and edge contour information. The article entitled "Generative adversarial network with a unified framework for multi-domain image translation" proposes a mask vector method that enables it to control all available domain labels, thereby guiding the precise transmission of content and style features. Based on it, the article entitled "Multi-domain multi-image synthesis network" replaces domain labels with style features of specific domains to achieve style diversity and scalability.

[0006] Most of the above style transfer methods are trained in a closed environment based on the network's own performance, or independent labels are assigned to features. It is difficult to avoid the influence of various confusing factors such as different target attributes and external labels during the training process, resulting in deviations between the actual network output and the theoretical output. Therefore, how to effectively use the deep features extracted from the image to ensure the consistency of the content of the style transfer process, the accurate transformation of the style, and better apply it to traffic scenes and industrial scenes has become an urgent problem to be solved. Summary of the invention

[0007] The purpose of the present invention is to provide a structure-guided style deviation correction style transfer method and system, which divides the noisy attributes generated by style features into multiple groups through spatial attention, exchanges information between different groups and the same group respectively, weakens the noise, and adds spatial structural attributes similar to content features to the style features, guides the precise transformation of style features, reduces the mapping deviation in the feature transfer process, and can effectively achieve style transfer with consistent image content.

[0008] To achieve the above object, the technical solution of the present invention is: a structure-guided style deviation correction style transfer method, comprising:

[0009] Prepare a dataset for training the style transfer network;

[0010] Get the target domain input image with feature channel c and the source domain input image Perform downsampling operations including convolution and nonlinear activation function processing on it to obtain a four-dimensional feature vector and

[0011] Processing source domain input images using multi-layer residual units and dual pyramid networks The downsampling result Get the content feature vector

[0012] Input image to the target domain The downsampling result of The global average pooling function is used to obtain a four-dimensional feature vector For the four-dimensional feature vector Use the full connection function to get the four-dimensional feature vector

[0013] The four-dimensional feature vector After being processed by residual pyramid, deep convolutional neural network, information exchange, and point convolutional neural network, a four-dimensional feature vector is obtained.

[0014] The four-dimensional feature vector and the four-dimensional eigenvector Multiply to generate a four-dimensional feature vector Realize the redistribution of target attributes in style features and correct feature mapping deviation;

[0015] For the four-dimensional feature vector Perform structural sinusoidal normalization to obtain the style feature vector

[0016] The content feature vector and style feature vector Add and fuse to get the four-dimensional feature vector Then upsample and output the style transfer result Y c×2h×2w .

[0017] Furthermore, the target domain input image and the source domain input image Perform downsampling operations, specifically:

[0018] Use the convolution kernel M c×3×3 Extract the target domain input image Style features in and the source domain input image Content features in The formula is:

[0019]

[0020]

[0021] in For the convolution process, each matrix represents a 3×3 feature vector;

[0022] The output feature vector and Using nonlinear activation function processing, when the eigenvalue of the activation processing is less than or equal to 0, the output value of the activation function is 0, as shown in formula (3); conversely, when the output value of the activation function is the same as the input value, as shown in formula (4):

[0023]

[0024]

[0025] Among them, the function A(·) is the activation function.

[0026] Furthermore, a multi-layer residual unit and a dual pyramid network are used to process the source domain input image. The downsampling result of Specifically:

[0027] Using multi-layer residual units to process 4D feature vectors And output the four-dimensional feature vector The formula is:

[0028]

[0029]

[0030] Among them, F(·) is the single-layer residual unit process function, ω3 is the weight matrix;

[0031] Using a dual pyramid network to process four-dimensional feature vectors And output the content feature vector The formula is:

[0032]

[0033] Among them, F h (·) is the process function of the double pyramid network.

[0034] Furthermore, the target domain input image The downsampling result of The global average pooling function is used to obtain a four-dimensional feature vector For the four-dimensional feature vector Use the full connection function to get the four-dimensional feature vector Specifically:

[0035] Use global average pooling to average each unit feature and obtain a four-dimensional feature vector The formula is:

[0036]

[0037] Among them, P average (·) is the global average pooling function, M c×2×2 Perform pixel-by-pixel operations on the convolution kernel features with filter k=2, select the average value and output it;

[0038] For the four-dimensional feature vector Use the fully connected function to process each feature channel one by one to obtain a four-dimensional feature vector The formula is:

[0039]

[0040] Among them, C fully (·) is the fully connected function, and M is selected c×1×1 That is, the convolution kernel of the filter k=1 is operated.

[0041] Furthermore, the four-dimensional feature vector After being processed by residual pyramid, deep convolutional neural network, information exchange, and point convolutional neural network, a four-dimensional feature vector is obtained. Specifically:

[0042] Processing 4D feature vectors via residual pyramid And get the four-dimensional feature vector The formula is:

[0043]

[0044]

[0045] Among them, F rp (·) is the residual pyramid process function, ω4 is the weight matrix;

[0046] Use a deep convolutional neural network to transform the four-dimensional feature vector According to the characteristic channel, it is evenly divided into p branches (p≤c), and the characteristic components of each characteristic channel are obtained The formula is as follows:

[0047]

[0048] Among them, F deep (·) is the process function of the deep convolutional neural network;

[0049] Each branch is divided into q groups for random information exchange, disrupting the inherent order of information between different channels, and outputting characteristic components The formula is as follows:

[0050]

[0051] Among them, Shuffle(·) is the information exchange function;

[0052] Use point convolutional neural network to merge the feature vectors after group exchange to obtain a four-dimensional feature vector The point convolutional neural network will randomly delete some neurons during the merging process. The formula is as follows:

[0053]

[0054] Among them, D ran is the random deletion function, m is the proportion of randomly deleted neurons;

[0055]

[0056] Among them, F poi (·) is the point convolutional neural network process function, using M c×1×1 The point-wise convolutional neural network in the form of is used to process feature vectors.

[0057] Furthermore, the four-dimensional feature vector and the four-dimensional eigenvector Multiply to generate a four-dimensional feature vector Specifically:

[0058]

[0059] in, and is the weight matrix, and × represents the multiplication of the feature matrix.

[0060] Furthermore, for the four-dimensional feature vector Perform structural sinusoidal normalization to obtain the style feature vector The formula is:

[0061]

[0062] Among them, sin IN (·) is the structural sinusoidal normalized process function, μ(x) and μ(y) are the means of the length and width dimensions of the eigenvector, respectively, and σ(x) and σ(y) are the standard deviations of the length and width dimensions of the four-dimensional eigenvector, respectively.

[0063] The present invention also provides a structure-guided style deviation correction style migration system, comprising an encoding module, a structure-guided module and a decoding module;

[0064] The encoding module includes a style encoding module and a content encoding module. The style encoding module inputs the target domain image. Perform downsampling operation to obtain a four-dimensional feature vector The content encoding module uses downsampling, residual units, and dual pyramid networks to process the source domain input image in sequence. Get the content feature vector

[0065] The structure guidance module includes a feature optimization unit and a style correction unit. The feature optimization unit performs a four-dimensional feature vector The global average pooling function is used to obtain a four-dimensional feature vector Then the four-dimensional feature vector Use the full connection function to get the four-dimensional feature vector The style correction unit performs a four-dimensional feature vector After being processed by residual pyramid, deep convolutional neural network, information exchange, and point convolutional neural network, a four-dimensional feature vector is obtained. The four-dimensional feature vector and the four-dimensional eigenvector Multiply ω1 and ω2 at a fixed ratio to obtain a four-dimensional feature vector For the four-dimensional feature vector Perform structural sinusoidal normalization to obtain the style feature vector

[0066] The decoding module converts the content feature vector and style feature vector Add and fuse to get the four-dimensional feature vector Then upsample and output the style transfer result Y c×2h×2w .

[0067] Furthermore, the structure guidance module is expressed as:

[0068]

[0069] Among them, F opt (x) is the characteristic optimization unit process function, F s (x) is the style correction unit process.

[0070] By adopting the above technical solution, the present invention can achieve the following beneficial effects: it can be applied to real-life scenarios such as autonomous driving, industrial and service robots, realize style changes in any weather and environmental scenes, and provide assistance for accurate identification of small and fuzzy targets. The following is a list of the beneficial effects of the present invention:

[0071] (1) Applicable to small target features

[0072] The style correction unit in the present invention can accurately separate any style and weather information in the image, fully tap the deep features, and ensure the accurate transformation of any weather and any scene style under unsupervised conditions, greatly improving the recognition accuracy of small target features and providing protection for driving safety.

[0073] (2) Applicable to high-speed moving target characteristics

[0074] The present invention uses a feature optimization unit and a style correction unit to respectively extract content features and target attributes in an input image, and corrects style mapping deviations based on the extracted style attributes, thereby effectively improving blurring caused by high-speed target motion and realizing the extraction of high-speed moving targets.

[0075] (3) Applicable to public security monitoring systems

[0076] The present invention can achieve precise style feature transformation from both global and local levels for any illumination intensity image acquired by security surveillance cameras while ensuring the consistency of content features. It greatly improves the adaptability of security surveillance to complex scenes and provides effective safety protection for production and life.

[0077] (4) Applicable to autonomous driving technology

[0078] The present invention is a computer vision environment perception technology, which is applicable to the field of autonomous driving. It can extract the target features and positions of pedestrians, vehicles, buildings, traffic signs, etc. around the driving environment, provide comprehensive feature information for the style migration model, and provide strong protection for driving safety.

[0079] (5) Applicable to situations with unclear vision

[0080] The present invention is applicable to style transfer situations in different complex scenes. It restores the unclear target features under infrared and visible light conditions obtained by camera lenses with different exposure and clarity, and performs style transfer after improving the image clarity. BRIEF DESCRIPTION OF THE DRAWINGS

[0081] Figure 1 This is a flowchart of the structure-guided style deviation correction style transfer method;

[0082] Figure 2 is a schematic diagram of the structure of the structure guidance module;

[0083] Figure 3 This is a schematic diagram of the security monitoring style migration in Example 1;

[0084] Figure 4 is a schematic diagram of autonomous driving style migration in Example 2;

[0085] Figure 5 This is a schematic diagram of the style transfer of the visually blurred scene in Example 3. DETAILED DESCRIPTION

[0086] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application, that is, the described embodiments are only part of the embodiments of the present application, rather than all of the embodiments. The components of the embodiments of the present application described and shown in the drawings here can be arranged and designed in various different configurations.

[0087] The present invention proposes a structure-guided style deviation correction style transfer method and system, and designs a structure-guided module: first, by deepening the network to enhance the ability of deep feature extraction, ensure the integrity of image style features during style transfer, and reduce feature redundancy; second, expand the style semantic features into spatial information expression, and the main channel interacts multiple times in batches to enhance the correlation between different channel features, add a correct content expression attribute to each style feature, improve the phenomenon of mismatch between target attributes and style filling during feature transfer, and ensure accurate transformation of style features. Figure 1 The specific migration method implementation steps are as follows:

[0088] Step 1: Prepare a dataset for training the style transfer network, where the dataset size may be 2h×2w;

[0089] Step 2: Get the target domain input image with feature channel c and the source domain input image Perform downsampling operations including convolution and nonlinear activation function processing on it, and output a four-dimensional feature vector and Specifically:

[0090] (1) Use convolution kernel M with step size s=2 and filter k=3 c×3×3 Extract the target domain input image Style features in and the source domain input image Content features in The formula is:

[0091]

[0092]

[0093] in For the convolution process, each matrix represents a 3×3 feature vector;

[0094] (2) Output the feature vector and Using nonlinear activation function processing, when the eigenvalue of the activation processing is less than or equal to 0, the output value of the activation function is 0, as shown in formula (3); conversely, when the output value of the activation function is the same as the input value, as shown in formula (4):

[0095]

[0096]

[0097] Among them, function A(·) is an activation function. Using the activation function to perform nonlinear processing on the feature vector can improve the effectiveness of the feature vector and reduce feature redundancy, which helps to achieve accurate style transfer.

[0098] Step 3: To reduce the redundancy of content features, use multiple layers (preferably 4 layers) of residual units and a dual pyramid network to process the four-dimensional feature vector Ensure that the image content is complete during the style transfer process, specifically:

[0099] (1) Using multi-layer residual unit processing And output the four-dimensional feature vector The formula is:

[0100]

[0101]

[0102] Among them, F(·) is the single-layer residual unit process function, ω3 is the weight matrix;

[0103] (2) Using a dual pyramid network to process four-dimensional feature vectors And output the four-dimensional feature vector Enhance the ability to extract and perceive multi-scale targets. The formula is:

[0104]

[0105] Among them, F h (·) is the process function of the double pyramid network;

[0106] Step 4: The feature optimization unit input is the target domain input image The downsampling result of Global average pooling and full connection functions are used in sequence to ensure the complete transmission of style features. Specifically:

[0107] (1) Use global average pooling to average each unit feature and obtain a four-dimensional feature vector The formula is:

[0108]

[0109] Among them, P average (·) is the global average pooling function, M c×2×2 Perform pixel-by-pixel operations on the convolution kernel with k=2, select the average value and output it;

[0110] (2) Yes Use the fully connected function to process each feature channel one by one, reduce the influence of pixel and feature position on feature classification, and output a four-dimensional feature vector The formula is:

[0111]

[0112] Among them, C fully (·) is the fully connected function, and M is selected c×1×1 That is, the convolution kernel of the filter k=1 is operated;

[0113] Step 5: The input of the style correction unit is The four-dimensional feature vector output after downsampling and global average pooling The four-dimensional feature vector After being processed by residual pyramid, deep convolutional neural network, information exchange, and point convolutional neural network, a four-dimensional feature vector is output. In the process of network training, the reflection mechanism is introduced to provide spatial information for the style features of semantic forms and enhance its spatial perception ability of three-dimensional objects. Specifically:

[0114] (1) In order to enhance the reading of multi-scale target attributes for style features, the present invention designs a residual pyramid to process the four-dimensional feature vector And output the four-dimensional feature vector The formula is:

[0115]

[0116]

[0117] Among them, F rp (·) is the residual pyramid process function, and ω4 is the weight matrix. ;

[0118] (2) Using deep convolutional neural network According to the characteristic channel, it is evenly divided into p branches (p≤c), and the characteristic components of each characteristic channel are obtained The formula is as follows:

[0119]

[0120] Among them, Fdeep (·) is the process function of the deep convolutional neural network;

[0121] (3) Each branch is divided into q groups for random information exchange, disrupting the inherent order of information between different channels and outputting characteristic components The formula is as follows:

[0122]

[0123] Among them, Shuffle(·) is an information exchange function, which divides the channel features on each branch into q groups, and randomly shuffles the order between each group and between different groups to seek new style and attribute matching relationships;

[0124] (4) Use the point convolutional neural network to merge the feature vectors after group exchange and output the style correction unit result The feature reorganization and fusion between different feature channels provide greater possibilities for accurate content feature transmission. The point convolutional neural network will randomly delete some neurons during the merging process. The formula is as follows:

[0125]

[0126] Among them, D ran is a random deletion function, and m is the ratio of randomly deleted neurons. This operation can prevent the network from overfitting.

[0127]

[0128] Among them, F poi (·) is the point convolutional neural network process function, using M c×1×1 The point convolutional neural network in the form of processing feature vectors;

[0129] Step 6: Output the output from step 4 and step 5 output Multiply to generate a four-dimensional feature vector To achieve the redistribution of target attributes in style features and correct feature mapping deviation, the formula is:

[0130]

[0131] in, and is the weight matrix, × represents the multiplication of the feature matrix;

[0132] Step 7: To change the style feature data distribution and realize accurate style feature transfer, the present invention performs a feature vector Perform structural sinusoidal normalization to suppress feature information irrelevant to the style and output a four-dimensional feature vector To prepare for integration with content features, the formula is:

[0133]

[0134] Among them, sin IN (·) is the structural sinusoidal normalized process function, μ(x) and μ(y) are the means of the length and width dimensions of the eigenvector, respectively, and σ(x) and σ(y) are the standard deviations of the length and width dimensions of the four-dimensional eigenvector, respectively.

[0135] Step 8: Features the content output from step 3 And the style features output in step 7 Add and fuse to get the four-dimensional feature vector Upsample and output the style transfer result Y in the decoding module c×2h×2w .

[0136] This embodiment also provides a system for implementing the above method, which includes an encoding module, a structure guidance module and a decoding module; each part is described in detail below:

[0137] The encoding module includes a style encoding module and a content encoding module. The style encoding module encodes the target domain input image Perform downsampling operation to obtain a four-dimensional feature vector The content encoding module uses downsampling, residual units, and dual pyramid networks to process the source domain input image in sequence. Get the content feature vector Enhance the perception of spatial structure of content features.

[0138] The structure guidance module includes a feature optimization unit and a style correction unit. It takes the four-dimensional feature vector of the target domain input image as input after downsampling, and uses global average pooling to abstract the style features into one-dimensional semantic information to ensure accurate extraction of style features. Specifically, the four-dimensional feature vector of a single feature channel is input into the feature optimization unit, and the advantages of the fully connected function's excellent classification performance and deep feature extraction are used to improve the diversity of style features and reduce feature redundancy; the style correction unit completes the spatial expansion of low-dimensional style features, adds spatial information with similar structures to the content features in the source domain and gives them structural attributes. Through multi-feature channel and batch training, each style feature is given an expression similar to the source domain, and the spatial structure perception ability of the style feature is enhanced, thereby achieving accurate correspondence between one-dimensional semantics and two-dimensional spatial pixels. The output results of different units are multiplied in the form of attention to correct the deviation of style feature mapping when there is only one-dimensional semantics, achieve accurate classification of style features, and implement consistency in image content and style transmission. The feature optimization unit and style correction unit in the structure guidance module are described in detail below: The input of the style correction unit is a four-dimensional feature vector with a size of 1×1 and containing c feature channels after global average pooling in the feature optimization unit. Among them, the characteristic optimization unit process function is F opt (x); the style correction unit process function is F s (x); Assume that after the output features of the two units are fused, the input to the next stage is a four-dimensional feature vector containing c feature channels and a size of h×w The structure guidance module is expressed as:

[0139]

[0140] The feature optimization unit uses a fully connected function to Focus on the attribute classification of style features, reduce the impact of target structure and position on the deep extraction of style features, and output a four-dimensional feature vector Provide assistance for the subsequent fusion of structural information and semantic style information.

[0141] The style correction unit uses a residual pyramid pair While performing dimensional expansion processing, it reduces feature redundancy, adds structural information such as detailed texture and contour, and outputs a four-dimensional feature vector In order to enhance the correlation between different feature channels and fully extract the hidden structural features in the input image of the target domain, a deep convolution with a convolution kernel of 3×3 is used to transform It is divided into p branches, and the style feature information on each feature channel is extracted respectively. Different branches learn and supervise each other to achieve reference across feature channels. At the same time, the feature channels in each branch are divided into q groups, and the channel order is disrupted within each group and between different groups to increase the randomness of hidden style attributes, improve the generalization ability of the network, and give each style feature different categories of target classification and structural attributes to reduce feature mapping deviation. In order to obtain accurate style attributes, the feature filtering of each branch is obtained. Use a 1×1 point convolutional neural network to integrate the feature information of p branches and output a four-dimensional feature vector Each style feature is given an enhanced feature expression to guide the precise filling of structural information in the style feature. and Multiply ω1 and ω2 at a fixed ratio and output a four-dimensional feature vector Hidden attributes and inherent style features are used for adversarial screening to fix the problem of incorrect style feature assignment and reduce the deviation of style feature mapping, thus helping to achieve style transfer with consistent content and precise style.

[0142] The decoding module performs the fusion operation of the source domain and the target domain feature vector: the four-dimensional feature vector and Add and fuse to get the four-dimensional feature vector Then upsample and output the style transfer result Y c×2h×2w .

[0143] In this embodiment, the characteristic parameter constraint condition may be:

[0144] (1) Downsample the input RGB three-channel image with a size of 256×256, and reduce the input image size to 128×128, feature channels n∈{4,8,16,64,256,512}, and output a four-dimensional feature vector containing image content features, any of {1,128,128,4}, {1,128,128,8}, {1,128,128,16}, {1,128,128,64}, {1,128,128,256}, and {1,128,128,512}.

[0145] (2) The structure guidance module selects four-dimensional feature vectors of different feature channels as input according to the different contents of the input image in the target domain: when the input image has complex colors and contains a large number of small targets and blurred targets, the four-dimensional feature vector with a feature channel of c = 256 is selected as the input of the content calibration module; when the input image has neutral colors and contains no small targets or blurred targets, the four-dimensional feature vector with a feature channel of c = 8 is selected as the input of the content calibration module.

[0146] (3) The feature channel transmitted in the style correction unit is a four-dimensional feature vector c∈{4,8,16,64,256,512}.

[0147] (4) A four-dimensional feature vector with a feature channel c=1 is transmitted in the feature optimization unit.

[0148] Structural unit constraints can be:

[0149] (1) The style correction unit contains p branches, and p∈{0,1,2,3,4}. When p=0, the structure guidance module only contains the feature optimization unit.

[0150] (2) Each branch in the style correction unit contains q groups, where q = {q|10≤q≤512,q∈Z+}.

[0151] (3) The style correction unit selects different numbers of groups according to the complexity of the input image style: when the input image has complex colors and contains small or blurred objects, the number of groups is q = {q|128≤q≤512,q∈Z+}; when the input image has neutral colors and contains no small or blurred objects, the number of groups is q = {q|10≤q≤128,q∈Z +}The number of groups.

[0152] Example 1: Security monitoring style transfer

[0153] This example is used to monitor unmanned places such as schools and crossroads. The application of this invention in outdoor security monitoring can effectively improve the recognition ability of targets under complex lighting conditions. Figure 3 shown.

[0154] Example 2: Autonomous driving style transfer

[0155] This example performs style transfer on an autonomous driving system. The present invention is applied to a vehicle-mounted camera to sense the driving environment, provide driving assistance, reduce traffic accident rates, and improve the vehicle's safe driving capabilities. The autonomous driving style transfer is shown in the following figure. Figure 4 shown.

[0156] Example 3: Style transfer in visually blurred scenes

[0157] This example aims at style transfer of visually blurred scenes caused by uneven lighting or natural weather. It can improve the image quality of style transfer in complex scenes and prepare for the next step of target detection or image segmentation. Figure 5 shown.

[0158] Those skilled in the art will appreciate that the embodiments described herein are intended to help readers understand the principles of the present invention, and should be understood that the scope of protection of the invention is not limited to such specific statements and embodiments. Any possible equivalent substitutions or changes made based on the above description are considered to fall within the scope of protection of the claims of the present invention.

Claims

1. A structure-guided style deviation correction style transfer method, characterized in that: include: Prepare a dataset for training the style transfer network, the dataset size is 2h×2w; Get the target domain input image with feature channel c and the source domain input image Perform downsampling operations including convolution and nonlinear activation function processing on it to obtain a four-dimensional feature vector and Processing source domain input images using multi-layer residual units and dual pyramid networks The downsampling result Get the content feature vector Input image to the target domain The downsampling result The global average pooling function is used to obtain a four-dimensional feature vector For the four-dimensional feature vector Use the full connection function to get the four-dimensional feature vector The four-dimensional feature vector After being processed by residual pyramid, deep convolutional neural network, information exchange, and point convolutional neural network, a four-dimensional feature vector is obtained. The four-dimensional feature vector and the four-dimensional eigenvector Multiply to generate a four-dimensional feature vector Y1 c×h×w , realize the redistribution of target attributes in style features and correct feature mapping deviation; For the four-dimensional feature vector Y1 c×h×w Perform structural sinusoidal normalization to obtain the style feature vector The content feature vector and style feature vector Add and fuse to get the four-dimensional feature vector Then upsample and output the style transfer result Y c×2h×2w ; Input image to the target domain The downsampling result The global average pooling function is used to obtain a four-dimensional feature vector For the four-dimensional feature vector Use the full connection function to get the four-dimensional feature vector Specifically: Use global average pooling to average each unit feature and obtain a four-dimensional feature vector The formula is: Among them, P average (·) is the global average pooling function, M c×2×2 Perform pixel-by-pixel operations on the convolution kernel features with filter k=2, select the average value and output it; For the four-dimensional feature vector Use the fully connected function to process each feature channel one by one to obtain a four-dimensional feature vector The formula is: Among them, C fully (·) is the fully connected function, and M is selected c×1×1 That is, the convolution kernel of the filter k=1 is operated; For the four-dimensional feature vector Y1 c×h×w Perform structural sinusoidal normalization to obtain the style feature vector The formula is: Among them, sin IN (·) is the structural sinusoidal normalized process function, μ(x) and μ(y) are the means of the length and width dimensions of the eigenvector, respectively, and σ(x) and σ(y) are the standard deviations of the length and width dimensions of the four-dimensional eigenvector, respectively.

2. According to claim 1, a structure-guided style deviation correction style transfer method is characterized in that: Input image to the target domain and the source domain input image Perform downsampling operations, specifically: Use the convolution kernel M c×3×3 Extract the target domain input image Style features in and the source domain input image Content features in The formula is: in For the convolution process, each matrix represents a 3×3 feature vector; The output feature vector and Using nonlinear activation function processing, when the eigenvalue of the activation processing is less than or equal to 0, the output value of the activation function is 0, as shown in formula (3); conversely, when the output value of the activation function is the same as the input value, as shown in formula (4): Among them, the function A(·) is the activation function.

3. According to claim 2, a structure-guided style deviation correction style transfer method is characterized in that: Processing source domain input images using multi-layer residual units and dual pyramid networks The downsampling result Specifically: Using multi-layer residual units to process 4D feature vectors And output the four-dimensional feature vector The formula is: Among them, F i is the process function of a single-layer residual unit, ω3 is the weight matrix; Using a dual pyramid network to process four-dimensional feature vectors And output the content feature vector The formula is: Among them, F h (·) is the process function of the double pyramid network.

4. According to claim 3, a structure-guided style deviation correction style transfer method is characterized in that: The four-dimensional feature vector After being processed by residual pyramid, deep convolutional neural network, information exchange, and point convolutional neural network, a four-dimensional feature vector is obtained. Specifically: Processing 4D feature vectors through residual pyramid And get the four-dimensional feature vector The formula is: Among them, F rp (·) is the residual pyramid process function, ω4 is the weight matrix; Use a deep convolutional neural network to transform the four-dimensional feature vector According to the characteristic channel, it is evenly divided into p branches, p≤c, and the characteristic components of each characteristic channel are obtained. The formula is as follows: Among them, F deep (·) is the process function of the deep convolutional neural network; Each branch is divided into q groups for random information exchange, disrupting the inherent order of information between different channels, and outputting characteristic components The formula is as follows: Among them, Shuffle(·) is the information exchange function; Use point convolutional neural network to merge the feature vectors after group exchange to obtain a four-dimensional feature vector The point convolutional neural network will randomly delete some neurons during the merging process. The formula is as follows: Among them, D ran is the random deletion function, m is the proportion of randomly deleted neurons; Among them, Fpoi(·) is the point convolutional neural network process function, using M c×1×1 The point-wise convolutional neural network in the form of is used to process feature vectors.

5. The structure-guided style deviation correction style transfer method according to claim 4, characterized in that: The four-dimensional feature vector and the four-dimensional eigenvector Multiply to generate a four-dimensional feature vector Y1 c×h×w , specifically: in, and is the weight matrix, and × represents the multiplication of the feature matrix.

6. A structure-guided style deviation correction style transfer system, used to implement the method according to any one of claims 1 to 5, characterized in that: It includes an encoding module, a structure guidance module and a decoding module; The encoding module includes a style encoding module and a content encoding module. The style encoding module inputs an image to the target domain. Perform downsampling operation to obtain a four-dimensional feature vector The content encoding module uses downsampling, residual units, and dual pyramid networks to process the source domain input image in sequence. Get the content feature vector The structure guidance module includes a feature optimization unit and a style correction unit. The feature optimization unit performs a four-dimensional feature vector The global average pooling function is used to obtain a four-dimensional feature vector Then the four-dimensional feature vector Use the full connection function to get the four-dimensional feature vector The style correction unit performs a four-dimensional feature vector After being processed by residual pyramid, deep convolutional neural network, information exchange, and point convolutional neural network, a four-dimensional feature vector is obtained. The four-dimensional feature vector and the four-dimensional eigenvector Multiply ω1 and ω2 at a fixed ratio to obtain the four-dimensional feature vector Y1 c×h×w , for the four-dimensional feature vector Y1 c×h×w Perform structural sinusoidal normalization to obtain the style feature vector The decoding module converts the content feature vector and style feature vector Add and fuse to get the four-dimensional feature vector Then upsample and output the style transfer result Y c×2h×2w .

7. The structure-guided style deviation correction style transfer system according to claim 6, characterized in that: The structure guidance module is expressed as: where F opt (x) is the characteristic optimization unit process function, F s (x) is the style correction unit process.

Images