Mobile terminal real-time image style migration method based on lightweight convolutional network

Abstract

The invention relates to the technical field of image processing, and particularly discloses a mobile terminal real-time image style migration method based on a lightweight convolutional network, and the method comprises the steps: obtaining an original image inputted by a user, and converting the original image into standardized image tensor data through a pixel normalization and dynamic partitioning strategy; performing feature coding on the image tensor data by using a lightweight convolutional neural network, and extracting a multi-scale feature map layer by layer through a deep separable convolutional layer and a bottleneck layer structure; based on dual-channel feature coding, dynamically performing weighted fusion of style features and content features through multi-scale convolution kernel parallel deployment and learnable parameters, and generating a fused feature map; according to the method, the pixel normalization and dynamic partitioning strategy is introduced, the image partitioning size is adaptively adjusted, the calculation efficiency and the feature retention degree are balanced, the calculation burden is reduced while the high image quality is ensured, and the model calculation amount is reduced by adopting the lightweight convolutional neural network and the depth separable convolution in combination with the bottleneck layer structure.

Description

Technical Field

[0001] The present invention belongs to the field of image processing technology, and in particular relates to a mobile terminal real-time image style transfer method based on a lightweight convolutional network. Background Art

[0002] With the improvement of smartphone and mobile device performance, image style transfer has gradually become widely used as an efficient artistic creation tool. Image style transfer aims to apply the style of one image to the content of another image, creating images with artistic effects. It is widely used in social media, photography, and creative design.

[0003] However, existing style transfer methods are typically based on complex convolutional neural networks, which have high requirements for computing resources. This results in large delays when running on mobile devices, making real-time processing difficult. Traditional image style transfer techniques often struggle to strike a balance between computational efficiency and image quality when processing high-resolution images. This is especially true in environments with limited computing resources, such as mobile devices, where the balance between style and content features can be imbalanced, resulting in suboptimal image effects. In addition, style transfer methods are typically unable to flexibly adjust the feature extraction and fusion methods of images, resulting in excessive or insufficient fusion of style and content information, affecting the quality of the final image. To overcome these problems, many methods have begun to explore how to improve the computational efficiency and image quality of style transfer algorithms based on lightweight convolutional networks, especially for mobile device applications.

[0004] Therefore, it is necessary to propose a real-time image style transfer method for mobile terminals based on lightweight convolutional networks to solve the problem of how to achieve efficient image style transfer on mobile devices with limited computing resources in the existing technology.

[0005] The above information disclosed in this background technology is only for enhancing understanding of the background technology of the present invention and therefore it may contain information that does not constitute the prior art that is already known to a person of ordinary skill in the art. Summary of the Invention

[0006] The purpose of the present invention is to provide a mobile terminal real-time image style transfer method based on a lightweight convolutional network to solve the problems raised in the above background technology.

[0007] To achieve the above object, the present invention provides the following technical solutions:

[0008] A mobile real-time image style transfer method based on a lightweight convolutional network, including:

[0009] Obtain the original image input by the user and convert it into standardized image tensor data through pixel normalization and dynamic blocking strategy;

[0010] Performing feature encoding on the image tensor data using a lightweight convolutional neural network to generate a dual-channel feature encoding containing semantic information and style information;

[0011] Based on the dual-channel feature encoding, weighted fusion of style features and content features is performed through parallel deployment of multi-scale convolution kernels and dynamic learnable parameters to generate a fused feature map;

[0012] The fused feature maps are decoded and reconstructed, and the spatial resolution is gradually restored by combining transposed convolution with sub-pixel upsampling. The multi-level features of the encoding stage are directly injected into the corresponding layers of the decoder through cross-layer connections, implementing fine-grained feature reuse and gradient flow.

[0013] An adaptive post-processing unit is embedded to dynamically adjust the color mapping curve and sharpening intensity of the output image through a lightweight convolutional network based on the local contrast and global tonal distribution of the feature map with restored spatial resolution, and output the target image after style transfer.

[0014] Preferably, the converting of the original image into standardized image tensor data by pixel normalization and dynamic blocking strategy includes:

[0015] Use bilinear interpolation algorithm to downsample the original image to generate a base image that matches the mobile screen resolution;

[0016] Calculate the local gradient amplitude of the base image based on the Canny edge detection algorithm to generate an edge density heat map;

[0017] Divide the heat map into non-overlapping blocks of 16×16 pixels and calculate the variance of the gradient amplitude in each block;

[0018] When the block variance exceeds the preset threshold, the block is subdivided into 8×8 pixel sub-blocks until the variances of all blocks meet the threshold requirement;

[0019] Variance threshold adaptive formula:

[0020] T block =μ grad +β·σ grad

[0021] β=log(1+λ·Complexity)

[0022] Where, T block is the block variance threshold, μ grad is the mean value of the gradient amplitude of the base image, σ gradis the standard deviation of the gradient amplitude of the base image, β is the dynamic adjustment coefficient, Complexity is the image content complexity index, defined as ω1·EdgeDensity+ω2·ColorEntropy, ω1 and ω2 are the weights of EdgeDensity and ColorEntropy respectively, and λ is a learnable scaling factor, initialized to 0.1;

[0023] Perform zero-mean-unit-variance normalization on the final block to generate a sequence of floating-point tensors that conform to the NCHW format.

[0024] Preferably, the steps of constructing the lightweight convolutional neural network include:

[0025] (1) Depthwise Separable Convolution Module Design:

[0026] Decompose the standard convolution kernel into a cascade structure of a 3×3 depth convolution kernel and a 1×1 point-by-point convolution kernel;

[0027] A channel separation strategy is adopted in the deep convolution layer to split the input feature map into multiple single-channel groups along the channel dimension, and each group performs spatial convolution operations independently;

[0028] A channel reorganization strategy is adopted in the point-by-point convolution layer to perform cross-channel information fusion through 1×1 convolution kernel;

[0029] (2) Construction of bottleneck layer compression unit:

[0030] Insert 1×1 convolution kernels between consecutive convolution layers to compress the number of feature map channels to 1 / 4 of the original size and generate a low-dimensional intermediate representation;

[0031] A parameterized ReLU activation function is used to optimize the nonlinear expression ability of the feature map channel, where the slope of the negative semi-axis is dynamically adjusted through learnable parameters;

[0032] The dynamic slope formula of the parameterized ReLU activation function is:

[0033]

[0034] Where x is the input eigenvalue, a=σ(W t f feat ), a is the negative semi-axis slope, σ(·) is the Sigmoid activation function, W is the learnable weight vector, f feat is the characteristic statistics vector;

[0035] (3) Integration of attention enhancement module:

[0036] The channel attention branch adopts the SE-Net structure, generates channel descriptors through global average pooling, and generates channel weight vectors through two layers of fully connected networks;

[0037] The spatial attention branch adopts the CBAM structure, generates a spatial weight matrix through a 7×7 convolution kernel, and performs an outer product operation with the channel attention weight to generate a feature mask.

[0038] Preferably, the multi-scale feature fusion and style transfer steps include:

[0039] (1) Multi-scale convolution kernel group configuration:

[0040] 1×1, 5×5, and 9×9 convolution kernels and global average pooling layers are deployed in parallel to extract texture details, structural contours, and semantic information features respectively;

[0041] Each scale convolution kernel is followed by a batch normalization layer and a LeakyReLU activation function to optimize multi-scale feature expression;

[0042] (2) Dynamic weight generator construction:

[0043] Construct a three-layer fully connected network, where the input layer receives edge density map, color entropy value, and texture complexity index;

[0044] The hidden layer uses 64 neurons, and the output layer generates a three-dimensional weight vector, which is normalized by the Softmax function and then assigned to each scale feature;

[0045] (3) Weighted fusion unit operation:

[0046] Perform tensor product operation on the content feature map and the style feature map according to the weight vector to generate a weighted feature set;

[0047] Adopting the residual connection strategy, the dual-channel feature encoding and weighted features are element-wise added to generate a fused feature map;

[0048] Residual connection formula:

[0049] F fused =F content Θw c +F style Θw s +γ·F enc

[0050] Where, F fused is the fused feature map, F content is the content feature map, F style is the style feature map, F enc is the encoder feature map, w c is the content feature weight vector, w s is the style feature weight vector, Θ is the channel-by-channel weighted operation, and γ is the residual strength coefficient, which is initialized to 1.

[0051] Preferably, the step of restoring spatial resolution and feature multiplexing includes:

[0052] (1) Design of transposed convolution upsampling layer:

[0053] Deploy 2×2 transposed convolution kernels at each decoder level with stride=2 for resolution multiplication;

[0054] Use depth-wise separable convolution structure to replace standard transposed convolution to optimize computational complexity;

[0055] (2) Construction of sub-pixel upsampling module:

[0056] A sub-pixel convolution layer is deployed in the final decoding layer to perform 4x super-resolution reconstruction through a periodic shuffle operation;

[0057] A pixel reorganization strategy is used to reorganize the channel dimension of the low-resolution feature map generated by the encoder into a spatial dimension;

[0058] (3) Cross-layer connection channel configuration:

[0059] Perform channel concatenation on the feature map of the encoder layer i and the feature map of the decoder layer i+1;

[0060] The number of feature channels is adaptively adjusted through a 1×1 convolution kernel to generate decoder feature maps of consistent size.

[0061] Preferably, the execution steps of the adaptive post-processing unit include:

[0062] Divide the decoder feature map into 8×8 non-overlapping blocks and calculate the mean and standard deviation of the RGB channels in each block;

[0063] Construct a 3×3 separable convolution layer, take the mean-standard deviation vector as input, and generate a local tone mapping curve;

[0064] Local tone mapping curve generation formula:

[0065] C out (j,k)=W curve Concat(μ RGB ,σ RGB )+b curve

[0066] Where Cout(j,k) is the output tone mapping curve, μ RGB is the RGB mean vector of the 8×8 image block, σ RGB is the RGB standard deviation vector of the 8×8 image block, W curve is the separable convolution kernel weight, b curve is the bias vector, Concat(·) is the channel concatenation operation;

[0067] The cubic spline interpolation algorithm is used to smooth the mapping curve and eliminate the color level break phenomenon.

[0068] Preferably, the execution step of the adaptive post-processing unit further includes:

[0069] Apply the Laplacian convolution kernel to analyze and process the decoder feature map to generate the initial edge response matrix;

[0070] Non-maximum suppression is used to perform non-maximum suppression on the initial edge response matrix, retaining only the local gradient maximum points to form a refined edge skeleton;

[0071] Based on the global threshold calculated by the Otsu algorithm, the edge skeleton is binarized to generate an edge response map with spatial continuity;

[0072] Apply the Laplacian operator to detect the edge response map and calculate the global edge intensity histogram;

[0073] Based on the global edge intensity histogram, the Otsu algorithm is used to maximize the inter-class variance and calculate the segmentation threshold of edge and non-edge areas;

[0074] Based on the segmentation threshold, the Unsharp Masking algorithm is applied to sharpen the edge area, and the bilateral filter is used to smooth the non-edge area to optimize the clarity of image details.

[0075] Compared with the prior art, the present invention has the following beneficial effects:

[0076] This invention introduces pixel normalization and dynamic blocking strategies to adaptively adjust the image block size, balance computational efficiency and feature retention, ensure high image quality while reducing the computational burden, and adopts lightweight convolutional neural networks and depthwise separable convolution, combined with a bottleneck layer structure, to reduce the amount of model calculations. By extracting features at different levels in parallel and dynamically adjusting the fusion weights, the weighted fusion of style and content is optimized, making the generated image more artistic and realistic. The decoder combines transposed convolution and sub-pixel upsampling technology to restore high-resolution images and enhance detail expression. Finally, color mapping and sharpening are dynamically adjusted to eliminate color cast and blurring effects in style transfer, further optimizing image quality. BRIEF DESCRIPTION OF THE DRAWINGS

[0077] Figure 1 This is a flow chart of the mobile terminal real-time image style transfer method based on lightweight convolutional network of the present invention. DETAILED DESCRIPTION

[0078] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0079] Example 1:

[0080] See also Figure 1 As shown in the figure, the mobile real-time image style transfer method based on lightweight convolutional network includes:

[0081] Obtain the original image input by the user, convert it into standardized image tensor data through pixel normalization and dynamic blocking strategy, and adaptively adjust the blocking size based on the complexity of the image content to balance computational efficiency and feature retention;

[0082] Use bilinear interpolation algorithm to downsample the original image to generate a base image that matches the mobile screen resolution;

[0083] Calculate the local gradient amplitude of the base image based on the Canny edge detection algorithm and generate an edge density heat map;

[0084] Divide the heat map into non-overlapping blocks of 16×16 pixels and calculate the variance of the gradient amplitude within each block;

[0085] When the block variance exceeds the preset threshold, the block is subdivided into 8×8 pixel sub-blocks until the variances of all blocks meet the threshold requirement;

[0086] Perform zero-mean-unit-variance normalization on the final block to generate a sequence of floating-point tensors that conform to the NCHW format.

[0087] Furthermore, through an intelligent adaptive blocking mechanism, the processing granularity is dynamically adjusted based on the local complexity of the image, significantly reducing the computational load while preserving key image features. Combined with a standardized tensor conversion process, this not only ensures the efficiency of subsequent neural network processing, but also significantly improves the mobile device's processing capabilities for high-resolution images, achieving an optimal balance between computational efficiency and image quality. This innovative design enables the system to maintain excellent real-time processing performance even in resource-constrained environments.

[0088] A lightweight convolutional neural network is used to encode the features of image tensor data. Multi-scale feature maps are extracted layer by layer through a deep separable convolutional layer and bottleneck layer structure. A channel attention mechanism is introduced to weightedly reorganize feature channels. A spatial attention mechanism is used to enhance the spatial dimension of key areas, generating a dual-channel feature code that contains both semantic and style information.

[0089] The steps to build a lightweight convolutional neural network include:

[0090] (1) Depthwise Separable Convolution Module Design:

[0091] Decompose the standard convolution kernel into a cascade structure of a 3×3 depth convolution kernel and a 1×1 point-by-point convolution kernel;

[0092] A channel separation strategy is adopted in the deep convolution layer to split the input feature map into multiple single-channel groups along the channel dimension, and each group performs spatial convolution operations independently;

[0093] A channel reorganization strategy is adopted in the point-by-point convolution layer to perform cross-channel information fusion through 1×1 convolution kernel;

[0094] (2) Construction of bottleneck layer compression unit:

[0095] Insert 1×1 convolution kernels between consecutive convolution layers to compress the number of feature map channels to 1 / 4 of the original size and generate a low-dimensional intermediate representation;

[0096] A parameterized ReLU activation function is used to optimize the nonlinear expression ability of the feature map channel, where the slope of the negative semi-axis is dynamically adjusted through learnable parameters;

[0097] (3) Integration of attention enhancement module:

[0098] The channel attention branch adopts the SE-Net structure, generates channel descriptors through global average pooling, and generates channel weight vectors through two layers of fully connected networks;

[0099] The spatial attention branch adopts the CBAM structure, generates a spatial weight matrix through a 7×7 convolution kernel, and performs an outer product operation with the channel attention weight to generate a feature mask.

[0100] Furthermore, through the design of depth-wise separable convolution and bottleneck layer structure, the computational complexity will be reduced while maintaining the feature extraction capability; the innovative dual-path architecture of channel and spatial attention mechanism enables the network to automatically focus on important feature channels and key image areas, improving feature expression capabilities; the combination of parameterized ReLU activation function and feature masking technology further enhances the network's ability to capture complex style features, achieving more accurate style semantic encoding while reducing model complexity.

[0101] Based on the multi-scale feature map, multi-scale convolution kernels are used to extract low-level texture features, mid-level structural features, and high-level semantic features in parallel. Based on the edge density, color distribution entropy, and texture complexity index of the target image, the fusion weights of each scale feature are dynamically generated through learnable parameters. Combined with dual-channel feature encoding, the weighted fusion of style features and content features is performed to generate a fused feature map.

[0102] The multi-scale feature fusion and style transfer steps include:

[0103] (1) Multi-scale convolution kernel group configuration:

[0104] 1×1, 5×5, and 9×9 convolution kernels and global average pooling layers are deployed in parallel to extract texture details, structural contours, and semantic information features respectively;

[0105] Each scale convolution kernel is followed by a batch normalization layer and a LeakyReLU activation function to optimize multi-scale feature expression;

[0106] (2) Dynamic weight generator construction:

[0107] Construct a three-layer fully connected network, where the input layer receives edge density map, color entropy value, and texture complexity index;

[0108] The hidden layer uses 64 neurons, and the output layer generates a three-dimensional weight vector, which is normalized by the Softmax function and then assigned to each scale feature;

[0109] (3) Weighted fusion unit operation:

[0110] Perform tensor product operation on the content feature map and the style feature map according to the weight vector to generate a weighted feature set;

[0111] A residual connection strategy is used to perform element-wise addition of the dual-channel feature encoding and weighted features to generate a fused feature map.

[0112] Furthermore, by deploying convolution kernels of different scales in parallel, multi-level feature extraction from local texture to global semantics is achieved, improving feature expression capabilities. An innovative dynamic weight generation mechanism automatically optimizes the feature fusion ratio based on image content characteristics, enhancing the naturalness of style transfer. Combined with a weighted fusion strategy using residual connections, this approach achieves an optimal balance between style and content features while preserving the original image's content details, improving the artistic expression score of the generated image. This design significantly improves the quality and controllability of style transfer.

[0113] The fused feature maps are decoded and reconstructed, and the spatial resolution is gradually restored by combining transposed convolution with sub-pixel upsampling. The multi-level features of the encoding stage are directly injected into the corresponding layers of the decoder through cross-layer connections, implementing fine-grained feature reuse and gradient flow.

[0114] The steps to restore spatial resolution and feature reuse include:

[0115] (1) Design of transposed convolution upsampling layer:

[0116] Deploy 2×2 transposed convolution kernels at each decoder level with stride=2 for resolution multiplication;

[0117] Use depth-wise separable convolution structure to replace standard transposed convolution to optimize computational complexity;

[0118] (2) Construction of sub-pixel upsampling module:

[0119] A sub-pixel convolution layer is deployed in the final decoding layer to perform 4x super-resolution reconstruction through a periodic shuffle operation;

[0120] A pixel reorganization strategy is used to reorganize the channel dimension of the low-resolution feature map generated by the encoder into a spatial dimension;

[0121] (3) Cross-layer connection channel configuration:

[0122] Perform channel concatenation on the feature map of the encoder layer i and the feature map of the decoder layer i+1;

[0123] The number of feature channels is adaptively adjusted through a 1×1 convolution kernel to generate decoder feature maps of consistent size.

[0124] Furthermore, the innovative combination of transposed convolution and sub-pixel upsampling reduces computational complexity while maintaining image reconstruction quality. The cross-layer connection design effectively addresses the feature loss problem in traditional codec networks, improving detail recovery. The application of a depthwise separable structure and a channel adaptive adjustment mechanism reduces memory usage while ensuring resolution restoration accuracy. This solution achieves an optimal balance between high-quality image reconstruction and mobile computing efficiency.

[0125] An adaptive post-processing unit is embedded to dynamically adjust the color mapping curve and sharpening strength of the output image based on the local contrast and global tonal distribution of the feature map with restored spatial resolution through a lightweight convolutional network. This eliminates the color cast and blurring effects that may occur during the style transfer process and outputs the target image after style transfer.

[0126] The execution steps of the adaptive post-processing unit include:

[0127] Divide the decoder feature map into 8×8 non-overlapping blocks and calculate the mean and standard deviation of the RGB channels in each block;

[0128] Construct a 3×3 separable convolution layer, take the mean-standard deviation vector as input, and generate a local tone mapping curve;

[0129] The cubic spline interpolation algorithm is used to smooth the mapping curve and eliminate the color level break phenomenon.

[0130] Furthermore, adaptive tone correction technology based on local color statistics eliminates color casts and improves the color fidelity of the output image. A lightweight convolutional network dynamically adjusts sharpening parameters to improve edge clarity while maintaining image naturalness. Block processing and separable convolution reduce post-processing computational complexity, ensuring real-time processing on mobile devices. This solution effectively addresses color distortion and blurred details in style transfer, enhancing the artistic expression of the final output image.

[0131] Apply the Laplacian convolution kernel to analyze and process the decoder feature map to generate the initial edge response matrix;

[0132] Non-maximum suppression is used to perform non-maximum suppression on the initial edge response matrix, retaining only the local gradient maximum points to form a refined edge skeleton;

[0133] Based on the global threshold calculated by the Otsu algorithm, the edge skeleton is binarized to generate an edge response map with spatial continuity;

[0134] Apply the Laplacian operator to detect the edge response map and calculate the global edge intensity histogram;

[0135] Based on the global edge intensity histogram, the Otsu algorithm is used to maximize the inter-class variance and calculate the segmentation threshold of edge and non-edge areas;

[0136] Based on the segmentation threshold, the Unsharp Masking algorithm is applied to sharpen the edge area, and the bilateral filter is used to smooth the non-edge area to optimize the clarity of image details.

[0137] Furthermore, through multi-level edge detection and optimization processing, the clarity of image edges is significantly improved while maintaining a natural transition effect; the innovative adaptive threshold segmentation technology combined with the regional differentiation processing strategy eliminates the blur effect while reducing the artifact generation rate; this solution has excellent computational efficiency and the edge processing speed is faster than traditional methods, achieving professional-level detail enhancement effects on mobile terminals.

[0138] Example 2:

[0139] App Name: Real-time Image Style Transfer App

[0140] The Real-Time Image Style Transfer app uses lightweight convolutional neural network technology to enable real-time image style transfer on mobile devices. Users can select different artistic styles and apply them to their own photos or target images, quickly generating artistically pleasing images.

[0141] 1. Functional Module

[0142] 1. Image input module:

[0143] Users can take a photo via camera or select a target image from the photo album.

[0144] The application supports multiple image formats, such as JPEG, PNG, etc.

[0145] 2. Style selection module:

[0146] Provides a variety of predefined art styles for users to choose from, such as Van Gogh style, Monet style, cartoon style, etc.

[0147] Users can preview the effects of each style to make their choice.

[0148] 3. Style transfer processing module (core module):

[0149] (1) Pixel normalization and dynamic blocking:

[0150] Perform pixel normalization on the input target image to make it meet the processing requirements of the neural network.

[0151] A dynamic blocking strategy is adopted to adaptively adjust the block size according to the complexity of image content, balancing computational efficiency and feature retention.

[0152] (2) Feature coding:

[0153] Use lightweight convolutional neural networks to perform feature encoding on image tensor data.

[0154] Multi-scale feature maps are extracted layer by layer through the depth-wise separable convolutional layer and bottleneck layer structure.

[0155] Channel attention mechanism and spatial attention mechanism are introduced to perform weighted reorganization and reinforcement of feature channels and key areas.

[0156] (3) Style feature fusion:

[0157] Based on the multi-scale feature map, low-level texture features, mid-level structural features and high-level semantic features are extracted in parallel through multi-scale convolution kernels.

[0158] According to the edge density, color distribution entropy and texture complexity index of the target image, the fusion weights of each scale feature are dynamically generated.

[0159] The style features and content features are weightedly fused together by combining dual-channel feature encoding to generate a fused feature map.

[0160] (4) Decoding and reconstruction:

[0161] The fused feature map is decoded and reconstructed, and the spatial resolution is gradually restored by combining transposed convolution with sub-pixel upsampling.

[0162] Through cross-layer connections, the multi-level features of the encoding stage are directly injected into the corresponding layers of the decoder, performing fine-grained feature reuse and gradient flow.

[0163] (5) Adaptive post-processing:

[0164] According to the local contrast and global tone distribution of the feature map of restored spatial resolution, the color mapping curve and sharpening intensity of the output image are dynamically adjusted through a lightweight convolutional network.

[0165] Eliminate color cast and blurring effects that may occur during style transfer and improve image quality.

[0166] (6) Result display and storage module:

[0167] Display the image effect after style transfer for users to preview.

[0168] Provide a save function, allowing users to save the results to the photo album or share them on social media platforms.

[0169] 2. Application Process Example

[0170] 1. The user launches the app and selects the "Style Transfer" feature.

[0171] 2. The user selects the target image: it can be taken from the camera or selected from the album.

[0172] 3. User selects art style: Browse and select the art style you like in the style selection module.

[0173] 4. Apply style transfer processing: Automatically perform pixel normalization, feature encoding, style feature fusion, decoding and reconstruction, and adaptive post-processing in the background.

[0174] 5. Display results: The application displays the image after style transfer, and users can preview it.

[0175] 6. Save or Share: Users can choose to save the results to their photo album or share them with friends via social media platforms.

[0176] 3. Technical advantages

[0177] Real-time: Utilizes lightweight convolutional neural networks and depthwise separable convolution technology to achieve real-time processing on mobile devices.

[0178] High quality: Through multi-scale feature extraction and dynamic weight fusion strategy, the style-transferred images are ensured to have high artistic sense and realism.

[0179] Flexibility: Supports multiple artistic style options to meet the personalized needs of different users.

[0180] Ease of use: The simple and clear user interface and operation process make it easy for users to get started quickly.

[0181] With this real-time image style transfer app, users can easily transfer the selected image style to the target image and enjoy the fun of artistic creation.

[0182] Example 3:

[0183] An embodiment of the present invention further provides a computer-readable storage medium, on which is stored a program for any of the above-mentioned methods for real-time image style transfer on a mobile terminal based on a lightweight convolutional network. When the program is executed by a processor, the various processes of the above-mentioned embodiments of the image style transfer method are implemented and the same technical effects are achieved. To avoid repetition, the details are not described here. The computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0184] In the description of this specification, the reference terms "one embodiment", "some embodiments", "examples", "specific examples" or "some examples" mean that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic expressions of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art can combine and combine different embodiments or examples described in this specification and features of different embodiments or examples without contradiction.

[0185] The drawings of the embodiments disclosed in the present invention only involve structures related to the embodiments disclosed in the present invention. Other structures may refer to common designs. In the absence of conflicts, the same embodiment and different embodiments of the present invention may be combined with each other.

[0186] The flowcharts shown in the accompanying drawings are for illustrative purposes only and do not necessarily include all contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps may be decomposed, combined, or partially merged, so the actual execution order may vary depending on the actual situation.

[0187] While embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions, and variations may be made to these embodiments without departing from the principles and spirit of the invention, and that the scope of the invention is defined by the appended claims and their equivalents.

Claims

1. A mobile real-time image style transfer method based on a lightweight convolutional network, characterized by: include: Obtain the original image input by the user and convert it into standardized image tensor data through pixel normalization and dynamic blocking strategy; Performing feature encoding on the image tensor data using a lightweight convolutional neural network to generate a dual-channel feature encoding containing semantic information and style information; Based on the dual-channel feature encoding, weighted fusion of style features and content features is performed through parallel deployment of multi-scale convolution kernels and dynamic learnable parameters to generate a fused feature map; The fused feature maps are decoded and reconstructed, and the spatial resolution is gradually restored by combining transposed convolution with sub-pixel upsampling. The multi-level features of the encoding stage are directly injected into the corresponding layers of the decoder through cross-layer connections, implementing fine-grained feature reuse and gradient flow. An adaptive post-processing unit is embedded to dynamically adjust the color mapping curve and sharpening intensity of the output image through a lightweight convolutional network based on the local contrast and global tonal distribution of the feature map with restored spatial resolution, and output the target image after style transfer.

2. The method for real-time image style transfer based on a lightweight convolutional network for mobile terminals according to claim 1, characterized in that: The method of converting the original image into standardized image tensor data through pixel normalization and dynamic blocking strategy includes: Use bilinear interpolation algorithm to downsample the original image to generate a base image that matches the mobile screen resolution; Calculate the local gradient amplitude of the base image based on the Canny edge detection algorithm to generate an edge density heat map; Divide the heat map into non-overlapping blocks of 16×16 pixels and calculate the variance of the gradient amplitude in each block; When the block variance exceeds the preset threshold, the block is subdivided into 8×8 pixel sub-blocks until the variances of all blocks meet the threshold requirement; Variance threshold adaptive formula: T block =μ grad +b·s grad β=log(1+λ·Complexity) Where, T block is the block variance threshold, μ grad is the mean value of the gradient amplitude of the base image, σ grad is the standard deviation of the gradient amplitude of the base image, β is the dynamic adjustment coefficient, Complexity is the image content complexity index, defined as ω1·EdgeDensity+ω2·ColorEntropy, ω1 and ω2 are the weights of EdgeDensity and ColorEntropy respectively, and λ is a learnable scaling factor, initialized to 0.1; Perform zero-mean-unit-variance normalization on the final block to generate a sequence of floating-point tensors that conform to the NCHW format.

3. The method for real-time image style transfer based on a lightweight convolutional network for mobile terminals according to claim 2, characterized in that: The steps of constructing the lightweight convolutional neural network include: (1) Depthwise Separable Convolution Module Design: Decompose the standard convolution kernel into a cascade structure of a 3×3 depth convolution kernel and a 1×1 point-by-point convolution kernel; A channel separation strategy is adopted in the deep convolution layer to split the input feature map into multiple single-channel groups along the channel dimension, and each group performs spatial convolution operations independently; A channel reorganization strategy is adopted in the point-by-point convolution layer to perform cross-channel information fusion through 1×1 convolution kernel; (2) Construction of bottleneck layer compression unit: Insert 1×1 convolution kernels between consecutive convolution layers to compress the number of feature map channels to 1 / 4 of the original size and generate a low-dimensional intermediate representation; A parameterized ReLU activation function is used to optimize the nonlinear expression ability of the feature map channel, where the slope of the negative semi-axis is dynamically adjusted through learnable parameters; The dynamic slope formula of the parameterized ReLU activation function is: Where x is the input eigenvalue, a=σ(W t f feat ), a is the negative semi-axis slope, σ(·) is the Sigmoid activation function, W is the learnable weight vector, f feat is the characteristic statistics vector; (3) Integration of attention enhancement module: The channel attention branch adopts the SE-Net structure, generates channel descriptors through global average pooling, and generates channel weight vectors through two layers of fully connected networks; The spatial attention branch adopts the CBAM structure, generates a spatial weight matrix through a 7×7 convolution kernel, and performs an outer product operation with the channel attention weight to generate a feature mask.

4. The method for real-time image style transfer based on a lightweight convolutional network for mobile terminals according to claim 3, characterized in that: The multi-scale feature fusion and style transfer steps include: (1) Multi-scale convolution kernel group configuration: 1×1, 5×5, and 9×9 convolution kernels and global average pooling layers are deployed in parallel to extract texture details, structural contours, and semantic information features respectively; Each scale convolution kernel is followed by a batch normalization layer and a LeakyReLU activation function to optimize multi-scale feature expression; (2) Dynamic weight generator construction: Construct a three-layer fully connected network, where the input layer receives edge density map, color entropy value, and texture complexity index; The hidden layer uses 64 neurons, and the output layer generates a three-dimensional weight vector, which is normalized by the Softmax function and then assigned to each scale feature; (3) Weighted fusion unit operation: Perform tensor product operation on the content feature map and the style feature map according to the weight vector to generate a weighted feature set; Adopting the residual connection strategy, the dual-channel feature encoding and weighted features are element-wise added to generate a fused feature map; Residual connection formula: F fused =F content Θw c +F style Θw s +γ·F enc Where, F fused is the fused feature map, F content is the content feature map, F style is the style feature map, F enc is the encoder feature map, w c is the content feature weight vector, w s is the style feature weight vector, Θ is the channel-by-channel weighted operation, and γ is the residual strength coefficient, which is initialized to 1.

5. The method for real-time image style transfer on a mobile terminal based on a lightweight convolutional network according to claim 4, characterized in that: The steps of restoring spatial resolution and feature multiplexing include: (1) Design of transposed convolution upsampling layer: Deploy 2×2 transposed convolution kernels at each decoder level with stride=2 for resolution multiplication; Use depth-wise separable convolution structure to replace standard transposed convolution to optimize computational complexity; (2) Construction of sub-pixel upsampling module: A sub-pixel convolution layer is deployed in the final decoding layer to perform 4x super-resolution reconstruction through a periodic shuffle operation; A pixel reorganization strategy is used to reorganize the channel dimension of the low-resolution feature map generated by the encoder into a spatial dimension; (3) Cross-layer connection channel configuration: Perform channel concatenation on the feature map of the encoder layer i and the feature map of the decoder layer i+1; The number of feature channels is adaptively adjusted through a 1×1 convolution kernel to generate decoder feature maps of consistent size.

6. The method for real-time image style transfer based on a lightweight convolutional network for mobile terminals according to claim 5, characterized in that: The execution steps of the adaptive post-processing unit include: Divide the decoder feature map into 8×8 non-overlapping blocks and calculate the mean and standard deviation of the RGB channels in each block; Construct a 3×3 separable convolution layer, take the mean-standard deviation vector as input, and generate a local tone mapping curve; Local tone mapping curve generation formula: C out (j,k)=W curve ·Concat(μ RGB ,σ RGB )+b curve Where Cout(j,k) is the output tone mapping curve, μ RGB is the RGB mean vector of the 8×8 image block, σ RGB is the RGB standard deviation vector of the 8×8 image block, W curve is the separable convolution kernel weight, b curve is the bias vector, Concat(·) is the channel concatenation operation; The cubic spline interpolation algorithm is used to smooth the mapping curve and eliminate the color level break phenomenon.

7. The method for real-time image style transfer based on a lightweight convolutional network for mobile terminals according to claim 6, characterized in that: The execution step of the adaptive post-processing unit further includes: Apply the Laplacian convolution kernel to analyze and process the decoder feature map to generate the initial edge response matrix; Non-maximum suppression is used to perform non-maximum suppression on the initial edge response matrix, retaining only the local gradient maximum points to form a refined edge skeleton; Based on the global threshold calculated by the Otsu algorithm, the edge skeleton is binarized to generate an edge response map with spatial continuity; Apply the Laplacian operator to detect the edge response map and calculate the global edge intensity histogram; Based on the global edge intensity histogram, the Otsu algorithm is used to maximize the inter-class variance and calculate the segmentation threshold of edge and non-edge areas; Based on the segmentation threshold, the Unsharp Masking algorithm is applied to sharpen the edge area, and the bilateral filter is used to smooth the non-edge area to optimize the clarity of image details.

Images