A multi-scene image style transfer and edge computing optimization method

Abstract

The present application relates to the technical field of deep learning and computer vision, in particular to a multi-scene image style transfer and edge computing optimization method, comprising: determining an optimal optimization strategy through an automatic strategy search agent; according to the optimal optimization strategy, simultaneously performing knowledge distillation and structured pruning on a student model using a unified loss function; and using a precision adaptive loss function containing a perception loss to perform quantization perception training on the model. The present application builds an automatic closed-loop optimization framework driven by a composite reward function, considers multiple links such as model pruning and quantization in a coordinated manner, solves the problem that traditional optimization methods rely on manual work, are inefficient and are difficult to balance model performance and artistic quality, and can automatically generate a customized model with the best overall performance for specific edge hardware and multi-scene application requirements.

Description

Technical Field

[0001] This invention relates to the fields of deep learning and computer vision technology, specifically to a multi-scene image style transfer and edge computing optimization method. Background Technology

[0002] Existing image style transfer techniques typically employ deep neural network-based methods. To run on edge devices such as mobile phones, a lightweight network is usually used to replace the large backbone network, followed by a series of independent compression steps such as model pruning and quantization to reduce the computational and storage overhead of the model.

[0003] However, its optimization process is step-by-step and relies heavily on manual experience, with a lack of coordination between the optimization steps, which can easily lead to the accumulation of model accuracy loss. At the same time, manually adjusting a large number of optimization parameters to find the best balance between model performance and perception quality under varying hardware platforms and diverse artistic scenes is a tedious, time-consuming process that is difficult to achieve the global optimum.

[0004] To address this, a multi-scene image style transfer and edge computing optimization method is proposed. Summary of the Invention

[0005] The purpose of this invention is to provide a multi-scene image style transfer and edge computing optimization method, which solves the problems of low optimization efficiency and difficulty in balancing model quality and performance in the prior art through a unified automated closed-loop optimization process.

[0006] To achieve the above objectives, the present invention provides the following technical solution: A multi-scene image style transfer and edge computing optimization method includes: The optimal optimization strategy is determined by an automated strategy search agent, and the optimal optimization strategy guides the following optimization process: A joint training approach is adopted, which uses a unified loss function to simultaneously perform knowledge distillation and structured pruning on the student model in a single training phase to generate an intermediate model; the unified loss function includes task loss, feature distillation loss and sparsity regularization loss. The intermediate model is trained using a precision adaptation loss function, which includes a perception loss term to measure the visual perception difference between the output of the full-precision model and the quantized model. The agent's policy search process determines the optimal optimization policy by maximizing the composite reward function. The composite reward function is generated based on the performance indicators of the compressed quantization model measured on the target edge hardware and the perceptual quality score measured on a preset multi-scene image verification set. The compressed quantization model is the final model obtained by executing the optimization process.

[0007] Preferably, in the joint training method, the sparsity regularization loss of the unified loss function is calculated for the learnable affine parameters in each batch normalized layer after the convolutional layer in the network topology of the student model; the sparsity regularization loss is a regularization term that penalizes the L1 norm of the scaling factor of all specified layers, and the regularization term, together with the task loss and feature distillation loss included in the unified loss function, participates in the iterative optimization process of gradient backpropagation and weight update of the student model; after the single training phase ends, a channel pruning operation is performed, which identifies channels with scaling factor values ​​lower than a preset threshold as channels to be pruned, removes the channels to be pruned and their associated convolutional kernels and batch normalized layer parameters from the model computation graph, and changes the network topology of the student model.

[0008] Preferably, the perceptual loss term is calculated using a deep neural network pre-trained on a large image dataset. This deep neural network is used to extract depth-sensing features from the image, and the calculation steps include: The output images of the intermediate model and the student model in the quantization perception training process are respectively passed as inputs to the deep neural network; the activation feature maps of the two output images on multiple pre-selected intermediate layers of the deep neural network are extracted and recorded; the norm distance of the tensor between the two sets of corresponding activation feature maps is calculated, and the norm distance is used as the value of the perception loss term to guide the fine-tuning of the parameters of the student model in the quantization perception training so as to minimize the distance between the two sets of activation feature maps.

[0009] Preferably, the automated policy search agent is a reinforcement learning agent that learns the optimal optimization strategy by interacting with the simulation environment. In each interaction, the reinforcement learning agent generates a complete optimization strategy by outputting a set of actions. The optimization strategy includes setting a structured pruning rate for each layer and module of the student model and selecting the number of quantization bits for quantized perception training. The simulation environment executes the complete optimization process according to the optimization strategy and generates the final compressed quantization model. The calculation steps of the composite reward function are as follows: the perception quality score of the compressed quantization model measured on a multi-scene image validation set is normalized with the inference latency, memory usage, and power consumption scores physically measured on the target edge hardware, and then weighted and summed according to preset weight coefficients to obtain the final reward value, which is returned to the reinforcement learning agent for updating the decision strategy.

[0010] Preferably, the task loss in the unified loss function includes content loss and style loss. The content loss measures the feature space difference between the content of the image generated by the student model and the content of the original content image. The style loss measures the feature statistical difference between the style of the image generated by the student model and the style of the target style image. The feature distillation loss is obtained by calculating the distance between the feature maps output by the student model and the pre-trained teacher model on the n intermediate layers of their respective networks.

[0011] Preferably, the accuracy adaptation loss function is a composite function, which includes a task loss for measuring content and style differences and a perceptual loss term for measuring visual perception differences.

[0012] Preferably, the multi-scene image verification set is a set of image pairs generated by combining a content image source dataset and a style image source dataset; wherein, the content image source dataset contains real-world images with diverse scene and object categories; and the style image source dataset contains images of artworks covering different art movements, periods and authors.

[0013] Preferably, the student model is built on a lightweight network architecture, the convolution operations of which consist of depthwise separable convolutions, and the lightweight network architecture uses an inverted residual structure containing a linear bottleneck as the basic building block of its network.

[0014] Compared with the prior art, the beneficial effects of the present invention are as follows: 1. The original sequential stages of "training-distillation-retraining-pruning-fine-tuning" are integrated into a unified training process, significantly shortening the model optimization cycle and saving substantial computational and human resources. Through a unified loss function, the model learns the knowledge of the teacher model while simultaneously learning the importance of its own network structure. This co-evolutionary approach avoids the potential for inadvertently damaging key knowledge points that might result from "learning first, then pruning," allowing knowledge to be more firmly embedded in a more compact network structure.

[0015] 2. By incorporating perceptual loss terms such as LPIPS into the fine-tuning objective of quantized perceptual training, this invention shifts the optimization focus from "pursuing mathematical similarity of pixel values" to "pursuing similarity in human visual perception." This allows the model to prioritize retaining features crucial to artistic style during parameter quantization, such as subtle texture details, smooth color gradients, and unique brushstroke textures. Especially in multi-scenario applications, this perceptual loss based on a general visual model ensures that the optimized model maintains high-quality, distortion-free visual effects when processing different types of images, such as portraits, landscapes, and abstract paintings, thereby significantly improving the artistic expression and user experience of the final application deployed on edge devices.

[0016] 3. The present invention proposes "determining the optimal optimization strategy by maximizing the composite reward function through an automated strategy search agent," fundamentally changing the current situation where model optimization relies on human experience, is time-consuming and labor-intensive, and is difficult to achieve optimality. By introducing a reinforcement learning agent, the present invention achieves end-to-end automation of the entire complex optimization process. Its composite reward function is directly linked to the measured performance of the target edge hardware and the perceptual quality of the multi-scenario validation set. This means that the present invention can automatically search and generate a customized compressed model that performs best under specific constraints for any specified hardware platform and application requirements. This not only achieves global optimization but also perfectly solves the adaptation problem caused by hardware fragmentation in the field of edge computing, greatly improving the scalability and efficiency of deploying advanced AI models across different devices. Attached Figure Description

[0017] Figure 1 This is a flowchart illustrating a multi-scene image style transfer and edge calculation optimization method proposed in an embodiment of the present invention. Figure 2 This is a flowchart illustrating the joint training process of the unified loss function proposed in this embodiment of the invention. Figure 3 This is a schematic diagram of the lightweight student model structure proposed in an embodiment of the present invention. Detailed Implementation

[0018] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0019] Example 1 Please see Figures 1 to 2 This invention provides a method for optimizing style transfer and edge computing in multi-scene images, the technical solution of which is as follows: A multi-scene image style transfer and edge computing optimization method, such as Figure 1 As shown, it includes: The optimal optimization strategy is determined by an automated strategy search agent, and the optimal optimization strategy guides the following optimization process: A joint training approach is adopted, which uses a unified loss function to simultaneously perform knowledge distillation and structured pruning on the student model in a single training phase to generate an intermediate model; the unified loss function includes task loss, feature distillation loss and sparsity regularization loss. The intermediate model is trained using a precision adaptation loss function, which includes a perception loss term to measure the visual perception difference between the output of the full-precision model and the quantized model. The agent's policy search process determines the optimal optimization policy by maximizing the composite reward function. The composite reward function is generated based on the performance indicators of the compressed quantization model measured on the target edge hardware and the perceptual quality score measured on a preset multi-scene image verification set. The compressed quantization model is the final model obtained by executing the optimization process.

[0020] Furthermore, in the joint training method, the sparsity regularization loss of the unified loss function is calculated for the learnable affine parameters in each batch normalized layer after the convolutional layer in the network topology of the student model. The sparsity regularization loss is a regularization term that penalizes the L1 norm of the scaling factor of all specified layers. The regularization term, together with the task loss and feature distillation loss included in the unified loss function, participates in the iterative optimization process of gradient backpropagation and weight update of the student model. After the single training phase is completed, a channel pruning operation is performed. The channel pruning operation identifies channels with scaling factor values ​​lower than a preset threshold as channels to be pruned, removes the channels to be pruned and their associated convolutional kernels and batch normalized layer parameters from the model computation graph, and changes the network topology of the student model.

[0021] To make the pruning process more instructive, the attention of the teacher model is incorporated as a reference when determining the channels to be pruned. Specifically, the system calculates and compares the impact of removing different channels on the feature distillation loss. For channels whose removal would significantly increase the distillation loss, even if their scaling factor has a low L1 norm, the system assigns a "protective weight" to reduce the probability of pruning them. This teacher-guided pruning strategy can more effectively preserve the network structure that is crucial to the model's perceptual quality, thereby achieving higher model accuracy at the same compression ratio.

[0022] During each iteration of model parameter updates, in addition to calculating the total gradient, the system pays special attention to the gradient values ​​generated separately by the feature distillation loss and flowing to the scaling factors of each batch normalization layer. The larger the distillation loss gradient received by the scaling factor of a channel, the more critical that channel is for mimicking the feature representation of the teacher model. The system continuously tracks and accumulates the distillation gradient magnitude of each channel using a moving average method, forming a stable "teacher guidance importance" score. Accordingly, the final channel pruning decision adopts a two-stage process: First, channels with scaling factor L1 norm values ​​below a first preset threshold are selected as candidate channels for pruning; then, these candidate channels are reviewed, and if their "teacher guidance importance" scores are higher than a second preset threshold, they are removed from the candidate set and protected; finally, pruning is performed only on channels that pass the two-stage selection.

[0023] The specific steps for a single iteration within joint training are as follows: First, take a batch of content images from the content dataset and a batch of style images from the style dataset.

[0024] The content image is input into the encoder of the student model to extract content features; at the same time, the style image is input into the same encoder to extract style features.

[0025] By applying adaptive instance normalization, statistics of style features are applied to content features to generate a fused target feature.

[0026] The fused target features are then input into the decoder of the student model for upsampling and reconstruction, ultimately generating a batch of stylized output images.

[0027] The preset threshold is not a fixed value, but is dynamically calculated from the numerical distribution of all or some scaling factors after the end of the single training phase, based on a preset global pruning rate or hierarchical pruning rate; its calculation method includes: After training, the scaling factors of all batch normalized layers to be pruned in the student model are extracted and their absolute values ​​are calculated. According to a pre-set target pruning rate (e.g., set to 70%), the absolute values ​​of all extracted scaling factors are sorted in ascending order, and the value located at the 70% percentile is selected as the preset threshold. By setting a pruning rate instead of a fixed value, precise control over the final sparsity and computational cost of the model is achieved. This overcomes the unpredictable pruning effect of traditional fixed threshold methods under different models or training epochs, greatly enhancing the stability and robustness of the optimization process. When combined with an automated policy search agent, which determines the pruning rate for each network layer, this method enables more refined structural optimization tailored to specific hardware and application goals. This allows for the discovery of optimal pruning strategies that surpass human experience within an automated process, resulting in a final compressed model with a better balance between performance and accuracy.

[0028] Furthermore, the perceptual loss term is calculated using a deep neural network pre-trained on a large image dataset. This deep neural network is used to extract depth-sensing features from the image, and the calculation steps include: The output images of the intermediate model and the student model in the quantization perception training process are respectively passed as inputs to the deep neural network; the activation feature maps of the two output images on multiple pre-selected intermediate layers of the deep neural network are extracted and recorded; the norm distance of the tensor between the two sets of corresponding activation feature maps is calculated, and the norm distance is used as the value of the perception loss term to guide the fine-tuning of the parameters of the student model in the quantization perception training so as to minimize the distance between the two sets of activation feature maps.

[0029] The pre-trained deep neural network used to calculate the perceptual loss term can employ a VGG-19 network pre-trained on the large image dataset ImageNet. The VGG-19 network was chosen as the feature extractor because its multi-level convolutional structure is widely recognized for its ability to effectively capture rich image features, from low-level textures to high-level semantics, which aligns well with the requirements for measuring the perceptual quality of artistic style. The VGG-19 network selects a set of convolutional layers distributed at different depths; for example, this set may include conv1_1, conv2_1, conv3_1, conv4_1, and conv5_1. Shallow networks such as conv1_1 can extract basic features such as edges and colors; mid-level networks such as conv3_1 can capture more complex textures and patterns; while deep networks such as conv5_1 can respond to the overall structure and high-level content of the image. By calculating and minimizing the distance between the activated feature maps of two images at all these selected levels, it can be ensured that the generated quantized model maintains a high degree of consistency with the output of the full-precision model across all perceptual dimensions, from basic brushstrokes to the overall composition.

[0030] This invention employs a deep network-based perceptual loss to guide quantization training. It shifts the optimization objective from traditional, pixel-level mathematical errors inconsistent with human vision to perceptual similarity alignment within the deep feature space. This allows the quantization process to accurately preserve texture details, color gradations, and structural contours crucial to artistic style, effectively suppressing visual artifacts such as blurring and color distortion often introduced by low-bit quantization. This method ensures that the model maintains maximum fidelity in artistic expression when handling diverse scenes such as portraits and landscapes, significantly improving the visual quality and user experience of style transfer applications ultimately deployed on edge devices.

[0031] Furthermore, the automated policy search agent is a reinforcement learning agent that learns the optimal optimization policy by interacting with the simulation environment. In each interaction, the reinforcement learning agent generates a complete optimization policy by outputting a set of actions. The optimization policy includes setting a structured pruning rate for each layer and module of the student model and selecting the number of quantization bits for quantized perception training. The simulation environment executes the complete optimization process according to the optimization policy and generates the final compressed quantization model. The calculation steps of the composite reward function are as follows: the perception quality score of the compressed quantization model measured on a multi-scene image validation set is normalized with the inference latency, memory usage, and power consumption scores physically measured on the target edge hardware, and then weighted and summed according to preset weight coefficients to obtain the final reward value, which is returned to the reinforcement learning agent for updating the decision policy.

[0032] The simulation environment includes an early termination mechanism when executing the optimization process. This mechanism monitors the loss convergence status of the model training under the current policy in real time. If it is determined that the policy cannot converge effectively or has poor performance, the current optimization process is terminated early, and a lower reward value is returned to accelerate the overall policy search process.

[0033] To ensure smooth performance in real-time applications, an additional "performance stability" penalty term is introduced into the composite reward function. This penalty term is calculated based on the variance or 99th percentile latency measured during multiple consecutive inference operations on the target hardware. A model with low average latency but occasional high-latency "glitch" will receive a lower reward than a model with slightly higher average latency but stable performance. This mechanism ensures a smooth final application, thus improving the user experience.

[0034] The weighting coefficients in the composite reward function are pre-set as hyperparameters by developers or deployers based on the specific application scenario requirements. These coefficients define the priority of each optimization objective. For example: For a real-time video filter application that strives for the smoothest experience, it may assign higher weights to the "inference latency" and "power consumption" scores, while giving a relatively lower weight to the "perceived quality score".

[0035] Conversely, for an offline rendering application used for professional image processing, the "perceived quality score" is given a very high weight, while the weight of performance metrics is relatively low.

[0036] The specific algorithm used in the reinforcement learning agent employs the proximal policy optimization algorithm. The proximal policy optimization algorithm is a policy gradient method. The reinforcement learning agent outputs its current optimization policy. After interacting with the simulated environment and obtaining a composite reward value, the proximal policy optimization algorithm updates the agent's network parameters by calculating the advantage function and using a pruned objective function.

[0037] To further improve the learning efficiency of the agent, this invention can also introduce a course-based learning strategy to train the agent. In the initial training phase, the agent learns on relatively simple tasks, such as searching policies on smaller models or using simplified reward functions. After initial convergence, the difficulty of the tasks is gradually increased, for example, switching to a complete model and a complete composite reward function for fine-tuning. This learning paradigm, from easy to difficult, helps the agent grasp the inherent rules of optimization strategies more quickly, thereby accelerating its convergence speed in complex search spaces.

[0038] This invention's automated optimization scheme fully automates the originally extremely tedious, time-consuming, and highly experience-dependent tuning process through reinforcement learning agents. It can shorten the development and debugging cycle from several weeks to the time required for automated machine search, significantly improving optimization efficiency. The agent can explore a high-dimensional parameter space far beyond the intuitive scope of human engineers, thereby discovering deeper synergistic relationships between optimization steps and finding a globally optimal strategy that better balances model performance and multi-scenario perception quality, rather than merely a locally optimal solution. The composite reward function is directly linked to the physical measured performance of the target edge hardware, ensuring that the final generated model is tailored to the characteristics of a specific chip, achieving true hardware-aware optimization. By adjusting the weight coefficients in the reward function, optimization objectives can be flexibly defined, allowing the same framework to produce optimal models with different characteristics according to different application needs, possessing extremely high adaptability and engineering application value.

[0039] Furthermore, the task loss in the unified loss function includes content loss and style loss. The content loss measures the feature space difference between the content of the image generated by the student model and the content of the original content image. The style loss measures the feature statistical difference between the style of the image generated by the student model and the style of the target style image. The feature distillation loss is obtained by calculating the distance between the feature maps output by the student model and the pre-trained teacher model on the n intermediate layers of their respective networks.

[0040] Among them, such as Figure 2 As shown, the steps for calculating the unified loss function are as follows: Content loss calculation: The stylized output image generated during joint training is input again into the encoder of the student model to extract features, and the L2 norm distance between the feature and the generated fused target feature is calculated.

[0041] Style loss calculation: The stylized output image and the original style image are respectively input into a fixed, pre-trained teacher model (e.g., VGG-19), and their respective feature maps are extracted on multiple pre-selected intermediate layers. The sum of the L2 norm distances between the mean and standard deviation of the two sets of feature maps is calculated.

[0042] Feature distillation loss calculation: The original content image is input into the encoders of the student model and the teacher model, respectively. The L2 norm distance between the feature maps output by the two models at several corresponding intermediate layers is calculated. Here, n can be 4, meaning 4 layers are selected for feature alignment.

[0043] Sparsity regularization loss calculation: Calculate the sum of the L1 norms of the scaling factors of all or specified batch normalized layers in the student model network.

[0044] The calculated content loss, style loss, feature distillation loss, and sparsity regularization loss are weighted and summed according to preset weight coefficients to obtain the final total loss value of the unified loss function.

[0045] Finally, gradient backpropagation is performed based on the total loss value, and the network parameters of the student model encoder and decoder are updated using the optimizer. This process is repeated for multiple iterations until the model converges.

[0046] By constructing and adopting a unified loss function that incorporates four types of losses—content, style, distillation, and sparsity—the goals of style learning, knowledge transfer, and structural simplification, which originally required multiple independent stages, are integrated into a single training iteration and a one-time gradient update, achieving collaborative optimization. This approach avoids the potential conflicts and cumulative accuracy losses between different optimization objectives in traditional step-by-step processes, significantly improving overall training efficiency and convergence stability. It forces the student model to simultaneously consider three mutually constraining goals at each step of parameter learning: style accuracy, inheritance of knowledge from the teacher model, and the sparsity of its own network structure.

[0047] Furthermore, the accuracy adaptation loss function is a composite function, which includes a task loss for measuring content and style differences and a perceptual loss term for measuring visual perception differences.

[0048] The specific implementation steps for using the precision-adaptive loss function are as follows: In the computational graph of the pruned intermediate model, pseudo-quantized nodes are inserted after the weights or activation values. These pseudo-quantized nodes simulate the precision rounding errors caused by low-bit integers during forward propagation, while allowing gradients to pass through without loss during backward propagation.

[0049] Then, taking a batch of content images and style images as input, perform two forward propagations: First step: Input data flows through a student model with pseudo-quantization nodes, generating a simulated quantized, low-precision output image.

[0050] The second time: the input data flows through the same model, but bypasses the quantization effect of the pseudo-quantization node, generating a full-precision baseline output image.

[0051] Then, the accuracy adaptation loss is calculated: Using the low-precision output image generated during the first propagation, calculate the content loss and style loss between the original content image and the style image; the sum of these two losses constitutes the task loss for this iteration. Then, simultaneously input the low-precision output image generated during the first propagation and the full-precision baseline output image generated during the second propagation into a pre-trained deep neural network, and calculate the perceptual distance between them to obtain the value of the perceptual loss term.

[0052] The calculated task loss and perception loss term are weighted and summed according to preset weighting coefficients to obtain the total loss value of the final accuracy-fitting loss function. Gradient backpropagation is then performed based on this total loss value, and the full-precision parameters of the student model are updated using the optimizer. This process is repeated multiple times for fine-tuning until the model's performance and visual quality converge under simulated quantization.

[0053] Before performing quantization-aware training, a one-time, layer-by-layer rapid quantization test is performed on the pruned intermediate model to evaluate the sensitivity of different layers to quantization errors and generate a "quantization sensitivity map". This map can serve as prior knowledge to guide the search space of the automated policy agent. For example, the agent can be guided to prioritize trying higher quantization bit counts for layers with high sensitivity, thereby performing more aggressive compression at other layers without sacrificing the accuracy of key layers, effectively improving the efficiency of automated search and the performance of the final model.

[0054] By employing a composite function that incorporates both task loss and perceptual loss, dual optimization is achieved in quantization-perceptual training. On the one hand, the task loss ensures that the model's core image style transfer function does not degrade under conditions simulating low bit precision; on the other hand, the perceptual loss, by directly minimizing the visual difference between the output images before and after quantization, can accurately suppress visual artifacts such as blurring and color distortion introduced by the quantization process itself.

[0055] Furthermore, the multi-scene image verification set is a collection of image pairs generated by combining a content image source dataset and a style image source dataset; wherein, the content image source dataset contains real-world images with diverse scene and object categories; and the style image source dataset contains images of artworks covering different art movements, periods and authors.

[0056] By constructing a validation set spanning diverse content and styles, we ensure that the evaluation of the model's multi-scenario capabilities is comprehensive, fair, and unbiased. The resulting perceptual quality score accurately reflects the model's generalization ability, thus preventing the optimization process from falling into overfitting to specific scenarios. Since the ultimate goal of the automated process is to achieve high scores on this diverse validation set, it drives the agent to find optimization strategies that produce more robust models.

[0057] Furthermore, the student model is built on a lightweight network architecture, the convolution operations of which consist of depthwise separable convolutions, and the lightweight network architecture uses an inverted residual structure containing a linear bottleneck as the basic building block of its network.

[0058] like Figure 3 As shown, the specific implementation steps for constructing the student model using the aforementioned lightweight network architecture are as follows: Select a model pre-trained on the aforementioned lightweight network architecture as the base. Remove the top classification layer of this pre-trained model, retaining its network structure from the input layer to the intermediate feature extraction part, and use this as the encoder of the student model. Use the pre-trained weights as initial values.

[0059] Design a decoder that is structurally roughly symmetrical to the encoder described above. The main function of this decoder is to receive stylized feature maps and upsample them to reconstruct the final output image. The decoder is constructed using multiple upsampling modules, each typically consisting of an upsampling layer and several convolutional layers. To maintain the lightweight nature of the entire student model, the convolutional layers in the decoder also employ efficient operations such as depthwise separable convolutions.

[0060] The encoder and decoder are connected. In the actual style transfer process, the content features and style features extracted by the encoder are fused through an adaptive instance normalization layer, and the resulting feature map is used as the input to the decoder. The entire network, consisting of a lightweight encoder, an adaptive instance normalization layer, and a lightweight decoder, constitutes the student model to be jointly trained, pruned, and quantized.

[0061] By employing techniques such as depthwise separable convolution to construct a lightweight student model, the number of parameters and computational cost are reduced by several orders of magnitude, significantly decreasing model size and memory usage, and greatly improving inference speed. This lays the foundation for real-time and smooth style transfer applications on resource-constrained devices such as mobile phones. Furthermore, using a pre-trained lightweight model as a starting point effectively leverages the advantages of transfer learning, ensuring that high image generation quality is maintained while significantly improving operational efficiency.

[0062] This invention introduces an automated strategy search agent, integrating multiple previously independent optimization steps—such as knowledge distillation, pruning, and quantization—into an end-to-end automated closed-loop process, significantly improving optimization efficiency and reproducibility. This process, through joint training and perceptual quantization, avoids the cumulative accuracy loss problem of traditional step-by-step optimization, finding a better global balance between model performance and artistic quality. Furthermore, this invention guides the entire automation process with a composite reward function that integrates perceptual quality across multiple scenarios and the actual measured performance of the target hardware. This enables the automatic and customized generation of the best-performing final model for any edge device and application requirement, effectively solving the deployment challenges of hardware fragmentation and diverse application scenarios.

[0063] Example 2 This embodiment aims to provide a multi-scene image style transfer and edge computing optimization method to develop a set of high-performance, real-time "AI artist" video filters for a mainstream mobile short video social application.

[0064] The short video app plans to launch a new feature that allows users to transform their footage into various artistic styles in real time while recording videos. The core technical challenges of this feature lie in the following aspects: (1) Video processing needs to achieve a smoothness of at least 25 frames per second, which means that the style transfer processing time of a single frame image must be strictly controlled within 40 milliseconds.

[0065] (2) The application needs to be compatible with various smartphones on the market, from mainstream mid-range to high-end, and these devices have huge differences in computing power and chip architecture.

[0066] (3) Real-time processing should not cause the phone to overheat or lose power rapidly, so as not to affect the user experience.

[0067] (4) The content recorded by users is ever-changing, and the filter effect must remain stable and of high quality in various scenarios.

[0068] To address the aforementioned challenges, a multi-scene image style transfer and edge computing optimization method is adopted.

[0069] First, a mid-range smartphone with a high market share was selected as the primary hardware target for optimization. Addressing the core requirements of real-time video, the team set preset weighting coefficients in the composite reward function, assigning higher weights to performance metrics, such as: inference latency weight = 0.6, power consumption weight = 0.2, and perceived quality weight = 0.2. Speed ​​and energy efficiency were the primary objectives of this optimization.

[0070] A lightweight MobileNetV2-based network is used as the student model, and VGG-19 is used as the teacher model. An automated policy search agent based on the proximal policy optimization algorithm is launched, specifying the target hardware and reward function weights. The agent begins to interact with the simulation environment. In each interaction, it outputs a complete optimization policy, such as setting pruning rates of 40%, 60%, and 70% for the 3rd, 5th, and 7th modules of MobileNetV2, respectively, and selecting 8-bit quantization. The simulation environment executes all steps of joint training, channel pruning, and quantization-aware training according to this policy, generating a final compressed quantized model.

[0071] The generated compressed quantization model is automatically deployed to the target hardware, where its inference latency and power consumption are physically measured.

[0072] Meanwhile, the model was run on a multi-scenario validation set built using MS-COCO and WikiArt to calculate its perceptual quality score. These measured performance and quality scores were combined into a final reward value, which was returned to the agent. The agent updated the decision network based on the reward value and then began the next interaction, trying new optimization strategies. After several hours of automated search, the agent converged and output a globally optimal optimization strategy.

[0073] To adapt to other hardware platforms, the team only needs to change the target hardware in the evaluation environment, fine-tune the reward weights as needed, and then rerun the automated search to obtain the optimal model tailored to the new platform in a short time, without a lot of manual repetitive adaptation work.

[0074] This embodiment fully demonstrates the practical application value and significant technical advantages of the present invention. It shows how to transform a complex business application requirement into a clear, actionable, automated optimization goal. By setting a customized composite reward function for specific hardware and application scenarios, this method can automatically generate the model with the best overall performance in that scenario. This not only ensures that the final function meets the stringent standards of smooth and low-power deployment on mainstream mid-range devices, but also provides an efficient and scalable solution to the problem of mobile hardware fragmentation, greatly reducing the development costs and time required for multi-platform adaptation.

[0075] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A method for optimizing style transfer and edge computing in multi-scene images, characterized in that, include: The optimal optimization strategy is determined by an automated strategy search agent, and the optimal optimization strategy guides the following optimization process: A joint training approach is adopted, which uses a unified loss function to simultaneously perform knowledge distillation and structured pruning on the student model in a single training phase to generate an intermediate model; the unified loss function includes task loss, feature distillation loss and sparsity regularization loss. The intermediate model is trained using a precision adaptation loss function for quantization perception. This precision adaptation loss function includes a perception loss term that measures the visual perception difference between the outputs of the full-precision model and the quantized model. This perception loss term is calculated using a deep neural network pre-trained on a large image dataset. This deep neural network is used to extract image depth perception features. The calculation steps include: The output images of the intermediate model and the student model in the quantization perception training process are respectively passed as inputs to the deep neural network; the activation feature maps of the two output images on multiple pre-selected intermediate layers of the deep neural network are extracted and recorded; the norm distance of the tensor between the two sets of corresponding activation feature maps is calculated, and the norm distance is used as the value of the perception loss term to guide the fine-tuning of the parameters of the student model in the quantization perception training so as to minimize the distance between the two sets of activation feature maps; The agent's policy search process determines the optimal optimization policy by maximizing a composite reward function. The composite reward function is generated based on the performance indicators of the compressed quantization model measured on the target edge hardware and the perceptual quality score measured on a preset multi-scene image verification set. The compressed quantization model is the final model obtained by executing the optimization process. The automated policy search agent is a reinforcement learning agent that learns the optimal optimization policy by interacting with the simulation environment. In each interaction, the reinforcement learning agent generates a complete optimization policy by outputting a set of actions. The optimization policy includes setting a structured pruning rate for each layer and module of the student model and selecting the number of quantization bits for quantized perception training. The simulation environment executes the complete optimization process according to the optimization policy and generates the final compressed quantized model. The calculation steps of the composite reward function are as follows: the perception quality score of the compressed quantized model measured on a multi-scene image validation set is normalized with the inference latency, memory usage, and power consumption scores physically measured on the target edge hardware, and then weighted and summed according to preset weight coefficients to obtain the final reward value, which is returned to the reinforcement learning agent for updating the decision policy.

2. The multi-scene image style transfer and edge calculation optimization method according to claim 1, characterized in that: In the joint training method, the sparse regularization loss of the unified loss function is calculated for the learnable affine parameters in each batch normalized layer after the convolutional layer in the network topology of the student model. The sparse regularization loss is a regularization term that penalizes the L1 norm of the scaling factor of all specified layers. The regularization term, together with the task loss and feature distillation loss included in the unified loss function, participates in the iterative optimization process of gradient backpropagation and weight update of the student model. After the single training phase is completed, a channel pruning operation is performed. The channel pruning operation identifies channels with scaling factor values ​​lower than a preset threshold as channels to be pruned, removes the channels to be pruned and their associated convolutional kernels and batch normalized layer parameters from the model computation graph, and changes the network topology of the student model.

3. The multi-scene image style transfer and edge calculation optimization method according to claim 1, characterized in that: The task loss in the unified loss function includes content loss and style loss. The content loss measures the feature space difference between the content of the image generated by the student model and the content of the original content image. The style loss measures the feature statistical difference between the style of the image generated by the student model and the style of the target style image. The feature distillation loss is obtained by calculating the distance between the feature maps output by the student model and the pre-trained teacher model at the n intermediate layers of their respective networks.

4. The multi-scene image style transfer and edge calculation optimization method according to claim 1, characterized in that: The accuracy adaptation loss function is a composite function, which includes a task loss to measure the difference between content and style and a perceptual loss term to measure the difference between visual perception.

5. The multi-scene image style transfer and edge calculation optimization method according to claim 1, characterized in that: The multi-scene image validation set is a collection of image pairs generated by combining a content image source dataset and a style image source dataset; wherein, the content image source dataset contains real-world images with diverse scene and object categories; and the style image source dataset contains images of artworks covering different art movements, periods and authors.

6. The multi-scene image style transfer and edge calculation optimization method according to claim 1, characterized in that: The student model is built on a lightweight network architecture, in which the convolutional operations are composed of depthwise separable convolutions, and the lightweight network architecture uses an inverted residual structure containing a linear bottleneck as the basic building block of its network.

Images