Image style transfer method and system based on webgpu technology

Abstract

The application provides an image style migration method and system based on WebGPU technology, which comprises the following steps: S1, constructing an image style migration model; S2, training the image style migration model; S3, exporting the trained and optimized image style migration model into an ONNX format, and then converting the image style migration model in the ONNX format into an intermediate format suitable for a Web-side inference engine, while generating necessary metadata and model atlas; S4, integrating a large language model, and automatically generating or optimizing style migration parameters and prompt words according to a style description or specified requirements input by a user; S5, integrating and deploying a complete image style migration model, and integrating and aligning the complete image style migration model with a prompt word optimization model to obtain a final image style migration model and convert the final image style migration model into a browser-side model suitable for WebGPU technology. The application realizes real-time migration of multi-style high-resolution images, and greatly improves the style migration speed and visual expressiveness.

Description

Technical Field

[0001] This invention belongs to the field of computer vision and artificial intelligence applications, and specifically relates to an image style transfer method and system based on WebGPU technology. Background Technology

[0002] With the continuous advancement of artificial intelligence and computer vision technologies, image style transfer, as a typical application of visual intelligence, has been widely used in fields such as artistic creation, image enhancement, and personalized recommendations. Traditional image style transfer models mostly rely on local high-performance computing platforms or servers for inference operations, typically using low-level interfaces such as CUDA to call graphics card resources to ensure efficient model execution. However, these methods require the construction of complex hardware and software environments, resulting in high deployment and usage barriers, making it difficult to meet users' needs for fast and convenient style transfer on the browser side.

[0003] While the previous generation of web-based graphics interfaces, WebGL, offered some GPU computing power, it still had many limitations in terms of parallel performance, resource allocation, and applicability to deep learning, making it insufficient to support more complex neural networks and diverse interactive scenarios. To address this, international organizations such as the W3C, in conjunction with major technology companies, developed the next-generation WebGPU standard. WebGPU boasts stronger low-level GPU resource scheduling capabilities, natively supports compute shaders and efficient asynchronous parallel operations, and significantly improves security and versatility. Currently, frameworks such as TensorFlow.js, ONNX.js, and WebNN support neural network inference in browsers. Based on WebGPU, web applications can achieve near-native GPU-accelerated experiences, greatly enhancing the feasibility and efficiency of deploying deep neural networks on the browser side. Major browsers are now gradually supporting the WebGPU API and related shader languages. There are cutting-edge resources available for reference, such as an introduction to WebGPU technology (https: / / dl.acm.org / doi / pdf / 10.1145 / 3532720.3535625) or a comparison of WebGL and WebGPU technologies (https: / / www.sciencedirect.com / science / article / pii / S1877050924006410). Meanwhile, large-scale language models have demonstrated powerful capabilities in natural language understanding and generation. Through semantic analysis and automatic optimization, they can intelligently generate style transfer parameters or prompts tailored to image content or user needs, driving image style transfer models towards interactivity and intelligence. However, there is currently a lack of a comprehensive solution that combines efficient WebGPU inference with intelligent optimization of large language models to enable users to conveniently perform high-performance image style transfer and personalized prompt generation on web pages. Summary of the Invention

[0004] This invention provides an image style transfer method and system based on WebGPU technology to solve the problems of low computational efficiency, slow response speed, insufficient interactive intelligence, and poor platform compatibility in the deployment of existing image style transfer models on the Web.

[0005] The technical solution of the present invention is as follows: According to one aspect of the present invention, an image style transfer method based on WebGPU technology is provided, comprising the following steps: S1. Constructing an image style transfer model, the image style transfer model including a style feature extraction module, a content feature extraction module, and a style transfer module; S2. Training the image style transfer model to reduce the image style transfer model size and inference latency; S3. Exporting the trained and optimized image style transfer model to ONNX format, and then converting the ONNX format image style transfer model into an intermediate format adapted to the Web-based inference engine, while generating necessary metadata and model graphs; S4. Integrating a large language model, performing semantic understanding based on user-input style descriptions or specified requirements, and automatically generating or optimizing style transfer parameters and prompt words; S5. Integrating and deploying the complete image style transfer model, and integrating and aligning it with the prompt word optimization model to obtain the final image style transfer model; and converting the final image style transfer model into a browser-based model suitable for WebGPU technology for browser deployment.

[0006] Optionally, in the above-mentioned image style transfer method based on WebGPU technology, in step S1, an image style transfer model is designed by combining a convolutional neural network (CNN) with a Transformer structure.

[0007] Optionally, in the above-mentioned image style transfer method based on WebGPU technology, the style feature extraction module uses a pre-trained network to extract intermediate layer features of the reference style image, and uses Transformer to further extract the global layout and long-distance dependency features of the style image; the content feature extraction module uses a hybrid structure of convolutional neural network and Transformer to capture multi-scale semantic relationships and extract the structure and spatial layout of the content image; the style transfer module uses a self-attention module to fuse content and style features, calculates style loss, content loss and perceptual loss, and after decoding, transfers the fused features into an output image of the target style.

[0008] Optionally, in the above-mentioned image style transfer method based on WebGPU technology, in step S2, for scenarios where browser-side computing and memory resources are limited, the image style transfer model is trained offline and optimized by combining model pruning, parameter quantization and operator fusion.

[0009] Optionally, in the above-mentioned image style transfer method based on WebGPU technology, in step S3, the trained and optimized image style transfer model is exported to ONNX format using the model export tool of the deep learning framework, and then the ONNX format image style transfer model is converted into an intermediate format adapted to the Web-based inference engine using the Hugging Face tool.

[0010] Optionally, in the above-mentioned image style transfer method based on WebGPU technology, in step S5, the three modules of style feature extraction module, content feature extraction module and style transfer module are integrated into a style transfer model.

[0011] According to another aspect of the present invention, an image style transfer system based on WebGPU technology is provided, including an image style transfer model and a prompt word optimization model, wherein the image style transfer model and the prompt word optimization model are integrated and aligned, and the integrated image style transfer model and prompt word optimization model have been converted into a browser-side model suitable for WebGPU technology.

[0012] Optionally, in the aforementioned WebGPU-based image style transfer system, the style feature extraction module is used to extract intermediate layer features of the reference style image using a pre-trained network, and further extract the global layout and long-distance dependency features of the style image using Transformer; the content feature extraction module is used to capture multi-scale semantic relationships and extract the structure and spatial layout of the content image using a hybrid structure of convolutional neural network and Transformer; the style transfer module is used to combine content and style features using a self-attention module, calculate style loss, content loss and perceptual loss, and after decoding, transfer the fused features to the output image of the target style; the prompt word optimization model is an intelligent human-computer interaction model trained by modifying and redesigning the open-source large language model and learning from the prompt word knowledge base to optimize the prompt words and parameters of the raw image.

[0013] The beneficial effects of the technical solution of the present invention are as follows: This invention proposes an image style transfer model and system based on WebGPU technology, overcoming the technical bottlenecks of traditional web-based style transfer methods in terms of computational efficiency, real-time response, and compatibility. Compared with existing solutions that rely on CPU or WebGL for inefficient inference, this invention fully leverages the high-performance parallel computing capabilities of browser-side GPUs. Combined with a neural network structure optimized for WebGPU and efficient shaders, it achieves real-time transfer of high-resolution images with multiple styles, significantly improving the speed and visual expressiveness of style transfer. Furthermore, by incorporating intelligent parsing and optimization of user style descriptions using a large language model, the system's interactive intelligence and user experience are significantly enhanced, enabling more personalized and diverse style transfer needs.

[0014] This invention lowers the technical threshold and cost for users, improves the scalability and security of the system, and is suitable for scenarios with high requirements for real-time performance, convenience and data security, such as online art creation, image enhancement, educational presentation, and media editing. It provides solid technical support and an excellent user experience for intelligent visual content production and innovative applications on the Web.

[0015] To better understand and illustrate the concept, working principle, and effects of this invention, the invention will be described in detail below with reference to the accompanying drawings and specific embodiments: Attached Figure Description

[0016] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the accompanying drawings used in the description of the specific embodiments or the prior art will be briefly introduced below.

[0017] Figure 1 This is a flowchart illustrating the steps of the image style transfer method based on WebGPU technology of the present invention; Figure 2 This is a flowchart of the image style transfer method based on WebGPU technology of the present invention. Detailed Implementation

[0018] To make the objectives, technical methods, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and specific examples. These examples are merely illustrative and not intended to limit the scope of the invention.

[0019] This invention fully leverages the GPU acceleration capabilities of the next-generation WebGPU browser, designing a highly efficient style transfer neural network structure adapted to the WebGPU architecture. This enables high-quality, low-latency processing of complex image style transfer tasks across multiple browser platforms. By deeply integrating the advantages of convolutional neural networks and Transformers, and combining local feature extraction with global self-attention mechanisms, the style transfer effect and image detail representation are significantly improved. Furthermore, this invention relies on Large Language Models (LLMs) to achieve semantic recognition of style parameters and intelligent generation of prompts through natural language processing techniques. This optimizes the style descriptions and adjustment suggestions input by the user, dynamically enhancing the intelligent interaction capabilities of the style transfer model. The model system supports automatic hardware environment detection and adaptive parameter configuration, is compatible with various mainstream browsers and graphics card architectures, and possesses advantages such as high scalability and easy deployment. This invention can be widely applied in Web visual intelligence fields such as online image enhancement, artistic creation, and personalized image processing, promoting the efficient application and popularization of advanced image style transfer technology on web pages, and greatly enriching the expressiveness of online visual content and user experience.

[0020] The principles of this invention are as follows: 1) The image style transfer model proposed in this invention utilizes Transformer and convolutional neural networks to achieve style feature extraction, content feature extraction, and style transfer mechanism; 2) By modifying the open-source large language model, a prompt word knowledge base is designed and learned, enabling the large language model to identify and optimize user prompt words through natural language processing technology; 3) The image style transfer model and the prompt word optimization large language model are integrated to improve the user experience and the quality of raw images; 4) For the heterogeneous GPU environment on the Web, a model structure and inference process adapted to the WebGPU interface standard are designed to enable efficient allocation of GPU resources on the browser side.

[0021] like Figure 1 and Figure 2 As shown, the image style transfer method based on WebGPU technology of the present invention fully utilizes the computing power of GPU resources on the browser side to achieve high-quality, high-resolution style transfer of images with different styles. The specific steps of the method are as follows: S1. Construct an image style transfer model, which includes a style feature extraction module, a content feature extraction module, and a style transfer module.

[0022] Specifically, an image style transfer model is designed using a combination of convolutional neural networks (CNN) and Transformer structures, where: The style feature extraction module is used to extract style features. Specifically, a pre-trained network is used to extract intermediate layer features from the reference style image, such as the Gram matrix representing style, and the mean and variance representing hue, texture, etc.; the Transformer is then used to further extract the global layout and long-range dependency features of the style image.

[0023] The content feature extraction module is used to extract content features. Specifically, it utilizes a hybrid structure of convolutional neural networks and Transformers to capture multi-scale semantic relationships, extract information such as the structure and spatial layout of the content image, and correspond to the style branches.

[0024] The style transfer module is used to perform style transfer. Specifically, it uses a self-attention module to fuse content and style features, calculates style loss, content loss, and perceptual loss, and decodes them to transfer the fused features into an output image with the target style.

[0025] S2. Train the image style transfer model to reduce its size and inference latency; For scenarios where browser-based computing and memory resources are limited, the constructed image style transfer model is trained offline and optimized using techniques such as model pruning, parameter quantization, and operator fusion, which significantly reduces the size of the image style transfer model and inference latency.

[0026] S3. Using the model export tool of the deep learning framework, export the trained and optimized image style transfer model to ONNX format, and then use the Hugging Face tool to convert the ONNX format image style transfer model into an intermediate format adapted to the web-based inference engine. At the same time, generate the necessary metadata and model graphs to facilitate efficient loading and parsing on the web.

[0027] S4. Integrating a large language model, based on the style description or specified requirements input by the user through text, it performs semantic understanding and automatically generates or optimizes style transfer parameters and prompt words, improving the interactive intelligence and artistic expression between the final image style transfer model and the user, and achieving a more personalized style transfer experience.

[0028] S5. Integrate and deploy the complete style transfer model, combining the style feature extraction module, content feature extraction module, and style transfer module into a single style transfer model, and aligning it with the prompt word optimization model to obtain the final image style transfer model; then, transform the final image style transfer model into a browser-side model suitable for WebGPU technology and deploy it on the browser side.

[0029] The image style transfer system based on WebGPU technology of the present invention includes an image style transfer model and a prompt word optimization model; wherein the image style transfer model and the prompt word optimization model are integrated and aligned, and the integrated image style transfer model and prompt word optimization model have been converted into a browser-side model suitable for WebGPU technology.

[0030] The image style transfer model includes a style feature extraction module, a content feature extraction module, and a style transfer module, wherein: The style feature extraction module uses a pre-trained network to extract intermediate layer features from reference style images, such as a Gram matrix to represent style, and mean and variance to represent hue, texture, etc. A Transformer is then used to further extract global layout and long-range dependency features from the style images.

[0031] The content feature extraction module is used to capture multi-scale semantic relationships by utilizing a hybrid structure of convolutional neural networks and Transformers, and to extract information such as the structure and spatial layout of the content image, which corresponds to the style branch.

[0032] The style transfer module is used to combine content and style features using the self-attention module, calculate style loss, content loss and perceptual loss, and decode them to transfer the fused features into an output image of the target style.

[0033] The prompt word optimization model is an intelligent human-computer interaction model that optimizes prompt words and parameters for raw images by modifying and redesigning an open-source large language model and learning from a prompt word knowledge base.

[0034] This invention presents an image style transfer model system based on WebGPU technology, which can efficiently utilize the parallel computing power of GPUs on the browser side to achieve high-resolution, multi-style art transfer. The neural network structure optimized for the WebGPU architecture significantly improves the model's inference efficiency and cross-platform adaptability on the web. The model structure integrates the advantages of Transformer and Convolutional Neural Networks, exhibiting higher expressiveness in multi-scale modeling and fusion of style and content features, and achieving accurate capture of image details and global style features.

[0035] Furthermore, this invention integrates a large language model for natural language parsing and a prompt word optimization model, enabling users to describe their style intent through text and receive intelligent style transfer settings recommendations, significantly improving the system's human-computer interaction intelligence and personalized experience. Based on the above innovations, this invention significantly expands the capabilities of image style transfer in web-based applications such as online art creation, image enhancement, and educational promotion, providing solid technical support for the popularization of intelligent visual processing and the improvement of user experience.

[0036] This invention utilizes the emerging Web GPU standard to efficiently access underlying graphics card resources, enabling image style transfer and personalized aesthetic transfer within a browser environment. Simultaneously, it introduces a large-scale language model, employing natural language recognition and analysis to optimize style transfer parameters and prompts, thereby enhancing the model's adaptability and user experience across diverse contexts. Compared to traditional WebGL or CPU-based computation methods, this invention significantly improves the performance, interactivity, and intelligence of image processing on the web.

[0037] The embodiments of this invention, through model design, training and integration, model transformation and deployment, achieve multi-style, high-resolution image generation on the browser side. As can be seen from the comparison of image generation time in Table 1, this method not only flexibly and fully utilizes the computing power of GPUs on the browser side, but also significantly reduces image generation time compared to other style transfer models that use local computing power, significantly improving the real-time performance and quality of style transfer image generation. It also supports large language model interaction and intelligent image generation parameter prompt adjustment, ensuring a good human-computer interaction experience and excellent image generation results for users. This invention can efficiently process large-size images, ensure seamless output of results and user data security, and the overall process has good scalability and ease of use, flexibly adapting to various application scenarios.

[0038] Table 1: Comparison of image generation time between the method of this invention and other local computing power style transfer models The above description represents the preferred embodiment based on the inventive concept and working principle. The above embodiments should not be construed as limiting the scope of protection of these claims; other embodiments and combinations of implementations based on the inventive concept are all within the scope of protection of this invention.

Claims

1. An image style transfer method based on WebGPU technology, characterized in that, Includes the following steps: S1. Construct an image style transfer model, which includes a style feature extraction module, a content feature extraction module, and a style transfer module; S2. Train the image style transfer model to reduce its size and inference latency; S3. Export the trained and optimized image style transfer model to ONNX format, then convert the ONNX format image style transfer model into an intermediate format adapted to the web-based inference engine, while generating the necessary metadata and model graph. S4. Integrate a large language model to perform semantic understanding and automatically generate or optimize style transfer parameters and prompt words based on the style description or specified requirements input by the user; S5. Integrate and deploy the complete image style transfer model, and integrate and align it with the prompt word optimization model to obtain the final image style transfer model; then convert the final image style transfer model into a browser-side model suitable for WebGPU technology and deploy it on the browser side.

2. The image style transfer method based on WebGPU technology according to claim 1, characterized in that, In step S1, the image style transfer model is designed using a combination of convolutional neural networks (CNN) and Transformer structures.

3. The image style transfer method based on WebGPU technology according to claim 1, characterized in that, in, The style feature extraction module uses a pre-trained network to extract intermediate layer features from the reference style image, and uses Transformer to further extract the global layout and long-distance dependency features of the style image; the content feature extraction module uses a hybrid structure of convolutional neural network and Transformer to capture multi-scale semantic relationships and extract the structure and spatial layout of the content image. The style transfer module uses a self-attention module to fuse content and style features, calculates style loss, content loss and perceptual loss, and decodes them to transfer the fused features into an output image of the target style.

4. The image style transfer method based on WebGPU technology according to claim 1, characterized in that, In step S2, for scenarios where browser-side computing and memory resources are limited, the image style transfer model is trained offline and optimized by combining model pruning, parameter quantization, and operator fusion.

5. The image style transfer method based on WebGPU technology according to claim 1, characterized in that, In step S3, the trained and optimized image style transfer model is exported to ONNX format using the model export tool of the deep learning framework, and then the ONNX format image style transfer model is converted into an intermediate format adapted to the web-based inference engine using the Hugging Face tool.

6. The image style transfer method based on WebGPU technology according to claim 1, characterized in that, In step S5, the style feature extraction module, the content feature extraction module, and the style transfer module are integrated into a style transfer model.

7. An image style transfer system based on WebGPU technology, characterized in that, It includes an image style transfer model and a prompt word optimization model, wherein the image style transfer model and the prompt word optimization model are integrated and aligned, and the integrated image style transfer model and the prompt word optimization model have been transformed into a browser-side model suitable for WebGPU technology.

8. The image style transfer system based on WebGPU technology according to claim 1, characterized in that, The style feature extraction module is used to extract intermediate layer features of the reference style image using a pre-trained network, and further extract the global layout and long-distance dependency features of the style image using Transformer. The content feature extraction module is used to capture multi-scale semantic relationships and extract the structure and spatial layout of the content image by utilizing a hybrid structure of convolutional neural network and Transformer. The style transfer module is used to combine content and style features using a self-attention module, calculate style loss, content loss and perceptual loss, and after decoding, transfer the fused features into an output image of the target style. The aforementioned prompt word optimization model is an intelligent human-computer interaction model that optimizes prompt words and parameters for raw images by modifying and redesigning an open-source large language model and learning from a prompt word knowledge base.

Images