Visual model system, visual model construction method, visual task processing method, image generation method, video generation method, information processing method based on visual model, and cloud training platform

Abstract

Provided in the embodiments of the present disclosure are a visual model system, a visual model construction method, a visual task processing method, an image generation method, a video generation method, an information processing method based on a visual model, and a cloud training platform. The visual model system comprises a visual model, wherein the visual model comprises at least two feature processing modules and at least one tuning module, each feature processing module comprises a plurality of feature processing layers, the tuning module comprises a plurality of tuning layers, and the plurality of tuning layers are respectively connected to feature processing layers of corresponding levels in the at least two feature processing modules; and each tuning layer is configured to receive feature data outputted by the feature processing layers connected thereto, and to perform tuning processing on the feature data to obtain a tuning result, and the tuning result is used for generating visual data. Detailed information on a plurality of levels is reserved, more accurate visual data is generated, and a feature processing module is decoupled from a tuning module, thereby improving the universality and the generation efficiency of a visual model system, reducing training overheads, and improving the training efficiency.

Description

Visual model system, visual model construction method, visual task processing method, image generation method, video generation method, information processing method based on visual model and cloud training platform

[0001] This disclosure claims priority to Chinese patent application number 202410251754.8 filed with the Patent Office of China on March 5, 2024, and entitled “Visual model system, visual model construction method, visual task processing method, image generation method, video generation method, information processing method based on visual model and cloud training platform”, the entire contents of which are incorporated by reference into this disclosure. Technical Field

[0002] The embodiments of the present disclosure relate to the field of deep learning technology, and in particular to a visual model system, a visual model construction method, a visual task processing method, an image generation method, a video generation method, an information processing method based on a visual model, and a cloud training platform. Background Art

[0003] With the development of deep learning technology, deep learning model systems represented by visual model systems have been widely used in the field of vision, and have performed different downstream visual generation tasks, such as virtual video generation, image enhancement to improve image quality, and image generation with style transfer.

[0004] At present, for different downstream visual generation tasks, different task samples need to be used accordingly, and fine-tuning training should be performed on the general visual model system with an increasing number of feature processing layers and parameters in the feature processing module in order to achieve more accurate downstream visual generation.

[0005] However, as the number of layers and parameters of the feature processing module continues to increase, there is a problem of loss of detailed information at multiple levels in the feature processing process, and the generated visual data is insufficient. Moreover, once different downstream visual generation tasks need to be performed, the feature processing module needs to be retrained. The visual model system is not versatile enough, and the efficiency of visual data generation is insufficient. In addition, the training overhead also continues to increase with the increasing number of layers and parameters, reducing the training efficiency. Summary of the Invention

[0006] In view of this, embodiments of the present disclosure provide a visual model system. One or more embodiments of the present disclosure also involve a visual model construction method, a visual task processing method, an image generation method, a video generation method, an information processing method based on a visual model, a network cloud platform, a visual model construction device, a visual task processing device, an image generation device, a video generation device, an information processing device based on a visual model, a computing device, a computer-readable storage medium, and a computer program product to address technical deficiencies in the prior art.

[0007] According to a first aspect of an embodiment of the present disclosure, a visual model system is provided, including a visual model, the visual model including at least two feature processing modules and at least one tuning module, the feature processing module including multiple feature processing layers, the tuning module including multiple tuning layers, the multiple tuning layers respectively connected to the feature processing layers of corresponding levels in the at least two feature processing modules;

[0008] The tuning layer is used to receive the feature data output by the connected feature processing layer, perform tuning processing on the feature data, and obtain a tuning result, wherein the tuning result is used to generate visual data.

[0009] According to a second aspect of an embodiment of the present disclosure, a method for constructing a visual model is provided, comprising:

[0010] Get target model parameters;

[0011] Based on the target model parameters, at least two feature processing modules and at least one tuning module are constructed, wherein the feature processing module includes multiple feature processing layers, and the tuning module includes multiple tuning layers. The tuning layers are used to perform tuning processing on the input feature data to obtain tuning results, and the tuning results are used to generate visual data;

[0012] The multiple tuning layers are respectively connected to the feature processing layers of corresponding levels in at least two feature processing modules to obtain a visual model.

[0013] According to a third aspect of an embodiment of the present disclosure, a method for processing a visual task is provided, including:

[0014] Obtain task data for the target visual generation task;

[0015] Inputting the task data into a target vision model corresponding to the target vision generation task, and obtaining feature data output by each feature processing layer in the target vision model, wherein the target vision model includes at least two feature processing modules and at least one tuning module, the feature processing module includes multiple feature processing layers, the tuning module includes multiple tuning layers, and the multiple tuning layers are respectively connected to the feature processing layers of corresponding levels in the at least two feature processing modules;

[0016] The feature data output by each feature processing layer are respectively input into the tuning layers connected to each feature processing layer for tuning processing, and the target visual data is generated based on the tuning results of each tuning layer.

[0017] According to a fourth aspect of the embodiments of the present disclosure, there is provided an image generation method, including:

[0018] Get the task data of the image generation task;

[0019] Inputting task data into an image generation model to obtain feature data output by each feature processing layer in the image generation model, wherein the image generation model includes at least two feature processing modules and at least one tuning module, the feature processing module includes multiple feature processing layers, the tuning module includes multiple tuning layers, and the multiple tuning layers are respectively connected to the feature processing layers of corresponding levels in the at least two feature processing modules;

[0020] The feature data output by each feature processing layer are respectively input into the tuning layers connected to each feature processing layer for tuning processing, and the target image is generated based on the tuning results of each tuning layer.

[0021] According to a fifth aspect of the embodiments of the present disclosure, a video generation method is provided, including:

[0022] Get the task data of the video generation task;

[0023] Inputting task data into a video generation model to obtain feature data output by each feature processing layer in the video generation model, wherein the target video model includes at least two feature processing modules and at least one tuning module, the feature processing module includes multiple feature processing layers, the tuning module includes multiple tuning layers, and the multiple tuning layers are respectively connected to the feature processing layers of corresponding levels in the at least two feature processing modules;

[0024] The feature data output by each feature processing layer are respectively input into the tuning layers connected to each feature processing layer for tuning processing, and the target video is generated based on the tuning results of each tuning layer.

[0025] According to a sixth aspect of an embodiment of the present disclosure, there is provided a visual model-based information processing method, which is applied to a cloud training platform, comprising:

[0026] Receiving a task generation request sent by a terminal device, wherein the task generation request includes request information;

[0027] Based on the request information, a target visual model is obtained, wherein the target visual model includes at least two feature processing modules and at least one tuning module, the feature processing module includes multiple feature processing layers, the tuning module includes multiple tuning layers, the multiple tuning layers are respectively connected to the feature processing layers of corresponding levels in the at least two feature processing modules, the tuning layers are used to perform tuning processing on feature data output by the connected feature processing layers to obtain tuning results, and the tuning results are used to generate visual data;

[0028] Based on the target visual model, task information is generated, wherein the task information is used by the terminal device to perform the target visual task.

[0029] According to a seventh aspect of an embodiment of the present disclosure, there is provided a cloud training platform, comprising a request interface and a response unit;

[0030] A request interface, configured to receive a task generation request sent by a terminal device, wherein the task generation request includes request information;

[0031] A response unit is used to obtain a target visual model based on the request information, wherein the target visual model includes at least two feature processing modules and at least one tuning module, the feature processing module includes multiple feature processing layers, the tuning module includes multiple tuning layers, and the multiple tuning layers are respectively connected to the feature processing layers of corresponding levels in the at least two feature processing modules, and the tuning layer is used to perform tuning processing on the feature data output by the connected feature processing layers to obtain a tuning result, and the tuning result is used to generate visual data; based on the target visual model, task information is generated, wherein the task information is used for the terminal device to perform the target visual task.

[0032] According to an eighth aspect of an embodiment of the present disclosure, there is provided a visual model construction apparatus, comprising:

[0033] A parameter acquisition module is configured to obtain target model parameters;

[0034] a module construction module configured to construct at least two feature processing modules and at least one tuning module based on target model parameters, wherein the feature processing module includes multiple feature processing layers, and the tuning module includes multiple tuning layers, wherein the tuning layers are used to perform tuning processing on input feature data to obtain tuning results, and the tuning results are used to generate visual data;

[0035] The connection module is configured to connect the multiple tuning layers with the feature processing layers of corresponding levels in at least two feature processing modules to obtain a visual model.

[0036] According to a ninth aspect of an embodiment of the present disclosure, there is provided a visual task processing apparatus, comprising:

[0037] A first acquisition module is configured to acquire task data of a target visual generation task;

[0038] A first processing module is configured to input task data into a target vision model corresponding to the target vision generation task, and obtain feature data output by each feature processing layer in the target vision model, wherein the target vision model includes at least two feature processing modules and at least one tuning module, the feature processing module includes multiple feature processing layers, the tuning module includes multiple tuning layers, and the multiple tuning layers are respectively connected to the feature processing layers of corresponding levels in the at least two feature processing modules;

[0039] The first tuning module is configured to input the feature data output by each feature processing layer into the tuning layer connected to each feature processing layer for tuning processing, and generate target visual data based on the tuning results of each tuning layer.

[0040] According to a tenth aspect of an embodiment of the present disclosure, there is provided an image generating apparatus, including:

[0041] A second acquisition module is configured to acquire task data of the image generation task;

[0042] a second processing module configured to input the task data into the image generation model and obtain feature data output by each feature processing layer in the image generation model, wherein the image generation model includes at least two feature processing modules and at least one tuning module, the feature processing module includes multiple feature processing layers, the tuning module includes multiple tuning layers, and the multiple tuning layers are respectively connected to the feature processing layers of corresponding levels in the at least two feature processing modules;

[0043] The second tuning module is configured to input the feature data output by each feature processing layer into the tuning layer connected to each feature processing layer for tuning processing, and generate a target image based on the tuning results of each tuning layer.

[0044] According to an eleventh aspect of the embodiments of the present disclosure, there is provided a video generating apparatus, including:

[0045] A third acquisition module is configured to acquire task data of the video generation task;

[0046] a third processing module configured to input the task data into the video generation model and obtain feature data output by each feature processing layer in the video generation model, wherein the target video model includes at least two feature processing modules and at least one tuning module, the feature processing module includes multiple feature processing layers, the tuning module includes multiple tuning layers, and the multiple tuning layers are respectively connected to the feature processing layers of corresponding levels in the at least two feature processing modules;

[0047] The third tuning module is configured to input the feature data output by each feature processing layer into the tuning layer connected to each feature processing layer for tuning processing, and generate a target video based on the tuning results of each tuning layer.

[0048] According to a twelfth aspect of the embodiments of the present disclosure, there is provided an information processing device based on a visual model, which is applied to a cloud training platform, comprising:

[0049] a request receiving module configured to receive a task generation request sent by a terminal device, wherein the task generation request includes request information;

[0050] a model acquisition module configured to acquire a target visual model based on the request information, wherein the target visual model includes at least two feature processing modules and at least one tuning module, the feature processing module includes multiple feature processing layers, the tuning module includes multiple tuning layers, the multiple tuning layers are respectively connected to the feature processing layers of corresponding levels in the at least two feature processing modules, the tuning layers are used to perform tuning processing on feature data output by the connected feature processing layers to obtain tuning results, and the tuning results are used to generate visual data;

[0051] The information generation module is configured to generate task information based on the target visual model, wherein the task information is used for the terminal device to perform the target visual task.

[0052] According to a thirteenth aspect of an embodiment of the present disclosure, there is provided a computing device, including:

[0053] memory and processor;

[0054] The memory is used to store computer programs / instructions, and the processor is used to execute the computer programs / instructions. When the computer programs / instructions are executed by the processor, the steps of the methods described in aspects 2 to 6 are implemented.

[0055] According to the fourteenth aspect of the embodiments of the present disclosure, a computer-readable storage medium is provided, which stores a computer program / instruction, and when the computer program / instruction is executed by a processor, the steps of the method described in the second to sixth aspects are implemented.

[0056] According to a fifteenth aspect of an embodiment of the present disclosure, a computer program product is provided, comprising a computer program / instruction, which, when executed by a processor, implements the steps of the method described in aspects 2 to 6.

[0057] In one embodiment of the present disclosure, a visual model system includes a visual model, the visual model includes at least two feature processing modules and at least one tuning module, the feature processing module includes multiple feature processing layers, the tuning module includes multiple tuning layers, and the multiple tuning layers are respectively connected to the feature processing layers of corresponding levels in the at least two feature processing modules; the tuning layer is used to receive feature data output by the connected feature processing layers, perform tuning processing on the feature data, and obtain a tuning result, wherein the tuning result is used to generate visual data. In the visual model system, multiple tuning layers of the tuning module are respectively connected to the feature processing layers of corresponding levels in at least two feature processing modules, so that complete feature data output by multiple feature processing layers are obtained, detailed information on multiple levels is retained, and tuning processing on corresponding levels is implemented in multiple tuning layers, thereby optimizing the output feature data and generating more accurate visual data. In addition, at least two feature processing modules and at least one tuning module in the visual model are decoupled, so that the tuning module can be flexibly selected during the visual data generation process, thereby improving the versatility of the visual model system. In the visual data generation process, the feature processing module can be shared to perform multi-task shared tuning, thereby improving the efficiency of visual data generation. In the training process of the visual model, only the tuning module can be trained for different downstream visual generation tasks without training the feature processing module, thereby reducing training overhead and improving training efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] FIG1 is a schematic diagram of the architecture of a visual model system;

[0059] FIG2 is a schematic diagram of the structure of a visual model system provided by an embodiment of the present disclosure;

[0060] FIG3 is a schematic diagram of a first architecture of a visual model in a visual model system provided by one embodiment of the present disclosure;

[0061] FIG4 is a schematic diagram of a second architecture of a visual model in a visual model system provided by an embodiment of the present disclosure;

[0062] FIG5 is a schematic diagram of a third architecture of a visual model in a visual model system provided by one embodiment of the present disclosure;

[0063] FIG6 is a schematic diagram of a fourth architecture of a visual model in a visual model system provided by one embodiment of the present disclosure;

[0064] FIG7 is a fifth schematic diagram of the architecture of a visual model in a visual model system provided by one embodiment of the present disclosure;

[0065] FIG8 is a sixth schematic diagram of the architecture of a visual model in a visual model system provided by one embodiment of the present disclosure;

[0066] FIG9 is a flowchart of a method for constructing a visual model provided by one embodiment of the present disclosure;

[0067] FIG10 is a flowchart of a visual task processing method provided by one embodiment of the present disclosure;

[0068] FIG11 is a flowchart of an image generation method provided by one embodiment of the present disclosure;

[0069] FIG12 is a flowchart of a video generation method provided by one embodiment of the present disclosure;

[0070] FIG13 is a flowchart of an information processing method based on a visual model provided by one embodiment of the present disclosure;

[0071] FIG14 is a schematic structural diagram of a cloud training platform provided by one embodiment of the present disclosure;

[0072] FIG15 is a schematic structural diagram of a visual model building device provided by one embodiment of the present disclosure;

[0073] FIG16 is a schematic diagram of the structure of a visual task processing device provided by one embodiment of the present disclosure;

[0074] FIG17 is a schematic structural diagram of an image generating device provided by one embodiment of the present disclosure;

[0075] FIG18 is a schematic structural diagram of a video generating device provided by one embodiment of the present disclosure;

[0076] FIG19 is a schematic structural diagram of an information processing device based on a visual model provided by one embodiment of the present disclosure;

[0077] FIG20 is a structural block diagram of a computing device provided by an embodiment of the present disclosure. DETAILED DESCRIPTION

[0078] The following description sets forth many specific details to facilitate a full understanding of the present disclosure. However, the present disclosure can be implemented in many other ways than those described herein, and those skilled in the art can make similar generalizations without violating the scope of the present disclosure. Therefore, the present disclosure is not limited to the specific implementations disclosed below.

[0079] The terms used in one or more embodiments of the present disclosure are for the purpose of describing specific embodiments only and are not intended to limit one or more embodiments of the present disclosure. The singular forms "a", "the", and "the" used in one or more embodiments of the present disclosure and the appended claims are also intended to include plural forms, unless the context clearly indicates otherwise. It should also be understood that the term "and / or" used in one or more embodiments of the present disclosure refers to and includes any or all possible combinations of one or more associated listed items.

[0080] It should be understood that although the terms first, second, etc. may be used to describe various information in one or more embodiments of the present disclosure, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from each other. For example, without departing from the scope of one or more embodiments of the present disclosure, the first may also be referred to as the second, and similarly, the second may also be referred to as the first. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0081] In addition, it should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in one or more embodiments of the present disclosure are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of the relevant regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0082] In one or more embodiments of the present disclosure, a large model refers to a deep learning model with large-scale model parameters, which usually contains hundreds of millions, tens of billions, hundreds of billions, trillions, or even more than ten trillion model parameters. A large model can also be called a cornerstone model / foundation model (Foundation Model), which is pre-trained by using large-scale unlabeled corpus to produce a pre-training task model with more than 100 million parameters. This model can adapt to a wide range of downstream tasks, and the model has good generalization ability, such as a large-scale language model (LLM), a multi-modal pre-training task model, etc.

[0083] When large models are used in practice, only a small number of samples are needed to fine-tune the pre-trained task model and it can be applied to different tasks. Large models can be widely used in natural language processing (NLP), computer vision and other fields. Specifically, they can be applied to computer vision tasks such as visual question answering (VQA), image caption (IC), and image generation, as well as natural language processing tasks such as text-based sentiment classification, text summary generation, and machine translation. The main application scenarios of large models include digital assistants, intelligent robots, search, online education, office software, e-commerce, and intelligent design.

[0084] First, the terms involved in one or more embodiments of the present disclosure are explained.

[0085] Pre-trained Model: A model produced after large-scale training and fine-tuning on a representative dataset. It is often used as an initialization model for training other downstream tasks to speed up training or achieve better results.

[0086] Foundation Model: In different fields, the model is trained with large amounts of data, powerful computing power, and carefully designed structures, and can adapt to a range of downstream tasks.

[0087] Deep Learning: A branch of machine learning, it is an algorithm that uses artificial neural networks as its architecture to represent and learn data.

[0088] Convolutional Neural Networks (CNN) model: A multi-layer deep learning model with forward propagation and backpropagation, and a convolution filter for processing feature data.

[0089] Recurrent Neural Network (RNN) model: A recursive deep learning model that processes vector representations recursively and connects intermediate layers in a chain-like manner.

[0090] Long Short Term Memory (LSTM) model: A deep learning model that has the ability to memorize long-term and short-term information and has a convolution filter for processing feature data.

[0091] Deep Self-Attention Model (Transformer Model): A deep learning architecture based on the attention mechanism for processing sequential data such as natural language.

[0092] Bidirectional Encoder Representations from Transformers (BERT): A special Transformer model trained using a bidirectional Transformer encoder and large-scale unlabeled text data.

[0093] U-Net model: A deep learning model architecture consisting of a downsampling encoder, an upsampling decoder, and skip connections.

[0094] Diffusion Model (DM): A generative deep learning model that gradually removes noise to generate a noise-free image by adding noise in a forward process and predicting noise in a backward process.

[0095] Stable Diffusion: A generative deep learning model consisting of an autoencoder, a text encoder, and a U-Net model structure.

[0096] Pre-training: refers to the pre-training phase conducted on large-scale pre-training data to generate processing results.

[0097] Fine-tuning: refers to the stage of adjusting and optimizing the pre-trained model on a specific task to improve the performance of the model on that task.

[0098] Reinforcement Learning (RL): A deep learning method that enables a model to learn how to make optimal decisions through continuous trial and error and feedback.

[0099] Transfer Learning: Improving learning of new tasks by transferring knowledge from previously learned related tasks.

[0100] Parameter-efficient transfer learning (PETL): A tuning training method that modifies a small number of parameters or adds a small number of additional parameters based on a pre-trained task model.

[0101] Memory-efficient Transfer Learning (METL): Based on the pre-trained task model, a training method is tuned using a relatively small amount of memory.

[0102] Skip Connection (SC): From the encoder input of the U-Net model to the decoder of the U-Net, it provides multi-level long-distance information.

[0103] Tensor Processing Unit (TPU): Optimized for handling large-scale tensor operations, particularly in deep learning, such as neural network training and inference. Compared to general-purpose processors like CPUs (Central Processing Units) or GPUs (Graphics Processing Units), TPUs offer higher performance and energy efficiency through a customized hardware architecture and instruction set. They excel at performing calculations such as matrix multiplication and convolution, which are common in machine learning models.

[0104] Neural Processing Unit (NPU): An application-specific integrated circuit (ASIC) designed specifically for neural network algorithm calculations, aimed at improving the efficiency and speed of running artificial intelligence applications. The NPU is particularly suitable for executing complex neural network models, including but not limited to tasks such as image recognition, speech recognition, and natural language processing. Similar to the TPU, the NPU also focuses on providing highly parallel processing capabilities to achieve fast neural network calculations and generally performs well in low-power environments. Therefore, it is often used in scenarios such as mobile devices, embedded systems, and data centers.

[0105] Application-Specific Integrated Circuit (ASIC): A custom chip designed and manufactured for a specific application. In an ASIC, the circuit design is fixed and cannot be changed; once manufactured, its functionality is permanently defined.

[0106] Field-Programmable Gate Array (FPGA): An integrated circuit with exceptional flexibility that allows users to reconfigure the internal logic structure to achieve different circuit functions after the device leaves the factory. An FPGA consists of a large number of programmable logic blocks, input and output units, embedded memory blocks, and other resources. By writing a design in a hardware description language (such as Verilog or VHDL) and compiling and configuring it using EDA tools, users can reprogram the FPGA multiple times in a short period of time to adapt to different or changing application requirements.

[0107] Currently, the encoder-decoder architecture has achieved widespread success in various task areas. For example, the diffusion model paradigm has achieved success in the field of visual generation due to its high-quality visual generation capabilities and flexible fine-tuning capabilities. On the one hand, this is based on the large-scale pre-trained basic model that provides a general basic capability. On the other hand, this pre-trained task model can be used to carry out further training on a series of downstream target tasks. Figure 1 shows a schematic diagram of the architecture of a visual model system, as shown in Figure 1:

[0108] The light pink color represents the encoding layer, the light yellow color represents the decoding layer, and the light blue color represents the intermediate layer. The visual model system consists of three feature processing modules: the encoding module, the decoding module, and the intermediate layer. The encoding module consists of multiple encoding layers (encoding layer_1, encoding layer_2, encoding layer_3, and encoding layer_4), and the decoding module consists of multiple decoding layers (decoding layer_1, decoding layer_2, decoding layer_3, and decoding layer_4). The encoding layers in the encoding module and the decoding layers in the decoding module exhibit a symmetrical structure. The encoding features output by each encoding layer (encoding feature_1, encoding feature_2, encoding feature_3, and encoding feature_4) are connected to the corresponding decoding layer via skip connections for layer-by-layer decoding. The final decoding layer (decoding layer_1) outputs the decoding features, based on which visual data is generated.

[0109] However, the architecture of the visual model system in Figure 1, including multiple feature processing layers in the feature processing module, requires the use of different task samples for different downstream visual generation tasks, and fine-tuning training is performed on the general visual model with an increasing number of feature processing layers and parameters in order to achieve more accurate downstream visual generation.

[0110] However, as the number of layers and parameters of the feature processing modules in the model system continues to increase, there is a problem of loss of detailed information at multiple levels in the feature processing process, and the generated visual data is insufficient. Moreover, once different downstream visual generation tasks need to be performed, the model needs to be retrained. The visual model system lacks versatility, and the efficiency of visual data generation is insufficient. In addition, the training overhead continues to increase with the increasing number of layers and parameters, reducing training efficiency.

[0111] In response to the above problems, the present disclosure provides a visual model system. The present disclosure also involves a visual model construction method, a visual task processing method, an image generation method, a video generation method, an information processing method based on a visual model, a network cloud platform, a visual model construction device, a visual task processing device, an image generation device, a video generation device, an information processing device based on a visual model, a computing device, a computer-readable storage medium and a computer program product, which are described in detail one by one in the following embodiments.

[0112] Referring to FIG2 , FIG2 shows a schematic diagram of the structure of a visual model system provided by an embodiment of the present disclosure, as shown in FIG2 :

[0113] The visual model system 200 includes a visual model 210, wherein the visual model 210 includes at least two feature processing modules 2110 and at least one tuning module 2112. The feature processing module 2110 includes multiple feature processing layers 21110, and the tuning module 2112 includes multiple tuning layers 21120. The multiple tuning layers 21120 are respectively connected to the feature processing layers 21110 of corresponding levels in the at least two feature processing modules 2110.

[0114] The tuning layer 21120 is used to receive the feature data output by the connected feature processing layer 21110, perform tuning processing on the feature data, and obtain a tuning result, wherein the tuning result is used to generate visual data.

[0115] Visual model system 200 is an integrated software and hardware architecture for processing visual information, built on deep learning technology. Visual model system 200 deploys a visual model 210, which can perform visual generation tasks such as image analysis and video understanding. This system performs deep learning-based visual information processing using hardware devices with corresponding functions, such as embedded device systems or distributed node systems.

[0116] The visual model 210 is a deep learning model with visual information processing capabilities, which is used to implement specific visual generation tasks. The visual model 210 includes at least two feature processing modules 2110 and at least one tuning module 2112. The multiple tuning layers 21120 in the tuning module 2112 are respectively connected to the feature processing layers 21110 of the corresponding levels in at least two feature processing modules 2110. According to the data processing modality, the visual model includes but is not limited to: image processing model and video processing model. According to the task division, the visual model includes but is not limited to: generation model, classification model (discriminant model), strategy selection model and regression prediction model. According to the model type division, the visual model includes but is not limited to: CNN model, RNN model, LSTM model, Transformer model, BERT model, U-Net model, diffusion model, stable diffusion model, GAN (Generative Adversarial Network, Generative Adversarial Network) model and large model.

[0117] The feature processing module 2110 is a module with visual feature processing capabilities, used to perform feature processing operations such as feature extraction and conversion on input data. This module is typically composed of multiple feature processing layers 21110, which perform different levels of feature processing on the input data to obtain feature data represented by abstract features, and then perform visual generation tasks based on the feature data at different levels. The feature processing module 2110 is installed on a hardware unit with corresponding functions, such as an AI (Artificial Intelligence) chip unit (such as a TPU and NPU), an ASIC, and an FPGA.

[0118] The feature processing layer 21110 is the basic unit that constitutes the feature processing module 2110. It performs feature processing on the input data at different levels to obtain feature data at different levels. It includes but is not limited to: encoding layer, decoding layer, convolution layer, pooling layer and normalization layer.

[0119] The feature data output by the feature processing layer 21110 is the feature data obtained by feature processing at different levels, and is generally expressed in the form of feature vectors (feature sequences, feature matrices). For example, in a visual model with a U-Net structure, the feature processing module has multiple feature processing layers of different sizes. Through feature processing methods such as convolution and downsampling operations, it outputs feature maps of different resolutions, that is, data encoding features at different levels. The 512×512-sized visual features output by the first encoding layer represent low-dimensional data features, and the 32×32-sized visual features output by the last encoding layer represent high-dimensional semantic features. As the level goes deeper, the dimension of the feature data will gradually decrease, but the expression feature data has a deep semantic level.

[0120] The tuning module 2112 is a module with the function of aggregating and tuning feature data, and is used to perform feature tuning on the input feature data, such as feature selection (selecting features with discrimination and target information from the input feature data, such as edges, textures, color histograms, key points, etc.), feature enhancement (using preprocessing techniques to improve the quality of features, such as normalization, convolution, activation, pooling, etc. to enhance the expressiveness of feature data), and feature dimensionality reduction (increasing or reducing feature dimensions by transposing matrices while keeping detail information from being lost, reducing computational complexity, and avoiding overfitting), which are not limited here. The tuning module 2112 is mounted on a hardware unit with corresponding functions, such as an AI chip unit (e.g., TPU and NPU), ASIC, and FPGA.

[0121] The tuning layer 21120 is the basic unit that constitutes the tuning module 2112. It performs feature tuning on feature data at different levels to achieve aggregate tuning of encoding features and decoding features at a specific level. Structurally, the tuning layer includes but is not limited to: a convolutional layer, a normalization layer, an activation layer, and a pooling layer. For example, the tuning layer consists of a convolutional layer and an activation layer (ReLU layer). Multiple tuning layers can be connected layer by layer (cascaded) or in parallel, which is not limited here.

[0122] Among them, multiple tuning layers 21120 are respectively connected to the feature processing layers 21110 of the corresponding levels in at least two feature processing modules 2110, and the connection method can be a cascade connection (Sequential Connection): the tuning layer receives the feature data of the corresponding levels from different feature processing modules in sequence, and tunes each input feature step by step. The connection method can also be a parallel connection (Parallel Connection): multiple tuning layers simultaneously obtain features from the corresponding levels of different feature processing modules, and perform tuning processing independently. Subsequently, these parallel-tuned features can be merged or selectively fused using methods such as attention mechanisms to generate a more comprehensive and refined feature representation. The connection method can also be a skip connection (Skip Connection): the tuning layer is not only connected to the feature processing layer at the same level, but can also cross levels to connect to the feature processing layer of the previous or next layer, to achieve long-range information transmission and multi-scale feature fusion. Another connection method is gated connection: when the tuning layer receives features from the feature processing module, it uses a gating mechanism to determine how to integrate and adjust the importance of features at different levels, thereby achieving dynamic tuning. This connection method is implemented between modules within the system and relies on an internal bus or chain data path.

[0123] The tuning result is the feature data obtained through aggregation and tuning by the tuning module 2112, which is an optimized representation obtained after aggregation and tuning of the input feature data. Specifically, the tuning layer 21120 receives the feature data output from the feature processing layer 21110, and performs targeted aggregation and tuning on it through methods such as feature selection, feature enhancement, or feature dimension increase and decrease to improve the quality, discrimination, and relevance of the features to the target task. The tuning result can be expressed as a more refined feature vector (or feature sequence, feature matrix) with higher information content. These optimized features will be used in subsequent visual information analysis, decision making, or other visual generation tasks.

[0124] Visual data refers to the processing results of a visual generation task based on the fine-tuning results. For example, in an image restoration or enhancement task, the fine-tuning results generate the restored or enhanced high-quality image as visual data. Another example is the fine-tuning results generate the video summary consisting of key frames or key segments as visual data in a video summary task.

[0125] In the visual model system of the embodiment of the present disclosure, multiple tuning layers of the tuning module are respectively connected to the feature processing layers of corresponding levels in at least two feature processing modules, so that complete feature data output by multiple feature processing layers are obtained, detailed information on multiple levels is retained, and tuning processing on corresponding levels is implemented in multiple tuning layers, thereby optimizing the output feature data and generating more accurate visual data. In addition, at least two feature processing modules and at least one tuning module in the visual model are decoupled, so that the tuning module can be flexibly selected during the visual data generation process, thereby improving the versatility of the visual model system. In the visual data generation process, the feature processing module can be shared to perform multi-task shared tuning, thereby improving the efficiency of visual data generation. In the training process of the visual model, only the tuning module can be trained for different downstream visual generation tasks without training the feature processing module, thereby reducing training overhead and improving training efficiency.

[0126] In an optional embodiment of the present disclosure, at least two feature processing modules 2110 include an encoding module and a decoding module, the encoding module includes multiple encoding layers, the decoding module includes multiple decoding layers, and multiple tuning layers 21120 are respectively connected to the encoding layers and decoding layers of the corresponding levels.

[0127] The encoding module is a module with visual feature encoding capabilities, used to encode features of input data. This module is typically composed of multiple encoding layers, which encode features at different levels of the input data, obtain encoded feature data represented by abstract features, and perform visual generation tasks based on the encoded features at different levels. The encoding module is installed on hardware units with corresponding functions, such as AI chip units (such as TPUs and NPUs), ASICs, and FPGAs.

[0128] The coding layer is the basic unit that constitutes the coding module. It encodes the features of the input data at different levels to obtain coding features at different levels.

[0129] The decoding module is a module with visual feature decoding capabilities. It is used to decode coded features. This module is usually composed of multiple decoding layers. It decodes coded features at different levels, obtains decoded feature data represented by abstract features, and performs visual generation tasks based on the decoded features at different levels. The decoding module is installed on hardware units with corresponding functions, such as AI chip units (such as TPU and NPU), ASICs, and FPGAs.

[0130] The decoding layer is the basic unit that constitutes the decoding module. It decodes the encoding features at different levels to obtain decoding features at different levels.

[0131] Among them, multiple tuning layers 21120 are respectively connected to the encoding layer and decoding layer of the corresponding level, and the connection method can be cascade connection, parallel connection, skip connection or gated connection.

[0132] Taking skip connections as an example, Figures 3 and 4 show the architectures of two visual models.

[0133] FIG3 shows a schematic diagram of a first architecture of a visual model in a visual model system provided by an embodiment of the present disclosure, as shown in FIG3 :

[0134] The light pink color represents the encoding layer, the light yellow color represents the decoding layer, the light blue color represents the intermediate layer, and the orange color represents the optimization layer. The task model consists of multiple symmetrical encoding layers (encoding layer_1, encoding layer_2, encoding layer_3, and encoding layer_4), multiple decoding layers (decoding layer_1, decoding layer_2, decoding layer_3, and decoding layer_4), intermediate layers, and multiple optimization layers. The optimization layers are connected to the encoding and decoding layers at the corresponding levels, and the optimization layers are connected layer by layer (cascaded). The encoding features (encoding features_1, encoding features_2, encoding features_3, and encoding features_4) output by each encoding layer and the decoding features (decoding features_1, decoding features_2, decoding features_3, and decoding features_4) output by each decoding layer are input to the optimization layer at the corresponding level. The optimization result output by the final optimization layer is used to generate visual data.

[0135] FIG4 shows a schematic diagram of a second architecture of a visual model in a visual model system provided by an embodiment of the present disclosure, as shown in FIG4 :

[0136] The light pink color represents the encoding layer, the light yellow color represents the decoding layer, the light blue color represents the intermediate layer, the orange color represents the optimization layer, and the light green color represents the feature fusion layer. The task model consists of multiple symmetrical encoding layers (encoding layer_1, encoding layer_2, encoding layer_3, and encoding layer_4), multiple decoding layers (decoding layer_1, decoding layer_2, decoding layer_3, and decoding layer_4), an intermediate layer, and multiple optimization layers. The optimization layers are connected to the encoding and decoding layers of the corresponding layers, and the optimization layers are connected in parallel. The optimization layers are connected to the feature fusion layer. The encoding features (encoding features_1, encoding features_2, encoding features_3, and encoding features_4) output by each encoding layer and the decoding features (decoding features_1, decoding features_2, decoding features_3, and decoding features_4) output by each decoding layer are input to the optimization layer of the corresponding layer. The feature fusion layer fuses the output features of the multiple optimization layers to obtain the optimization result, and visual data is generated based on this optimization result.

[0137] In the visual model system of the embodiment of the present disclosure, multiple tuning layers are used to connect with the encoding layer and decoding layer of the corresponding levels respectively, so as to obtain a complete plurality of encoding features and decoding features, retain the encoding and decoding detail information of multiple levels, complete feature tuning at the corresponding level through multiple tuning layers, optimize the output features, obtain more accurate visual data, and improve the accuracy of visual processing.

[0138] In an optional embodiment of the present disclosure, the encoding layer includes at least two sub-coding layers for encoding tasks, the decoding layer includes at least two sub-decoding layers for decoding tasks, the sub-coding layers are connected layer by layer, and the sub-decoding layers are connected layer by layer, and multiple tuning layers 21120 are respectively connected to the sub-coding layers of the target encoding tasks and the sub-decoding layers of the target decoding tasks at the corresponding levels.

[0139] Since the codec and multiple tuning layers have been decoupled in the disclosed embodiment, different sets of tuning layers can be trained for different visual generation tasks. In the face of different visual generation tasks, it is necessary to determine the corresponding multiple tuning layers and complete the layer-by-layer connection between the codec layers. However, there will inevitably be some cross-type visual generation tasks, and the choice of codec layers will inevitably affect the execution of the visual generation tasks. For example, visual generation task 1 is a face recognition task, and visual generation task 2 is a building recognition task. For the existence of two sets of codec layers, if there is visual generation task 3, which is a real scene recognition task, it is necessary to complete face recognition and building recognition at the same time. It is necessary to integrate the two sets of codec layers so that the visual model can understand the input data of visual generation task 1 and output the visual data of visual generation task 1, and can also understand the input data of visual generation task 2 and output the visual data of visual generation task 2.

[0140] The encoding task is the encoding of data in a specific domain. It is an upstream task of the target task and can be understood as the task of encoding the features of the input data of the visual generation task. For example, in the downstream target task of image enhancement, the encoding task involves encoding the original pixel-level source image into multi-level feature maps. These feature maps can capture visual information such as edges, textures, and shapes at different levels of the source image.

[0141] The sub-coding layers of at least two coding tasks are model sub-units whose coding tasks are capable of encoding data for visual generation tasks. At least two sub-coding layers constitute a coding layer, and the sub-coding layers are connected layer by layer. The sub-coding layers are obtained by pre-training based on the pre-training data of the coding task. The pre-training data of the coding task is a set of pre-training data used to pre-train the sub-coding layer, which contains multiple pre-training sample instances corresponding to the coding task, including the pre-training data and / or pre-training data results of the coding task. The pre-training of the initial sub-coding layer is completed in a supervised or unsupervised manner, so that the sub-coding layer has the ability to process the coding task, and can then understand the input data of the downstream task of the visual generation task. The sub-coding layer of the target coding task is a sub-coding layer that outputs the coding features of the coding layer, and there is at least one sub-coding layer.

[0142] The decoding task is the task of decoding data in a specific domain. It is an upstream task of the visual generation task and can be understood as the task of decoding features to obtain the output data of the visual generation task. For example, in the downstream visual generation task of image enhancement, the decoding task involves decoding multiple layers of feature maps to restore the target image at the target pixel level.

[0143] The sub-decoding layers of at least two decoding tasks are model sub-units whose decoding tasks are capable of decoding data for visual generation tasks. At least two sub-decoding layers constitute a decoding layer, and the sub-decoding layers are connected layer by layer. The sub-decoding layers are obtained by pre-training based on the pre-training data of the decoding task. The pre-training data of the decoding task is a set of pre-training data used to pre-train the sub-decoding layer, which contains multiple pre-training sample instances corresponding to the decoding task, including the pre-training data and / or pre-training data results of the decoding task. The pre-training of the initial sub-decoding layer is completed in a supervised or unsupervised manner, so that the sub-decoding layer has the ability to process decoding tasks, and can then understand the input data of the downstream task of the visual generation task. The sub-decoding layer of the target decoding task is a sub-decoding layer that outputs the decoding features of the decoding layer, and there is at least one sub-decoding layer.

[0144] The stacked structure of multiple encoding and decoding layers allows for better visual information processing by leveraging the encoding and decoding capabilities of different "experts." The multiple tuning layers 21120 can be connected to the corresponding sub-encoding and sub-decoding layers, or only to the last sub-encoding and sub-decoding layers, as shown in Figures 5-8.

[0145] FIG5 shows a third schematic diagram of the architecture of a visual model in a visual model system provided by an embodiment of the present disclosure, as shown in FIG5 :

[0146] The light pink is the encoding layer, the light yellow is the decoding layer, the light blue is the intermediate layer, and the orange is the tuning layer. The visual model consists of multiple encoding layers, multiple decoding layers, intermediate layers, and multiple tuning layers in a symmetrical structure. Any encoding layer includes two sub-coding layers (encoding layer_1.1, encoding layer_2.1, encoding layer_1.2, encoding layer_2.2, encoding layer_1.3, encoding layer_2.3, encoding layer_1.4, and encoding layer_2.4), and any decoding layer includes two sub-decoding layers (decoding layer_1.1, decoding layer_2.1, decoding layer_1.2, decoding layer_2.2, decoding layer_1.3, decoding layer_2.4). The multiple tuning layers are respectively connected to the sub-coding layers (coding layer_2.1, coding layer_2.2, coding layer_2.3, and coding layer_2.4) of the target encoding task at the corresponding level and the decoding layers (decoding layer_2.1, decoding layer_2.2, decoding layer_2.3, and decoding layer_2.4) of the target decoding task. The multiple tuning layers are respectively connected to the encoding layer and decoding layer of the corresponding level, and the multiple tuning layers are connected layer by layer (cascade). The encoding features (coding features_1, encoding features_2, encoding features_3, and encoding features_4) output by the sub-coding layers of each target encoding task and the decoding features (decoding features_1, decoding features_2, decoding features_3, and decoding features_4) output by the sub-decoding layers of each target decoding task are input to the tuning layer of the corresponding level. The tuning result output by the tuning layer of the last level is used to obtain visual data based on the tuning result.

[0147] FIG6 shows a fourth schematic diagram of the architecture of a visual model in a visual model system provided by an embodiment of the present disclosure, as shown in FIG6 :

[0148] The light pink is the encoding layer, the light yellow is the decoding layer, the light blue is the intermediate layer, the orange is the tuning layer, and the light green is the feature fusion layer. The visual model consists of multiple encoding layers, multiple decoding layers, intermediate layers, and multiple tuning layers in a symmetrical structure. Any encoding layer includes two sub-coding layers (encoding layer_1.1, encoding layer_2.1, encoding layer_1.2, encoding layer_2.2, encoding layer_1.3, encoding layer_2.3, encoding layer_1.4, and encoding layer_2.4), and any decoding layer includes two sub-decoding layers (decoding layer_1.1, decoding layer_2.1, decoding layer_1.2, decoding layer_2.2, decoding layer_ The optimization layers are connected to the sub-encoding layers (encoding layer_2.1, encoding layer_2.2, encoding layer_2.3, and encoding layer_2.4) of the target encoding task and the sub-decoding layers (decoding layer_2.1, decoding layer_2.2, decoding layer_2.3, and decoding layer_2.4) of the target decoding task at the corresponding level, respectively. The optimization layers are connected in parallel and are connected to the feature fusion layer. The encoding features (encoding features_1, encoding features_2, encoding features_3, and encoding features_4) output by the sub-encoding layers of each target encoding task and the decoding features (decoding features_1, decoding features_2, decoding features_3, and decoding features_4) output by the sub-decoding layers of each target decoding task are input to the optimization layers of the corresponding level. The feature fusion layer fuses the output features of the multiple optimization layers to obtain an optimization result, and visual data is obtained based on the optimization result.

[0149] FIG7 shows a fifth schematic diagram of the architecture of a visual model in a visual model system provided by an embodiment of the present disclosure, as shown in FIG7 :

[0150] Light pink represents the encoding layer, light yellow represents the decoding layer, light blue represents the intermediate layer, orange represents the optimization layer, and blue represents the feature fusion layer between the encoding and decoding layers. The visual model consists of multiple encoding layers, multiple decoding layers, intermediate layers, and multiple optimization layers in a symmetrical structure. Each encoding layer includes two sub-encoding layers (encoding layer_1.1, encoding layer_2.1, encoding layer_1.2, encoding layer_2.2, encoding layer_1.3, encoding layer_2.3, encoding layer_1.4, and encoding layer_2.4). Each decoding layer includes two sub-decoding layers (decoding layer_1.1, decoding layer_2.1, decoding layer_1.2, decoding layer_2.2, decoding layer_1.3, decoding layer_2.3, decoding layer_1.4, and decoding layer_2.4). The sub-encoding layers are connected through a feature fusion layer. Multiple optimization layers are connected to all sub-encoding layers and all sub-decoding layers of the corresponding level, and the multiple optimization layers are connected layer by layer (cascaded). The fused encoding features and fused decoding features output by each feature fusion layer are input into the tuning layer of the corresponding level, and the tuning result output by the tuning layer of the last layer is used to obtain visual data based on the tuning result.

[0151] FIG8 shows a sixth schematic diagram of the architecture of a visual model in a visual model system provided by an embodiment of the present disclosure, as shown in FIG8 :

[0152] Light pink represents the encoding layer, light yellow represents the decoding layer, light blue represents the intermediate layer, orange represents the optimization layer, blue represents the feature fusion layer between the encoding and decoding layers, and light green represents the final output feature fusion layer. The visual model consists of multiple encoding layers, multiple decoding layers, intermediate layers, and multiple optimization layers in a symmetrical structure. Each encoding layer includes two sub-encoding layers (encoding layer_1.1, encoding layer_2.1, encoding layer_1.2, encoding layer_2.2, encoding layer_1.3, encoding layer_2.3, encoding layer_1.4, and encoding layer_2.4). Each decoding layer includes two sub-decoding layers (decoding layer_1.1, decoding layer_2.1, decoding layer_1.2, decoding layer_2.2, decoding layer_1.3, decoding layer_2.3, decoding layer_1.4, and decoding layer_2.4). The sub-encoding layers are connected by a feature fusion layer. Multiple optimization layers are connected in parallel, and multiple optimization layers are connected to the feature fusion layer. The fused encoding features and fused decoding features output by each feature fusion layer are input into the tuning layer of the corresponding level. The feature fusion layer fuses the output features of multiple tuning layers to obtain the tuning result, and visual data is obtained based on the tuning result.

[0153] In the visual model system of the embodiment of the present disclosure, the encoding and decoding capabilities of different encoding and decoding tasks are integrated through a stacking structure, the accuracy of each encoding feature and the accuracy of each encoding feature are improved, and more accurate encoding and decoding features are provided for completing feature tuning.

[0154] In an optional embodiment of the present disclosure, the feature processing layer 21110 is used to process sample data of the visual generation task to obtain sample task features, and output the sample task features to the connected tuning layer 21120;

[0155] The optimization layer 21120 is further used to optimize the received sample task features to obtain prediction output features, wherein the prediction output features are used to obtain task prediction results;

[0156] The feature processing layer 21110 and the tuning layer 21120 are also used for training based on task prediction results.

[0157] In the disclosed embodiment, the visual model is pre-trained, and the feature processing layer is the feature processing layer in the general visual model. It needs to be specifically trained on the downstream visual generation task when connected to the tuning layer.

[0158] In the disclosed embodiment, feature processing layer 21110 is a deep learning model capable of performing feature processing on the input data of the visual generation task. That is, it can understand the input data of the visual generation task and perform feature processing on it. Feature processing layer 21110 is pre-trained based on pre-training data of a pre-training task. The pre-training task is an upstream task of the visual generation task, and the visual generation task is a downstream task of the pre-training task. For example, the pre-training task is an image processing task, and the visual generation task is an image generation task.

[0159] Visual generation tasks involve generating visual data based on input data (such as noise, conditional images, and text descriptions) using deep learning techniques. These tasks typically involve a process from constructing low-level features to understanding and synthesizing high-level semantics, aiming to output high-quality visual data such as images, videos, and 3D (three-dimensional) models.

[0160] Sample data for visual generation tasks is the data used to train and evaluate visual models. Generally, this sample data has corresponding labeled outputs. For example, in an image restoration task, the sample data might include a damaged image as input and the corresponding high-quality restored image as output. Alternatively, in a text-to-image generation task, the sample data might consist of pairs of text descriptions and their corresponding real-world images. This sample data provides the foundation for the model to learn how to generate appropriate visual content from given conditions and guides the learning and optimization of model parameters.

[0161] Sample task features are feature representations obtained after processing sample data from the visual generation task through the feature processing layer 21110. They are generally expressed as feature vectors (feature sequences, feature matrices). Sample task features are typically obtained by performing feature processing on sample data from the visual generation task at different levels, resulting in feature data at different levels.

[0162] The predicted output features are sample task features obtained through aggregation and optimization by the optimization module 2112 , and are optimized representations obtained after aggregation and optimization of the sample task features.

[0163] The task prediction result is the task processing result of the visual generation task generated based on the predicted output features.

[0164] Training is performed based on the task prediction results. Specifically, the loss value is determined based on the task prediction results, and the parameters of the feature processing layer 21110 and the tuning layer 21120 are adjusted based on the loss value.

[0165] In the visual model system of the embodiment of the present disclosure, a visual model adapted to the visual generation task is obtained by training the visual model connected to the feature processing layer and the tuning layer.

[0166] In an optional embodiment of the present disclosure, the feature processing layer 21110 is further configured to process target task samples of a downstream target visual task to obtain target task features, and output the target task features to a connected tuning layer 21120;

[0167] The optimization layer 21120 is further used to optimize the received target task features to obtain predicted output features, wherein the predicted output features are used to obtain the target task results;

[0168] The tuning layer 21120 is also used for training based on the target task results.

[0169] In the embodiment of the present disclosure, because the feature processing layer 21110 is a deep learning model with the ability to perform feature processing on the input data of the visual generation task, that is, it can understand the input data of the visual generation task and perform feature processing on it. Therefore, it is only necessary to train the tuning layer 21120. In the face of different downstream target visual tasks, multiple tuning modules 2112 can be pre-trained. During the application process, the tuning module 2112 corresponding to the downstream target visual task is determined, and the connection with the multiple feature processing layers 21110 in the feature processing module 2110 is completed. Then, a visual model 210 adapted to the downstream target visual task can be obtained. At this time, the visual model 210 can realize multi-task shared processing.

[0170] Downstream target vision tasks are visual generation tasks that, after pre-training, are further optimized and adjusted for specific application scenarios and requirements. These tasks are based on the already trained feature processing layer 21110 and achieve good adaptability to specific tasks by targeted retraining of the tuning module 2112 in the model. For example, based on a pre-trained image generation model, it can be applied to different downstream target vision tasks, such as image inpainting, super-resolution reconstruction, and style transfer.

[0171] Target task samples for downstream target vision tasks are data used to train and evaluate vision models for these tasks. Typically, these samples have corresponding labeled outputs. These samples help the model learn how to generate visual content suitable for the downstream target vision task from given conditions and guide the learning and optimization of model parameters.

[0172] The target task features are feature representations obtained after the target task samples of the downstream target vision task are processed by the feature processing layer 21110, and are generally expressed as feature vectors (feature sequences, feature matrices). The target task features are typically obtained by performing feature processing on the target task samples of the downstream target vision task at different levels, resulting in feature data at different levels.

[0173] The predicted output features are target task features obtained through aggregation and optimization by the optimization module 2112 , and are optimized representations obtained after aggregation and optimization of the target task features.

[0174] The target task result is the task processing result of the downstream target visual task generated based on the predicted output features.

[0175] Training is performed based on the task prediction results. Specifically, the loss value is determined based on the task prediction results, and the parameters of the tuning layer 21120 are adjusted based on the loss value.

[0176] In the visual model system of the embodiment of the present disclosure, for downstream target visual tasks, only multiple tuning layers are trained, avoiding training of other model layers, thereby obtaining a visual model adapted to downstream target visual tasks, reducing training overhead and improving training efficiency.

[0177] 9 , which shows a flowchart of a method for constructing a visual model according to an embodiment of the present disclosure, including the following specific steps:

[0178] Step 902: Obtain target model parameters.

[0179] Target model parameters are the configuration information parameters required to build and run a visual model. They cover multiple dimensions, from model architecture and weight data to hyperparameter settings during training and hardware resource management. They include model parameters of pre-trained visual models and tuning parameters, such as architecture parameters, model weight parameters, model hyperparameters, and resource parameters.

[0180] Architectural parameters include: layer depth (such as the number of layers in the feature processing module and the tuning module): This directly affects the required computing resources (such as the storage size of the NPU and TPU, the number of logic units in the FPGA or ASIC, etc.) and possible latency; network layer type (convolution, pooling, fully connected, etc.) and its configuration (such as convolution kernel size, stride, padding, etc.): Different layer structures and configurations correspond to different types of parallel computing requirements, which is crucial for hardware design. For example, large-scale parallel computing capabilities are extremely important for convolution layers, while pooling layers have high requirements for memory bandwidth.

[0181] Model weight parameters include: Storage requirements: Model weights need to be stored in hardware. For example, in embedded systems, they may be stored in ROM or Flash. In high-performance computing platforms, they may need to occupy the storage space of GPU video memory or TPU tensor cores. Accelerated computing: For weight data, hardware accelerators such as NPUs, TPUs, ASICs, or FPGAs typically optimize the speed of weight reading and matrix operations, reducing the time overhead of accessing main memory.

[0182] Model hyperparameters include: Computing resource allocation: Hyperparameters such as learning rate and batch size affect hardware resource utilization during training, such as determining NPU / TPU core utilization, memory usage, and I / O (Input / Output) operation frequency. Energy consumption control: In power-sensitive embedded scenarios, selecting appropriate hyperparameters helps balance computing performance and energy consumption, such as by adjusting model complexity to accommodate limited hardware computing power.

[0183] Resource parameters include: Quantization strategy: To meet low power consumption or high efficiency requirements, the hardware may need to perform model quantization. In this case, the target model parameters also include the number of quantization bits (such as 8-bit quantization) and the quantization method, which will affect hardware design and execution efficiency. Tensor processor configuration: In AI chip units such as TPUs, the target model parameters may also involve how to efficiently map to the layout and pipeline scheduling mechanism of the tensor core.

[0184] For example, when building a U-Net-structured diffusion vision model, a series of target model parameters are obtained, including: Architecture parameters: Network depth: Determines the number of layers contained in each encoder and decoder stage in the model. Layer types and their configurations: For example, different sizes and numbers of convolutional layers, max pooling or average pooling layers, upsampling layers, etc., and their parameters such as kernel size, stride, and padding are specified. Model weight parameters: Storage requirements: Allocate storage space for the weights in the model to ensure adaptability to different hardware environments, from embedded devices to high-performance GPU servers. Accelerated computing: Optimize weight data reading and computation efficiency. Model hyperparameters: Training process parameters: Set the learning rate, batch size, etc. to optimize training results and fully utilize computing resources. Energy consumption control: For low-power scenarios, select the appropriate model complexity and training strategy to balance performance and energy consumption. Resource parameters: Quantization strategy: If the model requires quantization to adapt to specific hardware, set the appropriate quantization bit number and method to reduce model accuracy loss while improving execution speed and energy efficiency. Tensor Processor Configuration: When using tensor processing units (TPUs), consider how to effectively map your model onto them for efficient execution.

[0185] Obtain the target model parameters and provide parameter support for the subsequent construction of the corresponding feature processing module and tuning module.

[0186] Step 904: Based on the target model parameters, construct at least two feature processing modules and at least one tuning module, wherein the feature processing module includes multiple feature processing layers, and the tuning module includes multiple tuning layers. The tuning layers are used to perform tuning processing on the input feature data to obtain tuning results, and the tuning results are used to generate visual data.

[0187] Exemplarily, based on the target model parameters, at least two feature processing modules are constructed, each module containing multiple convolutional or other types of feature extraction layers, and at least one tuning module composed of multiple levels, which is responsible for fine-tuning the input feature data to generate high-quality output results.

[0188] Based on the target model parameters, at least two feature processing modules and at least one tuning module are constructed to provide module support for subsequent connection to obtain a visual model.

[0189] Step 906: Connect the multiple tuning layers to the feature processing layers of corresponding levels in at least two feature processing modules to obtain a visual model.

[0190] For example, after building independent feature processing and optimization modules, each layer in the optimization module is connected to the corresponding layer in the feature processing module according to the U-Net structure. This forms a complete forward propagation path, ultimately resulting in a diffusion vision model with a U-Net structure that can be used for tasks such as image generation, image restoration, or image segmentation.

[0191] In the embodiment of the present disclosure, at least two feature processing modules and at least one tuning module are constructed based on the target model parameters, and the multiple tuning layers of the tuning module are respectively connected to the feature processing layers of the corresponding levels in the at least two feature processing modules, so that the complete feature data output by the multiple feature processing layers in the visual model retains the detailed information on the multiple levels, and the tuning processing on the corresponding levels is implemented in the multiple tuning layers, thereby optimizing the output feature data and generating more accurate visual data. In addition, the at least two feature processing modules and the at least one tuning module in the visual model are decoupled, so that the tuning module can be flexibly selected during the visual data generation process, thereby improving the versatility of the visual model system. The feature processing module can also be shared during the visual data generation process to perform multi-task shared tuning, thereby improving the efficiency of visual data generation. In the training process of the visual model, only the tuning module can be trained for different downstream visual generation tasks without training the feature processing module, thereby reducing training overhead and improving training efficiency.

[0192] In an optional embodiment of the present disclosure, step 902 includes the following specific steps:

[0193] Obtaining model parameters and tuning parameters of a pre-trained visual model corresponding to a visual generation task, wherein the pre-trained visual model includes at least two feature processing modules;

[0194] Correspondingly, step 904 includes the following specific steps:

[0195] Constructing at least two feature processing modules based on the pre-trained visual model, and setting initial parameters of the at least two feature processing modules according to model parameters of the pre-trained visual model;

[0196] Based on the tuning parameters, at least one tuning module is constructed, and parameters of the tuning module are initialized.

[0197] A pre-trained visual model is a pre-trained visual model with general visual information processing capabilities. It is a deep learning model capable of performing feature processing on input data for visual generation tasks. Pre-trained visual models are typically built based on various deep neural network architectures such as CNN, RNN, and Transformer. They are pre-trained on large-scale public datasets and have model weight parameters of a certain quality. Pre-trained models serve as foundational models for various downstream visual tasks. For example, a pre-trained visual model is a pre-trained U-Net model, which can extract high-level abstract features from input images for subsequent image recognition tasks or other visual generation tasks.

[0198] The model parameters of a pre-trained vision model are the model weight parameters learned by the model during the training process. They store the knowledge and patterns learned by the model from the pre-training data, determine how the model maps input to output, and the performance and ability of the model in solving specific problems. Model parameters include but are not limited to the convolution kernel weights of the convolutional layer, the weight matrix of the fully connected layer, bias terms, and other hyperparameters. For pre-trained vision models, their model parameters are optimized configurations based on a large amount of data and can be used as initialization parameters for downstream vision tasks to improve the model's convergence speed and performance. For example, in a pre-trained U-Net model, model parameters may include the filter weights of each convolutional layer, the parameters of the pooling layer, and the parameters of the upsampling layer in the decoding stage.

[0199] Initial parameters are the initial weight parameters set for at least two feature processing modules when building a visual model. When building a feature processing module based on a pretrained visual model, the initial parameters can be derived from the model parameters of the corresponding part of the pretrained visual model. This is done to leverage the feature representation capabilities already learned by the pretrained model.

[0200] Tuning parameters are the initial weight parameters set for at least one tuning module when building a visual model. When building a feature processing module based on a pre-trained visual model, tuning parameters can be manually set, randomly generated, or derived from model parameters of corresponding parts of other models, without limitation here.

[0201] In the embodiment of the present disclosure, by initializing the parameters of the feature processing module and the tuning module, support is provided for the subsequent construction of the visual model.

[0202] In an optional embodiment of the present disclosure, after step 906, the following specific steps are further included:

[0203] Based on the sample data of the visual generation task, the visual model is trained to obtain a trained visual model;

[0204] Based on target task samples of multiple downstream target visual tasks, the trained visual models are tuned respectively to obtain target tuning modules corresponding to the multiple downstream target visual tasks.

[0205] Each target tuning module is connected to at least two feature processing modules to obtain an updated visual model.

[0206] In the disclosed embodiments, the visual model is pre-trained, and the feature processing layer is the feature processing layer of the general visual model. It needs to be specifically trained on the downstream visual generation task when connected to the tuning layer. Because the codec and the multiple tuning layers have been decoupled, multiple different tuning modules can be trained for different downstream target visual tasks. For different downstream target visual tasks, it is necessary to determine the corresponding target tuning module and complete the layer-by-layer connection with at least two feature processing modules.

[0207] Visual generation tasks involve generating visual data based on input data (such as noise, conditional images, and text descriptions) using deep learning techniques. These tasks typically involve a process from low-level feature construction to high-level semantic understanding and synthesis, aiming to output high-quality visual data such as images, videos, and 3D models.

[0208] Sample data for visual generation tasks is the data used to train and evaluate visual models. Generally, this sample data has corresponding labeled outputs. For example, in an image restoration task, the sample data might include a damaged image as input and the corresponding high-quality restored image as output. Alternatively, in a text-to-image generation task, the sample data might consist of pairs of text descriptions and their corresponding real-world images. This sample data provides the foundation for the model to learn how to generate appropriate visual content from given conditions and guides the learning and optimization of model parameters.

[0209] Downstream target vision tasks are specific tasks that are completed by further fine-tuning or optimizing the pre-trained model for specific vision generation application scenarios. These tasks usually have clear goals and application scenarios, and the pre-trained model needs to be adjusted to adapt to the new data distribution and task requirements. Downstream target vision tasks are the process of fine-tuning the general capabilities of the pre-trained model, improving the model's performance in specific scenarios through targeted training and tuning modules. For example, based on a pre-trained diffusion vision model, there can be multiple downstream target vision tasks, such as image super-resolution, style transfer, image colorization, image-to-image translation, etc.

[0210] Target task samples for downstream target vision tasks are data used to train and evaluate vision models for these tasks. Typically, these samples have corresponding labeled outputs. These samples help the model learn how to generate visual content suitable for the downstream target vision task from given conditions and guide the learning and optimization of model parameters.

[0211] The target tuning module is a tuning module for downstream target visual tasks. It has the function of aggregating and tuning feature data for downstream target visual tasks, and is used to perform feature tuning on the input feature data of downstream target visual tasks.

[0212] In the visual model system of the disclosed embodiment, corresponding tuning modules are trained for multiple downstream target visual tasks. During the visual data generation process, the feature processing module can be shared to perform multi-task shared tuning, thereby improving the efficiency of visual data generation. During the training process of the visual model, only the tuning module can be trained for different downstream visual generation tasks without training the feature processing module, thereby reducing training overhead and improving training efficiency.

[0213] Referring to FIG10 , FIG10 shows a flowchart of a visual task processing method provided by an embodiment of the present disclosure, which includes the following specific steps:

[0214] Step 1002: Obtain task data of the target visual generation task.

[0215] The embodiments of the present disclosure are applied to applications, websites or mini-programs with visual task processing functions. On applications, websites or mini-programs, visual task processing functions are implemented. For example, a website with a visual model deployed can implement functions such as text-to-image, text-to-video, image-to-image, image-to-video, image-to-text, video-to-text, image enhancement, image denoising, image classification, and visual entity recognition. For example, a third-party application can implement corresponding visual task processing functions by calling a deployed visual model through an application programming interface (API).

[0216] The target visual generation task is a visual generation task to be processed, including but not limited to: image generation task, video generation task, image enhancement task, video enhancement task, image denoising task, video denoising task, image classification task, video classification task and visual entity recognition task.

[0217] The task data for the target visual generation task is the task input data for the target visual generation task. The task data is data in at least one modality, including but not limited to text, images, audio, and video. For example, if the target visual generation task is an image generation task, the task data for the target visual generation task is at least one source image or a text describing the target image.

[0218] The task data of the target visual generation task can be obtained by directly obtaining the task data sent from the front end, or by obtaining the task data from a database, which is not limited here.

[0219] For example, on a website where a visual model is deployed, the user clicks on the image generation function in the image field through the front end, determines that the target visual generation task is an image enhancement task, and the user uploads a 1080P source image to be enhanced.

[0220] Obtain the task data of the target visual generation task, which provides input data for subsequent feature processing.

[0221] Step 1004: Input the task data into the target vision model corresponding to the target vision generation task, and obtain the feature data output by each feature processing layer in the target vision model, wherein the target vision model includes at least two feature processing modules and at least one tuning module, the feature processing module includes multiple feature processing layers, the tuning module includes multiple tuning layers, and the multiple tuning layers are respectively connected to the feature processing layers of corresponding levels in the at least two feature processing modules.

[0222] The target visual model is a visual model used to process the target visual generation task. The target task model includes a codec and multiple tuning layers. The codec is a symmetrical structure with multiple encoding layers and multiple decoding layers. The multiple tuning layers are respectively connected to the encoding layer and decoding layer of the corresponding level. The target visual model is obtained by training multiple tuning layers in the pre-trained visual model. The training is further training adapted to the target task, including but not limited to: fine-tuning training, reinforcement learning, transfer learning, parameter-efficient transfer learning and memory-efficient transfer learning. Among them, fine-tuning training and reinforcement learning are visual information processing for the target visual generation task, that is, the target visual model has the ability to perform the target visual generation task. According to the data processing modality, the target visual model includes but is not limited to: image processing model and audio processing model, video processing model and structured data processing model. According to the task division, the target visual model includes but is not limited to: generation model, classification model (discriminant model), strategy selection model and regression prediction model. According to the model type, visual models include but are not limited to: CNN model, RNN model, LSTM model, Transformer model, BERT model, U-Net model, diffusion model, stable diffusion model, GAN model and large model.

[0223] This can be understood as the pre-trained visual model being the target visual model's additional model units, excluding the multiple fine-tuning layers, decoupling the codec from the multiple fine-tuning layers. When multiple sets of fine-tuning layers are trained for a target visual generation task, the target visual model can be obtained by simply identifying the corresponding fine-tuning layers and connecting them layer by layer with the codec. This allows for shared processing across multiple visual generation tasks.

[0224] Exemplarily, a 1080P source image is input into the image enhancement task model to obtain the image coding features Feature Encoding (image coding feature_1, image coding feature_2, ... image coding feature_N) output by each encoding layer in the image enhancement task model and the image decoding features Feature Decoding (image decoding feature_1, image decoding feature_2, ... image decoding feature_N) output by each decoding layer.

[0225] The task data is input into the target visual model corresponding to the target visual generation task, and the feature data output by each feature processing layer in the target visual model is obtained. In the target visual model, multiple tuning layers are used to connect with the encoding layer and decoding layer of the corresponding levels respectively, so as to obtain complete multiple encoding features and decoding features, retain the detailed information of multiple levels, and provide accurate encoding and decoding features for the subsequent feature tuning. Moreover, based on the sample data of the target visual generation task, only the multiple tuning layers in the pre-trained visual model are trained, avoiding the training of other model layers, and obtaining a deep learning model suitable for the target visual generation task, thereby reducing the training overhead and improving the training efficiency.

[0226] Step 1006: The feature data output by each feature processing layer is respectively input into the tuning layer connected to each feature processing layer for tuning processing, and the target visual data is generated based on the tuning results of each tuning layer.

[0227] The target visual data is the processing result of the target visual generation task generated based on the tuning results. For example, in an image restoration or enhancement task, the visual data of the restored or enhanced high-quality image is generated based on the tuning results. Another example is that in a video summarization task, the visual data of the video summary composed of key frames or key segments is generated based on the tuning results.

[0228] The feature data output by each feature processing layer are respectively input into the tuning layer connected to each feature processing layer for tuning processing. The specific method is: the encoding features output by each encoding layer and the decoding features output by each decoding layer are input into the corresponding tuning layer layer by layer, and each encoding feature and each decoding feature are aggregated and tuned to obtain the tuning results of each tuning layer.

[0229] Exemplarily, the image coding features Feature Encoding (image coding features_1, image coding features_2, ... image coding features_N) output by each encoding layer and the image decoding features Feature Decoding (image decoding features_1, image decoding features_2, ... image decoding features_N) output by each decoding layer are input layer by layer into the corresponding tuning layer (tuning layer_1, tuning layer_2 ... tuning layer_N), and each image coding feature and each image decoding feature is aggregated and tuned to obtain the tuning result Features Aggregate of the last tuning layer. Based on the tuning result, the image-enhanced 4K target image is obtained.

[0230] In the embodiment of the present disclosure, in the target visual model, multiple tuning layers are used to connect with the encoding layer and decoding layer of the corresponding levels respectively, so as to obtain multiple complete encoding features and decoding features, retain the detailed information of multiple levels, complete feature tuning at the corresponding levels through multiple tuning layers, optimize the output features, obtain more accurate target visual data, and improve the accuracy of visual generation. Moreover, based on the sample data of the target visual generation task, only the multiple tuning layers in the pre-trained visual model are trained, avoiding the training of other model layers, thereby obtaining a deep learning model suitable for the target visual generation task, reducing the training overhead and improving the training efficiency.

[0231] In an optional embodiment of the present disclosure, the at least two feature processing modules include an encoding module and a decoding module, the encoding module includes multiple encoding layers, the decoding module includes multiple decoding layers, and the multiple tuning layers are respectively connected to the encoding layers and decoding layers of corresponding levels;

[0232] Correspondingly, step 1004 includes the following specific steps:

[0233] Input the task data into the target visual model corresponding to the target visual generation task, and obtain the encoding features output by each encoding layer and the decoding features output by each decoding layer in the target visual model;

[0234] Correspondingly, step 1006 includes the following specific steps:

[0235] The encoding features output by each encoding layer and the decoding features output by each decoding layer are input into the connected tuning layer for tuning processing, and the target visual data is generated based on the tuning results of each tuning layer.

[0236] The hierarchical order of multiple tuning layers is the order in which the multiple tuning layers are connected, as shown in Figures 3 and 4. According to the encoding order, the tuning layer connected to the encoding layer that is encoded first is the later tuning layer, and the tuning layer connected to the encoding layer that is encoded later is the earlier tuning layer. According to the decoding order, the tuning layer connected to the decoding layer that is decoded first is the earlier tuning layer, and the tuning layer connected to the decoding layer that is decoded later is the later tuning layer. The previous tuning layer and the next tuning layer determine the aggregation tuning order of data features at different levels.

[0237] In an optional embodiment of the present disclosure, multiple tuning layers are connected layer by layer;

[0238] Correspondingly, the encoding features output by each encoding layer and the decoding features output by each decoding layer are input into the connected tuning layer for tuning processing, and the target visual data is generated based on the tuning results of each tuning layer, including the following specific steps:

[0239] The encoding features output by each encoding layer and the decoding features output by each decoding layer are input into the connected tuning layer for tuning processing. In accordance with the hierarchical order of multiple tuning layers, the tuning result of the previous tuning layer is input into the next tuning layer to obtain the output features of the last tuning layer;

[0240] Based on the output features of the last fine-tuning layer, target visual data is generated.

[0241] The encoding features output by each encoding layer and the decoding features output by each decoding layer are input into the connected tuning layer for tuning processing, and the target visual data is generated based on the tuning results of each tuning layer. An optional method is: the encoding features output by each encoding layer and the decoding features output by each decoding layer are respectively input into the tuning layer of the corresponding level, and the tuning result of the previous tuning layer is input into the next tuning layer in the hierarchical order of multiple tuning layers to obtain the tuning result of the last tuning layer, and the target visual data is generated based on the tuning result of the last tuning layer.

[0242] The encoding features output by each encoding layer and the decoding features output by each decoding layer are respectively input into the corresponding level of the tuning layer. And according to the hierarchical order of multiple tuning layers, the tuning result of the previous tuning layer is input into the next tuning layer to obtain the tuning result of the last tuning layer. The specific method is shown in Formula 1:

[0243] Formula 1

[0244] Among them, G() is the aggregation optimization process. The specific execution process is: input the encoding feature Feature Encoding_1 output by the first-level encoding layer and the decoding feature Feature Decoding_1 of the first level into the optimization layer of the first level to obtain the optimization result Feature Aggregate_1 of the optimization layer of the first level; input the optimization result Feature Aggregate_1 of the optimization layer of the first level, the encoding feature Feature Encoding_2 output by the second-level encoding layer, and the decoding feature Feature Decoding_2 of the second level into the optimization layer of the second level to obtain the optimization result Feature Aggregate_2 of the optimization layer of the second level; until the optimization result Feature Aggregate(N-1) of the optimization layer of the N-1 level is obtained, and the optimization result Feature Aggregate(N-1) of the optimization layer of the N-1 level, the encoding feature Feature Encoding_N output by the encoding layer of the N level, and the decoding feature Feature Decoding_2 of the N level are input into the optimization layer of the second level to obtain the optimization result Feature Aggregate_2 of the optimization layer of the second level. Decoding_N inputs the Nth level optimization layer to obtain the optimization result Feature Aggregate_N of the Nth level optimization layer.

[0245] Exemplarily, the image coding features Feature Encoding (image coding features_1, image coding features_2, ... image coding features_N) output by each encoding layer and the image decoding features Feature Decoding (image decoding features_1, image decoding features_2, ... image decoding features_N) output by each decoding layer are respectively input into the tuning layer of the corresponding level (tuning layer_1, tuning layer_2 ... tuning layer_N), and the tuning result of the previous tuning layer is input into the next tuning layer in the hierarchical order of multiple tuning layers to obtain the tuning result Features Aggregate of the last tuning layer, and based on the tuning results, the image-enhanced 4K target image is obtained.

[0246] In an optional embodiment of the present disclosure, multiple tuning layers are connected layer by layer;

[0247] Correspondingly, the encoding features output by each encoding layer and the decoding features output by each decoding layer are input into the connected tuning layer for tuning processing, and the target visual data is generated based on the tuning results of each tuning layer, including the following specific steps:

[0248] The encoding features output by each encoding layer and the decoding features output by each decoding layer are input into the connected tuning layer for tuning processing, and the tuning results are determined based on the tuning results of multiple tuning layers, and the target visual data is generated based on the tuning results.

[0249] Taking feature fusion as an example, the encoding features output by each encoding layer and the decoding features output by each decoding layer are respectively input into the tuning layer of the corresponding level to obtain the tuning results of multiple tuning layers. Based on the tuning results of multiple tuning layers, the tuning result is determined. The specific method is shown in Formula 2:

[0250] Formula 2

[0251] Among them, G() is the aggregation tuning process, and F() is the feature fusion process. The specific execution process is as follows: the encoding feature Feature Encoding_1 output by the encoding layer of the first level and the decoding feature Feature Decoding_1 of the first level are input into the optimization layer of the first level to obtain the optimization result Feature Aggregate_1 of the optimization layer of the first level; the encoding feature Feature Encoding_2 output by the encoding layer of the second level and the decoding feature Feature Decoding_2 of the second level are input into the optimization layer of the second level to obtain the optimization result Feature Aggregate_2 of the optimization layer of the second level; until the optimization result Feature Aggregate(N-1) of the optimization layer of the N-1 level is obtained, the encoding feature Feature Encoding_N output by the encoding layer of the N level and the decoding feature Feature Decoding_N of the N level are input into the optimization layer of the N level to obtain the optimization result Feature Aggregate_N of the optimization layer of the N level; the optimization results of the optimization layers of each level (Feature Aggregate_1, Feature Aggregate_2…Feature Aggregate_N) performs feature fusion to obtain the optimized result Feature Aggregate.

[0252] Exemplarily, the image coding features Feature Encoding (image coding features_1, image coding features_2, ... image coding features_N) output by each encoding layer and the image decoding features Feature Decoding (image decoding features_1, image decoding features_2, ... image decoding features_N) output by each decoding layer are respectively input into the tuning layer of the corresponding level (tuning layer_1, tuning layer_2 ... tuning layer_N), and the tuning results of multiple tuning layers are feature fused to obtain the tuning results Features Aggregate. Based on the tuning results, the image-enhanced 4K target image is obtained.

[0253] In the embodiment of the present disclosure, a plurality of tuning layers are connected to the encoding layer and decoding layer of the corresponding levels through a cascade architecture, and feature tuning is completed layer by layer through the plurality of tuning layers. The output features are gradually optimized at multiple levels to obtain more accurate task processing results.

[0254] In an optional embodiment of the present disclosure, the encoding layer includes at least two sub-encoding layers for encoding tasks, the decoding layer includes at least two sub-decoding layers for decoding tasks, the sub-encoding layers are connected layer by layer, and the sub-decoding layers are connected layer by layer, and the multiple tuning layers are respectively connected to the sub-encoding layers of the target encoding tasks and the sub-decoding layers of the target decoding tasks at the corresponding levels;

[0255] Correspondingly, the task data is input into the target visual model corresponding to the target visual generation task, and the encoding features output by each encoding layer and the decoding features output by each decoding layer in the target visual model are obtained, including the following specific steps:

[0256] The task data is input into the target visual model corresponding to the target visual generation task, and encoded layer by layer according to the hierarchical order of each sub-coding layer until the encoding features output by the sub-coding layer of the target encoding task are obtained, and decoded layer by layer according to the hierarchical order of each sub-decoding layer until the decoding features output by the sub-decoding layer of the target encoding task are obtained.

[0257] Exemplarily, a 1080P source image is input into the image enhancement task model, and encoding is performed layer by layer according to the hierarchical order of each sub-coding layer until the encoding features Feature Encoding (image encoding features_1, image encoding features_2, ... image encoding features_N) output by the sub-coding layer of the target encoding task are obtained, and decoding is performed layer by layer according to the hierarchical order of each sub-decoding layer until the decoding features Feature Decoding (image decoding features_1, image decoding features_2, ... image decoding features_N) output by the sub-decoding layer of the target encoding task are obtained.

[0258] In the embodiment of the present disclosure, the encoding and decoding capabilities of different encoding and decoding tasks are integrated through a stacking structure, the accuracy of each encoding feature and the accuracy of each encoding feature are improved, and more accurate encoding and decoding features are provided for subsequent feature optimization.

[0259] In an optional embodiment of the present disclosure, after step 1006, the following specific steps are further included:

[0260] Annotate the visual key points in the target visual data, where the visual key points are visual points that represent the task information of the target visual generation task;

[0261] Feedback target visual data to front-end users.

[0262] Visual keypoints are visual feature points in the target visual data that represent the task information for the target visual generation task. These feature points play a key role in describing the structure, shape, and motion of the target object or representing task-related information. Visual keypoints are an important form of high-level abstraction and understanding of visual data. They can be salient locations extracted through model calculations or manually defined specific regions that are critical to the task. After the visual generation task is completed, annotated visual keypoints can help users better understand the key information of the model generation results and provide a basis for subsequent analysis, editing, or application. For example, in a facial expression recognition task, visual keypoints may include the position and shape of parts such as the eyes, eyebrows, and corners of the mouth; in human pose estimation, visual keypoints correspond to the position coordinates of various human joints such as the shoulder, elbow, wrist, and knee; for image segmentation tasks, visual keypoints may be a collection of pixels representing the boundaries or internal features of different objects. After generating an image, annotating these keypoints can clearly demonstrate how the model captures and reproduces the important visual information in the input description or source image.

[0263] The visual key points in the target visual data are labeled based on the visual semantic understanding ability of the visual model.

[0264] For example, visual key points such as eyes, eyebrows, and corners of the mouth in the 4K target image after image enhancement are annotated, and the annotated 4K target image is fed back to the front-end user.

[0265] In the disclosed embodiment, by providing annotation feedback for visual key points, a basis is provided for subsequent analysis, editing or application.

[0266] In an optional embodiment of the present disclosure, after feeding back the target visual data to the front-end user, the following specific steps are further included:

[0267] Receiving key point feedback information sent by the front-end user for the visual key points in the target visual data, wherein the key point feedback information is the feedback information provided by the front-end user on the visual key points according to the task information;

[0268] Input key point feedback information, task data, and target visual data into the target visual model to obtain the adjusted feature data output by each feature processing layer;

[0269] The adjusted feature data output by each feature processing layer are respectively input into the tuning layers connected to each feature processing layer for tuning processing, and based on the tuning results of each tuning layer, the adjusted target visual data are generated.

[0270] Key point feedback information is the information returned by the front-end user after receiving the target visual data and evaluating, correcting or adjusting the visual key points marked therein according to the task information. These feedbacks can be manual modifications of key point positions and weights by the user, or additional key point annotations provided. Key point feedback information is interactive user feedback on the accuracy of visual key points in the target visual data generated by the model. Through this feedback mechanism, the system can obtain the user's judgment on whether the visual feature points meet expectations or requirements, and optimize the model output results accordingly. For example, in a facial expression recognition application, the user may find that the eye position marked by the model is inaccurate, so he manually adjusts the eye position coordinates and submits them to the system. This series of adjustment operations constitutes the key point feedback information.

[0271] Adjusting the feature data involves re-inputting the task data, target visual data, and key point feedback information into the target visual model after receiving the key point feedback information, and then processing the updated feature representations obtained through each feature processing layer. Based on the user's key point feedback information, the model will re-analyze and calculate the original task data to generate a new feature data set that reflects the user's feedback. These adjusted feature data contain visual semantic information after user intervention. For example, suppose that in an image segmentation task, the user is dissatisfied with the preliminary segmentation results and points out that some pixels are incorrectly marked as background, but should actually belong to the foreground object. The system integrates this feedback information into the feature processing process and recalculates the adjusted feature data that better meets the user's expectations.

[0272] The adjusted target visual data is the new visual generation task processing result generated by further tuning the adjusted feature data through the tuning layer based on the key point feedback information provided by the user. By further tuning and reconstructing the adjusted feature data, the system can produce a more accurate final visual data product that meets user needs. This not only reflects the human-computer interaction of the model, but also improves the model's adaptability and flexibility. For example, the system adjusts based on the user's feedback on eye position, and after tuning the adjusted feature data, it generates a 4K enhanced image with more accurate eye details as the adjusted target visual data.

[0273] For example, the system receives key point feedback information (e.g., "the eye color is incorrect, the eyebrow density is insufficient, and the corners of the mouth are reversed") sent by the front-end user for the annotated 4K target image. The key point feedback information, task data, and target visual data are input into the image enhancement task model to obtain the adjusted image encoding features Feature Encoding (image encoding feature_1', image encoding feature_2', ... image encoding feature_N') and adjusted image decoding features Feature Decoding (image decoding feature_1', image decoding feature_2', ... image decoding feature_N') output by each feature processing layer. The adjusted feature data output by each feature processing layer are respectively input into the optimization layers connected to each feature processing layer for optimization processing. Based on the optimization results Feature Aggregate' of each optimization layer, the adjusted 4K target image is generated.

[0274] In the disclosed embodiment, the target visual data generated by the model is adjusted through an interactive feedback mechanism, thereby improving the visual generation effect and user experience.

[0275] In an optional embodiment of the present disclosure, after generating adjusted target visual data based on the tuning results of each tuning layer, the following specific steps are further included:

[0276] Based on the key point feedback information and the adjusted target visual data, the visual generation result is integrated;

[0277] Feedback visual generation results to front-end users.

[0278] The visual generation result is the final output obtained after being processed by the target visual model, including feature extraction, tuning, and possible user feedback and adjustments. It is the new visual content generated after a series of complex calculations and optimizations on the original input data or task description. This result can be in the form of images, videos, audio, etc., and fully reflects the requirements of the target visual generation task, such as enhancing image quality, generating new visual content, and achieving specific image classification or entity recognition. The visual generation result is the terminal product of the entire visual task processing flow. It integrates the key feature information extracted from the source data, performs multi-level refined processing and optimization through the tuning layer, and makes targeted adjustments based on user feedback (if any).

[0279] For example, in an image enhancement task, based on a 1080P source image uploaded by a user, after being processed by multiple encoding, decoding, and tuning layers of the image enhancement task model, a high-quality enhanced image with 4K resolution is generated, in which key features such as eyes, eyebrows, and corners of the mouth are accurately enhanced. After receiving user feedback on the key points of the 4K target image and making corresponding adjustments, the generated adjusted 4K target image is a further visual generation result. In addition, after the user receives the visual generation result, he or she can edit it, such as adding a text description. The system can also automatically generate a recommended text description based on the image content, and display the text and the image together to the user. At this time, the combination of text and pictures constitutes the final complete visual generation result.

[0280] In the disclosed embodiment, an interactive feedback mechanism is used to ensure that the output results not only meet the preset task objectives but also meet the user's personalized needs, thereby improving the visual generation effect and user experience.

[0281] In an optional embodiment of the present disclosure, after step 1006, the following specific steps are further included:

[0282] Feedback target visual data to front-end users;

[0283] Receiving visual feedback information sent by a front-end user, wherein the visual feedback information is information provided by the front-end user regarding target visual data;

[0284] Construct sample task data based on visual feedback information;

[0285] The target vision model is trained based on sample task data.

[0286] Visual feedback information refers to the feedback information provided by users on the target visual data after the target visual data is generated and displayed to the front-end user, based on their subjective evaluation and requirements of the content, quality or task completion of the target visual data. This feedback information can be correction instructions on the position, color, shape and other aspects of visual key points, or it can be a satisfaction evaluation of the overall visual effect or other detailed descriptions that help improve the model output. Visual feedback information is an important part of realizing human-computer interaction. It can reflect the user's true feelings and expectations about the processing results of the target visual generation task. By collecting and utilizing this feedback information, the system can continuously optimize the performance of its visual generation model to more accurately meet the actual needs of users and improve the quality and accuracy of the final visual generation results. For example, in an image enhancement application, after receiving the 4K enhanced image generated by the system, the user found that the color of the eyes in the image did not match the original image, so he provided visual feedback information that "the eye color is wrong."

[0287] In the embodiment of the present disclosure, the sample task data is updated through an interactive feedback mechanism, and the target vision model is further trained based on the sample task data, thereby improving the model performance of the target vision model.

[0288] In an optional embodiment of the present disclosure, before step 1004, the following specific steps are further included:

[0289] Obtain target task samples for the target visual generation task;

[0290] Input the target task sample into the initial visual model to obtain the target task features output by each feature processing layer in the initial visual model, wherein the initial visual model is a visual model pre-trained based on sample data of a general visual generation task;

[0291] The target task features output by each feature processing layer are input into the tuning layer connected to each feature processing layer for tuning processing, and the target task results are obtained based on the predicted output features of each tuning layer;

[0292] Based on the target task results, each tuning layer is trained to obtain the target visual model corresponding to the target visual generation task.

[0293] Target visual generation tasks are specific tasks that require further fine-tuning or optimization of pre-trained models for specific visual generation application scenarios. These tasks typically have clear goals and application scenarios, requiring adjustments to the pre-trained models to adapt to the new data distribution and task requirements. Targeted visual generation tasks are a process of fine-tuning the general capabilities of pre-trained models, improving the model's performance in specific scenarios through targeted training and tuning modules. For example, a pre-trained diffusion vision model can have multiple downstream target visual tasks, such as image super-resolution, style transfer, image colorization, and image-to-image translation.

[0294] Target task samples for target generation vision tasks are data used to train and evaluate vision models for these tasks. Typically, these samples have corresponding labeled outputs. These samples provide the model with a framework for learning how to generate visual content suitable for the target generation vision task from given conditions and guide the learning and optimization of model parameters.

[0295] In the visual model system of the disclosed embodiment, corresponding tuning modules are trained for multiple downstream target visual tasks. During the visual data generation process, the feature processing module can be shared to perform multi-task shared tuning, thereby improving the efficiency of visual data generation. During the training process of the visual model, only the tuning module can be trained for different downstream visual generation tasks without training the feature processing module, thereby reducing training overhead and improving training efficiency.

[0296] Referring to FIG11 , FIG11 shows a flow chart of an image generation method provided by an embodiment of the present disclosure, which includes the following specific steps:

[0297] Step 1102: Obtain task data of the image generation task.

[0298] Step 1104: Input the task data into the image generation model to obtain feature data output by each feature processing layer in the image generation model, wherein the image generation model includes at least two feature processing modules and at least one tuning module, the feature processing module includes multiple feature processing layers, the tuning module includes multiple tuning layers, and the multiple tuning layers are respectively connected to the feature processing layers of corresponding levels in the at least two feature processing modules.

[0299] Step 1106: The feature data outputted by each feature processing layer is respectively inputted into the optimization layer connected to each feature processing layer for optimization processing, and a target image is generated based on the optimization results of each optimization layer.

[0300] The disclosed embodiments are applicable to applications, websites, or mini-programs that have image generation capabilities. The image generation functionality is implemented on the application, website, or mini-program. For example, a website that deploys an image generation model can implement functions such as text-to-image, image-to-image, and image-to-text. For another example, a third-party application can implement the corresponding image generation functionality by calling the deployed image generation model through an application programming interface (API).

[0301] The embodiments of the present disclosure and the embodiments of the specification shown in FIG. 1 , FIG. 9 and FIG. 10 are based on the same inventive concept. For the specific methods of steps 1102 to 1106 , please refer to the embodiments of the specification shown in FIG. 1 and will not be described again here.

[0302] In the embodiment of the present disclosure, in the image generation model, multiple tuning layers are used to connect with the encoding layer and decoding layer of the corresponding levels respectively, so as to obtain multiple complete encoding features and decoding features, retain the detailed information of multiple levels, complete feature tuning at the corresponding levels through multiple tuning layers, optimize the output features, obtain a more accurate target image, and improve the accuracy of image generation. Moreover, based on the sample data of the image generation task, only the multiple tuning layers in the pre-trained image model are trained, avoiding the training of other model layers, thereby obtaining a deep learning model suitable for the image generation task, reducing the training overhead and improving the training efficiency.

[0303] Referring to FIG12 , FIG12 shows a flow chart of a video generation method provided by an embodiment of the present disclosure, including the following specific steps:

[0304] Step 1202: Obtain task data for the video generation task.

[0305] Step 1204: Input the task data into the video generation model to obtain feature data output by each feature processing layer in the video generation model, wherein the target video model includes at least two feature processing modules and at least one tuning module, the feature processing module includes multiple feature processing layers, the tuning module includes multiple tuning layers, and the multiple tuning layers are respectively connected to the feature processing layers of corresponding levels in the at least two feature processing modules.

[0306] Step 1206: The feature data output by each feature processing layer is respectively input into the optimization layer connected to each feature processing layer for optimization processing, and the target video is generated based on the optimization results of each optimization layer.

[0307] The disclosed embodiments are applicable to applications, websites, or mini-programs with video generation capabilities. The video generation functionality is implemented on the application, website, or mini-program. For example, a website that deploys a video generation model can implement functions such as text-to-video, video-to-video, and video-to-text. For another example, a third-party application can implement the corresponding video generation functionality by calling the deployed video generation model through an application programming interface.

[0308] The embodiments of the present disclosure and the embodiments of the descriptions of Figures 1, 9, 10 and 11 are based on the same inventive concept. For the specific methods of steps 1202 to 1206, please refer to the embodiments of the descriptions above and will not be repeated here.

[0309] For example, on a platform deployed with a video generation model, a user selects the text-based video generation feature through the front-end interface and enters a detailed text script describing the HD 4K dynamic scene they wish to generate. In addition, the user uploads a reference static 1080P image, which represents the primary visual elements and style of the video. The system integrates the user's text description and the reference image as data sources for the video generation task, forming complete task data. This task data (including the text description and the reference image) is then fed into an advanced video generation model. The model architecture consists of at least two feature processing modules, each responsible for parsing and understanding the input data from different dimensions: one module focuses on semantically encoding the textual information, extracting features such as key actions, scene layout, and emotional overtones; the other module performs deep learning analysis on the reference image, extracting visual features such as spatial structure, color distribution, and texture details. These feature processing modules internally contain multiple layers of feature processing networks that gradually abstract and transform the raw input into high-level feature representations. The model also includes one or more tuning modules, consisting of multiple, sequentially connected tuning layers. These modules are connected to corresponding layers in the feature processing module to fine-tune the quality of the generated video. When the model runs, each feature processing layer outputs corresponding feature data, such as spatiotemporal sequence features from text vectorization and dynamic frame features from image feature mapping. The system passes the feature data generated by each feature processing layer to the corresponding tuning layer for in-depth optimization. For example, a motion smoothness tuning layer on the timeline may improve motion continuity, while a visual quality tuning layer may enhance image detail and color fidelity. After iterative optimization through a series of tuning layers, the final result, Features Aggregate, is obtained, which contains a set of enhanced spatiotemporal features. Based on this highly optimized feature set, the video generation model performs a reverse decoding process to gradually construct a high-quality video frame sequence that matches the style of the original text description and the reference image. Finally, the system combines these frames into a complete 4K resolution target video.

[0310] In the embodiment of the present disclosure, in the video generation model, multiple tuning layers are used to connect with the encoding layer and decoding layer of the corresponding levels respectively, so as to obtain multiple complete encoding features and decoding features, retain the detailed information of multiple levels, complete feature tuning at the corresponding levels through multiple tuning layers, optimize the output features, obtain a more accurate target video, and improve the accuracy of video generation. Moreover, based on the sample data of the video generation task, only the multiple tuning layers in the pre-trained video model are trained, avoiding the training of other model layers, thereby obtaining a deep learning model suitable for the video generation task, reducing the training overhead and improving the training efficiency.

[0311] Referring to FIG. 13 , FIG. 13 shows a flowchart of an information processing method based on a visual model provided by an embodiment of the present disclosure, which is applied to a cloud training platform and includes the following specific steps:

[0312] Step 1302: Receive a task generation request sent by a terminal device, wherein the task generation request includes request information.

[0313] Step 1304: Based on the request information, obtain the target visual model, wherein the target visual model includes at least two feature processing modules and at least one tuning module, the feature processing module includes multiple feature processing layers, the tuning module includes multiple tuning layers, the multiple tuning layers are respectively connected to the feature processing layers of corresponding levels in the at least two feature processing modules, the tuning layer is used to perform tuning processing on the feature data output by the connected feature processing layers to obtain a tuning result, and the tuning result is used to generate visual data.

[0314] Step 1306: Generate task information based on the target visual model, where the task information is used for the terminal device to perform the target visual task.

[0315] The cloud training platform is a cloud computing infrastructure that provides large-scale data processing and high-performance computing resources for training, optimizing, and deploying various machine learning models, particularly deep learning models. On the cloud training platform, users can upload data, select or customize algorithm models, and efficiently train and validate these models through distributed computing. The cloud training platform receives task generation requests from terminal devices, retrieves the corresponding pre-trained visual model or target visual model based on the request, and generates task information based on this model. The cloud training platform can quickly respond to the needs of different visual generation tasks, calling appropriate models for feature processing and tuning, ultimately generating high-quality target visual data.

[0316] A task generation request is a task instruction request for a visual generation task sent by a terminal device to the cloud training platform, requesting the execution of the target visual generation task. This request usually contains request information about the data to be processed, the task type, the expected output format, and other relevant parameters. For example, when a user selects the "Image Enhancement" function on the front-end interface of the cloud platform website and uploads an image, the application constructs a task generation request, which contains parameters such as image data, task type (i.e., image enhancement), and possibly the degree of enhancement, and sends it to the cloud training platform.

[0317] The request information is the target visual generation task parameters and description information carried in the task generation request, which is used to guide the cloud training platform on how to correctly identify and execute the requested target visual generation task. The request information includes but is not limited to: the task scenario identifier of the target visual task, the task model identifier, or the sample data of the target visual task. For example, in an image translation task, the request information may include various parameters such as the target language and whether the original layout style needs to be maintained; in a video classification task, the request information may include the video file address, a list of classification labels, and the model version number.

[0318] The task information is generated by the cloud training platform after parsing the received task generation request. It contains all the specific operation instructions and configuration items required to perform the target visual generation task, so that the terminal device or other server-side components can correctly use the target visual model to process the data. Task information usually includes the determined target visual model and its parameter configuration, the processing method of the input data, the specifications of the expected output results, as well as the intermediate steps and other auxiliary information that may be involved. For example, in an image super-resolution task, the task information may include details such as the selected super-resolution model address, the storage location of the input image, the target path for outputting 4K resolution images, and other environmental configurations required for the model to run. This information enables the terminal device to correctly load the model and execute the task on a local or remote server.

[0319] For example, the user selects the "image super-resolution" function on the front-end interface of the cloud training platform and uploads the source image file. The cloud training platform converts the user's operation into a task generation request, which includes: request type: image super-resolution; source image data: binary data or storage address of a low-resolution image; target resolution: 4K; optional parameters: whether to retain the original color style, model version number and other task information. After receiving this task generation request, the cloud training platform identifies a suitable pre-trained or optimized visual model based on the "image super-resolution" task type and possible other parameters (such as the model version number) in the request. The cloud training platform obtains a pre-trained super-resolution model from the model library. The model has multiple feature processing modules and a tuning module structure, which can perform multi-level feature extraction and tuning processing on the input low-resolution image. Based on the selected super-resolution model, the cloud training platform generates detailed task information, including: target visual model address (pointing to the super-resolution model file location stored in the cloud); input data processing method (how to read the source image data from the request and format it into the model's required input format); output specifications (expected output image resolution: 4K); intermediate steps and auxiliary information (which may include intermediate cache paths and hardware resource requirements for execution (NPU / TPU usage, etc.). The cloud training platform packages this task information and sends it to the end device. The application can follow these instructions to call the cloud training platform's service locally or remotely via an API interface to load the model and perform super-resolution processing on the source image. After receiving the task information, the end device loads the model according to the instructions and runs it locally or via an API call to the cloud service to process the source image data. After the model completes the computation, it returns a 4K resolution enhanced image to the end device. The end device displays the processed high-resolution image for the user to view or download.

[0320] In the disclosed embodiment, a cloud training platform integrated with cloud computing technology is utilized to generate task information for terminal devices to perform target visual tasks. This can reduce system deployment and operation and maintenance costs while ensuring the quality and efficiency of visual generation task processing, thereby providing users with convenient and efficient visual content generation services.

[0321] In an optional embodiment of the present disclosure, the request information includes a task scenario identifier of the target visual task, or a task model identifier;

[0322] Correspondingly, step 1304 includes the following specific steps:

[0323] Based on the task scene identifier, a target scene template is determined from a plurality of preset scene templates, and based on the target scene template, a target visual model is searched from a model library, wherein the model library stores a plurality of visual models;

[0324] or,

[0325] Based on the task model identifier, the target visual model is searched from the model library.

[0326] The task scenario identifier is a unique or specific label used to distinguish different visual generation application scenarios. It represents a certain type of specific visual processing task environment or requirement. In this embodiment, the task scenario identifier is part of the request information. The cloud training platform selects the appropriate model and configuration from a series of preset scene templates based on the identifier to perform the target visual task. For example, the task scenario identifier is "product image enhancement", which means that the terminal device hopes to perform targeted image quality improvement processing on the uploaded product images. For another example, if the user needs to perform diagnostic auxiliary analysis on medical images, the task scenario identifier is "medical image diagnosis".

[0327] Preset scene templates are predefined standard configuration scene templates for different vision task application scenarios. Each template contains information such as the target vision model, parameter settings, and data processing flow that matches the scenario. The cloud training platform stores a series of preset scene templates to quickly respond to different types of task generation requests. Different templates correspond to different vision task types and processing requirements, ensuring that the platform can automatically obtain the model and configuration that best suits the current task needs based on the task scenario identifier. For example, one of the preset scene templates may be specifically for face beautification processing, which includes a pre-trained facial feature recognition model and beautification algorithm.

[0328] The target scene template is a specific scene template identified by the identifier. It is a corresponding template selected from multiple preset scene templates. When parsing a task generation request, the cloud training platform locates the corresponding target scene template based on the task scene identifier and selects the corresponding visual model and other related configuration information from the model library based on this template. For example, if the task scene identifier is "Intelligent Security Surveillance Video Analysis," the target scene template is a template containing pre-trained models and related configuration parameters for modules such as object detection and behavior analysis, suitable for security surveillance scenarios.

[0329] The model library is a centralized repository for deep learning models that have been trained and optimized to solve different visual generation tasks. On the cloud training platform, the model library stores a large number of pre-trained visual models, including but not limited to models for various task types such as image classification, image super-resolution, image translation, and video analysis. Each model may include feature processing modules and tuning modules to meet complex and diverse visual processing needs. For example, the model library may contain multiple versions of image super-resolution models, such as different optimized models for facial images, landscape images, or document scans. The required model can be accurately extracted from the library based on the task scenario identifier or task model identifier in the task request.

[0330] For example, suppose a user selects the "Product Image Enhancement" function through the front-end interface of an e-commerce application. The application then sends a task generation request to the cloud training platform. The request contains the task scenario identifier "Product Image Enhancement." Upon receiving this task generation request, the cloud training platform identifies the task scenario identifier as "Product Image Enhancement." Based on a series of preset scene templates, the cloud training platform finds a target scene template that matches "Product Image Enhancement." This target scene template is pre-configured with specific visual models and parameter settings suitable for product image enhancement, such as a feature extraction module, color correction module, and texture enhancement module optimized for product characteristics. Based on the target scene template, the cloud training platform retrieves the corresponding pre-trained visual model from the model library and loads the relevant parameters and configuration files. Based on the information in the target scene template, the cloud training platform generates task information, including but not limited to details such as the model address, input data processing method, and output result specifications, so that the terminal device can correctly call and execute the product image enhancement task.

[0331] For example, imagine a user submitting a task generation request in the backend of an intelligent medical image diagnosis system. The request explicitly specifies the task model identifier, "DeepMedic v3.0," a deep learning model version specifically designed for medical image analysis. When the cloud training platform receives the request, it parses the request and identifies the task model identifier as "DeepMedic v3.0." Based on this model identifier, the cloud training platform directly searches the model library for the corresponding "DeepMedic v3.0" medical image analysis model. After finding the correct model, the cloud training platform generates the corresponding task information based on the model's characteristics and requirements, such as the location of the model file, input data format conversion rules, and the expected output diagnostic report structure. The terminal device or server component downloads the model based on the task information and uses it to perform feature extraction, lesion detection, and other processing on the uploaded medical images, ultimately generating accurate medical image diagnosis results.

[0332] In the disclosed embodiments, a flexible, efficient and standardized visual task processing mechanism is implemented by integrating cloud computing technology with predefined task scenario templates and model library resources.

[0333] In an optional embodiment of the present disclosure, the request information includes sample data of the target visual task;

[0334] Correspondingly, step 1304 includes the following specific steps:

[0335] Based on the sample data, the initial visual model corresponding to the target visual task is trained to obtain a trained target visual model.

[0336] In addition to selecting a pre-trained target visual model from the model library, you can also train a target visual model that meets the needs of the end user based on the sample data provided by the end user.

[0337] In the embodiment of this description, a flexible, efficient and accurate visual task processing mechanism is achieved by integrating cloud computing technology with pre-defined task scenario templates and model library resources.

[0338] Corresponding to the above method embodiments, the present disclosure further provides a cloud training platform embodiment. FIG14 shows a schematic structural diagram of a cloud training platform provided by one embodiment of the present disclosure. The cloud training platform 1400 includes a request interface 1410 and a response unit 1420.

[0339] The request interface 1410 is configured to receive a task generation request sent by a terminal device, wherein the task generation request includes request information;

[0340] Responding unit 1420 is used to obtain a target visual model based on the request information, wherein the target visual model includes at least two feature processing modules and at least one tuning module, the feature processing module includes multiple feature processing layers, the tuning module includes multiple tuning layers, the multiple tuning layers are respectively connected to the feature processing layers of corresponding levels in at least two feature processing modules, the tuning layer is used to perform tuning processing on the feature data output by the connected feature processing layers to obtain a tuning result, and the tuning result is used to generate visual data; based on the target visual model, task information is generated, wherein the task information is used for the terminal device to perform the target visual task.

[0341] In an optional embodiment of the present disclosure, the cloud training platform 1400 further includes a model library, wherein the model library stores a plurality of visual models; the request information includes a task scene identifier of the target visual task, or a task model identifier;

[0342] The response unit 1420 is specifically used to determine the target scene template from multiple preset scene templates based on the task scene identifier, and search for the target visual model from the model library based on the target scene template; or search for the target visual model from the model library based on the task model identifier.

[0343] In the disclosed embodiment, the cloud training platform generates task information for terminal devices to perform target visual tasks, which can reduce system deployment and operation and maintenance costs while ensuring the quality and efficiency of visual generation task processing, and provide users with convenient and efficient visual content generation services.

[0344] Corresponding to the above method embodiment, the present disclosure also provides an embodiment of a visual model construction device. FIG15 shows a schematic structural diagram of a visual model construction device provided by an embodiment of the present disclosure. As shown in FIG15 , the device includes:

[0345] Parameter acquisition module 1502, configured to acquire target model parameters;

[0346] The module construction module 1504 is configured to construct at least two feature processing modules and at least one tuning module based on the target model parameters, wherein the feature processing module includes multiple feature processing layers, and the tuning module includes multiple tuning layers. The tuning layers are used to perform tuning processing on the input feature data to obtain tuning results, and the tuning results are used to generate visual data;

[0347] The connection module 1506 is configured to connect the multiple tuning layers to the feature processing layers of corresponding levels in at least two feature processing modules to obtain a visual model.

[0348] Optionally, the parameter acquisition module 1502 is further configured to:

[0349] Obtaining model parameters and tuning parameters of a pre-trained visual model corresponding to a visual generation task, wherein the pre-trained visual model includes at least two feature processing modules;

[0350] Correspondingly, the module construction module 1504 is further configured to:

[0351] Based on the pre-trained visual model, at least two feature processing modules are constructed, and the initial parameters of the at least two feature processing modules are set according to the model parameters of the pre-trained visual model; based on the tuning parameters, at least one tuning module is constructed, and the parameters of the tuning module are initialized.

[0352] Optionally, the device further comprises:

[0353] The training module is configured to train the visual model based on sample data of the visual generation task to obtain a trained visual model; tune the trained visual model based on target task samples of multiple downstream target visual tasks to obtain target tuning modules corresponding to multiple downstream target visual tasks; and connect each target tuning module to at least two feature processing modules to obtain an updated visual model.

[0354] In the disclosed embodiment, by connecting multiple tuning layers of the tuning module with feature processing layers of corresponding levels in at least two feature processing modules, complete feature data output by multiple feature processing layers is obtained, detailed information on multiple levels is retained, tuning processing on corresponding levels is implemented in multiple tuning layers, the output feature data is optimized, and more accurate visual data can be generated. In addition, at least two feature processing modules and at least one tuning module in the visual model are decoupled, and the tuning module can be flexibly selected during the visual data generation process, thereby improving the versatility of the visual model. In the visual data generation process, the feature processing module can be shared to perform multi-task shared tuning, thereby improving the efficiency of visual data generation. In the training process of the visual model, only the tuning module can be trained for different downstream visual generation tasks, without training the feature processing module, thereby reducing training overhead and improving training efficiency.

[0355] The above is a schematic scheme of a visual model construction device of this embodiment. It should be noted that the technical scheme of the visual model construction device and the technical scheme of the above-mentioned visual model construction method belong to the same concept. For details not described in detail in the technical scheme of the visual model construction device, please refer to the description of the technical scheme of the above-mentioned visual model construction method.

[0356] Corresponding to the above method embodiments, the present disclosure also provides an embodiment of a visual task processing device. FIG16 shows a schematic structural diagram of a visual task processing device provided by an embodiment of the present disclosure. As shown in FIG16 , the device includes:

[0357] A first acquisition module 1602 is configured to acquire task data of a target visual generation task;

[0358] The first processing module 1604 is configured to input the task data into a target vision model corresponding to the target vision generation task, and obtain feature data output by each feature processing layer in the target vision model, wherein the target vision model includes at least two feature processing modules and at least one tuning module, the feature processing module includes multiple feature processing layers, the tuning module includes multiple tuning layers, and the multiple tuning layers are respectively connected to the feature processing layers of corresponding levels in the at least two feature processing modules;

[0359] The first tuning module 1606 is configured to input the feature data output by each feature processing layer into the tuning layer connected to each feature processing layer for tuning processing, and generate target visual data based on the tuning results of each tuning layer.

[0360] Optionally, the at least two feature processing modules include an encoding module and a decoding module, the encoding module includes multiple encoding layers, the decoding module includes multiple decoding layers, and the multiple tuning layers are respectively connected to the encoding layers and decoding layers of corresponding levels;

[0361] Correspondingly, the first processing module 1604 is further configured to:

[0362] Input the task data into the target visual model corresponding to the target visual generation task, and obtain the encoding features output by each encoding layer and the decoding features output by each decoding layer in the target visual model;

[0363] Correspondingly, the first tuning module 1606 is further configured to:

[0364] The encoding features output by each encoding layer and the decoding features output by each decoding layer are input into the connected tuning layer for tuning processing, and the target visual data is generated based on the tuning results of each tuning layer.

[0365] Optionally, the encoding layer includes at least two sub-encoding layers for encoding tasks, the decoding layer includes at least two sub-decoding layers for decoding tasks, the sub-encoding layers are connected layer by layer, and the sub-decoding layers are connected layer by layer, and the multiple tuning layers are respectively connected to the sub-encoding layers of the target encoding tasks and the sub-decoding layers of the target decoding tasks at the corresponding levels;

[0366] Correspondingly, the first processing module 1604 is further configured to:

[0367] The task data is input into the target visual model corresponding to the target visual generation task, and encoded layer by layer according to the hierarchical order of each sub-coding layer until the encoding features output by the sub-coding layer of the target encoding task are obtained, and decoded layer by layer according to the hierarchical order of each sub-decoding layer until the decoding features output by the sub-decoding layer of the target encoding task are obtained.

[0368] Optionally, multiple tuning layers are connected layer by layer;

[0369] Correspondingly, the first tuning module 1606 is further configured to:

[0370] The encoding features output by each encoding layer and the decoding features output by each decoding layer are input into the connected tuning layer for tuning processing, and the tuning result of the previous tuning layer is input into the next tuning layer in the hierarchical order of multiple tuning layers to obtain the output features of the last tuning layer; based on the output features of the last tuning layer, the target visual data is generated.

[0371] Optionally, the device further comprises:

[0372] The first feedback module is configured to annotate visual key points in the target visual data, wherein the visual key points are visual points that represent task information of the target visual generation task; and feed back the target visual data to the front-end user.

[0373] Optionally, the device further comprises:

[0374] The first adjustment module is configured to receive key point feedback information sent by the front-end user for the visual key points in the target visual data, wherein the key point feedback information is the information provided by the front-end user for the visual key points based on the task information; input the key point feedback information, task data and target visual data into the target visual model to obtain the adjusted feature data output by each feature processing layer; input the adjusted feature data output by each feature processing layer into the tuning layer connected to each feature processing layer for tuning processing, and generate the adjusted target visual data based on the tuning results of each tuning layer.

[0375] Optionally, the device further comprises:

[0376] The first integration module is configured to integrate the key point feedback information and the adjusted target visual data to obtain a visual generation result; and feed back the visual generation result to the front-end user.

[0377] Optionally, the device further comprises:

[0378] The first feedback training module is configured to feed back target visual data to a front-end user; receive visual feedback information sent by the front-end user, wherein the visual feedback information is information provided by the front-end user on the target visual data; construct sample task data based on the visual feedback information; and train the target visual model based on the sample task data.

[0379] Optionally, the device further comprises:

[0380] The first training module is configured to obtain target task samples of a target visual generation task; input the target task samples into an initial visual model to obtain target task features output by each feature processing layer in the initial visual model, wherein the initial visual model is a visual model pre-trained based on sample data of a general visual generation task; input the target task features output by each feature processing layer into the tuning layers connected to each feature processing layer for tuning processing, and obtain the target task results based on the predicted output features of each tuning layer; based on the target task results, train each tuning layer to obtain a target visual model corresponding to the target visual generation task.

[0381] In the disclosed embodiment, by connecting multiple tuning layers of the tuning module with feature processing layers of corresponding levels in at least two feature processing modules, complete feature data output by multiple feature processing layers is obtained, detailed information on multiple levels is retained, tuning processing on corresponding levels is implemented in multiple tuning layers, the output feature data is optimized, and more accurate visual data can be generated. In addition, at least two feature processing modules and at least one tuning module in the visual model are decoupled, and the tuning module can be flexibly selected during the visual data generation process, thereby improving the versatility of the visual model. In the visual data generation process, the feature processing module can be shared to perform multi-task shared tuning, thereby improving the efficiency of visual data generation. In the training process of the visual model, only the tuning module can be trained for different downstream visual generation tasks, without training the feature processing module, thereby reducing training overhead and improving training efficiency.

[0382] The above is a schematic diagram of a visual task processing device according to this embodiment. It should be noted that the technical solution of this visual task processing device and the technical solution of the visual task processing method described above are based on the same concept. For details not described in detail in the technical solution of the visual task processing device, please refer to the description of the technical solution of the visual task processing method described above.

[0383] Corresponding to the above method embodiment, the present disclosure also provides an embodiment of an image generation device. FIG17 shows a schematic structural diagram of an image generation device provided by an embodiment of the present disclosure. As shown in FIG17 , the device includes:

[0384] The second acquisition module 1702 is configured to acquire task data of the image generation task;

[0385] A second processing module 1704 is configured to input task data into an image generation model and obtain feature data output by each feature processing layer in the image generation model, wherein the image generation model includes at least two feature processing modules and at least one tuning module, the feature processing module includes multiple feature processing layers, the tuning module includes multiple tuning layers, and the multiple tuning layers are respectively connected to the feature processing layers of corresponding levels in the at least two feature processing modules;

[0386] The second optimization module 1706 is configured to input the feature data output by each feature processing layer into the optimization layer connected to each feature processing layer for optimization processing, and generate a target image based on the optimization results of each optimization layer.

[0387] In the embodiment of the present disclosure, by connecting multiple tuning layers of the tuning module with feature processing layers of corresponding levels in at least two feature processing modules, complete feature data output by multiple feature processing layers is obtained, detailed information on multiple levels is retained, tuning processing on corresponding levels is implemented in multiple tuning layers, the output feature data is optimized, and a more accurate target image can be generated. In addition, at least two feature processing modules and at least one tuning module in the image generation model are decoupled, and the tuning module can be flexibly selected during the image generation process, thereby improving the versatility of the image generation model. The feature processing module can also be shared during the image generation process to perform multi-task shared tuning, thereby improving the efficiency of image generation. In the training process of the image generation model, only the tuning module can be trained for different downstream image generation tasks, without the need to train the feature processing module, thereby reducing training overhead and improving training efficiency.

[0388] The above is a schematic diagram of an image generation device according to this embodiment. It should be noted that the technical solution of this image generation device and the technical solution of the aforementioned image generation method are based on the same concept. For details not described in detail in the technical solution of the image generation device, please refer to the description of the technical solution of the aforementioned image generation method.

[0389] Corresponding to the above method embodiment, the present disclosure also provides a video generation device embodiment. FIG18 shows a schematic structural diagram of a video generation device provided by an embodiment of the present disclosure. As shown in FIG18 , the device includes:

[0390] The third acquisition module 1802 is configured to acquire task data of the video generation task;

[0391] The third processing module 1804 is configured to input the task data into the video generation model and obtain feature data output by each feature processing layer in the video generation model, wherein the target video model includes at least two feature processing modules and at least one tuning module, the feature processing module includes multiple feature processing layers, the tuning module includes multiple tuning layers, and the multiple tuning layers are respectively connected to the feature processing layers of corresponding levels in the at least two feature processing modules;

[0392] The third optimization module 1806 is configured to input the feature data output by each feature processing layer into the optimization layer connected to each feature processing layer for optimization processing, and generate a target video based on the optimization results of each optimization layer.

[0393] In the disclosed embodiment, by connecting the multiple tuning layers of the tuning module with the feature processing layers of the corresponding levels in at least two feature processing modules, complete feature data output by the multiple feature processing layers is obtained, detailed information on the multiple levels is retained, and tuning processing on the corresponding levels is implemented in the multiple tuning layers, the output feature data is optimized, and a more accurate target video can be generated. In addition, at least two feature processing modules and at least one tuning module in the video generation model are decoupled, and the tuning module can be flexibly selected during the video generation process, thereby improving the versatility of the video generation model. The feature processing module can also be shared during the video generation process to perform multi-task shared tuning, thereby improving the efficiency of video generation. In the training process of the video generation model, only the tuning module can be trained for different downstream video generation tasks, without the need to train the feature processing module, thereby reducing training overhead and improving training efficiency.

[0394] The above is a schematic diagram of a video generation device according to this embodiment. It should be noted that the technical solution of the video generation device and the technical solution of the above-mentioned video generation method are based on the same concept. For details not described in detail in the technical solution of the video generation device, please refer to the description of the technical solution of the above-mentioned video generation method.

[0395] Corresponding to the above method embodiments, the present disclosure also provides an embodiment of an information processing device based on a visual model. FIG19 shows a schematic diagram of the structure of an information processing device based on a visual model provided by an embodiment of the present disclosure. As shown in FIG19 , the device is applied to a cloud training platform and includes:

[0396] The request receiving module 1902 is configured to receive a task generation request sent by a terminal device, wherein the task generation request includes request information;

[0397] Model acquisition module 1904 is configured to acquire a target visual model based on the request information, wherein the target visual model includes at least two feature processing modules and at least one tuning module, the feature processing module includes multiple feature processing layers, and the tuning module includes multiple tuning layers. The multiple tuning layers are respectively connected to the feature processing layers of corresponding levels in the at least two feature processing modules, and the tuning layers are used to perform tuning processing on feature data output by the connected feature processing layers to obtain tuning results, which are used to generate visual data;

[0398] The information generation module 1906 is configured to generate task information based on the target visual model, wherein the task information is used for the terminal device to perform the target visual task.

[0399] Optionally, the request information includes a task scenario identifier of the target visual task, or a task model identifier;

[0400] Correspondingly, the model acquisition module 1904 is further configured to:

[0401] Based on the task scene identification, a target scene template is determined from multiple preset scene templates, and based on the target scene template, a target visual model is searched from a model library, where the model library stores multiple visual models; or, based on the task model identification, a target visual model is searched from the model library.

[0402] Optionally, the request information includes sample data of the target vision task;

[0403] Correspondingly, the model acquisition module 1904 is further configured to:

[0404] Based on the sample data, the initial visual model corresponding to the target visual task is trained to obtain a trained target visual model.

[0405] In the disclosed embodiment, multiple tuning layers of the tuning module are respectively connected to the feature processing layers of corresponding levels in at least two feature processing modules, so that complete feature data output by multiple feature processing layers are obtained, detailed information on multiple levels is retained, tuning processing on corresponding levels is implemented in multiple tuning layers, and the output feature data is optimized. During the training process of the visual model, only the tuning module is trained for different downstream visual generation tasks, and there is no need to train the feature processing module, which reduces training overhead and improves training efficiency.

[0406] The above is a schematic diagram of a visual model-based information processing device according to this embodiment. It should be noted that the technical solution of the visual model-based information processing device and the technical solution of the visual model-based information processing method described above are based on the same concept. For details not described in detail in the technical solution of the visual model-based information processing device, please refer to the description of the technical solution of the visual model-based information processing method described above.

[0407] Figure 20 shows a block diagram of a computing device according to one embodiment of the present disclosure. Components of the computing device 2000 include, but are not limited to, a memory 2010 and a processor 2020. The processor 2020 is connected to the memory 2010 via a bus 2030, and a database 2050 is used to store data.

[0408] The computing device 2000 also includes an access device 2040 that enables the computing device 2000 to communicate via one or more networks 2060. Examples of such networks include a Public Switched Telephone Network (PSTN), a Local Area Network (LAN), a Wide Area Network (WAN), a Personal Area Network (PAN), or a combination of communication networks such as the Internet. The access device 2040 may include one or more of any type of network interface (e.g., a Network Interface Controller (NIC)) whether wired or wireless, such as an IEEE 802.11 Wireless Local Area Network (WLAN) wireless interface, a Worldwide Interoperability for Microwave Access (Wi-MAX) interface, an Ethernet interface, a Universal Serial Bus (USB) interface, a cellular network interface, a Bluetooth interface, or a Near Field Communication (NFC).

[0409] In one embodiment of the present disclosure, the aforementioned components of computing device 2000 and other components not shown in FIG. 20 may also be connected to one another, for example, via a bus. It should be understood that the computing device structure block diagram shown in FIG. 20 is for illustrative purposes only and does not limit the scope of the present disclosure. Those skilled in the art may add or replace other components as needed.

[0410] Computing device 2000 may be any type of stationary or mobile computing device, including a mobile computer or mobile computing device (e.g., a tablet computer, personal digital assistant, laptop computer, notebook computer, netbook computer, etc.), a mobile phone (e.g., a smartphone), a wearable computing device (e.g., a smartwatch, smart glasses, etc.), or other types of mobile devices, or a stationary computing device such as a desktop computer or personal computer (PC). Computing device 2000 may also be a mobile or stationary server.

[0411] Among them, the processor 2020 is used to execute the computer program / instructions, which, when executed by the processor, implements the steps of the above-mentioned visual model construction method, visual task processing method, image generation method, video generation method or information processing method based on visual model.

[0412] The above is a schematic scheme of a computing device of this embodiment. It should be noted that the technical scheme of the computing device and the technical schemes of the aforementioned visual model construction method, visual task processing method, image generation method, video generation method, and visual model-based information processing method are based on the same concept. For details not described in detail in the technical scheme of the computing device, please refer to the description of the technical schemes of the aforementioned visual model construction method, visual task processing method, image generation method, video generation method, or visual model-based information processing method.

[0413] An embodiment of the present disclosure also provides a computer-readable storage medium storing a computer program / instruction, which, when executed by a processor, implements the steps of the above-mentioned visual model construction method, visual task processing method, image generation method, video generation method, or information processing method based on a visual model.

[0414] The above is a schematic scheme of a computer-readable storage medium of this embodiment. It should be noted that the technical scheme of the storage medium is based on the same concept as the technical schemes of the aforementioned visual model construction method, visual task processing method, image generation method, video generation method, and visual model-based information processing method. For details not described in detail in the technical scheme of the storage medium, please refer to the description of the technical schemes of the aforementioned visual model construction method, visual task processing method, image generation method, video generation method, or visual model-based information processing method.

[0415] One embodiment of the present disclosure also provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the steps of the above-mentioned visual model construction method, visual task processing method, image generation method, video generation method or information processing method based on a visual model.

[0416] The above is a schematic scheme of a computer program product of this embodiment. It should be noted that the technical scheme of this computer program product and the technical schemes of the aforementioned visual model construction method, visual task processing method, image generation method, video generation method, and visual model-based information processing method are based on the same concept. For details not described in detail in the technical scheme of the computer program product, please refer to the description of the technical schemes of the aforementioned visual model construction method, visual task processing method, image generation method, video generation method, or visual model-based information processing method.

[0417] The foregoing description describes specific embodiments of the present disclosure. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims can be performed in an order different from that described in the embodiments and still achieve the desired results. Furthermore, the processes depicted in the accompanying drawings do not necessarily require the specific order shown or the sequential order to achieve the desired results. In certain embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0418] The computer instructions include computer program code, which may be in source code form, object code form, executable file, or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal, and software distribution medium. It should be noted that the content of the computer-readable medium may be appropriately increased or decreased according to the requirements of patent practice. For example, in some regions, according to patent practice, computer-readable media does not include electric carrier signals and telecommunication signals.

[0419] It should be noted that for the aforementioned method embodiments, for the sake of simplicity of description, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the embodiments of the present disclosure are not limited by the order of the actions described, because according to the embodiments of the present disclosure, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the embodiments of the present disclosure.

[0420] In the above embodiments, the description of each embodiment has its own focus. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0421] The preferred embodiments of the present disclosure disclosed above are only used to help illustrate the present disclosure. The optional embodiments do not describe all details in detail, nor do they limit the invention to only the specific embodiments described. Obviously, many modifications and variations can be made based on the content of the embodiments of the present disclosure. The present disclosure selects and specifically describes these embodiments in order to better explain the principles and practical applications of the embodiments of the present disclosure, so that those skilled in the art can better understand and utilize the present disclosure. The present disclosure is limited only by the claims and their full scope and equivalents.

Claims

1. A visual model system, comprising a visual model, the visual model comprising at least two feature processing modules and at least one tuning module, the feature processing modules comprising multiple feature processing layers, the tuning module comprising multiple tuning layers, the multiple tuning layers respectively connected to feature processing layers of corresponding levels in the at least two feature processing modules; The tuning layer is used to receive the feature data output by the connected feature processing layer, perform tuning processing on the feature data, and obtain a tuning result, wherein: The tuning results are used to generate visual data.

2. According to the system of claim 1, the at least two feature processing modules include an encoding module and a decoding module, the encoding module includes multiple encoding layers, the decoding module includes multiple decoding layers, and the multiple tuning layers are respectively connected to the encoding layers and decoding layers of corresponding levels.

3. According to the system of claim 2, the encoding layer includes at least two sub-encoding layers for encoding tasks, the decoding layer includes at least two sub-decoding layers for decoding tasks, the sub-encoding layers are connected layer by layer, and the sub-decoding layers are connected layer by layer, and the multiple tuning layers are respectively connected to the sub-encoding layers of the target encoding tasks and the sub-decoding layers of the target decoding tasks at the corresponding levels.

4. The system according to any one of claims 1 to 3, wherein the feature processing layer is configured to process sample data of the visual generation task to obtain sample task features, and output the sample task features to a connected tuning layer; The tuning layer is further used to perform tuning processing on the received sample task features to obtain predicted output features, wherein: The prediction output features are used to obtain task prediction results; The feature processing layer and the tuning layer are further used for training based on the task prediction results.

5. The system according to claim 4, wherein the feature processing layer is further configured to process target task samples of a downstream target vision task to obtain target task features, and output the target task features to a connected tuning layer; The tuning layer is further used to perform tuning processing on the received target task features to obtain predicted output features, wherein: The predicted output features are used to obtain target task results; The tuning layer is further used for training based on the target task results.

6. A method for constructing a visual model, comprising: Get target model parameters; Based on the target model parameters, construct at least two feature processing modules and at least one tuning module, wherein the feature processing module includes multiple feature processing layers, and the tuning module includes multiple tuning layers, wherein the tuning layers are used to perform tuning processing on the input feature data to obtain tuning results, and the tuning results are used to generate visual data; The multiple tuning layers are respectively connected to the feature processing layers of corresponding levels in the at least two feature processing modules to obtain a visual model.

7. The method according to claim 6, wherein obtaining target model parameters comprises: Obtaining model parameters and tuning parameters of a pre-trained visual model corresponding to a visual generation task, wherein the pre-trained visual model includes at least two feature processing modules; The step of constructing at least two feature processing modules and at least one tuning module based on the target model parameters includes: Based on the pre-trained visual model, construct at least two feature processing modules, and set initial parameters of the at least two feature processing modules according to model parameters of the pre-trained visual model; Based on the tuning parameters, at least one tuning module is constructed, and parameters of the tuning module are initialized.

8. The method according to claim 6 or 7, after respectively connecting the plurality of tuning layers to the feature processing layers of corresponding levels in the at least two feature processing modules to obtain a visual model, further comprising: Training the visual model based on sample data of the visual generation task to obtain the trained visual model; Based on target task samples of multiple downstream target visual tasks, respectively tuning the trained visual model to obtain target tuning modules corresponding to the multiple downstream target visual tasks; Each target tuning module is connected to the at least two feature processing modules respectively to obtain an updated visual model.

9. A method for processing a visual task, comprising: Obtain task data for the target visual generation task; Inputting the task data into a target vision model corresponding to the target vision generation task, and obtaining feature data output by each feature processing layer in the target vision model, wherein the target vision model includes at least two feature processing modules and at least one tuning module, the feature processing module includes multiple feature processing layers, the tuning module includes multiple tuning layers, and the multiple tuning layers are respectively connected to the feature processing layers of corresponding levels in the at least two feature processing modules; The feature data output by each feature processing layer is respectively input into the tuning layer connected to each feature processing layer for tuning processing, and the target visual data is generated based on the tuning results of each tuning layer.

10. The method according to claim 9, wherein the at least two feature processing modules include an encoding module and a decoding module, the encoding module includes multiple encoding layers, the decoding module includes multiple decoding layers, and the multiple tuning layers are respectively connected to the encoding layers and decoding layers at corresponding levels; The step of inputting the task data into a target vision model corresponding to the target vision generation task, and obtaining feature data output by each feature processing layer in the target vision model, comprises: Inputting the task data into a target visual model corresponding to the target visual generation task, and obtaining encoding features output by each encoding layer and decoding features output by each decoding layer in the target visual model; The step of inputting the feature data output by each feature processing layer into the optimization layer connected to each feature processing layer for optimization processing, and generating target visual data based on the optimization results of each optimization layer, includes: The encoding features output by each encoding layer and the decoding features output by each decoding layer are input into the connected tuning layer for tuning processing, and target visual data is generated based on the tuning results of each tuning layer.

11. The method according to claim 10, wherein the encoding layer includes at least two sub-encoding layers for encoding tasks, the decoding layer includes at least two sub-decoding layers for decoding tasks, the sub-encoding layers are connected layer by layer, and the sub-decoding layers are connected layer by layer, and the multiple tuning layers are respectively connected to the sub-encoding layers and sub-decoding layers of the target encoding tasks and the target decoding tasks at the corresponding levels; The step of inputting the task data into a target visual model corresponding to the target visual generation task, and obtaining encoding features output by each encoding layer and decoding features output by each decoding layer in the target visual model, comprises: The task data is input into the target visual model corresponding to the target visual generation task, and encoding is performed layer by layer according to the hierarchical order of the sub-coding layers until the encoding features output by the sub-coding layer of the target encoding task are obtained, and decoding is performed layer by layer according to the hierarchical order of the sub-decoding layers until the decoding features output by the sub-decoding layer of the target encoding task are obtained.

12. The method according to claim 10 or 11, wherein the plurality of tuning layers are connected layer by layer; The step of inputting the encoding features output by each encoding layer and the decoding features output by each decoding layer into a connected tuning layer for tuning processing, and generating target visual data based on the tuning results of each tuning layer, includes: Inputting the encoding features output by each encoding layer and the decoding features output by each decoding layer into a connected tuning layer for tuning processing, and inputting the tuning result of the previous tuning layer into the next tuning layer in the hierarchical order of the multiple tuning layers to obtain the output features of the last tuning layer; Based on the output features of the last fine-tuning layer, target visual data is generated.

13. The method according to any one of claims 9 to 12, further comprising, after generating target visual data based on the optimization results of each optimization layer: Annotating visual key points in the target visual data, wherein the visual key points are visual points representing task information of the target visual generation task; Feedback the target visual data to the front-end user.

14. The method according to claim 13, after feeding back the target visual data to the front-end user, further comprising: receiving key point feedback information sent by the front-end user for the visual key point in the target visual data, wherein the key point feedback information is feedback information provided by the front-end user on the visual key point according to the task information; Inputting the key point feedback information, the task data and the target visual data into the target visual model to obtain the adjusted feature data output by each feature processing layer; The adjusted feature data output by each feature processing layer are respectively input into the tuning layers connected to each feature processing layer for tuning processing, and based on the tuning results of each tuning layer, the adjusted target visual data are generated.

15. The method according to claim 14, after generating adjusted target visual data based on the tuning results of each tuning layer, further comprising: Integrating the key point feedback information and the adjusted target visual data to obtain a visual generation result; Feedback the visual generation result to the front-end user.

16. The method according to any one of claims 9 to 12, further comprising, after generating target visual data based on the optimization results of each optimization layer: Feeding back the target visual data to a front-end user; Receiving visual feedback information sent by the front-end user, wherein the visual feedback information is feedback information provided by the front-end user on the target visual data; constructing sample task data based on the visual feedback information; The target vision model is trained based on the sample task data.

17. The method according to any one of claims 9 to 12, further comprising, before inputting the task data into a target visual model corresponding to the target visual generation task and obtaining feature data output by each feature processing layer in the target visual model: Obtaining a target task sample of the target visual generation task; Inputting the target task sample into an initial visual model to obtain target task features output by each feature processing layer in the initial visual model, wherein the initial visual model is a visual model pre-trained based on sample data of a general visual generation task; Inputting the target task features output by each feature processing layer into the optimization layer connected to each feature processing layer for optimization processing, and obtaining the target task result based on the predicted output features of each optimization layer; Based on the target task result, each tuning layer is trained to obtain a target visual model corresponding to the target visual generation task.

18. A method for generating an image, comprising: Get the task data of the image generation task; Inputting the task data into an image generation model to obtain feature data output by each feature processing layer in the image generation model, wherein the image generation model includes at least two feature processing modules and at least one tuning module, the feature processing module includes multiple feature processing layers, the tuning module includes multiple tuning layers, and the multiple tuning layers are respectively connected to the feature processing layers of corresponding levels in the at least two feature processing modules; The feature data output by each feature processing layer is respectively input into the optimization layer connected to each feature processing layer for optimization processing, and a target image is generated based on the optimization results of each optimization layer.

19. A video generation method, comprising: Get the task data of the video generation task; Inputting the task data into a video generation model to obtain feature data output by each feature processing layer in the video generation model, wherein the target video model includes at least two feature processing modules and at least one tuning module, the feature processing module includes multiple feature processing layers, the tuning module includes multiple tuning layers, and the multiple tuning layers are respectively connected to the feature processing layers of corresponding levels in the at least two feature processing modules; The feature data output by each feature processing layer is respectively input into the optimization layer connected to each feature processing layer for optimization processing, and the target video is generated based on the optimization results of each optimization layer.

20. An information processing method based on a visual model, applied to a cloud training platform, comprising: Receiving a task generation request sent by a terminal device, wherein the task generation request includes request information; Based on the request information, a target visual model is obtained, wherein the target visual model includes at least two feature processing modules and at least one tuning module, the feature processing module includes multiple feature processing layers, the tuning module includes multiple tuning layers, the multiple tuning layers are respectively connected to the feature processing layers of corresponding levels in the at least two feature processing modules, the tuning layers are used to perform tuning processing on feature data output by the connected feature processing layers to obtain tuning results, and the tuning results are used to generate visual data; Based on the target visual model, task information is generated, wherein the task information is used by the terminal device to perform the target visual task.

21. The method according to claim 20, wherein the request information includes a task scenario identifier of the target visual task, or a task model identifier; The acquiring of a target visual model based on the request information includes: Based on the task scene identifier, determining a target scene template from a plurality of preset scene templates, and based on the target scene template, searching for a target visual model from a model library, wherein the model library stores a plurality of visual models; or, Based on the task model identifier, a target visual model is searched from the model library.

22. The method according to claim 20, wherein the request information includes sample data of a target visual task; The acquiring of a target visual model based on the request information includes: Based on the sample data, an initial visual model corresponding to the target visual task is trained to obtain a trained target visual model.

23. A cloud training platform, comprising a request interface and a response unit; The request interface is used to receive a task generation request sent by a terminal device, wherein: The task generation request includes request information; The response unit is used to obtain a target visual model based on the request information, wherein the target visual model includes at least two feature processing modules and at least one tuning module, the feature processing module includes multiple feature processing layers, and the tuning module includes multiple tuning layers, and the multiple tuning layers are respectively connected to the feature processing layers of corresponding levels in the at least two feature processing modules, and the tuning layer is used to perform tuning processing on the feature data output by the connected feature processing layers to obtain a tuning result, and the tuning result is used to generate visual data; based on the target visual model, task information is generated, wherein the task information is used for the terminal device to perform the target visual task.

24. The cloud training platform according to claim 23, further comprising a model library, wherein the model library stores a plurality of visual models; the request information comprises a task scenario identifier of the target visual task, or a task model identifier; The response unit is specifically used to determine a target scene template from multiple preset scene templates based on the task scene identifier, and search for a target visual model from the model library based on the target scene template; or, search for a target visual model from the model library based on the task model identifier.

25. A computing device comprising: memory and processor; The memory is used to store computer programs / instructions, and the processor is used to execute the computer programs / instructions. When the computer program / instructions are executed by the processor, the steps of the method according to any one of claims 6 to 22 are implemented.

26. A computer-readable storage medium storing a computer program / instruction, wherein the computer program / instruction, when executed by a processor, implements the steps of the method according to any one of claims 6 to 22.

27. A computer program product comprising a computer program / instruction, which, when executed by a processor, implements the steps of the method according to any one of claims 6 to 22.

Images