Model segmentation method, model content generation method and device

Abstract

The embodiment of the specification discloses a model segmentation method, a model content generation method and device, the segmentation method can include exchanging training of the model based on the exchange module, and the exchange training result is segmented and trained under the constraint condition, and the segmentation training result of the model is obtained; the content generation method can include loading the model based on the segmentation training result, and outputting the output result corresponding to the guide information.

Description

Technical Field

[0001] One or more embodiments of this specification relate to the field of artificial intelligence, and in particular to a method for segmenting a model, a method for generating model content, and an apparatus. Background Technology

[0002] With the development of the Internet, image generation methods based on diffusion models have achieved a major breakthrough in the field of AIGC, propelling AIGC technology from academia to industry.

[0003] However, although existing AIGC technology can generate a large amount of creative and high-quality image content, it requires a lot of resources when it is called up, and its computing power consumption is dozens of times higher than that of traditional artificial intelligence algorithms. Summary of the Invention

[0004] To address the problems existing in the prior art, embodiments of the present invention provide a model segmentation method, a model content generation method, and an apparatus.

[0005] According to one aspect of the embodiments of this specification, a method for segmenting a model is provided, comprising:

[0006] For the training parameters of the AIGC model, the training parameters are trained through the exchange module, and the exchange loss function generated during the training process is used as the convergence function to obtain the exchange result of the layer parameters of the AIGC model.

[0007] Based on the exchange results, the layer parameters of the AIGC model are replaced, and the reuse layer constraints after parameter replacement are determined.

[0008] Obtain the preset storage space of the target instruction object, and calculate the resident layer constraints based on the preset storage space;

[0009] Using the resident layer constraints and reuse layer constraints as constraints, the resident / dynamic preliminary segmentation of the AIGC model is planned;

[0010] Dynamically obtain resource usage information of the target instruction object;

[0011] Based on the resource usage information and constraints, the AIGC model is trained by segmentation, using the adjusted loss function of the initial resident / dynamic segmentation as the convergence function, to obtain the resident / dynamic real-time segmentation of the AIGC model.

[0012] Optionally, in one example of the above aspects, the switching module includes:

[0013] Switching layer prediction module and dynamic switching module;

[0014] The step of training the training parameters through the exchange module includes:

[0015] Using the training parameters as input data, the training switching layer of the AIGC model is trained by inputting it into the switching layer prediction module.

[0016] Using the training parameters and the training exchange layer as input data, the data is input into the dynamic exchange module to train and obtain the exchange results of the layer parameters of the AIGC model.

[0017] Optionally, in one example of the above aspects, the exchange loss function includes:

[0018] The training parameters are used as input data and input into the exchange layer prediction module to generate diffusion loss during training.

[0019] Using the training parameters and the training exchange layer as input data, the dynamic exchange module is trained to generate parameter consistency loss for dynamic exchange.

[0020] Optionally, in one example of the above aspects, the step of planning the initial resident / dynamic partitioning of the AIGC model using the resident layer constraints and reuse layer constraints as constraints includes:

[0021] Determine the layer vector corresponding to each layer of the AIGC model, and obtain the resident layer time and dynamic layer time corresponding to the layer vector of each layer.

[0022] The number of resident layer vectors in the layer vectors is determined based on the resident layer constraints, and the conflict vectors in the layer vectors are determined based on the reuse layer constraints.

[0023] Based on the number of resident layer vectors, conflict vectors, and the resident layer time and dynamic layer time corresponding to the layer vectors, linear programming is performed on the layer vectors corresponding to each layer to determine the initial resident / dynamic segmentation of the AIGC model.

[0024] Alternatively, in one example of the above aspects, the linear programming aims to minimize the total time consumption of the layer vectors at each layer of the AIGC model.

[0025] Optionally, in one example of the above aspects, the adjusted loss function includes:

[0026] The change in the total time consumption of the layer vectors;

[0027] The amount of change in the segmentation state of the resident / dynamic initial segmentation of the AIGC model.

[0028] Optionally, in one example of the above aspects, dynamically obtaining the resource usage information of the target instruction object includes:

[0029] Obtain a preset period duration, and periodically obtain resource usage information of the target instruction object based on the period duration;

[0030] Alternatively, a preset instruction time point can be obtained, and resource usage information of the target instruction object can be obtained at the instruction time point.

[0031] According to another aspect of the embodiments of this specification, a model segmentation apparatus is provided, comprising:

[0032] The exchange module training module is configured to train the training parameters of the AIGC model. The training parameters are trained through the exchange module, and the exchange loss function generated during the training process is used as the convergence function to obtain the exchange result of the layer parameters of the AIGC model.

[0033] The parameter replacement module is configured to replace the layer parameters of the AIGC model based on the exchange result, and determine the reuse layer constraints after parameter replacement.

[0034] The resident layer constraint calculation module is configured to obtain the preset storage space of the target instruction object and calculate the resident layer constraints based on the preset storage space.

[0035] The preliminary segmentation module is configured to use the resident layer constraints and reuse layer constraints as constraints to plan the resident / dynamic preliminary segmentation of the AIGC model;

[0036] The dynamic acquisition module is configured to dynamically acquire resource usage information of the target instruction object;

[0037] The real-time segmentation module is configured to perform segmentation training based on the resource usage information and constraints, using the adjustment loss function of the resident / dynamic initial segmentation as the convergence function, and train to obtain the resident / dynamic real-time segmentation of the AIGC model.

[0038] Optionally, in one example of the above aspects, it includes:

[0039] The training module for the switching layer is configured to use the training parameters as input data and input them into the switching layer prediction module to train the training switching layer of the AIGC model.

[0040] The exchange result training module is configured to take the training parameters and the training exchange layer as input data, input them into the dynamic exchange module, and train to obtain the exchange result of the layer parameters of the AIGC model.

[0041] Optionally, in one example of the above aspects, it includes:

[0042] The training parameter input module is configured to input the training parameters as input data into the exchange layer prediction module, and the diffusion loss generated during training.

[0043] The parameter consistency loss training module is configured to take the training parameters and the training exchange layer as input data and input them into the dynamic exchange module to train and generate the parameter consistency loss of dynamic exchange.

[0044] Optionally, in one example of the above aspects, it includes:

[0045] The time consumption acquisition module is configured to determine the layer vector corresponding to each layer of the AIGC model and acquire the resident layer time and dynamic layer time corresponding to the layer vector of each layer.

[0046] The conflict vector determination module is configured to determine the number of resident layer vectors in the layer vectors based on the resident layer constraints, and to determine the conflict vectors in the layer vectors based on the reuse layer constraints.

[0047] The linear programming module is configured to perform linear programming on the layer vectors corresponding to each layer based on the number of resident layer vectors, conflict vectors, and the resident layer time and dynamic layer time corresponding to the layer vectors, so as to determine the initial resident / dynamic segmentation of the AIGC model.

[0048] Optionally, in one example of the above aspects, it includes:

[0049] The solution module is configured to minimize the total time consumption of the layer vectors of each layer of the AIGC model when performing the linear programming.

[0050] Optionally, in one example of the above aspects, it includes:

[0051] The loss function determination module is adjusted to change the total time consumption of the layer vectors and the change in the segmentation state of the AIGC model's resident / dynamic initial segmentation.

[0052] Optionally, in one example of the above aspects, it includes:

[0053] The resource usage information acquisition module is configured to acquire a preset periodic duration and periodically acquire the resource usage information of the target instruction object based on the preset periodic duration, or to acquire a preset instruction time point and acquire the resource usage information of the target instruction object at the instruction time point.

[0054] According to another aspect of the embodiments of this specification, a method for generating content for a model is provided, comprising:

[0055] The persistent / dynamic real-time segmentation of the AIGC model obtained based on the features described in one or more embodiments is loaded into the target instruction object.

[0056] The system receives guidance information from the target instruction object, inputs the guidance information into the resident / dynamically real-time segmented AIGC model for training, temporarily loads the dynamic layer in the AIGC model through the guidance information during the training process, and outputs guidance results that match the guidance information.

[0057] According to another aspect of the embodiments of this specification, a model content generation apparatus is provided, comprising:

[0058] The model loading module is configured to perform resident / dynamic real-time segmentation of the AIGC model obtained based on the features described in one or more embodiments, and load the resident layer in the resident / dynamic real-time segmentation of the AIGC model into the target instruction object;

[0059] The guidance result output module is configured to receive guidance information from the target instruction object, input the guidance information into the resident / dynamically real-time segmented AIGC model for training, temporarily load the dynamic layer in the AIGC model through the guidance information during the training process, and output guidance results that match the guidance information.

[0060] According to another aspect of the embodiments of this specification, an electronic device is provided, including a processor and a memory;

[0061] The processor is connected to the memory;

[0062] The memory is used to store executable program code;

[0063] The processor runs a program corresponding to the executable program code stored in the memory to perform the methods described in one or more embodiments.

[0064] According to another aspect of the embodiments of this specification, a computer-readable storage medium is provided that stores a computer program thereon, which, when executed by a processor, implements one or more of the methods described in the embodiments.

[0065] In view of the above, in one or more embodiments of this specification, dynamic exchange technology is used to obtain an AIGC model with high reusability after dynamic exchange training. To overcome the problem that directly loading a highly reusable AIGC model would slow down training, the constraints of the model are obtained, and preliminary segmentation is performed to obtain a resident / dynamic preliminary segmentation of the AIGC model. Then, based on the execution status of the target instruction object, adaptive segmentation adjustments are made to obtain a resident / dynamic real-time segmentation result of the AIGC model. This approach can, to a certain extent, maintain the training speed of the highly reusable model while correspondingly reducing the amount of data called by the model and reducing resource consumption. Attached Figure Description

[0066] To more clearly illustrate the technical solutions in the embodiments of this specification, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this specification. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0067] Figure 1 A global schematic diagram illustrating an application scenario of a model segmentation method provided in the embodiments of this specification;

[0068] Figure 2 An interactive schematic diagram of a content generation method for a model provided in an embodiment of this specification;

[0069] Figure 3 A flowchart illustrating a model segmentation method provided in an embodiment of this specification;

[0070] Figure 4 A flowchart illustrating another model segmentation method provided in the embodiments of this specification;

[0071] Figure 5 A flowchart illustrating another model segmentation method provided in the embodiments of this specification;

[0072] Figure 6 A flowchart illustrating another model segmentation method provided in the embodiments of this specification;

[0073] Figure 7 This document provides a flowchart illustrating a method for generating content for a model, as exemplified in this specification.

[0074] Figure 8 A schematic diagram of the structure of a model segmentation device provided in the embodiments of this specification;

[0075] Figure 9 This is a schematic diagram of the structure of a model content generation device provided in the embodiments of this specification;

[0076] Figure 10 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this specification. Detailed Implementation

[0077] The subject matter described herein will now be discussed with reference to exemplary embodiments. It should be understood that these embodiments are discussed merely to enable those skilled in the art to better understand and implement the subject matter described herein, and are not intended to limit the scope, applicability, or examples set forth in the claims. The function and arrangement of the elements discussed may be changed without departing from the scope of this specification. Various processes or components may be omitted, substituted, or added as needed in the various examples. For example, the described methods may be performed in a different order than described, and steps may be added, omitted, or combined. Furthermore, features described in some examples may be combined in other examples.

[0078] As used herein, the term "comprising" and its variations are open terms meaning "including but not limited to". The term "based on" means "at least partially based on". The terms "one embodiment" and "an embodiment" mean "at least one embodiment". The term "another embodiment" means "at least one other embodiment". The terms "first", "second", etc., may refer to different or the same objects. Other definitions, whether explicit or implicit, may be included below. Unless explicitly indicated by the context, the definition of a term shall remain consistent throughout the specification.

[0079] With the development of the internet, image generation methods based on diffusion models have achieved significant breakthroughs in the field of AIGC (AI Generative Content Generation), propelling AIGC technology from academia to industry. However, in common AIGC models, each layer has different weights, meaning these layers cannot share weights. This results in each load requiring the separate loading of weights for each layer. Consequently, while existing AIGC technologies can generate a large amount of creative and high-quality image content, the computational resources consumed are dozens of times higher than traditional artificial intelligence algorithms, making runtime extremely resource-intensive.

[0080] To address this issue, existing technologies propose two schemes to optimize the computational efficiency of AIGC models. Scheme 1 involves optimizing the AIGC model using third-party acceleration libraries (such as xformer, deepspeed, and other general-purpose deep learning model acceleration libraries) before training it on a device. This scheme has low adaptation costs, often requiring only minor code modifications. However, its drawbacks are also significant: the efficiency improvement is not substantial, typically ranging from 20% to 50%, failing to adequately meet the requirements for running on terminal devices. Scheme 2 involves pruning the original model into smaller models, then using the unpruned model to distill the pruned model, improving the performance of the smaller models. The advantage of this scheme is that the entire model can be deployed and run on the terminal. The disadvantage is that while the pruned smaller models meet the requirements for running on terminal devices, the quality of generated AIGC content drops significantly, falling far short of the quality generated using the unpruned larger model.

[0081] In view of the above, the embodiments of this specification propose a model segmentation scheme. In this scheme, a highly reusable AIGC model is trained using dynamic exchange technology. The weight parameters of layers with the same structure are reused, and sub-units, such as corresponding layers in the model, are reused in different modules as much as possible, resulting in a highly reusable AIGC model. Then, based on the model's reusability and the storage of target instruction objects, intelligent segmentation is performed to obtain the initial resident / dynamic segmentation of the AIGC model. Adaptive adjustments are then made based on the running conditions to determine the real-time resident / dynamic segmentation of the AIGC model. Furthermore, after determining the dynamic layer for basic calls and the resident layer for targeted calls, deployment and application can be carried out accordingly based on the segmentation of the resident and dynamic layers, thereby reducing the amount of data called by the model and reducing resource consumption.

[0082] The following will describe in detail, with reference to the accompanying drawings, the model segmentation scheme and the model content generation scheme according to the embodiments of this specification.

[0083] Figure 1 Exemplary application scenarios of the AIGC model according to embodiments of this specification are shown.

[0084] exist Figure 1 In this system, the AIGC model is trained via a server / terminal. After training, it is configured on the platform or terminal to receive user prompts (input text or instructions provided by the user to the AIGC model. These prompts typically guide the model to generate specific images, scenes, or other instruction results. Users can provide different prompts to produce different results as needed) and generate the corresponding instruction results. The server / terminal can be any device capable of training the model, such as a dedicated server, graphics processing unit (GPU), cloud server, personal computer, GPU server, edge device, or any combination thereof.

[0085] The platform or terminal that receives the user's prompt can be configured with an AIGC model and can perform related tasks based on the user's prompt. This can include a command-line terminal, a Jupyter Notebook platform, a web application, a mobile application, or any combination thereof.

[0086] It should be understood that Figure 1 All network entities shown are exemplary; any other network entities may be involved in the application scenario depending on the specific application requirements.

[0087] Figure 2 An exemplary interface for executing a prompt using the AIGC model of a platform / terminal according to an embodiment of this specification is shown.

[0088] The user interface is included in the platform / terminal device and may include a statement input area and a display area. The statement input area is for the user to enter a prompt. For example, the target statement "sunset at the beach". The display area is based on the target result obtained after training the prompt using the AIGC model, such as an image corresponding to "sunset at the beach". The number of images can be set manually or arbitrarily.

[0089] It should be noted that Figure 2 All modules and their layouts shown are exemplary. Depending on specific application requirements, Figure 2 The user interface can omit or add any modules, and Figure 2 The layout of modules in the user interface can also be changed in various ways.

[0090] Figure 3 A flowchart illustrating the model segmentation process according to an embodiment of this specification is shown, such as... Figure 3 As shown, it includes:

[0091] Step 301: For the training parameters of the AIGC model, the training parameters are trained through the exchange module, and the exchange loss function generated during the training process is used as the convergence function to obtain the exchange result of the layer parameters of the AIGC model.

[0092] In step 301, the training parameters generated during the training of the AIGC model are trained using the exchange module, and then trained again using the exchange module to obtain a highly reusable AIGC model after layer parameter exchange. The exchange loss function generated during model data exchange in the AIGC model during training serves as the convergence function for model training convergence, thereby obtaining the exchange results of the corresponding AIGC model's layer parameters. Figure 4As shown, the exchange of layer parameters of the AIGC model generated during AIGC model training is achieved through the following steps.

[0093] Step 401: Input the prompt and random noise into the AIGC model to obtain training parameters during the training process. In this step, the AIGC model uses the input prompt and random noise. The random noise introduces random changes or errors into the dataset, improving the model's robustness and generalization ability. Then, using the training parameters of the AIGC model as input data, the subsequent exchange module is trained. The exchange module may include an exchange layer prediction module and a dynamic exchange module. The training process includes steps 402 and 403.

[0094] Step 402: Using the training parameters as input data, input them into the exchange layer prediction module to train and obtain the training exchange layer of the AIGC model. In this step, the parameters of all layers in the AIGC model are used as input to the exchange layer prediction module, which outputs the numbers of the layers suitable for exchange and performs parameter exchange. For example, taking all the layer parameters of the AIGC model (assuming layers 1 to 20 are input), the exchange layer prediction module performs feature extraction, feature exchange, and feature fusion on all layer parameters to predict the corresponding target layer, determine the corresponding layer information, and then outputs the corresponding layer numbers as 1 and 3; then the parameters of 1 and 3 are randomly exchanged in the dynamic exchange module.

[0095] Step 403: Input the dynamic exchange module to train and obtain the exchange results of the layer parameters of the AIGC model. This step uses the parameters of all layers in the AIGC model as the input of the dynamic exchange module, or it can use the parameters of the layer corresponding to the layer number output by the exchange layer prediction module as the input of the dynamic exchange module. The dynamic exchange module performs random parameter exchange between layers. The dynamic exchange module completes the corresponding dynamic exchange through feature selection, model component exchange, parameter adjustment, and output integration.

[0096] Step 404: Detect the exchange loss function generated during training. This step obtains the exchange loss function generated when exchanging layer parameters. The exchange loss function consists of two parts. The first part is the diffusion loss, which includes the propagation of prediction errors to neighboring pixels. In the exchange layer prediction module, some local or global contextual information is usually used for prediction. However, due to network limitations or other factors, the prediction results may have some errors. The second part is the consistency loss of dynamic exchange parameters. In the dynamic exchange module, different input samples may trigger different component exchanges or adjustments, causing changes in the model parameters and generating the consistency loss of dynamic exchange parameters. When calculating the above loss function, the final exchange loss function is determined through feature extraction, parameter recording, parameter consistency calculation, and loss optimization in the dynamic exchange module.

[0097] Step 405: Train the AIGC model until it converges, using the exchange loss function as the convergence function. In this step, during the training of the AIGC model through the exchange module, a determined exchange loss function is used as the convergence function during model training. When the exchange loss function converges, the corresponding AIGC model is obtained.

[0098] Step 302: Based on the exchange results, perform parameter replacement on the layer parameters of the AIGC model and determine the reuse layer constraints after parameter replacement.

[0099] In step 302, based on the AIGC model trained by the exchange module in the above steps, the parameters of the layers that can be dynamically exchanged are replaced with the same set of parameters. The same set of parameters can be the average parameters of the two exchangeable layers or other custom parameters. The model trained in this way has a higher reusability rate compared with the AIGC model before the exchange module training. After the parameter replacement, the reusability layer constraints corresponding to the parameters of the replacement layer are calculated. The data of each layer in the AIGC model is usually stored as a dynamic layer when configured in the memory of the target instruction object, that is, it does not occupy memory. When the dynamic layer data is needed, it is loaded into memory in real time for subsequent steps, which takes a relatively long time. However, the layer parameters of the replacement layer, due to the high reusability, do not exist in the form of dynamic layer data, but enter the memory in the form of resident layer. When the data of the resident layer is called, the time is shorter than that of the layer data stored in the dynamic layer because the data itself is in memory. The reuse layer constraint in the AIGC model is that layers that have undergone parameter replacement enter memory as resident layers, while layers that have not undergone parameter replacement can exist at the target instruction object as dynamic layers.

[0100] Step 303: Obtain the preset storage space of the target instruction object, and calculate the resident layer constraints based on the preset storage space.

[0101] In step 303, based on the AIGC model with a high reuse rate obtained in step 302, loading AIGC into the storage space of the corresponding target instruction object can reduce the requirements for computing resources. However, direct reuse will have problems. For some parallel structures (such as layers 1 and 2 can be reused, but they are in a parallel relationship, that is, they need to run at the same time), reuse loading will slow down the training of the model (reuse loading of parallel structures will turn them into serial structures). In order to solve the problem of slow model training, it is necessary to combine reuse layer constraints and resident layer constraints. Reuse layer constraints are used to constrain layers that have undergone parameter replacement to enter the preset storage space as resident layers, while resident layer constraints are used to constrain the maximum value that can enter the preset storage space as resident layers. The determination of resident layer constraints begins by obtaining the preset storage space of the target instruction object. The preset storage space can be the memory / video memory size in the target instruction object's configuration information, the storage space size defined by the target instruction object, or other available storage space sizes. Then, resident layer constraints are calculated using the preset storage space. The parameters corresponding to the resident layer need to be loaded into the storage space, which requires a certain amount of memory. The space occupied can be determined based on the number of parameters corresponding to the resident layer. Then, combined with the memory size of the preset storage space, the maximum number of resident layer parameters that the preset storage space can load can be determined. The resident layer constraint is that the number of resident layers should be less than the maximum number of resident layers that the preset storage space can load.

[0102] Step 304: Using the resident layer constraints and reuse layer constraints as constraints, plan the initial resident / dynamic segmentation of the AIGC model.

[0103] In step 304, the resident layer constraints and reuse layer constraints from the previous steps are used as constraints. The resident layer constraints determine the range and maximum number of resident layers, while the reuse layer constraints determine the layer parameters that have participated in parameter replacement and exist as resident layers in the AIGC model. This allows for the planning and determination of the initial resident / dynamic partitioning of the AIGC model. Figure 5 As shown, the initial resident / dynamic segmentation results of AIGC model training are achieved through the following steps.

[0104] Step 501: Determine the layer vector corresponding to each layer of the AIGC model, and obtain the resident layer time and dynamic layer time corresponding to the layer vector of each layer. In this step, the corresponding parameters in the layer vector of each layer of the AIGC model are obtained, and the resident layer time of each layer parameter after entering the preset storage space of the target instruction object and being called is calculated by factors such as the parameter data volume, data size, and data sequence. The dynamic layer time of each layer parameter when stored in a dynamic layer is then determined by the dynamic layer time during real-time dynamic loading.

[0105] Step 502: Determine the number of resident layer vectors in the layer vector based on the resident layer constraints, and determine the conflict vectors in the layer vector based on the reuse layer constraints. In this step, the range of the number of resident layer vectors in the layer vector can be determined by the maximum number of resident layers in the resident layer constraints. The reuse layer constraints determine the layers that have participated in parameter replacement as resident layers in the AIGC model. Then, the layers that have participated in parameter replacement are used as constraints that need to enter the preset storage space in the form of resident layers, and are used as conflict vectors in the layer vector.

[0106] Step 503: Based on the number of resident layer vectors, conflict vectors, and the resident layer time and dynamic layer time corresponding to each layer vector, perform linear programming on the layer vectors corresponding to each layer to determine the initial resident / dynamic segmentation of the AIGC model. The linear programming process in this step can be performed using the number of resident layer vectors, conflict vectors, and the resident layer time and dynamic layer time corresponding to each layer vector. For example, assuming the variable X in the linear programming is a vector, the number of elements in the vector is the same as the number of layers in the AIGC model. Each element of the vector can be a resident layer or a dynamic layer, and corresponds to its own resident layer time and dynamic layer time. Then, using the maximum value of the resident layer vector data and the conflict vector as constraints, the linear programming method is used to solve for the minimum time under the constraints, and the initial resident / dynamic segmentation result of the AIGC model corresponding to the minimum time is determined.

[0107] Step 305: Dynamically obtain resource usage information of the target instruction object.

[0108] In step 305, based on the determined initial resident / dynamic segmentation results of the AIGC model, an adaptive analysis of the environment is required. The initial resident / dynamic segmentation results of the AIGC model are obtained based on the model itself and common devices. However, in actual application, the available resources of the devices are dynamically changing. Therefore, it is necessary to perform real-time analysis based on the device resources and adjust the above-mentioned initial resident / dynamic segmentation to better suit the current situation. Dynamically acquire resource usage information of the target instruction object. The determination of resource usage information can be based on a comprehensive judgment from the perspectives of space utilization, read / write speed, disk I / O load, etc., of the target instruction object. The timing of dynamic adjustment can be achieved by acquiring a preset periodic duration and periodically acquiring the resource usage information of the target instruction object based on the periodic duration for subsequent adaptive adjustments. Alternatively, it can be achieved by acquiring a preset instruction time point and acquiring the resource usage information of the target instruction object at the instruction time point for subsequent adaptive adjustments. Other time rules are also possible, which are not limited here.

[0109] Step 306: Based on resource usage information and constraints, perform segmentation training, using the adjusted loss function of the resident / dynamic initial segmentation as the convergence function, to train and obtain the resident / dynamic real-time segmentation of the AIGC model.

[0110] In step 306, segmentation training is performed using the resource usage information of the target instruction object, as well as the constraints of the resident layer and reuse layer. Segmentation training can be implemented using an MLP model, which can perform segmentation training to complete classification and regression problems. During training, the adjusted loss function generated during training is used as the convergence function, and training continues until the model converges, resulting in the resident / dynamic real-time segmentation of the AIGC model. Figure 6 As shown, the resident / dynamic real-time segmentation results of AIGC model training are achieved through the following steps.

[0111] Step 601: Input the resource usage, constraints, and the initial resident / dynamic segmentation trained by the AIGC model into the MLP model. In this step, the segmentation training is completed through the MLP model. During the segmentation training process, the MLP model performs multi-sensor training of layer parameters based on the initial resident / dynamic segmentation trained by the AIGC model to obtain the subsequent adjustment results.

[0112] Step 602: Obtain the adjustment loss function during the MLP model training process. In this step, during the multi-sensory training of the MLP model's layer parameters, corresponding adjustment losses may be generated. These adjustment losses may include: minimizing the time increase loss, i.e., the loss caused by the increased time consumption of the adjusted segmentation after the initial static / dynamic segmentation in the AIGC model training; and minimizing the segmentation change loss, i.e., the loss caused by the change in static / dynamic segmentation relative to the change in the initial static / dynamic segmentation in the previous step. Thus, the adjustment loss function generated during the MLP model training process is determined through these two aspects.

[0113] Step 603: Using the adjusted loss function as the convergence function, train the resident / dynamic real-time segmentation of the AIGC model. In this step, during the MLP model training process, a determined adjusted loss function is used as the convergence function during model training. When the adjusted loss function converges, the resident / dynamic real-time segmentation of the AIGC model is obtained.

[0114] In the embodiments described in this specification, the purpose of the implementation is to keep frequently called parts of the model continuously loaded on the device, while loading less frequently used parts only when necessary for runtime. This effectively reduces the GPU memory requirements of the server / device. Specifically, firstly, dynamic swapping technology is used to obtain a highly reusable AIGC model through dynamic swapping training, that is, reusing model layers in different modules as much as possible. Secondly, directly reusing a highly reusable AIGC model may cause problems. For some parallel structures (such as layers 1 and 2 being reusable but in parallel, i.e., needing to run simultaneously), reusing loading can slow down model training. To overcome the problem of highly reusable AIGC models, intelligent segmentation is performed based on the model's reuse status to obtain an initial segmentation of the model's resident / dynamic parts. Then, the current device / server operating status is analyzed, and adaptive segmentation adjustments are made to obtain the final real-time segmentation result of resident / dynamic parts. This achieves the goal of reducing the amount of data called by the model and reducing resource consumption while ensuring that the training speed of the highly reusable model is not reduced.

[0115] In one or more embodiments of this specification, after the model is segmented, it needs to be loaded and configured to generate the corresponding target content. Figure 7 A flowchart illustrating the content generation method for the model is shown.

[0116] The content generation method for the model in the embodiments of this specification includes:

[0117] Step 701: Based on the features of any one of the embodiments, the resident / dynamic real-time segmentation of the AIGC model is obtained, and the resident layer in the resident / dynamic real-time segmentation of the AIGC model is loaded into the target instruction object.

[0118] In step 701, after obtaining the resident / dynamic real-time segmentation results, a server environment is deployed and configured in the preset storage space of the target instruction object. This includes ensuring that the Python environment, required libraries, and dependencies on the server are correctly installed and configured. Based on a deep learning framework, loading code corresponding to the AIGC model is written, and then the AIGC model is imported. After the AIGC model is imported into the target instruction object, the code for loading the model can be run on the server, and test data can be used to verify the model. This ensures that the model can be loaded and predicted normally.

[0119] Step 702: Receive the guidance information of the target instruction object, input the guidance information into the resident / dynamically real-time segmented AIGC model for training, temporarily load the dynamic layer in the AIGC model through the guidance information during the training process, and output the guidance result that matches the guidance information.

[0120] In step 703, when the target instruction object has an instruction requirement or under other circumstances, it can receive guidance information from the target instruction object. The guidance information usually refers to the initial input or guidance provided when using the AIGC model to generate content. This guidance information can be text, images, audio, or other forms of data, used to influence the results generated by the model. The guidance information is input into the resident / dynamically real-time segmented AIGC model for training. During the training process, based on the training requirements, the dynamic layer in the AIGC model with the requirements is temporarily loaded, and the parameters of the dynamic layer are referenced. Finally, the matching guidance result corresponding to the guidance information is obtained, which is the relevant output result required by the target instruction object. The guidance result can be in the form of images, videos, audio, etc.

[0121] The AIGC model in the embodiments of this specification, except for... Figure 7 In addition to the loading and training shown in the example, training can also be performed in other ways. For example, training can be performed locally by receiving guidance information from the target instruction object, or training can be performed by uploading to a cloud platform.

[0122] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0123] Please see Figure 8 The diagram shows a structural schematic of a model-splitting device according to an embodiment of this specification.

[0124] like Figure 8 As shown, the adaptive segmentation device of the AIGC model includes at least a swapping module training module 801, a parameter replacement module 802, a resident layer constraint calculation module 803, a preliminary segmentation module 804, a dynamic acquisition module 805, and a real-time segmentation module 806, wherein:

[0125] The exchange module training module 801 is configured to train the training parameters of the AIGC model. The training parameters are trained through the exchange module, and the exchange loss function generated during the training process is used as the convergence function to obtain the exchange result of the layer parameters of the AIGC model.

[0126] The parameter replacement module 802 is configured to replace the layer parameters of the AIGC model based on the exchange result, and determine the reuse layer constraints after parameter replacement.

[0127] The resident layer constraint calculation module 803 is configured to obtain the preset storage space of the target instruction object and calculate the resident layer constraints based on the preset storage space.

[0128] The preliminary segmentation module 804 is configured to plan the resident / dynamic preliminary segmentation of the AIGC model using the resident layer constraints and reuse layer constraints as constraints.

[0129] The dynamic acquisition module 805 is configured to dynamically acquire resource usage information of the target instruction object;

[0130] The real-time segmentation module 806 is configured to perform segmentation training based on the resource usage information and constraints, and to use the adjustment loss function of the resident / dynamic initial segmentation as the convergence function to train the resident / dynamic real-time segmentation of the AIGC model.

[0131] In some possible embodiments, the model segmentation device further includes at least:

[0132] The training module for the switching layer is configured to use the training parameters as input data and input them into the switching layer prediction module to train the training switching layer of the AIGC model.

[0133] The exchange result training module is configured to take the training parameters and the training exchange layer as input data, input them into the dynamic exchange module, and train to obtain the exchange result of the layer parameters of the AIGC model.

[0134] In some possible embodiments, the model segmentation device further includes at least:

[0135] The training parameter input module is configured to input the training parameters as input data into the exchange layer prediction module, and the diffusion loss generated during training.

[0136] The parameter consistency loss training module is configured to take the training parameters and the training exchange layer as input data and input them into the dynamic exchange module to train and generate the parameter consistency loss of dynamic exchange.

[0137] In some possible embodiments, the model segmentation device further includes at least:

[0138] The time consumption acquisition module is configured to determine the layer vector corresponding to each layer of the AIGC model and acquire the resident layer time and dynamic layer time corresponding to the layer vector of each layer.

[0139] The conflict vector determination module is configured to determine the number of resident layer vectors in the layer vectors based on the resident layer constraints, and to determine the conflict vectors in the layer vectors based on the reuse layer constraints.

[0140] The linear programming module is configured to perform linear programming on the layer vectors corresponding to each layer based on the number of resident layer vectors, conflict vectors, and the resident layer time and dynamic layer time corresponding to the layer vectors, so as to determine the initial resident / dynamic segmentation of the AIGC model.

[0141] In some possible embodiments, the model segmentation device further includes at least:

[0142] The solution module is configured to minimize the total time consumption of the layer vectors of each layer of the AIGC model when performing the linear programming.

[0143] In some possible embodiments, the model segmentation device further includes at least:

[0144] The loss function determination module is adjusted to change the total time consumption of the layer vectors and the change in the segmentation state of the AIGC model's resident / dynamic initial segmentation.

[0145] In some possible embodiments, the model segmentation device further includes at least:

[0146] The resource usage information acquisition module is configured to acquire a preset periodic duration and periodically acquire the resource usage information of the target instruction object based on the preset periodic duration, or to acquire a preset instruction time point and acquire the resource usage information of the target instruction object at the instruction time point.

[0147] In one or more embodiments of this specification, the content generation method of the model is through Figure 9 The content generation device for the model shown has been trained.

[0148] like Figure 9 As shown, the content generation device for this model includes at least a cross-model loading module 901 and a guidance result output module 902, wherein:

[0149] The model loading module 901 is configured in the persistent / dynamic real-time segmentation of the AIGC model obtained by the features described in one or more embodiments, and loads the persistent layer in the persistent / dynamic real-time segmentation of the AIGC model into the target instruction object;

[0150] The guidance result output module 902 is configured to receive guidance information from the target instruction object, input the guidance information into the resident / dynamically real-time segmented AIGC model for training, temporarily load the dynamic layer in the AIGC model through the guidance information during the training process, and output guidance results that match the guidance information.

[0151] Please see Figure 10 The diagram shown is a structural schematic of an electronic device provided in an embodiment of this specification.

[0152] like Figure 10 As shown, the electronic device 1100 may include: at least one processor 1101, at least one network interface 1104, user interface 1103, memory 1105, and at least one communication bus 1102.

[0153] The communication bus 1102 can be used to realize the connection and communication of the above components.

[0154] The user interface 1103 may include buttons, and the optional user interface may also include a standard wired interface or a wireless interface.

[0155] The network interface 1104 may include, but is not limited to, Bluetooth modules, NFC modules, Wi-Fi modules, etc.

[0156] The processor 1101 may include one or more processing cores. The processor 1101 connects to various parts within the electronic device 1100 using various interfaces and lines. It executes instructions, programs, code sets, or instruction sets stored in the memory 1105, and calls data stored in the memory 1105 to perform various functions and process data within the routing device 1100. Optionally, the processor 1101 may be implemented using at least one hardware form of DSP, FPGA, or PLA. The processor 1101 may integrate one or more of the following: CPU, GPU, and modem. The CPU primarily handles the operating system, user interface, and applications; the GPU is responsible for rendering and drawing the content required for display; and the modem handles wireless communication. It is understood that the modem may also not be integrated into the processor 1101 and may be implemented as a separate chip.

[0157] The memory 1105 may include RAM or ROM. Optionally, the memory 1105 may include a non-transitory computer-readable medium. The memory 1105 may be used to store instructions, programs, code, code sets, or instruction sets. The memory 1105 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for at least one function (such as touch function, sound playback function, image playback function, etc.), instructions for implementing the above-described method embodiments, etc.; the data storage area may store data involved in the above-described method embodiments, etc. Optionally, the memory 1105 may also be at least one storage device located remotely from the aforementioned processor 1101. Figure 10As shown, the memory 1105, as a computer storage medium, may include an operating system, a network communication module, a user interface module, and application programs. The processor 1101 can be used to call the application programs stored in the memory 1105 and execute the methods in one or more of the above embodiments.

[0158] This specification also provides a computer-readable storage medium storing instructions that, when executed on a computer or processor, cause the computer or processor to perform the above-described instructions. Figure 3 or Figure 4 or Figure 5 or Figure 6 or Figure 7 or Figure 8 One or more steps in the illustrated embodiment. If the constituent modules of the above-described electronic device are implemented as software functional units and sold or used as independent products, they can be stored in the computer-readable storage medium.

[0159] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this specification are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in or transmitted through a computer-readable storage medium. The computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, Digital Subscriber Line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium accessible to a computer or a data storage device such as a server or data center that integrates one or more available media. The available media may be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., Digital Versatile Discs (DVDs)), or semiconductor media (e.g., Solid State Disks (SSDs)).

[0160] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. This program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the methods described above. The aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks. Unless otherwise specified, the technical features of this embodiment and its implementation can be combined arbitrarily.

[0161] The embodiments described above are merely preferred embodiments of this specification and are not intended to limit the scope of this specification. Any modifications and improvements made by those skilled in the art to the technical solutions of this specification without departing from the spirit of this specification should fall within the protection scope defined by the claims of this specification.

Claims

1. A method for segmenting a model, wherein, include: For the training parameters of the AIGC model, the training parameters are trained through the exchange module, and the exchange loss function generated during the training process is used as the convergence function to obtain the exchange result of the layer parameters of the AIGC model. The switching module includes: Switching layer prediction module and dynamic switching module; The step of training the training parameters through the exchange module includes: Using the training parameters as input data, the training switching layer of the AIGC model is trained by inputting it into the switching layer prediction module. Using the training parameters and the training exchange layer as input data, the data is input into the dynamic exchange module to train and obtain the exchange results of the layer parameters of the AIGC model. The exchange loss function includes: The training parameters are used as input data and input into the exchange layer prediction module to generate diffusion loss during training. Using the training parameters and the training exchange layer as input data, the data is input into the dynamic exchange module to train and generate parameter consistency loss for dynamic exchange. Based on the exchange results, the layer parameters of the AIGC model are replaced, and the reuse layer constraints after parameter replacement are determined. Obtain the preset storage space of the target instruction object, and calculate the resident layer constraints based on the preset storage space; Using the resident layer constraints and reuse layer constraints as constraints, the resident / dynamic initial segmentation of the AIGC model is planned; Dynamically obtain resource usage information of the target instruction object; Based on the resource usage information and constraints, the AIGC model is trained by segmentation, using the adjusted loss function of the initial resident / dynamic segmentation as the convergence function, to obtain the resident / dynamic real-time segmentation of the AIGC model.

2. The method according to claim 1, wherein, The step of planning the initial resident / dynamic segmentation of the AIGC model using the resident layer constraints and reuse layer constraints as constraints includes: Determine the layer vector corresponding to each layer of the AIGC model, and obtain the resident layer time and dynamic layer time corresponding to the layer vector of each layer. The number of resident layer vectors in the layer vectors is determined based on the resident layer constraints, and the conflict vectors in the layer vectors are determined based on the reuse layer constraints. Based on the number of resident layer vectors, conflict vectors, and the resident layer time and dynamic layer time corresponding to the layer vectors, linear programming is performed on the layer vectors corresponding to each layer to determine the initial resident / dynamic segmentation of the AIGC model.

3. The method according to claim 2, wherein, The linear programming problem aims to minimize the total time consumption of the layer vectors at each layer of the AIGC model.

4. The method according to claim 3, wherein, The adjusted loss function includes: The change in the total time consumption of the layer vectors; The amount of change in the segmentation state of the resident / dynamic initial segmentation of the AIGC model.

5. The method according to claim 1, wherein, The dynamic acquisition of resource usage information of the target instruction object includes: Obtain a preset period duration, and periodically obtain resource usage information of the target instruction object based on the period duration; Alternatively, a preset instruction time point can be obtained, and resource usage information of the target instruction object can be obtained at the instruction time point.

6. A model segmentation device, wherein, include: The exchange module training module is configured to train the training parameters of the AIGC model. The training parameters are trained through the exchange module, and the exchange loss function generated during the training process is used as the convergence function to obtain the exchange result of the layer parameters of the AIGC model. The switching module includes: a switching layer prediction module and a dynamic switching module; The training module for the switching layer is configured to use the training parameters as input data and input them into the switching layer prediction module to train the training switching layer of the AIGC model. The exchange result training module is configured to take the training parameters and the training exchange layer as input data, input them into the dynamic exchange module, and train to obtain the exchange result of the layer parameters of the AIGC model; The training parameter input module is configured to input the training parameters as input data into the exchange layer prediction module, and the diffusion loss generated during training. The parameter consistency loss training module is configured to take the training parameters and the training exchange layer as input data, input them into the dynamic exchange module, and train to generate the parameter consistency loss of dynamic exchange. The parameter replacement module is configured to replace the layer parameters of the AIGC model based on the exchange result, and determine the reuse layer constraints after parameter replacement. The resident layer constraint calculation module is configured to obtain the preset storage space of the target instruction object and calculate the resident layer constraints based on the preset storage space. The preliminary segmentation module is configured to use the resident layer constraints and reuse layer constraints as constraints to plan the resident / dynamic preliminary segmentation of the AIGC model; The dynamic acquisition module is configured to dynamically acquire resource usage information of the target instruction object; The real-time segmentation module is configured to perform segmentation training based on the resource usage information and constraints, using the adjustment loss function of the resident / dynamic initial segmentation as the convergence function, and train to obtain the resident / dynamic real-time segmentation of the AIGC model.

7. The apparatus according to claim 6, wherein, include: The time consumption acquisition module is configured to determine the layer vector corresponding to each layer of the AIGC model and acquire the resident layer time and dynamic layer time corresponding to the layer vector of each layer. The conflict vector determination module is configured to determine the number of resident layer vectors in the layer vectors based on the resident layer constraints, and to determine the conflict vectors in the layer vectors based on the reuse layer constraints. The linear programming module is configured to perform linear programming on the layer vectors corresponding to each layer based on the number of resident layer vectors, conflict vectors, and the resident layer time and dynamic layer time corresponding to the layer vectors, so as to determine the initial resident / dynamic segmentation of the AIGC model.

8. The apparatus according to claim 7, wherein, include: The solution module is configured to minimize the total time consumption of the layer vectors of each layer of the AIGC model when performing the linear programming.

9. The apparatus according to claim 8, wherein, include: The loss function determination module is adjusted to change the total time consumption of the layer vectors and the change in the segmentation state of the AIGC model's resident / dynamic initial segmentation.

10. The apparatus according to claim 6, wherein, include: The resource usage information acquisition module is configured to acquire a preset periodic duration and periodically acquire the resource usage information of the target instruction object based on the preset periodic duration, or to acquire a preset instruction time point and acquire the resource usage information of the target instruction object at the instruction time point.

11. A method for generating content for a model, wherein, include: Based on the method of any one of claims 1 to 5, the resident / dynamic real-time segmentation of the AIGC model is obtained, and the resident layer in the resident / dynamic real-time segmentation of the AIGC model is loaded into the target instruction object; The system receives guidance information from the target instruction object, inputs the guidance information into the resident / dynamically real-time segmented AIGC model for training, temporarily loads the dynamic layer in the AIGC model through the guidance information during the training process, and outputs guidance results that match the guidance information.

12. A model content generation apparatus, wherein, include: The model loading module is configured to load the resident / dynamic real-time segmentation of the AIGC model obtained by the method according to any one of claims 1 to 5 into the target instruction object. The guidance result output module is configured to receive guidance information from the target instruction object, input the guidance information into the resident / dynamically real-time segmented AIGC model for training, temporarily load the dynamic layer in the AIGC model through the guidance information during the training process, and output guidance results that match the guidance information.

13. An electronic device, comprising a processor and a memory; The processor is connected to the memory; The memory is used to store executable program code; The processor runs a program corresponding to the executable program code stored in the memory to perform the method as described in any one of claims 1-5 and 11.

14. A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method as described in any one of claims 1-5 and 11.

Images