Model construction method and device, image generation method and device, equipment and medium

Abstract

The invention discloses a model construction method and device, an image generation method and device, equipment and a medium, and the method comprises the steps: carrying out the training of a super network through a training data set after the training data set is obtained, so as to enable each sub-network in the trained super network to have better image generation performance; searching a teacher model and a student model from the trained super network, so that the teacher model and the student model also have relatively good image generation performance; and finally, performing model distillation according to the training data set, the teacher model and the student model to obtain an image generation model, so that the image generation model not only has relatively good image generation performance, but also is relatively low in calculation complexity, and the image generation performance is relatively high. Therefore, the image generation model has relatively high model efficiency on the premise of ensuring the performance, so that the image generation model can be deployed on equipment with limited hardware.

Description

Technical Field

[0001] The present application relates to the field of image generation technology, and in particular to a model building method, image generation method, device, equipment, and medium. Background Art

[0002] For some image generation tasks, the task can be implemented with the help of pre-built models, such as the Generative adversarial networks (GAN) model.

[0003] In addition, for models with good image generation performance, they usually need to rely on explosive computational complexity, which makes deploying such resource-intensive models a daunting task, especially when hardware is limited. Therefore, how to alleviate this problem has become a technical problem that needs to be solved urgently. Summary of the invention

[0004] The present application provides a model building method, image generation method, device, equipment, and medium.

[0005] In order to achieve the above objectives, the technical solutions provided by this application are as follows:

[0006] The present application provides a model building method, the method comprising:

[0007] Acquire a training data set; the training data set includes a first sample image and label information corresponding to the first sample image;

[0008] Using the training data set to train the hypernetwork;

[0009] Searching for teacher and student models from the trained hypernetwork;

[0010] Model distillation is performed based on the training data set, the teacher model, and the student model to obtain an image generation model.

[0011] In a possible implementation manner, the training process of the hypernetwork includes:

[0012] determining a teacher generator and a student generator from the hypernetwork;

[0013] Model distillation is performed based on the training data set, the teacher generator and the student generator to obtain a trained super network.

[0014] In a possible implementation manner, determining the teacher generator and the student generator from the hypernetwork includes:

[0015] Sampling a first sub-network and a second sub-network from the super-network; the network size of the first sub-network is larger than the network size of the second sub-network;

[0016] Determine the teacher generator according to the first sub-network;

[0017] Based on the second sub-network, the student generator is determined.

[0018] In a possible implementation manner, the first subnetwork includes a largest subnetwork;

[0019] The second sub-network includes at least one of a minimum sub-network and a random sub-network.

[0020] In a possible implementation manner, the training data set further includes a second sample image;

[0021] The performing of model distillation according to the training data set, the teacher generator and the student generator to obtain a trained super network includes:

[0022] Determine loss representation data of the teacher generator using the first sample image and label information corresponding to the first sample image;

[0023] Determine loss representation data of the student generator using the second sample image and a first pseudo label corresponding to the second sample image; the first pseudo label is obtained by processing the second sample image by the teacher generator;

[0024] The hypernetwork, the teacher generator, and the student generator are updated according to the loss characterization data of the teacher generator and the loss characterization data of the student generator.

[0025] In a possible implementation manner, after updating the hypernetwork, the teacher generator, and the student generator, the method further includes:

[0026] Continue to perform the step of determining the loss representation data of the teacher generator using the first sample image and the label information corresponding to the first sample image until a first stopping condition is reached.

[0027] In a possible implementation manner, the training data set further includes a second sample image;

[0028] The performing of model distillation according to the training data set, the teacher generator and the student generator to obtain a trained super network includes:

[0029] updating the teacher generator using the first sample image and the label information corresponding to the first sample image;

[0030] Determine loss representation data of the student generator using the second sample image and a second pseudo label corresponding to the second sample image; the second pseudo label is obtained by processing the second sample image by the updated teacher generator;

[0031] According to the loss characterization data of the student generator, the hypernetwork, the teacher generator and the student generator are updated, and the step of updating the teacher generator using the first sample image and the label information corresponding to the first sample image is continued until the second stop condition is reached.

[0032] In a possible implementation manner, searching for a teacher model and a student model from a trained hypernetwork includes:

[0033] According to a first model constraint, searching for the student model from the trained hypernetwork so that the student model satisfies the first model constraint;

[0034] According to the second model constraint, at least one teacher model is searched from the trained hypernetwork so that each of the teacher models satisfies the first model constraint and the at least one teacher model satisfies a preset model architecture condition.

[0035] In a possible implementation manner, the at least one teacher model includes a teacher model that satisfies a preset model depth condition and a teacher model that satisfies a preset model width condition;

[0036] and / or,

[0037] The second model constraint is determined based on the first model constraint so that each of the teacher model and the student model satisfies a preset multiple condition on one or more model indicators.

[0038] In a possible implementation manner, the training data set further includes a second sample image;

[0039] The performing model distillation according to the training data set, the teacher model and the student model to obtain an image generation model includes:

[0040] updating the teacher model using the first sample image and label information corresponding to the first sample image;

[0041] The student model is updated using the second sample image and the third pseudo label corresponding to the second sample image, and the step of updating the teacher model using the first sample image and the label information corresponding to the first sample image is continued until the image generation model is determined according to the student model when the third stop condition is reached; the third pseudo label is obtained by processing the second sample image by the updated teacher model.

[0042] In a possible implementation manner, the updating the student model by using the second sample image and the third pseudo label corresponding to the second sample image includes:

[0043] The student model is updated using the first sample image, the fourth pseudo label corresponding to the first sample image, the second sample image, and the third pseudo label corresponding to the second sample image; the fourth pseudo label is obtained by processing the first sample image with the updated teacher model.

[0044] In a possible implementation manner, the updating process of the student model includes:

[0045] determining a first distillation loss of the student model using the second sample image and a third pseudo label corresponding to the second sample image;

[0046] Determining loss characterization data of the student model according to the first distillation loss;

[0047] The student model is updated according to the loss characterization data of the student model.

[0048] In a possible implementation manner, determining the loss characterization data of the student model according to the first distillation loss includes:

[0049] Determine the loss representation data of the student model according to the first distillation loss and the weighted weight corresponding to the first distillation loss; the weighted weight is determined according to the confidence of the third pseudo label; the confidence is determined according to the updated teacher model, the discriminator corresponding to the updated teacher model and the second sample image.

[0050] In a possible implementation manner, the process of determining the weighted weight includes:

[0051] If the confidence level is lower than the confidence threshold, randomly selecting a value from a preset interval and determining it as the weighted weight;

[0052] If the confidence level is not lower than the confidence threshold, a preset value is determined as the weighted weight.

[0053] In a possible implementation manner, the process of determining the confidence threshold includes:

[0054] Processing at least one image data using the updated teacher model to obtain a generated image corresponding to each image data; the at least one image data includes the second sample image and / or the first sample image;

[0055] Using the discriminator corresponding to the updated teacher model to perform discriminative processing on each of the generated images, to obtain a discriminative result corresponding to each of the generated images;

[0056] Data analysis is performed on at least one of the discrimination results corresponding to the generated image to obtain the confidence threshold.

[0057] The present application provides an image generation method, the method comprising:

[0058] Get the image to be processed;

[0059] The image to be processed is processed using a pre-constructed image generation model to obtain an image generation result; the image generation model is constructed using the model construction method provided in the present application.

[0060] The present application provides a model building device, comprising:

[0061] A first acquisition unit is used to acquire a training data set; the training data set includes a first sample image and label information corresponding to the first sample image;

[0062] A first training unit, used to train the super network using the training data set;

[0063] A network search unit, used to search for a teacher model and a student model from a trained hypernetwork;

[0064] The second training unit is used to perform model distillation based on the training data set, the teacher model and the student model to obtain an image generation model.

[0065] The present application provides an image generation device, comprising:

[0066] A second acquisition unit, used for acquiring an image to be processed;

[0067] The image generation unit is used to process the image to be processed using a pre-constructed image generation model to obtain an image generation result; the image generation model is constructed using the model construction method provided in the present application.

[0068] The present application provides an electronic device, the device comprising: a processor and a memory;

[0069] The memory is used to store instructions or computer programs;

[0070] The processor is used to execute the instructions or computer programs in the memory so that the electronic device executes the model building method or image generation method provided in the present application.

[0071] The present application provides a computer-readable medium, in which instructions or computer programs are stored. When the instructions or computer programs are executed on a device, the device executes the model building method or image generation method provided in the present application.

[0072] The present application provides a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, and the computer program contains program codes for executing the model building method or image generation method provided by the present application.

[0073] Compared with the prior art, this application has at least the following advantages:

[0074] In the technical solution provided by the present application, after obtaining a training data set (for example, some image samples with label information and some image samples without label information), the training data set is first used to train the super network so that each sub-network in the trained super network has good image generation performance (for example, image generation performance, etc.); then, a teacher model and a student model are searched from the trained super network so that the teacher model and the student model also have good image generation performance; finally, model distillation is performed based on the training data set, the teacher model and the student model to obtain an image generation model, so that the image generation model not only has good image generation performance, but also has a relatively low computational complexity, so that the image generation model has a relatively high model efficiency (for example, fewer resources are required when deploying the image generation model, etc.) while ensuring performance, so that the image generation model can be deployed on hardware-constrained devices.

[0075] In addition, since each sub-network in the trained super-network has good image generation performance, the teacher model and student model searched from the super-network have relatively good image generation performance, so that only fine-tuning of the teacher model and the student model is required in the future. This can effectively avoid the adverse effects caused by optimizing the teacher model and the student model from scratch (for example, slow model convergence speed, etc.), which is conducive to improving the efficiency of model training.

[0076] In addition, because the hypernetwork can provide a large number of various network structures, the present application can search for student models and teacher models that meet different constraints from the hypernetwork. This can effectively eliminate the additional resource consumption (such as human resource consumption, time consumption, etc.) caused by manual design and configuration of the teacher model, thereby helping to save resources. BRIEF DESCRIPTION OF THE DRAWINGS

[0077] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0078] Figure 1 A flowchart of a model building method provided in an embodiment of the present application;

[0079] Figure 2 A schematic diagram of a model building process provided in an embodiment of the present application;

[0080] Figure 3 A flowchart of an image generation method provided in an embodiment of the present application;

[0081] Figure 4 A schematic diagram of the structure of a model building device provided in an embodiment of the present application;

[0082] Figure 5 A schematic diagram of the structure of an image generating device provided in an embodiment of the present application;

[0083] Figure 6 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0084] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0085] In order to better understand the technical solution provided by this application, the model building method provided by this application is described below with reference to some drawings. Figure 1 As shown, the model building method provided in the embodiment of the present application includes the following S101-S104. Figure 1A flowchart of a model building method provided in an embodiment of the present application.

[0086] S101: Acquire a training data set; the training data set includes a first sample image and label information corresponding to the first sample image.

[0087] The training data set refers to the data set required for model training (for example, Figure 1 The dataset consists of labeled training data and unlabeled training data shown in ).

[0088] It should be noted that this application is not limited to the labeled training data in the above paragraph. For example, it can be used Among them, the represents a set of labeled training data; the {x i ,y i} represents the i-th labeled training data, i is a positive integer, 1≤i≤M, M is a positive integer, M represents the number of labeled training data; x i Represents the image data in the i-th labeled training data; y i Indicates that x i The corresponding label information can be obtained by manual annotation or other methods, which is not specifically limited in this application. i For example, it can be expressed using image data. i For example, in the field of image depth estimation, y i Can refer to x i The real depth map. For example, in the field of face recognition in images, y i Can refer to x i The real face label map of i It can indicate that the face area is in the x i For example, in the field of image style change, y i It can refer to an image in the target image domain whose domain style is the same as x i The data domain style of this image data is different, and this application does not limit the target image domain. For example, it can be an image domain with a watercolor style, an image domain with a cartoon style, an image domain with a stick figure style, or an image domain with a black and white style, etc.

[0089] It should also be noted that the present application is not limited to the unlabeled training data mentioned above. For example, it can be used Among them, the represents a set of unlabeled training data; z k represents the kth unlabeled training data, k is a positive integer, 1≤k≤NM, N is a positive integer, and N represents the number of training data in the above training data set.

[0090] In addition, the present application does not limit the implementation of the above training data set. For example, in some application scenarios (such as fully supervised training scenarios), the training data set may include a first sample image (such as x i ) and the label information corresponding to the first sample image (for example, y i ); wherein the first sample image refers to image data with label information required for use in the model training process; the label information corresponding to the first sample image refers to the label pre-annotated for the first sample image. It should be noted that the present application does not limit the number of the first sample images.

[0091] In fact, in order to reduce the difficulty of obtaining training data as much as possible, some labeled training data and some unlabeled training data can be used for model training, so that the model can not only learn knowledge from these labeled training data, but also learn structural features under a certain distribution domain from these unlabeled training data. Based on this, the present application also provides a possible implementation of the above training data set, in which the training data set can include not only the first sample image and the label information corresponding to the first sample image, but also the second sample image (for example, z k ). The second sample image refers to image data without label information required for model training. It should be noted that the present application does not limit the number of the second sample images.

[0092] It can be seen that in some application scenarios, the above training data set includes not only some labeled training data, but also some unlabeled training data, so that these unlabeled training data can bring some additional supervision information to the model training process, thereby enhancing the generalization of the model, which is beneficial for the model to learn the structural features under a certain distribution domain, so as to achieve the goal of increasing the amount of training data with as little labeling cost as possible.

[0093] In addition, this application does not limit the method for obtaining the above training data set.

[0094] S102: Train the hypernetwork using the training data set.

[0095] Among them, the hypernetwork is used to represent a search space (for example, Figure 2), so that the hypernetwork can provide some subnetworks with different network structures (for example, some subnetworks with different depths and widths, etc.). For example, the hypernetwork is used to represent the search space The G t represents a subnetwork sampled from the search space (ie, the supernetwork), t is a positive integer, and t≤the total number of subnetworks that the supernetwork can provide.

[0096] In fact, in order to better improve the model building effect, the present application provides a possible implementation of the above hypernetwork, in which the hypernetwork has dynamically adjustable depth and width, so that the depth and width of the hypernetwork can be dynamically adjusted along with the training process of the hypernetwork.

[0097] In addition, the present application does not limit the implementation method of S102 above. For example, when the training data set above only includes labeled training data, any existing or future method that can perform supervised training on a hypernetwork can be used for implementation. For another example, when the training data set above includes some labeled training data and some unlabeled training data, any existing or future method that can perform semi-supervised training on a hypernetwork can be used for implementation.

[0098] In fact, in order to better improve the model building effect, a distillation algorithm (eg, an online distillation algorithm) can be used to implement the training process for the hypernetwork. Based on this, the present application also provides a possible implementation of the above S102, which may specifically include the following steps 11-12.

[0099] Step 11: Determine the teacher generator and student generator from the hypernetwork.

[0100] The teacher generator refers to a sub-network with a larger network size sampled from a super-network; and the teacher generator has an image generation function (for example, an image generation function, etc.); and, the present application does not limit the teacher generator, for example, it can be Figure 2 The largest subnetwork shown. The network size is used to describe the size of a network in at least one dimension (e.g., depth and / or width) so that the network size can characterize the size of the network. It should be noted that the present application does not limit the implementation method of "at least one dimension", for example, it may include depth and / or width.

[0101] The student generator refers to another or more sub-networks sampled from the super network; and the student generator has an image generation function (for example, an image generation function, etc.); in addition, the network size of the student generator can be smaller than the network size of the above-mentioned teacher generator, so that the final trained student generator can represent the compression result for the teacher generator.

[0102] In addition, the present application does not limit the implementation of the above student generator. For example, the student generator can be Figure 2 As another example, the student generator may include Figure 2 The minimum subnetwork and random subnetwork shown. The random subnetwork refers to a subnetwork randomly sampled from the supernetwork. The minimum subnetwork is a subnetwork sampled from the supernetwork according to a pre-set minimum subnetwork sampling rule. It should be noted that the present application does not limit the minimum subnetwork sampling rule. For example, the minimum subnetwork sampling rule can be pre-set according to the application scenario.

[0103] In addition, the present application does not limit the implementation method of the above step 11. For example, it can be implemented by any existing or future method that can extract some sub-networks with different network sizes from the super network.

[0104] In fact, in order to better improve the training effect of the hypernetwork, the present application also provides a possible implementation of the above step 11, which may specifically include the following steps 111 to 113.

[0105] Step 111: sampling a first sub-network and a second sub-network from a super-network; the network size of the first sub-network is larger than the network size of the second sub-network.

[0106] The first subnetwork refers to a subnetwork sampled from the supernetwork and having a larger network size; and the present application does not limit the first subnetwork. For example, the first subnetwork may include the largest subnetwork sampled from the supernetwork.

[0107] The second subnetwork refers to a subnetwork sampled from the supernetwork and having a smaller network size; and the present application does not limit the second subnetwork. For example, the second subnetwork may include at least one of a minimum subnetwork and a random subnetwork sampled from the supernetwork. It can be seen that in a possible implementation, the second subnetwork may include a minimum subnetwork and one or more random subnetworks.

[0108] Step 112: Determine the teacher generator based on the first sub-network above.

[0109] It should be noted that the present application does not limit the implementation of step 112. For example, it can be specifically: after obtaining the first sub-network above, the first sub-network can be directly determined as the teacher generator. It can be seen that in one possible implementation, the teacher generator can refer to the largest sub-network sampled from the super network.

[0110] Step 113: Determine the student generator based on the second sub-network above.

[0111] It should be noted that the present application does not limit the implementation of step 113. For example, it can be specifically: after obtaining the second sub-network above, the second sub-network can be directly determined as the student generator. It can be seen that in a possible implementation, the student generator can include a minimum sub-network sampled from the super-network and one or more random sub-networks.

[0112] Based on the relevant content of step 11 above, it can be known that after obtaining the hypernetwork used to represent the search space, a teacher generator and a student generator can be sampled from the hypernetwork so that model distillation can be performed based on the teacher generator and the student generator to obtain a trained hypernetwork.

[0113] Step 12: Perform model distillation based on the training dataset, teacher generator, and student generator to obtain a trained hypernetwork.

[0114] It should be noted that the present application does not limit the implementation of step 12. For example, it can be implemented by any existing or future model distillation method (such as an online distillation algorithm, etc.). It can be seen that in a possible implementation, when the above training data set includes some labeled training data, step 12 can specifically be: using these labeled training data, the teacher generator and the student generator to perform model distillation to obtain a trained hypernetwork.

[0115] In fact, in order to better reduce the reliance of the training process of the hypernetwork on label information, the hypernetwork can be trained in a semi-supervised manner. Based on this, the present application also provides a possible implementation of step 12 above. In this implementation, when the training data set above includes a first sample image, label information corresponding to the first sample image, and a second sample image, step 12 can specifically include the following steps 121-123.

[0116] Step 121: Determine loss representation data of the teacher generator using the first sample image and the label information corresponding to the first sample image.

[0117] Among them, the loss characterization data of the teacher generator is used to characterize the image generation performance of the teacher generator (for example, image generation performance); and the present application does not limit the determination process of the loss characterization data of the teacher generator. For example, it can be implemented using any existing or future method that can determine the loss presented by the teacher generator on labeled training data.

[0118] In fact, in order to better improve the loss determination effect for the teacher generator, the present application also provides a possible implementation method of the loss characterization data determination process of the above teacher generator. In this implementation method, when the teacher generator is a generator in a generative adversarial network (GAN), the loss characterization data determination process of the teacher generator may specifically include the following steps 1211-1213.

[0119] Step 1211: Determine the adversarial loss corresponding to the teacher generator by using the first sample image, the label information corresponding to the first sample image, the teacher generator, and the discriminator corresponding to the teacher generator.

[0120] Among them, the discriminator corresponding to the teacher generator refers to the discriminator required to evaluate the performance of the teacher generator; and the discriminator corresponding to the teacher generator can be used to form a generative adversarial network with the teacher generator.

[0121] The adversarial loss corresponding to the teacher generator refers to the loss presented by the generative adversarial network composed of the teacher generator and the discriminator corresponding to the teacher generator.

[0122] In addition, the present application does not limit the determination process of the adversarial loss corresponding to the above teacher generator. For example, it can be implemented by using any existing or future method for determining the adversarial loss of any GAN (for example, the determination method shown in formula (1) below).

[0123]

[0124] Where, L GAN (G T ,D) represents the adversarial loss corresponding to the teacher generator above; G T represents the teacher generator; D represents the discriminator corresponding to the teacher generator; x represents sample image data (for example, the first sample image above); y represents the label information corresponding to the sample image data (for example, the label information corresponding to the first sample image); G T (x) indicates that the sample image data is input into the teacher generator G T Then the teacher generator G Tthe output image data; represents the expected function under the training data {x, y}; Represents the expected function under the sample image data.

[0125] Based on the relevant content of step 1211 above, it can be known that after obtaining the teacher generator G T After that, we can use some labeled training data The teacher generator G T , and the teacher generator G T The corresponding discriminator D determines the teacher generator G T The corresponding adversarial loss is used to optimize the teacher generator G based on the adversarial loss. T And the teacher generator G T The corresponding discriminator D, so that the final trained teacher generator G T Can realize x i Map to y i This goal enables the final trained discriminator D to achieve the goal that will be generated by the teacher generator G T The output generated images are distinguishable from real images.

[0126] Step 1212: Using the first sample image, the label information corresponding to the first sample image, and the teacher generator, determine the reconstruction loss corresponding to the teacher generator.

[0127] Among them, the reconstruction loss corresponding to the teacher generator is used to characterize the image generation performance of the teacher generator, so that the output image of the teacher generator can be made close to the true label with the help of the reconstruction loss.

[0128] In addition, the present application does not limit the process of determining the reconstruction loss corresponding to the above-mentioned teacher generator. For example, it can be implemented by adopting a method for determining the reconstruction loss of any existing or future image generator (for example, the determination method shown in formula (2) below).

[0129]

[0130] Where, L recon (G T ) represents the reconstruction loss corresponding to the teacher generator; G T represents the teacher generator; x represents sample image data (for example, the first sample image above); y represents the label information corresponding to the sample image data (for example, the label information corresponding to the first sample image); G T (x) indicates that the sample image data is input into the teacher generator G T Then the teacher generator G Tthe output image data; Represents the expected function under the training data {x,y}.

[0131] Based on the relevant content of step 1212 above, it can be known that when the label information corresponding to the first sample image above is image data, and the teacher generator above belongs to the generator in GAN, the first sample image above can be input into the teacher generator to obtain the generated image output by the teacher generator; and then based on the degree of similarity between the generated image and the label information corresponding to the first sample image, the reconstruction loss corresponding to the teacher generator is determined, so that the reconstruction loss can represent the image generation performance of the teacher generator, so that the teacher generator can be optimized based on the reconstruction loss subsequently, so that the optimized teacher generator has better image generation performance.

[0132] Step 1213: According to the adversarial loss corresponding to the teacher generator above and the reconstruction loss corresponding to the teacher generator, determine the loss representation data of the teacher generator (for example, Figure 2 The supervised loss shown in 1).

[0133] In the present application, after obtaining the adversarial loss corresponding to the above teacher generator and the reconstruction loss corresponding to the teacher generator, the loss representation data of the teacher generator can be determined based on the two losses (as determined by formula (3) below), so that the loss representation data can more comprehensively and accurately represent the image generation performance presented by the teacher generator.

[0134]

[0135] In the formula, Represents the loss representation data of the teacher generator above; L GAN (G T ,D) represents the adversarial loss corresponding to the teacher generator above; G T represents the teacher generator; D represents the discriminator corresponding to the teacher generator; L recon (G T ) represents the reconstruction loss corresponding to the teacher generator.

[0136] Based on the relevant content of steps 1211 to 1213 above, it can be known that for the teacher generator, some labeled training data (for example, the first sample image above and the label information corresponding to the first sample image) can be used to determine the loss (that is, the supervision loss) presented by the teacher generator on these labeled training data, so that the teacher generator can be optimized based on the loss subsequently.

[0137] Step 122: Determine loss representation data of the student generator using the second sample image and a first pseudo label corresponding to the second sample image; the first pseudo label is determined by the teacher generator through processing the second sample image.

[0138] Among them, the first pseudo label corresponding to the second sample image refers to the processing result of the second sample image by the above teacher generator, so that the processing result can be used as the pseudo label of the second sample image to guide the optimization process of the student generator. In this way, the purpose of the teacher generator guiding the student generator can be achieved, thereby completing the optimization processing of the student generator without relying on the discriminator.

[0139] The loss characterization data of the student generator is used to characterize the image generation performance of the student generator (for example, image generation performance); and the present application does not limit the determination process of the loss characterization data of the student generator. For example, it can be implemented using any existing or future method that can calculate the loss for the student generator (for example, online distillation loss, etc.).

[0140] In fact, in some application scenarios, in order to better improve the training effect of the super network, the student generator above may include at least two sub-networks sampled from the super network. Based on this, the present application also provides a possible implementation of step 122 above. In this implementation, when the student generator includes a minimum sub-network and a random sub-network, step 122 may specifically include the following steps 1221-1223.

[0141] Step 1221: Determine the online distillation loss of the minimum sub-network using the second sample image and the first pseudo-label corresponding to the second sample image, so that the online distillation loss can represent the image generation performance of the minimum sub-network.

[0142] It should be noted that the present application does not limit the implementation method of step 1221. For example, it can be implemented by any existing or future method that can calculate the online distillation loss of a model.

[0143] Step 1222: Determine the online distillation loss of the random sub-network using the second sample image and the first pseudo-label corresponding to the second sample image, so that the online distillation loss can represent the image generation performance of the random sub-network.

[0144] It should be noted that the present application does not limit the implementation method of step 1222. For example, it can be implemented by using any existing or future method that can calculate the online distillation loss of a model.

[0145] Step 1223: Determine the loss representation data of the student generator based on the online distillation loss of the minimum sub-network and the online distillation loss of the random sub-network.

[0146] In the present application, when the above student generator includes a minimum subnetwork and a random subnetwork, after obtaining the online distillation loss of the minimum subnetwork and the online distillation loss of the random subnetwork, the loss characterization data of the student generator can be determined based on these two losses (for example, it can be implemented using the following formula (4)).

[0147]

[0148] In the formula, G represents the loss representation data of the student generator; r represents a random subnetwork; G s represents the smallest subnetwork; L OD (G T (x u ),G r (x u )) represents the online distillation loss of the random subnetwork; x u represents unlabeled training data (e.g., the second sample image); G T (x u ) indicates that the x is processed by the teacher generator above u The image data generated when G is (for example, the first pseudo label corresponding to the second sample image); r (x u ) indicates that the x is processed by a random subnetwork u The image data generated when L OD (G T (x u ),G s (x u )) represents the online distillation loss of the smallest sub-network; G s (x u ) indicates that the x is processed by the smallest subnetwork u The image data generated when;.

[0149] Based on the relevant contents of steps 1221 to 1223 above, it can be known that in some application scenarios, if the student generator above includes multiple sub-networks, then under the guidance of the teacher generator above, the online distillation loss presented by each sub-network on these unlabeled training data can be first determined; and then the online distillation losses corresponding to all sub-networks are added together to obtain the loss representation data of the student generator, so that the loss representation data can better represent the image generation performance of the student generator.

[0150] Step 123: Update the hypernetwork, the teacher generator, and the student generator according to the loss characterization data of the teacher generator and the loss characterization data of the student generator.

[0151] In the present application, after obtaining the loss characterization data of the teacher generator and the loss characterization data of the student generator, the hypernetwork, teacher generator and student generator can be updated based on these two losses, so that the updated hypernetwork, updated teacher generator and updated student generator all have better image generation performance.

[0152] In addition, the present application does not limit the implementation method of the above step 123. For example, in some application scenarios, the step 123 may specifically include the following steps 1231-1233.

[0153] Step 1231: Determine the loss characterization data of the hypernetwork based on the loss characterization data of the teacher generator and the loss characterization data of the student generator, so that the loss characterization data can better represent the image generation performance of the hypernetwork.

[0154] It should be noted that the present application does not limit the implementation method of step 1231. For example, it can be implemented using the following formula (5).

[0155]

[0156] Where, L hypernetwork Representing the loss representation data of the hypernetwork; represents the loss representation data of the teacher generator; Represents the loss representation data of the student generator.

[0157] Step 1232: Update the hypernetwork according to the loss characterization data of the hypernetwork.

[0158] In the present application, after obtaining the loss characterization data of the hypernetwork, the hypernetwork can be updated (especially, the depth and / or width of the hypernetwork is updated) based on the loss characterization data of the hypernetwork, so that the updated hypernetwork has better image generation performance. It should be noted that the present application does not limit the update method of the hypernetwork.

[0159] Step 1233: Determine the teacher generator and the student generator from the updated hypernetwork to obtain an updated teacher generator and an updated student generator.

[0160] In the present application, after obtaining the updated supernetwork, some subnetworks (e.g., the maximum subnetwork, the minimum subnetwork, and the random subnetwork) can be resampled from the updated supernetwork, and based on these sampled subnetworks, an updated teacher generator and an updated student generator can be determined (e.g., the maximum subnetwork is determined as the updated teacher generator, and the minimum subnetwork and the random subnetwork are determined as the updated student generators).

[0161] Based on the relevant contents of steps 1231 to 1233 above, it can be known that after obtaining the loss characterization data of the teacher generator and the loss characterization data of the student generator, the loss characterization data of the above hypernetwork can be determined based on these two losses, so that the loss characterization data can better represent the image generation performance of the hypernetwork; and then based on the loss characterization data of the hypernetwork, the hypernetwork, the teacher generator and the student generator are updated, so that the updated hypernetwork, the updated teacher generator and the updated student generator all have better image generation performance.

[0162] Based on the relevant contents of steps 121 to 123 above, it can be known that in some application scenarios, the super network can be trained with the help of a semi-supervised method, and the process is roughly as follows: determine the loss of the largest subnetwork in the super network on some labeled training data sets, and use the largest subnetwork to guide the determination of the losses of some other subnetworks (such as the minimum subnetwork and random subnetwork, etc.) in the super network on some unlabeled training data sets, so that the super network can be optimized based on these two losses in the future. In this way, the super network can be optimized with the help of the mutual assistance between some subnetworks with relatively small network sizes and the largest subnetwork.

[0163] In fact, in order to better improve the image generation performance of the hypernetwork, the training process for the hypernetwork can be completed through multiple iterations. Based on this, the present application also provides another possible implementation of step 12 above, in which, when the training data set above includes a first sample image, label information corresponding to the first sample image, and a second sample image, step 12 includes not only steps 121 to 123 above, but also step 124 below.

[0164] Step 124: determine whether the first stop condition is met, if so, end the training process for the hypernetwork; if not, return to continue to execute the above step 121 and subsequent steps.

[0165] The first stop condition refers to the condition that needs to be achieved when the training process of the hypernetwork ends; and the present application does not limit the first stop condition, for example, it can be that the loss characterization data of the hypernetwork is lower than the first threshold, or the change rate of the loss characterization data of the hypernetwork is lower than the second threshold, or the number of updates of the hypernetwork reaches the third threshold. The first threshold, the second threshold and the third threshold can all be preset.

[0166] In addition, the present application does not limit the execution time of the above step 124. For example, the execution time of the above step 124 may be later than the execution time of the above step 123.

[0167] Based on the relevant contents of steps 121 to 124 above, it can be known that in some application scenarios, the super network can be trained with the help of a semi-supervised + iterative method. In this way, the super network can be continuously optimized with the help of the mutual assistance + progressive method between some sub-networks with relatively small network sizes and the largest sub-network in the super network, so that the trained super network has better image generation performance, and each sub-network in the trained super network also has better image generation performance.

[0168] In some application scenarios, the training process for the hypernetwork can be implemented by means of alternating updates of the teacher generator and the student generator. Based on this, the present application also provides another possible implementation of step 12 above. In this implementation, when the training data set above includes a first sample image, label information corresponding to the first sample image, and a second sample image, step 12 can specifically include the following steps 21 to 24.

[0169] Step 21: Update the teacher generator using the first sample image and the label information corresponding to the first sample image.

[0170] It should be noted that the present application does not limit the implementation method of step 21. For example, it can be specifically as follows: first, use the first sample image and the label information corresponding to the first sample image to determine the loss characterization data of the teacher generator; then, based on the loss characterization data of the teacher generator, update the teacher generator so that the updated teacher generator has better performance.

[0171] In fact, in some application scenarios, the hypernetwork can be updated along with the update of the teacher generator. Based on this, the present application does not limit the implementation method of step 21 above. For example, it can be specifically: using the first sample image and the label information corresponding to the first sample image to update the teacher generator and the hypernetwork. That is, in one possible implementation method, the loss characterization data of the teacher generator can be determined first using the first sample image and the label information corresponding to the first sample image; then, based on the loss characterization data of the teacher generator, the hypernetwork is updated so that the updated one has better model performance; then, a subnetwork (for example, the largest subnetwork) is sampled from the updated hypernetwork and determined as the updated teacher generator, so that the hypernetwork can be updated while the teacher generator is updated.

[0172] Step 22: Determine the loss representation data of the student generator using the second sample image and the second pseudo label corresponding to the second sample image; the second pseudo label is obtained by processing the second sample image by the updated teacher generator.

[0173] Among them, the second pseudo-label corresponding to the second sample image refers to the processing result of the second sample image by the updated teacher generator, so that the processing result can be used as the pseudo-label of the second sample image to guide the optimization process of the student generator. In this way, the purpose of guiding the student generator by the updated teacher generator can be achieved, thereby completing the optimization processing of the student generator without relying on the discriminator.

[0174] In addition, the implementation of step 22 is similar to the implementation of step 122 above, and for the sake of brevity, it will not be repeated here.

[0175] Step 23: Determine whether the second stop condition is met. If so, end the training process for the hypernetwork; if not, update the hypernetwork, teacher generator and student generator according to the loss representation data of the student generator, and return to continue executing step 21 and its subsequent steps.

[0176] The second stop condition refers to the condition that needs to be achieved when the training process of the hypernetwork ends; and the present application does not limit the second stop condition, for example, it can be that the loss characterization data of the hypernetwork is lower than the first threshold, or the change rate of the loss characterization data of the hypernetwork is lower than the second threshold, or the number of updates of the hypernetwork reaches the third threshold. The first threshold, the second threshold and the third threshold can all be preset.

[0177] In addition, the present application does not limit the implementation method of the above step "updating the super network, teacher generator and student generator according to the loss characterization data of the student generator". For example, it can be implemented using the implementation method shown in the above step 123. For another example, if the super network is updated at any time when the teacher generator is updated, then in order to better improve the model update efficiency, the update process can specifically include: first updating the super network according to the loss characterization data of the student generator; then sampling some sub-networks (for example, the maximum sub-network, the minimum sub-network and the random sub-network) from the updated super network, and determining the updated teacher generator and the updated student generator based on these sub-networks.

[0178] Based on the relevant content of steps 21 to 23 above, it can be known that in some application scenarios, another semi-supervised method can be used to train the super network. The process is roughly as follows: first, the largest subnetwork in the super network is updated on some labeled training data sets, and then the updated largest subnetwork is used to guide the training process of some other subnetworks in the super network (for example, the minimum subnetwork and random subnetwork, etc.) on some unlabeled training data sets, so that these other subnetworks can better absorb knowledge from these unlabeled training data and further optimize the super network through the weight sharing mechanism. In this way, the super network can be optimized by means of this alternating update method between some subnetworks with relatively small network sizes and the largest subnetwork.

[0179] Based on the relevant content of steps 11 to 12 above, it can be known that after constructing a supernetwork whose depth and width can change dynamically, supervised training can be performed on the largest subnetwork in the supernetwork, and under the guidance of the largest subnetwork, the training process of some other subnetworks in the supernetwork (for example, the smallest subnetwork and the random subnetwork, etc.) on some unlabeled training data can be completed, so that these other subnetworks can absorb knowledge from these unlabeled training data. In this way, the supernetwork can be continuously optimized by means of the mutual assistance strategy and progressive collaboration between some subnetworks with smaller network sizes in the supernetwork and the largest subnetwork, so that the trained supernetwork has better image generation performance, and thus each subnetwork in the trained supernetwork also has better image generation performance.

[0180] S103: Searching for a teacher model and a student model from the trained hypernetwork.

[0181] The teacher model refers to a sub-network searched from a trained super-network so that the teacher model can guide the training process of the student model.

[0182] The student model refers to another sub-network searched from the trained super-network so that the structure of the student model is simpler than that of the teacher model.

[0183] In addition, the present application does not limit the implementation of the above S103. For example, it may specifically include the following steps 31 and 32.

[0184] Step 31: According to the first model constraint, searching for a student model from the trained hypernetwork so that the student model satisfies the first model constraint.

[0185] Among them, the first model constraint refers to the constraint information required to search for the student model from the trained hypernetwork; and the first model constraint can be constraint information set in advance based on the application scenario (for example, in advance based on the configuration information of the terminal device deployed by the image generation model below, etc.), for example, it can be that the model running time is lower than a pre-set fourth threshold.

[0186] In addition, the present application does not limit the implementation of step 31. For example, it can be specifically: according to the first model constraint, using the evolutionary search algorithm to search for a student model from the trained hypernetwork so that the student model satisfies the first model constraint. It should be noted that the present application does not limit the implementation of the evolutionary search algorithm.

[0187] Step 32: Based on the second model constraint, search for at least one teacher model from the trained hypernetwork so that each teacher model satisfies the first model constraint, and the at least one teacher model satisfies the preset model architecture condition.

[0188] Among them, the second model constraint refers to the constraint information required to search for the teacher model from the trained hypernetwork; and the second model constraint can be the constraint information set in advance according to the application scenario, for example, it can be that the model running time is lower than the pre-set fifth threshold.

[0189] In fact, in order to better improve the training effect of the student model under the guidance of the teacher model, the present application also provides a determination process of the second model constraint above, for example, it can be specifically: according to the first model constraint above, the second model constraint is determined so that the teacher model that satisfies the second model constraint and the student model that satisfies the first model constraint meet the preset multiple conditions on one or more model indicators (for example, the computational cost of the teacher model is approximately 21 times the computational cost of the student model, etc.). Among them, the model indicator can be pre-set, for example, it can be the computational cost. The preset multiple condition can be determined based on the actual application scenario.

[0190] In the above step 32, "at least one teacher model" is used to represent the teacher model that is searched from the trained hypernetwork according to the second model constraint and is used to guide the training process of the student model; and the at least one teacher model satisfies the preset model architecture conditions so that the at least one teacher model can guide the training process of the student model in at least one dimension (for example, depth + width and other dimensions). It should be noted that the "at least one dimension" refers to one or more dimensions that are pre-set for the student model and need to be guided by the teacher model; and the present application does not limit the "at least one dimension", for example, it may include depth and / or width. It can be seen that under one possible implementation, the teacher model for model depth guidance and the teacher model for model width guidance can be used to participate in the training process for the student model so that the trained student model has better performance.

[0191] Among them, the preset model architecture conditions can be set according to the application scenario, and the present application does not limit the preset model architecture conditions. For example, the model architecture of at least one teacher model that meets the preset model architecture conditions can present a complementary state.

[0192] In addition, the present application does not limit the implementation method of the preset model architecture condition in the above paragraph. For example, the preset model architecture condition may include a preset model depth condition and a preset model width condition, so that at least one teacher model that meets the preset model architecture condition includes a teacher model that meets the preset model depth condition (for example, the above "teacher model for model depth guidance") and a teacher model that meets the preset model width condition (for example, the above "teacher model for model width guidance"), so that the at least one teacher model presents an architectural complementary state. Among them, the preset model depth condition refers to the condition required to be based on when searching for a teacher model for guiding the training process of the student model under the model depth; and the present application does not limit the preset model depth condition. For example, the preset model depth condition can be set in advance according to the actual application scenario, so that the teacher model that meets the preset model depth condition can be used later to guide the training process of the student model under the dimension of model depth. The preset model width condition refers to the condition required when searching for a teacher model for guiding the training process of the student model under the model width; and the present application does not limit the preset model width condition. For example, the preset model width condition can be set in advance based on the actual application scenario, so that the teacher model that meets the preset model width condition can be used later to guide the training process of the student model under the dimension of model width.

[0193] Based on the relevant contents of steps 31 to 32 above, it can be known that for some application scenarios, after obtaining the trained super network, an evolutionary search algorithm can be used to search out a depth-based teacher model, a width-based teacher model, and a student model from the super network, so that the depth-based teacher model can be used to guide the optimization adjustment of the student model in depth, and the width-based teacher model can be used to guide the optimization adjustment of the student model in width. Among them, because the trained super network has good image generation performance, each sub-network in the super network also has good image generation performance, so that the two teacher models and the student model searched from the super network have good performance, so that the adverse effects caused by optimizing the model from scratch (for example, the model convergence speed is relatively slow), so that the model training effect can be effectively improved (for example, the model convergence speed is accelerated). Among them, the depth-based teacher model refers to the teacher model that meets the preset model depth conditions above, so that the depth-based teacher model is used to guide the training process of the student model under the dimension of model depth. The width-based teacher model refers to the teacher model that meets the preset model depth conditions above, so that the width-based teacher model is used to guide the training process of the student model under the dimension of model width.

[0194] Also, because the trained hypernetwork can provide sub-networks that meet different constraints, the present application can use the hypernetwork to determine the various teacher models and student models that meet different constraints. This can effectively avoid the adverse effects caused by manually designing multiple teacher models and student models (for example, it takes a long time, etc.), thereby helping to improve the model building effect.

[0195] S104: Perform model distillation based on the training data set, the teacher model, and the student model to obtain an image generation model.

[0196] Among them, the image generation model is used to generate images for the input data of the image generation model; and the present application does not limit the implementation method of the image generation model. For example, in some application scenarios, the image generation model belongs to an image generator, so that the image generation model can generate a new image data based on the input data of the image generation model.

[0197] In addition, the present application does not limit the implementation method of the model distillation in S104. For example, it can be implemented using any existing or future model distillation method (for example, an online distillation method).

[0198] In fact, in order to better reduce the dependence of the model distillation process on label information, the present application also provides a possible implementation of the above S104. In this implementation, when the above training data set includes a first sample image, label information corresponding to the first sample image, and a second sample image, the S104 can specifically include the following steps 41-45.

[0199] Step 41: Update the teacher model using the first sample image and the label information corresponding to the first sample image.

[0200] It should be noted that the present application does not limit the implementation of step 41. For example, it may specifically be: firstly use the first sample image and the label information corresponding to the first sample image to determine the loss representation data of the teacher model (for example, Figure 2 The supervised loss 2 shown in the figure); then based on the loss representation data of the teacher model, the teacher model is updated so that the updated teacher model has better performance.

[0201] Step 42: Update the student model using the second sample image and the third pseudo label corresponding to the second sample image; the third pseudo label is obtained by processing the second sample image by the updated teacher model.

[0202] Among them, the third pseudo-label corresponding to the second sample image refers to the processing result of the second sample image by the updated teacher model, so that the processing result can be used as the pseudo-label of the second sample image to guide the optimization process of the student model. In this way, the purpose of guiding the student model by the updated teacher model can be achieved, thereby completing the optimization processing of the student model without relying on the discriminator.

[0203] In addition, the present application does not limit the implementation method of step 42. For example, it can be implemented using the student model update process involved in any existing or future model distillation method (for example, an online distillation method). It can be seen that in one possible implementation, step 42 can specifically be: first use the second sample image and the third pseudo-label corresponding to the second sample image to determine the loss representation data of the student model; then update the student model based on the loss representation data of the student model. It should be noted that the process of determining the loss representation data of the student model is similar to the process of determining the loss representation data of the student generator above. For the sake of brevity, it will not be repeated here.

[0204] In fact, since the update process of the student model does not need to use real label information, in order to better improve the training effect of the student model, the present application also provides another possible implementation of step 42 above, which can be specifically: using the first sample image, the fourth pseudo label corresponding to the first sample image, the second sample image and the third pseudo label corresponding to the second sample image to update the student model, which can effectively increase the number of training data of the student model, thereby helping to improve the training effect of the student model; the fourth pseudo label is obtained by processing the first sample image by the updated teacher model.

[0205] In addition, the present application does not limit the updating process of the above student model. For example, it may specifically include the following steps 421 to 423.

[0206] Step 421: Determine a first distillation loss of the student model using the second sample image and the third pseudo label corresponding to the second sample image.

[0207] Among them, the first distillation loss is used to characterize the loss presented by the student model on unlabeled training data.

[0208] In addition, the present application does not limit the determination process of the first distillation loss. For example, when the above "at least one teacher model" includes a depth-based teacher model and a width-based teacher model, the first distillation loss can be implemented using the following formula (6).

[0209]

[0210] In the formula, represents the first distillation loss of the student model; G * represents the student model; represents the online distillation loss presented by the student model on the unlabeled training data under the guidance of the depth-based teacher model; represents the online distillation loss of the student model on the unlabeled training data under the guidance of the width-based teacher model; represents unlabeled training data; Indicates that the depth-based teacher model processes the x u The image data generated when Indicates that the student model processes the x u The image data generated when Indicates that the width-based teacher model processes the x u The image data generated when .

[0211] Step 422: Determine loss characterization data of the student model according to the first distillation loss of the student model.

[0212] It should be noted that the present application does not limit the implementation method of step 422. For example, it may specifically be: directly determining the first distillation loss of the student model as the loss representation data of the student model.

[0213] For another example, when the above student model is trained with the aid of labeled training data and unlabeled training data, the process of determining the loss representation data of the student model may include the following steps 51 and 52.

[0214] Step 51: Determine a second distillation loss of the student model using the first sample image and a fourth pseudo label corresponding to the first sample image.

[0215] Among them, the fourth pseudo-label corresponding to the first sample image refers to the processing result of the first sample image by the updated teacher model, so that the processing result can be used as the pseudo-label of the first sample image to guide the optimization process of the student model. In this way, the purpose of guiding the student model by the updated teacher model can be achieved, thereby completing the optimization processing of the student model without relying on the discriminator.

[0216] The second distillation loss is used to characterize the loss exhibited by the student model on labeled training data.

[0217] In addition, the present application does not limit the determination process of the second distillation loss. For example, when the above "at least one teacher model" includes a depth-based teacher model and a width-based teacher model, the second distillation loss can be implemented using the following formula (7).

[0218]

[0219] In the formula, represents the second distillation loss of the student model; G * represents the student model; represents the online distillation loss presented by the student model on the labeled training data under the guidance of the depth-based teacher model; represents the online distillation loss of the student model on the labeled training data under the guidance of the width-based teacher model; Represents labeled training data; Indicates that the depth-based teacher model processes the The image data generated when Indicates that the student model handles the The image data generated when Indicates that the width-based teacher model processes the The image data generated when .

[0220] It should be noted that the present application does not limit the execution order between step 51 and the above step 421.

[0221] Step 52: Determine loss characterization data of the student model according to the first distillation loss of the student model and the second distillation loss of the student model.

[0222] It should be noted that the present application does not limit the implementation method of the above step 52. For example, it may specifically be: adding the first distillation loss of the student model and the second distillation loss of the student model to obtain the loss characterization data of the student model.

[0223] Based on the relevant contents of steps 51 to 52 above, it can be known that in some application scenarios, under the guidance of multiple teacher models, the distillation loss presented by the student model in the unlabeled training data and the distillation loss presented by the student model in the labeled training data can be first determined; and then the two losses are combined to determine the loss representation data of the student model, so that the loss representation data can better represent the performance of the student model.

[0224] In fact, because the image generation ability of the teacher model may fluctuate, some pseudo-labels generated based on the teacher model are not very credible. Therefore, in order to avoid these pseudo-labels with low credibility from interfering with the update process of the student model, the impact of these pseudo-labels with low credibility on the update process of the student model can be weakened as much as possible. Based on this, the present application also provides a possible implementation of step 422 above, which can be specifically: determining the loss characterization data of the student model based on the first distillation loss of the student model and the weighted weight corresponding to the first distillation loss. Among them, the weighted weight is determined according to the confidence of the third pseudo-label; the confidence of the third pseudo-label is determined based on the updated teacher model, the discriminator corresponding to the updated teacher model, and the second sample image.

[0225] Among them, the confidence of the third pseudo-label is used to characterize the credibility of the third pseudo-label; and the present application does not limit the determination process of the confidence of the third pseudo-label. For example, it can be specifically: first input the second sample image into the updated teacher model to obtain the third pseudo-label output by the teacher model; then the discriminator corresponding to the updated teacher model performs a discriminant process on the third pseudo-label to obtain a discriminant result of the third pseudo-label, and determines the discriminant result as the confidence of the third pseudo-label.

[0226] The weighted weight corresponding to the first distillation loss is used to characterize the influence of the first distillation loss. Moreover, the present application does not limit the determination process of the weighted weight corresponding to the first distillation loss. For example, it may specifically include the following steps 61-62.

[0227] Step 61: If the confidence of the third pseudo-label is lower than the confidence threshold, a value is randomly selected from a preset interval and determined as the weighted weight corresponding to the first distillation loss above.

[0228] The confidence threshold is used to describe the dividing line between credible and uncredible. The present application does not limit the process of determining the confidence threshold. For example, it can be pre-set according to the application scenario.

[0229] In addition, in order to better improve the rationality of the weights, the present application also provides a method for automatically determining a confidence threshold, which may specifically include the following steps 71 to 73.

[0230] Step 71: Process at least one image data using the updated teacher model to obtain a generated image corresponding to each image data; the at least one image data includes a second sample image and / or a first sample image.

[0231] Among them, at least one image data refers to image data required for reference when automatically determining the confidence threshold; and the present application does not limit the implementation method of the at least one image data, for example, it may include the second sample image and / or the first sample image mentioned above.

[0232] Step 72: Use the discriminator corresponding to the updated teacher model to perform discriminative processing on each generated image to obtain a discriminative result corresponding to each generated image.

[0233] The discriminator corresponding to the updated teacher model refers to the discriminator involved in the update process of the teacher model.

[0234] Step 73: Perform data analysis and processing on the discrimination result corresponding to at least one generated image to obtain a confidence threshold.

[0235] The data analysis process refers to a pre-set data analysis method; and the present application does not limit the implementation method of the data analysis process. For example, it can be implemented by any existing or future data analysis method (for example, maximum value, minimum value, mean value, median value, or exponential moving average value, etc.). It can be seen that in a possible implementation, the confidence threshold can refer to the result obtained by performing an exponential moving average (EMA) analysis on the discrimination results corresponding to all generated images.

[0236] Based on the relevant contents of steps 71 to 73 above, it can be known that in some application scenarios, after obtaining the updated teacher model, the teacher model can be used to process each sample image in the training data set to obtain the generated image corresponding to each sample image; then the discriminator corresponding to the teacher model is used to perform discrimination processing on each generated image to obtain the discrimination results of each generated image; finally, all these discrimination results are analyzed to obtain a confidence threshold, so that the confidence threshold can better describe the boundary between credible and unreliable.

[0237] The preset interval refers to a preset value selection interval, so that the above "randomly select a value in the preset interval" is greater than or equal to the minimum value in the preset interval, and the "randomly select a value in the preset interval" is less than or equal to the maximum value in the preset interval; and the present application does not limit the preset interval, for example, any value in the preset interval is less than or equal to a preset first value (for example, 1, etc.). For another example, any value in the preset interval is greater than or equal to a preset second value (for example, 0, etc.). For another example, when the second value is less than the first value, the preset interval can be the data interval of [second value, first value] (for example, the data interval of [0, 1]).

[0238] Based on the relevant content of step 61 above, it can be known that for the third pseudo-label corresponding to the second sample image, if the confidence of the third pseudo-label is lower than the confidence threshold, it can be determined that the credibility of the third pseudo-label is not very high. Therefore, in order to weaken the impact of the third pseudo-label on the update process of the student model, a value can be randomly selected from a preset interval (for example, a value is randomly selected between 0 and 1) as the weighted weight corresponding to the third pseudo-label, so that the weighted weight can represent the impact of the loss determined based on the third pseudo-label on the update process of the student model.

[0239] Step 62: If the confidence of the third pseudo-label is not lower than the confidence threshold, the preset value is determined as the weighted weight corresponding to the first distillation loss above.

[0240] In the present application, for the third pseudo-label corresponding to the second sample image, if the confidence of the third pseudo-label is higher than or equal to the confidence threshold, it can be determined that the credibility of the third pseudo-label is relatively high. Therefore, in order to maintain or even improve the impact of the third pseudo-label on the update process of the student model, a preset value (for example, 1) can be directly used as the weighted weight corresponding to the third pseudo-label, so that the weighted weight can represent the impact of the loss determined based on the third pseudo-label on the update process of the student model.

[0241] It should be noted that the present application does not limit the implementation of the above preset value. For example, the preset value is not lower than the highest value in the above preset range. For another example, the preset value may be 1.

[0242] Based on the relevant contents of steps 61 to 62 above, it can be known that in some application scenarios, when the first distillation loss above is determined based on the third pseudo-label corresponding to the second sample image, the weighted weight corresponding to the first distillation loss can be determined based on the confidence of the third pseudo-label. In this way, it is possible to weaken the impact of pseudo-labels with lower confidence on the update process of the student model, thereby better improving the model update effect.

[0243] In addition, the present application does not limit the implementation method of the above step of "determining the loss characterization data of the student model according to the first distillation loss of the student model and the weighted weight corresponding to the first distillation loss". For example, it can be implemented using the following formulas (8)-(10).

[0244]

[0245]

[0246]

[0247] In the formula, represents the first distillation loss of the student model; G * represents the student model; represents the online distillation loss presented by the student model on the unlabeled training data under the guidance of the depth-based teacher model; represents unlabeled training data; Indicates that the depth-based teacher model processes the x u The image data generated when Indicates that the student model processes the x u The image data generated when express The corresponding weighted weights; represents the online distillation loss of the student model on the unlabeled training data under the guidance of the width-based teacher model; express The corresponding weighted weights; Indicates that the width-based teacher model processes the x u The image data generated when Indicates that the corresponding discriminator is processed by the depth-based teacher model for The decision result (e.g., prediction score) given by this generated image; Represents the discriminator corresponding to the width-based teacher model for The judgment result given by this generated image; represents the EMA value of the judgment result given by the discriminator corresponding to the depth-based teacher model for the generated image output by the depth-based teacher model; It represents the EMA value of the judgment result given by the discriminator corresponding to the width-based teacher model for the generated image output by the width-based teacher model; RandInt([0,1]) represents a random selection of a value from 0 to 1.

[0248] Based on the relevant content of step 422 above, it can be known that in some application scenarios, after obtaining the first distillation loss of the student model, the loss characterization data of the student model can be determined based on the first distillation loss, so that the loss characterization data can better represent the image generation performance of the student model.

[0249] Step 423: Update the student model according to the loss characterization data of the student model.

[0250] It should be noted that the present application does not limit the implementation method of the above step 423.

[0251] Based on the relevant contents of steps 421 to 423 above, it can be known that in one possible implementation, after obtaining multiple updated teacher models, the updating process of the student model can be guided based on all training data (for example, the first sample image and the second sample image) and using these teacher models, so that the updated student model has better image generation performance.

[0252] Based on the relevant content of step 42 above, it can be seen that since the update process of the student model does not need to use real label information, all training data (for example, the first sample image and the second sample image) can be used to participate in the optimization process of the student model, which can better improve the model optimization effect.

[0253] Step 43: Determine whether the third stop condition is met, if so, execute the following step 45; if not, execute the following step 44.

[0254] The third stop condition refers to the condition that needs to be met when the training process of the student model ends; and the present application does not limit the third stop condition, for example, it can be that the loss characterization data of the student model is lower than the sixth threshold, or the change rate of the loss characterization data of the student model is lower than the seventh threshold, or the number of updates of the student model reaches the eighth threshold. The sixth threshold, the seventh threshold and the eighth threshold can all be preset.

[0255] Step 44: If the third stop condition is not met, return to continue executing the above step 41 and its subsequent steps.

[0256] In the present application, for the current round of training process for the student model, if it is determined that the third stop condition has not been met, it can be determined that the performance of the student model still needs to be improved, so the above step 41 and its subsequent steps can be re-executed based on the updated student model to start the next round of training process for the student model.

[0257] Step 45: If the third stop condition is reached, the image generation model is determined based on the student model.

[0258] In the present application, for the current round of training process for the student model, if it is determined that the third stop condition is reached, it can be determined that the performance of the student model is relatively good, so the image generation model can be determined based on the student model (for example, directly determining the student model as the image generation model, etc.), so that the image generation model can have better performance.

[0259] Based on the relevant contents of steps 41 to 45 above, it can be known that after searching for multiple teacher models and a student model from the trained hypernetwork, the training process for the student model can be implemented by alternately updating the teacher model and the student model, so that the final trained student model has better performance.

[0260] Based on the relevant contents of S101 to S104 above, it can be known that for the model building method provided in the embodiment of the present application, after obtaining a training data set (for example, some image samples with label information and some image samples without label information), the super network is first trained using the training data set so that each sub-network in the trained super network has good image generation performance (for example, image generation performance, etc.); then the teacher model and the student model are searched from the trained super network so that the teacher model and the student model also have good image generation performance; finally, model distillation is performed based on the training data set, the teacher model and the student model to obtain an image generation model, so that the image generation model not only has good image generation performance, but also has a relatively low computational complexity, so that the image generation model has a relatively high model efficiency (for example, fewer resources are required when deploying the image generation model, etc.) while ensuring performance, so that the image generation model can be deployed on hardware-constrained devices.

[0261] In addition, for the model building method provided in this application, since this method can realize the synergy of seamlessly promoting model efficiency and data efficiency learning by sequentially executing the semi-supervised driven network architecture search stage and the adaptive online semi-supervised distillation stage, it can achieve the purpose of unified compression of network data + model.

[0262] In addition, the present application provides a new learning paradigm with a unified optimization goal that can seamlessly promote the synergy of model-efficient and data-efficient learning. In this learning paradigm, semi-supervised-driven network architecture search and adaptive online semi-supervised distillation stages can be set in sequence to formulate heterogeneous mutual learning schemes to obtain models with flexible architecture, data efficiency, and excellent performance. A large number of experiments have shown that even with fewer labels, this learning paradigm can obtain state-of-the-art lightweight models, so that this learning paradigm can provide a feasible solution for deploying GAN models in hardware-constrained and low-data environments.

[0263] In addition, the present application does not limit the execution subject of the above model construction method. For example, the model construction method provided in the embodiment of the present application can be applied to a device with data processing function such as a terminal device or a server. For another example, the model construction method provided in the embodiment of the present application can also be implemented by means of a data communication process between different devices (for example, a terminal device and a server, two terminal devices, or two servers). Among them, the terminal device can be a smart phone, a computer, a personal digital assistant (PDA) or a tablet computer. The server can be a stand-alone server, a cluster server or a cloud server.

[0264] Based on the relevant content of the above model construction method, this application also provides an image generation method, which is described below in conjunction with the accompanying drawings. Figure 3 As shown, the image generation method provided in the embodiment of the present application includes the following S301-S302. Figure 3 A flowchart of an image generation method provided in an embodiment of the present application.

[0265] S301: Acquire an image to be processed.

[0266] The image to be processed refers to image data that needs to be generated with the help of the above-mentioned image generation model.

[0267] S302: Processing the image to be processed using an image generation model to obtain an image generation result; the image generation model is constructed using any implementation of the model construction method provided in the present application.

[0268] For more information about the image generation model, please refer to the above.

[0269] The image generation result refers to the result output by the image generation model after processing the above image to be processed.

[0270] Based on the above S301 to S302, it can be known that after obtaining the image to be processed, the image to be processed can be processed using the pre-built image generation model to obtain an image generation result. Among them, because the image generation model has good performance, the image generation result determined based on the image generation model is better, which is conducive to improving the image generation effect.

[0271] In addition, the present application does not limit the execution subject of the above image generation method. For example, the image generation method provided in the embodiment of the present application can be applied to a device with data processing function such as a terminal device or a server. For another example, the image generation method provided in the embodiment of the present application can also be implemented by means of a data communication process between different devices (for example, a terminal device and a server, two terminal devices, or two servers).

[0272] Based on the model building method provided in the embodiment of the present application, the embodiment of the present application also provides a model building device. Figure 4 Explain and illustrate. Figure 4 This is a schematic diagram of the structure of a model building device provided in an embodiment of the present application. It should be noted that for the technical details of the model building device provided in an embodiment of the present application, please refer to the relevant content of the model building method above.

[0273] like Figure 4 As shown, the model building device 400 provided in the embodiment of the present application includes:

[0274] A first acquisition unit 401 is used to acquire a training data set; the training data set includes a first sample image and label information corresponding to the first sample image;

[0275] A first training unit 402, configured to train the super network using the training data set;

[0276] A network search unit 403, used to search for a teacher model and a student model from a trained super network;

[0277] The second training unit 404 is used to perform model distillation based on the training data set, the teacher model and the student model to obtain an image generation model.

[0278] In a possible implementation manner, the first training unit 402 includes:

[0279] A first determination subunit, used to determine a teacher generator and a student generator from the super network;

[0280] The model distillation subunit is used to perform model distillation based on the training data set, the teacher generator and the student generator to obtain a trained super network.

[0281] In one possible implementation, the first determination subunit is specifically used to: sample a first subnetwork and a second subnetwork from the supernetwork; the network size of the first subnetwork is larger than the network size of the second subnetwork; determine the teacher generator based on the first subnetwork; and determine the student generator based on the second subnetwork.

[0282] In a possible implementation manner, the first subnetwork includes a maximum subnetwork; and the second subnetwork includes at least one of a minimum subnetwork and a random subnetwork.

[0283] In a possible implementation manner, the training data set further includes a second sample image;

[0284] The model distillation subunit is specifically used to: determine the loss representation data of the teacher generator using the first sample image and the label information corresponding to the first sample image; determine the loss representation data of the student generator using the second sample image and the first pseudo label corresponding to the second sample image; the first pseudo label is obtained by the teacher generator processing the second sample image; and update the hypernetwork, the teacher generator and the student generator according to the loss representation data of the teacher generator and the loss representation data of the student generator.

[0285] In one possible implementation, the model distillation subunit is further used to: after updating the hypernetwork, the teacher generator, and the student generator, continue to perform the step of determining the loss representation data of the teacher generator using the first sample image and the label information corresponding to the first sample image until a first stopping condition is reached.

[0286] In a possible implementation manner, the training data set further includes a second sample image;

[0287] The model distillation subunit is specifically used to: update the teacher generator using the first sample image and the label information corresponding to the first sample image; determine the loss representation data of the student generator using the second sample image and the second pseudo label corresponding to the second sample image; the second pseudo label is obtained by processing the second sample image by the updated teacher generator; update the hypernetwork, the teacher generator and the student generator according to the loss representation data of the student generator, and continue to execute the step of updating the teacher generator using the first sample image and the label information corresponding to the first sample image until the second stop condition is reached.

[0288] In one possible implementation, the network search unit 403 is specifically used to: search for the student model from the trained super network based on the first model constraint, so that the student model satisfies the first model constraint; and search for at least one from the trained super network based on the second model constraint, so that each of the teacher models satisfies the first model constraint, and the at least one teacher model satisfies the preset model architecture condition.

[0289] In one possible implementation, the at least one teacher model includes a teacher model that satisfies a preset model depth condition and a teacher model that satisfies a preset model width condition; and / or, the second model constraint is determined based on the first model constraint so that each of the teacher models and the student model satisfies a preset multiple condition on one or more model indicators.

[0290] In a possible implementation manner, the training data set further includes a second sample image;

[0291] The second training unit 404 is specifically used to: update the teacher model using the first sample image and the label information corresponding to the first sample image; update the student model using the second sample image and the third pseudo label corresponding to the second sample image, and continue to execute the step of updating the teacher model using the first sample image and the label information corresponding to the first sample image until the third stop condition is reached, and determine the image generation model according to the student model; the third pseudo label is obtained by processing the second sample image by the updated teacher model.

[0292] In one possible implementation, the second training unit 404 is specifically used to update the student model using the first sample image, the fourth pseudo label corresponding to the first sample image, the second sample image, and the third pseudo label corresponding to the second sample image; the fourth pseudo label is obtained by processing the first sample image by the updated teacher model.

[0293] In one possible implementation, the second training unit 404 is specifically used to: determine the first distillation loss of the student model using the second sample image and the third pseudo label corresponding to the second sample image; determine the loss representation data of the student model based on the first distillation loss; and update the student model based on the loss representation data of the student model.

[0294] In one possible implementation, the second training unit 404 is specifically used to: determine the loss representation data of the student model based on the first distillation loss and the weighted weight corresponding to the first distillation loss; the weighted weight is determined based on the confidence of the third pseudo label; the confidence is determined based on the updated teacher model, the discriminator corresponding to the updated teacher model, and the second sample image.

[0295] In one possible implementation, the process of determining the weighted weight includes: if the confidence level is lower than a confidence threshold, randomly selecting a value from a preset interval and determining it as the weighted weight; if the confidence level is not lower than the confidence threshold, determining the preset value as the weighted weight.

[0296] In one possible implementation, the process of determining the confidence threshold includes: processing at least one image data using an updated teacher model to obtain a generated image corresponding to each image data; the at least one image data includes the second sample image and / or the first sample image; using the discriminator corresponding to the updated teacher model to perform discriminant processing on each of the generated images to obtain a discrimination result corresponding to each of the generated images; performing data analysis processing on the discrimination result corresponding to at least one of the generated images to obtain the confidence threshold.

[0297] Based on the relevant content of the above-mentioned model building device 400, it can be known that for the model building device 400 provided in the present application, after obtaining a training data set (for example, some image samples with label information and some image samples without label information), the super network is first trained using the training data set so that each sub-network in the trained super network has good image generation performance (for example, image generation performance, etc.); then, a teacher model and a student model are searched from the trained super network so that the teacher model and the student model also have good image generation performance; finally, model distillation is performed based on the training data set, the teacher model and the student model to obtain an image generation model, so that the image generation model not only has good image generation performance, but also has a relatively low computational complexity, so that the image generation model has a relatively high model efficiency (for example, fewer resources are required when deploying the image generation model, etc.) while ensuring performance, so that the image generation model can be deployed on hardware-constrained devices.

[0298] Based on the image generation method provided in the embodiment of the present application, the embodiment of the present application also provides an image generation device. Figure 5 Explain and illustrate. Figure 5 This is a schematic diagram of the structure of an image generating device provided in an embodiment of the present application. It should be noted that for the technical details of the image generating device provided in an embodiment of the present application, please refer to the relevant contents of the image generating method above.

[0299] like Figure 5 As shown, the image generating device 500 provided in the embodiment of the present application includes:

[0300] The second acquisition unit 501 is used to acquire an image to be processed;

[0301] The image generation unit 502 is used to process the image to be processed using a pre-constructed image generation model to obtain an image generation result; the image generation model is constructed using any implementation of the model construction method provided in the embodiments of the present application.

[0302] Based on the above-mentioned contents of the image generation device 500, it can be known that, for the image generation device 500 provided in the embodiment of the present application, after obtaining the image to be processed, the image to be processed can be processed using the pre-built image generation model to obtain an image generation result. Among them, because the image generation model has good performance, the image generation result determined based on the image generation model is better, which is conducive to improving the image generation effect.

[0303] In addition, an embodiment of the present application also provides an electronic device, which includes a processor and a memory: the memory is used to store instructions or computer programs; the processor is used to execute the instructions or computer programs in the memory, so that the electronic device executes any implementation of the model building method or image generation method provided in the embodiment of the present application.

[0304] See also Figure 6 , which shows a schematic diagram of the structure of an electronic device 600 suitable for implementing the embodiment of the present disclosure. The terminal device in the embodiment of the present disclosure may include but is not limited to mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), vehicle terminals (such as vehicle navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 6 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0305] like Figure 6 As shown, the electronic device 600 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage device 608 into a random access memory (RAM) 603. In the RAM 603, various programs and data required for the operation of the electronic device 600 are also stored. The processing device 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0306] Typically, the following devices may be connected to the I / O interface 605: an input device 606 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 607 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 608 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 609. The communication device 609 may allow the electronic device 600 to communicate with other devices wirelessly or by wire to exchange data. Although Figure 6 The electronic device 600 is shown with various devices, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead.

[0307] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, and the computer program contains program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication device 609, or installed from the storage device 608, or installed from the ROM 602. When the computer program is executed by the processing device 601, the above-mentioned functions defined in the method of the embodiment of the present disclosure are executed.

[0308] The electronic device provided by the embodiment of the present disclosure and the method provided by the above embodiment belong to the same inventive concept, and the technical details not fully described in this embodiment can be referred to the above embodiment, and this embodiment has the same beneficial effects as the above embodiment.

[0309] An embodiment of the present application also provides a computer-readable medium, in which instructions or computer programs are stored. When the instructions or computer programs are executed on a device, the device executes any implementation of the model building method or image generation method provided in the embodiment of the present application.

[0310] It should be noted that the computer-readable medium disclosed above may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, device or device. In the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0311] In some embodiments, the client and the server may communicate using any currently known or future developed network protocol such as HTTP (Hyper Text Transfer Protocol), and may be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any currently known or future developed network.

[0312] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.

[0313] The computer-readable medium carries one or more programs, and when the one or more programs are executed by the electronic device, the electronic device can execute the method.

[0314] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages ​​or a combination thereof, including, but not limited to, object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0315] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0316] The units involved in the embodiments described in the present disclosure may be implemented by software or hardware, wherein the name of a unit / module does not, in some cases, constitute a limitation on the unit itself.

[0317] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0318] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0319] It should be noted that the various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments, and the same or similar parts between the various embodiments can be referred to each other. For the system or device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part description.

[0320] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0321] It should also be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0322] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be implemented directly using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0323] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A model building method, characterized in that: The method comprises: Acquire a training data set; the training data set includes a first sample image and label information corresponding to the first sample image; Using the training data set to train the hypernetwork; Searching for teacher and student models from the trained hypernetwork; Model distillation is performed based on the training data set, the teacher model, and the student model to obtain an image generation model.

2. The method according to claim 1, characterized in that The training process of the hypernetwork includes: Sampling a first sub-network and a second sub-network from the super-network; the network size of the first sub-network is larger than the network size of the second sub-network; Determine the teacher generator according to the first sub-network; Determining the student generator according to the second sub-network; Model distillation is performed based on the training data set, the teacher generator and the student generator to obtain a trained super network.

3. The method according to claim 2, characterized in that The first subnetwork includes a largest subnetwork; The second sub-network includes at least one of a minimum sub-network and a random sub-network.

4. The method according to claim 2, characterized in that: The training data set also includes a second sample image; The performing of model distillation according to the training data set, the teacher generator and the student generator to obtain a trained super network includes: Determine loss representation data of the teacher generator using the first sample image and label information corresponding to the first sample image; Determine loss representation data of the student generator using the second sample image and a first pseudo label corresponding to the second sample image; the first pseudo label is obtained by processing the second sample image by the teacher generator; The hypernetwork, the teacher generator, and the student generator are updated according to the loss characterization data of the teacher generator and the loss characterization data of the student generator.

5. The method according to claim 4, characterized in that After updating the super network, the teacher generator and the student generator, the method further comprises: Continue to perform the step of determining the loss representation data of the teacher generator using the first sample image and the label information corresponding to the first sample image until a first stopping condition is reached.

6. The method according to claim 2, characterized in that The training data set also includes a second sample image; The performing of model distillation according to the training data set, the teacher generator and the student generator to obtain a trained super network includes: updating the teacher generator using the first sample image and the label information corresponding to the first sample image; Determine loss representation data of the student generator using the second sample image and a second pseudo label corresponding to the second sample image; the second pseudo label is obtained by processing the second sample image by the updated teacher generator; According to the loss characterization data of the student generator, the hypernetwork, the teacher generator and the student generator are updated, and the step of updating the teacher generator using the first sample image and the label information corresponding to the first sample image is continued until the second stop condition is reached.

7. The method according to claim 1, characterized in that The searching for a teacher model and a student model from a trained hypernetwork includes: According to a first model constraint, searching for the student model from the trained hypernetwork so that the student model satisfies the first model constraint; According to the second model constraint, at least one teacher model is searched from the trained hypernetwork so that each of the teacher models satisfies the first model constraint and the at least one teacher model satisfies a preset model architecture condition.

8. The method according to claim 7, characterized in that The at least one teacher model includes a teacher model that satisfies a preset model depth condition and a teacher model that satisfies a preset model width condition; and / or, The second model constraint is determined based on the first model constraint so that each of the teacher model and the student model satisfies a preset multiple condition on one or more model indicators.

9. The method according to claim 1, characterized in that: The training data set also includes a second sample image; The performing model distillation according to the training data set, the teacher model and the student model to obtain an image generation model includes: updating the teacher model using the first sample image and label information corresponding to the first sample image; The student model is updated using the second sample image and the third pseudo label corresponding to the second sample image, and the step of updating the teacher model using the first sample image and the label information corresponding to the first sample image is continued until the image generation model is determined according to the student model when the third stop condition is reached; the third pseudo label is obtained by processing the second sample image by the updated teacher model.

10. The method according to claim 9, characterized in that The updating of the student model by using the second sample image and the third pseudo label corresponding to the second sample image includes: The student model is updated using the first sample image, the fourth pseudo label corresponding to the first sample image, the second sample image, and the third pseudo label corresponding to the second sample image; the fourth pseudo label is obtained by processing the first sample image with the updated teacher model.

11. The method according to claim 9 or 10, characterized in that: The updating process of the student model includes: determining a first distillation loss of the student model using the second sample image and a third pseudo label corresponding to the second sample image; Determining loss characterization data of the student model according to the first distillation loss; The student model is updated according to the loss characterization data of the student model.

12. The method according to claim 11, characterized in that Determining loss characterization data of the student model according to the first distillation loss includes: Determine the loss representation data of the student model according to the first distillation loss and the weighted weight corresponding to the first distillation loss; the weighted weight is determined according to the confidence of the third pseudo label; the confidence is determined according to the updated teacher model, the discriminator corresponding to the updated teacher model and the second sample image.

13. The method according to claim 12, characterized in that The process of determining the weighted weights includes: If the confidence level is lower than the confidence threshold, randomly selecting a value from a preset interval and determining it as the weighted weight; If the confidence level is not lower than the confidence threshold, a preset value is determined as the weighted weight.

14. The method according to claim 13, characterized in that The process of determining the confidence threshold includes: Processing at least one image data using the updated teacher model to obtain a generated image corresponding to each image data; the at least one image data includes the second sample image and / or the first sample image; Using the discriminator corresponding to the updated teacher model to perform discriminative processing on each of the generated images, to obtain a discriminative result corresponding to each of the generated images; Data analysis is performed on at least one of the discrimination results corresponding to the generated image to obtain the confidence threshold.

15. An image generation method, characterized in that: The method comprises: Get the image to be processed; The image to be processed is processed using a pre-constructed image generation model to obtain an image generation result; the image generation model is constructed using the model construction method described in any one of claims 1 to 14.

16. A model building device, characterized in that: include: A first acquisition unit, used to acquire a training data set; The training data set includes a first sample image and label information corresponding to the first sample image; A first training unit, used to train the super network using the training data set; A network search unit, used to search for a teacher model and a student model from a trained hypernetwork; The second training unit is used to perform model distillation based on the training data set, the teacher model and the student model to obtain an image generation model.

17. An image generating device, characterized in that: include: A second acquisition unit, used for acquiring an image to be processed; An image generation unit is used to process the image to be processed using a pre-constructed image generation model to obtain an image generation result; the image generation model is constructed using the model construction method described in any one of claims 1-14.

18. An electronic device, characterized in that: The device comprises: a processor and a memory; The memory is used to store instructions or computer programs; The processor is used to execute the instructions or computer programs in the memory so that the electronic device executes the method according to any one of claims 1 to 15.

19. A computer readable medium, characterized in that The computer-readable medium stores instructions or computer programs, and when the instructions or computer programs are executed on a device, the device executes the method according to any one of claims 1 to 15.

Images