Model construction method and apparatus, image generation method and apparatus, device, and medium

Abstract

The present disclosure provides a model construction method and apparatus, an image generation method and apparatus, a device, and a medium. The model construction method comprises: upon acquiring a training data set, first using the training data set to train a hypernetwork, so that each sub-network in the trained hypernetwork has good image generation performance; then searching the trained hypernetwork for a teacher model and a student model, so that the teacher model and the student model also have good image generation performance; and finally performing model distillation on the basis of the training data set, the teacher model, and the student model to obtain an image generation model.

Description

Model construction method, image generation method, device, equipment, medium

[0001] This application claims priority to Chinese patent application No. 202310798179.9 filed on June 30, 2023. The contents of the above-mentioned Chinese patent application disclosure are hereby incorporated by reference in their entirety as a part of this application. Technical Field

[0002] The present disclosure relates to a model construction method, an image generation method, an apparatus, a device, and a medium. Background Art

[0003] For some image generation tasks, the task can be implemented with the help of pre-built models, such as the Generative adversarial networks (GAN) model.

[0004] In addition, for models with good image generation performance, they usually need to rely on explosive computational complexity, which makes deploying such resource-intensive models a daunting task, especially when hardware is limited. Therefore, how to alleviate this problem has become a technical issue that needs to be solved urgently.

[0005] Summary of the Invention

[0006] The present disclosure provides a model construction method, an image generation method, an apparatus, a device, and a medium.

[0007] In order to achieve the above objectives, the technical solutions provided by the present disclosure are as follows:

[0008] The present disclosure provides a model building method, the method comprising:

[0009] Acquire a training data set; the training data set includes a first sample image and label information corresponding to the first sample image;

[0010] Training the hypernetwork using the training data set;

[0011] Searching for teacher and student models from the trained hypernetwork;

[0012] Model distillation is performed based on the training data set, the teacher model, and the student model to obtain an image generation model.

[0013] In one possible implementation, the training process of the hypernetwork includes:

[0014] determining a teacher generator and a student generator from the hypernetwork;

[0015] Model distillation is performed based on the training data set, the teacher generator, and the student generator to obtain a trained super network.

[0016] In one possible implementation, determining the teacher generator and the student generator from the hypernetwork includes:

[0017] Sampling a first sub-network and a second sub-network from the super-network; wherein the network size of the first sub-network is larger than the network size of the second sub-network;

[0018] Determining the teacher generator according to the first sub-network;

[0019] According to the second sub-network, the student generator is determined.

[0020] In one possible implementation manner, the first subnetwork includes a largest subnetwork;

[0021] The second sub-network includes at least one of a minimum sub-network and a random sub-network.

[0022] In one possible implementation, the training data set further includes a second sample image;

[0023] The model distillation is performed based on the training data set, the teacher generator, and the student generator to obtain a trained super network, including:

[0024] Determining loss representation data of the teacher generator using the first sample image and label information corresponding to the first sample image;

[0025] Determining loss representation data of the student generator using the second sample image and a first pseudo label corresponding to the second sample image, wherein the first pseudo label is obtained by processing the second sample image by the teacher generator;

[0026] The super network, the teacher generator and the student generator are updated according to the loss characterization data of the teacher generator and the loss characterization data of the student generator.

[0027] In a possible implementation manner, after updating the hypernetwork, the teacher generator, and the student generator, the method further includes:

[0028] Continue to perform the step of determining the loss representation data of the teacher generator using the first sample image and the label information corresponding to the first sample image until a first stopping condition is reached.

[0029] In one possible implementation, the training data set further includes a second sample image;

[0030] The model distillation is performed based on the training data set, the teacher generator, and the student generator to obtain a trained super network, including:

[0031] updating the teacher generator using the first sample image and the label information corresponding to the first sample image;

[0032] Determining loss representation data of the student generator using the second sample image and a second pseudo label corresponding to the second sample image, wherein the second pseudo label is obtained by processing the second sample image by the updated teacher generator;

[0033] According to the loss characterization data of the student generator, the super network, the teacher generator and the student generator are updated, and the step of updating the teacher generator using the first sample image and the label information corresponding to the first sample image is continued until the second stopping condition is reached.

[0034] In one possible implementation, searching for a teacher model and a student model from a trained hypernetwork includes:

[0035] Searching for the student model from the trained hypernetwork according to the first model constraint, so that the student model satisfies the first model constraint;

[0036] According to the second model constraint, at least one teacher model is searched from the trained hypernetwork so that each teacher model satisfies the second model constraint and the at least one teacher model satisfies a preset model architecture condition.

[0037] In one possible implementation, the at least one teacher model includes a teacher model that satisfies a preset model depth condition and a teacher model that satisfies a preset model width condition;

[0038] and / or,

[0039] The second model constraint is determined based on the first model constraint so that each of the teacher model and the student model satisfies a preset multiple condition on one or more model indicators.

[0040] In one possible implementation, the training data set further includes a second sample image;

[0041] The performing model distillation based on the training data set, the teacher model, and the student model to obtain an image generation model includes:

[0042] Updating the teacher model using the first sample image and label information corresponding to the first sample image;

[0043] The student model is updated using the second sample image and the third pseudo label corresponding to the second sample image, and the step of updating the teacher model using the first sample image and the label information corresponding to the first sample image is continued until the third stopping condition is reached, and the image generation model is determined according to the student model; the third pseudo label is obtained by processing the second sample image by the updated teacher model.

[0044] In a possible implementation manner, updating the student model using the second sample image and the third pseudo label corresponding to the second sample image includes:

[0045] The student model is updated using the first sample image, the fourth pseudo label corresponding to the first sample image, the second sample image, and the third pseudo label corresponding to the second sample image; the fourth pseudo label is obtained by processing the first sample image by the updated teacher model.

[0046] In one possible implementation, the updating process of the student model includes:

[0047] determining a first distillation loss of the student model using the second sample image and a third pseudo label corresponding to the second sample image;

[0048] Determining loss representation data of the student model according to the first distillation loss;

[0049] The student model is updated according to the loss characterization data of the student model.

[0050] In one possible implementation, determining loss representation data of the student model based on the first distillation loss includes:

[0051] Determine the loss representation data of the student model based on the first distillation loss and the weighted weight corresponding to the first distillation loss; the weighted weight is determined based on the confidence of the third pseudo label; the confidence is determined based on the updated teacher model, the discriminator corresponding to the updated teacher model, and the second sample image.

[0052] In one possible implementation, the process of determining the weighted weights includes:

[0053] If the confidence level is lower than the confidence threshold, randomly selecting a value from a preset interval and determining it as the weighted weight;

[0054] If the confidence level is not lower than the confidence threshold, a preset value is determined as the weighted weight.

[0055] In one possible implementation, the process of determining the confidence threshold includes:

[0056] Processing at least one image data using the updated teacher model to obtain a generated image corresponding to each image data; the at least one image data includes the second sample image and / or the first sample image;

[0057] Using the discriminator corresponding to the updated teacher model to perform discriminative processing on each of the generated images, to obtain a discrimination result corresponding to each of the generated images;

[0058] Data analysis and processing are performed on the discrimination result corresponding to at least one of the generated images to obtain the confidence threshold.

[0059] The present disclosure provides an image generation method, the method comprising:

[0060] Get the image to be processed;

[0061] The image to be processed is processed using a pre-constructed image generation model to obtain an image generation result; the image generation model is constructed using the model construction method provided in the present disclosure.

[0062] The present disclosure provides a model building device, comprising:

[0063] A first acquisition unit is configured to acquire a training data set, wherein the training data set includes a first sample image and label information corresponding to the first sample image;

[0064] A first training unit is used to train the super network using the training data set;

[0065] A network search unit, used to search for teacher models and student models from the trained super network;

[0066] The second training unit is used to perform model distillation based on the training data set, the teacher model and the student model to obtain an image generation model.

[0067] The present disclosure provides an image generating device, comprising:

[0068] A second acquiring unit, configured to acquire an image to be processed;

[0069] An image generation unit is used to process the image to be processed using a pre-built image generation model to obtain an image generation result; the image generation model is constructed using the model construction method provided in the present disclosure.

[0070] The present disclosure provides an electronic device, the device comprising: a processor and a memory;

[0071] The memory is used to store instructions or computer programs;

[0072] The processor is used to execute the instructions or computer programs in the memory, so that the electronic device executes the model building method or image generation method provided by the present disclosure.

[0073] The present disclosure provides a computer-readable medium having instructions or a computer program stored therein. When the instructions or the computer program are executed on a device, the device executes the model building method or the image generating method provided by the present disclosure.

[0074] The present disclosure provides a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, and the computer program contains program code for executing the model construction method or image generation method provided by the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0075] In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments recorded in the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0076] FIG1 is a flow chart of a model building method provided by an embodiment of the present disclosure;

[0077] FIG2 is a schematic diagram of a model building process provided by an embodiment of the present disclosure;

[0078] FIG3 is a flow chart of an image generation method provided by an embodiment of the present disclosure;

[0079] FIG4 is a schematic structural diagram of a model building device provided by an embodiment of the present disclosure;

[0080] FIG5 is a schematic structural diagram of an image generating device provided by an embodiment of the present disclosure; and

[0081] FIG6 is a schematic structural diagram of an electronic device provided by an embodiment of the present disclosure. DETAILED DESCRIPTION

[0082] In order to enable those skilled in the art to better understand the solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the embodiments described are only part of the embodiments of the present disclosure, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present disclosure without making any creative efforts shall fall within the scope of protection of the present disclosure.

[0083] To better understand the technical solutions provided by the present disclosure, the model building method provided by the present disclosure is described below with reference to some figures. As shown in Figure 1, the model building method provided by an embodiment of the present disclosure includes the following steps S101-S104. Figure 1 is a flow chart of a model building method provided by an embodiment of the present disclosure.

[0084] S101: Acquire a training data set; the training data set includes a first sample image and label information corresponding to the first sample image.

[0085] The training data set refers to the data set required for use in the model training process (for example, the data set consisting of labeled training data and unlabeled training data shown in FIG1 ).

[0086] It should be noted that the present disclosure is not limited to the labeled training data in the above paragraph. For example, it can be Among them, the Represents a set of labeled training data; the {x i ,y i} represents the i-th labeled training data, i is a positive integer, 1≤i≤M, M is a positive integer, M represents the number of labeled training data; x i Represents the image data in the i-th labeled training data; y i Indicates that x i The corresponding label information can be obtained by manual labeling or other methods, and this disclosure does not specifically limit this. i For example, it can be expressed using image data. i For example, in the field of image depth estimation, y i Can refer to x i The real depth map. For example, in the field of face recognition in images, y i Can refer to x i The real face label map of i Can indicate the face area in the x iFor example, in the field of image style change, y i It can refer to an image in the target image domain whose domain style is the same as x i The data domain styles of this image data are different, and the present disclosure does not limit the target image domain. For example, it can be an image domain with a watercolor style, an image domain with a cartoon style, an image domain with a stick figure style, or an image domain with a black and white style, etc.

[0087] It should also be noted that the present disclosure is not limited to the above unlabeled training data. For example, it can be used Among them, the represents a set of unlabeled training data; z k Represents the kth unlabeled training data, k is a positive integer, 1≤k≤NM, N is a positive integer, and N represents the number of training data in the training dataset above.

[0088] In addition, the present disclosure does not limit the implementation of the above training data set. For example, in some application scenarios (such as fully supervised training scenarios), the training data set may include a first sample image (such as x i ) and the label information corresponding to the first sample image (for example, y i ); wherein the first sample image refers to image data with label information required for use in the model training process; the label information corresponding to the first sample image refers to the label pre-annotated for the first sample image. It should be noted that the present disclosure does not limit the number of the first sample images.

[0089] In fact, in order to reduce the difficulty of obtaining training data as much as possible, some labeled training data and some unlabeled training data can be used for model training, so that the model can not only learn knowledge from these labeled training data, but also learn structural features under a certain distribution domain from these unlabeled training data. Based on this, the present disclosure also provides a possible implementation of the above training data set, in which the training data set can include not only the first sample image and the label information corresponding to the first sample image, but also the second sample image (for example, z k ). The second sample image refers to image data without label information required for use in the model training process. It should be noted that the present disclosure does not limit the number of the second sample images.

[0090] It can be seen that in some application scenarios, the training data set mentioned above not only includes some labeled training data, but also some unlabeled training data, so that these unlabeled training data can provide some additional supervision information for the model training process, thereby enhancing the generalization of the model, and then helping the model to learn the structural features under a certain distribution domain. In this way, the amount of training data can be increased while paying as little labeling cost as possible.

[0091] In addition, the present disclosure does not limit the method for obtaining the above training dataset.

[0092] S102: Train the hypernetwork using the training data set.

[0093] The hypernetwork is used to represent a search space (e.g., the search space shown in FIG2 ), so that the hypernetwork can provide some subnetworks with different network structures (e.g., some subnetworks with different depths and widths, etc.). For example, the hypernetwork is used to represent the search space The G t represents a subnetwork sampled from the search space (ie, the supernetwork), t is a positive integer, and t≤the total number of subnetworks that the supernetwork can provide.

[0094] In fact, in order to better improve the model building effect, the present disclosure provides a possible implementation of the above hypernetwork, in which the hypernetwork has dynamically adjustable depth and width, so that the depth and width of the hypernetwork can be dynamically adjusted as the training process of the hypernetwork progresses.

[0095] In addition, the present disclosure is not limited to the implementation of S102 above. For example, when the training dataset above includes only labeled training data, any existing or future method for performing supervised training on a hypernetwork can be used for implementation. For another example, when the training dataset above includes some labeled training data and some unlabeled training data, any existing or future method for performing semi-supervised training on a hypernetwork can be used for implementation.

[0096] In fact, in order to better improve the model building effect, a distillation algorithm (e.g., an online distillation algorithm) can be used to implement the training process for the hypernetwork. Based on this, the present disclosure also provides a possible implementation of the above S102, which can specifically include the following steps 11 and 12.

[0097] Step 11: Determine the teacher generator and student generator from the hypernetwork.

[0098] The teacher generator refers to a sub-network with a larger network size sampled from a super-network; and the teacher generator has an image generation function (for example, an image generation function, etc.); in addition, the present disclosure does not limit the teacher generator, for example, it can be the largest sub-network shown in Figure 2. The network size is used to describe the size of a network in at least one dimension (for example, depth and / or width) so that the network size can characterize the size of the network. It should be noted that the present disclosure does not limit the implementation of the "at least one dimension", for example, it can include depth and / or width.

[0099] The student generator refers to another sub-network or sub-networks sampled from the super-network; and the student generator has an image generation function (for example, an image generation function, etc.); in addition, the network size of the student generator can be smaller than the network size of the teacher generator above, so that the final trained student generator can represent the compression result for the teacher generator.

[0100] In addition, the present disclosure does not limit the implementation method of the student generator above. For example, the student generator can be the random subnetwork shown in Figure 2. For another example, the student generator can include the minimum subnetwork and random subnetwork shown in Figure 2. Among them, the random subnetwork refers to a subnetwork randomly sampled from the supernetwork. The minimum subnetwork is a subnetwork sampled from the supernetwork according to a pre-set minimum subnetwork sampling rule. It should be noted that the present disclosure does not limit the minimum subnetwork sampling rule. For example, the minimum subnetwork sampling rule can be set in advance based on the application scenario.

[0101] In addition, the present disclosure does not limit the implementation of the above step 11. For example, it can be implemented by any existing or future method that can extract some sub-networks with different network sizes from the super-network.

[0102] In fact, in order to better improve the training effect of the hypernetwork, the present disclosure also provides a possible implementation of the above step 11, which may specifically include the following steps 111 to 113.

[0103] Step 111: sampling a first sub-network and a second sub-network from the super-network; the network size of the first sub-network is larger than the network size of the second sub-network.

[0104] The first subnetwork refers to a subnetwork sampled from the supernetwork and having a larger network size; and the present disclosure does not limit the first subnetwork. For example, the first subnetwork may include the largest subnetwork sampled from the supernetwork.

[0105] The second subnetwork refers to a subnetwork sampled from the supernetwork and having a smaller network size. Furthermore, this disclosure does not limit the second subnetwork. For example, the second subnetwork may include at least one of a minimum subnetwork and a random subnetwork sampled from the supernetwork. Therefore, in one possible implementation, the second subnetwork may include a minimum subnetwork and one or more random subnetworks.

[0106] Step 112: Determine the teacher generator based on the first sub-network above.

[0107] It should be noted that the present disclosure does not limit the implementation of step 112. For example, it can be specifically: after obtaining the first sub-network mentioned above, the first sub-network can be directly determined as the teacher generator. It can be seen that in one possible implementation, the teacher generator can be the largest sub-network sampled from the super-network.

[0108] Step 113: Determine the student generator based on the second sub-network above.

[0109] It should be noted that the present disclosure does not limit the implementation of step 113. For example, it may specifically be: after obtaining the second sub-network described above, the second sub-network may be directly determined as the student generator. Thus, in one possible implementation, the student generator may include a minimum sub-network sampled from the super-network and one or more random sub-networks.

[0110] Based on the relevant content of step 11 above, it can be known that after obtaining the hypernetwork used to represent the search space, the teacher generator and the student generator can be sampled from the hypernetwork so that the model distillation can be performed based on the teacher generator and the student generator to obtain a trained hypernetwork.

[0111] Step 12: Perform model distillation based on the training dataset, teacher generator, and student generator to obtain a trained super network.

[0112] It should be noted that the present disclosure does not limit the implementation of step 12. For example, it can be implemented using any existing or future model distillation method (e.g., an online distillation algorithm). Therefore, in one possible implementation, when the training dataset includes some labeled training data, step 12 can specifically be: using this labeled training data, the teacher generator, and the student generator to perform model distillation to obtain a trained hypernetwork.

[0113] In fact, to further reduce the hypernetwork's reliance on label information during training, a semi-supervised approach can be used to train the hypernetwork. Based on this, the present disclosure also provides a possible implementation of step 12 above. In this implementation, when the training dataset includes a first sample image, label information corresponding to the first sample image, and a second sample image, step 12 can specifically include the following steps 121-123.

[0114] Step 121: Determine loss representation data of the teacher generator using the first sample image and the label information corresponding to the first sample image.

[0115] Among them, the loss characterization data of the teacher generator is used to characterize the image generation performance of the teacher generator (for example, image generation performance); and the present disclosure does not limit the determination process of the loss characterization data of the teacher generator. For example, it can be implemented using any existing or future method that can determine the loss presented by the teacher generator on labeled training data.

[0116] In fact, in order to better improve the loss determination effect for the teacher generator, the present disclosure also provides a possible implementation method of the process of determining the loss representation data of the above teacher generator. In this implementation method, when the teacher generator is a generator in a generative adversarial network (GAN), the process of determining the loss representation data of the teacher generator can specifically include the following steps 1211-1213.

[0117] Step 1211: Determine the adversarial loss corresponding to the teacher generator using the first sample image, the label information corresponding to the first sample image, the teacher generator, and the discriminator corresponding to the teacher generator.

[0118] Among them, the discriminator corresponding to the teacher generator refers to the discriminator required to evaluate the performance of the teacher generator; and the discriminator corresponding to the teacher generator can be used to form a generative adversarial network with the teacher generator.

[0119] The adversarial loss corresponding to the teacher generator refers to the loss presented by the generative adversarial network composed of the teacher generator and the discriminator corresponding to the teacher generator.

[0120] In addition, the present disclosure does not limit the determination process of the adversarial loss corresponding to the above teacher generator. For example, it can be implemented using any existing or future method for determining the adversarial loss of a GAN (for example, the determination method shown in formula (1) below).

[0121] In formula (1), LGAN (G T ,D) represents the adversarial loss corresponding to the teacher generator above; G T represents the teacher generator; D represents the discriminator corresponding to the teacher generator; x represents the sample image data (for example, the first sample image above); y represents the label information corresponding to the sample image data (for example, the label information corresponding to the first sample image); G T (x) indicates that the sample image data is input into the teacher generator G T Then the teacher generator G T the output image data; Represents the expected function under the training data {x, y}; Represents the expected function under the sample image data.

[0122] Based on the relevant content of step 1211 above, it can be seen that after obtaining the teacher generator G T After that, we can use some labeled training data The teacher generator G T , and the teacher generator G T The corresponding discriminator D determines the teacher generator G T The corresponding adversarial loss, so that the teacher generator G can be optimized based on the adversarial loss T And the teacher generator G T The corresponding discriminator D, so that the final trained teacher generator G T Can achieve x i Map to y i This goal enables the final trained discriminator D to achieve the goal of T The output generated images are distinguishable from real images.

[0123] Step 1212: Using the first sample image, the label information corresponding to the first sample image, and the teacher generator, determine the reconstruction loss corresponding to the teacher generator.

[0124] Among them, the reconstruction loss corresponding to the teacher generator is used to characterize the image generation performance of the teacher generator, so that the output image of the teacher generator can be made close to the true label with the help of the reconstruction loss.

[0125] In addition, the present disclosure does not limit the process of determining the reconstruction loss corresponding to the above-mentioned teacher generator. For example, it can be implemented using a method for determining the reconstruction loss of any existing or future image generator (for example, the determination method shown in formula (2) below).

[0126] In formula (2), Lrecon (G T ) represents the reconstruction loss corresponding to the teacher generator; G T represents the teacher generator; x represents sample image data (for example, the first sample image above); y represents the label information corresponding to the sample image data (for example, the label information corresponding to the first sample image); G T (x) indicates that the sample image data is input into the teacher generator G T Then the teacher generator G T the output image data; Represents the expected function under the training data {x,y}.

[0127] Based on the relevant content of step 1212 above, it can be known that when the label information corresponding to the above first sample image is image data, and the above teacher generator belongs to the generator in GAN, the above first sample image can be input into the teacher generator to obtain the generated image output by the teacher generator; and then based on the degree of similarity between the generated image and the label information corresponding to the first sample image, the reconstruction loss corresponding to the teacher generator is determined, so that the reconstruction loss can represent the image generation performance of the teacher generator, so that the teacher generator can be optimized based on the reconstruction loss subsequently, so that the optimized teacher generator has better image generation performance.

[0128] Step 1213: Based on the adversarial loss corresponding to the above teacher generator and the reconstruction loss corresponding to the teacher generator, determine the loss representation data of the teacher generator (for example, the supervision loss 1 shown in Figure 2).

[0129] In the present disclosure, after obtaining the adversarial loss corresponding to the above teacher generator and the reconstruction loss corresponding to the teacher generator, the loss representation data of the teacher generator can be determined based on the two losses (as determined in formula (3) below), so that the loss representation data can more comprehensively and accurately represent the image generation performance presented by the teacher generator.

[0130] In formula (3), Represents the loss representation data of the teacher generator above; L GAN (G T ,D) represents the adversarial loss corresponding to the teacher generator above; G T represents the teacher generator; D represents the discriminator corresponding to the teacher generator; L recon (G T ) represents the reconstruction loss corresponding to the teacher generator.

[0131] Based on the relevant content of steps 1211 to 1213 above, it can be known that for the teacher generator, some labeled training data (for example, the first sample image above and the label information corresponding to the first sample image) can be used to determine the loss (that is, the supervision loss) presented by the teacher generator on these labeled training data, so that the teacher generator can be optimized based on the loss subsequently.

[0132] Step 122: Determine loss representation data of the student generator using the second sample image and a first pseudo label corresponding to the second sample image; the first pseudo label is determined by the teacher generator by processing the second sample image.

[0133] Among them, the first pseudo label corresponding to the second sample image refers to the processing result of the second sample image by the above teacher generator, so that the processing result can be used as the pseudo label of the second sample image to guide the optimization process of the student generator. In this way, the purpose of the teacher generator guiding the student generator can be achieved, thereby completing the optimization processing of the student generator without relying on the discriminator.

[0134] The loss characterization data of the student generator is used to characterize the image generation performance of the student generator (for example, image generation performance); and the present disclosure does not limit the determination process of the loss characterization data of the student generator. For example, it can be implemented using any existing or future method that can calculate the loss for the student generator (for example, online distillation loss, etc.).

[0135] In fact, in some application scenarios, to better improve the training effect of the supernetwork, the student generator mentioned above can include at least two subnetworks sampled from the supernetwork. Based on this, the present disclosure also provides a possible implementation of the above step 122. In this implementation, when the student generator includes a minimum subnetwork and a random subnetwork, step 122 can specifically include the following steps 1221-1223.

[0136] Step 1221: Determine the online distillation loss of the minimum sub-network using the second sample image and the first pseudo-label corresponding to the second sample image, so that the online distillation loss can represent the image generation performance of the minimum sub-network.

[0137] It should be noted that the present disclosure does not limit the implementation of step 1221. For example, it can be implemented by using any existing or future method that can calculate the online distillation loss of a model.

[0138] Step 1222: Determine the online distillation loss of the random sub-network using the second sample image and the first pseudo-label corresponding to the second sample image, so that the online distillation loss can represent the image generation performance of the random sub-network.

[0139] It should be noted that the present disclosure does not limit the implementation of step 1222. For example, it can be implemented by using any existing or future method that can calculate the online distillation loss of a model.

[0140] Step 1223: Determine the loss representation data of the student generator based on the online distillation loss of the minimum sub-network and the online distillation loss of the random sub-network.

[0141] In the present disclosure, when the above student generator includes a minimum subnetwork and a random subnetwork, after obtaining the online distillation loss of the minimum subnetwork and the online distillation loss of the random subnetwork, the loss representation data of the student generator can be determined based on these two losses (for example, it can be implemented using the following formula (4)).

[0142] In formula (4), represents the loss representation data of the student generator; G r represents a random subnetwork; G s represents the smallest subnetwork; L OD (G T (x u ),G r (x u )) represents the online distillation loss of the random sub-network; x u represents unlabeled training data (e.g., the second sample image); G T (x u ) indicates that the x is processed by the above teacher generator u The image data generated when G is t(e.g., the first pseudo label corresponding to the second sample image); r (x u ) indicates that the x is processed by a random sub-network u The image data generated when L OD (G T (x u ),G s (x u )) represents the online distillation loss of the smallest sub-network; G s (x u ) indicates that the x is processed by the smallest sub-network u The image data generated when .

[0143] Based on the relevant content of steps 1221 to 1223 above, it can be seen that in some application scenarios, if the student generator above includes multiple sub-networks, then under the guidance of the teacher generator above, the online distillation loss presented by each sub-network on these unlabeled training data can be first determined; then the online distillation losses corresponding to all sub-networks are added together to obtain the loss representation data of the student generator, so that the loss representation data can better represent the image generation performance of the student generator.

[0144] Step 123: Update the hypernetwork, the teacher generator, and the student generator based on the loss representation data of the teacher generator and the loss representation data of the student generator.

[0145] In the present disclosure, after obtaining the loss characterization data of the teacher generator and the loss characterization data of the student generator, the hypernetwork, teacher generator and student generator can be updated based on these two losses, so that the updated hypernetwork, updated teacher generator and updated student generator all have better image generation performance.

[0146] In addition, the present disclosure does not limit the implementation of the above step 123. For example, in some application scenarios, the step 123 may specifically include the following steps 1231 to 1233.

[0147] Step 1231: Determine the loss characterization data of the hypernetwork based on the loss characterization data of the teacher generator and the loss characterization data of the student generator, so that the loss characterization data can better represent the image generation performance of the hypernetwork.

[0148] It should be noted that the present disclosure does not limit the implementation of step 1231. For example, it can be implemented using the following formula (5).

[0149] In formula (5), L hypernetwork Represents the loss representation data of the hypernetwork; Represents the loss representation data of the teacher generator; Represents the loss representation data of the student generator.

[0150] Step 1232: Update the hypernetwork according to the loss characterization data of the hypernetwork.

[0151] In the present disclosure, after obtaining the loss characterization data of the hypernetwork, the hypernetwork can be updated (particularly, the depth and / or width of the hypernetwork is updated) based on the loss characterization data of the hypernetwork, so that the updated hypernetwork has better image generation performance. It should be noted that the present disclosure does not limit the updating method of the hypernetwork.

[0152] Step 1233: Determine the teacher generator and the student generator from the updated hypernetwork to obtain an updated teacher generator and an updated student generator.

[0153] In the present disclosure, after obtaining the updated supernetwork, some subnetworks (for example, the maximum subnetwork, the minimum subnetwork, and the random subnetwork) can be resampled from the updated supernetwork, and based on these sampled subnetworks, the updated teacher generator and the updated student generator can be determined (for example, the maximum subnetwork is determined as the updated teacher generator, and the minimum subnetwork and the random subnetwork are determined as the updated student generator).

[0154] Based on the relevant content of steps 1231 to 1233 above, it can be known that after obtaining the loss characterization data of the teacher generator and the loss characterization data of the student generator, the loss characterization data of the above hypernetwork can be determined based on these two losses, so that the loss characterization data can better represent the image generation performance of the hypernetwork; and then based on the loss characterization data of the hypernetwork, the hypernetwork, the teacher generator and the student generator are updated, so that the updated hypernetwork, the updated teacher generator and the updated student generator all have better image generation performance.

[0155] Based on the relevant content of steps 121 to 123 above, it can be seen that in some application scenarios, the super network can be trained with the help of a semi-supervised method. The process is roughly as follows: determine the loss of the largest subnetwork in the super network on some labeled training data sets, and use the largest subnetwork to guide the determination of the losses of some other subnetworks in the super network (for example, the minimum subnetwork and random subnetwork, etc.) on some unlabeled training data sets, so that the super network can be optimized based on these two losses. In this way, the super network can be optimized with the help of this mutual assistance between some subnetworks with relatively small network sizes and the largest subnetwork.

[0156] In fact, to better improve the image generation performance of the hypernetwork, the training process for the hypernetwork can be completed through multiple iterations. Based on this, the present disclosure also provides another possible implementation of step 12 above. In this implementation, when the training data set above includes a first sample image, label information corresponding to the first sample image, and a second sample image, step 12 not only includes steps 121 to 123 above, but also may include step 124 below.

[0157] Step 124: Determine whether the first stopping condition is met. If so, end the training process for the hypernetwork; if not, return to and continue to execute step 121 and subsequent steps.

[0158] The first stopping condition refers to the condition that must be met at the end of the hypernetwork training process. The present disclosure does not limit the first stopping condition. For example, it may be that the loss representation data of the hypernetwork is lower than a first threshold, or the rate of change of the loss representation data of the hypernetwork is lower than a second threshold, or the number of updates of the hypernetwork reaches a third threshold. The first threshold, the second threshold, and the third threshold can all be preset.

[0159] In addition, the present disclosure does not limit the execution time of the above step 124. For example, the execution time of the above step 124 may be later than the execution time of the above step 123.

[0160] Based on the relevant content of steps 121 to 124 above, it can be seen that in some application scenarios, the super network can be trained with the help of a semi-supervised method + an iterative method. In this way, the super network can be continuously optimized with the help of this mutual assistance + progressive method between some sub-networks with smaller network sizes and the largest sub-network in the super network, so that the trained super network has better image generation performance, and thus each sub-network in the trained super network also has better image generation performance.

[0161] In some application scenarios, the training process for the hypernetwork can be implemented by alternating updates of the teacher generator and the student generator. Based on this, the present disclosure also provides another possible implementation of step 12 above. In this implementation, when the training dataset above includes a first sample image, label information corresponding to the first sample image, and a second sample image, step 12 can specifically include the following steps 21-24.

[0162] Step 21: Update the teacher generator using the first sample image and the label information corresponding to the first sample image.

[0163] It should be noted that the present disclosure does not limit the implementation method of step 21. For example, it can be specifically as follows: first use the first sample image and the label information corresponding to the first sample image to determine the loss representation data of the teacher generator; then update the teacher generator based on the loss representation data of the teacher generator, so that the updated teacher generator has better performance.

[0164] In fact, in some application scenarios, the hypernetwork can be updated as the teacher generator is updated. Based on this, the present disclosure does not limit the implementation method of step 21 above. For example, it can be specifically: using the first sample image and the label information corresponding to the first sample image to update the teacher generator and the hypernetwork. That is, in one possible implementation method, the loss characterization data of the teacher generator can be determined first using the first sample image and the label information corresponding to the first sample image; then, based on the loss characterization data of the teacher generator, the hypernetwork is updated so that the updated one has better model performance; then, a subnetwork (for example, the largest subnetwork) is sampled from the updated hypernetwork and determined as the updated teacher generator, so that the hypernetwork can be updated while the teacher generator is updated.

[0165] Step 22: Determine the loss representation data of the student generator using the second sample image and the second pseudo label corresponding to the second sample image; the second pseudo label is obtained by processing the second sample image by the updated teacher generator.

[0166] Among them, the second pseudo-label corresponding to the second sample image refers to the processing result of the second sample image by the updated teacher generator, so that the processing result can be used as the pseudo-label of the second sample image to guide the optimization process of the student generator. In this way, the purpose of the updated teacher generator guiding the student generator can be achieved, thereby completing the optimization processing of the student generator without relying on the discriminator.

[0167] In addition, the implementation of step 22 is similar to the implementation of step 122 above, and for the sake of brevity, it will not be repeated here.

[0168] Step 23: Determine whether the second stopping condition is met. If so, end the training process for the hypernetwork; if not, update the hypernetwork, teacher generator, and student generator based on the loss representation data of the student generator, and return to continue executing step 21 above and its subsequent steps.

[0169] The second stopping condition refers to the condition that must be met at the end of the hypernetwork training process. The present disclosure does not limit the second stopping condition. For example, it may be that the loss representation data of the hypernetwork is lower than a first threshold, or the rate of change of the loss representation data of the hypernetwork is lower than a second threshold, or the number of updates of the hypernetwork reaches a third threshold. The first threshold, the second threshold, and the third threshold can all be preset.

[0170] In addition, the present disclosure does not limit the implementation method of the above step of "updating the super network, teacher generator, and student generator according to the loss characterization data of the student generator". For example, it can be implemented using the implementation method shown in the above step 123. For another example, if the super network is updated at any time when the teacher generator is updated, then in order to better improve the model update efficiency, the update process can specifically include: first updating the super network according to the loss characterization data of the student generator; then sampling some sub-networks (for example, the maximum sub-network, the minimum sub-network, and the random sub-network) from the updated super network, and determining the updated teacher generator and the updated student generator based on these sub-networks.

[0171] Based on the relevant content of steps 21 to 23 above, it can be seen that in some application scenarios, another semi-supervised method can be used to train the super network. The process is roughly as follows: first, the largest subnetwork in the super network is updated on some labeled training data sets, and then the updated largest subnetwork is used to guide the training process of some other subnetworks in the super network (such as the minimum subnetwork and random subnetwork, etc.) on some unlabeled training data sets, so that these other subnetworks can better absorb knowledge from these unlabeled training data and further optimize the super network through the weight sharing mechanism. In this way, the super network can be optimized with the help of this alternating update method between some subnetworks with relatively small network sizes and the largest subnetwork.

[0172] Based on the relevant content of steps 11 to 12 above, it can be seen that after constructing a supernetwork whose depth and width can change dynamically, supervised training can be performed on the largest subnetwork in the supernetwork, and under the guidance of the largest subnetwork, the training process of some other subnetworks in the supernetwork (for example, the minimum subnetwork and random subnetwork, etc.) on some unlabeled training data is completed, so that these other subnetworks can absorb knowledge from these unlabeled training data. In this way, the supernetwork can be continuously optimized with the help of this mutual assistance strategy and progressive collaboration between some subnetworks with smaller network sizes and the largest subnetwork in the supernetwork, so that the trained supernetwork has better image generation performance, and thus each subnetwork in the trained supernetwork also has better image generation performance.

[0173] S103: Searching for teacher models and student models from the trained hypernetwork.

[0174] The teacher model refers to a sub-network searched from the trained super-network so that the teacher model can guide the training process of the student model.

[0175] The student model refers to another sub-network searched from the trained super-network so that the structure of the student model is simpler than that of the teacher model.

[0176] In addition, the present disclosure does not limit the implementation of the above S103. For example, it may specifically include the following steps 31 and 32.

[0177] Step 31: Based on the first model constraint, search for a student model from the trained hypernetwork so that the student model satisfies the first model constraint.

[0178] Among them, the first model constraint refers to the constraint information required to search for the student model from the trained hypernetwork; and the first model constraint can be constraint information set in advance based on the application scenario (for example, in advance based on the configuration information of the terminal device deployed by the image generation model below, etc.), for example, it can be that the model running time is lower than a pre-set fourth threshold.

[0179] In addition, the present disclosure does not limit the implementation of step 31. For example, it can specifically be: based on the first model constraint, using an evolutionary search algorithm to search for a student model from the trained hypernetwork, so that the student model satisfies the first model constraint. It should be noted that the present disclosure does not limit the implementation of the evolutionary search algorithm.

[0180] Step 32: Based on the second model constraint, search for at least one teacher model from the trained hypernetwork so that each teacher model satisfies the second model constraint, and the at least one teacher model satisfies the preset model architecture condition.

[0181] Among them, the second model constraint refers to the constraint information required to search for the teacher model from the trained hypernetwork; and the second model constraint can be the constraint information set in advance based on the application scenario, for example, it can be that the model running time is lower than the pre-set fifth threshold.

[0182] In fact, in order to better improve the training effect of the student model under the guidance of the teacher model, the present disclosure also provides a determination process of the second model constraint above. For example, it can be specifically: according to the first model constraint above, the second model constraint is determined so that the teacher model that satisfies the second model constraint and the student model that satisfies the first model constraint meet the preset multiple conditions on one or more model indicators (for example, the computational cost of the teacher model is approximately 21 times the computational cost of the student model, etc.). Among them, the model indicator can be pre-set, for example, it can be the computational cost. The preset multiple condition can be determined based on the actual application scenario.

[0183] In the above step 32, "at least one teacher model" is used to represent a teacher model that is searched from the trained hypernetwork according to the second model constraint and is used to guide the training process of the student model; and the at least one teacher model satisfies the preset model architecture conditions so that the at least one teacher model can guide the training process of the student model in at least one dimension (for example, depth + width and other dimensions). It should be noted that the "at least one dimension" refers to one or more dimensions that are set in advance for the student model and need to be guided by the teacher model; and the present disclosure does not limit the "at least one dimension", for example, it may include depth and / or width. It can be seen that under one possible implementation method, the teacher model for model depth guidance and the teacher model for model width guidance can be used to participate in the training process for the student model, so that the trained student model has better performance.

[0184] Among them, the preset model architecture condition can be set according to the application scenario, and the present disclosure does not limit the preset model architecture condition. For example, the model architecture of at least one teacher model that meets the preset model architecture condition can present a complementary state.

[0185] In addition, the present disclosure does not limit the implementation method of the preset model architecture condition in the above paragraph. For example, the preset model architecture condition may include a preset model depth condition and a preset model width condition, so that at least one teacher model that meets the preset model architecture condition includes a teacher model that meets the preset model depth condition (for example, the above "teacher model for model depth guidance") and a teacher model that meets the preset model width condition (for example, the above "teacher model for model width guidance"), so that the at least one teacher model presents an architectural complementary state. Among them, the preset model depth condition refers to the condition required to be based on when searching for a teacher model for guiding the training process of the student model under the model depth; and the present disclosure does not limit the preset model depth condition. For example, the preset model depth condition can be set in advance according to the actual application scenario, so that the teacher model that meets the preset model depth condition can be used later to guide the training process of the student model under the dimension of model depth. The preset model width condition refers to the condition required when searching for a teacher model for guiding the training process of the student model under the model width; and the present disclosure does not limit the preset model width condition. For example, the preset model width condition can be set in advance based on the actual application scenario, so that the teacher model that meets the preset model width condition can be used later to guide the training process of the student model under the dimension of model width.

[0186] Based on the relevant content of steps 31 to 32 above, it can be known that for some application scenarios, after obtaining the trained super network, an evolutionary search algorithm can be used to search out a depth-based teacher model, a width-based teacher model, and a student model from the super network, so that the depth-based teacher model can be used to guide the optimization adjustment of the student model in terms of depth, and the width-based teacher model can be used to guide the optimization adjustment of the student model in terms of width. Among them, because the trained super network has good image generation performance, each sub-network in the super network also has good image generation performance, so that the two teacher models and student models searched from the super network have good performance, which can effectively avoid the adverse effects caused by optimizing the model from scratch (for example, the model convergence speed is relatively slow), thereby effectively improving the model training effect (for example, accelerating the model convergence speed). Among them, the depth-based teacher model refers to the teacher model that meets the preset model depth condition above, so that the depth-based teacher model is used to guide the training process of the student model under the dimension of model depth. The width-based teacher model refers to the teacher model that meets the preset model depth conditions above, so that the width-based teacher model is used to guide the training process of the student model under the dimension of model width.

[0187] Also, because the trained hypernetwork can provide sub-networks that meet different constraints, the present disclosure can use the hypernetwork to determine the various teacher models and student models that meet different constraints. This can effectively avoid the adverse effects caused by manually designing multiple teacher models and student models (for example, it takes a long time, etc.), thereby helping to improve the model construction effect.

[0188] S104: Perform model distillation based on the training dataset, the teacher model, and the student model to obtain an image generation model.

[0189] In which, the image generation model is used to generate images based on the input data of the image generation model; and the present disclosure does not limit the implementation method of the image generation model. For example, in some application scenarios, the image generation model belongs to an image generator, so that the image generation model can generate a new image data based on the input data of the image generation model.

[0190] In addition, the present disclosure does not limit the implementation method of the model distillation in S104. For example, it can be implemented using any existing or future model distillation method (such as an online distillation method).

[0191] In fact, in order to better reduce the dependence of the model distillation process on label information, the present disclosure also provides a possible implementation of the above S104. In this implementation, when the above training data set includes a first sample image, label information corresponding to the first sample image, and a second sample image, the S104 can specifically include the following steps 41-45.

[0192] Step 41: Update the teacher model using the first sample image and the label information corresponding to the first sample image.

[0193] It should be noted that the present disclosure does not limit the implementation method of step 41. For example, it can be specifically as follows: first use the first sample image and the label information corresponding to the first sample image to determine the loss representation data of the teacher model (for example, the supervision loss 2 shown in Figure 2); then update the teacher model based on the loss representation data of the teacher model so that the updated teacher model has better performance.

[0194] Step 42: Update the student model using the second sample image and the third pseudo label corresponding to the second sample image; the third pseudo label is obtained by processing the second sample image by the updated teacher model.

[0195] Among them, the third pseudo label corresponding to the second sample image refers to the processing result of the second sample image by the updated teacher model, so that the processing result can be used as the pseudo label of the second sample image to guide the optimization process of the student model. In this way, the purpose of guiding the student model by the updated teacher model can be achieved, thereby completing the optimization processing of the student model without relying on the discriminator.

[0196] In addition, the present disclosure does not limit the implementation method of step 42. For example, it can be implemented using the student model update process involved in any existing or future model distillation method (for example, an online distillation method). It can be seen that in one possible implementation method, step 42 can specifically be: first use the second sample image and the third pseudo-label corresponding to the second sample image to determine the loss representation data of the student model; then update the student model based on the loss representation data of the student model. It should be noted that the process of determining the loss representation data of the student model is similar to the process of determining the loss representation data of the student generator above. For the sake of brevity, it will not be repeated here.

[0197] In fact, since the update process of the student model does not require the use of real label information, in order to better improve the training effect of the student model, the present disclosure also provides another possible implementation of step 42 above, which can be specifically: using the first sample image, the fourth pseudo label corresponding to the first sample image, the second sample image and the third pseudo label corresponding to the second sample image to update the student model, so as to effectively increase the number of training data of the student model, thereby helping to improve the training effect of the student model; the fourth pseudo label is obtained by processing the first sample image by the updated teacher model.

[0198] In addition, the present disclosure does not limit the updating process of the above student model. For example, it may specifically include the following steps 421 to 423.

[0199] Step 421: Determine a first distillation loss of the student model using the second sample image and the third pseudo label corresponding to the second sample image.

[0200] Among them, the first distillation loss is used to characterize the loss presented by the student model on unlabeled training data.

[0201] In addition, the present disclosure does not limit the determination process of the first distillation loss. For example, when the above "at least one teacher model" includes a depth-based teacher model and a width-based teacher model, the first distillation loss can be implemented using the following formula (6).

[0202] In formula (6), represents the first distillation loss of the student model; G * represents the student model; represents the online distillation loss of the student model on unlabeled training data under the guidance of the depth-based teacher model; represents the online distillation loss of the student model on the unlabeled training data under the guidance of the width-based teacher model; represents unlabeled training data; Indicates that the depth-based teacher model processes the x u The image data generated when Indicates that the student model processes the x u The image data generated when Indicates that the width-based teacher model processes the x u The image data generated when .

[0203] Step 422: Determine loss representation data of the student model based on the first distillation loss of the student model.

[0204] It should be noted that the present disclosure does not limit the implementation of step 422. For example, it may specifically be: directly determining the first distillation loss of the student model as the loss representation data of the student model.

[0205] For another example, when the above student model is trained with the help of labeled training data and unlabeled training data, the process of determining the loss representation data of the student model may include the following steps 51 and 52.

[0206] Step 51: Determine a second distillation loss of the student model using the first sample image and the fourth pseudo label corresponding to the first sample image.

[0207] Among them, the fourth pseudo-label corresponding to the first sample image refers to the processing result of the updated teacher model for the first sample image, so that the processing result can be used as the pseudo-label of the first sample image to guide the optimization process of the student model. In this way, the purpose of guiding the student model by the updated teacher model can be achieved, thereby completing the optimization processing of the student model without relying on the discriminator.

[0208] The second distillation loss is used to characterize the loss exhibited by the student model on labeled training data.

[0209] In addition, the present disclosure does not limit the determination process of the second distillation loss. For example, when the above "at least one teacher model" includes a depth-based teacher model and a width-based teacher model, the second distillation loss can be implemented using the following formula (7).

[0210] In formula (7), represents the second distillation loss of the student model; G * represents the student model; represents the online distillation loss of the student model on the labeled training data under the guidance of the depth-based teacher model; represents the online distillation loss of the student model on the labeled training data under the guidance of the width-based teacher model; Represents labeled training data; Indicates that the depth-based teacher model processes the The image data generated when Indicates that the student model handles the The image data generated when Indicates that the width-based teacher model processes the The image data generated when .

[0211] It should be noted that the present disclosure does not limit the execution order between step 51 and the above step 421.

[0212] Step 52: Determine loss representation data of the student model based on the first distillation loss of the student model and the second distillation loss of the student model.

[0213] It should be noted that the present disclosure does not limit the implementation of the above step 52. For example, it can specifically be: adding the first distillation loss of the student model and the second distillation loss of the student model to obtain the loss representation data of the student model.

[0214] Based on the relevant content of steps 51 to 52 above, it can be seen that in some application scenarios, under the guidance of multiple teacher models, the distillation loss presented by the student model in the unlabeled training data and the distillation loss presented by the student model in the labeled training data can be determined first; then these two losses are combined to determine the loss representation data of the student model, so that the loss representation data can better represent the performance of the student model.

[0215] In fact, since the image generation ability of the teacher model may fluctuate, the credibility of some pseudo labels generated based on the teacher model is not high. Therefore, in order to avoid these pseudo labels with low credibility from interfering with the update process of the student model, the impact of these pseudo labels with low credibility on the update process of the student model can be weakened as much as possible. Based on this, the present disclosure also provides a possible implementation of step 422 above, which can be specifically: determining the loss representation data of the student model based on the first distillation loss of the student model and the weighted weight corresponding to the first distillation loss. The weighted weight is determined based on the confidence of the third pseudo label; the confidence of the third pseudo label is determined based on the updated teacher model, the discriminator corresponding to the updated teacher model, and the second sample image.

[0216] Among them, the confidence of the third pseudo label is used to characterize the credibility of the third pseudo label; and the present disclosure does not limit the determination process of the confidence of the third pseudo label. For example, it can be specifically: first input the second sample image into the updated teacher model to obtain the third pseudo label output by the teacher model; then the discriminator corresponding to the updated teacher model performs a discriminant process on the third pseudo label to obtain a discrimination result of the third pseudo label, and determines the discrimination result as the confidence of the third pseudo label.

[0217] The weighted weight corresponding to the first distillation loss is used to characterize the influence of the first distillation loss. The present disclosure does not limit the process of determining the weighted weight corresponding to the first distillation loss. For example, it may specifically include the following steps 61 and 62.

[0218] Step 61: If the confidence of the third pseudo label is lower than the confidence threshold, a value is randomly selected from a preset interval and determined as the weighted weight corresponding to the first distillation loss above.

[0219] The confidence threshold is used to describe the dividing line between trustworthy and untrustworthy. The present disclosure does not limit the determination process of the confidence threshold. For example, it can be pre-set according to the application scenario.

[0220] In addition, in order to better improve the rationality of the weights, the present disclosure also provides a method for automatically determining the confidence threshold, which may specifically include the following steps 71 to 73.

[0221] Step 71: Process at least one image data using the updated teacher model to obtain a generated image corresponding to each image data; the at least one image data includes a second sample image and / or a first sample image.

[0222] Among them, at least one image data refers to the image data required for reference when automatically determining the confidence threshold; and the present disclosure does not limit the implementation method of the at least one image data, for example, it may include the second sample image and / or the first sample image mentioned above.

[0223] Step 72: Use the discriminator corresponding to the updated teacher model to perform discrimination processing on each generated image to obtain a discrimination result corresponding to each generated image.

[0224] The discriminator corresponding to the updated teacher model refers to the discriminator involved in the update process of the teacher model.

[0225] Step 73: Perform data analysis on the discrimination result corresponding to at least one generated image to obtain a confidence threshold.

[0226] The term "data analysis and processing" refers to a pre-defined data analysis method; the present disclosure does not limit the implementation of the data analysis and processing; for example, it may be implemented using any existing or future data analysis method (e.g., maximum, minimum, mean, median, or exponential moving average). Thus, in one possible implementation, the confidence threshold may be the result of performing an exponential moving average (EMA) analysis on the discrimination results corresponding to all generated images.

[0227] Based on the relevant content of steps 71 to 73 above, it can be seen that in some application scenarios, after obtaining the updated teacher model, the teacher model can be used to process each sample image in the training data set to obtain the generated image corresponding to each sample image; then the discriminator corresponding to the teacher model is used to perform discrimination processing on each generated image to obtain the discrimination results of each generated image; finally, all these discrimination results are analyzed to obtain a confidence threshold, so that the confidence threshold can better describe the boundary line between credible and untrustworthy.

[0228] The preset interval refers to a preset numerical selection interval, so that the above "randomly select a numerical value in the preset interval" is greater than or equal to the minimum value in the preset interval, and the "randomly select a numerical value in the preset interval" is less than or equal to the maximum value in the preset interval; and the present disclosure does not limit the preset interval. For example, any numerical value in the preset interval is less than or equal to a preset first numerical value (for example, 1, etc.). For another example, any numerical value in the preset interval is greater than or equal to a preset second numerical value (for example, 0, etc.). For another example, when the second numerical value is less than the first numerical value, the preset interval can be the data interval of [second numerical value, first numerical value] (for example, the data interval of [0, 1]).

[0229] Based on the relevant content of step 61 above, it can be known that for the third pseudo-label corresponding to the second sample image, if the confidence of the third pseudo-label is lower than the confidence threshold, it can be determined that the credibility of the third pseudo-label is not very high. Therefore, in order to weaken the impact of the third pseudo-label on the update process of the student model, a value can be randomly selected from a preset interval (for example, a value can be randomly selected from between 0 and 1) as the weighted weight corresponding to the third pseudo-label, so that the weighted weight can represent the impact of the loss determined based on the third pseudo-label on the update process of the student model.

[0230] Step 62: If the confidence of the third pseudo label is not lower than the confidence threshold, the preset value is determined as the weighted weight corresponding to the first distillation loss above.

[0231] In the present disclosure, for the third pseudo-label corresponding to the second sample image, if the confidence of the third pseudo-label is higher than or equal to the confidence threshold, it can be determined that the credibility of the third pseudo-label is relatively high. Therefore, in order to maintain or even improve the impact of the third pseudo-label on the update process of the student model, a preset value (for example, 1) can be directly used as the weighted weight corresponding to the third pseudo-label, so that the weighted weight can represent the impact of the loss determined based on the third pseudo-label on the update process of the student model.

[0232] It should be noted that the present disclosure does not limit the implementation of the above preset value. For example, the preset value is not lower than the highest value in the above preset range. For another example, the preset value can be 1.

[0233] Based on the relevant content of steps 61 to 62 above, it can be seen that in some application scenarios, when the first distillation loss above is determined based on the third pseudo-label corresponding to the second sample image, the weighted weight corresponding to the first distillation loss can be determined based on the confidence of the third pseudo-label. In this way, the impact of the pseudo-label with lower confidence on the update process of the student model can be weakened, thereby better improving the model update effect.

[0234] In addition, the present disclosure does not limit the implementation method of the above step of "determining the loss representation data of the student model based on the first distillation loss of the student model and the weighted weight corresponding to the first distillation loss". For example, it can be implemented using the following formulas (8)-(10).

[0235] In formulas (8)-(10), represents the first distillation loss of the student model; G * represents the student model; represents the online distillation loss of the student model on unlabeled training data under the guidance of the depth-based teacher model; represents unlabeled training data; Indicates that the depth-based teacher model processes the x u The image data generated when Indicates that the student model processes the x u The image data generated when express The corresponding weighted weights; represents the online distillation loss of the student model on the unlabeled training data under the guidance of the width-based teacher model; express The corresponding weighted weights; Indicates that the width-based teacher model processes the x u The image data generated when Indicates that the corresponding discriminator is processed by the depth-based teacher model for The decision result (e.g., prediction score) given by this generated image; Represents the discriminator corresponding to the width-based teacher model for The judgment result given by this generated image; Represents the EMA value of the judgment result given by the discriminator corresponding to the depth-based teacher model for the generated image output by the depth-based teacher model; It represents the EMA value of the judgment result given by the discriminator corresponding to the width-based teacher model for the generated image output by the width-based teacher model; RandInt([0,1]) represents a random selection of a value from 0 to 1.

[0236] Based on the relevant content of step 422 above, it can be known that in some application scenarios, after obtaining the first distillation loss of the student model, the loss representation data of the student model can be determined based on the first distillation loss, so that the loss representation data can better represent the image generation performance of the student model.

[0237] Step 423: Update the student model based on the loss representation data of the student model.

[0238] It should be noted that the present disclosure does not limit the implementation of the above step 423.

[0239] Based on the relevant content of steps 421 to 423 above, it can be seen that in one possible implementation, after obtaining multiple updated teacher models, the update process of the student model can be guided based on all training data (for example, the first sample image and the second sample image) and using these teacher models, so that the updated student model has better image generation performance.

[0240] Based on the relevant content of step 42 above, it can be seen that since the update process of the student model does not need to use real label information, all training data (for example, the first sample image and the second sample image) can be used to participate in the optimization process of the student model, which can better improve the model optimization effect.

[0241] Step 43: Determine whether the third stop condition is met. If so, execute the following step 45; if not, execute the following step 44.

[0242] The third stopping condition refers to the condition that must be met when the student model training process ends. The present disclosure does not limit the third stopping condition. For example, it may be that the loss representation data of the student model is lower than a sixth threshold, or the rate of change of the loss representation data of the student model is lower than a seventh threshold, or the number of updates of the student model reaches an eighth threshold. The sixth threshold, the seventh threshold, and the eighth threshold can all be preset.

[0243] Step 44: If the third stop condition is not met, return to execute step 41 and subsequent steps.

[0244] In the present disclosure, for the current round of training process for the student model, if it is determined that the third stop condition has not been met, it can be determined that the performance of the student model still needs to be improved, so the above step 41 and its subsequent steps can be re-executed based on the updated student model to start the next round of training process for the student model.

[0245] Step 45: If the third stopping condition is reached, the image generation model is determined based on the student model.

[0246] In the present disclosure, for the current round of training process for the student model, if it is determined that the third stop condition is reached, it can be determined that the performance of the student model is relatively good, so the image generation model can be determined based on the student model (for example, directly determining the student model as the image generation model, etc.) so that the image generation model can have better performance.

[0247] Based on the relevant content of steps 41 to 45 above, it can be seen that after searching for multiple teacher models and a student model from the trained hypernetwork, the training process for the student model can be implemented by alternatingly updating the teacher model and the student model, so that the final trained student model has better performance.

[0248] Based on the relevant contents of S101 to S104 above, it can be seen that for the model construction method provided by the embodiment of the present disclosure, after obtaining a training data set (for example, some image samples with label information and some image samples without label information), the super network is first trained using the training data set so that each sub-network in the trained super network has good image generation performance (for example, image generation performance, etc.); then, a teacher model and a student model are searched from the trained super network so that the teacher model and the student model also have good image generation performance; finally, model distillation is performed based on the training data set, the teacher model and the student model to obtain an image generation model, so that the image generation model not only has good image generation performance, but also has a relatively low computational complexity of the image generation model, so that the image generation model has a relatively high model efficiency (for example, fewer resources are required when deploying the image generation model, etc.) while ensuring performance, so that the image generation model can be deployed on hardware-constrained devices.

[0249] In addition, since each sub-network in the trained super-network has good image generation performance, the teacher model and student model searched from the super-network have relatively good image generation performance, so that only the teacher model and student model need to be fine-tuned in the future. This can effectively avoid the adverse effects caused by optimizing the teacher model and student model from scratch (for example, slow model convergence speed, etc.), thereby helping to improve the efficiency of model training.

[0250] In addition, because the hypernetwork can provide a large variety of network structures, the present disclosure can search for student models and teacher models that meet different constraints from the hypernetwork, which can effectively eliminate the additional resource consumption (such as human resource consumption, time consumption, etc.) caused by manual design and configuration of the teacher model, thereby helping to save resources.

[0251] In addition, for the model construction method provided by the present invention, this method can achieve the synergy of seamlessly promoting model-efficient and data-efficient learning by sequentially executing the semi-supervised driven network architecture search stage and the adaptive online semi-supervised distillation stage, thereby achieving the purpose of unified compression of network data + model.

[0252] In addition, the present disclosure provides a new learning paradigm with a unified optimization objective that can seamlessly promote the synergy of model-efficient and data-efficient learning. In this learning paradigm, a semi-supervised driven network architecture search and an adaptive online semi-supervised distillation phase can be set up in sequence to formulate a heterogeneous mutual learning scheme to obtain a model with flexible architecture, data efficiency, and excellent performance. Extensive experiments show that even with fewer labels, this learning paradigm can obtain a state-of-the-art lightweight model, so that this learning paradigm can provide a feasible solution for deploying GAN models in hardware-constrained and low-data environments.

[0253] In addition, the present disclosure does not limit the execution subject of the above model construction method. For example, the model construction method provided in the embodiment of the present disclosure can be applied to a device with data processing function such as a terminal device or a server. For another example, the model construction method provided in the embodiment of the present disclosure can also be implemented with the help of a data communication process between different devices (for example, a terminal device and a server, two terminal devices, or two servers). Among them, the terminal device can be a smart phone, a computer, a personal digital assistant (PDA) or a tablet computer. The server can be a standalone server, a cluster server or a cloud server.

[0254] Based on the above model building method, the present disclosure also provides an image generation method, which is described below with reference to the accompanying drawings. As shown in Figure 3, the image generation method provided by the present embodiment includes the following steps S301-S302. Figure 3 is a flowchart of the image generation method provided by the present embodiment.

[0255] S301: Acquire an image to be processed.

[0256] The image to be processed refers to image data that needs to be generated with the help of the above image generation model.

[0257] S302: Processing the image to be processed using an image generation model to obtain an image generation result; the image generation model is constructed using any implementation of the model construction method provided in the present disclosure.

[0258] For more information about the image generation model, please refer to the above text.

[0259] The image generation result refers to the result output by the image generation model after processing the above image to be processed.

[0260] Based on the relevant content of S301 to S302 above, it can be seen that after obtaining the image to be processed, the pre-built image generation model can be used to process the image to be processed to obtain an image generation result. In particular, because the image generation model has good performance, the image generation result determined based on the image generation model is relatively good, which is conducive to improving the image generation effect.

[0261] Furthermore, the present disclosure does not limit the execution subject of the above-described image generation method. For example, the image generation method provided in the embodiments of the present disclosure can be applied to a device with data processing capabilities, such as a terminal device or a server. Furthermore, the image generation method provided in the embodiments of the present disclosure can also be implemented via a data communication process between different devices (e.g., a terminal device and a server, two terminal devices, or two servers).

[0262] Based on the model building method provided in the embodiments of the present disclosure, the embodiments of the present disclosure also provide a model building device, which will be explained and illustrated below in conjunction with Figure 4. Figure 4 is a schematic diagram of the structure of a model building device provided in the embodiments of the present disclosure. It should be noted that for the technical details of the model building device provided in the embodiments of the present disclosure, please refer to the relevant content of the model building method above.

[0263] As shown in FIG4 , the model building device 400 provided in an embodiment of the present disclosure includes:

[0264] A first acquisition unit 401 is configured to acquire a training data set, wherein the training data set includes a first sample image and label information corresponding to the first sample image;

[0265] A first training unit 402 is configured to train the supernetwork using the training data set;

[0266] A network search unit 403 is used to search for a teacher model and a student model from a trained super network;

[0267] The second training unit 404 is used to perform model distillation based on the training data set, the teacher model and the student model to obtain an image generation model.

[0268] In a possible implementation manner, the first training unit 402 includes:

[0269] A first determining subunit, configured to determine a teacher generator and a student generator from the super network;

[0270] The model distillation subunit is used to perform model distillation based on the training data set, the teacher generator and the student generator to obtain a trained super network.

[0271] In one possible implementation, the first determination subunit is specifically used to: sample a first subnetwork and a second subnetwork from the supernetwork; the network size of the first subnetwork is larger than the network size of the second subnetwork; determine the teacher generator based on the first subnetwork; and determine the student generator based on the second subnetwork.

[0272] In a possible implementation manner, the first subnetwork includes a maximum subnetwork; and the second subnetwork includes at least one of a minimum subnetwork and a random subnetwork.

[0273] In one possible implementation, the training data set further includes a second sample image;

[0274] The model distillation subunit is specifically used to: determine the loss representation data of the teacher generator using the first sample image and the label information corresponding to the first sample image; determine the loss representation data of the student generator using the second sample image and the first pseudo label corresponding to the second sample image; the first pseudo label is obtained by the teacher generator processing the second sample image; and update the super network, the teacher generator and the student generator according to the loss representation data of the teacher generator and the loss representation data of the student generator.

[0275] In one possible implementation, the model distillation subunit is further used to: after updating the hypernetwork, the teacher generator, and the student generator, continue to perform the step of determining the loss representation data of the teacher generator using the first sample image and the label information corresponding to the first sample image until a first stopping condition is reached.

[0276] In one possible implementation, the training data set further includes a second sample image;

[0277] The model distillation subunit is specifically used to: update the teacher generator using the first sample image and the label information corresponding to the first sample image; determine the loss representation data of the student generator using the second sample image and the second pseudo label corresponding to the second sample image; the second pseudo label is obtained by processing the second sample image by the updated teacher generator; update the hypernetwork, the teacher generator and the student generator according to the loss representation data of the student generator, and continue to execute the step of updating the teacher generator using the first sample image and the label information corresponding to the first sample image until the second stop condition is reached.

[0278] In one possible implementation, the network search unit 403 is specifically used to: search for the student model from the trained super network based on the first model constraint, so that the student model satisfies the first model constraint; and search for at least one from the trained super network based on the second model constraint, so that each of the teacher models satisfies the second model constraint, and the at least one teacher model satisfies the preset model architecture condition.

[0279] In one possible implementation, the at least one teacher model includes a teacher model that satisfies a preset model depth condition and a teacher model that satisfies a preset model width condition; and / or, the second model constraint is determined based on the first model constraint so that each of the teacher models and the student model satisfies a preset multiple condition on one or more model indicators.

[0280] In one possible implementation, the training data set further includes a second sample image;

[0281] The second training unit 404 is specifically used to: update the teacher model using the first sample image and the label information corresponding to the first sample image; update the student model using the second sample image and the third pseudo label corresponding to the second sample image, and continue to execute the step of updating the teacher model using the first sample image and the label information corresponding to the first sample image until the third stop condition is reached, and determine the image generation model according to the student model; the third pseudo label is obtained by processing the second sample image by the updated teacher model.

[0282] In one possible implementation, the second training unit 404 is specifically used to update the student model using the first sample image, the fourth pseudo label corresponding to the first sample image, the second sample image, and the third pseudo label corresponding to the second sample image; the fourth pseudo label is obtained by processing the first sample image by the updated teacher model.

[0283] In one possible implementation, the second training unit 404 is specifically used to: determine the first distillation loss of the student model using the second sample image and the third pseudo label corresponding to the second sample image; determine the loss representation data of the student model based on the first distillation loss; and update the student model based on the loss representation data of the student model.

[0284] In one possible implementation, the second training unit 404 is specifically used to: determine the loss representation data of the student model based on the first distillation loss and the weighted weight corresponding to the first distillation loss; the weighted weight is determined based on the confidence of the third pseudo label; the confidence is determined based on the updated teacher model, the discriminator corresponding to the updated teacher model, and the second sample image.

[0285] In one possible implementation, the process of determining the weighted weight includes: if the confidence level is lower than the confidence threshold, randomly selecting a value from a preset interval and determining it as the weighted weight; if the confidence level is not lower than the confidence threshold, determining the preset value as the weighted weight.

[0286] In one possible implementation, the process of determining the confidence threshold includes: processing at least one image data using an updated teacher model to obtain a generated image corresponding to each image data; the at least one image data includes the second sample image and / or the first sample image; using the discriminator corresponding to the updated teacher model to perform discrimination processing on each of the generated images to obtain a discrimination result corresponding to each of the generated images; performing data analysis processing on the discrimination result corresponding to at least one of the generated images to obtain the confidence threshold.

[0287] Based on the relevant content of the above-mentioned model building device 400, it can be known that for the model building device 400 provided by the present disclosure, after obtaining a training data set (for example, some image samples with label information and some image samples without label information), the super network is first trained using the training data set so that each sub-network in the trained super network has good image generation performance (for example, image generation performance, etc.); then, a teacher model and a student model are searched from the trained super network so that the teacher model and the student model also have good image generation performance; finally, model distillation is performed based on the training data set, the teacher model and the student model to obtain an image generation model, so that the image generation model not only has good image generation performance, but also has a relatively low computational complexity of the image generation model, so that the image generation model has a relatively high model efficiency (for example, fewer resources are required when deploying the image generation model, etc.) while ensuring performance, so that the image generation model can be deployed on hardware-constrained devices.

[0288] Based on the image generation method provided in the embodiments of the present disclosure, the embodiments of the present disclosure also provide an image generation device, which will be explained and illustrated below in conjunction with Figure 5. Figure 5 is a schematic structural diagram of the image generation device provided in the embodiments of the present disclosure. It should be noted that for the technical details of the image generation device provided in the embodiments of the present disclosure, please refer to the relevant content of the image generation method above.

[0289] As shown in FIG5 , an image generating apparatus 500 provided by an embodiment of the present disclosure includes:

[0290] The second acquisition unit 501 is used to acquire an image to be processed;

[0291] The image generation unit 502 is used to process the image to be processed using a pre-built image generation model to obtain an image generation result; the image generation model is constructed using any implementation of the model construction method provided in the embodiments of the present disclosure.

[0292] Based on the above description of the image generation device 500, it can be seen that, after obtaining an image to be processed, the image generation device 500 provided in the embodiments of the present disclosure can use a pre-built image generation model to process the image to be processed to obtain an image generation result. The image generation model has good performance, so the image generation result determined based on the image generation model is relatively good, which is conducive to improving the image generation effect.

[0293] In addition, an embodiment of the present disclosure also provides an electronic device, which includes a processor and a memory: the memory is used to store instructions or computer programs; the processor is used to execute the instructions or computer programs in the memory, so that the electronic device executes any implementation of the model building method or image generation method provided in the embodiment of the present disclosure.

[0294] Referring to FIG6 , a schematic diagram of the structure of an electronic device 600 suitable for implementing embodiments of the present disclosure is shown. Terminal devices in embodiments of the present disclosure may include, but are not limited to, mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. The electronic device shown in FIG6 is merely an example and should not limit the functionality or scope of use of embodiments of the present disclosure.

[0295] As shown in Figure 6, electronic device 600 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage device 608 into a random access memory (RAM) 603. Various programs and data required for the operation of electronic device 600 are also stored in RAM 603. Processing device 601, ROM 602, and RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to bus 604.

[0296] Typically, the following devices may be connected to the I / O interface 605: an input device 606 including, for example, a touch screen, a touchpad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 607 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 608 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 609. The communication device 609 may allow the electronic device 600 to communicate with other devices wirelessly or by wire to exchange data. Although FIG. 6 shows the electronic device 600 with various devices, it should be understood that not all of the devices shown are required to be implemented or present. More or fewer devices may alternatively be implemented or present.

[0297] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication device 609, or installed from the storage device 608, or installed from the ROM 602. When the computer program is executed by the processing device 601, the above-mentioned functions defined in the method of the embodiment of the present disclosure are performed.

[0298] The electronic device provided by the embodiment of the present disclosure and the method provided by the above embodiment belong to the same inventive concept. For technical details not fully described in this embodiment, please refer to the above embodiment, and this embodiment has the same beneficial effects as the above embodiment.

[0299] The embodiments of the present disclosure further provide a computer-readable medium having instructions or computer programs stored therein. When the instructions or computer programs are executed on a device, the device executes any implementation of the model building method or image generation method provided in the embodiments of the present disclosure.

[0300] It should be noted that the computer-readable medium mentioned above in the present disclosure may be a computer-readable signal medium or a computer-readable storage medium, or any combination of the two. A computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or component, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, device, or component. In the present disclosure, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, which carries computer-readable program code. Such a propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transport a program for use by or in conjunction with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium may be transmitted using any suitable medium, including but not limited to wires, optical cables, RF (radio frequency), etc., or any suitable combination thereof.

[0301] In some embodiments, the client and server can communicate using any currently known or later developed network protocol, such as HTTP (Hypertext Transfer Protocol), and can be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any currently known or later developed network.

[0302] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.

[0303] The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device can perform the method.

[0304] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages, or a combination thereof, including, but not limited to, object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on the remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0305] The flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flowchart or block diagram can represent a module, program segment, or a part of code, and the module, program segment, or a part of code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order than that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of the boxes in the block diagram and / or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0306] The units involved in the embodiments described in this disclosure may be implemented in software or hardware, wherein the name of a unit / module does not, in some cases, limit the unit itself.

[0307] The functions described above herein may be performed, at least in part, by one or more hardware logic components. For example, and without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chip (SOCs), complex programmable logic devices (CPLDs), and the like.

[0308] In the context of the present disclosure, a machine-readable medium can be a tangible medium that can contain or store a program for use by or in conjunction with an instruction execution system, device or equipment. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium can include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0309] It should be noted that the various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Reference can be made to the common and similar parts between the various embodiments. For the systems or devices disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple, and the relevant parts can be referred to the method description.

[0310] It should be understood that in the present disclosure, "at least one (item)" refers to one or more, and "plurality" refers to two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following items" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0311] It should also be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of additional identical elements in the process, method, article, or device comprising the element.

[0312] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein may be implemented directly using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), internal memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0313] The above description of the disclosed embodiments is intended to enable one skilled in the art to implement or use the present disclosure. Various modifications to these embodiments will be readily apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present disclosure. Therefore, the present disclosure is not limited to the embodiments shown herein, but is intended to be construed in the widest manner consistent with the principles and novel features disclosed herein.

Claims

1. A model building method, comprising: Acquire a training data set; wherein the training data set includes a first sample image and label information corresponding to the first sample image; Using the training data set to train the hypernetwork; Searching for teacher and student models from a trained hypernetwork; and Model distillation is performed based on the training data set, the teacher model, and the student model to obtain an image generation model.

2. The method according to claim 1, wherein: The training process of the hypernetwork includes: Sampling a first sub-network and a second sub-network from the super-network; wherein the network size of the first sub-network is larger than the network size of the second sub-network; According to the first sub-network, determining a teacher generator; According to the second sub-network, determining a student generator; Model distillation is performed based on the training data set, the teacher generator and the student generator to obtain the trained super network.

3. The method according to claim 2, wherein: The first subnetwork includes a largest subnetwork; The second sub-network includes at least one of a minimum sub-network and a random sub-network.

4. The method according to claim 2, wherein: The training data set also includes a second sample image; The performing model distillation according to the training data set, the teacher generator and the student generator to obtain the trained super network includes: Determine loss representation data of the teacher generator using the first sample image and label information corresponding to the first sample image; Determine loss representation data of the student generator using the second sample image and a first pseudo label corresponding to the second sample image; wherein the first pseudo label is obtained by the teacher generator processing the second sample image; The hypernetwork, the teacher generator, and the student generator are updated according to the loss characterization data of the teacher generator and the loss characterization data of the student generator.

5. The method according to claim 4, wherein: After updating the super network, the teacher generator and the student generator, the method further comprises: Continue to perform the step of determining the loss representation data of the teacher generator using the first sample image and the label information corresponding to the first sample image until a first stopping condition is reached.

6. The method according to claim 2, wherein: The training data set also includes a second sample image; The performing of model distillation according to the training data set, the teacher generator and the student generator to obtain the trained super network includes: updating the teacher generator using the first sample image and the label information corresponding to the first sample image; Determine loss representation data of the student generator using the second sample image and a second pseudo label corresponding to the second sample image; wherein the second pseudo label is obtained by processing the second sample image by the updated teacher generator; According to the loss characterization data of the student generator, the hypernetwork, the teacher generator and the student generator are updated, and the step of updating the teacher generator using the first sample image and the label information corresponding to the first sample image is continued until the second stop condition is reached.

7. The method according to claim 1, wherein: The searching of the teacher model and the student model from the trained hypernetwork includes: According to a first model constraint, searching for the student model from the trained hypernetwork so that the student model satisfies the first model constraint; According to the second model constraint, at least one teacher model is searched from the trained hypernetwork so that each of the teacher models satisfies the second model constraint and the at least one teacher model satisfies a preset model architecture condition.

8. The method according to claim 7, wherein: The at least one teacher model includes a teacher model that satisfies a preset model depth condition and a teacher model that satisfies a preset model width condition; and / or, The second model constraint is determined based on the first model constraint so that each of the teacher model and the student model satisfies a preset multiple condition on one or more model indicators.

9. The method according to claim 1, wherein: The training data set also includes a second sample image; The performing model distillation according to the training data set, the teacher model and the student model to obtain an image generation model includes: updating the teacher model using the first sample image and label information corresponding to the first sample image; The student model is updated using the second sample image and the third pseudo label corresponding to the second sample image, and the step of updating the teacher model using the first sample image and the label information corresponding to the first sample image is continued until the image generation model is determined according to the student model when the third stop condition is reached; wherein the third pseudo label is obtained by processing the second sample image by the updated teacher model.

10. The method according to claim 9, wherein: The updating of the student model by using the second sample image and the third pseudo label corresponding to the second sample image includes: The student model is updated using the first sample image, the fourth pseudo label corresponding to the first sample image, the second sample image, and the third pseudo label corresponding to the second sample image; wherein the fourth pseudo label is obtained by processing the first sample image by the updated teacher model.

11. The method according to claim 9 or 10, wherein: The updating process of the student model includes: determining a first distillation loss of the student model using the second sample image and a third pseudo label corresponding to the second sample image; Determining loss characterization data of the student model according to the first distillation loss; The student model is updated according to the loss characterization data of the student model.

12. The method according to claim 11, wherein: Determining loss characterization data of the student model according to the first distillation loss includes: Determine the loss representation data of the student model according to the first distillation loss and the weighted weight corresponding to the first distillation loss; wherein the weighted weight is determined according to the confidence of the third pseudo label; and the confidence is determined according to the updated teacher model, the discriminator corresponding to the updated teacher model, and the second sample image.

13. The method according to claim 12, wherein: The process of determining the weighted weights includes: If the confidence level is lower than the confidence threshold, a value is randomly selected from the preset interval to determine the confidence level. is defined as the weighted weight; If the confidence level is not lower than the confidence threshold, a preset value is determined as the weighted weight.

14. The method according to claim 13, wherein: The process of determining the confidence threshold includes: Processing at least one image data using the updated teacher model to obtain a generated image corresponding to each image data; wherein the at least one image data includes the second sample image and / or the first sample image; Using the discriminator corresponding to the updated teacher model to perform discriminative processing on each of the generated images, to obtain a discriminative result corresponding to each of the generated images; Data analysis is performed on at least one of the discrimination results corresponding to the generated image to obtain the confidence threshold.

15. A method for generating an image, wherein: The method comprises: Get the image to be processed; The image to be processed is processed using a pre-constructed image generation model to obtain an image generation result; wherein the image generation model is constructed using the model construction method described in any one of claims 1 to 14.

16. A model building device comprising: A first acquisition unit is configured to acquire a training data set; wherein the training data set includes a first sample image and label information corresponding to the first sample image; A first training unit, configured to train the super network using the training data set; A network search unit configured to search for a teacher model and a student model from a trained hypernetwork; and The second training unit is configured to perform model distillation based on the training data set, the teacher model and the student model to obtain an image generation model.

17. An image generating device, comprising: A second acquisition unit, configured to acquire an image to be processed; as well as An image generation unit is configured to process the image to be processed using a pre-constructed image generation model to obtain an image generation result; wherein the image generation model is constructed using the model construction method described in any one of claims 1-14.

18. An electronic device, comprising: Processor and memory; The memory is used to store instructions or computer programs; The processor is configured to execute the instructions or the computer program in the memory so that the electronic device executes the method as claimed in any one of claims 1 to 15.

19. A computer readable medium, wherein: The computer-readable medium stores instructions or a computer program, and when the instructions or the computer program are executed on a device, the device is caused to perform the method according to any one of claims 1 to 15.

Images