Model updating method and device, equipment, storage medium and program product

Abstract

The invention relates to a model updating method and device, equipment, a storage medium and a program product. The method comprises the following steps: respectively utilizing a teacher model and a student model to determine first visual content and second visual content based on sample cue words; determining at least one key area in the sample visual content corresponding to the sample cue word; determining at least one local difference between predicted region information corresponding to the at least one key region in the second visual content and sample region information corresponding to the at least one key region in the sample visual content; and updating the student model based on the global difference and the at least one local difference between the first visual content and the second visual content.

Description

Technical Field

[0001] Example embodiments of the present disclosure generally relate to the field of computers, and more particularly, to methods, apparatuses, electronic devices, computer-readable storage media, and computer program products for model updating. Background Art

[0002] With the development of machine learning technology, machine learning models can now be used to perform tasks in a variety of application environments. For example, trained machine learning models can be used to generate content, which can be any suitable content such as video, images, and text. To meet the ever-increasing task requirements, machine learning models are being constructed with increasingly complex structures and an increasing number of parameters to support these complex tasks. Summary of the Invention

[0003] In a first aspect of the present disclosure, a method for model updating is provided. The method comprises: determining first visual content and second visual content based on a sample cue word using a teacher model and a student model, respectively; determining at least one key region in the sample visual content corresponding to the sample cue word; determining at least one local difference between predicted region information corresponding to the at least one key region in the second visual content and sample region information corresponding to the at least one key region in the sample visual content; and updating the student model based on the global difference and the at least one local difference between the first visual content and the second visual content.

[0004] In a second aspect of the present disclosure, a device for model updating is provided. The device includes: a visual content determination module configured to determine first visual content and second visual content based on sample prompt words using a teacher model and a student model respectively; a key area determination module configured to determine at least one key area in the sample visual content corresponding to the sample prompt word; a local difference determination module configured to determine at least one local difference between predicted area information corresponding to the at least one key area in the second visual content and sample area information corresponding to the at least one key area in the sample visual content; and a student model updating module configured to update the student model based on the global difference and at least one local difference between the first visual content and the second visual content.

[0005] In a third aspect of the present disclosure, an electronic device is provided. The device includes at least one processor; and at least one memory coupled to the at least one processor and storing instructions for execution by the at least one processor. When executed by the at least one processor, the instructions cause the electronic device to perform the method of the first aspect.

[0006] In a fourth aspect of the present disclosure, a computer-readable storage medium is provided, wherein computer-executable instructions are stored on the computer-readable storage medium, and when the computer-executable instructions are executed by a processor, the method of the first aspect is implemented.

[0007] In a fifth aspect of the present disclosure, a computer program product is provided, which includes computer-executable instructions, wherein the computer-executable instructions implement the method according to the first aspect of the present disclosure when executed by a processor.

[0008] It should be understood that the content described in this summary section is not intended to limit the key features or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. In the accompanying drawings, the same or similar reference numerals represent the same or similar elements, wherein:

[0010] Figure 1 A schematic diagram illustrating an example environment in which embodiments of the present disclosure can be implemented;

[0011] Figure 2 shows an example architecture for model updating according to some embodiments of the present disclosure;

[0012] Figure 3 A flowchart of a method for model updating according to some embodiments of the present disclosure is shown;

[0013] Figure 4 An exemplary structural block diagram of an apparatus for model updating according to some embodiments of the present disclosure is shown; and

[0014] Figure 5 A block diagram of an electronic device in which one or more embodiments of the present disclosure may be implemented is shown. DETAILED DESCRIPTION

[0015] The following describes embodiments of the present disclosure in more detail with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as limited to the embodiments described herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are for illustrative purposes only and are not intended to limit the scope of protection of the present disclosure.

[0016] In the description of the embodiments of the present disclosure, the term "including" and similar terms should be understood as open inclusion, that is, "including but not limited to". The term "based on" should be understood as "based at least in part on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The term "some embodiments" should be understood as "at least some embodiments". The terms "first", "second", etc. may refer to different or the same objects. Other explicit and implicit definitions may also be included below.

[0017] Herein, unless explicitly stated otherwise, executing a step “in response to A” does not mean executing the step immediately after “A” but may include one or more intermediate steps.

[0018] It is understandable that the data involved in this technical solution (including but not limited to the data itself, the acquisition, use, storage or deletion of the data) shall comply with the requirements of relevant laws, regulations and relevant provisions.

[0019] It is understandable that before using the technical solutions disclosed in the various embodiments of this disclosure, the type, scope of use, usage scenarios, etc. of the personal information involved in this disclosure should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations.

[0020] For example, in response to receiving a user's active request, a prompt message is sent to the user to clearly remind the user that the operation requested to be performed will require obtaining and using the user's personal information, so that the user can independently choose whether to provide personal information to the electronic device, application, server or storage medium and other software or hardware that performs the operation of the technical solution of the present disclosure based on the prompt message.

[0021] As an optional but non-limiting implementation, in response to receiving a user's active request, a prompt message may be sent to the user, for example, in the form of a pop-up window, in which the prompt message may be presented in text form. Furthermore, the pop-up window may also include a selection control for the user to select "agree" or "disagree" to provide personal information to the electronic device.

[0022] It is understandable that the above notification and the process of obtaining user authorization are merely illustrative and do not constitute a limitation on the implementation of the present disclosure. Other methods that comply with relevant laws and regulations may also be applied to the implementation of the present disclosure.

[0023] As used herein, the term "model" can learn the association between corresponding inputs and outputs from training data, so that after training is completed, corresponding outputs can be generated for given inputs. The generation of the model can be based on machine learning technology. Deep learning is a machine learning algorithm that processes inputs and provides corresponding outputs by using multiple layers of processing units. A neural network model is an example of a model based on deep learning. In this article, "model" may also be referred to as "machine learning model", "learning model", "machine learning network" or "learning network", and these terms are used interchangeably in this article.

[0024] A "neural network" is a machine learning network based on deep learning. A neural network is capable of processing inputs and providing corresponding outputs. It typically includes an input layer, an output layer, and one or more hidden layers between the input and output layers. Neural networks used in deep learning applications typically include many hidden layers, thereby increasing the depth of the network. The layers of a neural network are connected in sequence so that the output of the previous layer is provided as input to the next layer, where the input layer receives the input of the neural network and the output of the output layer serves as the final output of the neural network. Each layer of a neural network includes one or more nodes (also called processing nodes or neurons), each of which processes the input from the previous layer.

[0025] Generally speaking, machine learning can be roughly divided into three stages, namely the training stage, the testing stage, and the application stage (also called the inference stage). In the training stage, a given model can be trained using a large amount of training data, and the parameter values ​​are continuously updated iteratively until the model can obtain consistent inferences that meet the expected goals from the training data. Through training, the model can be considered to be able to learn the association between input and output (also called input-to-output mapping) from the training data. The parameter values ​​of the trained model are determined. In the testing stage, the test input is applied to the trained model to test whether the model can provide the correct output, thereby determining the performance of the model. The testing stage can sometimes be integrated into the training stage. In the application or inference stage, the trained model can be used to process the actual model input based on the parameter values ​​obtained through training to determine the corresponding model output.

[0026] Diffusion models, also known as diffusion probability models, are a type of generative model. They generate data by simulating the diffusion process, which is inspired by physical processes such as heat diffusion. Diffusion models include forward diffusion and reverse diffusion. Diffusion models simulate a forward diffusion process that gradually adds noise and then learn how to reverse this process to generate new data samples.

[0027] During forward diffusion, noise is gradually added to the data, making it increasingly random over a series of steps until the data resembles pure noise. This process can be thought of as a Markov chain, where Gaussian noise is added to the data at each step. The forward diffusion process can be expressed as: where x t is the noise data of step t, α t Used to control the amount of noise added. The forward diffusion process is performed during model training, and the data used to add noise are the training samples.

[0028] In the reverse diffusion process (also known as reverse denoising), the model learns to reverse the steps of adding noise. Starting with pure noise, the diffusion model gradually removes the noise to generate data that matches the training distribution. The reverse diffusion process is usually simulated using a neural network that predicts the noise added at each step: where u θ and σ θ are the learned model parameters. After completing model training, the model performing the back-diffusion process can first start sampling from the noise distribution and iteratively denoise by using until the desired data is obtained.

[0029] In diffusion models, the time step refers to the number of noise addition steps during the forward diffusion process. The total number of steps, T, is typically a preset value, representing the number of steps required to transform the original data into pure noise. At each time step, t, Gaussian noise is added to the data according to a predetermined noise scheme. This process is continuous, and each step depends on the results of the previous step.

[0030] When generating data, the diffusion model inference step (Inference Step) refers to the number of steps required to recover the original data from pure noise during the back diffusion process. The number of inference steps directly affects the quality and speed of generated data. Generally, the more inference steps there are, the higher the quality of the generated data is, but it also increases the computational cost and time. In practical applications, the generation quality and efficiency can be balanced by adjusting the number of inference steps. In some embodiments, the inference step corresponds to the time step, and each inference step can correspond to one or more time steps. For example, if the total time step of the diffusion model is 1000 steps and the inference step is set to 50 steps, then each inference step can correspond to 20 time steps.

[0031] The basic principle of model distillation is to transfer the knowledge of a complex model to a simplified student model by introducing a teacher model. The teacher model is usually a large, complex model with high accuracy and generalization performance, while the student model is a smaller, simpler model with lower computational and storage costs.

[0032] The content generation process based on the diffusion model generates content by iteratively performing multiple denoising operations. To reduce model complexity and improve model inference speed, model distillation aims to reduce the denoising operations in the content generation process of the diffusion model. For example, a model that originally required multiple denoising inference steps can be reduced to a model that performs fewer denoising inference steps.

[0033] Classifier-Free Guidance (CFG) technology can improve the generation effect of machine learning models. The core of CFG technology is to use the model to determine the positive output corresponding to the positive prompt word and the negative output corresponding to the negative prompt word, and determine the final model output based on the guidance scale, positive output and negative output. Positive prompt words can guide the model to generate content that matches the positive prompt words, while negative prompt words can guide the model to avoid generating content related to the negative prompt words. In addition, the guidance scale used can be used to control the degree of influence of the positive prompt words and negative prompt words on the model output. The model output using CFG technology will be higher in quality than the model output obtained directly based on a single prompt word (for example, only positive prompt words), which can effectively improve the quality of the model output and improve the task execution capability of the model.

[0034] Figure 1 A schematic diagram of an example environment 100 in which embodiments of the present disclosure can be implemented is shown. In this example environment 100, a model update system 120 can be deployed in an electronic device 110. The model update system 120 can also be referred to as a model distillation system. Model distillation is a technique for transferring knowledge from a complex model (also referred to as a teacher model) to a simpler model (also referred to as a student model). Its core goal is to significantly reduce the computational complexity and storage requirements of the model while maintaining model performance.

[0035] The model update system 120 can, for example, use the training samples 105 to perform model distillation on the teacher model 130 and the student model 140. The teacher model 130 is typically a large, complex model with high accuracy and generalization performance, while the student model 140 is a smaller, simpler model with lower computational and storage costs. The teacher model 130 and the student model 140 can be used to perform the same task. As an example, the teacher model 130 and the student model 140 can both be configured to perform a visual generation task. In a visual generation task, the model input can include text, images, and / or videos, and the model output is visual content generated based on the model input. The output visual content includes images or videos. Both the teacher model 130 and the student model 140 can be based on any appropriate model structure, including but not limited to a Transformer model, a convolutional neural network (CNN), a recurrent neural network (RNN), a deep neural network (DNN), and the like. In some embodiments, both the teacher model 130 and the student model 140 can be implemented based on a diffusion model.

[0036] During the training phase, a machine learning model based on a diffusion model can train the model by continuously adding noise to known data and predicting the added noise. During the application phase, a machine learning model based on a diffusion model can continuously add noise to the model input to determine the noise corresponding to the model input, and then continuously perform a denoising process on the noise to obtain the model output corresponding to the model input. For example, when generating a video, a machine learning model based on a diffusion model can generate a video that meets the conditions by continuously performing a denoising process on random noise under certain constraints.

[0037] The model updating system 120 may, for example, process the training samples 105 using the teacher model 130 and the student model 140, respectively, and obtain the model outputs of the teacher model 130 and the student model 140. The model updating system 120 may, for example, update the student model 140 based on the difference between the model outputs of the teacher model 130 and the model outputs of the student model 140.

[0038] The electronic device 110 may be any type of device with computing capabilities, including a terminal device or a server device. The terminal device may be any type of mobile terminal, fixed terminal, or portable terminal, including a mobile phone, a desktop computer, a laptop computer, a notebook computer, a netbook computer, a tablet computer, a media computer, a multimedia tablet, a personal communication system (PCS) device, a personal navigation device, a personal digital assistant (PDA), an audio / video player, a digital camera / camcorder, a positioning device, a television receiver, a radio broadcast receiver, an e-book device, a gaming device, or any combination thereof, including accessories and peripherals of these devices or any combination thereof.

[0039] A server-side device can be a standalone physical server, a server cluster or distributed system consisting of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content distribution networks, and big data and artificial intelligence platforms. Server-side devices may include, for example, computing systems / servers such as mainframes, edge computing nodes, and computing devices in cloud environments.

[0040] It should be understood that the structure and function of the various elements in the environment 100 are described for illustrative purposes only and do not imply any limitation on the scope of the present disclosure.

[0041] As mentioned earlier, to meet the increasing demands of tasks, machine learning models are being constructed with increasingly complex structures and larger numbers of parameters to support these complex tasks. Therefore, model distillation techniques are needed to help accelerate these models.

[0042] When both the teacher model and the student model used for distillation are configured to perform visual generation tasks, traditionally, methods such as consistency loss-based distillation, generative adversarial network (GAN) loss-based distillation, and CFG distillation are used to update the student model to obtain a student model that meets the model distillation objectives. However, these distillation methods generally improve the overall distillation effect of the student model, that is, they can improve the overall effect of the visual content output by the student model. These distillation methods generally ignore feature changes in a few key areas of the visual content. Taking the visual content output by the student model including people as an example, these distillation methods generally ignore facial and mouth movements, hand distortion, etc.

[0043] In view of this, according to an embodiment of the present disclosure, an improved scheme for model updating is provided, which proposes determining the global difference between the output of the student model and the output of the teacher model, as well as the local difference between the key area of ​​the student model and the key area of ​​the sample visual content, and updating the student model based on the global difference and the local difference to ensure the details of the model output of the student model. This can help improve the effect of model distillation. According to the scheme of an embodiment of the present disclosure, the teacher model and the student model are respectively used to determine the first visual content and the second visual content based on the sample prompt word. Determine at least one key area in the sample visual content corresponding to the sample prompt word. Determine at least one local difference between the predicted area information corresponding to the at least one key area in the second visual content and the sample area information corresponding to the at least one key area in the sample visual content. Update the student model based on the global difference and at least one local difference between the first visual content and the second visual content.

[0044] In this way, local differences between the visual content output by the student model and the sample visual content are introduced, and the student model is updated jointly with global differences. This allows the student model to focus on processing local features, significantly reducing local distortion after distillation, thereby helping to improve the performance of the distilled student model. Such a student model can support more efficient and accurate visual content generation.

[0045] The following will continue to describe some example embodiments of the present disclosure with reference to the accompanying drawings. The model updating method involved in the present disclosure can be implemented at the electronic device 110, and specifically can be implemented at the model updating system 120. The present disclosure uses the teacher model 130 and the student model 140 as an example to describe the example of performing a visual content generation task (that is, the model outputs of the teacher model 130 and the student model 140 are both visual content). The visual content includes images and / or videos.

[0046] Figure 2 An example architecture 200 for model updating according to some embodiments of the present disclosure is shown. The example architecture 200 can be implemented at the model updating system 120. Block 210 in the example architecture 200 shows an example process of performing a model update (which can be referred to as performing a model iteration) on the student model 140.

[0047] The training samples 105 for model distillation may include sample prompt words 211, and sample visual content 214 or sample visual features corresponding to the sample prompt words 211, also called true visual content or true visual features, which are used to measure whether the model output in the model distillation process deviates from the true output. The sample visual content 214 is the specific visual content in the pixel space, and the sample visual features are the visual feature representations in the feature space. It can be understood that the visual content in the pixel space and the visual feature representations in the feature space can be converted into each other. For example, the sample visual features in the feature space can be obtained by performing feature extraction on the sample visual content 214 in the pixel space, and the sample visual content 214 in the pixel space can be obtained by performing image generation on the sample visual features in the feature space. Of course, the training samples 105 may also include both sample visual content 214 and sample visual features, which is not limited in this disclosure. In some embodiments, when both the teacher model 130 and the student model 140 are based on the diffusion model, the training sample 105 may also include a time step. The time step may be any appropriate value, which may be input by the user or determined randomly. This disclosure does not limit this.

[0048] In some embodiments, CFG technology can be used to implement model distillation. In this case, during the distillation process, a weighted average (e.g., an exponential moving average (EMA)) of the student model 140 can be used as the teacher model 130, while distillation is performed in conjunction with CFG. That is, the student model 140 is allowed to learn the CFG reasoning results of the teacher model 130. By performing model consolidation based on CFG reasoning, while ensuring the high reasoning efficiency of the student model 140, for example, the learning model 140 can only perform reasoning based on positive prompt words and not on negative prompt words, while still achieving the better generation effect brought by CFG reasoning.

[0049] When performing model distillation using training samples 105, the teacher model 130 and the student model 140 are respectively input with sample prompt words 211. Based on the current model parameters, the teacher model 130 generates visual content 215, while the student model 140 generates visual content 216. In some embodiments, if the teacher model 130 and the student model 140 are constructed based on a diffusion model, then during the visual content generation process, the teacher model 130 and the student model 140 also perform visual content generation based on the time step indicated in the training samples 105. In each training sample 105, the time step is randomly specified. This time step may not be the time step expected by the teacher model 130 or the student model 140 to generate visual content that ultimately meets quality requirements, but is a random value between 0 and the maximum time step.

[0050] In some embodiments, when the CFG technology is used to implement model distillation, the sample prompt word 211 may include a negative prompt word 212 and a positive prompt word 213. The teacher model 130 is configured to generate visual content 215 (for example, which may be referred to as first visual content) based on the negative prompt word 212, the positive prompt word 213, and the guidance ratio. The guidance ratio may indicate the degree of influence of the positive prompt word 213 and the negative prompt word 212 on the visual content 215. As an example, the teacher model 130 may generate the visual content 215 based on the following formula:

[0051] t i =scale×t i c -(scale-1)×t i uc (1)

[0052] where t i , t i c and t i uc Respectively represent the final model output (i.e., visual content 215) of the teacher model 130 for the i-th positive prompt word, the positive output determined based on the positive prompt word 213, and the negative output determined based on the negative prompt word 214, and scale indicates the guidance ratio. In formula (1), the smaller the guidance ratio scale, the smaller the impact of the negative prompt word on the model output. When the guidance ratio scale = 1, it means that the negative prompt word has no effect on the model output, that is, the model does not need to perform reasoning based on the negative prompt word. It can be understood that formula (1) is only an example, and the CFG technology adopted by the teacher model 130 can actually be expressed based on other appropriate formulas. For example, the formula can be designed so that the larger the value of scale, the higher the impact of the negative prompt word on the model output.

[0053] In the following, for the purpose of illustration, some embodiments of the present disclosure are described by taking the model distillation based on CFG technology as an example, but it should be understood that the embodiments of the present disclosure can be applied to the model distillation not based on CFG.

[0054] In the initial stage of model distillation, the student model 140 is configured to be the same as the teacher model 130 (that is, initially, the guidance ratios of the student model 140 and the teacher model 130 are often set to the same value.). The purpose of model distillation is to enable the student model 140 to generate visual content 216 (e.g., which can be referred to as second visual content) based only on the positive cue words 213. For example, in the CFG reasoning based on formula (1), the guidance ratio scale used by the student model 140 is 1. Therefore, as the number of iterations of model training increases, the guidance ratio can be gradually reduced to reduce the impact of negative cue words on the model output of the student model, thereby avoiding oversaturation and other quality issues in the generated results.

[0055] In some embodiments, the model updating system 120 may further determine a global difference 217 between the visual content 215 output by the teacher model 130 and the visual content 216 output by the student model 140. The model updating system 120 may use any appropriate loss function, such as mean squared error (MSE), cross-entropy loss, or categorical cross-entropy loss, to calculate the global difference 217 between the visual content 215 and the visual content 216. For example, in the process of performing inference based on the MSE loss, the loss function may be expressed as L s =MSE(s i ,t i ), where s i represents the visual content output by the student model 140 for the i-th positive prompt word; t i represents the visual content output by the teacher model 130 for the i-th positive prompt word (and possible negative prompt word).

[0056] MSE loss measures the mean square difference between the visual content output by the teacher model and the student model. This loss calculates the square of the Euclidean distance between the two model output vectors. Cross entropy loss measures the difference between the output distribution of the student model and the output distribution of the teacher model. It reflects the information difference between the predicted distribution of the student model and the predicted distribution of the teacher model. Multi-classification cross entropy loss is a specific form of cross entropy loss in multi-classification problems. It is applicable to situations where the model output has multiple categories. It measures the difference between the category distribution predicted by the student model in the multi-classification task and the category distribution of the teacher model. It should be understood that the present disclosure does not limit the loss function of model distillation.

[0057] In an embodiment of the present disclosure, in addition to global loss, the model update system 120 also considers local loss. Specifically, the model update system 120 determines at least one key area 221 in the sample visual content 214 included in the current training sample 105. The key area refers to an area in the overall visual content that needs to be focused on or interested in. The model update system 120 can obtain a predetermined condition, and the predetermined condition can indicate an object category or a requirement for a key area. The model update system 120 can then determine the area in the sample visual content 214 that meets the predetermined condition as a key area. For example, if the object category indicates that it includes faces and hands, the model update system 120 can determine the area of ​​the sample visual content 214 that includes the image of the face and the area of ​​the image that includes the hand as the key area. For another example, if the requirement for the key area indicates that the key area is located in the foreground of the image, the model update system 120 can determine the area in the sample visual content 214 that meets the requirement for the key area as the key area. It can be understood that these are just some examples. In fact, specific predetermined conditions can be determined based on actual needs. For example, the predetermined conditions can indicate that the object categories include goods, animals, specified items, etc., and the requirements for key areas can indicate that the key areas are located at a specified position (such as the center of the image, etc.).

[0058] The model update system 120 can determine at least one key area 221 in the sample visual content 214 in any appropriate manner. As an example only, in some embodiments, the model update system 120 can use a trained detection model to determine at least one key area 221 that meets predetermined conditions from the sample visual content 214. The detection model can be based on any appropriate model such as a classification model, a foreground segmentation model, a language model, etc., and the present disclosure is not limited to this. The model update system 120 can provide the predetermined conditions and the sample visual content 214 to the detection model and obtain the model output of the detection model. The model output of the detection model can indicate the location information of at least one key area 221 in the sample visual content 214 that meets the predetermined conditions in the sample visual content 214.

[0059] It should be noted that in some embodiments, the size of the key region is related to the size of the object within the key region. The detection model can classify each pixel of the sample visual content 214 in the pixel space at the pixel level and determine the key region based on the classification results. For example, if the predetermined condition indicates that the key region includes a face and the sample visual content 214 is an image, the detection model can determine multiple pixels in the image that are classified as faces and, based on these multiple pixels, completely frame the region within the image where the face is located. The size of this region depends on the size of the face.

[0060] In some embodiments, the size of the key region may also be fixed. The model updating system 120 may divide the sample visual content 214 into multiple regions based on the size of the sample visual content 214 and a predetermined division rule. By way of example only, this division rule may indicate the number of regions obtained by dividing the sample visual content 214, the size of each region, and the like. The model updating system 120 may then determine, among these multiple regions, those that meet the predetermined conditions as key regions.

[0061] The model updating system 120 can then determine sample region information corresponding to at least one key region 221 in the sample visual content 214 based on the position information of the at least one key region 221 in the sample visual content 214. Taking the example of a training sample 105 directly including the sample visual content 214, the sample region information can be pixel information in pixel space, which can indicate the content of each of the at least one key region 221 in the sample visual content 214. For example, the model updating system 120 can directly determine the content of each of the at least one key region 221 based on the position of each of the at least one key region 221 output by the detection model, and then determine the sample region information in pixel space based on the content of each of the at least one key region 221.

[0062] Taking the training sample 105 including the sample visual feature as an example, the sample visual feature can be represented as a feature sequence. The feature sequence can include multiple feature units (or tokens), each feature unit being able to characterize an area of ​​the sample visual content 214. For example, if the feature sequence corresponding to the visual feature representation is [A, B, C, D], then the feature unit A in the feature sequence can correspond to the upper left corner area A in the sample visual content 214, the feature unit B can correspond to the upper right corner area B in the sample visual content 214, the feature unit C can correspond to the lower left corner area C in the sample visual content 214, and the feature unit D can correspond to the lower right corner area D in the sample visual content 214.

[0063] In this case, the sample area information determined by the model updating system 120 can be feature information in the feature space. Specifically, the model updating system 120 can, for example, determine the position of at least one feature unit corresponding to at least one key area in the feature sequence. The model updating system 120 can, for example, first convert the sample visual features into sample visual content 214, and determine at least one key area from the sample visual content with the help of the detection model. The model updating system 120 can then determine the position of at least one feature unit corresponding to at least one key area in the feature sequence based on the correspondence between each feature unit in the feature sequence of the sample visual features and multiple areas of the sample visual content 214. Of course, the model updating system 120 can also directly use the detection model to determine the position of at least one feature unit in the feature sequence based on the sample visual features, and the present disclosure is not limited to this. The model updating system 120 can then determine the feature value of each of the at least one feature units in the feature sequence based on the respective positions of the at least one feature unit, and determine the sample area information in the feature space based on the feature value of each of the at least one feature unit. That is, the sample area information can indicate the feature value of each of the at least one feature units.

[0064] The model updating system 120 can determine the predicted region information corresponding to the at least one key region 221 in the visual content 216 output by the student model 140. The size of the visual content 216 output by the student model 140 should be the same as the size of the sample visual content 214. For example, the model updating system 120 can determine the position of each of the at least one predicted region in the visual content 216 corresponding to the at least one key region based on the position of each of the at least one key region 221 in the sample visual content 214. For example, if region A at position A in the sample visual content 214 is the key region in the sample visual content 214, then region A′ at position A in the visual content 216 can be determined as the predicted region in the visual content 216.

[0065] Model updating system 120 can determine prediction region information corresponding to at least one prediction region in visual content 216. Similar to sample region information, prediction region information can be feature information in a feature space or pixel information in a pixel space. If the prediction region information is pixel information in a pixel space, it can indicate the content of each of the at least one prediction region in visual content 216. If the prediction region information is feature information in a feature space, it can indicate the feature value of each of at least one feature unit corresponding to the at least one prediction region in a feature sequence of a visual feature representation corresponding to visual content 216.

[0066] The model updating system 120 can determine at least one local difference 222 between the sample region information and the prediction region information. For example, taking the sample region information and the prediction region information as pixel information in a pixel space as an example, if the visual content 216 includes prediction region A', prediction region B', and prediction region C', and the sample visual content 214 includes key region A, key region B, and key region C, wherein the corresponding positions of prediction region A' and key region A are the same, the corresponding positions of prediction region B' and key region B are the same, and the corresponding positions of prediction region C' and key region C are the same, then the model updating system 120 can determine the local difference between prediction region A' and key region A, the local difference between prediction region B' and key region B, and the local difference between prediction region B' and key region B based on the respective contents of prediction region A', prediction region B', prediction region C', key region A, key region B, and key region C.

[0067] Similarly, if the sample area information and the predicted area information are feature information in the feature space, the sample area information may indicate at least one field in the sequence corresponding to the sample visual feature, and the predicted area information may indicate at least one field in the sequence corresponding to the visual content 216. The model updating system 120 may be based on the local differences between the corresponding fields in at least one field in the sequence corresponding to the sample visual feature and at least one field in the sequence corresponding to the visual content 216.

[0068] The model updating system 120 may use any appropriate loss function to calculate at least one local difference 222 between the sample region information and the predicted region information. For example, the model updating system 120 may use any appropriate loss function to calculate at least one local difference 222 between the sample region information and the predicted region information. gt =gt_func(s i ,gt,Area) is a true value-based loss function to calculate at least one local difference 222, where gt is the abbreviation of groundtruth, which can represent the content of at least one key area or the feature value of at least one feature unit, and Area represents the position information of at least one key area in the sample visual content 214.

[0069] The model updating system 120 can update the student model 140 based on the global difference 217 and the at least one local difference 222 between the first visual content and the second visual content. For example, the model updating system 120 can determine a first weight corresponding to the global difference 217 and a second weight corresponding to each of the at least one local difference 222. If the at least one key region includes multiple key regions, the model updating system 120 can determine multiple corresponding second weights.

[0070] Multiple second weights may be the same or different. For example, in some embodiments, the second weight corresponding to each of the multiple key areas may at least indicate the importance of the corresponding key area. This importance refers to the importance of the key area relative to the sample visual content. This importance can be determined based on any appropriate method. For example, if two key areas are included, key area A includes a person's face image, and key area B includes a person's hand image, it can be determined that the importance of key area A is higher than the importance of key area B. If both key areas include a person's hand image, but the size of the hand image in key area A is larger than the size of the hand image in key area B, it can be determined that the importance of key area A is higher than the importance of key area B. This disclosure does not limit the specific method of determining importance. It can be understood that the greater the importance corresponding to the key area, the greater the corresponding second weight.

[0071] The model updating system 120 may then determine a loss function for updating the student model 140 based on the first weight, at least one second weight corresponding to at least one local difference, the global difference, and at least one local difference. For example, the loss function may be based on L = αL s +(β1L gt1 +β2L gt2 +…+β N L gtN ) is used to calculate the loss L for updating the student model 140, where α represents the global difference L s The first weight, β N Represents the Nth local difference L gtN The model updating system 120 can update the student model 140 by reducing the loss L of the loss function.

[0072] In some embodiments, the first weight can be set to 1. In some embodiments, the user can also adjust the first weight and at least one second weight in real time based on their own needs. For example, if the user wants the student model to pay more attention to details, the user can increase the second weight; if the user wants the student model to pay more attention to the overall situation, the user can decrease the second weight.

[0073] The model update system 120 may perform multiple updates (or may be referred to as multiple iterations) on the student model 140 by repeatedly executing the example process shown in block 201. It is understood that the positive prompt words and negative prompt words applied to different updates in the multiple updates are different. After each execution of the example process shown in block 201, the model update system 120 may determine (230) whether the model update target corresponding to the student model 140 is satisfied. The model update target may be, for example, based on the number of updates of the student model 140 reaching a predetermined number (e.g., reaching a maximum number of iterations S). max). Taking the predetermined number of times as 500 as an example, the model update system 120 can also determine that the model update target is met in response to the number of updates of the student model 140 reaching 500. The model update target can be, for example, based on the content quality of the visual content output by the updated student model 140 reaching a threshold quality. That is, the model update system 120 can determine that the model update target is met in response to the content quality of the visual content output by the updated student model 140 being high. Currently, the model update target can also be based on any other appropriate content, and the present disclosure does not limit the model update target.

[0074] In response to the model update target being met, the model update system 120 may stop (240) updating the student model 140. At this point, the model distillation process for the learning model 140 may be considered complete. The model update system 120 may also continue updating the student model 140 in response to the model update target not being met, i.e., continue to repeatedly execute the process shown in block 210.

[0075] In summary, according to the embodiments of the present disclosure, local differences between the visual content output by the student model and the sample visual content are introduced, and the student model is updated jointly with the local and global differences. This can focus on the learning ability of local features and significantly reduce the local distortion problem after the student model distillation. This helps to improve the updating effect of the student model.

[0076] Figure 3 FIG. 3 is a flow chart showing a method 300 for model updating according to some embodiments of the present disclosure. The method 300 may be implemented in Figure 1 The electronic device 110 will refer to Figure 1 The method 300 is described with reference to the environment 100 of FIG.

[0077] In block 310 , the electronic device 110 determines first visual content and second visual content based on sample cue words using a teacher model and a student model, respectively.

[0078] In block 320 , the electronic device 110 determines at least one key area in the sample visual content corresponding to the sample prompt word.

[0079] In block 330 , the electronic device 110 determines at least one local difference between the predicted region information corresponding to the at least one key region in the second visual content and the sample region information corresponding to the at least one key region in the sample visual content.

[0080] In block 340 , the electronic device 110 updates the student model based on the global difference and the at least one local difference between the first visual content and the second visual content.

[0081] In some embodiments, determining at least one key area in the sample visual content corresponding to the sample prompt word includes: using a trained detection model to determine at least one key area that meets predetermined conditions from the sample visual content, where the predetermined conditions indicate an object category or requirements for the key area.

[0082] In some embodiments, the prediction region information indicates at least one prediction region corresponding to at least one key region.

[0083] In some embodiments, updating the student model based on the global difference and the local difference includes: determining a first weight corresponding to the global difference and a second weight corresponding to at least one local difference; determining the loss of a loss function used to update the student model based on the first weight, at least one second weight corresponding to at least one local difference, the global difference and at least one local difference; and updating the student model by reducing the loss of the loss function.

[0084] In some embodiments, the at least one key area includes a plurality of key areas, and the second weight of each of the plurality of key areas indicates at least the importance of the corresponding key area.

[0085] In some embodiments, the prediction region information and the sample region information are both feature information in a feature space or pixel information in a pixel space.

[0086] In some embodiments, the teacher model is configured to generate a first visual content based on a positive cue word, a negative cue word, and a guidance ratio, wherein the guidance ratio indicates the degree of influence of the positive cue word and the negative cue word on the first visual content, and the student model is configured to generate a second visual content based on the positive cue word.

[0087] In some embodiments, method 300 further includes: stopping updating the student model in response to a model update target of the student model being met, wherein the model update target is based on at least one of the following: the number of updates of the student model reaches a predetermined number, or the content quality of the visual content output by the updated student model reaches a threshold quality.

[0088] The embodiments of the present disclosure also provide corresponding devices for implementing the above methods or processes. Figure 4 FIG4 shows an exemplary structural block diagram of an apparatus 400 for model updating according to some embodiments of the present disclosure. The apparatus 400 may be implemented as or included in the electronic device 110. Each module / component in the apparatus 400 may be implemented by hardware, software, firmware, or any combination thereof.

[0089] like Figure 4As shown, the device 400 includes a visual content determination module 410, which is configured to determine the first visual content and the second visual content based on the sample prompt word using a teacher model and a student model respectively. The device 400 also includes a key area determination module 420, which is configured to determine at least one key area in the sample visual content corresponding to the sample prompt word. The device 400 also includes a local difference determination module 430, which is configured to determine at least one local difference between the predicted area information corresponding to the at least one key area in the second visual content and the sample area information corresponding to the at least one key area in the sample visual content. The device 400 also includes a student model update module 440, which is configured to update the student model based on the global difference and at least one local difference between the first visual content and the second visual content.

[0090] In some embodiments, the key area determination module 420 is further configured to: utilize the trained detection model to determine at least one key area satisfying a predetermined condition from the sample visual content, where the predetermined condition indicates an object category or a requirement for the key area.

[0091] In some embodiments, the prediction region information indicates at least one prediction region corresponding to at least one key region.

[0092] In some embodiments, the student model update module 440 is further configured to: determine a first weight corresponding to a global difference and a second weight corresponding to at least one local difference; determine the loss of a loss function for updating the student model based on the first weight, at least one second weight corresponding to at least one local difference, the global difference and at least one local difference; and update the student model by reducing the loss of the loss function.

[0093] In some embodiments, the at least one key area includes a plurality of key areas, and the second weight of each of the plurality of key areas indicates at least the importance of the corresponding key area.

[0094] In some embodiments, the prediction region information and the sample region information are both feature information in a feature space or pixel information in a pixel space.

[0095] In some embodiments, the teacher model is configured to generate a first visual content based on a positive cue word, a negative cue word, and a guidance ratio, wherein the guidance ratio indicates the degree of influence of the positive cue word and the negative cue word on the first visual content, and the student model is configured to generate a second visual content based on the positive cue word.

[0096] In some embodiments, the device 400 also includes: a stop update module, configured to stop updating the student model in response to the model update target of the student model being met, wherein the model update target is based on at least one of the following: the number of updates of the student model reaches a predetermined number of times, or the content quality of the visual content output by the updated student model reaches a threshold quality.

[0097] The units and / or modules included in the device 400 can be implemented in various ways, including software, hardware, firmware, or any combination thereof. In some embodiments, one or more units and / or modules can be implemented using software and / or firmware, such as machine executable instructions stored on a storage medium. In addition to or as an alternative to machine executable instructions, some or all of the units and / or modules in the device 400 can be implemented at least in part by one or more hardware logic components. By way of example and not limitation, exemplary types of hardware logic components that can be used include field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chip (SOCs), complex programmable logic devices (CPLDs), and the like.

[0098] It should be understood that one or more steps in the above method can be performed by a suitable electronic device or combination of electronic devices. Such an electronic device or combination of electronic devices may include, for example, Figure 1 The electronic device 110 in.

[0099] Figure 5 1 shows a block diagram of an electronic device 500 in which one or more embodiments of the present disclosure may be implemented. Figure 5 The illustrated electronic device 500 is merely exemplary and should not be construed as limiting the functionality and scope of the embodiments described herein. Figure 5 The electronic device 500 shown can be used to implement Figure 1 electronic device 110 or Figure 4 device 400.

[0100] like Figure 5 As shown, electronic device 500 is in the form of a general electronic device. Components of electronic device 500 may include, but are not limited to, one or more processing units or processors 510, memory 520, storage device 530, one or more communication units 540, one or more input devices 550, and one or more output devices 560. Processor 510 may be a real or virtual processor and is capable of performing various processes according to programs stored in memory 520. In a multi-processor system, multiple processors execute computer-executable instructions in parallel to increase the parallel processing capabilities of electronic device 500.

[0101] The electronic device 500 typically includes a plurality of computer storage media. Such media can be any accessible media that can be obtained by the electronic device 500, including but not limited to volatile and non-volatile media, removable and non-removable media. The memory 520 can be a volatile memory (e.g., registers, cache, random access memory (RAM)), a non-volatile memory (e.g., read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory), or some combination thereof. The storage device 530 can be a removable or non-removable medium and can include a machine-readable medium, such as a flash drive, a disk, or any other medium that can be used to store information and / or data and can be accessed within the electronic device 500.

[0102] The electronic device 500 may further include additional removable / non-removable, volatile / non-volatile storage media. Figure 5 As shown in FIG, a magnetic disk drive for reading from or writing to a removable, non-volatile magnetic disk (e.g., a "floppy disk") and an optical disk drive for reading from or writing to a removable, non-volatile optical disk may be provided. In these cases, each drive may be connected to a bus (not shown) by one or more data media interfaces. Memory 520 may include a computer program product 525 having one or more program modules configured to perform various methods or actions of various embodiments of the present disclosure.

[0103] The communication unit 540 enables communication with other electronic devices via a communication medium. Additionally, the functions of the components of the electronic device 500 can be implemented in a single computing cluster or multiple computing machines that can communicate via a communication connection. Thus, the electronic device 500 can operate in a networked environment using a logical connection with one or more other servers, a network personal computer (PC), or another network node.

[0104] Input device 550 may be one or more input devices, such as a mouse, keyboard, or trackball. Output device 560 may be one or more output devices, such as a display, a speaker, or a printer. Electronic device 500 may also communicate with one or more external devices (not shown) via communication unit 540 as needed, such as a storage device, a display device, or the like, with one or more devices that allow a user to interact with electronic device 500, or with any device that allows electronic device 500 to communicate with one or more other electronic devices (e.g., a network card, a modem, etc.). Such communication may be performed via an input / output (I / O) interface (not shown).

[0105] According to an exemplary implementation of the present disclosure, a computer-readable storage medium is provided, on which computer-executable instructions are stored, wherein the computer-executable instructions are executed by a processor to implement the method described above. According to an exemplary implementation of the present disclosure, a computer program product is also provided, which is tangibly stored on a non-transitory computer-readable medium and includes computer-executable instructions, and the computer-executable instructions are executed by a processor to implement the method described above.

[0106] Various aspects of the present disclosure are described herein with reference to flowcharts and / or block diagrams of methods, apparatuses, devices, and computer program products implemented according to the present disclosure. It should be understood that each block of the flowcharts and / or block diagrams, and combinations of blocks in the flowcharts and / or block diagrams, can be implemented by computer-readable program instructions.

[0107] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine such that when these instructions are executed by the processor of the computer or other programmable data processing device, a device is generated that implements the functions / actions specified in one or more blocks in the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium, where these instructions cause the computer, programmable data processing device, and / or other device to operate in a specific manner. Thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing various aspects of the functions / actions specified in one or more blocks in the flowchart and / or block diagram.

[0108] Computer-readable program instructions can be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operational steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0109] The flow charts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the systems, methods and computer program products according to multiple implementations of the present disclosure. In this regard, each box in the flow chart or block diagram can represent a part of a module, program segment or instruction, and a part of a module, program segment or instruction comprises one or more executable instructions for realizing the logical function of the specification. In some updated implementations, the functions marked in the box can also occur in a sequence different from that marked in the accompanying drawings. For example, two continuous boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be realized by a special hardware-based system that performs the function or action of the specification, or can be realized by a combination of special hardware and computer instructions.

[0110] While various implementations of the present disclosure have been described above, the foregoing description is intended to be illustrative, not exhaustive, and not limited to the disclosed implementations. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described implementations. The terminology used herein is selected to best explain the principles of the implementations, their practical applications, or improvements to existing technologies, or to enable others skilled in the art to understand the various implementations disclosed herein.

Claims

1. A method for model updating, comprising: The teacher model and the student model are used to determine the first visual content and the second visual content based on the sample prompt words respectively; Determining at least one key area in the sample visual content corresponding to the sample prompt word; determining at least one local difference between the predicted region information corresponding to the at least one key region in the second visual content and the sample region information corresponding to the at least one key region in the sample visual content; and The student model is updated based on the global difference and the at least one local difference between the first visual content and the second visual content.

2. The method according to claim 1, wherein determining at least one key area in the sample visual content corresponding to the sample prompt word comprises: At least one key region satisfying a predetermined condition is determined from the sample visual content using the trained detection model, where the predetermined condition indicates an object category or a requirement for the key region. The method according to claim 1 , wherein the prediction region information indicates at least one prediction region corresponding to the at least one key region.

4. The method of claim 1 , wherein updating the student model based on the global difference and the local difference comprises: Determine a first weight corresponding to the global difference and a second weight corresponding to each of the at least one local difference; Determining a loss of a loss function for updating the student model based on the first weight, at least one second weight corresponding to the at least one local difference, the global difference, and the at least one local difference; and The student model is updated by reducing the loss of the loss function. The method according to claim 4 , wherein the at least one key area comprises a plurality of key areas, and the second weight of each of the plurality of key areas indicates at least the importance of the corresponding key area. 6 . The method according to claim 1 , wherein the prediction region information and the sample region information are both feature information in a feature space or pixel information in a pixel space.

7. The method according to claim 1, wherein the teacher model is configured to generate the first visual content based on a positive cue word, a negative cue word and a guidance ratio, the guidance ratio indicating the degree of influence of the positive cue word and the negative cue word on the first visual content, and the student model is configured to generate the second visual content based on the positive cue word.

8. The method according to claim 1, further comprising: In response to a model update target of the student model being met, stopping updating the student model, wherein the model update target is based on at least one of the following: the number of updates of the student model reaches a predetermined number, or the content quality of the visual content output by the updated student model reaches a threshold quality.

9. A device for model updating, comprising: a visual content determination module configured to determine a first visual content and a second visual content based on a sample prompt word using a teacher model and a student model respectively; a key area determination module, configured to determine at least one key area in the sample visual content corresponding to the sample prompt word; a local difference determining module configured to determine at least one local difference between the predicted region information corresponding to the at least one key region in the second visual content and the sample region information corresponding to the at least one key region in the sample visual content; as well as The student model updating module is configured to update the student model based on the global difference and the at least one local difference between the first visual content and the second visual content.

10. An electronic device comprising: at least one processor; as well as At least one memory is coupled to the at least one processor and stores instructions for execution by the at least one processor, the instructions causing the electronic device to perform the method according to any one of claims 1 to 8 when executed by the at least one processor.

11. A computer-readable storage medium having computer-executable instructions stored thereon, wherein the computer-executable instructions can be executed by a processor to implement the method according to any one of claims 1 to 8.

12. A computer program product comprising computer executable instructions, wherein the computer executable instructions, when executed by a processor, implement the method according to any one of claims 1 to 8.

Images