A model training method, an image generation method, a computing device and a storage medium

Abstract

This disclosure provides a model training method, comprising: extracting features of an image, the features including channel dimension, height dimension, and width dimension; dividing the features into multiple feature blocks along the channel dimension; calculating entropy constraints of the features based on the encoding entropy of each feature block; and training an encoder based on the entropy constraints of the features, the encoder being used to generate compressed codes for the image. This disclosure also provides a method for training a generator using the encoder trained by this model training method, and for generating compressed codes for input image labels based on the generator.

Description

Technical Field

[0001] This disclosure relates to the field of computer vision, and more particularly to a model training method, an image generation method, a computing device, and a storage medium. Background Technology

[0002] In recent years, generative models based on diffusion models and Generative Language Model (GPT) architectures have become mainstream due to their good generation quality and the ability to embed rich control conditions. Both diffusion models and GPT-based generative models rely on a core component: the compression and reconstruction unit. For tasks such as image generation and video generation, the reconstruction quality of the compression and reconstruction unit indicates the upper limit of the quality of the generated samples. Current compression and reconstruction units often suffer from excessive computational cost and insufficient accuracy and efficiency in compression or reconstruction. Therefore, a more accurate and efficient compression and reconstruction method is needed. Summary of the Invention

[0003] Embodiments of this disclosure provide a model training method, an image generation method, a computing device, and a storage medium to at least solve the aforementioned problems.

[0004] To achieve the above objectives, the embodiments of this disclosure adopt the following technical solutions:

[0005] A first aspect of this disclosure provides a model training method, comprising:

[0006] Extract features from the image, including channel dimension, height dimension, and width dimension;

[0007] The feature is divided into multiple feature blocks along the channel dimension;

[0008] The entropy constraint of the feature is calculated based on the encoding entropy of each feature block; and

[0009] An encoder is trained based on the entropy constraints of the features, which is used to generate compressed encodings for images.

[0010] A second aspect of this disclosure provides an image encoding method, including:

[0011] The encoder is trained using the model training method described above;

[0012] The input image is compressed and encoded based on the encoder obtained from training.

[0013] A third aspect of this disclosure provides an image generation method, comprising:

[0014] The encoder is trained using the model training method described above;

[0015] The generator is trained based on the encoder obtained from the training process.

[0016] The generator, trained from scratch, generates compressed codes for the labels of the input image.

[0017] A fourth aspect of this disclosure provides a computing device, including: a processor, a memory, and a computer program stored in the memory and executable on the processor; wherein, when the processor runs the computer program, it performs the method described above.

[0018] A fifth aspect of this disclosure provides a computer-readable storage medium having a computer program stored thereon, the computer program being executed by a processor to implement the method described above.

[0019] The technical solution provided in this disclosure employs a grouped entropy constraint method, dividing image features into multiple feature blocks along the channel dimension, and calculating the overall entropy constraint of the features based on the encoding entropy of each feature block. The encoder is then trained based on this entropy constraint, enabling the training of a very large codebook, significantly reducing computational load and improving computational accuracy. Furthermore, this disclosure embodiment can repeatedly and randomly shuffle feature channels, allowing for arbitrary grouping of features and improving the balance of the generated codebook. Moreover, this disclosure embodiment can train a generator based on the trained encoder to generate the encoding corresponding to the input image label, improving image reconstruction accuracy. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in the embodiments of this disclosure or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this disclosure. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1 A flowchart of a model training method 100 provided in this embodiment of the disclosure;

[0022] Figure 2 A schematic diagram of a feature channel grouping provided in an embodiment of this disclosure;

[0023] Figure 3 A flowchart of a model training method 100 provided in this embodiment of the disclosure;

[0024] Figure 4 This is a schematic diagram of the network structure of a generator provided in an embodiment of the present disclosure;

[0025] Figure 5This is a structural block diagram of a computing device 500 provided in an embodiment of the present disclosure. Detailed Implementation

[0026] The technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this disclosure, and not all embodiments. Based on the embodiments of this disclosure, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this disclosure.

[0027] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this disclosure are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate for the embodiments of this disclosure described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0028] Figure 1 A flowchart of a model training method 100 according to an embodiment of the present disclosure is shown, adapted to be executed on a computing device, wherein the encoder is used to generate compressed codes for images, such as binary codes. Figure 1 As shown, the method includes:

[0029] S110: Extract image features, including channel dimension, height dimension, and width dimension. This feature can also be called the initial feature. Image features can be extracted using feature extraction methods. The feature size is usually smaller than the original image size. If the feature height is H, the width is W, and the number of channels is K, then the feature has H*W pixel bits and H*W*K feature bits. Each pixel bit corresponds to K channel bits. These parameters are all positive integers. If there are B frames of images, extract the features of each frame separately, such as extracting the features of the m-th frame.

[0030] S120: Divide the features into multiple feature blocks along the channel dimension. For example, divide K channels into G groups, and combine the features of each group in the height and width dimensions to obtain G feature blocks.

[0031] like Figure 2As shown, the feature height H is 6, the width W is 8, and the number of channels is 8. The 8 channels are divided into 2 groups, each group including 4 channels, which yields 2 corresponding image blocks, namely the feature blocks corresponding to channels 1-4 and the feature blocks corresponding to channels 5-8.

[0032] S130: Calculate the entropy constraint of the feature based on the coding entropy of each feature block.

[0033] Entropy coding, also known as entropy encoding, is a lossless encoding method that utilizes the statistical information of data for compression. The basic idea of ​​entropy coding is to leverage the statistical characteristics of data to make the average length of the encoded codeword approach the limit of information entropy, thereby reducing redundancy and achieving efficient data compression. Entropy coding uses the concept of entropy from information theory for data compression. Information entropy is the average amount of information in a source, representing the degree of regularity in the information. The higher the information entropy value, the less regular the information, and the more difficult it is to compress. Entropy coding represents the probability of the output code appearing. A variable-length code table is used to encode the source codeword, making the average length of the encoded codeword approach the information entropy, thus achieving efficient compression. The encoding is binary; typically, if the feature value of each feature bit is greater than 0, it is encoded as 1, and vice versa. Thus, for K channels, the equivalent codebook size is 2^k. K This increases the codebook size and codebook density.

[0034] S140: Train the encoder based on the entropy constraints of the features. Entropy constraints represent the conditions that the encoding entropy should satisfy. The purpose of entropy constraints is to maximize the entropy of the probability distribution while satisfying the known information. The encoder can be trained by continuously optimizing the relevant parameters of the encoder, such as the weights and biases in the neural network, based on the entropy constraints.

[0035] In some embodiments, calculating the entropy constraint of the feature based on the encoding entropy of each feature block includes:

[0036] The entropy constraint of a feature is calculated based on the coding entropy of each feature bit within the feature and the average coding entropy of each feature block.

[0037] Furthermore, the entropy constraints of the features include:

[0038]

[0039] Where H, W, and K represent the feature height, width, and number of channels, respectively; B represents the number of images; G represents the number of groups; i represents the i-th channel; j represents the j-th pixel; m represents the m-th frame; g represents the g-th group; H(x) represents the output encoding entropy; ρ is the similarity function; z represents the feature value at the corresponding position; q is the step function; and the bin function is used to convert integers into binary strings. g Represents the eigenvalues ​​of the g-th group. Let represent the feature value of the j-th pixel in the i-th channel. The first term after the equals sign represents the sum of the coding entropy of each feature bit, and the second term represents the average coding entropy of each feature block. The first term of the formula minimizes the information entropy of all codes, making the feature values ​​as far away from 0 as possible, thus making the coding more deterministic. The second term increases the information entropy of the average code, making the equivalent codebook more balanced and avoiding the problem of invalid coding.

[0040] Alternatively, the encoding entropy, step function, and bin function can be expressed as the following expressions:

[0041] H(x)) = —x·log(x)

[0042] q(x) = sgn(x)

[0043] bin:int→bin

[0044] ρ:cosine similarity

[0045] In some other embodiments, the order of the feature channels can be adjusted, i.e., the feature channels can be shuffled or reshuffled to achieve a balance of encoded features among different groups. The formula for calculating the entropy constraint can then be further expressed as:

[0046]

[0047] Where, φ(z) g This includes the feature value of the g-th group after adjusting the feature channels. φ represents the operation of adjusting the feature channels, i.e., the shuffling operation.

[0048] Furthermore, the order of multiple channels can be adjusted randomly, that is, the feature channels are randomized. For example, if the initial feature channel order is 1-8, after one random adjustment, the feature channel order becomes 3, 2, 5, 6, 1, 7, 4, 8. Of course, they can also be adjusted according to a predetermined order, such as each channel moving one or more channels forward within the feature volume each time. For example, assuming the initial nine channel positions are 1-8, if one channel position is moved each time, after the first adjustment, the feature channel order is 8, 1, 2, 3, ..., 7. After the second adjustment, the feature channel order is 7, 8, 1, 2, 3, ..., 6. And so on. As long as there are enough adjustments, a balanced feature channel partitioning can be achieved.

[0049] In one implementation, an entropy constraint can be calculated once based on the initial features, and then the entropy constraint can be calculated again for each of the adjusted features after adjusting the initial feature channels individually. Method 100 may also include, for example: Figure 3 The steps shown are as follows:

[0050] S150: Adjust the order of multiple channels in the feature to obtain a new feature;

[0051] S160: Divide the new feature into multiple new feature blocks along the channel dimension.

[0052] S170: Calculate the entropy constraint of the new feature based on the encoding entropy of each new feature block; and

[0053] S190: Retrain the encoder based on the entropy constraints of the new features.

[0054] That is, in step S140, the encoder was initially trained, and the entropy constraints of the new features were generated using the initially trained encoder. The encoder was then trained again based on the entropy constraints of the new features to obtain the relevant parameters of the encoder after further optimization. In other words, the encoder was trained again based on the relevant parameters of the encoder obtained in the previous training.

[0055] Based on this, such as Figure 3 As shown, method 100 also includes:

[0056] Repeat steps S150-S190 until a predetermined number of adjustments is reached or the entropy constraint meets a predetermined condition. For example, iteration stops after 300 adjustments and training iterations, or when the entropy constraint is less than or equal to a predetermined threshold. That is, each time, the previously trained encoder is used to calculate the new entropy constraint, and the encoder is retrained based on the new entropy constraint.

[0057] In another implementation, the order of the feature channels can be adjusted before each grouping. Then, step S140, which divides the features into multiple feature blocks along the channel dimension, may include:

[0058] Adjusting the order of multiple channels in a feature yields a new feature;

[0059] The adjusted features are divided into multiple feature blocks along the channel dimension.

[0060] Accordingly, step S150 calculates the entropy constraint of the new feature based on the encoding entropy of each feature block, and step S160 trains the encoder based on the entropy constraint of the new feature.

[0061] Therefore, as Figure 3 As shown, after the features are extracted, method 100 can also repeatedly perform the steps of adjusting the feature channel order, dividing the new features into multiple feature blocks along the channel dimension, calculating the entropy constraint of the new features, and training the encoder according to the entropy constraint of the new features, until a predetermined number of adjustments is reached or the entropy constraint meets the predetermined conditions.

[0062] Furthermore, during training, the previously trained encoder can be used each time to calculate the entropy constraints of the adjusted features, and the encoder can be retrained based on the new entropy constraints. This iterative process is repeated multiple times to gradually train the encoder to the desired goal. Alternatively, the entropy constraint of one feature can be calculated each time the order of the feature channels is adjusted, and the entropy constraints calculated after multiple adjustments can be summed to obtain the overall entropy constraint for multiple adjustments. The encoder can then be trained based on this overall entropy constraint.

[0063] Optionally, in each embodiment of this disclosure, the order of feature channels can be adjusted from the initial features of the image each time the feature channels are adjusted, or the order of the current feature channels can be adjusted again based on the features after the last adjustment to obtain the features after the current adjustment. When the number of adjustments is sufficient, the balance of feature grouping can be achieved.

[0064] The above provides a method for achieving balanced group coding by shuffling feature channels. In other embodiments, this disclosure can also achieve balanced group coding by dividing the code into multiple groups, that is, by using multiple grouping methods to obtain multiple grouping results, and calculating an entropy constraint based on each grouping result.

[0065] Each encoding group can be a regular grouping, aiming to approximate the average grouping. For example, the first grouping might consist of two feature channels per group. The second grouping might consist of three feature channels per group. The third grouping might consist of four feature channels per group, and so on.

[0066] Of course, each encoding group can also be randomly grouped. For example, in the first grouping, the first group includes 4 channels, the second group includes 1 channel, and the third group includes 2 channels. In the second grouping, the first group contains 2 channels, the second group contains 3 channels, the third group contains 2 channels, and so on.

[0067] It's important to understand that either shuffling the channel order or shuffling the grouping method can be chosen, or both can be performed. For example, after each shuffling of the channel order, entropy constraints can be calculated separately for each of the various grouping methods. The entropy constraints calculated each time can be directly used for the current encoder training, and the next entropy constraint can be calculated based on the trained encoder. Alternatively, the entropy constraints calculated multiple times can be combined, and the encoder can be trained based on the results of these calculations; this disclosure does not impose any restrictions on this approach.

[0068] After training the encoder using the above method, a compressed code for the input image can be generated based on the trained encoder. Upon receiving this compressed code, the receiving end can decode it using a decoder to obtain the corresponding image. The encoder trained in this embodiment improves the accuracy and efficiency of image compression, reduces the computational load of image compression, and reduces the network bandwidth required for data transmission.

[0069] In some embodiments, this disclosure may also train a generator based on a trained encoder. The generator is used to generate an encoding corresponding to the input image label. The encoding generated by the encoder or generator is decoded by a decoder to obtain the corresponding image.

[0070] Image tags include, but are not limited to, object categories, text descriptions, or image information. Object categories, such as cats, dogs, trees, people, vehicles, and buildings, can be used to generate or extract images with corresponding categories. Text descriptions can include phrases like "generate an image of a group meeting" or "generate an image of an office." Image information may include one or more known images, in which case the task is to "generate new images or videos based on these images." Based on these image tags, images corresponding to those tags can be generated. Therefore, this solution can be widely applied in fields such as text-to-image, image-to-image, and text-to-video generation.

[0071] In one implementation, embodiments of this disclosure may first train the encoder according to the entropy constraints described above, and then train the decoder and generator based on the trained encoder. The decoder decodes the encoded data generated by the encoder into an output image, while the image used to extract features in step S140 is the original image. Therefore, the decoder can be trained based on the differences between the output image and the original image.

[0072] In another implementation, embodiments of this disclosure may simultaneously train the encoder and decoder, and train the generator based on the trained encoder, while the trained decoder is used to decode the code generated by the generator.

[0073] Therefore, step S140, training the encoder based on the entropy constraints of the features, includes:

[0074] The encoder and decoder are trained simultaneously based on the entropy constraints of the features and the reconstruction loss of the decoder.

[0075] The reconstruction loss of the decoder includes the difference between the image decoded by the decoder and the original image. This includes the differences in all pixel values ​​and all channel values ​​between the original and output images, specifically:

[0076]

[0077] Where B represents the number of images in the training set, and each image has C feature channels (such as red, yellow, and green channels, grayscale and texture channels, etc.). Let I be the true value of the k-th channel of the j-th pixel in the m-th frame. mjk This is the predicted value for the k-th channel of the j-th pixel in the m-th frame. Training stops when the predetermined number of iterations or the loss function is less than or equal to a predetermined threshold.

[0078] Then, by summing or weighted summing the entropy constraint of the encoder and the reconstruction loss of the decoder, the total loss function for training the encoder and decoder can be obtained. The encoder and decoder are trained simultaneously based on this total loss function. Both the encoder and decoder are neural network models, and those skilled in the art can set the hyperparameters of these two models as needed. This disclosure does not impose any restrictions on this.

[0079] Once the codec is trained, the generator can be trained next. Figure 4 A schematic diagram of the structure of a generator according to an embodiment of the present disclosure is shown. Figure 4 As shown, the generator includes a spatial transformation layer and multiple inner transformation layers. The combination of the spatial transformation layer and the j-th inner transformation layer can be used to extract the code corresponding to the j-th pixel. The inner transformation layer is suitable for extracting the code corresponding to the K channel bits under that pixel.

[0080] Furthermore, the spatial transformation layer converts the first input data into an intermediate vector and uses the encoding of the first j pixels as the input for the (j+1)th prediction, thereby predicting the (j+1)th output, j∈(1,H*W).

[0081] The inner transformation layer is adapted to generate the encoding of the K channels corresponding to the (j+1)th pixel based on the output of the (j+1)th transformation. For each pixel, the inner transformation layer is adapted to concatenate the intermediate vector output by the spatial transformation layer with the encoding of the first i channels obtained from the previous i transformations to form a concatenation vector, and use this concatenation vector as the input for the (i+1)th transformation to predict the encoding corresponding to the (i+1)th channel, where i∈(1,K).

[0082] Optionally, the spatial transformation layer can be a spatial Transformer network, and the inner transformation layer can be an inner coding Transformer network.

[0083] like Figure 4As shown, the spatial transformation layer takes a category vector SS as input and outputs vector V1. Vector V1 is then input to the inner transformation layer to obtain the K-channel encoded output O1 for the first pixel. Output O1 is then input to the spatial transformation layer to obtain output vector V2. Vector V2 is input to the inner transformation layer to obtain the K-channel encoded output O2 for the second pixel. The outputs O1 and O2 for the first two pixels are then input to the spatial transformation layer to obtain output vector V3. Vector V3 is input to the inner transformation layer to obtain the K-channel encoded output O3 for the third pixel. This process continues until the output for the last pixel (the nth pixel in the diagram) is obtained. It should be understood that each pixel's encoding O1, O2, O3, etc., has a K-bit encoding.

[0084] In each inner conversion layer, V j Generate O j For example, the process:

[0085] First, V j The input is fed into the inner transformation layer to obtain the output O of the first channel bit of the j-th pixel. j1 Then, vector V j The output O of the first channel bit of the j-th pixel bit j1 Input to the inner transformation layer to obtain the output O of the second channel bit of the j-th pixel. j2。 Then vector V i The output of the first two channels of the j-th pixel is input together with the output of the third channel of the j-th pixel and fed into the inner transformation layer to obtain the output O of the third channel of the j-th pixel. j3 And so on, until the code of the Kth bit of the j-th pixel is output. jk Then, these K encoded bits are concatenated together to obtain the output code O corresponding to the j-th pixel. j .

[0086] Since the images used to train the encoder all have image labels, the generator can generate the corresponding code based on the image label and compare it with the code output by the encoder itself to train the generator.

[0087] Therefore, method 100 may also include the steps of:

[0088] Determine the first encoding of the image features when the encoder is trained, as the ground value;

[0089] A generator is used to generate a second code corresponding to the image label, which serves as the predicted value.

[0090] The generator is trained based on the true value and the predicted value.

[0091] Of course, in this embodiment of the present disclosure, the generator can also be trained using a new image with image labels instead of the image data used to train the codec. In this case, the new image can be input into the encoder to generate the compressed code corresponding to the new image as the ground truth, and then the generator can generate the output code as the predicted value based on the image label of the new image, thereby training the generator.

[0092] Optionally, during the training of the generator, method 100 may further include a loss function to supervise the binary encoding of the output:

[0093]

[0094] in, This represents the e-th correctly encoded bit (if there are multiple frames, each frame has H*W*K feature bits), β e Let represent the e-th bit of the predicted encoding. In this embodiment, a loss function is calculated based on the predicted value and the true value of the output encoding, and the generator is trained using this loss function. Training stops when a predetermined number of iterations or when the loss function is less than or equal to a predetermined threshold.

[0095] Once the generator is trained, it can be used to predict the encoding of the input image label. Therefore, method 100 may further include the step of:

[0096] The preset image labels are input into the trained generator to obtain a third code corresponding to the image labels; and

[0097] The third code is decoded using a decoder to obtain an image corresponding to the image label. For example, the third code can be decoded using a decoder trained using the method described above.

[0098] According to another embodiment of this disclosure, an image encoding method is also provided, comprising:

[0099] The encoder trained using the model training method described above is used to generate compressed codes for the image.

[0100] According to another embodiment of this disclosure, an image generation method is also provided, comprising:

[0101] The encoder, trained using the model training method described above, is used to train the generator; and

[0102] The generator, trained from scratch, generates compressed codes for the input image labels.

[0103] Optionally, the image generation method further includes: decoding the generated compressed code using a decoder to obtain an image corresponding to the image label.

[0104] The model method and image generation method provided in this disclosure can be widely applied to the fields of text-to-image, text-to-video, image-to-image, and image-to-video. By introducing group entropy constraints and feature channel randomization, the feature dimension is no longer limited by the number of features, improving the balance of the large codebook and the accuracy of the encoder, thereby improving the accuracy of the generator.

[0105] Figure 5 An illustration shows a machine in an example form of a computing device 500, which may be a device for performing model training methods, image encoding methods, and image generation methods. Within this computing device, an instruction set, when executed, and / or processing logic, when initiated, may cause the machine to perform any or more of the methods described and / or claimed herein. In alternative embodiments, the machine operates as a standalone device or may be connected (e.g., networked) to other machines. In a networked deployment, the machine may operate as a server or client machine in a server-client network environment, or as a peer-to-peer (or distributed) network environment. The machine may be a personal computer (PC), laptop computer, tablet computing system, personal digital assistant (PDA), cellular phone, smartphone, web application, set-top box (STB), network router, switch, or bridge, or any machine capable of executing an instruction set (sequentially or otherwise) specifying actions to be taken by the machine or initiating processing logic. Furthermore, although only a single machine is illustrated, the term "machine" may also be understood to include any collection of machines that individually or jointly execute one or more sets of instructions (or multiple sets of instructions) for performing any one or more of the methods described and / or required herein.

[0106] Example computing device 500 may include a data processor 502 (e.g., a system-on-a-chip (SoC), a general-purpose processing core, a graphics core, and optional other processing logic) and a memory 504 (e.g., RAM) capable of communicating with each other via a bus 506 or other data transmission system. Computing device 400 may also include various input / output (I / O) devices and / or interfaces 510, such as a touchscreen display, an audio jack, a voice interface, and an optional network interface 512. In an example embodiment, network interface 512 may include one or more radio transceivers configured to communicate with any one or more standard wireless and / or cellular protocols or access technologies (e.g., second-generation (2G), 2.5G, third-generation (3G), fourth-generation (4G), and next-generation radio access for cellular systems, Global System for Mobile Communications (GSM), General Packet Radio Service (GPRS), Enhanced Data GSM Environment (EDGE), Wideband Code Division Multiple Access (WCDMA), LTE, CDMA2000, WLAN, Wireless Router (WR) mesh, etc.). Network interface 512 can also be configured to be used with a variety of other wired and / or wireless communication protocols, including TCP / IP, UDP, SIP, SMS, RTP, WAP, CDMA, TDMA, UMTS, UWB, WiFi, WiMax, Bluetooth, IEEE 802.11x, etc. Essentially, network interface 512 can virtually include or support any wired and / or wireless communication and data processing mechanism through which information / data can be propagated between computing device 500 and another computing or communication system via network 514.

[0107] Memory 504 may represent a machine-readable medium (or computer-readable storage medium) storing one or more instruction sets, software, firmware, or other processing logic (e.g., logic 508) that implements any one or more of the methods or functions described and / or claimed herein. During execution by computing device 500, logic 508 or a portion thereof may also be wholly or at least partially located within processor 502. Thus, memory 404 and processor 402 may also constitute a machine-readable medium (or computer-readable storage medium). Logic 508 or a portion thereof may also be configured as processing logic or logic, at least a portion of which is partially implemented in hardware. Logic 508 or a portion thereof may also be transmitted or received via network 514 via network interface 512. While the machine-readable medium (or computer-readable storage medium) of the example embodiments may be a single type of medium, the term "machine-readable medium" (or computer-readable storage medium) should be understood to include a single non-transitory medium or multiple non-transitory media (e.g., centralized or distributed databases and / or associated caches and computing systems) storing one or more instruction sets. The term "machine-readable medium" (or computer-readable storage medium) can also be understood to include any non-transitory medium capable of storing, encoding, or carrying a set of instructions for machine execution and causing the machine to perform any one or more methods of various embodiments, or capable of storing, encoding, or carrying data structures utilized or associated with such set of instructions. The term "machine-readable medium" (or computer-readable storage medium) can therefore be understood to include, but is not limited to, solid-state memory, optical media, and magnetic media.

[0108] A7. The method as described in A2, wherein the entropy constraint of the feature includes:

[0109]

[0110] Where, φ(z) g This includes features representing the g-th group after adjusting the feature channels.

[0111] A8. The method as described in A1, wherein the feature includes multiple channels, and the feature is divided into multiple feature blocks along the channel dimension; the multiple channels are divided into G groups, and the features of each group in the height and width dimensions are combined to obtain G feature blocks.

[0112] A9. The method as described in any one of A1-A8, wherein the encoder is connected to the decoder, and the decoder is adapted to decode the encoding generated by the encoder to obtain the corresponding image; the step of training the encoder according to the entropy constraint of the feature includes: training the encoder and the decoder simultaneously according to the entropy constraint of the feature and the reconstruction loss of the decoder.

[0113] A10, the method of any one of A1-A8, wherein the image has an image label, the method further comprising: determining a first encoding of the feature when the encoder is trained, as a ground truth; generating a second encoding corresponding to the image label using a generator, as a predicted value; and training the generator based on the ground truth and the predicted value.

[0114] A11. The method as described in A10, wherein the generator is adapted to generate an image corresponding to the image label based on the image label, the image label including an object category and / or a text description.

[0115] A12. The method described in A10 further includes: inputting a preset image label into a trained generator to obtain a third encoding corresponding to the image label; and decoding the third encoding using a decoder to obtain an image corresponding to the image label.

[0116] A13. The method as described in A1, wherein the encoding is binary encoding.

[0117] The disclosed and other embodiments, modules, and functional operations described in this document can be implemented in digital electronic circuit systems, or in computer software, firmware, or hardware (including the structures disclosed in this document and their structural equivalents), or combinations thereof. The disclosed and other embodiments can be implemented as one or more computer program products, that is, one or more modules of computer program instructions encoded on a computer-readable medium for execution by a data processing apparatus or for controlling the operation of such data processing apparatus. The computer-readable medium can be a machine-readable storage device, a machine-readable storage substrate, a memory device, a composition of material influencing machine-readable propagation signals, or combinations thereof. The term "data processing apparatus" covers all means, devices, and machines for processing data, including, for example, a programmable processor, a computer, or multiple processors or computers. In addition to hardware, the apparatus may also include code that creates an execution environment for the computer program under investigation, such as code constituting processor firmware, a protocol stack, a database management system, an operating system, or combinations thereof. Propagation signals are artificially generated signals, such as machine-generated electrical, optical, or electromagnetic signals, generated to encode information to be transmitted to a suitable receiver device.

[0118] Computer programs (also known as programs, software, software applications, scripts, or code) can be written in any programming language (including compiled or interpreted languages), and can be deployed in any form, including as a standalone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A computer program does not necessarily correspond to a file in a file system. A program can be stored as part of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), or as a single file dedicated to the program under development, or as multiple collaborative files (e.g., files storing one or more modules, subroutines, or portions of code). A computer program can be deployed to execute on a single computer or on multiple computers located at a single site or distributed across multiple sites and interconnected via a communication network.

[0119] The processes and logic flows described in this document can be executed by one or more programmable processors that execute one or more computer programs to perform functions by manipulating input data and generating outputs. The processes and logic flows can also be executed by special-purpose logic circuit systems (e.g., FPGAs (Field-Programmable Gate Arrays) or ASICs (Application-Specific Integrated Circuits)), and the devices can also be implemented as special-purpose logic circuits (e.g., FPGAs or ASICs).

[0120] Processors suitable for executing computer programs include, for example, both general-purpose microprocessors and special-purpose microprocessors, as well as any one or more processors in any kind of digital computer. Typically, a processor receives instructions and data from read-only memory or random access memory, or both. Essential components of a computer are a processor for executing instructions and one or more memory devices for storing instructions and data. Typically, a computer also includes one or more mass storage devices (e.g., magnetic disks, magneto-optical disks, or optical disks) for storing data, or the computer is operatively coupled to receive data from or transfer data to or from such mass storage devices, or both. However, a computer does not necessarily need to have such devices. Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, including, for example, semiconductor memory devices such as EPROM, EEPROM, and flash memory devices; magnetic disks, such as internal hard disks or removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks. The processor and memory may be supplemented by or incorporated into a special-purpose logic circuit system.

[0121] While this disclosure contains numerous details, these details should not be construed as limiting the scope of any of this disclosure or any content that may be claimed, but rather as a description of features that may be implemented for specific embodiments of a particular disclosure. Certain features described in the context of individual embodiments in this disclosure may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments. Furthermore, although features may be described above as functioning in certain combinations and initially or even equally claimed, in some cases one or more features from the claimed combination may be removed from the combination, and the claimed combination may be for sub-combinations or variations thereof.

[0122] Similarly, although operations are depicted in a specific order in the accompanying drawings, this should not be construed as requiring such operations to be performed in the specific order shown or in a sequential order, or as requiring all illustrated operations to achieve the desired result. Furthermore, the separation of various system components in the embodiments described in this disclosure should not be construed as requiring such separation in all embodiments. Only some implementations and examples have been described, and other implementations, enhancements, and variations may be made based on what is described and illustrated in this disclosure.

[0123] The embodiments described herein are intended to provide a general understanding of the structure of various embodiments, and are not intended to be used as a complete description of all elements and features of components and systems that may utilize the structures described herein. Many other embodiments will be apparent to those skilled in the art upon review of the descriptions provided herein. Other embodiments can be utilized and derived, allowing for structural and logical substitutions and changes without departing from the scope of this disclosure. The figures herein are representative only and may not be drawn to scale. Some scales may be enlarged, while others may be minimized. Therefore, the specification and figures are to be considered illustrative rather than restrictive.

[0124] Some embodiments implement functionality in two or more specific interconnected hardware modules or devices, wherein control and data signals are communicated between and through the modules, or as part of an application-specific integrated circuit. Therefore, the example system is applicable to software, firmware, and hardware implementations.

[0125] While exemplary embodiments or examples of this disclosure have been described with reference to the accompanying drawings, it should be understood that the exemplary discussions above are not intended to be exhaustive or to limit this disclosure to the specific forms disclosed. Many modifications and variations are possible in light of the above teachings. Therefore, the subject matter disclosed should not be limited to any single embodiment or example described herein, but should be interpreted in accordance with the breadth and scope of the appended claims.

Claims

1. A method for training a model, comprising: extracting features of an image, the features comprising a channel dimension, a height dimension, and a width dimension; dividing the features into a plurality of feature blocks along the channel dimension; calculating an entropy constraint of the features according to the encoding entropy of each feature block; and training an encoder for generating compressed encoding of an image according to the entropy constraint of the features. 2.The method of claim 1, further comprising: adjusting the order of a plurality of channels in the features to obtain new features; dividing the new features into a plurality of new feature blocks along the channel dimension. calculating an entropy constraint of the new features according to the encoding entropy of each new feature block; and re-training the encoder according to the entropy constraint of the new features. 3.The method of claim 2, further comprising: repeating the steps of adjusting the order of a plurality of channels in the features, dividing the new features into a plurality of new feature blocks, calculating an entropy constraint of the new features, and training the encoder until a predetermined number of adjustments is reached or the entropy constraint meets a predetermined condition.

4. The method of claim 2, wherein, the adjusting the order of a plurality of channels in the features comprises: randomly adjusting the order of a plurality of channels in the features.

5. The method of claim 1, wherein, the calculating an entropy constraint of the features according to the encoding entropy of each feature block comprises: calculating the entropy constraint of the features according to the encoding entropy of each feature bit within the features and the average encoding entropy of each feature block.

6. The method of claim 1, wherein, the entropy constraint of the features comprises: wherein H, W, K are the height, width, and number of channels of the features, B represents the number of images, G is the number of groups, i represents the i th channel, j represents the j th pixel bit, m represents the m th image, g represents the g th group, H(x) represents the output encoding entropy, ρ is a similarity function, z represents the feature value at the corresponding position, q is a step function, and the bin function is used to convert an integer to a binary string. 7.A method for generating an image, comprising: training a generator using the trained encoder obtained by the method for training a model according to any one of claims 1-6; generating compressed encoding of an input image label according to the trained generator. 8.The method of claim 7, further comprising: decoding the generated compressed encoding by a decoder to obtain an image corresponding to the image label. 9.A computing device, comprising: a processor, a memory, and a computer program stored on the memory and executable on the processor; wherein the processor, when executing the computer program, performs the method according to any one of claims 1-8. 10.A computer-readable storage medium having a computer program stored thereon, the computer program being executable by a processor to implement the method according to any one of claims 1-8.

Images