Method executed by electronic device, electronic device, storage medium and program product

Abstract

The embodiment of the invention provides a method executed by electronic equipment, the electronic equipment, a storage medium and a program product, and relates to the technical field of artificial intelligence. The method comprises the following steps: acquiring a first top-layer feature and a first bottom-layer feature obtained by encoding a first image; performing semantic prediction based on the quantitative representation of the first top-layer feature to obtain a target semantic element corresponding to the first bottom-layer feature; sparse coding is carried out based on the target semantic element, and quantitative representation of the first bottom-layer feature is obtained; the quantized representation of the first top layer feature and the quantized representation of the first bottom layer feature are used for decoding input of image generation to obtain a second image. According to the embodiment of the invention, the relationship between the hierarchies can be fully utilized, and the calculation amount of multi-level representation is reduced. Optionally, the method performed by the electronic device may be performed using an artificial intelligence model.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, and in particular to a method executed by an electronic device, an electronic device, a storage medium, and a program product. Background Art

[0002] In image generation technology based on multi-level representation, the mutual complementation of top-level features containing global information and bottom-level features containing detail information can effectively improve the quality of reconstructed images; however, the relationship between the levels is not fully utilized. Summary of the invention

[0003] The embodiments of the present disclosure provide a method executed by an electronic device, an electronic device, a storage medium, and a program product, aiming to fully utilize the relationship between hierarchical features. The technical solution is as follows:

[0004] According to one aspect of an embodiment of the present disclosure, a method performed by an electronic device is provided, the method including:

[0005] Obtaining a first top-level feature and a first bottom-level feature obtained by encoding the first image;

[0006] Performing semantic prediction based on the quantized representation of the first top-level feature to obtain a target semantic element corresponding to the first bottom-level feature;

[0007] Performing sparse coding based on the target semantic element to obtain a quantized representation of the first underlying feature;

[0008] The quantized representation of the first top-level feature and the quantized representation of the first bottom-level feature are used as decoding input for image generation to obtain a second image.

[0009] In a feasible embodiment, the step of obtaining the first top-level feature and the first bottom-level feature obtained by encoding the first image includes:

[0010] Obtaining a first top-level feature and a second bottom-level feature obtained by encoding the first image;

[0011] Performing semantic feature extraction on the first top-level feature through a first module in a neural network to obtain a second top-level feature;

[0012] A first bottom layer feature is obtained based on the second top layer feature and the second bottom layer feature.

[0013] In a feasible embodiment, performing semantic prediction based on the quantized representation of the first top-level feature to obtain a target semantic element corresponding to the first bottom-level feature includes:

[0014] Obtaining a quantized representation of the first top-level feature obtained by quantizing the first top-level feature;

[0015] Obtaining, by a second module in a neural network, a target semantic group index of the first bottom-level feature based on the quantized representation of the first top-level feature;

[0016] Obtaining a target semantic element corresponding to the first underlying feature based on the target semantic group index;

[0017] The codebook of the first underlying feature is divided into a plurality of semantic groups, each semantic group includes a plurality of semantic elements, and each semantic group has a one-to-one correspondence with a semantic group index.

[0018] In a feasible embodiment, obtaining the target semantic group index of the first bottom-level feature based on the quantized representation of the first top-level feature by the second module in the neural network includes:

[0019] Performing a convolution operation on the quantized representation of the first top-level feature through at least two cascaded convolution layers included in the second module to obtain a probability distribution of the semantic group to which the first top-level feature belongs;

[0020] Through the activation function layer included in the second module, the target semantic group index of the first bottom-level feature is determined based on the probability distribution of the semantic group to which the first top-level feature belongs, and the activation function layer is composed of differentiable functions.

[0021] In a feasible embodiment, the at least two cascaded convolutional layers included in the second module include several first convolutional layers and several second convolutional layers arranged in cascade; the first convolutional layer includes a convolutional sublayer, a group normalization sublayer and an activation function sublayer arranged in cascade; and the second convolutional layer includes a convolutional sublayer.

[0022] In a feasible embodiment, the step of performing sparse coding based on the target semantic element to obtain a quantitative representation of the first underlying feature includes:

[0023] The target semantic elements are linearly combined through the third module in the neural network to obtain a quantitative representation of the first underlying feature.

[0024] In a feasible embodiment, the linear combination of the target semantic elements to obtain a quantitative representation of the first underlying feature includes:

[0025] Based on a linear combination of a learnable parameter matrix and the target semantic element, a quantitative representation of the first underlying feature is obtained; the parameter matrix is ​​obtained by constraining the target loss function of the neural network as a whole.

[0026] In a feasible embodiment, the L1 norm is used as a sparsity constraint on the parameter matrix in the objective loss function.

[0027] According to another aspect of an embodiment of the present disclosure, an electronic device is provided. The electronic device includes a memory, a processor, and a computer program stored in the memory. The processor executes the computer program to implement the steps of the method performed by the electronic device.

[0028] According to another aspect of the embodiments of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of the method performed by the electronic device are implemented.

[0029] According to another aspect of an embodiment of the present disclosure, a computer program product is provided, including a computer program, which implements the steps of the method performed by the electronic device when the computer program is executed by a processor.

[0030] The technical solution provided by the embodiments of the present disclosure brings the following beneficial effects:

[0031] The present disclosure provides a method, an electronic device, a storage medium and a program product executed by an electronic device. Specifically, in the image generation process, after obtaining the first top-level feature and the first bottom-level feature obtained by encoding the first image, semantic prediction is performed based on the quantized representation of the quantized first top-level feature to obtain the target semantic element corresponding to the first bottom-level feature, and then sparse coding is performed based on the target semantic element to achieve quantization of the first bottom-level feature to obtain the quantized representation of the first bottom-level feature. The obtained quantized representation of the first top-level feature and the quantized representation of the first bottom-level feature are used as decoding input for image generation to obtain a second image. The implementation of the present disclosure performs semantic prediction and sparse coding based on the quantized representation of the first top-level feature, and can adaptively calculate the corresponding bottom-level feature (including detail information) based on the top-level feature (including global information) to obtain local detail features that are more suitable for global features, thereby fully utilizing the relationship between multiple layers of features in image generation. At the same time, it is unnecessary to model the sequence of bottom-level features in the transformer stage, which can effectively reduce the amount of calculation required for image generation of multi-level representations. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] In order to more clearly illustrate the technical solutions in the embodiments of the present disclosure, the drawings required for describing the embodiments of the present disclosure are briefly introduced below.

[0033] Figure 1 A flowchart of a method performed by an electronic device provided in an embodiment of the present disclosure;

[0034] Figure 2A schematic diagram of the architecture of a VQ-GAN network provided in an embodiment of the present disclosure;

[0035] Figure 3 A schematic diagram of the architecture of a HR-VQVAE network provided in an embodiment of the present disclosure;

[0036] Figure 4 A schematic diagram of the architecture of an AHQ-VAE network provided in an embodiment of the present disclosure;

[0037] Figure 5 A schematic diagram of a spatial semantic consistency module provided in an embodiment of the present disclosure;

[0038] Figure 6 A schematic diagram of a semantic group prediction module provided by an embodiment of the present disclosure;

[0039] Figure 7 A schematic diagram of another semantic group prediction module provided by an embodiment of the present disclosure;

[0040] Figure 8 A schematic diagram of the operation flow related to the semantic group prediction module and the sparse coding module provided in the embodiment of the present disclosure;

[0041] Fig. 9 A schematic diagram for comparing visual effects provided by an embodiment of the present disclosure;

[0042] Fig.10 A schematic diagram of the structure of an electronic device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0043] The following description with reference to the accompanying drawings is provided to facilitate a comprehensive understanding of the various embodiments of the present disclosure as defined by the claims and their equivalents. This description includes various specific details to facilitate understanding but should be considered as exemplary only. Therefore, one of ordinary skill in the art will recognize that various changes and modifications can be made to the various embodiments described herein without departing from the scope and spirit of the present disclosure. In addition, for the sake of clarity and conciseness, descriptions of well-known functions and structures may be omitted.

[0044] The terms and expressions used in the following specification and claims are not limited to their dictionary meanings, but are merely used by the inventor to enable a clear and consistent understanding of the present disclosure. Therefore, it should be apparent to those skilled in the art that the following description of various embodiments of the present disclosure is provided for illustration purposes only and not for the purpose of limiting the present disclosure as defined in the appended claims and their equivalents.

[0045] It should be understood that the singular forms "a", "an", and "the" may also include plural references unless the context clearly indicates otherwise. Thus, for example, reference to a "component surface" includes reference to one or more such surfaces. When we refer to an element as being "connected" or "coupled" to another element, the one element may be directly connected or coupled to the other element, or the one element and the other element may be connected through an intermediate element. In addition, "connected" or "coupled" as used herein may include wireless connection or wireless coupling.

[0046] The term "include" or "may include" refers to the presence of the corresponding disclosed functions, operations or components that can be used in various embodiments of the present disclosure, rather than limiting the presence of one or more additional functions, operations or features. In addition, the term "include" or "have" may be interpreted as indicating certain characteristics, numbers, steps, operations, constituent elements, components or combinations thereof, but should not be interpreted as excluding the possibility of the presence of one or more other characteristics, numbers, steps, operations, constituent elements, components or combinations thereof.

[0047] The term "or" used in various embodiments of the present disclosure includes any of the listed terms and all combinations thereof. For example, "A or B" may include A, may include B, or may include both A and B. When describing multiple (two or more) items, if the relationship between the multiple items is not clearly defined, the multiple items may refer to one, multiple, or all of the multiple items. For example, the description of "parameter A includes A1, A2, A3" may be implemented as parameter A including A1 or A2 or A3, or may be implemented as parameter A including at least two of the three items A1, A2, and A3.

[0048] Unless defined differently, all terms (including technical terms or scientific terms) used in the present disclosure have the same meanings as understood by those skilled in the art described in the present disclosure. Common terms as defined in dictionaries are interpreted as having meanings consistent with the context in the relevant technical field, and should not be interpreted ideally or overly formally unless clearly defined in the present disclosure.

[0049] At least some functions of the device or electronic device provided in the embodiments of the present disclosure may be implemented by an AI (Artificial Intelligence) model, such as at least one module among multiple modules of the device or electronic device may be implemented by an AI model. Functions associated with AI may be performed by a non-volatile memory, a volatile memory, and a processor.

[0050] The processor may include one or more processors. In this case, the one or more processors may be general-purpose processors, such as a central processing unit (CPU), an application processor (AP), etc., or pure graphics processing units, such as a graphics processing unit (GPU), a visual processing unit (VPU), and / or an AI-specific processor, such as a neural processing unit (NPU).

[0051] The one or more processors control the processing of input data according to predefined operating rules or artificial intelligence (AI) models stored in non-volatile memory and volatile memory. The predefined operating rules or artificial intelligence models are provided by training or learning.

[0052] Here, providing by learning means obtaining a predefined operating rule or an AI model with desired characteristics by applying a learning algorithm to a plurality of learning data. The learning can be performed in the device or electronic device itself in which the AI ​​according to the embodiment is executed, and / or can be implemented by a separate server / system.

[0053] The AI ​​model may include multiple neural network layers. Each layer has multiple weight values, and each layer performs neural network calculations by calculating between the input data of the layer (such as the calculation results of the previous layer and / or the input data of the AI ​​model) and the multiple weight values ​​of the current layer. Examples of neural networks include, but are not limited to, convolutional neural networks (CNNs), deep neural networks (DNNs), recurrent neural networks (RNNs), restricted Boltzmann machines (RBMs), deep belief networks (DBNs), bidirectional recurrent deep neural networks (BRDNNs), generative adversarial networks (GANs), and deep Q networks.

[0054] A learning algorithm is a method of using a plurality of learning data to train a predetermined target device (e.g., a robot) to enable, allow, or control the target device to make a determination or prediction. Examples of the learning algorithm include, but are not limited to, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning.

[0055] According to the present disclosure, at least one step performed in an electronic device, such as convolving the quantized representation of the first top-level feature to obtain the semantic group index of the first bottom-level feature, can be implemented using an artificial intelligence model. The processor of the electronic device can perform preprocessing operations on the data to convert it into a form suitable for use as an input to the artificial intelligence model. The artificial intelligence model can be obtained through training. Here, "obtained through training" means training a basic artificial intelligence model with multiple training data through a training algorithm to obtain a predefined operation rule or artificial intelligence model configured to perform the desired feature (or purpose).

[0056] Among the related image generation technologies, the auto-regressive (AR) model, as one of the image generation methods, has achieved remarkable results in recent years. These models usually adopt a two-stage framework. The first stage uses a set of discrete codes to represent the image, and the second stage performs auto-regressive modeling on these codes. Vector quantization (VQ) is an important technology for discretizing continuous latent features, which enables AR models to effectively generate high-resolution images without relying on pixel-to-pixel predictions. Among them, VQ-GAN (vector quantization-Generative Adversarial Network, a quantized adversarial generative network, corresponding to an autoregressive image generation method) involves two-stage processing, including an encoder, VQ, and a decoder in stage 1; the encoder represents the input image as Then replace the vector by finding the most similar vector in the codebook Get z q , this step is also called vector quantization; the decoder Decoder z q Mapped into an image, the discriminator is used to determine whether it is a generated image or a real image. In stage 2, according to the quantization sequence obtained in stage 1 (such as Figure 2 1, 42, 3...) use the network transformer to fit this sequence. According to the formula in the dashed box, given the first i-1 sequences, the transformer will predict the element at the i-th position (autoregression). Transformer predicts all z q After the sequence, the decoder is used to generate the image; by minimizing the cross entropy loss, the transformer is used to model the discrete code.

[0057] In the related art, on the one hand, different VQ methods are used to narrow the gap between continuous latent features and quantized discrete representations. On the other hand, in order to make the reconstructed image have high fidelity, hierarchical discrete representations (multi-level representations), especially the representations of the top layer and the bottom layer, are used to capture global semantic information while retaining local fine details. The mutual complementation of global information and detail information can effectively improve the quality of the reconstructed image. Although this intuitive method shows good performance, the relationship between the multiple levels is not fully utilized.

[0058] In response to the problems or areas that need improvement in related technologies, this disclosure proposes an Adaptively Hierarchical Quantization Variational AutoEncoder (AHQ-VAE) to explore the relationship in hierarchical discrete representations. Specifically, a semantic group prediction (SGP) module is used to map the top code to the corresponding semantic group, so as to quantize the underlying latent features and use sparse coding (SC) technology to calculate the linear combination of elements in the selected semantic group, generate a quantized representation for the bottom layer, and obtain local detail features that are more suitable for global features, make full use of the relationship between hierarchical features, and reduce the computational complexity of multi-level representations.

[0059] The following describes several exemplary embodiments to illustrate the technical solutions of the embodiments of the present disclosure and the technical effects produced by the technical solutions of the present disclosure. It should be noted that the following embodiments can refer to, draw on or combine with each other, and the same terms, similar features and similar implementation steps in different embodiments will not be described repeatedly.

[0060] Figure 1 A flow chart of a method performed by an electronic device provided by an embodiment of the present disclosure is shown, and the method can be performed by a terminal or a server. The terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, a vehicle-mounted device, etc.; the server can be an independent physical server, or a server cluster or a distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms, but is not limited to this. The specific content executed by each device will be explained through specific examples in subsequent embodiments.

[0061] Before describing the solution provided by the present disclosure in detail, the related operations in image generation are described.

[0062] Given an input image x∈R H×W×3 , first use encoder Encoder E to encode the input image, and get Then use VQ to quantize the obtained z, and get Then use the decoder decoder D to get the reconstructed image Here, Q(·) is used to represent the VQ process. The codebook learned in the VQ process is a set of tuples C = (k, e k ) k∈[K] , where K is the size of the codebook, is the kth element in the codebook. The quantization process is to find the element in the codebook that matches z i,j The nearest element k Get z q , as shown in the following formula (1):

[0063]

[0064] In the above process, the objective function used can be expressed as the following formula (2):

[0065]

[0066] In formula (2), the first term sg represents the stop-gradient operation (i.e., no parameter update is performed on the variable), and γ is the hyperparameter of the last loss function. The first loss function is the reconstruction loss, which represents the generated image The second term represents the loss function used to optimize codebook learning, and the third term represents similarity commitmentloss, which aims to encourage the encoder output to be as consistent as possible with the selected codebook element.

[0067] In the image generation method based on multi-level representation, two levels are usually considered, which can be represented as z top , z bot The codebook they use for quantization is denoted as C top and C bot The quantized representation is written as and Therefore, the loss function of the image generation method based on hierarchical representation is expressed as follows:

[0068]

[0069] The upper encoding captures the global information of the input, allowing the model to understand the overall structure and context. On the other hand, the lower encoding compensates for the local fine details, allowing the model to capture complex and precise features. By leveraging this hierarchical structure, the framework can generate high-fidelity images.

[0070] The following describes a method performed by an electronic device according to an embodiment of the present disclosure.

[0071] Specifically, Figure 1 As shown, the method performed by the electronic device provided in the embodiment of the present disclosure includes the following steps S101 to S103:

[0072] Step S101: obtaining a first top-level feature and a first bottom-level feature obtained by encoding a first image.

[0073] Step S102: Perform semantic prediction based on the quantized representation of the first top-level feature to obtain a target semantic element corresponding to the first bottom-level feature.

[0074] Step S103: performing sparse coding based on the target semantic element to obtain a quantitative representation of the first underlying feature.

[0075] The quantized representation of the first top-level feature and the quantized representation of the first bottom-level feature are used as decoding input for image generation to obtain a second image.

[0076] In the embodiment of the present disclosure, the above steps S101 to S103 can be performed by a neural network. Figure 4 As shown, the network framework provided by the present disclosure may include a semantic group prediction module and a sparse coding module.

[0077] The first image can be an input image in image generation, which is suitable for different application scenarios, or any other image. The first top-layer feature can be a feature of the top layer, which contains global semantic information; the first bottom-layer feature can be a feature of the bottom layer, which contains detail information in the input image. The second image can be an output image in image generation, that is, a generated image.

[0078] Alternatively, if Figure 4 As shown, the first top-level feature can be the output obtained by the encoder encoding the input image (n z is the dimension of the feature); the first underlying feature can be the output z obtained by the encoder encoding the input image bot ; Then, the quantified (Quantized representation of the first top layer feature) is used as the input of SGP. The output of this module can be used as the input of SC module to quantize the features of the bottom layer. After obtaining the target semantic element corresponding to the first bottom layer feature, the element can be sparsely encoded by SC to obtain the final quantized representation of the bottom layer.

[0079] Among them, the codebook used to quantize the first top-level features can be expressed as C top , the codebook used to quantize the first bottom-level features can be expressed as C bot .

[0080] In the VQ step, the network framework provided by the present disclosure uses a nearest neighbor search to calculate the top-level encoding, and the output (first top-level feature) containing global semantic information is used to guide the semantic group prediction of the bottom latent features; the top-level encoding is used as input through the SGP module, and the semantic group information is output, which is then used to quantize the bottom latent features. In order to obtain the quantitative representation of the bottom layer, the present disclosure applies the SC operation to calculate the linear combination of the selected semantic elements. This process ensures that the bottom layer encoding adapts to the relevant top-level encoding, resulting in reconstruction and generation with exquisite details and improved fidelity. The AHQ-VAE framework of the present disclosure adopts a hierarchical representation to reduce the reconstruction error without increasing the computational cost of the second stage. In addition, the framework including SGP and SC can effectively utilize the relationship between the top-level encoding and the bottom-level encoding, so that image generation is significantly improved.

[0081] In a feasible embodiment, in order to make full use of the correlation between top-level features and bottom-level features so that the top-level features are consistent with the bottom-level features in spatial semantics, the network provided in the embodiment of the present disclosure also includes a spatially consistent semantics mapping (Spatially Consistent Semantics Embedding, SCSE) module (also known as the first module), which can embed global semantic information into the bottom-level representation to enhance the semantic consistency between the hierarchical representations.

[0082] Among them, Figure 4 and Figure 5 As shown, the input of the module includes the first top-level feature and the second bottom-level feature output by the encoder. Optionally, obtaining the first top-level feature and the first bottom-level feature obtained by encoding the first image in step S101 includes steps A1 to A3:

[0083] Step A1: Obtain a first top-level feature and a second bottom-level feature obtained by encoding a first image.

[0084] Step A2: extracting semantic features from the first top-level features through the first module in the neural network to obtain second top-level features.

[0085] Step A3: obtaining a first bottom-level feature based on the second top-level feature and the second bottom-level feature.

[0086] Optionally, after the introduction of SCSE, the first underlying feature may be obtained by obtaining the second underlying feature z output by the encoder. bot and the first top-level feature z top After that, the output result after inputting the two into SCSE is The first bottom-level feature is , it indicates that the feature is connected to the first top-level feature z after passing through the SCSE module. top Before the VQ step, the SCSE module can effectively inject the semantic information captured by the high-level features into the low-level features. The structure of this module is as follows: Figure 5 As shown in Figure 1, it includes a convolutional layer that is responsible for incorporating semantic information and improving the consistency between latent features. Optionally, a 3*3 convolution conv is used to convolution the first top feature z top Semantic feature extraction can obtain the second top-level feature, and then the second top-level feature and the second bottom-level feature z bot Perform dot product processing (other feature combinations may also be used, which is not limited in this disclosure), and use the (i, j) position and z top Parts with the same semantics are activated to achieve spatial semantic consistency; on this basis, the first underlying feature It can be expressed as shown in the following formula (4):

[0087]

[0088] In the disclosed embodiment, the proposed AHQ-VAE utilizes the SCSE module to promote the semantic consistency property to calculate adaptive local details corresponding to global semantic features, so that the image generated by the AHQ-VAE framework can capture both fine-grained features and global features with high fidelity.

[0089] This paper proposes an adaptive hierarchical quantization framework AHQ-VAE, which uses two layers of discrete representation to perform high-fidelity image reconstruction, such as Figure 4 As shown. AHQ-VAE first uses a convolutional encoder to encode the input image into latent features at the top and bottom layers. The top-level information captures global semantics, while the bottom-level information focuses on local details. In order to strengthen the consistency of semantics, the SCSE module is introduced, which spatially adjusts the semantic information in the top and bottom-level latent features. By injecting the global semantic information captured by the top-level latent features, the bottom-level features can exhibit consistent semantics while retaining local details. In the quantization step, the present disclosure proposes two modules, namely SGP and SC, to calculate the adaptive bottom-level encodings of the corresponding top-level encodings. SGP predicts the semantic groups of each bottom-level encoding, while SC combines the elements of the selected group to enhance the bottom-level discrete representation capabilities. With the support of AHQVAE, the relationships within the hierarchical representations are fully mined to achieve high-fidelity reconstruction.

[0090] The following describes the network structure and processing content involved in the SGP module provided in the embodiments of the present disclosure.

[0091] like Figure 6As shown, the SGP module (also called the second module) includes several convolutional layers and activation function layers arranged in cascade. Among them, at least two cascaded convolutional layers include several first convolutional layers and several second convolutional layers arranged in cascade. Optionally, the first convolutional layer includes a convolutional sublayer (such as a 3*3 convolution conv), a group normalization sublayer (GN, groupnormalization) and an activation function sublayer (such as Leaky ReLU, or other activation functions) arranged in cascade; the second convolutional layer includes a convolutional sublayer (such as a 3*3 convolution conv).

[0092] Optionally, in step S102, semantic prediction is performed based on the quantized representation of the first top-level feature to obtain a target semantic element corresponding to the first bottom-level feature, including steps B1 to B3:

[0093] Step B1: Obtain a quantized representation of the first top-level feature obtained by quantizing the first top-level feature.

[0094] Step B2: Obtain a target semantic group index of the first bottom-level feature based on the quantized representation of the first top-level feature through the second module in the neural network.

[0095] Step B3: Obtaining a target semantic element corresponding to the first underlying feature based on the target semantic group index.

[0096] The codebook of the first underlying feature is divided into a plurality of semantic groups, each semantic group includes a plurality of semantic elements, and each semantic group has a one-to-one correspondence with a semantic group index.

[0097] SGP is a component of the adaptive hierarchical quantization disclosed in the present invention, which can make reasonable assumptions about dividing the underlying coding codebook into multiple semantic groups based on making full use of spatial semantic consistency. The process of the SGP module is as follows Figure 6 As shown in Figure 2, the SGP module can be composed of a convolutional layer, a group normalization layer, and an activation function layer. This structure can effectively predict the semantic group of the underlying latent features. Regarding the semantic group, the codebook C can be used in the framework. bot Divided into M groups (for example, assuming that the codebook includes 9 elements, it can be divided into three semantic groups, each including three elements, according to the order of the elements), denoted as C bot = {g 1 , g 2 , …, g M}. Each group, use g m =(h, e n ) n∈[N] Represented by N elements. The elements between groups represent different semantic information, while the elements within the group capture different concepts with similar semantics. This module converts the quantized top-level representation As input, and output the selected group The target semantic group index corresponding to the underlying latent features of .

[0098] Optionally, the output of the SGP module is as shown in the following formula (5):

[0099]

[0100] It represents the index in the bottom codebook corresponding to the semantic group group predicted by the SGP module. With the assistance of SCSE, the SGP module effectively identifies relevant semantic groups and assigns them to the underlying latent features, and also ensures the utilization of global semantic information from the top layer and local details from the bottom layer.

[0101] In one example, if Figure 6 As shown, the output of SGP may be a semantic group index G (corresponding to step B2); in another example, Figure 7 As shown, the output of SGP can be the target semantic group obtained by searching the codebook group of the first bottom-level feature based on the semantic group index G. The target semantic elements included (corresponding to step B3). In the present disclosure, for each spatial position (i, j) Quantization assigns a set of elements Alternatively, if Figure 8 As shown, based on the semantic group index G, the corresponding target semantic group is searched in the codebook group of the first bottom-level feature. The operation can be performed independently of SGP and SC.

[0102] The process of finding the target semantic group is explained by example: Figure 4 As shown, for index 2, the corresponding second semantic group g can be found in the codebook. 2 As the target semantic group; for index 11, the corresponding eleventh semantic group g can be found in the codebook 11 As the target semantic group; and by analogy, the target semantic group corresponding to the semantic group index can be obtained. It can be understood that each target semantic group index corresponds to a group of target semantic groups, each target semantic group includes a number of semantic elements, and thus the target semantic element can refer to the elements included in all target semantic groups.

[0103] Optionally, in step B2, obtaining the target semantic group index of the first bottom-level feature based on the quantized representation of the first top-level feature by the second module in the neural network includes steps B21 and B22:

[0104] Step B21: Perform a convolution operation on the quantized representation of the first top-level feature through at least two cascaded convolutional layers included in the second module to obtain a probability distribution of the semantic group to which the first top-level feature belongs.

[0105] Step B22: Determine the target semantic group index of the first bottom-level feature based on the probability distribution of the semantic group to which the first top-level feature belongs, through the activation function layer included in the second module, wherein the activation function layer is composed of differentiable functions.

[0106] Among them, the embodiments of the present disclosure can use Figure 6 The corresponding network structure implements the proposed SGP. Optionally, the first top-level feature z obtained by encoding the input image x by the encoder can be top Quantize (the quantization operation of the first top-level feature can be performed by using the modeling operation of the sequence in the transformer stage, which will not be described here) to obtain the quantized As the input of this module, it is then convolved and normalized through several cascaded first convolutional layers, and the obtained output is input into several cascaded second convolutional layers to obtain the probability distribution of the semantic group to which the first top-level feature belongs, and then sampled from the probability distribution through the differentiable function Gumbel Softmax (that is, the maximum value from the obtained several semantic values ​​is taken as the target semantic group index of the first bottom-level feature).

[0107] The following describes the processing content of the SC module (third module) provided in the embodiment of the present disclosure.

[0108] In a feasible embodiment, in step S103, sparse coding is performed based on the semantic group index of the first underlying feature to obtain a quantized representation of the first underlying feature, including step C1:

[0109] Step C1: linearly combine the target semantic elements through the third module in the neural network to obtain a quantitative representation of the first underlying feature.

[0110] Among them, Figure 4 As shown, the target semantic group index is used to query the bottom codebook to obtain the target semantic group This matrix indicates that for each position (i, j), a semantic group with N elements can be assigned to the underlying feature of the position, and the bottom feature is quantized using these N elements to obtain a quantized representation of the first underlying feature.

[0111] Optionally, the quantized representation of the first underlying features can be based on a learnable parameter matrix The target semantic group in the codebook grouping based on the first bottom-level feature obtained by G The result obtained by linearly combining the included elements is shown in the following formula (6):

[0112]

[0113] Wherein, ⊙ is a dot multiplication operation. The parameter matrix α is obtained by constraining the target loss function of the entire neural network; it can be understood that the parameter matrix α is a learnable parameter matrix created by the present disclosure, and its learning process is completed by constraining the target loss function.

[0114] Optionally, the L1 norm is used as a sparsity constraint on the parameter matrix in the objective loss function; wherein the objective loss function is shown in the following formula (7):

[0115]

[0116] In formula (7), l is added 1 The norm is used as a constraint on the sparsity of α. τ is the sparsity weight; γ is the loss weight of the item; sg represents the stop-gradient operator, and no parameter update is performed on the variable.

[0117] Specifically, the first loss function is the reconstruction loss, which represents the difference between the generated image and the input image; the second loss function is the loss function for top-level features used to optimize codebook learning; the third loss function is the similarity loss commitment loss for top-level features, which is used to encourage the encoder's output to be consistent with the selected codebook elements as much as possible; the fourth loss function corresponds to the second loss function, the difference is that the fourth loss function is for the underlying features; the fifth loss function corresponds to the third loss function, the difference is that the fifth loss function is for the underlying features; the sixth loss function is a constraint on the sparsity of α.

[0118] In the present disclosure, based on the E (encoder) and D (decoder) trained in the first stage above, the transformer can be trained to model the prior distribution of discrete codes generated by the image. Specifically, The codebook indices are rearranged into a sequence s∈{0, 1, …K} h×w Since the present disclosure aims to improve the performance of hierarchical quantization in the first stage, the transformer model in the related technology (such as VQGAN) can be applied to fit the distribution of s. The objective function of the second stage can be expressed as the following formula (8):

[0119]

[0120] The disclosed embodiment proposes an adaptive hierarchical quantization framework AHQ-VAE for image generation, which aims to explore the relationship in hierarchical representation to achieve high-quality image generation. First, the SCSE module is proposed, which makes the features of the top and bottom layers semantically consistent. Then, using this feature, the bottom layer quantization based on semantic mapping is designed; in the bottom layer quantization process, the quantized As the input of the SGP module, SGP outputs the index of the semantic group. The index (i, j) represents the selected semantic group, which is used to quantize the (i, j) position of the bottom layer. Then the SC module is introduced, using the coefficient α i,j Perform a linear combination of the elements in the selected group to obtain The present invention utilizes the proposed SCSE, SGP and SC. In the transformer stage, only the top-level sequence needs to be modeled, which not only obtains the high fidelity brought by the multi-level representation, but also does not increase the computational complexity of the second-stage transformer.

[0121] exist Fig. 9 In the figure, some visualization results are shown. The results are experimental results on the FFHQ (Flickr-Faces-Hight-Quality, high-definition face) dataset, combined with Fig. 9 The visual effects shown show that the disclosed embodiment has better fidelity than the existing VQGAN and HQ-VAE methods.

[0122] An embodiment of the present disclosure also provides an electronic device, which includes a processor and, optionally, may also include a transceiver and / or a memory coupled to the processor, and the processor is configured to execute the steps of the method provided in any optional embodiment of the present disclosure.

[0123] Fig.10 A schematic diagram of the structure of an electronic device applicable to the embodiment of the present disclosure is shown in FIG. Fig.10 As shown, Fig.10 The electronic device 4000 shown includes: a processor 4001 and a memory 4003. The processor 4001 and the memory 4003 are connected, such as through a bus 4002. Optionally, the electronic device 4000 may also include a transceiver 4004, which may be used for data interaction between the electronic device and other electronic devices, such as data transmission and / or data reception. It should be noted that in actual applications, the transceiver 4004 is not limited to one, and the structure of the electronic device 4000 does not constitute a limitation on the embodiments of the present disclosure.

[0124] Processor 4001 may be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array) or other programmable logic devices, transistor logic devices, hardware components or any combination thereof. It may implement or execute various exemplary logic blocks, modules and circuits described in conjunction with the disclosure of the present invention. Processor 4001 may also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, etc.

[0125] The bus 4002 may include a path to transmit information between the above components. The bus 4002 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus. The bus 4002 may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Fig.10 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0126] The memory 4003 may be a ROM (Read Only Memory) or other types of static storage devices that can store static information and instructions, a RAM (Random Access Memory) or other types of dynamic storage devices that can store information and instructions, or an EEPROM (Electrically Erasable Programmable Read Only Memory), a CD-ROM (Compact Disc Read Only Memory) or other optical disk storage, optical disk storage (including compressed optical disk, laser disk, optical disk, digital versatile disk, Blu-ray disk, etc.), magnetic disk storage media, other magnetic storage devices, or any other medium that can be used to carry or store computer programs and can be read by a computer, without limitation herein.

[0127] The memory 4003 is used to store the computer program for executing the embodiment of the present disclosure, and the execution is controlled by the processor 4001. The processor 4001 is used to execute the computer program stored in the memory 4003 to implement the steps shown in the above method embodiment.

[0128] An embodiment of the present disclosure provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps and corresponding contents of the aforementioned method embodiment can be implemented.

[0129] The embodiments of the present disclosure also provide a computer program product, including a computer program, which can implement the steps and corresponding contents of the aforementioned method embodiments when executed by a processor.

[0130] The terms "first", "second", "third", "fourth", "1", "2", etc. (if any) in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the numbers used in this way can be interchanged where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than that shown or described in the text.

[0131] It should be understood that, although the flowchart of the embodiment of the present disclosure indicates each operation step by arrows, the implementation order of these steps is not limited to the order indicated by the arrows. Unless clearly stated herein, in some implementation scenarios of the embodiment of the present disclosure, the implementation steps in each flowchart can be executed in other orders according to demand. In addition, some or all of the steps in each flowchart may include multiple sub-steps or multiple stages based on the actual implementation scenario. Some or all of these sub-steps or stages can be executed at the same time, and each sub-step or stage in these sub-steps or stages can also be executed at different times. In scenarios with different execution times, the execution order of these sub-steps or stages can be flexibly configured according to demand, and the embodiment of the present disclosure does not limit this.

[0132] The above text and drawings are provided only as examples to help readers understand the present disclosure. They are not intended and should not be interpreted as limiting the scope of the present disclosure in any way. Although certain embodiments and examples have been provided, based on the contents disclosed herein, it is obvious to those skilled in the art that the embodiments and examples shown can be changed without departing from the scope of the present disclosure, and other similar implementation means based on the technical ideas of the present disclosure are adopted, which also fall within the protection scope of the embodiments of the present disclosure.

Claims

1. A method performed by an electronic device, characterized in that: include: Obtaining a first top-level feature and a first bottom-level feature obtained by encoding the first image; Performing semantic prediction based on the quantized representation of the first top-level feature to obtain a target semantic element corresponding to the first bottom-level feature; Performing sparse coding based on the target semantic element to obtain a quantized representation of the first underlying feature; The quantized representation of the first top-level feature and the quantized representation of the first bottom-level feature are used as decoding input for image generation to obtain a second image.

2. The method according to claim 1, characterized in that The obtaining of the first top-level feature and the first bottom-level feature obtained by encoding the first image includes: Obtaining a first top-level feature and a second bottom-level feature obtained by encoding the first image; Performing semantic feature extraction on the first top-level feature through a first module in a neural network to obtain a second top-level feature; A first bottom layer feature is obtained based on the second top layer feature and the second bottom layer feature.

3. The method according to claim 1, characterized in that The performing semantic prediction based on the quantized representation of the first top-level feature to obtain a target semantic element corresponding to the first bottom-level feature includes: Obtaining a quantized representation of the first top-level feature obtained by quantizing the first top-level feature; Obtaining, by a second module in a neural network, a target semantic group index of the first bottom-level feature based on the quantized representation of the first top-level feature; Obtaining a target semantic element corresponding to the first underlying feature based on the target semantic group index; The codebook of the first underlying feature is divided into a plurality of semantic groups, each semantic group includes a plurality of semantic elements, and each semantic group has a one-to-one correspondence with a semantic group index.

4. The method according to claim 3, characterized in that The step of obtaining the target semantic group index of the first bottom-level feature based on the quantized representation of the first top-level feature by the second module in the neural network includes: Performing a convolution operation on the quantized representation of the first top-level feature through at least two cascaded convolution layers included in the second module to obtain a probability distribution of the semantic group to which the first top-level feature belongs; Through the activation function layer included in the second module, the target semantic group index of the first bottom-level feature is determined based on the probability distribution of the semantic group to which the first top-level feature belongs, and the activation function layer is composed of differentiable functions.

5. The method according to claim 4, characterized in that The at least two cascaded convolutional layers included in the second module include several first convolutional layers and several second convolutional layers arranged in cascade; the first convolutional layer includes a convolutional sublayer, a group normalization sublayer and an activation function sublayer arranged in cascade; the second convolutional layer includes a convolutional sublayer.

6. The method according to claim 1, characterized in that The step of performing sparse coding based on the target semantic element to obtain a quantitative representation of the first underlying feature includes: The target semantic elements are linearly combined through the third module in the neural network to obtain a quantitative representation of the first underlying feature.

7. The method according to claim 6, characterized in that The linear combination of the target semantic elements to obtain a quantitative representation of the first underlying feature includes: Based on a linear combination of a learnable parameter matrix and the target semantic element, a quantitative representation of the first underlying feature is obtained; the parameter matrix is ​​obtained by constraining the target loss function of the neural network as a whole.

8. The method according to claim 7, characterized in that In the objective loss function, the L1 norm is used as a sparsity constraint on the parameter matrix.

9. An electronic device comprising a memory, a processor and a computer program stored in the memory, characterized in that: The processor executes the computer program to implement the steps of the method according to any one of claims 1 to 8.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 8 are implemented.

11. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 8 are implemented.

Images