Vector quantization method and system based on grouping optimization
By dividing the codebook into subcodebook groups and performing group optimization training, the problems of low codebook utilization and codebook crash are solved, and flexible adjustment and efficient reconstruction of the codebook are realized, improving the adaptability and practicality of the model.
Patent Information
- Application Number
- CN202510381515.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-03-28
- Publication Date
- 2025-08-08
AI Technical Summary
The codebook utilization rate in traditional vector quantization models is low, which causes codebook crashes, affects model performance, and it is difficult to flexibly adjust the codebook size after training.
The codebook is divided into multiple independent subcodebook groups, and the codebook self-scaling is achieved through shared parameters within the group and independent parameters between groups, and training is carried out by combining reconstruction losses and promised losses. After training, new code vectors are generated through resampling to achieve self-scaling of the codebook.
Improve codebook utilization and reconstruction performance, avoid codebook crashes, realize flexible adjustment of the model and improve adaptability, and enhance the stability and interpretability of the model.
Smart Images
Figure CN120449936A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of machine learning and deep learning, and in particular to a vector quantization method and system based on grouping optimization. Background Art
[0002] In the fields of machine learning and deep learning, vector quantization models, such as vector quantized variational autoencoders (VQ-VAEs), have demonstrated great potential in areas such as image generation and speech synthesis by performing self-supervised learning through reconstruction tasks. They represent continuous vectors using the closest vector in a codebook, thereby discretizing data.
[0003] However, traditional vector quantization training suffers from serious drawbacks, notably low codebook utilization. During training, only a portion of the code vectors are used and updated, leading to a phenomenon known as "codebook collapse." Codebook collapse occurs when a large number of code vectors become ineffective, leaving only a few performing the primary encoding task. This significantly limits the model's encoding capabilities, impacting its overall performance.
[0004] To address these issues, researchers have proposed a variety of improvement methods, such as reducing the dimensionality of the latent space, initializing the codebook with a pre-trained model, and jointly optimizing the entire codebook (Joint VQ). While Joint VQ has achieved some success in improving codebook utilization, achieving near 100% codebook utilization, it also limits the codebook's learning capabilities. This results in the model's reconstruction performance being less than expected at the same utilization, and failing to meet the high-quality reconstruction requirements of practical applications. Traditional models struggle to flexibly adjust the codebook size after training. Changing the codebook size often requires a complex retraining process, consuming significant time and computing resources. Therefore, there is still significant room for improvement in both the training and application of vector quantization models. Summary of the Invention
[0005] The present invention provides a vector quantization method based on group optimization to balance codebook utilization and reconstruction performance, while achieving flexible adjustment of the codebook size after training, thereby improving the adaptability and practicality of the model.
[0006] A first embodiment of the present invention provides a vector quantization method based on group optimization, comprising the following steps:
[0007] The codebook is divided into multiple independent sub-codebook groups, and each sub-codebook group is jointly parameterized so that the code vectors within the group are generated by shared projection parameters, and the parameters of different sub-codebook groups are independent of each other. Each sub-codebook group contains multiple code vectors, and the total codebook size remains unchanged.
[0008] During training, the parameters of each sub-codebook group are optimized independently by minimizing the reconstruction loss and the intra-group commitment loss;
[0009] When the input feature vector is quantized, the input feature vector is automatically assigned to the sub-codebook group with the closest distance based on the distance between the input feature vector and the code vector in each sub-codebook group. When calculating the loss and performing backpropagation, only the parameters of the sub-codebook group to which the input feature vector belongs are updated;
[0010] After training, the codebook is resampled by randomly sampling the core part of each sub-codebook group and generating new code vectors using the trained projection matrix to achieve self-expansion of the codebook.
[0011] Optionally, in one embodiment of the present invention, the codebook is divided to obtain multiple independent sub-codebook groups, including:
[0012] The total codebook C∈R n×d Divided into k sub-codebook groups, denoted as {C0, C1, ..., C k-1}, d is the dimension of the code vector, n is the number of code vectors in the total code book, where each sub-code book group Satisfy the total number of code vectors For each subcodebook group C i , generated by the following formula:
[0013]
[0014] in, is the fixed initialized codebook core matrix, n i is the number of code vectors contained in the i-th sub-codebook, r i is the rank of the i-th subcodebook, is the learnable projection matrix, b i ∈R d is the learnable bias vector.
[0015] Optionally, in one embodiment of the present invention, jointly parameterizing each sub-codebook group includes:
[0016] For each sub-codebook group, a codebook core matrix and a projection matrix are defined. The codebook core matrix is linearly transformed through the projection matrix to generate a sub-codebook group.
[0017] Optionally, in one embodiment of the present invention, the codebook core matrix is initialized by random sampling from a standard normal distribution and remains fixed during the training process, and only the projection matrix and the bias vector are optimized.
[0018] Optionally, in one embodiment of the present invention, the calculation method of the intra-group commitment loss is:
[0019] For each sub-codebook group, the corresponding commitment loss is calculated. The total commitment loss is the sum of the losses of each sub-codebook group, and the gradient is only back-propagated to the parameters of the corresponding sub-codebook group.
[0020] Optionally, in one embodiment of the present invention, resampling the codebook after training is completed further includes:
[0021] A new codebook core vector is sampled from a predefined distribution, mapped to an extended code vector through a trained projection matrix, and the extended code vector is added to the atomic codebook group to generate an adjusted codebook without retraining the model.
[0022] A second aspect of the present invention provides a vector quantization system based on group optimization, using the vector quantization method based on group optimization described in the above embodiment. The system includes:
[0023] An encoder, which maps input data into a continuous input feature vector;
[0024] A processor configured to train a group codebook using a vector quantization method based on group optimization;
[0025] The decoder reconstructs the input data based on the quantized input feature vector.
[0026] The vector quantization method and system based on group optimization according to the embodiments of the present invention have the following beneficial effects:
[0027] 1. Through a group optimization strategy, the codebook is divided into multiple sub-codebooks for independent optimization. Feature vectors are automatically routed to the appropriate sub-codebook, and gradients are propagated only within the sub-codebook to which they belong. This ensures full codebook utilization while allowing each sub-codebook to focus on learning specific patterns. This effectively balances codebook utilization and reconstruction performance, resulting in superior reconstruction quality in image reconstruction experiments.
[0028] 2. If the codebook size is to be changed after existing model training, retraining is often required, which consumes a lot of resources and time. The post-training codebook resampling and self-expansion method of the present invention can resample the codebook core without retraining the model, generate new code vectors using the trained projection matrix, and flexibly adjust the codebook size. The codebook can be easily expanded or reduced according to the requirements of different application scenarios, thereby improving the adaptability and practicality of the model. Group optimization allows each sub-codebook to learn different patterns, and through statistical analysis and visualization of the code vectors within the group, the model learning process and feature distribution can be better understood, thereby enhancing the interpretability of the model. At the same time, it avoids the collapse problem that is prone to occur in a single codebook in traditional models, improves the stability of the model, and enables it to maintain good performance in different data sets and tasks.
[0029] Additional aspects and advantages of the present invention will be set forth in part in the description which follows and, in part, will be obvious from the description which follows, or may be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS
[0030] The above and / or additional aspects and advantages of the present invention will become apparent and readily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:
[0031] Figure 1 Schematic diagram of a framework of a vector quantization method based on group optimization according to an embodiment of the present invention. DETAILED DESCRIPTION
[0032] The following describes embodiments of the present invention in detail, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present invention, and are not to be construed as limiting the present invention.
[0033] Figure 1 Schematic diagram of a framework of a vector quantization method based on group optimization according to an embodiment of the present invention.
[0034] like Figure 1 As shown, the vector quantization method based on group optimization includes the following steps:
[0035] In step S101, the codebook is divided into multiple independent sub-codebook groups. Each sub-codebook group is jointly parameterized so that the code vectors within the group are generated using shared projection parameters, and the parameters of different sub-codebook groups are independent of each other. Each sub-codebook group contains multiple code vectors, and the total codebook size remains unchanged.
[0036] The codebook is divided into multiple independent sub-codebooks. Each sub-codebook is an independent group with its own independent parameters. The parameters of each sub-codebook are randomly initialized using the standard normal distribution. Specifically, the total codebook C∈R n×d Divided into k sub-codebook groups, denoted as {C0, C1, ..., C k-1}, d is the dimension of the code vector, n is the number of code vectors in the total code book, where each sub-code book group Satisfy the total number of code vectors For each subcodebook group C i , generated by the following formula:
[0037]
[0038] in, is a fixed initialized codebook core matrix (e.g., sampled from a standard normal distribution), n iis the number of code vectors contained in the i-th sub-codebook, r i is the rank of the i-th subcodebook. is the learnable projection matrix. b i ∈R d is the learnable bias vector. This step ensures that the code vectors within a group are jointly optimized through shared parameters, while the parameters between groups are independent.
[0039] Optionally, in one embodiment of the present invention, jointly parameterizing each sub-codebook group includes:
[0040] For each sub-codebook group, a codebook core matrix and a projection matrix are defined. The codebook core matrix is linearly transformed through the projection matrix to generate a sub-codebook group.
[0041] Optionally, in one embodiment of the present invention, the codebook core matrix is initialized by random sampling from a standard normal distribution and remains fixed during the training process, and only the projection matrix and the bias vector are optimized.
[0042] In step S102 , during the training process, the parameters of each sub-codebook group are optimized independently by minimizing the reconstruction loss and the intra-group commitment loss.
[0043] Optionally, in one embodiment of the present invention, the calculation method of the intra-group commitment loss is:
[0044] For each sub-codebook group, the corresponding commitment loss is calculated. The total commitment loss is the sum of the losses of each sub-codebook group, and the gradient is only back-propagated to the parameters of the corresponding sub-codebook group.
[0045] Optionally, in one embodiment of the present invention, the total committed loss is defined as:
[0046]
[0047] Among them, Q i and Z i are the quantized features and original features of group i, respectively, and sg[·] represents the gradient truncation. The commitment loss of each group The gradient of only updates the W of the corresponding sub-codebook group i and b i , avoiding inter-group interference. During the training process, the codebook utilization rate of each group gradually increased with each epoch, eventually reaching more than 99%.
[0048] In step S103, when the input feature vector is quantized, the input feature vector is automatically assigned to the sub-codebook group with the closest distance based on the distance between the input feature vector and the code vector in each sub-codebook group. When calculating the loss and performing backpropagation, only the parameters of the sub-codebook group to which the input feature vector belongs are updated.
[0049] In one embodiment of the present invention, a grouping optimization strategy is employed during the model training phase. When an input feature vector is quantized, it is automatically assigned to the sub-codebook that best matches it based on a specific distance metric. During loss calculation and backpropagation, only the parameters of the sub-codebook to which the feature vector belongs are updated, avoiding interference between different sub-codebooks and improving codebook learning efficiency and utilization.
[0050] In step S104, after the training is completed, the codebook is resampled by randomly sampling the core part of each sub-codebook group and generating a new code vector using the trained projection matrix to achieve self-expansion of the codebook.
[0051] Specifically, after model training, the codebook is resampled. This involves randomly sampling the core portion of each sub-codebook and generating new code vectors using the trained projection matrix, enabling codebook self-expansion. This approach allows for flexible adjustment of the codebook size to suit different application scenarios and requirements without retraining the entire model.
[0052] Optionally, in one embodiment of the present invention, resampling the codebook after training is completed further includes:
[0053] A new codebook core vector is sampled from a predefined distribution, mapped to an extended code vector through a trained projection matrix, and the extended code vector is added to the atomic codebook group to generate an adjusted codebook without retraining the model.
[0054] The predefined distribution is the standard normal distribution, and the projection matrix has been optimized during the training phase.
[0055] In one embodiment of the present invention, from a fixed distribution Sampling new kernel vectors Using the trained projection matrix W i Generate a new code vector:
[0056]
[0057] q new Join Atomic Codebook Group C i , achieving flexible adjustment of the codebook size. After sampling and adding new vectors, the sub-codebook size is expanded, enhancing the representation capability of the codebook.
[0058] In one embodiment of the present invention, the optimal reconstruction performance is achieved when the number of sub-codebook groups is 32 to 64.
[0059] The group-optimization-based vector quantization method proposed in this embodiment of the present invention effectively balances codebook utilization and reconstruction performance, ensuring good reconstruction quality while improving codebook utilization. The post-training codebook sampling method enables the model to flexibly adjust the codebook size without requiring complex retraining, improving the model's adaptability and practicality and providing strong support for the application of vector quantization models in a wider range of fields.
[0060] The embodiment of the present invention also proposes a vector quantization system based on grouping optimization.
[0061] The vector quantization system based on group optimization includes:
[0062] The encoder is used to map the input data into a continuous input feature vector;
[0063] A processor configured to train a group codebook using a vector quantization method based on group optimization;
[0064] The decoder reconstructs the input data based on the quantized input feature vector.
[0065] It should be noted that the above explanation of the embodiment of the vector quantization method based on group optimization is also applicable to the vector quantization system based on group optimization in this embodiment, and will not be repeated here.
[0066] The vector quantization system based on group optimization in an embodiment of the present invention includes codebook partitioning, initialization, quantization, training, and codebook adjustment. The codebook is divided into multiple sub-codebooks, and joint optimization within each group and independent optimization between groups are performed to adapt the codebook to complex feature distributions and balance codebook utilization and reconstruction performance. During training, a group optimization strategy is used to update parameters. After training, new code vectors are generated by randomly sampling the codebook core to achieve self-expansion of the codebook and improve reconstruction quality. This system effectively avoids codebook collapse, exhibits excellent performance in image reconstruction experiments, and can flexibly adjust the codebook size, improving model adaptability and practicality.
[0067] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic expressions of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or N embodiments or examples in a suitable manner. In addition, those skilled in the art can combine and combine different embodiments or examples described in this specification and the features of different embodiments or examples without contradiction.
[0068] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be understood to indicate or imply relative importance or implicitly specify the number of technical features indicated. Thus, a feature specified as "first" or "second" may explicitly or implicitly include at least one such feature. In the description of the present invention, "N" means at least two, such as two, three, etc., unless otherwise specifically defined.
[0069] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, segment or portion of code comprising one or N executable instructions for implementing a custom logical function or step of a process, and the scope of the preferred embodiments of the present invention includes alternative implementations in which functions may be performed out of the order shown or discussed, including performing functions in a substantially simultaneous manner or in the reverse order depending on the functions involved, which should be understood by those skilled in the art to which the embodiments of the present invention pertain.
Claims
1. A vector quantization method based on group optimization, characterized in that: The following steps are involved: The codebook is divided into multiple independent sub-codebook groups, and each sub-codebook group is jointly parameterized so that the code vectors within the group are generated by shared projection parameters, and the parameters of different sub-codebook groups are independent of each other. Each sub-codebook group contains multiple code vectors, and the total codebook size remains unchanged. During training, the parameters of each sub-codebook group are optimized independently by minimizing the reconstruction loss and the intra-group commitment loss; When the input feature vector is quantized, the input feature vector is automatically assigned to the sub-codebook group with the closest distance based on the distance between the input feature vector and the code vector in each sub-codebook group. When calculating the loss and performing backpropagation, only the parameters of the sub-codebook group to which the input feature vector belongs are updated; After training, the codebook is resampled by randomly sampling the core part of each sub-codebook group and generating new code vectors using the trained projection matrix to achieve self-expansion of the codebook.
2. The method according to claim 1, characterized in that The codebook is divided into multiple independent sub-codebook groups, including: The total codebook C∈R n×d Divided into k sub-codebook groups, denoted as {C0, C1, ..., C k-1 }, d is the dimension of the code vector, n is the number of code vectors in the total code book, where each sub-code book group Satisfy the total number of code vectors For each subcodebook group C i , generated by the following formula: in, is the fixed initialized codebook core matrix, n i is the number of code vectors contained in the i-th sub-codebook, r i is the rank of the i-th subcodebook, is the learnable projection matrix, b i ∈R d is the learnable bias vector.
3. The method according to claim 1, characterized in that Joint parameterization of each sub-codebook group includes: For each sub-codebook group, a codebook core matrix and a projection matrix are defined. The codebook core matrix is linearly transformed through the projection matrix to generate a sub-codebook group.
4. The method according to claim 3, characterized in that The codebook core matrix is initialized by randomly sampling from a standard normal distribution and remains fixed during training. Only the projection matrix and bias vector are optimized.
5. The method according to claim 1, wherein The calculation method of the intra-group commitment loss is: For each sub-codebook group, the corresponding commitment loss is calculated. The total commitment loss is the sum of the losses of each sub-codebook group, and the gradient is only back-propagated to the parameters of the corresponding sub-codebook group.
6. The method according to claim 1, characterized in that After the training is completed, the codebook is resampled further including: A new codebook core vector is sampled from a predefined distribution, mapped to an extended code vector through a trained projection matrix, and the extended code vector is added to the atomic codebook group to generate an adjusted codebook without retraining the model.
7. A vector quantization system based on group optimization, using the vector quantization method based on group optimization according to any one of claims 1 to 6, characterized in that: The system comprises: An encoder, which maps input data into a continuous input feature vector; A processor configured to train a group codebook using a vector quantization method based on group optimization; The decoder reconstructs the input data based on the quantized input feature vector.