Feature encoding method, decoding method and related devices

Abstract

The application discloses a feature coding method, a decoding method, an electronic device and a computer readable storage medium. The feature coding method comprises the following steps: transforming a to-be-coded feature into a plurality of to-be-quantized features based on a preset vector dimension; quantizing each to-be-quantized feature by using at least two codebooks pre-trained, so as to obtain a quantized feature corresponding to each to-be-quantized feature; and coding a target quantized vector composed of each quantized feature, so as to obtain a coded feature, wherein the dimension of the target quantized vector is the same as the preset vector dimension. In the foregoing manner, the accuracy of feature coding can be improved.

Description

Technical Field

[0001] This application relates to the field of image encoding and decoding technology, and in particular to a feature encoding method, a decoding method, an electronic device, and a computer-readable storage medium. Background Technology

[0002] In the field of artificial intelligence, target recognition (images, speech, text, etc.) is frequently involved. Before target recognition, the target needs to be transmitted from the acquisition end (also known as the encoding end) to the recognition end (also known as the decoding end). There are two transmission methods in related technologies. One method treats the target as the transmission object, encodes it at the acquisition end, transmits the encoded result to the recognition end, and decodes the result to obtain the target. The other method considers that target recognition is based on target features. Therefore, it first extracts the target's features, uses these features as the transmission object, encodes them at the acquisition end, transmits the encoded result to the recognition end, and decodes the result to obtain the target's features. The former transmission method has lower transmission efficiency, affecting the efficiency of target recognition. The latter transmission method has higher transmission efficiency, but existing feature encoding methods for implementing the latter method still do not have high enough accuracy in feature encoding. Summary of the Invention

[0003] This application provides a feature encoding method, a decoding method, an electronic device, and a computer-readable storage medium, which can solve the problem that the accuracy of feature encoding in existing feature encoding methods is not high enough.

[0004] To address the aforementioned technical problems, this application provides a feature encoding method. This method includes: transforming the feature to be encoded into multiple features to be quantized based on a preset vector dimension; quantizing each of the features to be quantized using at least two pre-trained codebooks to obtain quantized features corresponding to each feature; and encoding a target quantization vector composed of the quantized features to obtain encoded features, wherein the dimension of the target quantization vector is the same as the preset vector dimension.

[0005] To address the aforementioned technical problems, another technical solution adopted in this application is to provide a feature decoding method. This method includes: acquiring encoded features; decoding the encoded features to obtain a target quantization vector; determining inverse quantization features from the corresponding codebook based on each quantization feature included in the target quantization vector; and inversely transforming each inverse quantization feature into a decoded feature; wherein the encoded features are obtained based on the aforementioned encoding method.

[0006] To solve the above-mentioned technical problems, another technical solution adopted in this application is: to provide an electronic device, which includes a processor and a memory connected to the processor, wherein the memory stores program instructions; the processor is used to execute the program instructions stored in the memory to implement the above-mentioned method.

[0007] To solve the above-mentioned technical problems, another technical solution adopted in this application is to provide a computer-readable storage medium storing program instructions that, when executed, can implement the above-mentioned method.

[0008] In this application, the feature to be encoded is transformed into multiple features to be quantized. Using at least two pre-trained codebooks, each feature is quantized separately to obtain its corresponding quantized feature. The target quantization vector composed of these quantized features is then encoded to obtain the encoded feature. Therefore, this application does not directly perform codebook quantization on the feature to be encoded, but rather performs multi-codebook quantization on the multiple features transformed from the feature to be encoded. This allows for a more comprehensive utilization of codebook quantization, improving the quantization precision of the feature to be encoded, reducing quantization errors, and increasing the accuracy of feature encoding. Attached Figure Description

[0009] Figure 1 This is a flowchart illustrating an embodiment of the feature encoding method of this application;

[0010] Figure 2 This is a schematic diagram of non-uniform quantization;

[0011] Figure 3 This is a schematic diagram of codebook quantization;

[0012] Figure 4 This is a flowchart illustrating another embodiment of the feature encoding method of this application;

[0013] Figure 5 This is a schematic diagram illustrating the transformation of the features to be encoded.

[0014] Figure 6 This is another schematic diagram illustrating the transformation of the features to be encoded;

[0015] Figure 7 This is another schematic diagram illustrating the transformation of the coded features;

[0016] Figure 8 This is another schematic diagram illustrating the transformation of the coded features;

[0017] Figure 9 This is a flowchart illustrating another embodiment of the feature encoding method of this application;

[0018] Figure 10 This is a schematic diagram of the first spatial transformation performed on the features to be encoded.

[0019] Figure 11 This is a schematic diagram of performing a second-space transformation on the features to be encoded;

[0020] Figure 12 This is another schematic diagram of performing the first spatial transformation on the features to be encoded;

[0021] Figure 13 This is a flowchart illustrating another embodiment of the feature encoding method of this application;

[0022] Figure 14 This is a schematic diagram of the structure of the coding network in this application;

[0023] Figure 15 This is a flowchart illustrating another embodiment of the feature encoding method of this application;

[0024] Figure 16 This is a training diagram of the quantization module;

[0025] Figure 17 This is a structural diagram of a quantization module;

[0026] Figure 18 This is a flowchart illustrating an embodiment of the feature decoding method of this application;

[0027] Figure 19 This is a schematic diagram of the structure of an embodiment of the electronic device of this application;

[0028] Figure 20 This is a schematic diagram of the structure of an embodiment of the computer-readable storage medium of this application. Detailed Implementation

[0029] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of the embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0030] The terms "first," "second," and "third" used in this application are for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined as "first," "second," or "third" may explicitly or implicitly include at least one of that feature. In the description of this application, "multiple" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0031] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments without conflict.

[0032] Figure 1 This is a flowchart illustrating an embodiment of the feature encoding method of this application. It should be noted that if substantially the same result is obtained, this embodiment does not necessarily replace it. Figure 1 The illustrated process sequence is limited. For example... Figure 1 As shown, this embodiment may include:

[0033] S11: Based on the preset vector dimension, transform the feature to be encoded into multiple features to be quantized.

[0034] The execution subject of the method embodiment of this application is a feature encoding device (encoding end). The feature encoding device can be any electronic device with feature encoding capability, such as a mobile phone, computer, image processing device, voice processing device, text processing device, or server, such as physical server, cloud server, or distributed server.

[0035] The features to be encoded are the original features of the target, which can be speech features, image features, text features, etc. These features can be extracted using neural networks. Speech features can be used for speech recognition, such as identifying a person through speech and speech-to-text conversion. Image features can be used for image recognition, such as image classification, object detection, and semantic segmentation. Text features can be used for text recognition, such as text translation and text summarization.

[0036] The transformation methods for the features to be encoded can include spatial transformation, splitting, and combinations of spatial transformation and splitting, etc.

[0037] S12: Using at least two pre-trained codebooks, quantize each feature to be quantized to obtain the quantized features corresponding to each feature.

[0038] The number of codebooks can be the same as the number of features to be quantized. In this case, there is a one-to-one correspondence between the features to be quantized and the codebooks, with each feature to be quantized corresponding to a codebook with a different identifier. Alternatively, the number of codebooks can be different from the number of features to be quantized. In this case, at least two features to be quantized correspond to codebooks with the same identifier. A feature to be quantized can be quantized into a quantized feature by its corresponding codebook.

[0039] The codebook comprises several codebook features that can be used to measure the feature to be quantized and whose dimensions are consistent with those of the feature to be quantized. Each codebook feature has its own index value. Quantizing the feature to be quantized using the codebook involves finding the codebook feature that is most similar to the feature to be quantized, using this most similar codebook feature to represent the feature to be quantized, and then using the index value of this most similar codebook feature as the result of the quantization (quantized feature).

[0040] The number of codebook features included in the codebook is determined based on the number of encoded bits of the quantized feature. For example, the quantization result of feature i to be quantized is quantized feature i. The number of codebook features in codebook i corresponding to feature i is determined based on the number of encoded bits b of quantized feature i, and the number of codebook features Mi = 2. b For each feature to be quantized, the codebook feature with the highest similarity to the feature to be quantized can be selected from the corresponding codebook as the target codebook feature; the index value of the target codebook feature is used as the quantization feature. The feature with the highest similarity to the feature to be quantized can be the codebook feature with the closest distance to the feature to be quantized, or the codebook feature with the largest weighted sum of the features to be quantized, or it can be a codebook feature selected in other ways.

[0041] In a codebook, the number of codebook features corresponding to each feature to be quantized can be the same, and correspondingly, the number of encoded bits for each quantized feature is the same; this is called uniform quantization. Alternatively, the number of codebook features corresponding to at least two features to be quantized can be different, and correspondingly, the number of encoded bits for each codebook corresponding to at least two features to be quantized is different; this is called non-uniform quantization. In non-uniform quantization, the more encoded bits / number of codebook features, the higher the encoding accuracy, and vice versa. This provides greater flexibility for codebook quantization, making it more suitable for user needs and improving encoding accuracy compared to uniform quantization.

[0042] Combination Figure 2 Examples of non-uniform quantization are provided, such as Figure 2 As shown, the transformation of the features to be encoded yields 256 8-dimensional features to be quantized. Features to be quantized 1–256 correspond to codebooks 1–256 and quantized features 1–256, respectively. The number of encoded bits for quantized features 1–64 is 2, and the number of codebook features in codebooks 1–64 is 2. 2 =4. The value range of quantization features 1 to 64 is {1, 2, 3, 4}. The number of encoded bits for quantization features 65 to 128 is 4, and the number of codebook features in codebook 65 to 128 is 2. 4=16. The value range of quantization feature 65-128 is {1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16}. The number of encoded bits for quantization feature 129-256 is 1. The number of codebook features in codebook 129-256 is 2. 1 =2. The value range of quantization feature 129 to 256 is {1, 2}. The dimensions of features in different codebooks within the same codebook are the same, and the dimensions of features in different codebooks within the same codebook are the same as the dimensions of their corresponding features to be quantized.

[0043] The following combination Figure 3 Example of S12:

[0044] like Figure 3 As shown, the transformation of the features to be encoded yields 256 8-dimensional features to be quantized. Features 1 to 256 correspond to codebooks 1 to 256, and each codebook includes four codebook features. Codebooks 1 to 256 are {A1} 1 = (a1, ..., a8), ..., A1 4};…;{A 256 1 A 256 4 The features to be quantized, 1 to 256, are X1 = (x1, ..., x8); ...; X 256 .

[0045] When the feature with the highest similarity to the feature to be quantized is the codebook feature with the largest weighted sum of similarities to the feature to be quantized, the weighted sum of each feature to be quantized and each codebook feature in the corresponding codebook can be calculated:

[0046] Y1 1 =A1 1 *X1=a1*x1+…+a8*x8;

[0047] …

[0048] Y1 4 =A1 4 *X1;

[0049] …

[0050] Y 256 1 =A 256 1 *X 256 ;

[0051] …

[0052] Y 256 4 =A 256 4 *X256 .

[0053] For X1, determine its corresponding Y1. 1 ~Y1 4 The largest one (e.g., Y1) 4 ), Y1 4 Corresponding codebook feature A1 4 As a feature of the target codebook, A1 4 The index value 4 is used as the quantization feature of X1. Similar processing can be used to obtain X2 to X... 256 Quantitative characteristics.

[0054] If the feature with the highest similarity to the feature to be quantized is the codebook feature that is closest to the feature to be quantized, then the distance between each feature to be quantized and each codebook feature in the corresponding codebook can be calculated:

[0055] L1 1 =L(A1) 1 ,X1)=|a1-x1|+...+|a8-x8|;

[0056] …

[0057] L1 4 =L(A1) 4 (X1);

[0058] …

[0059] L 256 1 =L(A 256 1 (X256);

[0060] …

[0061] L 256 4 =L(A 256 4 (X256).

[0062] For X1, determine its corresponding L1. 1 ~L1 4 The smallest one (e.g., L1) 4 ), L1 4 Corresponding codebook feature A1 4 As a feature of the target codebook, A1 4 The index value 4 is used as the quantization feature of X1. Similar processing can be used to obtain X2 to X... 256 Quantitative characteristics.

[0063] S13: Encode the target quantization vector composed of various quantization features to obtain the encoded features.

[0064] The dimension of the target quantization vector is the same as the dimension of the preset vector.

[0065] The quantization result of a feature to be quantized is a quantized feature. Therefore, the dimension of the target quantization vector composed of various quantized features is the same as the number of features to be quantized, and the preset vector dimension is also the same as the number of features to be quantized. It is understandable that feature encoding methods in related technologies directly use a codebook to quantize the features to be encoded in the codebook quantization stage, which does not give full play to the role of codebook quantization, resulting in insufficient quantization precision and large quantization errors, affecting the accuracy of encoding.

[0066] Through the implementation of this embodiment, this application transforms the feature to be encoded into multiple features to be quantized. Using at least two pre-trained codebooks, each feature to be quantized is quantized separately to obtain the quantized features corresponding to each feature. The target quantization vector composed of these quantized features is then encoded to obtain the encoded features. Therefore, this application does not directly perform codebook quantization on the feature to be encoded, but rather performs multi-codebook quantization on the multiple features to be quantized, thus more fully utilizing the role of codebook quantization, improving the quantization precision of the feature to be encoded, reducing quantization errors, and improving the accuracy of feature encoding.

[0067] Figure 4 This is a flowchart illustrating another embodiment of the feature encoding method of this application. It should be noted that if substantially the same result is obtained, this embodiment does not necessarily reflect that result. Figure 4 The illustrated process sequence is limited. This embodiment is a further extension of S11, with the transformation involving at least one split. For example... Figure 4 As shown, this embodiment may include:

[0068] S21: Perform at least one splitting process on the feature to be encoded.

[0069] The first splitting process splits the feature to be encoded into the splitting object, while the splitting objects of the subsequent splitting processes are the splitting results of the previous splitting process. The splitting process splits the splitting object into at least two splitting results whose sum or concatenation result is equal to the splitting object (i.e., the sum or concatenation result of the splitting results obtained by the splitting process is the splitting object).

[0070] S22: Take each splitting result obtained from the last splitting in at least one splitting process as a feature to be quantified.

[0071] The following combination Figures 5-8 Examples of S21 to S22 (the dimension of the feature to be encoded is D):

[0072] like Figure 5As shown, the feature to be encoded is split once to obtain features 1 to N to be quantized. The dimensions of features 1 to N to be quantized are d1 to dn, d1 = ... = dn = D. The sum of features 1 to N to be quantized is the feature to be encoded.

[0073] like Figure 6 As shown, the features to be encoded are split once to obtain features 1 to N to be quantized. The dimensions of features 1 to N to be quantized are d1 to dn, and d1+…+dn=D. The concatenation result of features 1 to N to be quantized is the feature to be encoded.

[0074] like Figure 7 As shown, the first splitting process is performed on the features to be encoded to obtain intermediate features 1 to N, with dimensions d1 to dn respectively, where d1 = ... = dn = D. The sum of intermediate features 1 to N is the feature to be encoded. Further, the intermediate features 1 to N are split a second time to obtain the quantized features 1_1 to 1_N1, ..., corresponding to intermediate feature 1, and the quantized features N_1 to 1_Nm corresponding to intermediate feature N. The total dimension of the quantized features 1_1 to 1_N1 is equal to the dimension d1 of intermediate feature 1. The concatenation result of the quantized features 1_1 to 1_N1 is intermediate feature 1, ..., the total dimension of the quantized features N_1 to 1_Nm is equal to the dimension dn of intermediate feature N. The concatenation result of the quantized features N_1 to 1_Nm is intermediate feature N.

[0075] like Figure 8 As shown, the first splitting process is performed on the feature to be encoded to obtain intermediate features 1 to N, with dimensions d1 to dn respectively, and d1 + ... + dn = D. The concatenation result of intermediate features 1 to N is the feature to be encoded. Further, the intermediate features 1 to N are split into the second splitting process to obtain the quantized features 1_1 to 1_N1, ..., corresponding to intermediate feature 1, and the quantized features N_1 to 1_Nm corresponding to intermediate feature N. The dimensions of quantized features 1_1 to 1_N1 are all equal to the dimension d1 of intermediate feature 1. The sum of quantized features 1_1 to 1_N1 is intermediate feature 1, ..., the dimensions of quantized features N_1 to 1_Nm are all equal to the dimension dn of intermediate feature N. The sum of quantized features N_1 to 1_Nm is intermediate feature N.

[0076] Figure 9 This is a flowchart illustrating another embodiment of the feature encoding method of this application. It should be noted that if substantially the same result is obtained, this embodiment does not necessarily reflect that result. Figure 9 The illustrated process sequence is limited. This embodiment is a further extension of S11, using a combination of spatial transformation and splitting. For example... Figure 9 As shown, this embodiment may include:

[0077] S31: Perform a first spatial transformation on the feature to be encoded to obtain the feature to be encoded after the first spatial transformation, or perform a second spatial transformation on the feature to be encoded to obtain the feature to be encoded after the second spatial transformation.

[0078] The feature to be encoded after the first spatial transformation includes one first latent space feature, and the feature to be encoded after the second spatial transformation includes multiple second latent space features. The dimension of the first latent space feature is equal to the dimension of the feature to be encoded, and the total dimension of the multiple second latent space features is equal to the dimension of the feature to be encoded. Compared to the first spatial transformation, the second spatial transformation has a larger error, and the feature to be encoded after the second spatial transformation is used for codebook quantization, requiring a lower degree of quantization. Compared to the second spatial transformation, the first spatial transformation has a smaller error, requiring a higher degree of quantization.

[0079] The first / second spatial transformation changes the feature space to which the feature to be encoded belongs; that is, the feature space to which the feature to be encoded after the first / second spatial transformation belongs is different from the feature space to which the feature to be encoded before the spatial transformation belongs. Through the first / second spatial transformation, the feature to be quantized can be fully compressed during subsequent quantization.

[0080] The first / second spatial transformation can be implemented through a spatial transformation module, which can be any type of neural network, such as a fully connected network, a convolutional network, or a combination of fully connected networks and convolutional networks.

[0081] The following combination Figures 10-12 Example of S31 (the dimension of the feature to be encoded is 2048):

[0082] like Figure 10 As shown, a fully connected network is used to perform a first spatial transformation on the feature to be encoded, reducing the dimension of the feature to be encoded to 256, thus obtaining a 256-dimensional first latent space feature.

[0083] like Figure 11 As shown, a fully connected network is used to perform a second spatial transformation on the features to be encoded, resulting in 256 8-dimensional second latent space features.

[0084] like Figure 12 As shown, a first spatial transformation is performed on the feature to be encoded using a fully connected network and a convolutional network, reducing the dimension of the feature to be encoded to 256, thus obtaining a 256-dimensional first latent space feature.

[0085] S32: Perform at least one splitting process on the spatially transformed features to be encoded.

[0086] S33: Take each split result obtained from the last split as a feature to be quantized.

[0087] For further detailed descriptions of this embodiment, please refer to the preceding embodiments, which will not be repeated here.

[0088] Figure 13 This is a flowchart illustrating another embodiment of the feature encoding method of this application. It should be noted that if substantially the same result is obtained, this embodiment does not necessarily reflect that result. Figure 13 The illustrated process sequence is limited. This embodiment is a further extension of S11, using a combination of spatial transformation and splitting. For example... Figure 13 As shown, this embodiment may include:

[0089] S41: Perform a second space transformation on the features to be encoded to obtain multiple second latent space features.

[0090] The total dimension of multiple second latent space features is equal to the dimension of the feature to be encoded.

[0091] S42: Each of the second latent space features is taken as a feature to be quantized.

[0092] For further detailed descriptions of this embodiment, please refer to the preceding embodiments, which will not be repeated here.

[0093] Furthermore, the feature encoding method of this application can be implemented based on an encoding network, in conjunction with reference to [reference needed]. Figure 14 The encoding network can include a transformation module, a quantization module, and an encoding module. The transformation module transforms the feature to be encoded into multiple quantized features based on a preset vector dimension. The quantization module quantizes each feature using at least two pre-trained codebooks to obtain the quantized features corresponding to each feature. The encoding module encodes the target quantization vector composed of these quantized features to obtain the encoded features, where the dimension of the target quantization vector is the same as the preset vector dimension.

[0094] Continue reading Figure 14 The encoded features obtained based on the encoding network can be further decoded by the decoding network. The decoding network can include a decoding module (used to decode to obtain the target quantization vector, corresponding to the encoding module), an inverse quantization module (used to inverse quantize to obtain inverse quantization features, corresponding to the quantization module), and an inverse transform module (used to inverse transform to obtain the decoded features, corresponding to the transform module).

[0095] Before applying the encoding network to the feature encoding method, the encoding network can be trained in conjunction with the decoding network.

[0096] During the training phase, the encoding network can be trained using the loss between the input of the quantization module (sample to be quantized) and the output of the dequantization module (dequantized sample), and the loss between the input of the transformation module (sample to be encoded) and the output of the inverse transformation module (decoded sample).

[0097] The trainable parameters of the encoding network include the trainable parameters of the transformation module, the trainable parameters of the quantization module, and the trainable parameters of the encoding module.

[0098] The trainable parameters of the quantization module are the features of each codebook within each codebook. It's understood that features to be encoded under the same application scenario or the same target recognition task have similar characteristics. During the training phase, the quantization module can be specifically trained using samples from the same application scenario or the same target recognition task. This allows the quantization module to learn the feature characteristics of the same application scenario or the same target recognition task, resulting in a codebook suitable for that application scenario or target recognition task. Therefore, during the application phase, using this codebook to quantize the same application scenario or the same target recognition task ensures that the encoding process retains the information needed for that specific application scenario or the same target recognition task, thereby improving the accuracy of subsequent target recognition.

[0099] Training of individual modules within an encoding network can be performed synchronously, meaning the parameters of each module are adjusted based on the overall output of the encoding network and the overall loss between outputs (the loss between the input of the transform module and the output of the inverse transform module). Alternatively, training of individual modules can be performed step-by-step. This involves training a subset of modules first, then fixing the trainable parameters of that subset, and then training another subset based on those parameters. For example, training all modules except the quantization module first, then fixing the trainable parameters of those modules, and finally training the quantization module based on those parameters.

[0100] The following is a detailed explanation of the synchronous execution scenario:

[0101] Figure 15 This is a flowchart illustrating another embodiment of the feature encoding method of this application. It should be noted that if substantially the same result is obtained, this embodiment does not necessarily reflect that result. Figure 15 The illustrated process sequence is limited. For example... Figure 15 As shown, this embodiment may include:

[0102] S51: The transformation module transforms the sample to be encoded into multiple samples to be quantized based on the preset vector dimension.

[0103] S52: Using at least two codebooks, the quantization module quantizes each sample to be quantized, thereby obtaining the quantized sample corresponding to each sample to be quantized.

[0104] S53: The quantization vector composed of each quantized sample is encoded by the encoding module to obtain the encoded sample.

[0105] S54: The encoded sample is decoded by the decoding module to obtain the sample quantization vector.

[0106] The dimension of the sample quantization vector is the same as the preset vector dimension.

[0107] S55: The dequantization module determines the dequantization sample from the corresponding codebook based on each quantization sample included in the sample quantization vector.

[0108] S56: The inverse transformation module transforms each inverse quantized sample into a decoded sample.

[0109] S57: Construct the loss of the encoding and decoding networks based on the decoded samples and / or inverse quantized samples.

[0110] The loss can be constructed based on the decoded samples, i.e., based on the difference between the sample to be encoded and the decoded samples (e.g., L2). The loss of the encoding network can also be constructed based on the dequantized samples, i.e., based on the difference between the dequantized samples and the sample to be quantized. When multiple losses are constructed, they can be weighted to obtain the final loss.

[0111] S58: Trainable parameters of the encoding network based on loss adjustment.

[0112] In some embodiments, the trainable parameters of the decoding network can also be adjusted based on the loss.

[0113] Adjusting the trainable parameters of the encoding and / or decoding networks based on the loss can include adjusting the parameters of any module in the encoding and / or decoding networks. The method for adjusting the trainable parameters can be a deep learning-based gradient backpropagation algorithm or a non-deep learning algorithm.

[0114] In the gradient backpropagation algorithm based on deep learning, the direction of the encoder and decoder networks—transform module → quantization module → encoder module → decoder module → inverse quantization module → inverse transform module—is called the forward path, and the direction of the inverse transform module → inverse quantization module → decoder module → encoder module → quantization module → transform module—is called the backward path. Based on this, starting from the first module on the backward path and ending with the last module, the gradients of the trainable parameters of each module are obtained sequentially based on the loss. The trainable parameters of the corresponding modules are then adjusted based on these gradients. Obtaining the gradient of a module's trainable parameters based on the loss is essentially taking the partial derivative of the loss with respect to the module's trainable parameters to obtain the gradient of that module's trainable parameters.

[0115] According to the chain rule of gradient backpropagation, when calculating the partial derivatives of the trainable parameters of a module with respect to the loss, the gradients of each module preceding that module on the backpropagation path are needed. For details, please refer to the principles of gradient backpropagation in related technologies; they will not be elaborated here. For example, the gradients of the inverse transform module, inverse quantization module, decoding module, encoding module, quantization module, and transform module are w6, w5, w4, w3, w2, and w1, respectively, and the loss is E. Taking the partial derivative of E with respect to w6 yields the gradient of w6; taking the partial derivative of E with respect to w5 yields the gradient of w5, and the gradient of w6 is needed when taking the partial derivative of E with respect to w5; ...; taking the partial derivative of E with respect to w2 yields the gradient of w2, and the gradients of w3 through w6 are needed when taking the partial derivative of E with respect to w2; and so on.

[0116] The processing of each module is implemented based on its corresponding function. Taking the quantization module as an example, quantization in the quantization module is implemented based on the quantization function. The parameters of the quantization function are the trainable parameters of the quantization module. The quantization function is either the first quantization function or the second quantization function. The parameters of the first quantization function are differentiable with respect to the loss (the partial derivative of the loss with respect to the parameters of the first quantization function can be calculated, and the gradient can continue to propagate back to the quantization module). The parameters of the second quantization function are not differentiable with respect to the loss (the partial derivative of the loss with respect to the parameters of the first quantization function cannot be calculated, and the gradient cannot continue to propagate back to the quantization module).

[0117] See also Figure 16 When the quantization of the quantization module is implemented based on the second quantization function, the training of the quantization module can be achieved using non-deep learning algorithms. The training of other modules in the encoding network besides the quantization module can still be achieved using the gradient backpropagation algorithm of deep learning. When the gradient is backpropagated to the quantization module, the gradient of the module preceding the quantization module (e.g., the encoding module) on the backpropagation path can be directly used as the gradient of the quantization module. That is, the gradient at the output q of the quantization module is copied to the input p of the quantization module for gradient acquisition and training of the transformation module. Therefore, training the transformation module can include: obtaining the gradient of the encoding module based on the loss; using the gradient of the encoding module as the gradient of the quantization module; obtaining the gradient of the transformation module based on the gradient of the quantization module; and adjusting the trainable parameters of the transformation module based on the gradient of the transformation module. Thus, even if the trainable parameters of the quantization module are not differentiable relative to the loss and the gradient cannot be backpropagated at the quantization module, the transformation module can still be trained using the gradient copying method, thereby achieving end-to-end training of the encoding network and improving its performance.

[0118] When the quantization of the quantization module is implemented based on the first quantization function, the training of the quantization module can be achieved using a deep learning-based gradient backpropagation algorithm. The gradient of the quantization module can be obtained by taking the partial derivative of the loss with respect to the parameters of the first quantization function; based on the gradient of the quantization module, the trainable parameters of the quantization module can be adjusted. After the quantization module is trained, various codebooks can be obtained. Furthermore, the training of the transformation module can also be achieved using a deep learning-based gradient backpropagation algorithm. That is, the gradient of the transformation module can be obtained based on the gradient of the quantization module; based on the gradient of the transformation module, the trainable parameters of the transformation module can be adjusted.

[0119] The following combination Figure 17 Here's an example illustrating the case where the quantization function of the quantization module is the first quantization function:

[0120] The quantization module is a fully connected layer. Its input is the feature to be quantized, Y = (y1, ..., yN), and its output is the quantized feature, x = (x1, ..., xN). The trainable parameters of the quantization module can be expressed as:

[0121]

[0122] Where w2_kj represents the trainable parameters (values ​​in the codebook features) between the k-th (belonging to 1 to N) quantized feature xi and the j-th (belonging to 1 to N) quantized feature yj.

[0123] The first quantization function is:

[0124]

[0125] Figure 18 This is a flowchart illustrating an embodiment of the feature decoding method of this application. It should be noted that if substantially the same result is obtained, this embodiment does not necessarily replace it. Figure 18 The sequence of processes shown is limited.

[0126] like Figure 18 As shown, this embodiment may include:

[0127] S61: Obtain the encoded features.

[0128] The encoding features are obtained using the encoding method provided in any of the preceding embodiments.

[0129] The execution subject of this embodiment is a feature decoding device (decoding end). The feature decoding device is similar to the feature encoding device, and will not be described in detail here.

[0130] S62: Decode the encoded features to obtain the target quantization vector.

[0131] S63: Determine the inverse quantization features from the corresponding codebook based on each quantization feature included in the target quantization vector.

[0132] S64: Transform each inverse quantization feature into a decoded feature.

[0133] For a detailed description of this embodiment, please refer to the preceding embodiments; it will not be repeated here.

[0134] Through the implementation of this embodiment, since multiple codebooks are used to quantize the features to be encoded during the encoding process, correspondingly, the encoded features are decoded to obtain the target quantization vector. Based on each quantization feature included in the target quantization vector, inverse quantization features are determined from the corresponding codebook, and each inverse quantization feature is inversely transformed into a decoded feature. Therefore, in the decoding process, more accurate decoded features can be obtained, improving the accuracy of decoding.

[0135] Figure 19 This is a schematic diagram of the structure of an embodiment of the electronic device of this application. For example... Figure 19 As shown, the electronic device includes a processor 21 and a memory 22 coupled to the processor 21.

[0136] The memory 22 stores program instructions for implementing the methods of any of the above embodiments; the processor 21 executes the program instructions stored in the memory 22 to implement the steps of the above method embodiments. The processor 21 may also be referred to as a CPU (Central Processing Unit). The processor 21 may be an integrated circuit chip with signal processing capabilities. The processor 21 may also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. The general-purpose processor may be a microprocessor or any conventional processor.

[0137] The electronic device can be either the aforementioned feature encoding device with feature encoding capability or the aforementioned feature decoding device with feature decoding capability.

[0138] Figure 20 This is a schematic diagram of the structure of an embodiment of the computer-readable storage medium of this application. Figure 20As shown, the computer-readable storage medium 30 of this application embodiment stores program instructions 31, which, when executed, implement the methods provided in the above embodiments of this application. The program instructions 31 can form a program file and be stored in the computer-readable storage medium 30 in the form of a software product, so that a computer device (which may be a personal computer, server, or network device, etc.) or processor can execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned computer-readable storage medium 30 includes various media capable of storing program code, such as a USB flash drive, portable hard drive, read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk, or terminal devices such as computers, servers, mobile phones, and tablets.

[0139] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces, or indirect coupling or communication connection between apparatuses or units, and may be electrical, mechanical, or other forms.

[0140] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated units described above can be implemented in hardware or as software functional units. The above are merely embodiments of this application and do not limit the patent scope of this application. Any equivalent structural or procedural transformations made based on the description and drawings of this application, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

Claims

1. A feature encoding method, characterized in that, include: Based on a preset vector dimension, the feature to be encoded is transformed into multiple features to be quantized, including: performing a spatial transformation on the feature to be encoded to obtain a spatially transformed feature to be encoded, wherein the spatial transformation is a first spatial transformation or a second spatial transformation, the first spatially transformed feature to be encoded includes a first latent space feature, the second spatially transformed feature to be encoded includes multiple second latent space features, performing at least one splitting process on the spatially transformed feature to be encoded to obtain multiple splitting results, and using each of the splitting results as the feature to be quantized; or, performing the second spatial transformation on the feature to be encoded to obtain multiple second latent space features; and using each of the second latent space features as the feature to be quantized. Using at least two pre-trained codebooks, each of the features to be quantized is quantized to obtain the quantized features corresponding to each feature to be quantized. The number of codebooks is the same as or different from the number of features to be quantized. One feature to be quantized is quantized into one quantized feature by its corresponding codebook. The target quantization vector composed of each of the quantization features is encoded to obtain the encoded features, wherein the dimension of the target quantization vector is the same as the dimension of the preset vector; Wherein, the feature to be encoded is one of speech features, image features, and text features. The transformation, quantization, and encoding are implemented using the transformation module, quantization module, and encoding module of the encoding network, respectively. The trainable parameters of the quantization module are the features of each codebook in each codebook. The training samples of the quantization module are the samples to be encoded under the same application scenario or the same target recognition task as the feature to be encoded.

2. The method according to claim 1, characterized in that, The initial splitting process targets the feature to be encoded, while subsequent splitting processes target the various splitting results from the previous splitting process. Each splitting process involves dividing the target feature into at least two splitting results whose sum or concatenation result equals the target feature. The process of performing at least one splitting process on the spatially transformed feature to be encoded to obtain multiple splitting results includes: Each of the splitting results obtained from the last splitting in the at least one splitting process is taken as the feature to be quantized.

3. The method according to claim 2, characterized in that, The dimension of the first latent space feature is equal to the dimension of the feature to be encoded, and the total dimension of the multiple second latent space features is equal to the dimension of the feature to be encoded.

4. The method according to claim 1, characterized in that, The codebook includes multiple codebook features, and the number of codebook features is determined based on the number of encoded bits of the quantization feature; The step of quantizing each of the features to be quantized using at least two pre-trained codebooks to obtain the quantized features corresponding to each feature to be quantized includes: For each feature to be quantized, select the codebook feature that has the greatest similarity to the feature to be quantized from the corresponding codebook and use it as the target codebook feature; The index value of the target codebook feature is used as the quantization feature.

5. The method according to claim 4, characterized in that, At least two of the features to be quantized have different numbers of codebook features in their corresponding codebooks.

6. The method according to claim 1, characterized in that, The number of codebooks is the same as the preset vector dimension, and the features to be quantized correspond one-to-one with the codebooks.

7. The method according to claim 1, characterized in that, The transformation module is used to transform the feature to be encoded into multiple features to be quantized based on a preset vector dimension; The quantization module is used to quantize each of the features to be quantized using at least two pre-trained codebooks, so as to obtain the quantized features corresponding to each of the features to be quantized. The encoding module is used to encode the target quantization vector composed of each of the quantization features to obtain the encoded features, wherein the dimension of the target quantization vector is the same as the dimension of the preset vector.

8. The method according to claim 7, characterized in that, The encoded features can be further decoded by a decoding network, which includes a decoding module, an inverse quantization module, and an inverse transform module; the method further includes: The transformation module transforms the sample to be encoded into multiple samples to be quantized based on the preset vector dimension. The quantization module uses at least two codebooks to quantize each of the samples to be quantized, thereby obtaining the quantized samples corresponding to each of the samples to be quantized. The encoding module encodes the sample quantization vector composed of each quantized sample to obtain an encoded sample. The encoded sample is decoded by the decoding module to obtain a sample quantization vector; The dequantization module determines the dequantized sample from the corresponding codebook based on each of the quantization features included in the sample quantization vector. The inverse transformation module transforms each of the inverse quantized samples into decoded samples. Based on the decoded samples and / or the inverse quantization samples, construct the loss of the encoding network and the decoding network; The trainable parameters of the encoding network are adjusted based on the loss.

9. The method according to claim 8, characterized in that, The quantization of the quantization module is implemented based on a first quantization function, the parameters of which are trainable parameters of the quantization module. The parameters of the first quantization function are differentiable with respect to the loss. Adjusting the trainable parameters of the encoding network based on the loss includes: The gradient of the quantization module is obtained by taking the partial derivative of the loss with respect to the parameters of the first quantization function. Based on the gradient of the quantization module, the trainable parameters of the quantization module are adjusted.

10. The method according to claim 8, characterized in that, The trainable parameters of the quantization module are the features of each codebook in each codebook. The quantization of the quantization module is implemented based on a second quantization function, the parameters of which are the trainable parameters of the quantization module. The parameters of the second quantization function are not differentiable with respect to the loss. Adjusting the trainable parameters of the encoding network based on the loss includes: The gradient of the encoding module is obtained based on the loss. The gradient of the encoding module is used as the gradient of the quantization module; Based on the gradient of the quantization module, the gradient of the transformation module is obtained; Based on the gradient of the transformation module, the trainable parameters of the transformation module are adjusted.

11. A feature decoding method, characterized in that, include: Obtain encoded features; Decoding the encoded features yields the target quantization vector; Based on each of the quantization features included in the target quantization vector, the inverse quantization features are determined from the corresponding codebook. Each of the aforementioned inverse quantization features is inversely transformed into a decoding feature; The encoding feature is obtained based on the encoding method described in any one of claims 1 to 10.

12. An electronic device, characterized in that, Includes a processor and a memory connected to the processor, wherein, The memory stores program instructions; The processor is used to execute the program instructions stored in the memory to implement the method of any one of claims 1-10 or 11.

13. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores program instructions that can be executed by a processor to implement the method as described in any one of claims 1-10 or 11.

Images