Image processing method and device, medium and product

Abstract

The embodiment of the invention relates to the technical field of computers, and discloses an image processing method and device, a medium and a product. The method comprises the following steps: determining a target codebook vector corresponding to each channel vector according to each channel vector of an input image and a codebook vector in a codebook; determining quantized channel characteristics according to the target codebook vector and the channel vector; and according to the quantized channel features, determining quantized image feature representation. The image generation method and device can at least solve the technical problems of low image generation efficiency and low overall representation capability of the image in the prior art.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to an image processing method, device, medium, and product. Background Art

[0002] Current mainstream image generation technologies include diffusion models and vector quantization models. The latter, with its autoregressive properties, naturally fits well with large language models, and has therefore attracted considerable attention.

[0003] The operation of a vector quantization model is divided into two main stages: first, codebook learning. As the fundamental element of image representation, the quality of the codebook plays a decisive role in model performance; second, codebook-based probabilistic image modeling. Research has found that a high-quality codebook must possess two key characteristics: the ability to simultaneously model both global and local information, and sufficient scale and high utilization.

[0004] Related technologies have adopted different strategies to achieve the desired properties of codebooks. One approach, such as VAR, employs a hierarchical modeling approach, enabling different tokens to focus on global and local information, respectively. Another approach, such as MagViT v2, attempts to expand the codebook size and improve its utilization by reducing the codebook channel dimensionality and adopting a binary representation.

[0005] However, the inventors found that there are at least the following technical problems in the relevant technology: the implementation of solution one will increase the amount of word data and reduce the efficiency of subsequent image generation; the implementation of solution two will weaken the representation ability of a single word, thereby resulting in a decrease in the overall representation ability of a fixed number of words for the image. Summary of the Invention

[0006] One purpose of the present application is to provide a method for at least solving the technical problems of low image generation efficiency and low overall image representation capability in related technologies.

[0007] To achieve the above objectives, some embodiments of the present application provide the following aspects:

[0008] In a first aspect, some embodiments of the present application also provide an image processing method, the method comprising: determining a target codebook vector corresponding to each channel vector based on each channel vector of the input image and a codebook vector in a codebook; determining a quantized channel feature based on the target codebook vector and the channel vector; and determining a quantized image feature representation based on the quantized channel feature.

[0009] In a second aspect, some embodiments of the present application further provide an electronic device comprising: one or more processors; and a memory storing computer program instructions, wherein the computer program instructions, when executed, cause the processor to perform the steps of the method described above.

[0010] In a third aspect, some embodiments of the present application further provide a computer-readable medium having computer program instructions stored thereon, wherein the computer program instructions can be executed by a processor to implement the method described above.

[0011] In a fourth aspect, some embodiments of the present application further provide a computer program product, comprising a computer program / instruction, which implements the steps of the above-described method when executed by a processor.

[0012] Compared with the related art, the solution provided in the embodiment of the present application is to determine the target codebook vector corresponding to each channel vector according to each channel vector of the input image and the codebook vector in the codebook, and then determine the quantized channel features according to the target codebook vector and the channel vector, and then determine the quantized image feature representation according to the quantized channel features, which can effectively solve the problems of low image generation efficiency and weak overall characterization ability in the related art. Specifically, by quantizing the channel vector using the codebook vector, the originally complex and large-data channel information can be cleverly converted into a finite number of representative codebook vector representations, greatly reducing the amount of data processing and reducing the computational complexity, thereby significantly improving the data processing speed in the image generation process and improving the generation efficiency; at the same time, since the codebook vector is the feature essence extracted from a large amount of image data, it can accurately abstract important features such as local details and textures of the image. Compared with the original channel vector, the quantized image feature representation discards redundant information, retains and strengthens key features, so that the image has stronger feature expression capabilities while reducing dimensionality, thereby improving the overall characterization ability of the image, and providing a better feature basis for subsequent image analysis, recognition and other tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] One or more embodiments are exemplarily illustrated by pictures in the corresponding drawings. These exemplifications do not constitute limitations on the embodiments. Elements with the same reference numerals in the drawings are represented as similar elements. Unless otherwise stated, the figures in the drawings do not constitute proportional limitations.

[0014] Figure 1 An exemplary flowchart of an image processing method provided in some embodiments of the present application;

[0015] Figure 2 An exemplary schematic diagram of an image processing method provided in some embodiments of the present application;

[0016] Figure 3 A schematic diagram comparing a spatial marker in related art and a channel quantization compressor of the present application is provided for some embodiments of the present application;

[0017] Figure 4 An exemplary effect diagram provided for some embodiments of the present application;

[0018] Figure 5 An exemplary schematic diagram of a visualization result of a codebook vector provided in some embodiments of the present application;

[0019] Figure 6 An exemplary structural diagram of an electronic device provided for some embodiments of the present application. DETAILED DESCRIPTION

[0020] To make the purpose, technical solutions, and advantages of the embodiments of this application more clear, the technical solutions in the embodiments of this application will be clearly and completely described below in conjunction with the drawings in the embodiments of this application. Obviously, the described embodiments are part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0021] The following terms are used in this document.

[0022] Diffusion models, diffusion models.

[0023] Vector Quantization models.

[0024] An encoder is an algorithm or model component that takes an input image as its processing object.

[0025] Codebook, codebook, an image representation primitive.

[0026] First embodiment

[0027] The first embodiment of this application relates to an image processing method. This method can address the technical issues of low image generation efficiency and low overall image representation capabilities in related technologies. The implementation details of this embodiment's image processing method are described below. The following content is provided solely for ease of understanding and is not essential for implementing this solution.

[0028] like Figure 1 As shown, the method may include the following steps:

[0029] Step S101, determining a target codebook vector corresponding to each channel vector according to each channel vector of the input image and a codebook vector in the codebook;

[0030] Step S102, determining a quantized channel feature according to the target codebook vector and the channel vector;

[0031] Step S103: determining a quantized image feature representation based on the quantized channel features.

[0032] Exemplarily, the method can be applied to an image compressor. The following describes each of the above steps in detail.

[0033] With respect to step S101, specifically, in some examples, the input image can be specifically the original image input to the image compressor. Each channel vector of the input image refers to the input image arranging the pixel values ​​in each channel in a specific order (such as row by row or column by column) according to its color mode, and expanding the two-dimensional pixel matrix into a one-dimensional vector. The color mode can be the red, green, and blue channels of an RGB image, a single channel of a grayscale image, etc. For example, each channel of an RGB image can be regarded as a two-dimensional pixel matrix, thereby generating three vectors corresponding to the red, green, and blue channels respectively. Each channel vector completely contains the information of all pixels in the corresponding channel. For example, in the vector of the red channel, the value of each element represents the red intensity of the pixel at the corresponding position, reflecting the distribution of red information in the image.

[0034] Specifically, a codebook is a pre-built set containing multiple codebook vectors. These are feature vectors, representative vectors derived from clustering and other processing of large amounts of image data. They can abstractly represent various image features. For example, codebook vectors can represent local features and texture features of an image.

[0035] Regarding step S102, specifically, in some examples, since the target codebook vector can represent the information contained in the channel vector to a certain extent, the quantized channel feature can be determined based on the target codebook vector and the channel vector.

[0036] Specifically, in some examples, regarding step S103, these quantized channel features can be integrated based on the image's color model. For example, for an RGB image, the quantized results of the red, green, and blue channels are sequentially combined to form a holistic, more streamlined image representation, which is the quantized image feature representation. This representation significantly reduces the amount of data while preserving key image features, making image information easier to store and transmit, and also facilitating subsequent image analysis, retrieval, classification, and other operations.

[0037] It can be found that, compared with the related art, the scheme provided in the embodiment of the application can effectively solve the problems of low image generation efficiency and weak overall representation ability in the related art by determining a target codebook vector corresponding to each channel vector according to the codebook vector in the codebook and the input image, then determining a quantized channel feature according to the target codebook vector and the channel vector, and further determining a quantized image feature representation according to the quantized channel feature. Specifically, by quantizing the channel vector using the codebook vector, the originally complex and large data amount channel information can be ingeniously converted into a limited number of representative codebook vector representations, greatly reducing the data processing amount and reducing the computational complexity, thereby significantly improving the data processing speed in the image generation process and improving the generation efficiency. At the same time, since the codebook vector is a feature essence extracted from a large amount of image data, it can accurately abstract important features such as local details and textures of the image. Compared with the original channel vector, the quantized image feature representation discards redundant information and retains and strengthens key features, so that the image has stronger feature expression ability while being reduced in dimension, thereby improving the overall representation ability of the image and providing a better feature basis for subsequent image analysis, recognition and other tasks.

[0038] Second embodiment

[0039] The second embodiment of the application relates to a processing method of an image. The second embodiment is an improvement based on the first embodiment, and the specific improvement is that in the embodiment, a specific implementation manner of how to determine a channel vector of an input image is provided.

[0040] Specifically, the determination method of the channel vector of the input image can include the following steps:

[0041] In step S201, a feature map is determined according to the input image and an encoder; the dimension of the feature map is CxHxW; C represents the number of channels, and H and W represent the height and width of the feature map, respectively;

[0042] In step S202, the channel vector is determined according to the feature map; the number of the channel vectors is C, and the dimension of each channel vector is HxW.

[0043] Regarding step S201, specifically, in some examples, the encoder can perform feature extraction on the input image to extract more representative information from the input image to generate a feature map. The feature map is a representation of image features, and its dimensions can be expressed as C×H×W. Among them, the number of channels C is used to characterize the number of different feature dimensions contained in the feature map. For example, in color images, the common C=3 corresponds to the three channels of red, green, and blue; the height H and width W are used to define the size range of the feature map in space, describing the size scale of the feature map on the two-dimensional plane.

[0044] Regarding step S202, specifically, in some examples, after obtaining the feature map in step S201, the channel vector can be determined based on the dimensional structure of the feature map. Since the feature map has C channels, it can be split according to the channel dimension to obtain C channel vectors. The size of each channel vector in the spatial dimension is related to the feature map. Figure 1 These channel vectors carry the feature information of the image in different channel dimensions. For example, in the feature map corresponding to a color image, different channel vectors can correspond to the feature information of the red, green, and blue channels respectively.

[0045] For example, suppose the input image is a 256×256 pixel color photograph. This input image can be fed into an encoder based on a convolutional neural network, which operates through a series of convolutional and pooling layers. For example, it first uses several convolutional layers to extract basic features such as edges and textures, and then uses pooling layers to downsample these features. After processing by the encoder, a feature map is obtained. Assume that the feature map has 64 channels (C), meaning it contains 64 different feature dimensions; and that its height (H) and width (W) are 32, meaning its spatial size is 32×32. Therefore, the dimensions of the feature map are 64×32×32.

[0046] Furthermore, channel vectors are determined based on the feature map. Since the number of channels C = 64, the feature map can be split along the channel dimension to obtain 64 channel vectors. Each channel vector is a two-dimensional matrix with a dimension of 32×32. They reflect the feature information of the input image from different angles. For example, some channel vectors may mainly represent the contour features of objects in the image, while others may represent color distribution features.

[0047] It is not difficult to find that in the embodiment of the present application, the input image is processed by the encoder through step S201 to obtain a feature map, and the original pixel information of the image is converted into a structured feature representation. Then, the channel vector is split out from the feature map through step S202, so that each channel vector independently carries the image feature information of a specific dimension. This method of determining the channel vector of the input image, because it orderly layers and splits the image features, can more clearly and efficiently perform targeted analysis and processing on the features of different dimensions of the image, which is convenient for subsequent tasks such as image compression and image recognition. The characteristics of the channel vector are used to mine the intrinsic features of the image, thereby improving the accuracy and efficiency of task processing, and at the same time providing a standardized and operational basis for channel-based image quantization, feature extraction and other operations.

[0048] Third embodiment

[0049] The third embodiment of the present application relates to a method for processing an image. The third embodiment is an improvement on the first embodiment, specifically comprising: providing a specific implementation method for determining a target codebook vector corresponding to each channel vector of an input image based on each channel vector and a codebook vector in a codebook.

[0050] Optionally, in this embodiment, determining the target codebook vector corresponding to each channel vector according to each channel vector of the input image and the codebook vector in the codebook, that is, step S101 may include the following steps:

[0051] Step S1011, respectively calculating the distance between each channel vector and all codebook vectors in the codebook;

[0052] Step S1012: Determine a target codebook vector according to the distance value.

[0053] Specifically, with respect to step S1011, in some examples, when calculating the distance between each channel vector and all codebook vectors in the codebook, a method such as Euclidean distance or Manhattan distance may be used to measure the degree of similarity between them. In other words, a distance calculation is performed between each channel vector and all codebook vectors in the codebook to obtain a set of distance values, which are used to represent the degree of similarity between the channel vector and different codebook vectors.

[0054] Regarding step S1012 , specifically, in some examples, a target codebook vector that best represents the channel vector may be obtained according to the distance value.

[0055] Optionally, in some embodiments, determining the target codebook vector according to the distance value, that is, step S1012, may further include the following steps:

[0056] Step S10121, determining the minimum value in the distance values corresponding to each channel vector;

[0057] Step S10122, determining the target codebook vector according to the minimum value.

[0058] Specifically, in some examples, in the set of distance values corresponding to each channel vector, the codebook vector with the minimum distance can be selected as the target codebook vector. Because the minimum distance means that the codebook vector is most similar to the corresponding channel vector, it can better replace the original channel vector, achieve the quantized representation of the channel vector, and thus reduce the data volume while preserving the important features of the image, achieving the purpose of image compression or feature processing.

[0059] For example, the feature map has three channel vectors, denoted as v1, v2, and v3, each of which is a 2x2 matrix. If At the same time, there is a codebook containing four codebook vectors, c1, c2, c3, and c4. For channel v1, the Euclidean distance formula can be used to calculate the distances between it and c1, c2, c3, and c4, respectively. Similarly, the above operations can also be performed on v2 and v3, respectively, to obtain multiple sets of distance values, which are used to judge the similarity between each channel vector and the codebook vector.

[0060] Further, after calculating the distances between channel vector v1 and codebook vectors c1, c2, c3, and c4, suppose the obtained distances are d11, d12, d13, and d14, respectively. If d12 has the minimum value, it indicates that codebook vector c2 is most similar to channel vector v1, so c2 can be determined as the target codebook vector of v1. Similarly, by comparing the distance values of channel vectors v2 and v3 with the codebook vectors, the codebook vectors with the minimum distance can be found as the target codebook vectors of v2 and v3, respectively. Finally, based on the three channel vectors, the corresponding target codebook vectors can be determined.

[0061] It should be noted that the present embodiment can also be an improvement based on the second embodiment.

[0062] It can be found that in the present embodiment, by calculating the distance values between each channel vector and all codebook vectors in the codebook, the similarity between each channel vector and the codebook vector can be comprehensively measured, providing accurate data support for subsequent screening. Therefore, based on these distance values, the target codebook vector that best matches each channel vector can be accurately determined, thereby achieving effective quantization of the channel vector, reducing data redundancy while preserving key feature information of the image, and improving the efficiency of image data storage and transmission.

[0063] Fourth embodiment

[0064] The fourth embodiment of the present application relates to a method for processing an image. The fourth embodiment is an improvement on the first embodiment, specifically comprising: providing a specific implementation method for determining quantized channel features based on a target codebook vector and the channel vector.

[0065] Optionally, in this embodiment, determining the quantized channel features based on the target codebook vector and the channel vector may specifically include: step S1021, replacing the image information carried by the channel vector with the target codebook vector to determine the quantized channel features.

[0066] Specifically, the target codebook vector replaces the image information carried by the channel vector. This can be achieved by completely replacing the original values ​​of the channel vector with the numerical content of the target codebook vector, representing the originally complex and diverse channel features with a limited number of typical feature patterns in the codebook. This can achieve quantization of the channel vector, converting the feature information of the original image into a more concise and unified form. While retaining the key features of the image, it effectively reduces the amount of data and facilitates subsequent storage and transmission of image features.

[0067] For example, suppose the input image is processed to obtain a channel vector v, whose value is , the target codebook vector c value corresponding to the channel vector is In this embodiment, the target codebook vector c replaces the image information carried by the channel vector v, that is, the value of the channel vector v is directly replaced by the value of the target codebook vector c. After the replacement, the original channel vector v becomes , the new vector is the quantized channel feature. It can be seen that through the above substitution operation, the information of the original channel vector can be represented by the typical feature pattern in the codebook, thereby realizing the quantization of the channel vector, making the data more streamlined while retaining the main features of the image in this channel.

[0068] Optionally, in some embodiments, replacing the image information carried by the channel vector with the target codebook vector to determine the quantized channel characteristics may include: replacing the image information carried by the channel vector according to the token index of the target codebook vector to determine the quantized channel characteristics.

[0069] Specifically, each target codebook vector corresponds to a unique token index in the codebook, and this index can be regarded as the "identity identifier" of the target codebook vector. When performing the above-mentioned replacement operation, the index can essentially be used to replace the specific numerical information in the original channel vector. Specifically, there is no need to directly store or transmit the high-dimensional target codebook vector itself, only its token index needs to be recorded. Later, in the decoding or image feature recovery stage, the corresponding target codebook vector can be quickly located and extracted from the codebook with the help of this index, thereby replacing the information carried by the original channel vector, thereby realizing the quantization of the channel features. This method can ensure that the key features of the image are retained while greatly reducing the amount of data storage and transmission, further improving the efficiency of image data processing.

[0070] For example, assume that the codebook contains four codebook vectors, namely c1, c2, c3 and c4, and the corresponding token indexes are 1, 2, 3, and 4, respectively. For a channel vector of the input image, its target codebook vector is determined to be c3 after calculation, that is, the token index corresponding to the channel vector is 3. When executing the steps, the original value of the channel vector is no longer directly stored or transmitted, but the image information it carries is replaced by token index 3. In subsequent processing (such as decoding), the corresponding target codebook vector c3 can be taken out from the codebook according to index 3, and the target codebook vector c3 can be used as the quantized feature of the channel.

[0071] Optionally, in some embodiments, replacing the channel vector with the target codebook vector to determine the quantized channel feature, that is, step S1021 may include:

[0072] Step S10211, traversing each channel vector; wherein, replacing the channel vector with the target codebook vector corresponding to the channel vector currently being processed, and storing the replaced target codebook vector as the quantized channel feature, until the traversal is completed, so that all channel vectors have completed the replacement operation;

[0073] Step S10212: Determine a quantized channel feature set based on all obtained target codebook vectors.

[0074] Specifically, for step S10211, for the channel vector currently being processed, its corresponding target codebook vector is determined through the previous step, and then the target codebook vector is directly used to replace the original channel vector. After the replacement is complete, the replaced target codebook vector is stored as the quantized channel feature. This process is performed sequentially for all channel vectors until all channel vectors are traversed, ensuring that each channel vector is replaced by the corresponding target codebook vector. In this way, the original set of channel vectors is converted into a new set of target codebook vectors, achieving quantization of the channel features.

[0075] Specifically, with respect to step S10212, after completing the traversal and substitution operations of step S10211, the target codebook vectors corresponding to all channel vectors are obtained. The target codebook vectors are arranged in the order of the original channel vectors to form a new set. This set is the quantized channel feature set, which represents the feature information of the original image in a compact and standardized manner. By combining all the target codebook vectors together, a unified feature representation is formed, which provides a basis for subsequent image processing tasks (such as image reconstruction, feature matching, etc.).

[0076] For example, suppose the input image is processed to obtain three channel vectors, namely The corresponding target codebook vectors are During the traversal process, we can first process v1 and replace it with c2 to obtain the quantized channel feature v1′=c2 and store it; then process v2 and replace it with c1 to obtain v2′=c1 and store it; finally process v3 and replace it with c3 to obtain v3′=c3 and store it. At this point, all channel vectors have completed the replacement operation.

[0077] Furthermore, all target codebook vectors obtained in step S10211, namely v1′ = c2, v2′ = c1, and v3′ = c3, are combined into a set. This set is the quantized channel feature set, which can be expressed as {v1′, v2′, v3′} = {c2, c1, c3}. This set represents the feature information of the original image in a compact and standardized manner, providing a foundation for subsequent image processing tasks.

[0078] It should be noted that this embodiment may also be an improvement based on the second embodiment and / or the third embodiment.

[0079] It is not difficult to find that in the embodiment of the present application, by replacing the image information of the channel vector with the target codebook vector, since the target codebook vector is a representative vector closest to the original channel feature selected from the codebook, it is possible to retain the key features of the image while converting the high-dimensional and diversified original channel data into an index representation of a finite number of vectors in the codebook, thereby significantly reducing data redundancy and storage space; and since the replacement operation discretizes continuous floating-point values ​​into discrete token indexes, it facilitates subsequent efficient compression, transmission and storage, and at the same time provides a standardized feature representation for image reconstruction, further improving the efficiency of the entire image processing process.

[0080] Fifth embodiment

[0081] The fifth embodiment of the present application relates to an image processing method. The fifth embodiment provides a specific application example and explains the above methods in principle.

[0082] Assume that the input image is given and the feature map Z∈R is obtained after processing H×W×C . Flatten it into a one-dimensional sequence Zc∈R C×H×W Then, for each channel vector Z in Zc (i,j) . Among them, Z∈R H×W×C Where Z represents the feature map obtained after the input image is processed by the encoder, H represents the height of the feature map, W represents the width of the feature map, and C represents the number of channels in the feature map, that is, the dimension of the feature. Zc represents the one-dimensional sequence of feature map Z flattened by channel. This flattening can be done by sequentially arranging the two-dimensional matrices (H×W) for each channel (C in total).

[0083] 1. Based on codebook C'∈R K×H1×W′ (where each code vector e' k ∈C', dimension is H1×W1), through the image compressor Q' according to the formula Determine the channel vector Z from the codebook C' (i,j) The closest target codebook vector. Where C'∈R K×H1×W′ The codebook used by the C' image compressor in the codebook, K represents the number of codebook vectors contained in the codebook, and H1×W1 represents the dimension of each codebook vector, that is, its corresponding spatial size. k represents the kth codebook vector in codebook C', with dimension H1×W1. All e' k (k=1, 2, ..., k) together constitute the codebook C'.

[0084] Where Q' represents the image compressor, which is used to map the channel vector to the index in the codebook. (i,j) After the feature map is flattened, the channel vector located in the i-th row and j-th column has a dimension of H1×W1. (i,j) ; C ') represents the calculation of Z (i,j) The square of the Euclidean distance to all codebook vectors e′k (|Z (i,j) -e' k | 2 ), select the index k with the smallest distance as the index of the target codebook vector.

[0085] 2. Replace the channel vector Z with the target codebook vector (i,j) , and obtain the quantized features After all channels have been subjected to this operation, the quantized channel features are determined. In other words, the codebook vector e′ corresponding to index k k Replace the original channel vector Z (i,j) . Represents the quantized channel feature, which is the target codebook vector e′ k The result after substitution. e(Q'(Z (i,j) ,C')) represents the function of obtaining the vector corresponding to the index in the codebook.

[0086] 3. Convert the quantized channel features to Z q ∈R H×W×C , in order to determine the quantized image feature representation. After combining the quantization results of all channels, it is restored to a tensor of the same dimension (H×W×C) as the original feature map Z, but each channel vector has been replaced by the codebook vector.

[0087] The core process of the solution provided in this embodiment is: input image → feature extraction (Z) → channel flattening (Zc) → channel vector matching with codebook (Q') → quantization replacement → reconstruction of feature map (Z q ). It can be seen that in this application, the compression and representation of image features are achieved by quantizing the channel dimension (rather than the traditional spatial dimension) and using the codebook vector to replace the original channel vector.

[0088] It can be understood that in related technologies, image quantization is achieved through a space-based image compressor. Specifically, the encoder first extracts features from a given image, and in this process the feature size is changed based on the downsampling factor. The image features are then converted into discrete tags in the spatial dimension, that is, for each vector in the feature, the closest tag index is found from the codebook to replace the original vector. Finally, the decoder takes the quantized features as input and outputs a reconstructed image. Current advanced compressors use high codebook usage and multi-scale feature quantization strategies. The former improves codebook utilization by reducing the code embedding dimension, but reduces the capacity of the image compressor; the latter uses multiple sets of compressors to capture global and local features of the image, but brings high computational costs.

[0089] The present application proposes an image compressor based on channel quantization. Unlike existing spatial compressors, it quantizes image features to channel dimensions. First, it is flattened into a 1D sequence. For each vector in the sequence, the image compressor finds the closest token index from the codebook through a specific similarity calculation. The quantized features can be converted into a specific form and then mapped to the reconstructed image through the decoder. An image is taken as input, and the image feature F (with dimensions CxHxW) is obtained after being processed by the encoder. Then, for each channel vector (with dimensions HxW), the nearest codebook vector is found from the codebook as a replacement representation of the channel vector. Finally, the newly obtained representation is processed by the decoder and the reconstructed image is output. The difference between the input image and the reconstructed image is used as the loss to train the encoder, codebook, and decoder.

[0090] It can be seen that in order to solve the problems existing in codebook design in related technologies, this embodiment proposes a new image compressor based on channel quantization. Existing image compressors usually quantize in the spatial dimension (H×W), while the method provided in this embodiment directly quantizes the channel dimension (C).

[0091] See also Figure 2 , Figure 2 This is an exemplary schematic diagram of the image processing method provided by the present application, and the process is: input image (Image) → encoder (Encoder) → obtain feature map, dimensions are H, W, C) → decoder (Decoder) → reconstruct image (Reconstruction). Specifically, the beginning is "Image", which is a picture, such as the picture of the puppy in the picture. The picture enters the "Encoder". The role of the encoder is to convert the picture from the original pixel representation to a more abstract feature representation. In this process, the picture is converted into a three-dimensional feature map with height (H), width (W) and number of channels (C), just like the set of three-dimensional blocks shown in the middle. The feature map obtained by the encoder then enters the "Decoder". The task of the decoder is to convert the feature map back into image form to obtain "Reconstruction". In theory, the reconstructed image should be as close to the original input image as possible.

[0092] See also Figure 3 As shown, it is a comparative diagram of the spatial marker in the related art and the channel quantization compressor of the present application. The three-dimensional block on the left is a schematic diagram of the channel quantization compressor of the present application, and the three-dimensional block on the right is a schematic diagram of the spatial marker in the related art, and the dimensions of the feature map are H, W, and C. It can be found that the spatial marker in the related art, that is, the three-dimensional block on the right, pays more attention to the information of the feature map in the spatial dimension (height H and width W). The figure shows the division of the feature map in the spatial dimension, and the feature map is divided into blocks in space, and each block is a token. The channel quantization compressor of the present application, that is, the three-dimensional block on the left, focuses on the channel dimension (C) of the feature map, processes the feature map in the channel dimension, takes different channels as tokens, and pays attention to the differences and information between channels.

[0093] See also Figure 4 As shown, Figure 4To in some examples, the effect diagram of the image processing method provided by the present application is shown. The figure shows the influence of different codebook sizes on reconstruction performance (rFID), unique ratio and usage. Among them, the codebook size (Codebook size): represents the number of elements contained in the codebook, from small to large, 1024, 16384, 65536 and 131072. Reconstruction performance (rFID): with the increase of codebook size, the rFID value gradually decreases from 2.25 to 1.33. The lower the rFID value, the better the reconstruction performance. Unique ratio (Unique ratio): with the increase of codebook size, the unique ratio gradually increases from 72.5% to 96.9%. The higher the unique ratio, the greater the proportion of unique elements in the codebook. Usage (Usage): the usage of all codebooks is 100%, indicating that all elements in the codebook are fully utilized. In summary, it can be seen that, by the technical solution provided by the present application, with the increase of codebook size, the reconstruction performance is improved, the unique ratio is improved, and the usage remains at 100%.

[0094] Referring to Figure 5 As shown, a schematic diagram of the visualization result of a codebook vector in an example is shown, where each small square represents a codebook vector. The color and pattern changes in the figure show the different response patterns of the codebook vector globally and locally. It can be understood that the global response refers to the consistency or regularity exhibited by the entire codebook vector set as a whole. The global feature is shown in the figure by the repeated appearance of a certain color or pattern. The local response refers to the difference or specific pattern exhibited by the codebook vector in the local area. The local feature can be shown in the figure by the change of color or pattern. The different global and local responses show that the method provided by the present application can effectively capture the features of the data at different scales, thereby achieving more precise modeling.

[0095] The step division of the above various methods is only for the purpose of clear description, and when implemented, one step can be combined or some steps can be split and decomposed into multiple steps, as long as the same logical relationship is included, all within the protection scope of the present application; adding insignificant modifications or introducing insignificant designs in the algorithm or process, but not changing the core design of the algorithm and process, are within the protection scope of the present application.

[0096] In addition, some embodiments of the present application also provide an electronic device. The electronic device can be various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, etc. The electronic device can also be various forms of mobile devices, such as personal digital assistants, cellular phones, smart phones, wearable devices and other similar computing devices.

[0097] The electronic device includes: one or more processors; and a memory storing computer program instructions, wherein the computer program instructions, when executed, enable the processor to perform the steps of the method provided in any one or more of the above embodiments. Figure 6 An exemplary structural diagram of the electronic device is disclosed. The electronic device includes: one or more processors 1101, a memory 1102, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. The various components are connected to each other using different buses and can be installed on a common mainboard or installed in other ways as needed. The processor can process instructions executed in the electronic device, including instructions stored in or on the memory to display graphical information of the GUI on an external input / output device (such as a display device coupled to the interface). In some other embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Similarly, multiple electronic devices can be connected, and each device provides some necessary operations. Among them, the components shown in this article, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present application described and / or required herein.

[0098] The electronic device may further include an input device 1103 and an output device 1104. The processor 1101, the memory 1102, the input device 1103 and the output device 1104 may be connected via a bus or other means, with the bus connection being used as an example in the figure.

[0099] The input device 1103 can receive input digital or character information and generate key signal input related to user settings and function control of the electronic device, such as input devices such as a touch screen, a keypad, a mouse, a trackpad, a touch pad, an indicator stick, one or more mouse buttons, a trackball, and a joystick. The output device 1104 may include a display device, an auxiliary lighting device (e.g., an LED), and a tactile feedback device (e.g., a vibration motor). The display device may include, but is not limited to, a liquid crystal display, a light emitting diode display, and a plasma display. In some embodiments, the display device may be a touch screen.

[0100] To provide for interaction with a user, the electronic device can be a computer. The computer has a display device (e.g., a cathode ray tube or LCD monitor) for displaying information to the user and a keyboard and a pointing device (e.g., a mouse) by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, etc.); and input from the user can be received in any form (e.g., acoustic input, speech input, tactile input, etc.).

[0101] In the embodiments of the present application, the computer program / instruction stored on the computer readable medium is executed by the processor to implement the steps of the method provided by any one or more of the above embodiments. The computer readable medium can be included in the electronic device described in the above embodiments; or can exist separately and not be assembled into the device. The computer readable medium carries one or more computer readable instructions.

[0102] The memory 1102 can be used as a non-transitory computer readable storage medium to store non-transitory software programs, non-transitory computer executable programs and modules. The processor 1101 executes various functions and data processing of the server by running the non-transitory software programs, instructions and modules stored in the memory 1102, so as to implement the program instructions / modules corresponding to the method provided by any one or more of the above embodiments in the embodiments of the present application.

[0103] The memory 1102 can include a program storage area and a data storage area, wherein the program storage area can store an operating system, at least one application required by a function; and the data storage area can store data created according to the use of the electronic device, etc. In addition, the memory 1102 can include a high-speed random access memory, and can also include a non-transitory memory, such as at least one magnetic disk storage device, a flash memory device, or other non-transitory solid-state memory device. In some embodiments, the memory 1102 can optionally include a memory disposed remotely with respect to the processor 1101, and these remote memories can be connected to the electronic device through a network. Examples of the above network include but are not limited to the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof.

[0104] It should be noted that the computer-readable medium described in this application may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above. Computer-readable media may be, for example, but not limited to: electrical, magnetic, optical, electromagnetic, infrared or semiconductor systems, devices or components, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory, a read-only memory, an erasable programmable read-only memory, an optical fiber, a portable compact disk read-only memory, an optical storage device, a magnetic storage device, or any suitable combination of the above. In this application, a computer-readable medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, device or device.

[0105] Computer-readable media includes both permanent and non-permanent, removable and non-removable media, and can be implemented using any method or technology for information storage. The information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change memory, static random access memory, dynamic random access memory, other types of random access memory, read-only memory, electrically erasable programmable read-only memory, flash memory or other memory technology, compact discs, digital versatile discs or other optical storage, magnetic cassettes, magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information that can be accessed by a computing device.

[0106] Computer program code for performing the operations of the present application can be written in one or more programming languages, or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as C or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer can be connected to the user's computer through any type of network, including a local area network or a wide area network, or can be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0107] In the above-described embodiment, can realize wholly or in part by software, hardware, firmware or its arbitrary combination.For example, can adopt application-specific integrated circuit, general-purpose computer or any other similar hardware device to realize.In certain embodiments, the software program of the present application can be carried out to realize above steps or function by processor.Similarly, the software program of the present application (comprising relevant data structure) can be stored in computer-readable recording medium, for example, RAM memory, magnetic or optical drive or floppy disk and similar device.In addition, some steps or functions of the present application can adopt hardware to realize, for example, as the circuit that cooperates with processor to perform each step or function.

[0108] The computer program product provided by the embodiment of the present application includes one or more computer programs / instructions, and when the computer program / instructions are executed by the processor, the process or function described in the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instruction can be stored in a computer-readable storage medium, or transmitted from a computer-readable storage medium to another computer-readable storage medium. For example, the computer instruction can be transmitted from a website, a computer, a server or a data center by wired (such as coaxial cable, optical fiber, digital subscriber line) or wireless (such as infrared, wireless, microwave, etc.) mode to another website, a computer, a server or a data center. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server, a data center that includes one or more available media integrations. The available medium can be a magnetic medium (such as a floppy disk, a hard disk, a magnetic tape), an optical medium (such as a DVD), or a semiconductor medium (such as a solid-state hard disk) etc.

[0109] The flowcharts or block diagrams in the accompanying drawings illustrate the possible architectures, functions and operations of the devices, methods and computer program products according to various embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a module, program segment or part of code, and the module, program segment or part of code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, as well as the combination of boxes in the block diagram and / or flowchart, can be implemented with a dedicated hardware-specific system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0110] The scope of this application is defined by the appended claims rather than the foregoing description and is therefore intended to encompass within this application all changes that come within the meaning and range of equivalents of the claims. Any reference signs in the claims should not be construed as limiting the claims to which they relate. In addition, it is clear that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units or devices stated in a device claim may also be implemented by one unit or device through software or hardware. Words such as "first" and "second" are only used to distinguish the description and do not indicate any particular order, nor should they be understood as indicating or implying relative importance.

[0111] The above descriptions are merely specific embodiments of the present application, but the scope of protection of the present application is not limited thereto. Any person skilled in the art may easily propose variations or substitutions within the technical scope disclosed in the present application, and such variations or substitutions shall be encompassed within the scope of protection of the present application. Therefore, the scope of protection of the present application shall be subject to the scope of protection of the claims, and the above descriptions shall be regarded as exemplary and non-limiting.

Claims

1. A method for processing an image, characterized in that: The method comprises: Determine the target codebook vector corresponding to each channel vector according to each channel vector of the input image and the codebook vector in the codebook; Determining quantized channel features according to the target codebook vector and the channel vector; A quantized image feature representation is determined based on the quantized channel features.

2. The method according to claim 1, characterized in that The method for determining the channel vector of the input image includes: Determine a feature map according to the input image and the encoder; the dimension of the feature map is C×H×W, where C represents the number of channels, and H and W represent the height and width of the feature map, respectively; The channel vector is determined according to the feature map; wherein the number of the channel vectors is C, and the dimension of each channel vector is H×W.

3. The method according to claim 1, characterized in that The determining, based on each channel vector of the input image and a codebook vector in the codebook, a target codebook vector corresponding to each channel vector includes: Calculate the distance between each channel vector and all codebook vectors in the codebook respectively; A target codebook vector is determined according to the distance value.

4. The method according to claim 3, characterized in that The determining of the target codebook vector according to the distance value is specifically as follows: Determine the minimum value among the distance values ​​corresponding to each channel vector; The target codebook vector is determined according to the minimum value.

5. The method according to any one of claims 1 to 4, characterized in that The determining of the quantized channel feature according to the target codebook vector and the channel vector includes: The target codebook vector replaces the image information carried by the channel vector to determine the quantized channel feature.

6. The method according to claim 5, characterized in that The step of replacing the image information carried by the channel vector with the target codebook vector to determine the quantized channel feature includes: The image information carried by the channel vector is replaced according to the token index of the target codebook vector to determine the quantized channel feature.

7. The method according to claim 5, characterized in that Replacing the channel vector with the target codebook vector to determine the quantized channel feature includes: Traversing each channel vector; wherein, replacing the channel vector with the target codebook vector corresponding to the channel vector currently being processed, and storing the replaced target codebook vector as the quantized channel feature until the traversal is completed, so that all channel vectors complete the replacement operation; According to all the obtained target codebook vectors, the quantized channel feature set is determined.

8. An electronic device, characterized in that: The electronic device comprises: one or more processors; and A memory storing computer program instructions, which, when executed, cause the processor to perform the steps of the method according to any one of claims 1 to 7.

9. A computer-readable medium having a computer program / instruction stored thereon, characterized in that: When the computer program / instructions are executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images