A method, device and electronic device for tokenizing an image
By performing time-domain to frequency-domain transformation and quantization on images or videos, the problem of visual feature extractors relying on training set quality and consuming large amounts of resources in existing technologies is solved, and a fast and flexible image or video tokenization process is realized.
Patent Information
- Application Number
- CN202411287882.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-14
- Publication Date
- 2024-11-22
- Estimated Expiration
- 2044-09-14
AI Technical Summary
In the process of tokenizing images or videos, existing technologies rely on the quality of the training set for visual feature extractors, making it difficult to autonomously adjust the removal or retention of high-frequency and low-frequency information. This results in slow inference speeds and a large consumption of video memory resources.
By performing time-domain to frequency-domain transformation on the three channels of the image, quantizing the frequency domain matrix using a quantization table of the same size, a sparse integer matrix is obtained, which is then converted into a symbol sequence. Finally, the high-frequency co-occurring symbols are combined to complete tokenization.
It enables a fast image or video tokenization process, reduces GPU resource consumption, and has the ability to flexibly adjust high-frequency and low-frequency information to adapt to the needs of different task scenarios.
Smart Images

Figure CN118803262B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a method, device and electronic device for tokenizing an image. Background Art
[0002] Tokenizing an image or video is a method of encoding and representing an image or video with a sequence of discrete symbols similar to language. In the field of artificial intelligence, this process allows visual tasks to adapt to the structure and technology of language models.
[0003] The existing methods usually first train a visual feature extractor to extract image and video features, and then discretize them by code table or quantization to form a token sequence (in computer science and information technology, "token" is a term with multiple meanings and uses. It is usually regarded as a unit representing a certain information or data, and has a specific meaning and function in a specific context. In the parsing algorithm, the input data is usually decomposed into a series of tokens (for example, tokenize an image or video to obtain a series of tokens), and then the parser constructs the data structure or performs corresponding operations based on these tokens). The shortcomings of this type of method include:
[0004] (1) Visual feature extractors are data-driven, their effectiveness depends heavily on the quality of the training set, and they are difficult to train.
[0005] (2) Under a certain limit on the total number of tokens, a lot of visual details are lost, and it is impossible to autonomously adjust the removal or retention of high-frequency and low-frequency information.
[0006] (3) The inference speed is relatively slow and will occupy a large amount of graphics card memory (GPU) resources. Summary of the invention
[0007] In order to solve the problems existing in the prior art, the present invention provides the following technical solutions.
[0008] A first aspect of the present invention provides a method for tokenizing an image, comprising:
[0009] Perform time domain-frequency domain conversion on the three channels of the image respectively, and obtain the frequency domain matrix accordingly;
[0010] quantizing the frequency domain matrix using a quantization table of the same size as the frequency domain matrix to obtain a sparse integer matrix;
[0011] Convert the sparse integer matrix obtained after quantization into a symbol sequence;
[0012] The high-frequency co-occurring symbols in the transformed symbol sequence are combined into new symbols to obtain a new symbol sequence, thus completing the tokenization of the image.
[0013] Preferably, the three channels of the image are respectively converted from time domain to frequency domain, and the corresponding frequency domain matrix includes:
[0014] Convert the RGB format of the image into the brightness-chrominance format;
[0015] The three channels of the brightness-color format are converted from time domain to frequency domain respectively, and the frequency domain matrix is obtained accordingly.
[0016] Preferably, the image comprises an image converted from a video in a time-sequential frame extraction manner.
[0017] Preferably, the step of quantizing the frequency domain matrix using a quantization table of the same size as the frequency domain matrix to obtain a sparse integer matrix comprises:
[0018] The elements in the frequency domain matrix are divided by the elements in the corresponding position in the quantization table, and rounded to get a sparse integer matrix.
[0019] Preferably, the elements in the quantization table are compression coefficients of elements at corresponding positions in the frequency domain matrix.
[0020] Preferably, the compression coefficient is set as follows:
[0021] The size of the compression coefficient is set according to the category of the downstream task scenario. When facing downstream task scenarios such as image understanding or image retrieval, the elements with row numbers greater than 4 and column numbers greater than 4 in the quantization table are set to 200-300; when facing downstream task scenarios such as image storage or image transmission, the elements with row numbers greater than 4 and column numbers greater than 4 in the quantization table are set to 1-200.
[0022] Preferably, converting the sparse integer matrix obtained after quantization into a symbol sequence comprises:
[0023] Each tuple in each row of the sparse integer matrix is mapped to a unique symbol and listed in order from left to right; wherein the tuple consists of a non-zero element value and a vertical coordinate;
[0024] Arrange all symbols in the order of the rows of the matrix from top to bottom, and use special symbols to mark the end of each row and the beginning and end of each channel to form a symbol sequence.
[0025] A second aspect of the present invention provides a device for tokenizing an image, comprising:
[0026] The time domain-frequency domain conversion module is used to perform time domain-frequency domain conversion on the three channels of the image respectively, and obtain the corresponding frequency domain matrix;
[0027] A quantization module, used to quantize the frequency domain matrix using a quantization table of the same size as the frequency domain matrix to obtain a sparse integer matrix;
[0028] A symbol sequence conversion module, used for converting the sparse integer matrix obtained after quantization into a symbol sequence;
[0029] The Tokenize module is used to combine the high-frequency co-occurring symbols in the converted symbol sequence into new symbols, obtain a new symbol sequence, and complete the tokenization of the image.
[0030] A third aspect of the present invention provides a memory storing a plurality of instructions, wherein the instructions are used to implement the method for tokenizing an image as described in the first embodiment.
[0031] A fourth aspect of the present invention provides an electronic device, comprising a processor and a memory connected to the processor, wherein the memory stores a plurality of instructions, and the instructions can be loaded and executed by the processor so that the processor can execute the method for tokenizing an image as described in Example 1.
[0032] The beneficial effects of the present invention are as follows: the present invention provides a method, device and electronic device for tokenizing an image, which solves the core deficiencies of the existing technical solutions by performing frequency domain transformation on the image or video, and then quantizing, compressing and tokenizing the frequency domain information. The frequency domain transformation method is based on mathematical convolution transformation, has a fast algorithm, and does not require a training model, solving the problems of data dependence and slow reasoning speed. At the same time, the result of the frequency domain transformation has a clear physical meaning, and the restoration quality of the image or video can be controlled by adjusting the retention and compression ratio of high-frequency and low-frequency information. BRIEF DESCRIPTION OF THE DRAWINGS
[0033] Figure 1 This is a flow chart of the method for tokenizing an image according to the present invention;
[0034] Figure 2 The figure is a functional structure diagram of the device for tokenizing an image according to the present invention. DETAILED DESCRIPTION
[0035] In order to better understand the above technical solution, the above technical solution will be described in detail below in conjunction with the accompanying drawings and specific implementation methods.
[0036] The method provided by the present invention can be implemented in the following terminal environment, and the terminal may include one or more of the following components: a processor, a memory, and a display screen. The memory stores at least one instruction, and the instruction is loaded and executed by the processor to implement the method described in the following embodiment.
[0037] The processor may include one or more processing cores. The processor uses various interfaces and lines to connect various parts of the entire terminal, and executes various functions of the terminal and processes data by running or executing instructions, programs, code sets or instruction sets stored in the memory, and calling data stored in the memory.
[0038] The memory may include random access memory (RAM) or read-only memory (ROM). The memory may be used to store instructions, programs, codes, code sets or instructions.
[0039] The display screen is used to display the user interface of each application.
[0040] In addition, those skilled in the art can understand that the structure of the above terminal does not constitute a limitation on the terminal, and the terminal may include more or fewer components, or combine certain components, or arrange the components differently. For example, the terminal also includes components such as a radio frequency circuit, an input unit, a sensor, an audio circuit, and a power supply, which will not be described in detail here.
[0041] Embodiment 1
[0042] like Figure 1 As shown, an embodiment of the present invention provides a method for tokenizing an image, including: S101, performing time domain-frequency domain conversion on three channels of the image respectively, and obtaining a corresponding frequency domain matrix; S102, quantizing the frequency domain matrix using a quantization table of the same size as the frequency domain matrix, and obtaining a sparse integer matrix; S103, converting the sparse integer matrix obtained after quantization into a symbol sequence; S104, combining high-frequency co-occurring symbols in the converted symbol sequence into new symbols, obtaining a new symbol sequence, and completing the tokenization of the image.
[0043] The above method provided by the present invention can not only realize the tokenization of images, but also extract frames of a video in time sequence to obtain multiple images, and then tokenize each of the obtained images, thereby realizing the tokenization of videos.
[0044] In step S101, performing time-domain-frequency domain conversion on the three channels of the image respectively to obtain a corresponding frequency domain matrix may include: converting the RGB format of the image into a luminance-chrominance format; performing time-domain-frequency domain conversion on the three channels of the luminance-chrominance format respectively to obtain a corresponding frequency domain matrix.
[0045] In the specific implementation process, the RGB format of the image can be converted into the brightness-chrominance format through linear transformation. For example, Y, U and V can be obtained using the following conversion formula:
[0046] ;
[0047] The Y channel represents brightness, and the U and V channels represent chrominance.
[0048] Then, the three channels Y, U and V can be converted from time domain to frequency domain respectively, and the frequency domain matrices can be obtained respectively. The high-frequency components with larger intensity in the image will be concentrated in the upper left corner of the frequency domain matrix. The specific conversion method can use two-dimensional discrete cosine transform (DCT) and wavelet transform (DWT).
[0049] In step S102, the quantization table of the same size as the frequency domain matrix is used to quantize the frequency domain matrix to obtain a sparse integer matrix, which includes: dividing the elements in the frequency domain matrix by the elements at the corresponding positions in the quantization table, and rounding to obtain a sparse integer matrix. The elements in the quantization table are compression coefficients of the elements at the corresponding positions in the frequency domain matrix.
[0050] Since each element of the frequency domain matrix corresponds to a spectral component, the above method can be used to adjust the compression coefficient corresponding to each element in the frequency domain matrix, that is, the quantization table value, to control whether to retain the spectral components corresponding to each element of the frequency domain matrix, which has good flexibility. At the same time, taking integer values after quantization can make the obtained matrix discrete, laying the foundation for the subsequent tokenize process.
[0051] During the specific implementation process, the compression coefficient can be set as follows: the size of the compression coefficient is set according to the category of the downstream task scenario. When facing downstream task scenarios such as image understanding or image retrieval, the elements with row numbers greater than 4 and column numbers greater than 4 in the quantization table are set to 200-300; when facing downstream task scenarios such as image storage or image transmission, the elements with row numbers greater than 4 and column numbers greater than 4 in the quantization table are set to 1-200.
[0052] In the present invention, corresponding quantization tables are independently designed for different scenarios, so that the sparsity of the frequency domain matrix after quantization (the number of elements with zero in the matrix) can meet expectations. Specifically, for downstream task scenarios such as image understanding and image retrieval, the image clarity requirements are relatively low, so the high-frequency components in the quantization table can be increased (for example, the following quantization table Q can be Y The elements with row numbers greater than 4 and column numbers greater than 4 are increased to 300), to ensure that the high-frequency signal of the original image is compressed to 0, and a higher compression ratio is obtained. On the contrary, in the task scenario with high-definition requirements for images, the following quantization table Q can be maintained or reduced Y The high frequency components in (for example, the following quantization table Q Y The elements with row numbers greater than 4 and column numbers greater than 4 are reduced to 1) to reduce high-frequency signal loss and meet clarity requirements.
[0053] In the embodiment of the present invention, each element in the frequency domain matrix is divided by a corresponding compression coefficient and rounded. A matrix composed of all compression coefficients and having the same size as the frequency domain matrix is a quantization table. For example, taking the brightness channel of the DCT of 8*8 frequency components as an example, a common quantization table is as follows:
[0054] .
[0055] In step S103, converting the sparse integer matrix obtained after quantization into a symbol sequence may include:
[0056] Each pair (non-zero element value, vertical coordinate) in each row of the sparse integer matrix is mapped to a unique symbol and listed in order from left to right;
[0057] Arrange all symbols in the order of the rows of the matrix from top to bottom, and use special symbols to mark the end of each row and the beginning and end of each channel to form a symbol sequence.
[0058] In this method, the sparsity of the quantized matrix is utilized to discard elements with a value of 0, and only the non-zero elements and their positions, i.e., the binary (non-zero element value, vertical coordinate), are encoded, which greatly reduces the total length of the sequence after tokenization.
[0059] The purpose of the common JPG and JPEG2000 encoding methods is to convert an image into a 0-1 bit stream. In the present invention, the purpose is to convert an image into a shorter discrete symbol sequence. Specifically, each tuple (non-zero element value, ordinate) can be uniquely represented using coding symbols, thereby completely describing the quantized sparse integer matrix in a manner similar to a structured language.
[0060] In one embodiment of the present invention, the specific steps may be as follows:
[0061] (1) For each row of the sparse integer matrix, use a pair of (non-zero element value, vertical coordinate) to list all non-zero elements from left to right.
[0062] (2) Map each binary tuple into a unique encoding symbol.
[0063] (3) Use special characters to mark the end of each row of the sparse integer matrix, as well as the beginning and end of each channel (e.g., the three channels Y, U, and V).
[0064] For example, the quantized sparse integer matrix corresponding to channel Y is:
[0065] ;
[0066] The converted symbol sequence is: ;
[0067] in <y>and <\Y> represent the beginning and end of the channel respectively. Represents the end of each line, that is, a line break. , which are the unique identifiers of each tuple (non-zero element value, vertical coordinate).
[0068] If the possible values of the elements of the sparse integer matrix are M, the number of frequency domain components (i.e. the maximum number of columns of the sparse integer matrix) is N, and the number of special symbols required in the symbol sequence conversion process is C, then the total vocabulary size is For example, in one embodiment of the present invention, , the total vocabulary size is about 30k, which is similar to the size of common language model vocabulary.
[0069] In step S104, by recombining the symbol sequence obtained by the transformation, the high-frequency co-occurring symbols are combined into new symbols, so that the total length of the sequence can be compressed. In one embodiment of the present invention, a common algorithm for language model tokenize can be used: Byte Pair Encoding (BPE).
[0070] Compared with the existing technical solutions based on feature extractors, the method provided by the present invention has the following advantages:
[0071] 1. Fast running speed: Based on time-domain-frequency transformations such as DCT and DWT, the algorithm matures quickly and does not occupy GPU resources.
[0072] 2. Good scalability: Each step has a clear physical meaning, which can easily adjust the trade-off between high-frequency and low-frequency information, as well as the total number of tokens after encoding, and can be flexibly adapted to different fields and tasks.
[0073] 3. Simple implementation: Existing methods require high-quality training data and a large number of targeted hyperparameter searches, so they are difficult to implement and take a long time to implement. The method of the present invention does not require a training model, and is more convenient to implement and deploy.
[0074] Embodiment 2
[0075] like Figure 2 As shown, another aspect of the present invention also includes a functional module architecture that is completely consistent with the aforementioned method flow, that is, an embodiment of the present invention also provides a device for tokenizing an image, including: a time domain-frequency domain conversion module 201, used to perform time domain-frequency domain conversion on the three channels of the image respectively, and obtain a corresponding frequency domain matrix; a quantization module 202, used to quantize the frequency domain matrix using a quantization table of the same size as the frequency domain matrix, and obtain a sparse integer matrix; a symbol sequence conversion module 203, used to convert the sparse integer matrix obtained after quantization into a symbol sequence; a Tokenize module 204, used to combine high-frequency co-occurring symbols in the converted symbol sequence into new symbols, obtain a new symbol sequence, and complete the tokenization of the image.
[0076] Furthermore, in the time domain-frequency domain conversion module, the three channels of the image are respectively converted from time domain to frequency domain to obtain a corresponding frequency domain matrix, which may include: converting the RGB format of the image into a luminance-chrominance format; and performing a time domain-frequency domain conversion on the three channels of the luminance-chrominance format to obtain a corresponding frequency domain matrix.
[0077] The image may include an image converted from a video in a time-sequential frame extraction manner.
[0078] Furthermore, in the quantization module, quantizing the frequency domain matrix using a quantization table of the same size as the frequency domain matrix to obtain a sparse integer matrix may include: dividing the elements in the frequency domain matrix by the elements at corresponding positions in the quantization table, and rounding to obtain a sparse integer matrix.
[0079] The elements in the quantization table are compression coefficients of the elements at the corresponding positions in the frequency domain matrix. The compression coefficient can be set as follows: the size of the compression coefficient is set according to the category of the downstream task scenario. When facing downstream task scenarios such as image understanding or image retrieval, the elements in the quantization table with row numbers greater than 4 and column numbers greater than 4 are set to 200-300; when facing downstream task scenarios such as image storage or image transmission, the elements in the quantization table with row numbers greater than 4 and column numbers greater than 4 are set to 1-200.
[0080] Furthermore, in the symbol sequence conversion module, converting the sparse integer matrix obtained after quantization into a symbol sequence may include: mapping each binary group (non-zero element value, vertical coordinate) in each row of the sparse integer matrix into a unique symbol, and listing them in order from left to right; arranging all symbols in order from top to bottom according to the rows of the matrix, and marking the end of each row and the beginning and end of each channel with special symbols to form a symbol sequence.
[0081] The device can be implemented by the method for tokenizing an image provided in the above-mentioned embodiment 1. The specific implementation method can be found in the description of embodiment 1 and will not be described again here.
[0082] The present invention also provides a memory storing a plurality of instructions, wherein the instructions are used to implement the method for tokenizing an image as described in the first embodiment.
[0083] The present invention also provides an electronic device, including a processor and a memory connected to the processor, wherein the memory stores a plurality of instructions, and the instructions can be loaded and executed by the processor so that the processor can execute the method for tokenizing an image as described in Example 1.
[0084] Although preferred embodiments of the present invention have been described, additional changes and modifications may be made to these embodiments by those skilled in the art once the basic inventive concepts are known. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention. Obviously, those skilled in the art may make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.< / y>
Claims
1. A method for tokenizing an image, characterized in that: include: Perform time domain-frequency domain conversion on the three channels of the image respectively, and obtain the frequency domain matrix accordingly; quantizing the frequency domain matrix using a quantization table of the same size as the frequency domain matrix to obtain a sparse integer matrix; Convert the sparse integer matrix obtained after quantization into a symbol sequence; The high-frequency co-occurring symbols in the transformed symbol sequence are combined into new symbols to obtain a new symbol sequence, thus completing the tokenization of the image. The step of quantizing the frequency domain matrix using a quantization table of the same size as the frequency domain matrix to obtain a sparse integer matrix comprises: The elements in the frequency domain matrix are divided by the elements in the corresponding position in the quantization table, and the sparse integer matrix is obtained after rounding; The elements in the quantization table are compression coefficients of elements at corresponding positions in the frequency domain matrix; The compression factor is set as follows: The compression coefficient is set according to the type of downstream task scenario. When facing downstream task scenarios such as image understanding or image retrieval, the elements with row numbers greater than 4 and column numbers greater than 4 in the quantization table are set to 200-300; when facing downstream task scenarios such as image storage or image transmission, the elements with row numbers greater than 4 and column numbers greater than 4 in the quantization table are set to 1-200; The step of converting the sparse integer matrix obtained after quantization into a symbol sequence comprises: Each tuple in each row of the sparse integer matrix is mapped to a unique symbol and listed in order from left to right; wherein the tuple consists of a non-zero element value and a vertical coordinate; Arrange all symbols in the order of the rows of the matrix from top to bottom, and use special symbols to mark the end of each row and the beginning and end of each channel to form a symbol sequence.
2. The method for tokenizing an image according to claim 1, wherein: The three channels of the image are respectively converted from time domain to frequency domain, and the corresponding frequency domain matrix includes: Convert the image's RGB format to brightness-chrominance format; The three channels of the brightness-color format are converted from time domain to frequency domain respectively, and the frequency domain matrix is obtained accordingly.
3. The method for tokenizing an image according to claim 1, wherein: The image includes an image converted from a video in a time-sequential frame extraction manner.
4. A device for tokenizing an image, characterized in that: include: The time domain-frequency domain conversion module is used to perform time domain-frequency domain conversion on the three channels of the image respectively, and obtain the corresponding frequency domain matrix; A quantization module, used to quantize the frequency domain matrix using a quantization table of the same size as the frequency domain matrix to obtain a sparse integer matrix; A symbol sequence conversion module, used for converting the sparse integer matrix obtained after quantization into a symbol sequence; The Tokenize module is used to combine the high-frequency co-occurring symbols in the converted symbol sequence into new symbols, obtain a new symbol sequence, and complete the tokenization of the image; The step of quantizing the frequency domain matrix using a quantization table of the same size as the frequency domain matrix to obtain a sparse integer matrix comprises: dividing the elements in the frequency domain matrix by the elements at corresponding positions in the quantization table, and rounding to obtain a sparse integer matrix; the elements in the quantization table are compression coefficients of the elements at corresponding positions in the frequency domain matrix; The compression coefficient is set as follows: the size of the compression coefficient is set according to the category of the downstream task scenario. When facing the downstream task scenario of image understanding or image retrieval, the elements with row numbers greater than 4 and column numbers greater than 4 in the quantization table are set to 200-300; when facing the downstream task scenario of image storage or image transmission, the elements with row numbers greater than 4 and column numbers greater than 4 in the quantization table are set to 1-200; The method of converting the sparse integer matrix obtained after quantization into a symbol sequence includes: mapping each binary group in each row of the sparse integer matrix into a unique symbol and listing them in order from left to right; wherein the binary group consists of a non-zero element value and a vertical coordinate; arranging all the symbols in order from top to bottom of the rows of the matrix, and marking the end of each row and the beginning and end of each channel with special symbols to form a symbol sequence.
5. A memory, characterized in that: A plurality of instructions are stored, and the instructions are used to implement the method for tokenizing an image as described in any one of claims 1 to 3.
6. An electronic device, characterized in that: The invention comprises a processor and a memory connected to the processor, wherein the memory stores a plurality of instructions, and the instructions can be loaded and executed by the processor, so that the processor can execute the method for tokenizing an image as described in any one of claims 1 to 3.
