Vision transformer apparatus and vision transformer method using token merging

Abstract

The present invention relates to a vision transformer device and method using token merging. A vision transformer device according to one embodiment of the present invention may include: an input processing unit that divides a target image for recognition into a plurality of patches and processes one of the plurality of patches as input data in the form of a token; a transformer encoding unit that performs transformer encoding on the input data using a plurality of transformer blocks, and performs token merging on selected tokens based on the similarity between a selected reference token and an adjacent token and the reference similarity among a plurality of tokens that constitute a token matrix based on the input data only in one of the plurality of transformer blocks; and an inference processing unit that performs object recognition inference in the target image for recognition based on the token merging processing result output through the plurality of transformer blocks.

Description

Technology Field

[0001] The present invention relates to a vision transformer device and method using token merging, and more specifically, to a technology that implements accelerated inference related to image recognition and processing based on a vision transformer by utilizing the features of image data to compare similarity between adjacent tokens and performing token merging at once without additional model training. Background Technology

[0002] With the emergence of transformer models that demonstrate powerful performance in identifying and generating sentence structures and semantic relationships, transformer-based models are showing remarkable performance in various fields, including natural language processing, computer vision, and speech recognition.

[0003] In particular, among them, the Vision Transformer (ViT) model demonstrates superior performance in image classification tasks compared to classical CNN models.

[0004] However, despite the excellent accuracy of the Vision Transformer model, the Vision Transformer suffers from the problem of inference time delays caused by its massive model size.

[0005] To address this, research has been proposed to apply token pruning techniques to Vision Transformers, which reduce the amount of computation required by pruning unnecessary tokens within images.

[0006] However, in conventional technology, accuracy decreases because information loss occurs as tokens are removed, and fine-tuning is required to improve model performance.

[0007] Token merging was proposed to address the shortcomings of the token pruning technique.

[0008] In conventional technology, similar tokens are gradually combined in token merging to reduce the total number of tokens and prevent information loss.

[0009] However, there is a limit to the acceleration of inference speed as gradual merging across multiple transformer blocks is required. Prior art literature

[0010] Korean Published Patent No. 10-2023-0149554, "Apparatus and method for normalization of a vision transformer" Korean Published Patent No. 10-2023-0085495, "Image outpainting method and apparatus using image adaptive hint generation" Korean Registered Patent No. 10-2479817, "Vision transformer apparatus for small datasets and method of operation thereof" The problem to be solved

[0011] The present invention aims to provide a vision transformer device and method that implements accelerated inference related to image recognition and processing based on a vision transformer by utilizing the features of image data to compare similarity between adjacent tokens and performing token merging at once without additional model training.

[0012] The present invention aims to provide a vision transformer device and method that improves overall inference speed while maintaining high accuracy by comparing the similarity between adjacent tokens based on one of a plurality of tokens constituting a token matrix in one of a plurality of transformer blocks. means of solving the problem

[0013] A vision transformer device according to one embodiment of the present invention may include an input processing unit that divides a target image for recognition into a plurality of patches and processes one of the plurality of patches into input data in the form of a token; a transformer encoding unit that performs transformer encoding on the input data using a plurality of transformer blocks, and performs token merging on selected tokens based on the similarity between a selected reference token and an adjacent token and the reference similarity among a plurality of tokens that constitute a token matrix based on the input data only in one of the plurality of transformer blocks; and an inference processing unit that performs object recognition inference in the target image for recognition based on the token merging processing result output through the plurality of transformer blocks.

[0014] The transformer encoding processing unit selects at least one similar token based on the similarity between the selected reference token and a token adjacent to at least one of the top, bottom, left, and right sides of the selected reference token and the reference similarity, stores information about the selected similar token, selects at least one additional similar token based on the similarity between the selected at least one similar token and a token adjacent to at least one of the top, bottom, left, and right sides and the reference similarity, stores information about the at least one additional similar token, and merges the selected reference token, the selected at least one similar token, and the selected at least one additional similar token at once to output the token merging processing result through the plurality of transformer blocks.

[0015] The transformer encoding processing unit performs an attention distribution operation on the tokens constituting the token matrix, and based on the calculated attention distribution and the spatial locality of the tokens, compares the similarity between the selected reference token and a token adjacent to at least one of the top, bottom, left, and right of the selected reference token with the reference similarity, and can select a token whose compared similarity is greater than the reference similarity as the at least one similar token.

[0016] The above transformer encoding processing unit can compute the attention distribution by applying a softmax function to the attention scores of the tokens constituting the token matrix.

[0017] The above transformer encoding processing unit can decrease the reference similarity to increase the token merging ratio for the tokens constituting the token matrix, and increase the reference similarity to decrease the token merging ratio.

[0018] The transformer encoding processing unit above can decrease the reference similarity by considering the background image ratio in the recognition target image, and increase the reference similarity by considering the number of multiple identifiable target objects in the recognition target image.

[0019] The above inference processing unit can perform object recognition inference to classify the class of an object in the recognition target image through head processing of a multilayer perceptron based on the reduced number of tokens constituting the token matrix as adjacent backgrounds with uniform color are merged into one token in the result of the token merging process.

[0020] According to one embodiment of the present invention, a vision transformer method may include: a step in an input processing unit of dividing a recognition target image into a plurality of patches and processing one of the plurality of patches into input data in the form of a token; a step in a transformer encoding unit of performing transformer encoding on the input data using a plurality of transformer blocks, wherein only in one of the plurality of transformer blocks, token merging is performed on selected tokens based on the similarity between a selected reference token and an adjacent token and the reference similarity among a plurality of tokens constituting a token matrix based on the input data; and a step in an inference processing unit of performing object recognition inference on the recognition target image based on the token merging processing result output through the plurality of transformer blocks.

[0021] The step of performing token merging on selected tokens based on similarity between a selected reference token and an adjacent token and reference similarity in only one of the plurality of transformer blocks, among the plurality of tokens constituting a token matrix based on the input data, may include: selecting at least one similar token based on similarity between the selected reference token and a token adjacent to at least one of the top, bottom, left, and right of the selected reference token and reference similarity, storing information about the selected similar token, selecting at least one additional similar token based on similarity between the selected at least one similar token and a token adjacent to at least one of the top, bottom, left, and right and reference similarity, storing information about the at least one additional similar token, merging the selected reference token, the selected at least one similar token, and the selected at least one additional similar token at once and outputting the token merging processing result through the plurality of transformer blocks.

[0022] The step of performing token merging on selected tokens based on the similarity between a selected reference token and an adjacent token among a plurality of tokens constituting a token matrix based on the input data in only one of the plurality of transformer blocks may include the step of performing an attention distribution operation on the tokens constituting the token matrix, comparing the similarity between the selected reference token and a token adjacent to at least one of the up, down, left, and right of the selected reference token with the reference similarity based on the calculated attention distribution and the spatial locality of the tokens, and selecting a token whose compared similarity is greater than the reference similarity as the at least one similar token. Effects of the invention

[0023] The present invention can provide a vision transformer device and method that implements accelerated inference related to image recognition and processing based on a vision transformer by utilizing the features of image data to compare similarity between adjacent tokens and performing token merging at once without additional model training.

[0024] The present invention can provide a vision transformer device and method that improves overall inference speed while maintaining high accuracy by comparing the similarity between adjacent tokens based on one of a plurality of tokens constituting a token matrix in one of a plurality of transformer blocks. Brief explanation of the drawing

[0025] FIG. 1 is a drawing illustrating a vision transformer device using token merging according to an embodiment of the present invention. FIG. 2 is a diagram illustrating a token merging block of a transformer encoding unit in a vision transformer device according to an embodiment of the present invention. FIGS. 3 and 4 are drawings illustrating a vision transformer method using token merging according to an embodiment of the present invention. FIGS. 5a to 5e are drawings illustrating a token merging procedure in a transformer encoding unit of a vision transformer device according to an embodiment of the present invention. FIG. 6 is a diagram illustrating an additional token merging procedure in a transformer encoding unit of a vision transformer device according to an embodiment of the present invention. FIG. 7 is a diagram illustrating the result of token merging processing in the transformer encoding unit of a vision transformer device according to an embodiment of the present invention. FIGS. 8 and 9 are drawings illustrating simulation results for inference performance in a vision transformer device according to an embodiment of the present invention. Specific details for implementing the invention

[0026] Hereinafter, various embodiments of this document are described with reference to the attached drawings.

[0027] The embodiments and terms used therein are not intended to limit the technology described in this document to specific embodiments and should be understood to include various modifications, equivalents, and / or substitutions of said embodiments.

[0028] In describing various embodiments below, if it is determined that a detailed description of related known functions or configurations could unnecessarily obscure the essence of the invention, such detailed description will be omitted.

[0029] Furthermore, the terms described below are defined considering their functions in various embodiments, and these may vary depending on the intentions or practices of the user or operator. Therefore, their definitions should be based on the content throughout this specification.

[0030] In relation to the description of the drawings, similar reference numerals may be used for similar components.

[0031] A singular expression may include a plural expression unless the context clearly indicates otherwise.

[0032] In this document, expressions such as "A or B" or "at least one of A and / or B" may include all possible combinations of the items listed together.

[0033] Expressions such as "first," "second," "first," or "second" may modify the corresponding components regardless of order or importance, and are used merely to distinguish one component from another without limiting the components.

[0034] Where it is stated that a certain (e.g., first) component is "(functionally or telecommunicationally) connected" or "connected" to another (e.g., second) component, said certain component may be directly connected to said other component or connected through another component (e.g., third component).

[0035] In this specification, "configured to" may be used interchangeably with, depending on the context, for example, in hardware or software, "suitable for," "capable of," "modified to," "made to," "capable of," or "designed to."

[0036] In some situations, the expression "device configured to..." may mean that the device is "able to..." together with other devices or parts.

[0037] For example, the phrase “a processor configured (or set) to perform A, B, and C” may mean a dedicated processor for performing said operations (e.g., an embedded processor), or a general-purpose processor capable of performing said operations by executing one or more software programs stored in a memory device (e.g., a CPU or an application processor).

[0038] Also, the term 'or' means an inclusive or rather an exclusive or.

[0039] That is, unless otherwise noted or is not clear from the context, the expression 'x uses a or b' means any one of the natural inclusive permutations.

[0040] Terms such as '..bu', '..gi' used below refer to a unit that processes at least one function or operation, and this may be implemented in hardware or software, or a combination of hardware and software.

[0042] FIG. 1 is a drawing illustrating a vision transformer device using token merging according to an embodiment of the present invention.

[0043] FIG. 1 illustrates the components of a vision transformer device using token merging according to an embodiment of the present invention.

[0044] Referring to FIG. 1, a vision transformer device (100) using token merging according to an embodiment of the present invention includes an input processing unit (110), a transformer encoding unit (120), and an inference processing unit (130).

[0045] For example, the vision transformer device (100) is a deep learning model that divides an input image into small pieces, infers an object located within each piece, and recognizes the entire image based on the object inferred from each piece.

[0046] A vision transformer device (100) according to one embodiment of the present invention may be a deep learning model-based device that performs object inference and image recognition without additional model training by reducing the time consumed in token merging based on a framework that merges tokens at once.

[0047] A vision transformer device (100) according to one embodiment of the present invention can reduce the inference time by reducing the time consumed for token merging compared to conventional technology that performs token merging at once before deep learning model inference and performs merging for every transformer block.

[0048] An input processing unit (110) according to an embodiment of the present invention can divide a recognition target image into a plurality of patches and process one of the plurality of patches into input data in the form of a token.

[0049] For example, the input processing unit (110) can divide the image into patches and perform data processing in the form of tokens as input data to be applied to a vision transformer model, which is a deep learning model that performs various image recognition and processing tasks.

[0050] For example, the input processing unit (110) performs input data processing for token merging to reduce the number of tokens and prevent information loss.

[0051] According to one embodiment of the present invention, the transformer encoding unit (120) performs transformer encoding on input data using a plurality of transformer blocks, and can perform token merging on selected tokens based on the similarity between a selected reference token and an adjacent token and the reference similarity among a plurality of tokens that constitute a token matrix based on input data in only one of the plurality of transformer blocks.

[0052] The transformer encoding processing unit (120) can select at least one similar token based on the similarity between a selected reference token and a token adjacent to at least one of the top, bottom, left, and right sides of the selected reference token, and the reference similarity.

[0053] The transformer encoding processing unit (120) stores information about the selected similar token and can select at least one additional similar token based on the similarity between the selected at least one similar token and a token adjacent to at least one of the top, bottom, left, and right, and the reference similarity.

[0054] The transformer encoding processing unit (120) stores information about at least one additional similar token and can output a token merging processing result through multiple transformer blocks by merging a selected reference token, at least one selected similar token, and at least one selected additional similar token at once.

[0055] The transformer encoding processing unit (120) performs an attention distribution operation on the tokens constituting the token matrix, compares the similarity between a selected reference token and a token adjacent to at least one of the top, bottom, left, and right of the selected reference token based on the calculated attention distribution and the spatial locality of the tokens with the reference similarity, and can select at least one similar token whose compared similarity is greater than the reference similarity.

[0056] The transformer encoding processing unit (120) can calculate the attention distribution by applying a softmax function to the attention scores of the tokens constituting the token matrix.

[0057] The transformer encoding processing unit (120) can decrease the reference similarity to increase the token merging ratio for tokens constituting the token matrix, and increase the reference similarity to decrease the token merging ratio.

[0058] The transformer encoding processing unit (120) can reduce the reference similarity by considering the background image ratio in the image to be recognized, and increase the reference similarity by considering the number of multiple identifiable objects in the image to be recognized.

[0059] Unlike conventional technology that performs token merging for each transformer block, the transformer encoding processing unit (120) can reduce the overall time required for merging by performing token merging on only one of the multiple transformer blocks.

[0060] The transformer encoding processing unit (120) can adjust the degree of token merging based on the reference similarity according to the setting, and the smaller this value, the more tokens are merged.

[0061] The fact that adjacent patches in an image are highly likely to have similar backgrounds or features can be utilized.

[0062] Adjacent backgrounds with uniform colors can be merged at once, and based on this, only tokens adjacent to each other in all directions (up, down, left, and right) can be compared to significantly reduce the number of similarity comparisons between tokens.

[0063] A single token merging operation according to one embodiment of the present invention repeatedly performs the process of merging similar tokens after visiting adjacent tokens for all tokens present in a given image.

[0064] In other words, a single token merge recursively repeats the process of comparing the similarity of a given token with four adjacent tokens in all directions (up, down, left, and right).

[0065] The inference processing unit (130) can perform object recognition inference in the recognition target image based on the token merging processing result output through a plurality of transformer blocks.

[0066] The inference processing unit (130) can perform object recognition inference to classify the class of an object in a recognition target image through head processing of a multilayer perceptron based on the reduced number of tokens constituting the token matrix as adjacent backgrounds with uniform colors are merged into one token in the token merging processing result.

[0067] Accordingly, the present invention can provide a vision transformer device and method that implements accelerated inference related to image recognition and processing based on a vision transformer by utilizing the features of image data to compare similarity between adjacent tokens and performing token merging at once without additional model training.

[0069] FIG. 2 is a diagram illustrating a token merging block of a transformer encoding unit in a vision transformer device according to an embodiment of the present invention.

[0070] FIG. 2 more specifically illustrates and explains the procedures performed in the token merging block of the transformer encoding unit in a vision transformer device according to an embodiment of the present invention.

[0071] Referring to FIG. 2, the transformer encoding unit (200) of the present invention includes a first transformer block (210), a second transformer block (220), and a third transformer block (230).

[0072] The number of the first transformer block (210), the second transformer block (220), and the third transformer block (230) can be increased or decreased as an example.

[0073] The transformer encoding unit (200) of the present invention performs token merging only in the first transformer block (210) and provides accuracy similar to and improved inference speed of the prior art, which performs token merging in all of the first transformer block (210), the second transformer block (220), and the third transformer block (230).

[0074] In step (S201), the transformer encoding unit (200) selects a reference token and compares the similarity between the selected reference token and four adjacent tokens based on the reference similarity.

[0075] In step (S202), the transformer encoding unit (200) selects a token among four adjacent tokens whose similarity is higher than the reference similarity.

[0076] In step (S203), the transformer encoding unit (200) additionally selects a reference token as an expansion target among the selected adjacent tokens, and then proceeds with a similarity comparison as in step (S201).

[0077] In step (S204), the transformer encoding unit (200) proceeds with the process of selecting a token among four adjacent tokens that has a similarity higher than the reference similarity, as in step (S202).

[0078] That is, the transformer encoding unit (200) performs a similarity comparison with four adjacent tokens above, below, left, and right based on one token.

[0079] In relation to spatial locality, the fact that adjacent tokens in an image have similar characteristics is utilized.

[0080] Similarity threshold values ​​selected based on the attention distribution are used for similarity comparison.

[0081] In a vision transformer device according to one embodiment of the present invention, the procedures performed in the token merging block of the transformer encoding unit improve the overall inference speed while maintaining high accuracy without additional model training.

[0082] The present invention considers token similarity comparison and spatial locality with a small number of times.

[0083] The transformer encoding unit reduces the time required for token merging by performing token merging on only one of the multiple transformer blocks.

[0084] Preferably, token merging is performed in the first Transformer block.

[0086] FIGS. 3 and 4 are drawings illustrating a vision transformer method using token merging according to an embodiment of the present invention.

[0087] FIG. 3 illustrates a procedure for a vision transformer method using token merging according to an embodiment of the present invention to accelerate a vision transformer.

[0088] Referring to FIG. 3, in step (S301), a vision transformer method using token merging according to an embodiment of the present invention processes input data.

[0089] That is, a vision transformer method using token merging according to an embodiment of the present invention can divide a recognition target image into a plurality of patches and process any one of the plurality of patches as input data in the form of a token.

[0090] In step (S302), the vision transformer method using token merging according to an embodiment of the present invention performs token merging in only one of the plurality of transformer blocks.

[0091] That is, a vision transformer method using token merging according to an embodiment of the present invention performs transformer encoding on input data using a plurality of transformer blocks, and can perform token merging on selected tokens based on the similarity between a selected reference token and an adjacent token and the reference similarity among a plurality of tokens constituting a token matrix based on input data in only one of the plurality of transformer blocks.

[0092] In step (S303), the vision transformer method using token merging according to an embodiment of the present invention performs object recognition inference based on the result of token merging processing.

[0093] That is, the vision transformer method using token merging according to one embodiment of the present invention can perform object recognition inference in a recognition target image based on the token merging processing result output through a plurality of transformer blocks.

[0094] FIG. 4 illustrates a vision transformer method using token merging according to an embodiment of the present invention, which is a token merging at once (ToMato) procedure related to vision transformer acceleration.

[0095] Referring to FIG. 4, in step (S401), a vision transformer method using token merging according to an embodiment of the present invention inputs a token matrix.

[0096] That is, the vision transformer method using token merging according to one embodiment of the present invention can input a token matrix transformed from an input image to be recognized.

[0097] In step (S402), the vision transformer method using token merging according to an embodiment of the present invention computes an attention distribution.

[0098] That is, the vision transformer method using token merging according to one embodiment of the present invention can compute an attention distribution by applying an attention score and a softmax function.

[0099] In step (S403), the vision transformer method using token merging according to an embodiment of the present invention compares the similarity between an adjacent token and a selected reference token.

[0100] That is, the vision transformer method using token merging according to one embodiment of the present invention compares the similarity between the reference token currently being compared and four adjacent tokens.

[0101] In step (S404), the vision transformer method using token merging according to an embodiment of the present invention determines whether the similarity of the comparison tokens is greater or less than the reference similarity.

[0102] That is, the vision transformer method using token merging according to an embodiment of the present invention can determine whether there is a token among adjacent tokens that has a similarity greater than a reference similarity, and if there is a comparison token greater than the reference similarity, proceed to step (S405), and if there is a comparison token less than the reference similarity, proceed to step (S406).

[0103] In step (S405), the vision transformer method using token merging according to an embodiment of the present invention stores comparison token information and selects the comparison token as a reference token.

[0104] That is, the vision transformer method using token merging according to one embodiment of the present invention stores token information when the similarity is greater than the reference similarity.

[0105] In step (S406), the vision transformer method using token merging according to an embodiment of the present invention merges all previously stored tokens.

[0106] That is, the vision transformer method using token merging according to one embodiment of the present invention can store all tokens corresponding to the stored token information, where the similarity is determined to be greater than the reference similarity, at once.

[0107] In step (S407), the vision transformer method using token merging according to an embodiment of the present invention checks whether merging has been completed for all tokens.

[0108] That is, the vision transformer method using token merging according to an embodiment of the present invention determines whether the merging of a specific token is completed based on a reference similarity judgment result, moves to the next token to perform a new merging process, and sequentially determines whether the merging is completed; if the merging is completed, proceeds to step (S408), and if the merging is not completed, proceeds to step (S403).

[0109] In step (S408), the vision transformer method using token merging according to an embodiment of the present invention outputs a token merging processing result.

[0110] That is, the vision transformer method using token merging according to one embodiment of the present invention can output a token merging processing result in which the merging of tokens within the token matrix is ​​completed.

[0111] Accordingly, the present invention can provide a vision transformer device and method that improves the overall inference speed while maintaining high accuracy by comparing the similarity with adjacent tokens based on one of the tokens constituting a token matrix in one of the transformer blocks among the plurality of transformer blocks.

[0113] FIGS. 5a to 5e are drawings illustrating a token merging procedure in a transformer encoding unit of a vision transformer device according to an embodiment of the present invention.

[0114] Referring to FIG. 5a, a token matrix (500) and an attention distribution (501) for the token matrix (500) are shown.

[0115] With respect to the token matrix (500), an attention distribution (501) is calculated as a distribution of the results of attention execution among tokens excluding class tokens and distillation tokens after multi-head attention is finished.

[0116] Referring to FIG. 5b, the similarity comparison between adjacent tokens starting from the i-th token, which is the reference token in the token matrix (510), is performed in the attention distribution (511).

[0117] The attention distribution (511) shows that the solid line is similar and the dotted line is different.

[0118] Referring to FIG. 5c, the result of selecting adjacent tokens from the i-th token, which is the reference token in the token matrix (520), excluding the token (i+14), is shown.

[0119] The attention distribution (521) shows that the token corresponding to the dotted line is not selected in the token matrix (520).

[0120] That is, it can be confirmed that only tokens exceeding the sim (similarity threshold which affects merging rate) value corresponding to the similarity threshold in the token matrix (520) and attention distribution (521) are selected.

[0121] For example, in relation to similarity comparison, the similarity criterion value may be set to 0.1e-7.

[0122] Referring to FIG. 5d, the token matrix (530) and attention distribution (531) show recursive similarity comparison based on the newly merged tokens.

[0123] Referring to FIG. 5e, the token matrix (540) and the attention distribution (541) show that token information is stored after recursively comparing similarity based on the newly merged tokens.

[0124] The token merging procedure according to the images and tables in Figures 5a to 5e is explained in an integrated manner.

[0125] A token merging procedure according to one embodiment of the present invention starts token merging immediately after multi-head attention is performed in the first decoder, and proceeds with a procedure to determine the token to be merged for four adjacent tokens in the up, down, left, and right directions.

[0126] Repeat the same process of merging 4 adjacent tokens based on the tokens to be merged.

[0127] After recursively finding similar tokens, when no more similar tokens can be found among adjacent tokens, all found similar tokens are merged into one.

[0128] The recursive token merging process starts from the first token and proceeds to the last token.

[0130] FIG. 6 is a diagram illustrating an additional token merging procedure in a transformer encoding unit of a vision transformer device according to an embodiment of the present invention.

[0131] FIG. 6 illustrates an additional token merging procedure in the transformer encoding unit of a vision transformer device according to an embodiment of the present invention using an image.

[0132] Referring to FIG. 6, the image (600) is expanded to an adjacent token in part (601) and merged as in part (611) in image (610).

[0133] As the token moves to the next image (620), a new merging process is performed in the part (621).

[0134] When there are no longer any adjacent tokens that exceed the similarity threshold, the stored tokens are merged into a single token, and then the process of merging is repeated to the next token. If the merging process is already completed, the process proceeds to the next part (621).

[0135] The merger is determined to be complete from the image (620) to the part (621), and the merger process for the entire token is performed by merging from the image (630) to the part (631).

[0136] Regarding the merging of all tokens, the token merging processing result is provided as an actual final result image (640).

[0138] FIG. 7 is a diagram illustrating the result of token merging processing in the transformer encoding unit of a vision transformer device according to an embodiment of the present invention.

[0139] FIG. 7 illustrates the result of token merging processing in the transformer encoding unit of a vision transformer device according to an embodiment of the present invention.

[0140] Referring to FIG. 7, an input recognition target image (700) is shown, and a token merging processing result (710) is shown that is output by applying the recognition target image (700) to the present invention.

[0141] The result of the token merging process (710) represents the result of effectively merging the patches in semantics.

[0143] FIGS. 8 and 9 are drawings illustrating simulation results for inference performance in a vision transformer device according to an embodiment of the present invention.

[0144] FIG. 8 compares the latency and accuracy with the prior art in relation to the simulation results of inference performance in a vision transformer device according to an embodiment of the present invention.

[0145] Referring to FIG. 8, the graph (800) shows a guide line (801) related to the present invention and a guide line (802) related to the prior art.

[0146] When comparing the indicator line (801) and the indicator line (802), it can be confirmed that the present invention has a minor decrease in accuracy due to the reduction in delay speed.

[0147] That is, it can be seen that the indicator line (802) has a large range of change in accuracy according to the inference time, while the indicator line (801) does not have a large change in accuracy even when the inference time is reduced.

[0148] FIG. 9 illustrates the value of a similar reference figure and the latency related to the inference speed in relation to the simulation results for inference performance in a vision transformer device according to an embodiment of the present invention.

[0149] Referring to FIG. 9, the graph (900) shows the relationship between the value of a similar reference map and the delay speed related to the inference speed with the indicator line (901) related to the present invention.

[0150] That is, the graph (900) shows that the inference speed increases as the similarity threshold value decreases.

[0151] In relation to the similarity threshold value, the more uniform the background or features the image has, the higher the merging rate.

[0152] In relation to the similarity threshold value, the more important or complex features an image has, the lower the merging ratio becomes.

[0153] For example, the present invention sets the similarity criterion value to 10 -7 Set to decrease the total number of tokens from approximately 8.16% to 20.92%.

[0154] When the value representing the number of tokens merged in each transformer block is set to "2", approximately 15% of the tokens are ultimately merged.

[0155] Accordingly, the present invention can flexibly set the delay speed and merging range based on the setting of the similarity reference value.

[0157] The device described above may be implemented as a hardware component, a software component, and / or a combination of a hardware component and a software component. For example, the device and components described in the embodiments may be implemented using one or more general-purpose or special-purpose computers, such as, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable array (FPA), a programmable logic unit (PLU), a microprocessor, or any other device capable of executing and responding to instructions. The processing unit may execute an operating system (OS) and one or more software applications executed on said operating system. Additionally, the processing unit may access, store, manipulate, process, and generate data in response to the execution of the software. For ease of understanding, the processing unit may be described as being used as a single unit, but those skilled in the art will understand that the processing unit may include multiple processing elements and / or multiple types of processing elements. For example, the processing unit may include multiple processors or one processor and one controller. In addition, other processing configurations, such as parallel processors, are also possible.

[0158] Software may include computer programs, code, instructions, or a combination of one or more of these, and may configure a processing unit to operate as desired or command the processing unit independently or collectively. Software and / or data may be permanently or temporarily embodied in any type of machine, component, physical device, virtual equipment, computer storage medium or device, or transmitted signal wave so as to be interpreted by the processing unit or to provide instructions or data to the processing unit. Software may be distributed over networked computer systems and may be stored or executed in a distributed manner. Software and data may be stored on one or more computer-readable recording media.

[0159] Although the embodiments have been described above with reference to the limited drawings, those skilled in the art can make various modifications and variations from the description above. For example, suitable results can be achieved even if the described techniques are performed in a different order than described, and / or the components of the described system, structure, device, circuit, etc. are combined or assembled in a form different from described, or replaced or substituted by other components or equivalents.

[0160] Therefore, other implementations, other embodiments, and equivalents to the claims also fall within the scope of the claims set forth below. Explanation of the symbols

[0161] 100: Vision Transformer Device 110: Input Processing Unit 120: Transformer Encoding Unit 130: Inference processing unit

Claims

Claim 1 A vision transformer device comprising: an input processing unit that divides a recognition target image into a plurality of patches and processes one of the plurality of patches into input data in the form of a token; a transformer encoding unit that performs transformer encoding on the input data using a plurality of transformer blocks, and performs token merging on selected tokens based on the similarity between a selected reference token and an adjacent token and the reference similarity among a plurality of tokens constituting a token matrix based on the input data only in one of the plurality of transformer blocks; and an inference processing unit that performs object recognition inference in the recognition target image based on the token merging processing result output through the plurality of transformer blocks, wherein the transformer encoding processing unit sets the reference similarity to decrease so that the token merging ratio in a single token merging of tokens constituting the token matrix increases by considering the background image ratio in the recognition target image, and sets the reference similarity to increase so that the token merging ratio in a single token merging of tokens constituting the token matrix decreases by considering the number of multiple identifiable target objects in the recognition target image. Claim 2 A vision transformer device according to claim 1, wherein the transformer encoding processing unit selects at least one similar token based on the similarity between the selected reference token and a token adjacent to at least one of the top, bottom, left, and right of the selected reference token and the reference similarity, stores information about the selected similar token, selects at least one additional similar token based on the similarity between the selected at least one similar token and a token adjacent to at least one of the top, bottom, left, and right and the reference similarity, stores information about the at least one additional similar token, and merges the selected reference token, the selected at least one similar token, and the selected at least one additional similar token at once to output the token merging processing result through the plurality of transformer blocks. Claim 3 A vision transformer device according to claim 2, wherein the transformer encoding processing unit performs an attention distribution operation on tokens constituting the token matrix, compares the similarity between the selected reference token and a token adjacent to at least one of the top, bottom, left, and right of the selected reference token with the reference similarity based on the calculated attention distribution and the spatial locality of the tokens, and selects the token whose compared similarity is greater than the reference similarity as the at least one similar token. Claim 4 A vision transformer device according to claim 3, wherein the transformer encoding processing unit calculates the attention distribution by applying a softmax function to the attention scores of the tokens constituting the token matrix. Claim 5 delete Claim 6 delete Claim 7 A vision transformer device according to claim 1, wherein the inference processing unit reduces the number of tokens constituting the token matrix as adjacent backgrounds having uniform colors are merged into one token in the result of the token merging process, and performs object recognition inference to classify the class of an object in the recognition target image through head processing of a multilayer perceptron based on the reduced tokens. Claim 8 A vision transformer method performed by one or more processors controlling the operation of an input processing unit, a transformer encoding unit, and an inference processing unit, comprising: a step in which, in the input processing unit, a recognition target image is divided into a plurality of patches and one of the plurality of patches is processed as input data in the form of a token; and a step in which, in the transformer encoding unit, transformer encoding is performed on the input data using a plurality of transformer blocks, wherein token merging is performed on selected tokens based on the similarity between an adjacent token and a selected reference token among a plurality of tokens constituting a token matrix based on the input data, and the reference similarity, only in one of the plurality of transformer blocks. A vision transformer method comprising: a step of performing object recognition inference in the recognition target image based on the token merging processing result output through the plurality of transformer blocks in the inference processing unit, wherein the step of performing token merging for selected tokens based on the similarity between a selected reference token and an adjacent token and the reference similarity among a plurality of tokens constituting a token matrix based on the input data in only one of the plurality of transformer blocks; a step of setting the reference similarity to decrease so that the token merging ratio in a single token merging of the tokens constituting the token matrix is ​​increased by considering the background image ratio in the recognition target image; and a step of setting the reference similarity to increase so that the token merging ratio in a single token merging is decreased by considering the number of multiple identifiable target objects in the recognition target image. Claim 9 In claim 8, the step of performing token merging on tokens selected based on similarity between a selected reference token and an adjacent token and reference similarity among a plurality of tokens constituting a token matrix based on the input data in only one of the plurality of transformer blocks comprises: selecting at least one similar token based on similarity between the selected reference token and a token adjacent to at least one of the top, bottom, left, and right of the selected reference token and reference similarity, storing information about the selected similar token, selecting at least one additional similar token based on similarity between the selected at least one similar token and a token adjacent to at least one of the top, bottom, left, and right and reference similarity, storing information about the at least one additional similar token, merging the selected reference token, the selected at least one similar token, and the selected at least one additional similar token at once and outputting the token merging processing result through the plurality of transformer blocks. Claim 10 In claim 9, the step of performing token merging on tokens selected based on the similarity between a selected reference token and an adjacent token among a plurality of tokens constituting a token matrix based on the input data in only one of the plurality of transformer blocks comprises: performing an attention distribution operation on the tokens constituting the token matrix; comparing the similarity between the selected reference token and a token adjacent to at least one of the up, down, left, and right of the selected reference token with the reference similarity based on the calculated attention distribution and the spatial locality of the tokens; and selecting a token whose compared similarity is greater than the reference similarity as the at least one similar token.

Images