Image recognition performance optimization method and system

Abstract

The present invention relates to an image recognition performance optimization method and system, and provides an image recognition performance optimization method and system, in which a visual token is re-tokenized in units of semantics in order to improve image recognition efficiency.

Description

Image Recognition Performance Optimization Method and System

[0001] The present invention relates to a method and system for optimizing image recognition performance, and provides a method and system for optimizing image recognition performance that re-tokenizes visual tokens into semantic units to improve the efficiency of image recognition.

[0002] Tokenization is the process of dividing text into small units, such as words (e.g., tokens), and is an essential preprocessing step in natural language processing (NLP). For example, modern NLP models such as BERT (Bidirectional Encoder Representations from Transformers), GPT (Generative Pre-trained Transformer), and XLNet (eXtreme Language Net) tokenize text into subword units.

[0003] These subword units maintain a balance between words and characters to preserve linguistic meaning, while minimizing vocabulary scarcity issues even with relatively small amounts of vocabulary. For example, subword tokenizers such as WordPiece use a greedy longest-match-first strategy to repeatedly select the longest prefix from the remaining text that matches a vocabulary token. This method is called Maximum Matching or MaxMatch.

[0004] Building on the pioneering success of Transformers in the field of Natural Language Processing (NLP), Vision Transformers (ViTs) have demonstrated remarkable performance in various computer vision tasks. For images, tokenization involves dividing the image into non-overlapping grid patches and inputting a sequence of linear embeddings of these patches into the Transformer. In this process, the image patches are treated as tokens, similar to words in natural language processing.

[0005] However, unlike tokens derived from text subwords, which mostly have meaning and contribute to contextual understanding, image patch-based tokens are significantly different because i) they often contain meanings that are completely unrelated to the overall meaning (e.g., road, sky, background, etc.), or ii) they require a broader context to convey meaning.

[0006] This phenomenon becomes a more serious problem in various subsequent tasks aimed at improving the efficiency of the Vision Transformer, namely, the task of reducing the number of input tokens. Token reduction in the Vision Transformer is primarily performed through two main methods. For example, these may include i) token pruning, a method of removing tokens with low attention, and ii) token merging, a method of merging duplicate or similar tokens into a single new abstract token. Both of these methods contribute to effectively reducing the computational complexity of the Vision Transformer.

[0007] However, due to the limitations of patch-based tokenization, this method may i) remove tokens too early before their meaning is sufficiently contextualized, or ii) prematurely combine semantic tokens based on similarity with surrounding non-semantic noise, resulting in performance degradation that is more severe than simple token pruning in some scenes.

[0008] Therefore, there is still a need for methods to efficiently process image data and improve image recognition performance.

[0009] The present invention is intended to provide an image recognition performance optimization method and system capable of improving image recognition performance by efficiently processing image data.

[0010] Furthermore, the present invention aims to provide a method and system for optimizing image recognition performance that solves the problem of information loss during the process of processing image data and preserves important information.

[0011] Furthermore, the present invention aims to provide a method and system for optimizing image recognition performance that can improve the expressiveness of visual information in an artificial intelligence model and effectively reduce computational load.

[0012] In particular, the present invention aims to provide a method and system for optimizing image recognition performance that can simultaneously improve the efficiency and accuracy of Vision Transformers (ViTs).

[0013] Furthermore, the present invention aims to provide a method and system for optimizing image recognition performance that can improve efficiency and productivity in industrial settings.

[0014] To solve the problem described above, the image recognition performance optimization method according to the present invention, which is performed by a computer, may include the steps of receiving at least one input image; dividing the at least one input image into a plurality of patches of a predetermined size; obtaining a plurality of tokens by performing a transformation on the plurality of patches; calculating an attention score for each of the plurality of tokens to evaluate the importance of each of the plurality of tokens; performing grouping by dividing at least some of the plurality of tokens into a first group and a second group, respectively, based on the importance of each of the evaluated plurality of tokens; merging at least one token included in the first group and at least one token included in the second group to generate at least one merged token; re-evaluating the importance of the at least one merged token and the importance of at least some of the tokens among the plurality of tokens; and selecting at least one of the at least one merged token and at least some of the tokens based on the re-evaluation result.

[0015] In an embodiment, in the step of re-evaluating importance, the attention score of the at least one merged token and the attention score of the at least a portion of tokens are calculated to re-evaluate the importance of the at least one merged token and the importance of the at least a portion of tokens, and in the step of selecting, based on the re-evaluation result, at least one token to be retained and at least one token to be removed can be selected among the at least one merged token and the at least a portion of tokens.

[0016] In an embodiment, the method further includes the step of applying a local inductive bias to at least some of the plurality of patches using at least one specific module, and the plurality of tokens may include at least some tokens obtained by performing a transformation on at least some of the patches to which the local inductive bias has been applied.

[0017] In an embodiment, in the step of applying the local inductive bias, the local inductive bias may be applied to the at least some patches such that, in the at least one specific module, the at least some patches that are positionally adjacent among the plurality of patches have high similarity.

[0018] In an embodiment, the step of applying the local inductive bias may be a step of applying the local inductive bias to at least some of the positionally adjacent patches among the plurality of patches in at least one specific module to enhance the feature similarity between at least some of the positionally adjacent tokens among the plurality of tokens.

[0019] In an embodiment, the plurality of patches includes at least one of at least one patch having a first attribute and at least one patch having a second attribute, and in the step of imparting the local inductive bias, at least some of the non-meaning patches that are positionally adjacent among the non-meaning patches can be induced to have a high similarity through the at least one specific module.

[0020] In an embodiment, at least one patch having the first attribute may include at least one semantic patch, and at least one patch having the second attribute may include at least one non-semantic patch.

[0021] In an embodiment, in the step of evaluating the importance of each of the plurality of tokens, the importance of each of the plurality of tokens can be evaluated based on the attension score calculated for each of the plurality of tokens.

[0022] In an embodiment, in the step of performing the grouping, at least some of the tokens to be grouped among the plurality of tokens are identified based on the importance of each of the evaluated plurality of tokens, and the identified at least some of the tokens to be grouped are classified into the first group and the second group, respectively, to perform the grouping; and in the step of generating the at least one merged token, each token included in the first group is merged with a semantically similar token within the second group to generate the at least one merged token.

[0023] In an embodiment, the at least some of the grouping target tokens may include at least some of the tokens with low importance among the plurality of tokens.

[0024] In an embodiment, each token included in the first group can be merged with a semantically similar token in the second group through a pre-set matching technique.

[0025] In an embodiment, the at least some tokens may include at least some tokens among the plurality of tokens for which grouping has not been performed, and at least one of the remaining tokens among the tokens included in each of the first group and the second group that have not been merged.

[0026] In an embodiment, the selection step involves selecting, based on the re-evaluation result, at least one token to be retained and at least one token to be removed, each having at least one attribute among at least one merged token and at least one part of the tokens, and the at least one attribute may include at least one of a first attribute and a second attribute.

[0027] In the embodiment, in the selection step, based on the re-evaluation result, among the at least one merged token and the at least one partial token, the at least one token to be retained having the first attribute and the at least one token to be removed having the second attribute can be selected respectively.

[0028] In an embodiment, the at least one retention target token having the first attribute may include at least one semantic token, and the at least one retention target token having the second attribute may include at least one non-semantic token.

[0029] In an embodiment, the step of maintaining at least one retention target token having the first attribute and removing at least one removal target token having the second attribute may be further included.

[0030] In an embodiment, the at least one specific module may be composed of at least one overlapping convolution layer.

[0031] An image recognition performance optimization system according to the present invention, comprising a memory configured to store executable instructions and one or more processors configured to perform operations by executing one or more instructions, receives at least one input image, divides the at least one input image into a plurality of patches of a predetermined size, performs a transformation on the plurality of patches to obtain a plurality of tokens, calculates an attention score for each of the plurality of tokens to evaluate the importance of each of the plurality of tokens, performs grouping by dividing at least some of the plurality of tokens into a first group and a second group respectively based on the evaluated importance of each of the plurality of tokens, merges at least one token included in the first group and at least one token included in the second group to generate at least one merged token, re-evaluates the importance of the at least one merged token and the importance of at least some of the tokens among the plurality of tokens, and selects at least one of the at least one merged token and at least some of the tokens based on the re-evaluation result.

[0032] A program according to the present invention is a program that is executed by one or more processes in an electronic device and can be stored on a computer-readable recording medium, and may include instructions for performing the following steps: receiving at least one input image; dividing the at least one input image into a plurality of patches of a predetermined size; performing a transformation on the plurality of patches to obtain a plurality of tokens; calculating an attention score for each of the plurality of tokens to evaluate the importance of each of the plurality of tokens; performing grouping by dividing at least some of the plurality of tokens into a first group and a second group, respectively, based on the importance of each of the evaluated plurality of tokens; merging at least one token included in the first group and at least one token included in the second group to generate at least one merged token; re-evaluating the importance of the at least one merged token and the importance of at least some of the tokens among the plurality of tokens; and selecting at least one of the at least one merged token and at least some of the tokens based on the re-evaluation result.

[0033] As described above, according to the image recognition performance optimization method and system of the present invention, by re-tokenizing visual tokens into semantic units, non-semantic tokens that were previously judged to be meaningless can be merged (or combined) with surrounding information to acquire meaning. Through this, the present invention not only preserves tokens containing important information but also improves the overall performance of the model. That is, the present invention allows non-semantic tokens to be merged into a single meaningful token when they have the potential to have contextual meaning, thereby effectively reducing computational load while maintaining the expressiveness of the model.

[0034] Furthermore, according to the image recognition performance optimization method and system of the present invention, information loss can be minimized by reconstructing (or re-tokenizing) meaningless tokens (i.e., non-meaningful tokens) in the initial stage based on the importance of each of a plurality of tokens. That is, by reconstructing meaningless tokens in the initial stage, the present invention can prevent the problem in which potentially important patches are considered unimportant tokens and removed in the initial stage.

[0035] Furthermore, according to the image recognition performance optimization method and system of the present invention, a local inductive bias can be applied to positionally adjacent patches using a specific module so that positionally adjacent patches among a plurality of patches have high similarity. Through this, the present invention enables effective merging of meaningless tokens. That is, by strengthening the feature similarity between positionally close tokens using a specific module, the present invention enables non-meaningful tokens to be merged more naturally and meaningfully within an actual visual scene.

[0036] Furthermore, according to the image recognition performance optimization method and system of the present invention, non-meaningful tokens can be re-tokenized into meaningful units, and tokens to be retained and / or removed can be selected through re-evaluation. Through this, the present invention can improve the performance of the model while reducing information loss. That is, the present invention can enhance the expressiveness of the model's visual information and effectively reduce the amount of computation.

[0037] FIG. 1 illustrates an example of a block diagram of a computing system in which the present invention can be implemented.

[0038] FIG. 2 illustrates an example of a block diagram of a computing device that may be included in a user computing device, a server computing system, and a training computing system, as an embodiment of a computing system in which the present invention can be implemented.

[0039] Figure 3 illustrates an example of a block diagram from another perspective of a computing device, which is one of the components of a computing system.

[0040] FIG. 4 is a conceptual diagram illustrating an image recognition performance optimization system according to the present invention.

[0041] FIG. 5 is a flowchart illustrating an image recognition performance optimization method according to the present invention.

[0042] FIGS. 6, FIGS. 7, FIGS. 8, and FIGS. 9 are conceptual diagrams and formulas for explaining an image recognition performance optimization method according to the present invention.

[0043] FIGS. 10 and 11 illustrate an example of the performance evaluation results of an image recognition performance optimization method and system according to the present invention.

[0044] FIGS. 12 and FIGS. 13 are conceptual diagrams for explaining another embodiment of an image recognition performance optimization method and system according to the present invention.

[0045] Hereinafter, embodiments disclosed in this specification will be described in detail with reference to the attached drawings. Identical or similar components are assigned the same reference number regardless of the drawing symbols, and redundant descriptions thereof will be omitted. The suffixes "module" and "part" used for components in the following description are assigned or used interchangeably solely for the ease of drafting the specification and do not have distinct meanings or roles in themselves. Furthermore, in describing the embodiments disclosed in this specification, if it is determined that a detailed description of related prior art could obscure the essence of the embodiments disclosed in this specification, such detailed description will be omitted. Additionally, the attached drawings are intended only to facilitate understanding of the embodiments disclosed in this specification; the technical concept disclosed in this specification is not limited by the attached drawings, and it should be understood that they include all modifications, equivalents, and substitutions that fall within the spirit and technical scope of the present invention.

[0046] Terms including ordinal numbers, such as first, second, etc., may be used to describe various components, but said components are not limited by said terms. These terms are used solely for the purpose of distinguishing one component from another.

[0047] When it is stated that one component is "connected" or "connected" to another component, it should be understood that while it may be directly connected or connected to that other component, there may also be other components in between. On the other hand, when it is stated that one component is "directly connected" or "directly connected" to another component, it should be understood that there are no other components in between.

[0048] A singular expression includes a plural expression unless the context clearly indicates otherwise.

[0049] In this application, terms such as “comprising” or “having” are intended to specify the existence of the features, numbers, steps, actions, components, parts, or combinations thereof described in the specification, and should be understood as not precluding the existence or addition of one or more other features, numbers, steps, actions, components, parts, or combinations thereof.

[0050] Hereinafter, the present invention will be examined in more detail with reference to the attached drawings. FIG. 1 illustrates an example of a block diagram of a computing system in which the present invention can be implemented. FIG. 2 illustrates an example of a block diagram of a computing device that may be included in a user computing device, a server computing system, and a training computing system as an embodiment of a computing system in which the present invention can be implemented. FIG. 3 illustrates an example of a block diagram of a computing device in another aspect that is one of the components of a computing system.

[0051] Furthermore, FIG. 4 is a conceptual diagram for explaining an image recognition performance optimization system according to the present invention. FIG. 5 is a flowchart for explaining an image recognition performance optimization method according to the present invention, and FIG. 6, FIG. 7, FIG. 8 and FIG. 9 are conceptual diagrams and formulas for explaining an image recognition performance optimization method according to the present invention.

[0052] Meanwhile, FIG. 1 illustrates an example of a block diagram of a computing system in which the present invention can be implemented. In this regard, the image recognition performance optimization system according to the present invention can be implemented through a computing device described below and can perform data processing related to the image recognition performance optimization method described in this specification.

[0053] Referring to FIG. 1, a computing system (10000) that performs a method of re-tokenizing visual tokens into semantic units and selecting tokens to be retained and / or removed through re-evaluation according to one embodiment of the present invention may include at least one computing device. At this time, the at least one computing device may be a single processor or a multi-processor computing device.

[0054] The components of at least one computing device of the present invention may include various hardware components such as one or more processors, memory, other hardware, and a system bus (not shown) that connects various system components so that they can transmit and receive data to and from each other (e.g., telecommutatively connected, physically connected, electrically connected), and the components of at least one computing device are not limited thereto and may be very diverse.

[0055] Meanwhile, at least one computing device included in a computing system (10000) that performs a method of re-tokenizing (or reconstructing) visual tokens in semantic units and selecting tokens to be retained and / or removed through re-evaluation may be connected to communicate via a network (1070). For example, at least one computing device included in the computing system (10000) may be clustered or part of a local area network (LAN). Additionally, at least one computing device may be part of a wide area network (WAN) or connected to at least one of a client-server network and a peer-to-peer network within the cloud.

[0056] Meanwhile, when at least one computing device is used in at least one of a network environment and a cloud computing environment, the at least one computing device may be connected to at least one of a public and private network through a network interface or adapter. In one embodiment, other communication connection devices, such as a modem, may be used to establish communication through the network. The modem may be at least one of an internal modem and an external modem, and may be connected to a system bus through a network interface or a specific mechanism, etc. A wireless network component consisting of an interface and an antenna may be coupled to the network through a device such as an access point, a peer computer, etc. In the present invention, the method of connecting at least one computing device to communicate through the network (1070) is not limited, and it may be connected to communicate in a manner different from the described example.

[0057] Furthermore, other computer-type devices and / or systems not shown in FIG. 1 may also interact technically with at least one computing device or other system through one or more connections to the network (1070) via a network interface. Here, the network interface may include network interface equipment such as a physical network interface controller (NIC) or a virtual network interface (VIF).

[0058] The network (1070) of the present invention may include various forms such as the Internet, WLAN (Wireless LAN), Wi-Fi (Wireless-Fidelity), Wi-Fi (Wireless Fidelity) Direct, DLNA (Digital Living Network Alliance), WiBro (Wireless Broadband), WiMAX (World Interoperability for Microwave Access), HSDPA (High Speed ​​Downlink Packet Access), HSUPA (High Speed ​​Uplink Packet Access), LTE (Long Term Evolution), LTE-A (Long Term Evolution-Advanced), 5G (5th Generation Mobile Telecommunication), Bluetooth (Bluetooth™ RFID (Radio Frequency Identification), Infrared Data Association (IrDA), UWB (Ultra-Wideband), ZigBee, NFC (Near Field Communication), Wi-Fi Direct, Wireless USB (Wireless Universal Serial Bus), etc., and in the present invention, data transmission may be performed based on standard communication protocols such as TCP / IP, HTTP, SSL, etc.

[0059] A computing system (10000) that performs a method of re-tokenizing visual tokens into semantic units according to the present invention and selecting tokens to be maintained and / or removed through re-evaluation may include at least one of a user computing device (1010), a training computing system (1050), and a server computing system (1030).

[0060] A user computing device (1010) according to the present invention may be understood as a computing device comprising at least one processor (1011) and at least one memory (1012) that perform a method of re-tokenizing visual tokens in semantic units and selecting tokens to be retained and / or removed through re-evaluation. For example, the user computing device (1010) may include at least one computing device among a smartphone, a smart TV, a laptop computer, a desktop computer, a digital broadcasting terminal, a PDA (personal digital assistants), a PMP (portable multimedia player), a navigation device, a slate PC, a tablet PC, an ultrabook, a wearable device (e.g., a smartwatch, a smart glass, and a head-mounted display).

[0061] At least one and / or at least one processor (1011) constituting the user computing device (1010) may include one or more general-purpose processors and / or one or more special-purpose processors. For example, at least one and / or at least one processor (1011) constituting the user computing device (1010) may be composed of at least one or a plurality of electrically connected processors among a central processing unit (CPU), a graphics processing unit (GPU), a tensor processing unit (TPU), a neural network processing unit (NPU), an arithmetic logic unit (ALU), a floating-point arithmetic unit (FPU), an application integrated circuit, an application semiconductor (ASIC), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, and / or other electrical units for performing functions.

[0062] Furthermore, at least one and / or at least one processor (1011) may be configured to execute computer-readable instructions contained in memory (1012) and / or other instructions described herein.

[0063] The memory (1012) constituting the user computing device (1010) according to the present invention may include volatile memory, non-volatile memory, fixed media, removable media, magnetic media, optical media, semiconductor media and / or other types of physically durable storage media.

[0064] For example, memory (1012) may include one or more non-transient / transient computer-readable storage media such as RAM, ROM, HDD (Hard Disk Drive), SSD (Solid State Disk), SSD (Silicon Disk Drive), EEPROM, EPROM, flash memory device, magnetic disk, etc., and may include web storage of a server that performs the storage function of memory on the internet. Such memory (1012) may store data and instructions necessary for the operation of an application for the at least one and / or at least one processor (1011) to re-tokenize visual tokens in semantic units and to select tokens to be retained and / or removed through re-evaluation.

[0065] A user computing device (1010) may include one or more user input components (1021) that detect user input. For example, the user input component (1021) may also be referred to as a user interface module. The user input component (1021) may include a touch screen, a computer mouse, a keyboard, a keypad, a touchpad, a trackball, a joystick, a voice recognition module, or other similar devices. However, the present invention does not limit the type of user input component (1021). In this case, the user input component (1021) in the present invention does not necessarily mean a hardware means, but can be understood as a channel for receiving input from a user. Meanwhile, the user of the present invention may refer to an automated agent, script, playback software, etc., that operates on behalf of one or more people.

[0066] A user can interact with a computing system (10000) including at least one computing device through input text, touch, voice, movement, computer vision, gestures and / or other forms of input / output using a user input component (1021). For example, the user input component (1021) may include one or more of a command line interface (CLI), a graphical user interface (GUI), a natural user interface (NUI), a voice command interface and / or other user interface (UI) representations.

[0067] Between the user input component (1021) and the user computing device (1010), one or more application programming interface (API) calls may be made based on user input received from the user interface and / or network.

[0068] Here, the expression "based on" may be interpreted to include cases where it is based on the use of a specific configuration, modified from, derived from, influenced by, dependent on, or otherwise derived from a specific configuration. In some embodiments, an API call may be configured for a specific API, which may be interpreted or converted into an API call configured for another API. Here, an API may refer to a defined interface or connection between computers or between computer programs.

[0069] In one embodiment, the user computing device (1010) may store at least one machine learning model (1020). For example, the user computing device (1010) may be various machine learning models, such as a plurality of neural networks (e.g., deep neural networks) that perform the process of re-tokenizing visual tokens obtained by processing images into semantic units and selecting tokens to be retained and / or removed through re-evaluation, or other types of machine learning models including non-linear models and / or linear models, and may be composed of a combination thereof.

[0070] According to an embodiment of the present invention, a user computing device (1010) may perform a method of re-tokenizing visual tokens in semantic units and selecting tokens to be retained and / or removed through re-evaluation using a local or / and external machine learning model (1020). Alternatively, the user computing device (1010) may perform a method of re-tokenizing visual tokens in semantic units and selecting tokens to be retained and / or removed through re-evaluation using a machine learning model (1040) provided by a server.

[0071] Additionally, according to another embodiment of the present invention, a server computing system (1030) communicating with a user computing device (1010) may provide information regarding a token to be retained and / or a token to be removed to the user computing device (1010) on an application or / and the web, in accordance with a request from a user received through the user computing device (1010).

[0072] In addition, according to another embodiment of the present invention, at least a part of a user computing device (1010) and a server computing system (1030) are linked together to re-tokenize visual tokens in semantic units and perform a method of selecting tokens to be maintained and / or removed through re-evaluation, thereby providing information to the user regarding tokens to be maintained and / or tokens to be removed.

[0073] Additionally, according to various embodiments of the present invention, a user computing device (1010) and / or a server computing system (1030) may learn a machine learning model (1020, 1040) performed in a method of re-tokenizing visual tokens in semantic units and selecting tokens to be retained and / or removed through re-evaluation by interacting with a training computing system (1050) that is communicated via a network (1070). In this case, the training computing system (1050) may be a computing system separate from the server computing system (1030). Alternatively, in some embodiments, the training computing system (1050) may be part of the server computing system (1030) or part of the user computing device (1010).

[0074] Meanwhile, the server computing system (1030) may include at least one processor (1031) and memory (1032). Here, the processor (1031) may be composed of at least one or a plurality of electrically connected processors among a central processing unit (CPU), a graphics processing unit (GPU), a tensor processing unit (TPU), a neural network processing unit (NPU), an application integrated circuit, an application semiconductor (ASIC), an arithmetic logic unit (ALU), a floating-point arithmetic unit (FPU), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, and / or other electrical units for performing functions. For example, at least one processor (1031) may include a circuit and a transistor configured to execute instructions from memory (1032).

[0075] The memory (1032) constituting the server computing system (1030) according to the present invention may include volatile memory, non-volatile memory, fixed media, removable media, magnetic media, optical media, semiconductor media, and / or other types of physically durable storage media. For example, the memory (1032) may include one or more non-transient / transient computer-readable storage media such as RAM, ROM, HDD (Hard Disk Drive), SSD (Solid State Disk), SSD (Silicon Disk Drive), EEPROM, EPROM, flash memory device, magnetic disk, etc., and combinations thereof, and may include web storage of a server that performs the storage function of memory over the internet. Additionally, the server computing system (1030) may further include a data storage (data store). For example, the data storage may be composed of at least one of a relational database, a NoSQL database, a data warehouse, and a local file system.

[0076] In the memory (1032) constituting the server computing system (1030) according to the present invention, data and instructions necessary for the operation of an application for the at least one processor (1031) to re-tokenize visual tokens in semantic units and select tokens to be maintained and / or removed through re-evaluation may be stored.

[0077] In one embodiment, the server computing system (1030) may be composed of a single device or a plurality of computing devices, and these may be configured to operate according to a sequential or parallel computing architecture. Additionally, a distributed processing system may be configured with a plurality of networked devices.

[0078] Meanwhile, the training computing system (1050) may include at least one processor (1051) and memory (1052). The model trainer (1060) is a logical component that executes the training of at least one machine learning model (1020, 1040) and may be implemented in the form of hardware, firmware, or software. For example, the model trainer (1060) may be executed by the processor (1051) after loading training data (1061) stored in a storage device into memory (1052). For example, the model trainer (1060) may be configured to execute one or more operations (e.g., model training, model reconstruction, model validation, model testing) on ​​at least one machine learning model.

[0079] The machine learning model of the present invention may include at least one of a statistical model, an algorithm, a neural network (NN), a convolutional neural network (CNN), a generative neural network (GNN), a Word2Vec model, a Bag of Words model, a TF-IDF (document frequency-inverse document frequency) model, a GPT (Generative Pre-trained Transformer) model (or other autoregressive models), a PPO (Proximal Policy Optimization) model, a nearest neighbor model (e.g., a k-nearest neighbor model), a linear regression model, a K-means clustering model, a Q-learning model, a TD (Temporal Difference) model, a Deep Adversarial Network model, and all other types of models further described herein.

[0080] Specifically, the model trainer (1060) may execute operations to train a machine learning model, and said operations may include at least one of adding, removing, and modifying model parameters. At this time, the training of the machine learning model may be at least one of supervised learning, semi-supervised learning, and unsupervised learning. In one embodiment, the training of the machine learning model may include the step of repeatedly inputting training data (1061) based on epochs and repeatedly performing the machine learning model training process configured in this way. Here, an epoch may refer to a unit in which the entire set of training data (1061) undergoes forward and backpropagation processing once. In some implementations, different levels of training methods (e.g., supervised learning, semi-supervised learning, unsupervised learning) may be used for different epochs.

[0081] The training data (1061) of the present invention may include input data and / or data previously output from at least one machine learning model (e.g., recursive learning feedback).

[0082] At least one parameter of a machine learning model may include at least one of a seed value, a model node, a model layer, an algorithm, a function, connections between different machine learning models, connections between parameters, machine learning model constraints, and other digital components that influence the output of the machine learning model. In this case, model connections between different machine learning models may include or represent relationships between model parameters and / or models, which may be dependent or interdependent, hierarchical, and / or static or dynamic. The combinations and configurations of model parameters described herein may be too complex to be maintained or utilized by human cognitive abilities.

[0083] In the present invention, the machine learning parameters described according to the embodiments are not limited, and a single machine learning model may further include a plurality of model parameters.

[0084] Meanwhile, FIG. 2 illustrates an example of a block diagram of a computing device (1100) that may be included in a user computing device (1010), a server computing system (1030), and a training computing system (1050), as an embodiment of a computing system (10000) in which the present invention can be implemented.

[0085] As illustrated in FIG. 2, the computing device (1100) may include at least one application (e.g., Application 1 to Application N), and each of the at least one application may include a machine learning library and a model execution environment for performing a method of re-tokenizing visual tokens into machine learning-based semantic units and selecting tokens to retain and / or remove through re-evaluation. The at least one application included in the computing device (1100) may communicate with the sensor, context manager, device state manager, or additional component(s) within the computing device (1100) via an Application Programming Interface (API). In one embodiment, the at least one application may interface with device components, such as receiving sensor data or state data or transmitting prediction results to an output device via a public or private API.

[0086] Meanwhile, FIG. 3 illustrates an example of a block diagram in another aspect of a computing device (1200), which is one of the components of a computing system (10000) that performs a method of re-tokenizing visual tokens into semantic units according to an embodiment of the present invention and selecting tokens to be maintained and / or removed through re-evaluation.

[0087] A computing device (1200) according to the present invention may include at least one application (e.g., Application 1 to Application N), and at least one application may communicate with a central intelligence layer (1210). Each application may interact with a shared model within the central intelligence layer (1210) through an API (e.g., a common API).

[0088] The central intelligence layer (1210) includes one or more machine learning models and may share them among multiple applications or provide them independently to each. In one embodiment, the central intelligence layer (1210) may be integrated as part of an operating system or implemented as a separate logical layer.

[0089] Additionally, the central intelligence layer (1210) can communicate with the central device data layer (1220). The central device data layer (1220) can integrate and store at least one and / or at least one image stored within the computing device (1200), re-tokenize visual tokens into semantic units, and provide this as input data necessary to select tokens to retain and / or remove through re-evaluation. Each device component (e.g., sensor, state manager, etc.) can communicate with the central device data layer (1220) via a private API, etc.

[0090] The technology described herein may be composed of a single or multiple computing devices, and a machine learning model that performs a method of re-tokenizing visual tokens into semantic units and selecting tokens to retain and / or remove through re-evaluation may be executed sequentially or in parallel on a single component or multiple distributed components. The data store, machine learning model, and application may be distributed and operated locally or over a network, and these configurations can be flexibly applied to various system architectures.

[0091] Meanwhile, the present invention relates to a method and system for optimizing image recognition performance capable of improving image recognition performance by efficiently processing image data (or images). More specifically, the present invention relates to a method and system for optimizing image recognition performance that re-tokenizes visual tokens in semantic units and selects tokens to be retained and / or removed through re-evaluation in order to improve image recognition efficiency. In this case, the method and system for optimizing image recognition performance according to the present invention may also be referred to as an “image recognition efficiency optimization method and system,” an “image recognition performance improvement method and system,” an “image recognition efficiency improvement method and system,” or a “content recognition re-tokenization method and system for efficient image recognition.”

[0092] Furthermore, the image recognition performance optimization system according to the present invention includes at least one artificial intelligence model, and the present invention aims to improve image recognition performance by efficiently processing image data.

[0093] In this case, at least one artificial intelligence model mentioned in this specification may include various models that can be utilized depending on various situations or purposes. For example, the artificial intelligence model in the present invention includes at least one of a machine learning (ML) model, a deep learning model, a deep neural network (DNN), a language model (LM), a large language model (LLM), a foundation model, a super-large foundation model, a generative AI model, a transformer-based model, a self-supervised learning (SSL) model, a supervised learning (SL) model, a reinforcement learning (RL) model, and a special purpose model (e.g., a time series forecasting model (e.g., ARIMA model, SARIMA model, etc.), a time series foundation model, a graph neural network (GNN), a multimodal model, a natural language processing (NLP) model, a computer vision model, a speech recognition / synthesis model, a recommendation system model, etc.). It is possible.

[0094] Furthermore, the artificial intelligence model used in the image recognition performance optimization system according to the present invention may be implemented with a single or multiple components. More specifically, the artificial intelligence model used in the present invention may be implemented with a single component (1) or with multiple components (2, 3, 4 or more, etc.).

[0095] In this regard, when multiple artificial intelligence models are implemented in the present invention, the multiple artificial intelligence models may be implemented as models of the same type (e.g., each implemented as a first transformer-based model, a second transformer-based model, etc.). In this case, the multiple artificial intelligence models may be implemented to perform different functions (or roles) in the present invention (e.g., the first transformer-based model performs a first function, and the second transformer-based model performs a second function).

[0096] Alternatively, multiple AI models may each be implemented as different models (for example, at least one AI model may be implemented as a Transformer-based model, and at least one other AI model may be implemented as a large-scale language model). In this case, the multiple AI models may be physically separated models (or modules) or multiple functional units within a single piece of hardware. However, this specification does not limit the implementation of AI models to a singular or multiple forms, and for convenience, they will be described as "at least one AI model" or "AI model."

[0097] Meanwhile, as illustrated in FIG. 4, the image recognition performance optimization system (100) according to the present invention may include at least one of an input unit (110), an output unit (120), a communication unit (130), a storage unit (140), a specific module (150), a token processing module (160), an artificial intelligence model (170), and a control unit (180). However, the components of the image recognition performance optimization system (100) according to the present invention are not necessarily limited thereto and may further include various hardware components that perform the same or similar roles as described in the description of the present specification.

[0098] Although not illustrated, the image recognition performance optimization system (100) according to the present invention may include one or more processors, and such processors may include one or more general-purpose processors and / or one or more special-purpose processors (e.g., digital signal processor, tensor processing unit (TPU), graphics processing unit (GPU), neural network processing unit (NPU), application integrated circuit, application semiconductor (ASIC), field programmable gate array (FPGA), quantum processing unit (or quantum processor, QPU), etc.). One or more processors may be configured to execute instructions, computer-readable instructions, and / or other instructions described herein that are stored (or included) in the storage unit (140). The image recognition performance optimization method and system according to the present invention may perform data processing described below in cooperation with memory and at least one processor. The processor may perform a series of operations and data processing using data and information stored in memory. In this case, memory may be a component of the storage unit (140).

[0099] In addition, the image recognition performance optimization system (100) according to the present invention can perform data processing and computation processes using quantum gates, quantum entanglement, and quantum superposition states, taking into consideration implementation in a quantum computer environment. For example, the present invention can perform parallel computations based on qubits, and such quantum computations can operate complementarily with existing classical computers.

[0100] Such quantum computers may include parallel computation using qubits and high-speed data processing devices utilizing quantum entanglement, and hardware-based computational optimization using FPGAs and ASICs is possible. In addition, quantum computers may utilize quantum processors capable of qubit-based parallel computation, and data processing efficiency can be improved through a hybrid structure with existing classical computers.

[0101] Meanwhile, the input unit (110) can be configured in various ways as a means of data input. For example, the input unit (110) can be configured to receive user input. The input unit (110) can be configured to receive user input from a user terminal. Here, “receiving input” may mean receiving an input signal (or selection signal) corresponding to the user’s input based on input made by the user through the configuration of the input unit provided in the user terminal.

[0102] Here, the user terminal may include at least one of a mobile phone, a smartphone, a notebook computer, a laptop computer, a slate PC, a tablet PC, an ultrabook, a desktop computer, a digital broadcasting terminal, a PDA (personal digital assistants), a PMP (portable multimedia player), a navigation device, and a wearable device (e.g., a smartwatch, a smart glass, a head-mounted display).

[0103] In addition, the input unit (110) in the present invention does not necessarily mean a hardware means, but can be understood as a channel for receiving input from a user.

[0104] The input unit (110) may also be referred to as a user interface module. The input unit (110) may include a touch screen, a computer mouse, a keyboard, a keypad, a touchpad, a trackball, a joystick, a voice recognition module, or other similar devices. However, the present invention does not limit the type of input unit (110).

[0105] Here, user input may include documents, text, images (or videos), voice, etc. In this case, the image recognition performance optimization system (100) may further include a module that converts voice into text.

[0106] Next, the output unit (120) can output information through an output unit configuration (e.g., a display unit, a touch screen, a speaker, etc.) provided in a user terminal linked with the image recognition performance optimization system (100) according to the present invention. For example, the output unit (120) can output information regarding a token to be maintained and / or a token to be removed to be removed to the display unit of the user terminal. As another example, the output unit (120) can output at least one page (or service page) linked with the image recognition performance optimization system (100) according to the present invention to the display unit of the user terminal. Furthermore, the output unit (120) does not necessarily mean a hardware means, but can be understood as a channel for outputting results to a user.

[0107] Next, the communication unit (130) may be connected via a wireless or wired network to a user terminal, at least one server (e.g., a server linked to the image recognition performance optimization system (100), a central server, an external server, etc.), at least one device, and at least one network, etc., to receive or transmit overall data and information necessary for the operation of the image recognition performance optimization system (100) according to the present invention.

[0108] The communication unit (130) can support various communication methods depending on the communication standard of the communicating device.

[0109] For example, the communication unit (130) may be configured to communicate with a communication target using at least one of the following technologies: WLAN (Wireless LAN), Wi-Fi (Wireless-Fidelity), Wi-Fi (Wireless Fidelity) Direct, DLNA (Digital Living Network Alliance), WiBro (Wireless Broadband), WiMAX (World Interoperability for Microwave Access), HSDPA (High Speed ​​Downlink Packet Access), HSUPA (High Speed ​​Uplink Packet Access), LTE (Long Term Evolution), LTE-A (Long Term Evolution-Advanced), 5G (5th Generation Mobile Telecommunication), Bluetooth (Bluetooth™ RFID (Radio Frequency Identification), Infrared Communication (Infrared Data Association; IrDA), UWB (Ultra-Wideband), ZigBee, NFC (Near Field Communication), Wi-Fi Direct, and Wireless USB (Wireless Universal Serial Bus).

[0110] Next, the storage unit (140, or memory) serves to store various data related to the present invention and may include one or more non-transient computer-readable storage media that can be read and / or accessed by at least one of one or more processors.

[0111] One or more computer-readable storage media may include volatile and / or non-volatile storage components, such as optical, magnetic, organic, or other memory or disk storage devices. In some examples, the storage unit (140) may be implemented using a single physical device (e.g., one optical, magnetic, organic, or other memory or disk storage device), whereas in other examples, the storage unit (140) may be implemented using two or more physical devices.

[0112] The storage unit (140) may include computer-readable instructions and additional data. The storage unit (140) may include a storage necessary to perform at least some of the methods, scenarios, and techniques described herein and / or at least some of the functions of the device and network.

[0113] Furthermore, at least a portion of the storage unit (140) may be a cloud storage or a cloud server. At least a portion of the data corresponding to user input received from the input unit (110) and the training data may be stored in the storage unit (140).

[0114] That is, the storage unit (140) is sufficient as a space where information necessary for the operation of the image recognition performance optimization system (100) according to the present invention is stored, and it can be understood that there are no restrictions on the physical space.

[0115] Furthermore, the storage unit (140) may store a computer program including computer program instructions. Furthermore, the storage unit (140) may store a computer program including computer program instructions that control the operation of the system (100) or control the operation of the control unit (180) when loaded into the processor of the system (100).

[0116] Next, a specific module (150) may be configured to perform the role of applying a local inductive bias to the visual scene represented by the input image (20). The specific module (150) may apply a local inductive bias so that positionally adjacent patches have a higher similarity (or similarity), thereby facilitating re-tokenization until non-semantic tokens are merged (or combined) with adjacent tokens to form a meaningful token (or removed if not related to the overall meaning). In the present invention, the specific module (150) may also be named a “Local Coherence Bias module” or a “Local Inductive Bias module,” etc.

[0117] Here, local inductive bias (or local induction bias) refers to a bias implied within the model structure (or within the model) and / or computational process to induce the preferential consideration of elements that are positionally (or spatially) adjacent in the input data, based on the premise (or assumption) that such elements possess similar meanings (or characteristics) or continuous semantic structures. In other words, it assumes that even if an individual input element lacks sufficient meaning on its own, a clearer semantic representation can be derived when processed together with nearby elements. For example, in image (or video) processing and / or Vision Transformer-based models, local inductive bias is based on the assumption that patches or tokens located close to each other share high correlation or continuous meaning; accordingly, it may include structural characteristics that cause the model to operate by reinforcing features between adjacent tokens or forming meaning at the local area level.

[0118] Furthermore, a specific module (150) may be composed of at least one overlapping convolution layer. The convolution layer constituting this specific module (150) enables each patch to acquire meaning through contextual relationships with adjacent patches. That is, the specific module (150) enhances the similarity between non-semantic patches, thereby enabling the creation of new meaningful tokens or the removal of tokens unrelated to meaning during the re-tokenization process.

[0119] Next, the token processing module (160, or token processing unit, tokenizer, etc.) can convert (or embedding) a plurality of tokens (21, 22, 23, 24, 25, 26) through linear projection on a plurality of patches divided from the input image (20).

[0120] Additionally, the token processing module (160) may re-tokenize at least some of the converted multiple tokens (21, 22, 23, 24, 25, 26) into semantic units and select tokens to be retained and / or removed through re-evaluation. For example, the token processing module (160) may re-tokenize meaningless tokens (i.e., non-semantic tokens) into semantic units in the initial stage and, based on the re-evaluation results through semantic (or importance) re-evaluation, select tokens to be retained (first token (e.g., “T1”, 21), third token (e.g., “T3”, 23), fifth token (e.g., “T5”, 25)) and tokens to be removed (second token (e.g., “T2”, 22), fourth token (e.g., “T4”, 24), sixth token (e.g., “T6”, 26)), respectively.

[0121] In one embodiment, the token processing module (160) may be a component included in the artificial intelligence model (170). Alternatively, the token processing module (160) may be a component included in the image recognition performance optimization system (100) as a separate component from the artificial intelligence model (170). The present invention is not limited to any one of these.

[0122] Next, the artificial intelligence model (170) may include at least one model for processing image data (i.e., input image (20)). For example, the artificial intelligence model (170) may include a Vision Transformers (ViTs) based model.

[0123] The vision transformer divides the input image (20) into non-overlapping PxP size patches and projects (or transforms) each patch into a token embedding (or token, image token, visual token, visual token, etc.). In one embodiment, when the patch size is “16x16 (p=16)” and the image size is “224x224”, a total of 196 image tokens can be generated. At this time, an additional class token, indicated by [CLS], can be added to the image token sequence. The class token serves to aggregate the entire image information and is utilized for final classification. All tokens are combined with positional embeddings, and these patch embeddings can be input into a transformer encoder.

[0124] Next, the control unit (180) can perform the role of controlling the overall operation of the image recognition performance optimization system (100) related to the present invention. The control unit (180) can process signals, data, information, etc. that are input or output through the components of the image recognition performance optimization system (100) described above, or perform a series of data processing to provide or process appropriate information and functions to the user. The control unit (180) can be physically implemented by the processor described above.

[0125] Meanwhile, as described above, the image recognition performance optimization system (100) according to the present invention may be a system that efficiently processes image data (or images) to improve image recognition performance. More specifically, the image recognition performance optimization system (100) may be a system that re-tokenizes visual tokens into semantic units and selects tokens to be maintained and / or removed through re-evaluation in order to improve image recognition efficiency. Below, based on the configuration of the image recognition performance optimization system (100) described above, we will examine the image recognition performance optimization method according to the present invention in more detail.

[0126] Meanwhile, as illustrated in FIG. 5, the image recognition performance optimization method according to the present invention may include the steps of: receiving at least one input image (S510); dividing at least one input image into a plurality of patches of a preset size (S520); performing a transformation on the plurality of patches to obtain a plurality of tokens (S530); calculating an attention score for each of the plurality of tokens to evaluate the importance of each of the plurality of tokens (S540); performing grouping by dividing at least some of the plurality of tokens into a first group and a second group, respectively, based on the evaluated importance of each of the plurality of tokens (S550); merging at least one token included in the first group and at least one token included in the second group to generate at least one merged token (S560); re-evaluating the importance of at least one merged token and the importance of at least some of the tokens among the plurality of tokens (S570); and selecting at least one of the at least one merged token and at least some of the tokens based on the re-evaluation result (S580).

[0127] At this time, in the present invention, at least some of the processes S510 to S580 may be performed using a token processing module (160) that is executed (or performed) by at least one processor (or control unit (180)). Alternatively, in the present invention, at least some of the processes S510 to S580 may be performed using an artificial intelligence model (170) that is executed by at least one processor. However, for the convenience of explanation, the present invention will be described without distinguishing between them.

[0128] First, the image recognition performance optimization system (100) receives at least one input image and can divide the received input image into a plurality of patches of a preset size. For example, as shown in FIGS. 6 and 7, the image recognition performance optimization system (100) can divide the input image (700) into a plurality of patches (701, 702, 703, 704, 705, 706, 707, 708, 709, 710, 711, 712, 713, 714, 715, 716) of a preset size that do not overlap each other.

[0129] And, the image recognition performance optimization system (100) may use at least one specific module (e.g., “Local Coherence Bias”, 150) to impart (or add) a local inductive bias to the visual scene (or visual scene) represented by the input image (700).

[0130] As an example, even when it is difficult to distinguish meaning at the individual patch level, meaningful semantics can be formed through contextual combination with positionally (or spatially) adjacent patches. As discussed above, this phenomenon can be referred to as local inductive bias.

[0131] The image recognition performance optimization system (100) can apply local inductive bias to at least some of a plurality of patches (701, 702, 703, 704, 705, 706, 707, 708, 709, 710, 711, 712, 713, 714, 715, 716) by using a specific module (150) to apply such local inductive bias to tokens (or image tokens, visual tokens, visual tokens, etc.).

[0132] More specifically, a specific module (150) can apply the local inductive bias to at least some of the patches so that at least some of the positionally adjacent patches among the plurality of patches (701, 702, 703, 704, 705, 706, 707, 708, 709, 710, 711, 712, 713, 714, 715, 716) have high similarity.

[0133] In this case, a plurality of patches (701, 702, 703, 704, 705, 706, 707, 708, 709, 710, 711, 712, 713, 714, 715, 716) may include at least one of at least one patch having a first attribute and at least one patch having a second attribute.

[0134] Here, at least one patch having the first attribute may include at least one semantic patch. A semantic patch is a patch that contributes to the formation of the overall visual meaning of the input image (700), and can be understood as a patch that contains information (or visual information) that contributes to the formation of the overall meaning of the input image (700) by itself as a single patch. In one embodiment, the semantic patch may include information essential for image recognition, such as the outline, boundary, feature point, and texture of an object.

[0135] Additionally, at least one patch having a second attribute may include at least one non-semantic patch. A non-semantic patch is a patch that contributes little to the formation of the overall visual meaning of the input image (700), and can be understood as a patch that does not sufficiently express meaningful visual features on its own or contributes little to the semantic judgment of the entire input image (700). In one embodiment, the non-semantic patch may include areas such as the sky, a road, a uniform background, or partial or incomplete patterns that are difficult to form semantic features on their own.

[0136] Among the plurality of patches (701, 702, 703, 704, 705, 706, 707, 708, 709, 710, 711, 712, 713, 714, 715, 716), at least some of the patches to which a local inductive bias is applied may include at least one positionally adjacent and / or at least one non-semantic patch.

[0137] For example, let us assume that the first patch (701) and the fifth patch (705), and the eleventh patch (711) and the twelveth patch (712) that are positionally adjacent are specified as target patches (i.e., non-meaning patches) to which local inductive bias is applied. The specific module (150) can apply local inductive bias to at least some of the positionally adjacent patches (e.g., the first patch (701) and the fifth patch (705) and the eleventh patch (711) and the twelveth patch (712), etc.) among the plurality of patches (701, 702, 703, 704, 705, 706, 707, 708, 709, 710, 711, 712, 713, 714, 715, 716). In this case, applying a local inductive bias to at least some positionally adjacent patches in the present invention can also be understood as enhancing the feature similarity between at least some positionally adjacent tokens among a plurality of tokens. Alternatively, it can also be understood as enhancing the feature similarity between at least some positionally adjacent patches among a plurality of patches (701, 702, 703, 704, 705, 706, 707, 708, 709, 710, 711, 712, 713, 714, 715, 716) (e.g., the first patch (701) and the fifth patch (705), and the eleventh patch (711) and the twelfth patch (712), etc.).

[0138] That is, in the present invention, applying a local inductive bias to at least some patches can also be understood as applying a local inductive bias to at least some positionally adjacent tokens among a plurality of tokens. Alternatively, in the present invention, applying a local inductive bias to positionally adjacent non-semantic patches can also be understood as applying a local inductive bias to positionally adjacent non-semantic tokens among a plurality of tokens.

[0139] Accordingly, as adjacent patches become entangled with overlapping features, geometrically close non-semantic patches can be induced to have higher similarity. That is, through a specific module (150) that applies a local inductive bias, at least some of the non-semantic patches that are positionally adjacent can be induced to have high similarity. This can also be interpreted to mean that at least some of the positionally adjacent non-semantic tokens generated (or obtained) by performing a transformation on at least some of the positionally adjacent non-semantic patches are induced to have high similarity.

[0140] Furthermore, multiple patches (701, 702, 703, 704, 705, 706, 707, 708, 709, 710, 711, 712, 713, 714, 715, 716) can each be converted into multiple tokens (701a, 702a, 703a, 704a, 705a, …, 892a, 893a, 894a, 895a, 896a). More specifically, the token processing module (160) can perform a conversion on each of the plurality of patches (701, 702, 703, 704, 705, 706, 707, 708, 709, 710, 711, 712, 713, 714, 715, 716) to obtain (or generate) the plurality of tokens (701a, 702a, 703a, 704a, 705a, …, 892a, 893a, 894a, 895a, 896a). For example, assuming that the size of multiple patches (701, 702, 703, 704, 705, 706, 707, 708, 709, 710, 711, 712, 713, 714, 715, 716) is “16x16” and the size of the input image (700) is “224x224”, a total of 196 tokens (e.g., the first token (701a) to the 196th token (896a)) can be obtained.

[0141] At this time, a plurality of tokens (701a, 702a, 703a, 704a, 705a, …, 892a, 893a, 894a, 895a, 896a) may include at least some tokens obtained by performing a transformation on at least some patches to which a local inductive bias is applied (e.g., a first token (701a), a fifth token (705a), an eleventh token (711a), a twelfth token (not shown), etc.). For example, the at least some tokens may be transformed (or embedded) through a linear projection on at least some patches to which a local inductive bias is applied (e.g., a first patch (701) and a fifth patch (705), and an eleventh patch (711) and a twelfth patch (712), etc.). These tokens may also be understood as tokens to which a local inductive bias is applied. That is, the plurality of tokens may include tokens without local inductive bias (e.g., tokens transformed through linear projection onto a patch having a first attribute (semantic patch)) and tokens with local inductive bias (e.g., tokens transformed through linear projection onto a patch having a second attribute (non-semantic patch)).

[0142] Meanwhile, as illustrated in FIGS. 7 and 8, the token processing module (160) can calculate the attention score of each of the plurality of tokens (701a, 702a, 703a, 704a, 705a, …, 892a, 893a, 894a, 895a, 896a) and evaluate the importance of each of the plurality of tokens (701a, 702a, 703a, 704a, 705a, …, 892a, 893a, 894a, 895a, 896a).

[0143] In one embodiment, the attention score (or class attention score) measures the degree of interaction between a class token and each of a plurality of tokens (701a, 702a, 703a, 704a, 705a, …, 892a, 893a, 894a, 895a, 896a), which may indicate the importance (or degree) contributing to the overall visual semantic of the input image (700). Here, the length of the query vector and the number of input tokens may be represented as shown in FIG. 9 (a) and (b). In each transformer block, the input token may consist of a class token and a remainder token. The token sequence may be projected onto a query matrix, a key matrix, and a value matrix (see FIG. 9 (c) through FIG. 9 (f)). In this case, the attention score, class token, and output can be represented as in Fig. 9 (g). At this time, the query vector of the class token can be represented as in Fig. 9 (h). Consequently, the output of the class token can be expressed as a linear combination of value vectors (see Fig. 9 (i)). The coefficients of this combination (see Fig. 9 (j)) may represent the attention values ​​that the class token has for all tokens in the formula illustrated in Fig. 9 (g).

[0144] The token processing module (160) can evaluate the importance of each of the multiple tokens (701a, 702a, 703a, 704a, 705a, …, 892a, 893a, 894a, 895a, 896a) based on the attention score with the class token ([CLS]).

[0145] More specifically, the token processing module (160) can evaluate (or calculate) the importance that each of the multiple tokens (701a, 702a, 703a, 704a, 705a, …, 892a, 893a, 894a, 895a, 896a) contributes to the overall visual semantic of the input image (700), based on the attention score calculated for each of the multiple tokens (701a, 702a, 703a, 704a, 705a, …, 892a, 893a, 894a, 895a, 896a).

[0146] And, the token processing module (160) can perform grouping on at least some of the multiple tokens (701a, 702a, 703a, 704a, 705a, …, 892a, 893a, 894a, 895a, 896a) based on the importance of each of the multiple tokens (701a, 702a, 703a, 704a, 705a, …, 892a, 893a, 894a, 895a, 896a).

[0147] More specifically, the token processing module (160) can identify (or select, sort, etc.) at least some of the tokens to be grouped among the multiple tokens based on the importance of each of the multiple tokens (701a, 702a, 703a, 704a, 705a, …, 892a, 893a, 894a, 895a, 896a).

[0148] Here, the tokens to be grouped may be specified according to a pre-set criterion (or a pre-set ratio criterion). For example, the pre-set criterion may be set so that the lower K tokens of the entire set of tokens are specified. Or, the pre-set criterion may be set so that the lower 30% of the ratio of the entire set of tokens is specified. In this case, the tokens to be grouped may include at least some (or multiple) tokens (701a, 711a, 722a, 732a, 743a, 753a, ..., 864a, 874a, 885a, 896a) of low importance among a plurality of tokens (701a, 702a, 703a, 704a, 705a, …, 892a, 893a, 894a, 895a, 896a). The low-importance tokens identified as such grouping target tokens may include at least one and / or at least one non-meaning token. Alternatively, the grouping target tokens may include at least one and / or at least one token endowed with a local inductive bias.

[0149] Based on the grouping target tokens (701a, 711a, 722a, 732a, 743a, 753a, …, 864a, 874a, 885a, 896a) being specified, the token processing module (160) can perform grouping by dividing the grouping target tokens (701a, 711a, 722a, 732a, 743a, 753a, …, 864a, 874a, 885a, 896a) into a first group (e.g., “Group 1”) and a second group (e.g., “Group 2”), respectively. For example, multiple grouping target tokens (701a, 711a, 722a, 732a, 743a, 753a, …, 864a, 874a, 885a, 896a) may be assigned (or included) to each of the first group and the second group alternately (or alternately) according to the order of attention scores.

[0150] Subsequently, the token processing module (160) may merge at least one of the tokens included in the first group (701a, 722a, 743a, …, 864a, 885a, or a plurality of tokens) and at least one of the tokens included in the second group (711a, 732a, 753a, …, 874a, 896a, or a plurality of tokens) to generate at least one and / or at least one or more merged tokens. In the present invention, this merging process can also be understood as a process of re-tokenizing (or reconstructing) tokens of low importance (i.e., non-meaning tokens) among a plurality of tokens tokenized from a plurality of divided patches into meaningful units. That is, it can be understood as a process of re-tokenizing tokens into meaningful units.

[0151] As seen above, the present invention is based on the premise that positionally adjacent non-meaning tokens may be candidates for having meaning together. In order to induce better merging of such token pairs (i.e., non-meaning token pairs), a specific module (150) can be used to strengthen the similarity between positionally adjacent patches (or tokens). Through this, the present invention can enable more effective merging of non-meaning tokens. That is, the present invention promotes the condensation of positionally close non-meaning tokens together through the local inductive biasing process described above, thereby allowing adjacent non-meaning tokens to be combined and merged into meaningful tokens during the re-tokenization process, or, if irrelevant to the overall meaning, to be removed for the efficiency of data processing.

[0152] In this regard, the token processing module (160) can generate at least one merged token by merging each token (701a, 722a, 743a, …, 864a, 885a) included in the first group with a semantically similar token in the second group. Each token (701a, 722a, 743a, …, 864a, 885a) included in the first group can be merged with a semantically similar token in the second group through a pre-set matching technique. For example, the pre-set matching technique may include a bipartite soft matching technique (or bipartite soft matching, bipartite soft matching, etc.). Such a matching technique may be a technique (or algorithm) that places two sets of tokens (e.g., a first group and a second group) in the form of a bipartite graph and performs soft matching based on the similarity of each pair of tokens to select or merge the optimal pair of tokens. Here, soft matching may be a method that preferentially matches the most similar token pairs through similarity-weighted probabilistic and / or continuous matching, rather than complete deterministic matching. That is, based on the similarity matrix between tokens, it calculates which token in the second group best merges (or combines) with each token in the first group, and merges those pairs to generate a new token.

[0153] The token processing module (160) can generate at least one merged token by using a preset matching technique to merge each token (701a, 722a, 743a, …, 864a, 885a) included in the first group with a semantically similar token in the second group.

[0154] Alternatively, the token processing module (160) may use a pre-configured matching technique to merge at least one token with a local inductive bias included in the first group (e.g., Token 22 (722a)) with at least one token with a local inductive bias among the tokens included in the second group that is semantically similar to the at least one token and is therefore given a local inductive bias (e.g., Token 53 (896a)), thereby generating at least one merged token. This means that at least one and / or at least one more token with a local inductive bias included in the first group may be merged through the pre-configured matching technique with at least one and / or at least one more token that is semantically similar among the tokens included in the second group and is given a local inductive bias.

[0155] In one embodiment, the token processing module (160) may generate a merged first token (722) by merging the 22nd token (722a) included in the first group with the 53rd token (753a) which is semantically similar among the tokens (711a, 732a, 753a, …, 874a, 896a) included in the second group. Additionally, the token processing module (160) may generate a merged second token (743) by merging the 43rd token (743a) included in the first group with the 11th token (711a) which is semantically similar among the tokens (711a, 732a, 753a, …, 874a, 896a) included in the second group. Additionally, the token processing module (160) can merge the 164 token (864a) included in the first group with the 196 token (896a), which is semantically similar among the tokens (711a, 732a, 753a, …, 874a, 896a) included in the second group, to generate a merged 3 token (870).

[0156] Furthermore, the token processing module (160) can re-evaluate the importance of at least one merged token and the importance of at least some of the tokens among the plurality of tokens. More specifically, the token processing module (160) can re-evaluate the importance of the merged token and the importance of at least some of the tokens by calculating the attention score of at least one and / or at least one (or plurality) merged token and the attention score of at least some of the tokens.

[0157] For example, the above at least some of the tokens may include at least some of the tokens among the plurality of tokens (701a, 702a, 703a, 704a, 705a, …, 892a, 893a, 894a, 895a, 896a) for which grouping has not been performed, and at least one of the remaining tokens among the tokens included in the first group and the second group that have not been merged (e.g., 701a, 732a, …, 874a, 885a, etc.). In this case, the tokens for which grouping has not been performed may include tokens of high importance (see FIG. 7).

[0158] As illustrated in FIGS. 6 and 8, the token processing module (160) can calculate the attention score of each of the merged tokens (722, 743, …, 870) and the attention score of each of at least some of the tokens (701a, 702a, 723a, …, 865a, 874a, 885a, 896a), and based thereon, re-evaluate the importance of each of the merged tokens (722, 743, …, 870) and the importance of each of at least some of the tokens (701a, 702a, 723a, …, 865a, 874a, 885a, 896a). At this time, among the merged tokens (722, 743, …, 870), at least one token that has a high relevance (or relevance) to the overall visual meaning of the input image (700) can be reorganized and / or selected as a semantic token.

[0159] Subsequently, the token processing module (160) can select, based on the re-evaluation result, at least one token to be retained and at least one token to be removed from among at least one merged token and at least some tokens. In one embodiment, the selection process in the present invention can also be understood as a filtering process.

[0160] Specifically, the token processing module (160) can select, based on the re-evaluation result, at least one merged token and at least one token to be retained and at least one token to be removed, each having at least one attribute among the at least one portion of the tokens. Here, the at least one attribute may include at least one of a first attribute and a second attribute.

[0161] More specifically, the token processing module (160) may perform a first selection process (or a first filtering process) to select at least one token to be retained having a first attribute among at least one merged token and at least some of the tokens based on the re-evaluation result. For example, as shown in FIG. 8, the token processing module (160) may select tokens to be retained having a first attribute (702a, 722, 723a …, 865a, 870, 896a) among the merged tokens (722, 743, …, 870) and at least some of the tokens (701a, 702a, 723a, …, 865a, 874a, 885a, 896a) based on the re-evaluation result. In this case, the selected retention target tokens (702a, 722, 723a …, 865a, 870, 896a) may include at least one and / or at least one merged token (722, 870) that has a high relevance (or association) to the overall visual meaning of the input image (700).

[0162] The retained target tokens (702a, 722, 723a …, 865a, 870, 896a) having these first attributes may include at least one and / or at least one (or multiple) semantic tokens. A semantic token is a token that contributes to the formation of the overall visual meaning of the input image (700), and can be understood as a token that contains information (or visual information) that contributes to the formation of the overall meaning of the input image (700) by itself as a single token. In one embodiment, the semantic token may include information essential for image recognition, such as the outline, boundary, feature point, and texture of an object.

[0163] Additionally, the token processing module (160) may perform a second selection process (or a second filtering process) to select at least one token to be removed having a second attribute among at least one merged token and at least some tokens based on the re-evaluation result. For example, as illustrated in FIG. 8, the token processing module (160) may select tokens to be removed having a second attribute (701a, 743, 770 …, 874a, 885a) among the merged tokens (722, 743, …, 870) and at least some tokens (701a, 702a, 723a, …, 865a, 874a, 885a, 896a) based on the re-evaluation result. In this case, the selected removal target tokens (701a, 743, 770 …, 874a, 885a) may include at least one and / or at least one merged token (743, 770) that still has low relevance (or association) to the overall visual meaning of the input image (700).

[0164] The retained target tokens (701a, 743, 770 …, 874a, 885a) having these second attributes may include at least one and / or at least one (or multiple) non-semantic tokens. A non-semantic token is a token that contributes to a low degree to the formation of the overall visual meaning of the input image (700), and may be understood as a token that does not sufficiently express meaningful visual features on its own as a single token unit, or contributes to a low degree to the semantic judgment of the entire input image (700). In one embodiment, the non-semantic token may include areas such as the sky, a road, a uniform background, or partial or incomplete patterns that are difficult to form semantic features on their own.

[0165] Finally, the image recognition performance optimization system (100) can retain at least one retention target token having a first attribute and remove at least one removal target token having a second attribute.

[0166] For example, as illustrated in FIGS. 6 and FIGS. 8, the token processing module (160) may retain tokens (702a, 722, 723a …, 865a, 870, 896a) having a first attribute and store them in a designated storage (e.g., storage unit (140) or memory). In this case, the tokens (702a, 722, 723a …, 865a, 870, 896a) having a first attribute may be used for inference related to image recognition of the artificial intelligence model (170). Alternatively, the tokens (702a, 722, 723a …, 865a, 870, 896a) having a first attribute may be constructed as training data (or training data set) and used for training to optimize the image recognition performance of the artificial intelligence model (170).

[0167] As another example, the token processing module (160) can remove tokens (701a, 743, 770 …, 874a, 885a) having a second attribute so that they are not used in the learning and / or inference process of the artificial intelligence model (170). This ensures that only necessary tokens (tokens to be retained) are used and unnecessary tokens (tokens to be removed) are not used, thereby improving the expressiveness of visual information of the artificial intelligence model (170) and effectively reducing the amount of computation.

[0168] Meanwhile, FIGS. 10 and 11 illustrate an example of the performance evaluation results of an image recognition performance optimization method and system according to the present invention.

[0169] In one embodiment, with reference to FIG. 10 (a) and (b), it can be seen that the image recognition performance optimization method and system according to the present invention achieve higher accuracy than conventional token removal methods (or techniques, methods, etc.) in an image classification task. In particular, it is not only superior to conventional lightweighting techniques but also shows higher performance than the original model that is not lightweighted, which indicates that the semantic unit re-tokenization method of the present invention contributes to performance improvement.

[0170] In another embodiment, referring to FIG. 10(c), the effect of the semantic unit re-tokenization method according to the present invention was quantitatively analyzed, and the ratio of tokens that were initially evaluated as unimportant but were subsequently reconstructed into meaningful chunks and re-evaluated as important tokens was measured by layer. In particular, at the 6th layer, it was confirmed that 58.77% of the tokens were converted into important tokens. This supports the fact that the present invention effectively discovers meaningful information even within non-meaningful tokens that are easily overlooked in the beginning.

[0171] In another embodiment, referring to FIG. 10(d), as a result of testing the compatibility between the present invention and existing token removal methods, it can be seen that when the image recognition optimization method according to the present invention is applied, an average accuracy improvement of 0.79% is observed. This indicates that the present invention can produce high synergy with other token removal and / or merging-based methods.

[0172] In another embodiment, referring to FIG. 11, it can be seen that the image recognition performance optimization method according to the present invention exhibits high performance in terms of inference speed and accuracy. That is, the image recognition performance optimization method according to the present invention not only maintains relatively robust accuracy but also demonstrates excellent compatibility even at high speeds.

[0173] In conclusion, the present invention demonstrates that it is an effective method for improving the efficiency of a vision transformer through a strategy of reorganizing non-meaningful tokens into meaningful units. Furthermore, it confirms that the present invention can improve performance while reducing information loss compared to conventional simple token removal methods.

[0174] Meanwhile, the image recognition performance optimization method and system according to the present invention described above can be applied to various industrial and / or service fields and utilized effectively.

[0175] In this regard, FIGS. 12 and FIGS. 13 are conceptual diagrams for illustrating another embodiment of an image recognition performance optimization method and system according to the present invention.

[0176] In one embodiment, as illustrated in FIG. 12, the control unit (180) may provide at least one page (or service page) associated with the image recognition performance optimization system (100) according to the present invention to a user terminal (10). In this case, the user may be understood as a user and / or client company (or client enterprise) using the image recognition performance optimization system (100, or service) according to the present invention for at least one of the following purposes: i) to improve image recognition performance by efficiently processing image data; ii) to solve the problem of information loss during the process of processing image data and to preserve important information; iii) to improve the expressiveness of visual information of an artificial intelligence model and to effectively reduce the amount of computation; iv) to simultaneously improve the efficiency and accuracy of Vision Transformers (ViTs); and v) to improve efficiency and productivity in industrial sites.

[0177] The control unit (180) may receive at least one user input (or user request) required for the image recognition performance optimization system (100) from the user terminal (10). For example, the input data may include at least one of the following: a model type (e.g., “first artificial intelligence model”, 1201), input data (e.g., “Image set_20251102”, 1202), input data type (e.g., “image”, 1203), whether to maintain data security (e.g., “Yes”, 1204), purpose of use (e.g., “image recognition and classification”, 1205), and a goal or expected effect (e.g., “1. Improvement of the model’s visual information representation, 2. Reduction of computational load, 3. Improvement of performance, 4. Minimization of information loss”, 1206). The control unit (180) can receive user input required for the image recognition performance optimization system (100) based on the selection of a graphic object (1200a) associated with a user input receiving function from the user terminal (10).

[0178] Subsequently, the image recognition performance optimization system (100) can generate and provide a processing result (or output, output data, response (or answer), etc.) for the user input based on the received user input. For example, let us assume that the user input includes at least one and / or at least one image. An image recognition performance optimization system (100) can provide an appropriate processing result corresponding to the user input by: a process of dividing at least one image included in the user input into a plurality of patches of a preset size; a process of obtaining a plurality of tokens by performing a transformation on the plurality of patches; a process of calculating an attention score for each of the plurality of tokens and evaluating the importance of each of the plurality of tokens; a process of grouping at least some of the plurality of tokens into a first group and a second group based on the importance of each of the evaluated plurality of tokens; a process of merging at least one token included in the first group and at least one token included in the second group to generate at least one merged token; a process of re-evaluating the importance of at least one merged token and the importance of at least some of the tokens among the plurality of tokens; and a process of selecting at least one of the at least one merged token and at least some of the tokens based on the re-evaluation result.

[0179] In one embodiment, as illustrated in FIG. 13, an image recognition performance optimization system (100) may provide information about a token to be retained having a first attribute selected through a selection process (e.g., “semantic token”, 1210), and information about a token to be removed having a second attribute (e.g., “non-semantic Token”, 1211) to a user terminal (10).

[0180] At this time, the selected tokens may further include a token having a third attribute processed according to user input (or request), in addition to a token to be retained having a first attribute and a token to be removed having a second attribute. In one embodiment, the token having the third attribute may include information that may infringe on privacy or a token related to sensitive information. The image recognition performance optimization system (100) may provide information about the token having the third attribute (e.g., “other token”, 1212) to the user terminal (10).

[0181] Furthermore, the selected tokens may be retained and / or removed according to the user's selection (or request). For example, the user may retain and / or remove the corresponding tokens by selecting at least one and / or at least one graphic object that is output (or provided) in the surrounding area of ​​information (1211) about the token to be removed having a second attribute and information (1212) about the token having a third attribute. The control unit (180) may retain the tokens without removing all of them when a graphic object (1211a, 1212a) associated with retention is selected from the user terminal (10). Additionally, the control unit (180) may remove only some of the tokens when a graphic object (1211b, 1212b) associated with partial removal is selected from the user terminal (10). Additionally, the control unit (180) can remove all corresponding tokens when a graphic object (1211c, 1212c) associated with removal from all is selected from the user terminal (10).

[0182] As described above, according to the image recognition performance optimization method and system of the present invention, by re-tokenizing visual tokens into semantic units, non-semantic tokens that were previously judged to be meaningless can be merged (or combined) with surrounding information to acquire meaning. Through this, the present invention not only preserves tokens containing important information but also improves the overall performance of the model. That is, the present invention allows non-semantic tokens to be merged into a single meaningful token when they have the potential to have contextual meaning, thereby effectively reducing computational load while maintaining the expressiveness of the model.

[0183] Furthermore, according to the image recognition performance optimization method and system of the present invention, information loss can be minimized by reconstructing (or re-tokenizing) meaningless tokens (i.e., non-meaningful tokens) in the initial stage based on the importance of each of a plurality of tokens. That is, by reconstructing meaningless tokens in the initial stage, the present invention can prevent the problem in which potentially important patches are considered unimportant tokens and removed in the initial stage.

[0184] Furthermore, according to the image recognition performance optimization method and system of the present invention, a local inductive bias can be applied to positionally adjacent patches using a specific module so that positionally adjacent patches among a plurality of patches have high similarity. Through this, the present invention enables effective merging of meaningless tokens. That is, by strengthening the feature similarity between positionally close tokens using a specific module, the present invention enables non-meaningful tokens to be merged more naturally and meaningfully within an actual visual scene.

[0185] Furthermore, according to the image recognition performance optimization method and system of the present invention, non-meaningful tokens can be re-tokenized into meaningful units, and tokens to be retained and / or removed can be selected through re-evaluation. Through this, the present invention can improve the performance of the model while reducing information loss. That is, the present invention can enhance the expressiveness of the model's visual information and effectively reduce the amount of computation.

[0186] Meanwhile, the present invention described above can be implemented based on a quantum computer. The present invention implemented based on a quantum computer may include a qubit-based quantum processor and quantum memory, and may include software and hardware interfaces optimized for quantum computation.

[0187] Quantum processors in quantum computers utilize qubits to efficiently process complex operations through parallel computation, quantum entanglement, and quantum superposition, which cannot be performed by the binary bits of classical computers. Quantum processors process data using quantum gates and can provide exponential speed improvements for specific problems.

[0188] Meanwhile, the present invention described above can be implemented as a program that is executed by one or more processes on a computer and can be stored on a computer-readable medium (or recording medium).

[0189] Furthermore, the present invention described above can be implemented as computer-readable code or instructions on a medium on which a program is recorded. That is, the present invention can be provided in the form of a program.

[0190] Meanwhile, computer-readable media include all types of recording devices in which data that can be read by a computer system is stored. Examples of computer-readable media include HDD (Hard Disk Drive), SSD (Solid State Disk), SSD (Silicon Disk Drive), ROM, RAM, CD-ROM, magnetic tape, floppy disk, optical data storage device, etc.

[0191] Furthermore, the computer-readable medium may be a server or cloud storage that includes a storage and is accessible to an electronic device via communication. In this case, the computer may download the program according to the present invention from the server or cloud storage via wired or wireless communication.

[0192] A computer program may reach the system (100) through various suitable transmission mechanisms. The transmission mechanism may be, for example, a computer-readable storage medium, a computer program product, a memory device, a recording medium such as a CD-ROM or DVD, or a product that tangibly embodies the computer program. The transmission mechanism may be a signal configured to reliably transmit the computer program through air or an electrical connection. The system (100) may propagate or transmit the computer program as a computer data signal.

[0193] Furthermore, references to 'computer-readable storage media,' 'computer program products,' 'computer programs embodied in a tangible form,' etc., or to 'controller,' 'computer,' 'processor,' etc., should be understood to include not only computers with various architectures such as single / multi-processor architectures and sequential (Von Neumann) / parallel architectures, but also specialized circuits such as Field-Programmable Gate Arrays (FPGAs), Application Specific Circuits (ASICs), signal processing units, and other devices. References to computer programs, instructions, code, etc., should be understood to include software for programmable processors or firmware, such as programmable content for hardware devices, whether it is instructions for a processor or configuration settings for a fixed-function device, gate array, or programmable logic device.

[0194] Furthermore, in the present invention, the computer described above is an electronic device equipped with a processor, namely a CPU (Central Processing Unit), and no special limitations are placed on its type.

[0195] Meanwhile, the above detailed description should not be interpreted restrictively in all respects but should be considered exemplary. The scope of the invention shall be determined by a reasonable interpretation of the appended claims, and all modifications within the equivalent scope of the invention are included within the scope of the invention.

Claims

1. Regarding methods performed by a computer, A step of receiving at least one input image; A step of dividing the above at least one input image into a plurality of patches of a preset size; A step of obtaining a plurality of tokens by performing a transformation on the plurality of patches mentioned above; A step of calculating an attention score for each of the plurality of tokens and evaluating the importance of each of the plurality of tokens; A step of performing grouping by dividing at least some of the plurality of tokens into a first group and a second group, respectively, based on the importance of each of the plurality of tokens evaluated above; A step of merging at least one token included in the first group and at least one token included in the second group to generate at least one merged token; A step of re-evaluating the importance of at least one merged token and the importance of at least some of the plurality of tokens; and An image recognition performance optimization method characterized by including the step of selecting at least one of the at least one merged token and at least one of the at least some tokens based on the above re-evaluation result.

2. In Paragraph 1, In the step of re-evaluating the above importance, Calculate the attention score of at least one merged token and the attention score of at least some tokens, re-evaluate the importance of at least one merged token and the importance of at least some tokens, and In the above screening step, An image recognition performance optimization method characterized by selecting, based on the above re-evaluation results, at least one token to be retained and at least one token to be removed from among the at least one merged token and the at least one partial token.

3. In Paragraph 1, The method further includes the step of applying a local inductive bias to at least some of the plurality of patches using at least one specific module, and The above plurality of tokens are, An image recognition performance optimization method characterized by including at least some tokens obtained by performing a transformation on at least some patches to which the above-mentioned local inductive bias is applied.

4. In Paragraph 3, In the step of imparting the local inductive bias mentioned above, An image recognition performance optimization method characterized by applying a local inductive bias to at least some patches in at least one specific module such that at least some patches that are positionally adjacent among the plurality of patches have high similarity.

5. In Paragraph 3, The step of imparting the above local inductive bias is, An image recognition performance optimization method characterized by a step of applying the local inductive bias to at least some positionally adjacent patches among the plurality of patches in at least one specific module, thereby enhancing the feature similarity between at least some positionally adjacent tokens among the plurality of tokens.

6. In Paragraph 5, The above plurality of patches includes at least one of at least one patch having a first attribute and at least one patch having a second attribute, and In the step of imparting the local inductive bias mentioned above, An image recognition performance optimization method characterized by inducing at least some positionally adjacent non-meaning patches among the non-meaning patches to have high similarity through at least one specific module.

7. In Paragraph 6, At least one patch having the first attribute above includes at least one semantic patch, and An image recognition performance optimization method characterized in that at least one patch having the above-mentioned second attribute includes at least one non-semantic patch.

8. In Paragraph 1, In the step of evaluating the importance of each of the aforementioned plurality of tokens, An image recognition performance optimization method characterized by evaluating the importance of each of the plurality of tokens based on the attentive score calculated for each of the plurality of tokens.

9. In Paragraph 1, In the step of performing the above grouping, Based on the importance of each of the plurality of tokens evaluated above, at least some of the tokens to be grouped among the plurality of tokens are identified, and Grouping is performed by dividing at least some of the specified grouping target tokens into the first group and the second group, respectively. In the step of generating at least one merged token, An image recognition performance optimization method characterized by merging each token included in the first group with a semantically similar token within the second group to generate at least one merged token.

10. In Paragraph 9, An image recognition performance optimization method characterized in that the at least some of the grouping target tokens include at least some of the tokens with low importance among the plurality of tokens.

11. In Paragraph 9, An image recognition performance optimization method characterized in that each token included in the first group is merged with a semantically similar token in the second group through a pre-set matching technique.

12. In Paragraph 2, At least some of the above tokens are, At least some of the tokens among the plurality of tokens above for which the grouping was not performed, and Image recognition performance optimization method characterized by including at least one of the remaining unmerged tokens among the tokens included in each of the first group and the second group.

13. In Paragraph 2, In the above screening step, Based on the above re-evaluation results, among the at least one merged token and the at least one partial token, the at least one token to be retained and the at least one token to be removed, each having at least one attribute, are selected, and An image recognition performance optimization method characterized in that the above-mentioned at least one attribute includes at least one of a first attribute and a second attribute.

14. In Paragraph 13, In the above screening step, An image recognition performance optimization method characterized by selecting, based on the above re-evaluation results, at least one token to be retained having the first attribute and at least one token to be removed having the second attribute, respectively, among the at least one merged token and the at least one partial token.

15. In Paragraph 14, The at least one maintenance target token having the above-mentioned first attribute includes at least one semantic token, and An image recognition performance optimization method characterized in that the at least one maintenance target token having the above second attribute includes at least one non-semantic token.

16. In Paragraph 14, An image recognition performance optimization method characterized by further including the step of maintaining at least one token to be maintained having the first attribute and removing at least one token to be removed having the second attribute.

17. In Paragraph 3, An image recognition performance optimization method characterized in that the above-mentioned at least one specific module is composed of at least one overlapping convolution layer.

18. A system comprising memory configured to store executable instructions and one or more processors configured to perform operations by executing one or more instructions, The above system is, Receive at least one input image, and The above at least one input image is divided into a plurality of patches of a preset size, and A plurality of tokens are obtained by performing a transformation on the above plurality of patches, and The attention score of each of the plurality of tokens is calculated to evaluate the importance of each of the plurality of tokens, and Based on the importance of each of the plurality of tokens evaluated above, grouping is performed by dividing at least some of the plurality of tokens into a first group and a second group, respectively. At least one token included in the first group and at least one token included in the second group are merged to produce at least one merged token, Re-evaluate the importance of at least one merged token and the importance of at least some of the plurality of tokens, and An image recognition performance optimization system characterized by selecting at least one of the at least one merged token and at least one of the at least some tokens based on the above re-evaluation result.

19. A program that is executed by one or more processes in an electronic device and stored on a computer-readable recording medium, The above program is, A step of receiving at least one input image; A step of dividing the above at least one input image into a plurality of patches of a preset size; A step of obtaining a plurality of tokens by performing a transformation on the plurality of patches mentioned above; A step of calculating an attention score for each of the plurality of tokens and evaluating the importance of each of the plurality of tokens; A step of performing grouping by dividing at least some of the plurality of tokens into a first group and a second group, respectively, based on the importance of each of the plurality of tokens evaluated above; A step of merging at least one token included in the first group and at least one token included in the second group to generate at least one merged token; A step of re-evaluating the importance of at least one merged token and the importance of at least some of the plurality of tokens; and A program stored on a computer-readable recording medium characterized by including instructions for performing a step of selecting at least one of the at least one merged token and at least one of the at least some tokens based on the above re-evaluation result.

Images