ViT model improvement method, system and device and storage medium

Abstract

The invention provides a ViT model improvement method, system and device and a storage medium, and belongs to the technical field of computer vision. The method comprises the following steps: firstly, dividing an input image into a plurality of non-overlapped patches, and carrying out embedding processing on each patch to generate initial feature sequence representation; clustering the features of the adjacent patches by using a K-means clustering algorithm with spatial constraint, combining the adjacent patches with similar semantic features into K classes, and extracting the feature of each clustering center as a new feature representation; and finally, inputting the clustered feature sequence into ViT for self-attention calculation and feedforward network processing, and generating final image feature representation. According to the method, semantic similar patches are aggregated, the length of an input sequence is reduced, the calculation complexity is reduced, redundant information is reduced, the local semantic understanding ability is enhanced, and the method is suitable for visual tasks such as image classification and target detection.

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence and computer vision technology, and more specifically relates to a ViT model improvement method, system, device and storage medium. Background Art

[0002] In recent years, deep learning technology has made significant progress in the field of computer vision, especially convolutional neural networks (CNNs), which have performed well in tasks such as image classification, object detection, and semantic segmentation. However, the limitation of CNNs is that they have a fixed local receptive field and cannot effectively capture long-distance dependencies and global contextual information in images. To solve this problem, the Vision Transformer (ViT) model was proposed, and the self-attention mechanism of the Transformer was used to significantly improve the modeling ability of global information.

[0003] The ViT model divides the input image into a series of fixed-size patches (usually 16×16 or 32×32 pixel blocks), then processes these patches into vectors through linear mapping and feeds them into the Transformer network as an input sequence. The Transformer calculates the global dependencies between all patches through the self-attention mechanism, thereby achieving a global understanding of the image.

[0004] Although the ViT model has achieved success on many visual tasks, there are still several obvious problems: (1) The sequence length is too long: As the image resolution increases, the number of patches into which the image is divided increases rapidly, resulting in a significant increase in the sequence length of the input Transformer. This not only increases the computational complexity of the model, but also affects the model’s reasoning efficiency.

[0005] (2) Existence of redundant information: Adjacent patches usually have highly similar semantic information. Especially when processing images with smooth textures, this similarity leads to the introduction of a large amount of redundant information, which in turn affects the model performance.

[0006] (3) Loss of local semantics: After the standard ViT model divides the image into independent patches, the spatial continuity and local dependencies between patches are not fully modeled, resulting in a decrease in the ability to understand the local object structure or details.

[0007] Therefore, existing ViT models still face efficiency and performance bottlenecks when processing complex image tasks. Summary of the invention

[0008] In view of the above problems, the purpose of the present invention is to provide a ViT model improvement method, system, device and storage medium, which can effectively improve the comprehensive performance of the ViT model by clustering the features of adjacent pixel blocks, reducing redundant information and enhancing local feature expression.

[0009] In order to achieve the above object, the present invention is implemented through the following technical solutions: In a first aspect, an embodiment of the present application provides a ViT model improvement method, comprising: The input image is divided into multiple non-overlapping patches, and each patch is embedded to generate an initial feature sequence representation; The features of adjacent patches are clustered using the K-means clustering algorithm with spatial constraints. Adjacent patches with similar semantic features are merged into K clusters. The features of each cluster center are extracted to generate a clustered feature sequence. The clustered feature sequence is input into the ViT model for self-attention calculation and feedforward network processing to generate the final image feature representation.

[0010] In an optional implementation, the step of dividing the input image into a plurality of non-overlapping patches and performing embedding processing on each patch to generate an initial feature sequence representation includes: Get the input image ;in, is the number of image channels, is the spatial size of the image; When the size of each patch is , the input image is divided into N non-overlapping patches; among them, ; For each patch, flatten it into a one-dimensional vector with dimension ; The one-dimensional vector is embedded into a d-dimensional vector space using a linear mapping operation to generate an initial feature sequence representation.

[0011] In an optional implementation, the linear mapping operation is expressed as:

[0012] in, represents the flattened vector of the ith patch, is the weight matrix, is the bias vector, Represents the semantic embedding vector of the i-th patch after linear projection.

[0013] In an optional implementation, the features of adjacent patches are clustered using a K-means clustering algorithm with spatial constraints, adjacent patches with similar semantic features are merged into K clusters, and the features of each cluster center are extracted to generate a clustered feature sequence, including: Normalize the semantic embedding vectors of N patches; Based on the K-means clustering algorithm with spatial constraints, the spatial position of the patch is used as an additional feature dimension, and the Euclidean distance of the feature vector is used to aggregate patches with similar semantic features and merge them into K clusters; The average value of the semantic embedding vectors of all patches in each cluster is extracted as the class aggregation feature to generate a clustered feature sequence.

[0014] In an optional implementation, the method of clustering the features of adjacent patches using a K-means clustering algorithm with spatial constraints, merging adjacent patches with similar semantic features into K clusters, and extracting the features of each cluster center to generate a clustered feature sequence specifically includes: The semantic embedding vector of the i-th patch is normalized by the following formula:

[0015] in, is the mean of the semantic embedding vectors of N patches, is the corresponding standard deviation, Represents the standardized semantic embedding vector; The center coordinates of each patch Normalize it so that its range is between [0, 1]; in, ; Concatenate the patch’s normalized semantic embedding vector with its spatial position vector to construct a joint feature vector ;in, is a scaling factor; Randomly select the joint feature representation of K patches as the initial cluster center, denoted as ; Execute the cluster allocation process, including: for the joint feature vector of the i-th patch , through the formula Calculate the Euclidean distance between the patch and all cluster centers: assign the i-th patch to the cluster with the closest distance according to the Euclidean distance until all patches are assigned to K clusters; Execute the cluster center update process, including: updating the cluster center of the cluster by calculating the mean of the joint feature vectors of all patches in each cluster.

[0016] Repeat the cluster assignment process and class center update process until the maximum number of iterations is reached to end clustering; After clustering, for K clusters, take the average value of the semantic embedding vectors of all patches in each cluster as the class aggregation feature representation ;

[0017] in, represents the kth cluster The number of patches in Class-based aggregation feature representation , generate the clustered feature sequence.

[0018] In an optional embodiment, the clustered feature sequence is input into the ViT model for self-attention calculation and feedforward network processing to generate the final image feature representation, including: The clustered feature sequence is used as the input sequence of the subsequent ViT model, and a learnable token is added before the input sequence. , then the final input sequence is represented as ; The final input sequence is summed with the position encoding vector sequence and then input into the Transformer encoder to model long-distance dependencies, and the output algorithm is used to calculate the output of the L+1 layer encoder; After passing through L Transformer encoder layers, the output of the learnable token position is taken as the global feature representation and input into the fully connected layer to generate the predicted probability distribution for subsequent downstream tasks.

[0019] In an optional embodiment, the output algorithm includes:

[0020]

[0021] in, Indicates The output feature sequence of the encoder layer is Normalize the layers in the Transformer network structure. Represents the multi-head self-attention in the Transformer network structure, Represents the multi-layer perceptron in the Transformer network structure.

[0022] In a second aspect, the embodiment of the present application further provides a ViT model improvement system, including: The image patch segmentation and embedding module is used to divide the input image into multiple non-overlapping patches and embed each patch to generate an initial feature sequence representation; The similar semantic feature clustering module is used to cluster the features of adjacent patches using the K-means clustering algorithm with spatial constraints, merge adjacent patches with similar semantic features into K clusters, and extract the features of each cluster center to generate a clustered feature sequence; The Transformer processing module is used to input the clustered feature sequence into the ViT model for self-attention calculation and feedforward network processing to generate the final image feature representation.

[0023] In a third aspect, an embodiment of the present application further provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps of the ViT model improvement method as described in any one of the above items are implemented.

[0024] In a fourth aspect, an embodiment of the present application further provides a storage medium on which a computer program is stored, and when the computer program is executed by a processor, the steps of the ViT model improvement method as described in any one of the above items are implemented.

[0025] It can be seen from the above technical solutions that the present invention has the following advantages: In the ViT model improvement method provided in this application, by designing a method based on clustering of adjacent patch features, similar patch features can be merged into one token, reducing the length of the sequence input to the Transformer, reducing the computational complexity, and enhancing the ability to express local information.

[0026] This application divides the image into patches and embeds them, and then clusters the features of adjacent patches, which effectively reduces the number of features and reduces the computational complexity while retaining key semantic information, thereby improving the computational efficiency and feature representation capability of the ViT model.

[0027] In the clustering process, the spatial position of the patch is considered as an additional feature dimension, so that the clustering result takes into account both semantic similarity and spatial position relationship, which helps the model better understand the local and global structures in the image.

[0028] This application uses the clustered feature sequence as the input of the ViT model, which not only reduces the length of the input sequence and improves processing efficiency, but also adds a learnable token before the input sequence to extract the global feature representation, thereby enhancing the model's ability to capture the overall information of the image.

[0029] This application enables the ViT model to process image data more efficiently and better model long-distance dependencies in images, thereby showing higher performance and better generalization ability in downstream tasks such as image classification and object detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] In order to more clearly illustrate the technical solution of the present invention, the accompanying drawings required for use in the description will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0031] Figure 1 A flowchart of the ViT model improvement method provided in this application.

[0032] Figure 2 A schematic diagram of the structure of the ViT model improvement system provided in this application.

[0033] Figure 3 A schematic diagram of the structure of the electronic device provided in this application. DETAILED DESCRIPTION

[0034] In the specific steps of the ViT model improvement method described in detail below, various embodiments of the present disclosure will be described more comprehensively. The present disclosure may have various embodiments, and adjustments and changes may be made therein. However, it should be understood that there is no intention to limit the various embodiments of the present disclosure to the specific embodiments disclosed herein, but rather the present disclosure should be understood to cover all adjustments, equivalents and / or alternatives that fall within the spirit and scope of the various embodiments of the present disclosure.

[0035] Hereinafter, the terms "include" or "may include" used in various embodiments of the present disclosure indicate the presence of disclosed functions, operations, or elements, and do not limit the addition of one or more functions, operations, or elements. In addition, as used in various embodiments of the present disclosure, the terms "include", "have", and their cognates are intended only to indicate specific features, numbers, steps, operations, elements, components, or a combination of the foregoing, and should not be understood as first excluding the presence of one or more other features, numbers, steps, operations, elements, components, or a combination of the foregoing or the possibility of adding one or more features, numbers, steps, operations, elements, components, or a combination of the foregoing.

[0036] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0037] See also Figure 1 The figure is a method flow chart of a ViT model improvement method in a specific embodiment, the method comprising: S1: Divide the input image into multiple non-overlapping patches and embed each patch to generate an initial feature sequence representation.

[0038] In this step, the input image is preprocessed and divided into non-overlapping patches, and each patch is linearly mapped to obtain the semantic embedding representation of the initial patch.

[0039] S2: Use the K-means clustering algorithm with spatial constraints to cluster the features of adjacent patches, merge adjacent patches with similar semantic features into K clusters, and extract the features of each cluster center to generate a clustered feature sequence.

[0040] In this step, the similarity of the features of adjacent patches is calculated and similar patch features are clustered into a token so that it contains semantic information of a larger spatial area, thereby reducing the length of the input sequence.

[0041] For example, first, the semantic embedding vectors of N patches are normalized.

[0042] Then, based on the K-means clustering algorithm with spatial constraints, the spatial position of the patch is used as an additional feature dimension, and the Euclidean distance of the feature vector is used to cluster patches with similar semantic features and merge them into K clusters.

[0043] Finally, the average value of the semantic embedding vectors of all patches in each cluster is extracted as the class aggregation feature to generate a clustered feature sequence.

[0044] S3: The clustered feature sequence is input into the ViT model for self-attention calculation and feedforward network processing to generate the final image feature representation. In this step, the clustered new features are input into the Transformer for multi-head self-attention and feedforward network processing to obtain the final image feature representation for downstream tasks such as image classification.

[0045] In this embodiment, the input image is divided into patches and embedded, and then the adjacent patches with similar semantic features are merged using the K-means clustering algorithm with spatial constraints, which effectively reduces the number of features and retains key information. Finally, the clustered feature sequence is input into the ViT model, which improves the computational efficiency and feature representation ability of the model, thereby generating a more accurate and efficient image feature representation.

[0046] In an embodiment of the present invention, based on step S1, a possible embodiment is given below to illustrate its specific implementation scheme in a non-limiting manner.

[0047] S101: Obtain input image ;in, is the number of image channels, is the spatial size of the image.

[0048] As an example, given an input image ,in is the number of image channels, is the spatial size of the image.

[0049] S102: When the size of each patch is , the input image is divided into N non-overlapping patches; among them, .

[0050] As an example, divide the image into non-overlapping patches, assuming that the size of each patch is , then the total generation The calculation formula is as follows: .

[0051] S103: For each patch, flatten it into a one-dimensional vector with dimension .

[0052] S104: Embed the one-dimensional vector into a d-dimensional vector space using a linear mapping operation to generate an initial feature sequence representation.

[0053] Among them, the linear mapping operation is expressed as:

[0054] in, represents the flattened vector of the ith patch, is the weight matrix, is the bias vector, Represents the semantic embedding vector of the i-th patch after linear projection.

[0055] In this step, in order to map the flattened patch vector to a high-dimensional feature space, it is embedded into a d-dimensional vector space through a linear mapping layer.

[0056] In an embodiment of the present invention, based on step S2, a possible embodiment is given below to illustrate its specific implementation scheme in a non-limiting manner.

[0057] This embodiment discloses a similar semantic feature clustering method, which first normalizes the embedded feature vectors of N patches, and then uses K-means clustering to cluster patches with similar semantic features. In standard K-means clustering, only the Euclidean distance of the feature vector is used for clustering, and the patches in the image have a spatial position relationship. In order to make it more likely that adjacent patches will be clustered together, this method uses the spatial position of the patch as an additional feature dimension, and proposes a K-means clustering method with spatial constraints, which considers both feature similarity and spatial proximity in clustering. Specifically, it includes the following steps: S201: In order to accelerate the convergence of clustering, the features are standardized so that they obey a distribution with a mean of 0 and a variance of 1. The semantic embedding vector of the i-th patch is standardized using the following formula:

[0058] in, is the mean of the semantic embedding vectors of N patches, is the corresponding standard deviation, Represents the normalized semantic embedding vector.

[0059] S202: Center position coordinates of each patch Normalize it so that its range is between [0, 1]; in, .

[0060] S203: In order to allow clustering to consider both feature similarity and spatial proximity, we concatenate the normalized semantic embedding vector of the patch with its spatial position vector to construct a joint representation , is a scaling factor that balances feature similarity and spatial proximity.

[0061] S204: Randomly select the joint feature representation of K patches as the initial cluster center, denoted as .

[0062] S205: Use the K-means clustering method to iterate through the following process until the maximum number of iterations is reached.

[0063] The specific process includes: For the joint feature vector of the i-th patch , through the formula Calculate the Euclidean distance from all cluster centers.

[0064] The i-th patch is assigned to the cluster with the closest distance according to the Euclidean distance until all patches are assigned to K clusters, each with a specific cluster center.

[0065] The cluster centers are updated by calculating the mean of the joint feature vectors of all patches in each cluster.

[0066] Repeat the above process of assigning clusters and updating cluster centers until the maximum number of iterations is reached.

[0067] S206: After clustering, for the K clusters, take the average value of the semantic embedding vectors of all patches in each cluster as the class aggregation feature representation ;

[0068] in, represents the kth cluster The number of patches in .

[0069] Finally, based on the class aggregation feature representation , generate the clustered feature sequence.

[0070] In an embodiment of the present invention, based on step S3, a possible embodiment is given below to illustrate its specific implementation scheme in a non-limiting manner.

[0071] First, the feature sequence after K-class clustering is used as the input sequence of the subsequent Vision Transformer (ViT) model, and a learnable token is added before the input sequence , then the final input sequence is represented as .

[0072] Then, the final input sequence is summed with the position encoding vector sequence and input into the Transformer encoder to model long-distance dependencies. The output of the l+1 layer encoder is calculated as follows:

[0073]

[0074] in, Indicates The output feature sequence of the encoder layer is Normalize the layers in the Transformer network structure. Represents the multi-head self-attention in the Transformer network structure, Represents the multi-layer perceptron in the Transformer network structure.

[0075] After passing through L Transformer encoder layers, the output of the learnable token position is taken as the global feature representation, and then input into the fully connected layer to generate the predicted probability distribution for subsequent downstream tasks such as image classification and object detection.

[0076] like Figure 2 As shown, the following is an embodiment of the ViT model improvement system provided by the embodiment of the present disclosure. The system and the ViT model improvement method of the above-mentioned embodiments belong to the same inventive concept. For details not described in detail in the embodiment of the ViT model improvement system, please refer to the embodiment of the above-mentioned ViT model improvement method.

[0077] A ViT model improvement system includes: an image patch segmentation and embedding module 1, a similar semantic feature clustering module 2 and a Transformer processing module 3.

[0078] The image patch segmentation and embedding module 1 is used to divide the input image into multiple non-overlapping patches, embed each patch, and generate an initial feature sequence representation.

[0079] For example, the image patch segmentation and embedding module 1 is used to preprocess the input image, divide it into non-overlapping patches, perform linear mapping on each patch, and obtain the semantic embedding representation of the initial patch.

[0080] The similar semantic feature clustering module 2 is used to cluster the features of adjacent patches using the K-means clustering algorithm with spatial constraints, merge adjacent patches with similar semantic features into K clusters, and extract the features of each cluster center to generate a clustered feature sequence.

[0081] For example, the similar semantic feature clustering module 2 is used to calculate the similarity of the features of adjacent patches and cluster similar patch features into a token so that it contains semantic information of a larger spatial area, thereby reducing the length of the input sequence.

[0082] Transformer processing module 3 is used to input the clustered feature sequence into the ViT model for self-attention calculation and feedforward network processing to generate the final image feature representation.

[0083] For example, Transformer processing module 3 is used to input the clustered new features into Transformer for multi-head self-attention and feedforward network processing to obtain the final image feature representation for downstream tasks such as image classification.

[0084] The ViT model improvement system provided in this embodiment can merge similar patch features into one token, reduce the length of the sequence input to the Transformer, reduce the computational complexity, and enhance the ability to express local information.

[0085] Figure 3 A schematic diagram of the hardware structure of an electronic device for implementing various embodiments of the present invention.

[0086] The ViT model improvement method provided in the embodiment of the present application can be applied to electronic devices. Those skilled in the art will appreciate that the electronic device structure involved in the embodiment of the present invention does not constitute a limitation on the electronic device, and the electronic device may include more or fewer components than shown, or combine certain components, or arrange different components. In an embodiment of the present invention, the electronic device includes but is not limited to a laptop computer, a desktop computer, a workbench, a personal digital assistant, a server, a blade server, a mainframe computer, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the embodiments of the present application described and / or required herein.

[0087] The electronic device may include a processor, an external memory interface, an internal memory, a universal serial bus (USB) interface, a charging management module, a power management module, a battery, a wireless communication module, an audio module, a speaker, a microphone, a sensor module, buttons, a camera, a display, and a SIM card interface, etc.

[0088] The processor may include one or more processing units, for example, the processor may include a central processing unit (CPU), an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Among them, different processing units may be independent devices or integrated into one or more processors.

[0089] The processor can be the nerve center and command center of the electronic device. The controller can generate an operation control signal according to the instruction operation code and timing signal to complete the control of fetching and executing instructions.

[0090] A memory may also be provided in the processor for storing instructions and data. In some embodiments, the memory in the processor is a cache memory. The memory may store instructions or data that the processor has just used or is cyclically used. If the processor needs to use the instruction or data again, it may be directly called from the memory. This avoids repeated access, reduces the waiting time of the processor, and thus improves system efficiency.

[0091] The external memory interface can be used to connect an external memory card, such as a MicroSD card, to expand the storage capacity of the electronic device. The external memory card communicates with the processor through the external memory interface to implement data storage functions. For example, files such as music and videos can be saved in the external memory card.

[0092] The internal memory can be used to store computer executable program codes, which include instructions. The processor executes various functional applications and data processing of the electronic device by running the instructions stored in the internal memory. The internal memory may include a program storage area and a data storage area. The internal memory may include a high-speed random access memory and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc.

[0093] The wireless communication function of an electronic device can be realized through an antenna, a wireless communication module, a modem processor, and a baseband processor.

[0094] The wireless communication module can provide wireless communication solutions for electronic devices, including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared technology (IR), etc.

[0095] Electronic devices can implement audio functions, etc. through audio modules, speakers, receivers, microphones, headphone jacks, and application processors.

[0096] Electronic devices can achieve shooting functions through ISP, camera, video codec, GPU, display and application processor.

[0097] Electronic devices can achieve display functions through GPU, display screen and application processor.

[0098] The GPU is a microprocessor for image processing that connects the display screen and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor may include one or more GPUs that execute program instructions to generate or change display information.

[0099] The display screen is used to display images, videos, etc. The display screen includes a display panel.

[0100] The above-mentioned electronic device implements the architectural ViT model improvement method of the present application by improving the input processing method and clustering method of the ViT model, effectively improving the computational efficiency and feature representation ability of the model, while integrating spatial position information and optimizing the input of the model, ultimately improving the performance of the model in downstream tasks such as image classification.

[0101] The storage medium provided in the present application stores a program product that can implement the ViT model improvement method.

[0102] The ViT model improvement method includes: dividing the input image into multiple non-overlapping patches, and embedding each patch to generate an initial feature sequence representation; clustering the features of adjacent patches using a K-means clustering algorithm with spatial constraints, merging adjacent patches with similar semantic features into K clusters, extracting the features of each cluster center to generate a clustered feature sequence; inputting the clustered feature sequence into the ViT model for self-attention calculation and feedforward network processing to generate the final image feature representation.

[0103] In some possible implementations, the ViT model improvement method of the present disclosure may be implemented in the form of a program product, which includes a program code. When the program product is run on a terminal device, the program code is used to enable the terminal device to execute the steps described in the above “Exemplary Method” section of this specification according to various exemplary implementations of the present disclosure.

[0104] The storage medium of the present disclosure can adopt any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium can be, for example, but not limited to, a system, device or device of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0105] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A ViT model improvement method, characterized in that: include: The input image is divided into multiple non-overlapping patches, and each patch is embedded to generate an initial feature sequence representation; The features of adjacent patches are clustered using the K-means clustering algorithm with spatial constraints. Adjacent patches with similar semantic features are merged into K clusters. The features of each cluster center are extracted to generate a clustered feature sequence. The clustered feature sequence is input into the ViT model for self-attention calculation and feedforward network processing to generate the final image feature representation.

2. The ViT model improvement method according to claim 1, characterized in that: The input image is divided into multiple non-overlapping patches, and each patch is embedded to generate an initial feature sequence representation, including: Get the input image ;in, is the number of image channels, is the spatial size of the image; When the size of each patch is , the input image is divided into N non-overlapping patches; among them, ; For each patch, flatten it into a one-dimensional vector with dimension The one-dimensional vector is embedded into a d-dimensional vector space using a linear mapping operation to generate an initial feature sequence representation.

3. The ViT model improvement method according to claim 2, characterized in that: The linear mapping operation is expressed as: in, represents the flattened vector of the ith patch, is the weight matrix, is the bias vector, Represents the semantic embedding vector of the i-th patch after linear projection.

4. The ViT model improvement method according to claim 3, characterized in that: The K-means clustering algorithm with spatial constraints is used to cluster the features of adjacent patches, adjacent patches with similar semantic features are merged into K clusters, and the features of each cluster center are extracted to generate a clustered feature sequence, including: Normalize the semantic embedding vectors of N patches; Based on the K-means clustering algorithm, the spatial position of the patch is used as an additional feature dimension, and the Euclidean distance of the feature vector is used to aggregate patches with similar semantic features and merge them into K clusters; The average value of the semantic embedding vectors of all patches in each cluster is extracted as the class aggregation feature to generate a clustered feature sequence.

5. The ViT model improvement method according to claim 4, characterized in that: The method of clustering the features of adjacent patches using a K-means clustering algorithm with spatial constraints, merging adjacent patches with similar semantic features into K clusters, and extracting the features of each cluster center to generate a clustered feature sequence specifically includes: The semantic embedding vector of the i-th patch is normalized by the following formula: in, is the mean of the semantic embedding vectors of N patches, is the corresponding standard deviation, Represents the standardized semantic embedding vector; The center coordinates of each patch Normalize it so that its range is between [0, 1]; in, ; Concatenate the normalized semantic embedding vector of the patch with its spatial position vector to construct a joint feature vector ;in, is a scaling factor; Randomly select the joint feature representation of K patches as the initial cluster center, denoted as ; Execute the cluster allocation process, including: for the joint feature vector of the i-th patch , through the formula Calculate the Euclidean distance between the patch and all cluster centers: assign the i-th patch to the cluster with the closest distance according to the Euclidean distance until all patches are assigned to K clusters; Execute the cluster center update process, including: updating the cluster center of the cluster by calculating the mean of the joint feature vectors of all patches in each cluster; Repeat the cluster assignment process and class center update process until the maximum number of iterations is reached to end clustering; After clustering, for K clusters, take the average value of the semantic embedding vectors of all patches in each cluster as the class aggregation feature representation ; in, represents the kth cluster The number of patches in Class-based aggregation feature representation , generate the clustered feature sequence.

6. The ViT model improvement method according to claim 5, characterized in that: The clustered feature sequence is input into the ViT model for self-attention calculation and feedforward network processing to generate the final image feature representation, including: The clustered feature sequence is used as the input sequence of the subsequent ViT model, and a learnable token is added before the input sequence. , then the final input sequence is represented as ; The final input sequence is summed with the position encoding vector sequence and then input into the Transformer encoder to model long-distance dependencies, and the output algorithm is used to calculate the output of the L+1 layer encoder; After passing through L Transformer encoder layers, the output of the learnable token position is taken as the global feature representation and input into the fully connected layer to generate the predicted probability distribution for subsequent downstream tasks.

7. The ViT model improvement method according to claim 6, characterized in that: The output algorithm includes: in, Indicates The output feature sequence of the encoder layer is Normalize the layers in the Transformer network structure. Represents the multi-head self-attention in the Transformer network structure, Represents the multi-layer perceptron in the Transformer network structure.

8. A ViT model improvement system, characterized in that: The system adopts the ViT model improvement method as claimed in any one of claims 1 to 7; The system comprises: The image patch segmentation and embedding module is used to divide the input image into multiple non-overlapping patches and embed each patch to generate an initial feature sequence representation; The similar semantic feature clustering module is used to cluster the features of adjacent patches using the K-means clustering algorithm with spatial constraints, merge adjacent patches with similar semantic features into K clusters, and extract the features of each cluster center to generate a clustered feature sequence; The Transformer processing module is used to input the clustered feature sequence into the ViT model for self-attention calculation and feedforward network processing to generate the final image feature representation.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the steps of the ViT model improvement method according to any one of claims 1 to 7 are implemented.

10. A storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the ViT model improvement method according to any one of claims 1 to 7 are implemented.

Images