VIT model training and image classification method

Abstract

The embodiment of the invention provides a VIT model training and image classification method. The method comprises the following steps: acquiring a first sample image; for each first window of the first sample image, removing a first number of first tokens from the first window, and taking the remaining first tokens as a first model input result; inputting the first model input result into the original VIT model to obtain a first classification result output by the original VIT model; calculating a first target loss based on the first classification result and a first sample label corresponding to the first sample image; and adjusting model parameters of the original VIT model according to the target loss, returning to the step of inputting the first model input result into the original VIT model to obtain a first classification result output by the original VIT model, and continuing execution until a preset convergence condition is met, thereby obtaining a trained VIT model. By applying the scheme provided by the embodiment of the invention, the reasoning efficiency of the VIT model can be improved.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to a VIT model training and image classification method. Background Art

[0002] VIT (Vision Transformer) is a visual model based on the Transformer architecture. It introduces the self-attention mechanism into the field of computer vision, breaking the traditional CNN (Convolutional Neural Networks)-dominated situation. It can achieve comparable or even better results than CNN in processing image data.

[0003] The core operation of Transformer is Self-attention. The Self-attention operation performs attention calculation on all tokens (vectors). The calculation process is complex and time-consuming, which limits the applicability of VIT in actual scenarios.

[0004] Therefore, it is necessary to provide a VIT model training method to improve the reasoning efficiency of the VIT model. Summary of the invention

[0005] The purpose of the embodiment of the present invention is to provide a VIT model training and image classification method to improve the reasoning efficiency of the VIT model. The specific technical solution is as follows:

[0006] In a first aspect of an embodiment of the present invention, a VIT model training method is first provided, the method comprising:

[0007] Acquire a first sample image;

[0008] For each first window of the first sample image, a first number of first vector tokens are removed from the first window, and the remaining first tokens are used as first model input results; wherein the first tokens included in each first window are adjacent to each other in the first sample image, and the first number is less than the total number of first tokens included in one first window;

[0009] Inputting the first model input result into the original visual converter VIT model to obtain a first classification result output by the original VIT model;

[0010] Calculating a first target loss based on the first classification result and a first sample label corresponding to the first sample image, wherein the first sample label indicates the category to which the first sample image belongs;

[0011] The model parameters of the original VIT model are adjusted according to the target loss, and the step of inputting the first model input result into the original VIT model to obtain the first classification result output by the original VIT model is returned to continue execution until a preset convergence condition is met to obtain a trained VIT model; the target loss is determined based on the first target loss.

[0012] In a possible embodiment, the method further includes:

[0013] Acquire a second sample image, where the second sample image has the same size as the first sample image;

[0014] For each first window of the first sample image, remove a second number of first tokens from the first window; wherein the second number is less than the total number of first tokens included in one first window;

[0015] For each second window of the second sample image, remove a third number of second tokens from the second window; wherein the second window of the second sample image corresponds one-to-one to the first window in the first sample image, and for the corresponding first window and second window, the position of the first token contained in the first window in the first sample image is the same as the position of the second token contained in the second window in the second sample image, and the third number is less than the total number of second tokens contained in one second window; the sum of the second number and the third number is equal to the total number of the second tokens; and the total number of the first tokens is equal to the total number of the second tokens;

[0016] Superimposing the remaining first target tokens in the first sample image and the remaining second target tokens in the second sample image to obtain a second model input result;

[0017] Fusing the first sample label with the second sample label corresponding to the second sample image to obtain a first fused sample label; wherein the second sample label indicates the category to which the second sample image belongs;

[0018] Inputting the second model input result into the original VIT model to obtain a second classification result output by the original VIT model;

[0019] Calculate a second target loss based on the second classification result and the first fusion sample label;

[0020] Calculating a target loss according to the first target loss and the second target loss;

[0021] The step of adjusting the model parameters of the original VIT model according to the target loss and returning to the step of inputting the first model input result into the original VIT model to obtain the first classification result output by the VIT model includes:

[0022] Adjust the model parameters of the original VIT model according to the target loss, and return to the steps of inputting the first model input result into the original VIT model to obtain the first classification result output by the original VIT model, and inputting the second model input result into the original VIT model to obtain the second classification result output by the original VIT model to continue execution.

[0023] In a possible embodiment, the position of the first target token in the first sample image is the same as the position of the second token removed from the second sample image in the second sample image.

[0024] In a possible embodiment, the method further includes:

[0025] Acquire a third sample image, where the third sample image has the same size as the first sample image;

[0026] Fusing the first sample image with the third sample image to obtain a third model input result;

[0027] Fusing the first sample label and the third sample label corresponding to the third sample image to obtain a second fused sample label, wherein the third sample label indicates the category to which the third sample image belongs;

[0028] Inputting the third model input result into the original VIT model to obtain a third classification result output by the original VIT model;

[0029] Calculating a third target loss based on the third classification result and the second fusion sample label;

[0030] Calculating a target loss according to the first target loss and the third target loss;

[0031] The step of adjusting the model parameters of the original VIT model according to the target loss and returning to the step of inputting the first model input result into the original VIT model to obtain the first classification result output by the VIT model includes:

[0032] Adjust the model parameters of the original VIT model according to the target loss, and return to the steps of inputting the first model input result into the original VIT model to obtain the first classification result output by the original VIT model, and inputting the third model input result into the original VIT model to obtain the third classification result output by the original VIT model to continue execution.

[0033] In a possible embodiment, for each first window of the first sample image, removing a first number of first tokens from the first window, and using the remaining first tokens as the first model input result, includes:

[0034] Arrange the first tokens belonging to the same first window in the same row of the token matrix;

[0035] For each row of the token matrix, a first number of first tokens are removed from the row, and the remaining first tokens are used as first model input results.

[0036] In a second aspect of the embodiments of the present invention, an image classification method is further provided, the method comprising:

[0037] Get the image to be classified;

[0038] For each window in the image to be classified, removing a first number of vector tokens from the window;

[0039] Input the remaining tokens into a pre-trained visual converter VIT model to obtain a classification result of the image to be classified, wherein the pre-trained VIT model is trained by any one of the method steps described in the first aspect above;

[0040] The first number is equal to the number of first tokens taken from the first sample image when obtaining the first model input result during the process of training the VIT model.

[0041] In a third aspect of the embodiments of the present invention, a VIT model training device is also provided, the device comprising:

[0042] A first acquisition module, used for acquiring a first sample image;

[0043] A first removal module is used for removing a first number of first vector tokens from each first window of the first sample image, and using the remaining first tokens as first model input results; wherein the first tokens contained in each first window are adjacent to each other in the first sample image, and the first number is less than the total number of first tokens contained in one first window;

[0044] A first input module, used for inputting the first model input result into the original visual converter VIT model to obtain a first classification result output by the original VIT model;

[0045] A first calculation module, configured to calculate a first target loss based on the first classification result and a first sample label corresponding to the first sample image; the first sample label indicates the category to which the first sample image belongs;

[0046] A parameter adjustment module is used to adjust the model parameters of the original VIT model according to the target loss, return to trigger the execution of the first input module until a preset convergence condition is met, and obtain the trained VIT model; the target loss is determined based on the first target loss.

[0047] In a possible embodiment, the device further includes:

[0048] A second acquisition module, used for acquiring a second sample image, where the second sample image has the same size as the first sample image;

[0049] A second removal module is used to remove a second number of first tokens from each first window of the first sample image; wherein the second number is less than the total number of first tokens included in one first window;

[0050] a third removal module, configured to remove a third number of second tokens from each second window of the second sample image; wherein the second window of the second sample image corresponds one-to-one to the first window in the first sample image, and for the corresponding first window and second window, the position of the first token contained in the first window in the first sample image is the same as the position of the second token contained in the second window in the second sample image, and the third number is less than the total number of second tokens contained in one second window; the sum of the second number and the third number is equal to the total number of the second tokens; and the total number of the first tokens is equal to the total number of the second tokens;

[0051] a vector superposition module, used to superimpose the remaining first target tokens in the first sample image with the remaining second target tokens in the second sample image to obtain a second model input result;

[0052] a first fusion module, configured to fuse the first sample label with a second sample label corresponding to the second sample image to obtain a first fused sample label; wherein the second sample label indicates the category to which the second sample image belongs;

[0053] A second input module, used for inputting the second model input result into the original VIT model to obtain a second classification result output by the original VIT model;

[0054] A second calculation module, used to calculate a second target loss based on the second classification result and the first fusion sample label;

[0055] A third calculation module, configured to calculate a target loss according to the first target loss and the second target loss;

[0056] The parameter adjustment module is specifically used for:

[0057] The model parameters of the original VIT model are adjusted according to the target loss, and the first input module and the second input module are triggered back.

[0058] In a possible embodiment, the position of the first target token in the first sample image is the same as the position of the second token removed from the second sample image in the second sample image.

[0059] In a possible embodiment, the device further includes:

[0060] A third acquisition module, used for acquiring a third sample image, wherein the third sample image has the same size as the first sample image;

[0061] A second fusion module, used for fusing the first sample image with the third sample image to obtain a third model input result;

[0062] A third fusion module, configured to fuse the first sample label and a third sample label corresponding to the third sample image to obtain a second fused sample label, wherein the third sample label indicates the category to which the third sample image belongs;

[0063] A third input module, used for inputting the third model input result into the original VIT model to obtain a third classification result output by the original VIT model;

[0064] A fourth calculation module, used to calculate a third target loss based on the third classification result and the second fusion sample label;

[0065] a fifth calculation module, configured to calculate a target loss according to the first target loss and the third target loss;

[0066] The parameter adjustment module is specifically used for:

[0067] The model parameters of the original VIT model are adjusted according to the target loss, and the first input module and the third input module are triggered to execute.

[0068] In a possible embodiment, the first removal module is specifically configured to:

[0069] Arrange the first tokens belonging to the same first window in the same row of the token matrix;

[0070] For each row of the token matrix, a first number of first tokens are removed from the row, and the remaining first tokens are used as first model input results.

[0071] In a fourth aspect of the embodiments of the present invention, an image classification device is further provided, the device comprising:

[0072] An image acquisition module, used to acquire images to be classified;

[0073] A vector removal module, configured to remove a first number of vector tokens from each window in the image to be classified;

[0074] A vector input module, used for inputting the remaining tokens into a pre-trained visual converter VIT model to obtain a classification result of the image to be classified, wherein the pre-trained VIT model is trained by any one of the method steps described in the first aspect above;

[0075] The first number is equal to the number of first tokens taken from the first sample image when obtaining the first model input result during the process of training the VIT model.

[0076] In a fifth aspect of the embodiments of the present application, there is further provided an electronic device, including a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus;

[0077] Memory, used to store computer programs;

[0078] The processor is used to implement any method step described in the first aspect when executing the program stored in the memory.

[0079] In a sixth aspect of the embodiments of the present application, there is further provided an electronic device, including a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus;

[0080] Memory, used to store computer programs;

[0081] The processor is used to implement the method steps described in the second aspect when executing the program stored in the memory.

[0082] In another aspect of the embodiments of the present application, a computer-readable storage medium is provided, wherein a computer program is stored in the computer-readable storage medium. When the computer program is executed by a processor, the method steps described in any one of the first aspect or the second aspect are implemented.

[0083] Beneficial effects of the embodiments of the present invention:

[0084] In the VIT model training method provided by an embodiment of the present invention, since the positions of the first tokens contained in each first window in the first sample image are adjacent to each other, and the first number is less than the total number of first tokens contained in a first window, for each first window of the first sample image, a first number of first vector tokens are removed from the first window, and the remaining first tokens are used as the first model input result, so that some of the first tokens in each first window of the first sample image can be eliminated.

[0085] The first model input result is input into the original VIT model to obtain a first classification result output by the VIT model, and a first target loss is calculated based on the first classification result and a first sample label corresponding to the first sample image. The model parameters of the original VIT model are adjusted according to the target loss determined by the first target loss, and the original VIT model is continuously trained until a preset convergence condition is met, thereby completing the training process of the VIT model.

[0086] Since some of the first tokens have been eliminated before the first model input results are input into the original VIT model, the VIT model does not need to screen and eliminate the first tokens, which reduces the time of eliminating the first tokens layer by layer according to self-attention in the VIT model, thereby improving the reasoning efficiency of the VIT model.

[0087] Of course, it is not necessary to achieve all of the advantages described above at the same time to implement any product or method of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0088] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other embodiments can also be obtained based on these drawings.

[0089] Figure 1 A schematic diagram of a flow chart of a first VIT model training method provided by an embodiment of the present invention;

[0090] Figure 2 A schematic diagram of a flow chart of a second VIT model training method provided in an embodiment of the present invention;

[0091] Figure 3 A position comparison diagram provided by an embodiment of the present invention;

[0092] Figure 4 A schematic diagram of a flow chart of a third VIT model training method provided by an embodiment of the present invention;

[0093] Figure 5 A schematic diagram of a fourth VIT model training method provided by an embodiment of the present invention;

[0094] Figure 6 A schematic diagram of a fifth VIT model training method provided by an embodiment of the present invention;

[0095] Figure 7 A schematic diagram of a flow chart of an image classification method provided by an embodiment of the present invention;

[0096] Figure 8 A schematic diagram of the structure of a VIT model training device provided by an embodiment of the present invention;

[0097] Fig. 9 A schematic diagram of the structure of an image classification device provided by an embodiment of the present invention;

[0098] Fig.10 A schematic diagram of the structure of a first electronic device provided by an embodiment of the present invention;

[0099] Fig.11 A schematic structural diagram of a second electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0100] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field based on the present invention belong to the scope of protection of the present invention.

[0101] The present invention provides a VIT model training method, see Figure 1 , is a flow chart of a first VIT model training method provided in an embodiment of the present invention, the method includes S101-S105.

[0102] S101, obtaining a first sample image.

[0103] In S101, the first sample image may be a photo taken by a camera, a mobile phone or other professional equipment, or a picture obtained from a video frame, or a picture downloaded from the Internet. The first sample image may display an object, an animal, a plant, etc., such as a car, a cat, or a willow tree.

[0104] The size of the first sample image may be 224 pixels×224 pixels, or may be other sizes such as 200 pixels×200 pixels. The embodiment of the present invention does not specifically limit the size of the first sample image.

[0105] S102: For each first window of the first sample image, remove a first number of first tokens from the first window, and use the remaining first tokens as first model input results.

[0106] In S102, for each first window of the first sample image, a first number of first tokens may be randomly removed from the first window, or a first number of first tokens may be removed from the first window at intervals of a preset number. For example, the first window 1 contains 14 first tokens, namely x1, x2, x3, x4, ..., x 14 , you can randomly remove x1, x2, x3, x4, ..., x7, or you can randomly remove x1, x3, x5, x7, x9, x 11 , x 13 Remove.

[0107] The first tokens contained in each first window are located adjacent to each other in the first sample image. For example, it is assumed that the number of first tokens is 196, namely x1, x2, x3, x4, ..., x 195 , x 196 , where the 196 first tokens are arranged in a 14×14 pattern, with the first row being x1, x2, x3, x4, …, x 14 , the second line is x 15 , x 16 , x 17 , x 18 , …, x 28 , the third line is x 29 , x 30 , x 31 , x 32 , …, x 42 , since x1, x2, x 15 , x 16 The positions of x1, x2, x3 are adjacent in the first sample image. 15 , x 16 In the same first window, x3, x4, x 17 , x 18In the first sample image, the positions of x3, x4, x 17 , x 18 In the same first window, and so on.

[0108] The first number is less than the total number of first tokens contained in a first window. Exemplarily, assuming that the total number of first tokens contained in a first window is M, the first number is less than M. For example, M=4, then the first number is less than 4, that is, the first number can be 1, 2, or 3, which means that for each first window of the first sample image, 1 first token can be removed from the first window, 2 first tokens can be removed from the first window, or 3 first tokens can be removed from the first window.

[0109] In a possible embodiment, a position code is added to the first token to obtain a new first token.

[0110] Specifically, it is assumed that the first sample image can be expressed by the following formula:

[0111] X i =[x1, x2, x, x n ]

[0112] in,… i Indicates that the first sample image is the i-th image, x1, x2, ..., x n They are n first tokens respectively.

[0113] Add E to the first token pos After that, the first sample image can be expressed by the following formula:

[0114] X i =[x1, x2, …, x n ]+E pos

[0115] Among them, E pos Encode the position.

[0116] Assume n = 196, rearrange the 196 first tokens in a 14×14 manner, and get:

[0117]

[0118] From the above formula, we can see that x1, x2, x 15 , x 16 The positions are adjacent and in the same first window.

[0119] S103, inputting the first model input result into the original VIT model to obtain the first classification result output by the original VIT model.

[0120] In S103, the original VIT model may be created based on MAE-B (Masked Autoencoders-Base) or MAE-L (Masked Autoencoders-Large).

[0121] S104: Calculate a first target loss based on the first classification result and a first sample label corresponding to the first sample image.

[0122] In S104, the first sample label indicates the category to which the first sample image belongs. For example, assuming that the content displayed by the first sample image is a moving jeep, the first sample label is jeep.

[0123] The first target loss is used to measure the difference between the first classification result and the first sample label. The smaller the first target loss, the smaller the difference between the first classification result and the first sample label.

[0124] S105, adjusting the model parameters of the original VIT model according to the target loss.

[0125] After adjusting the model parameters of the original VIT model according to the target loss, it is determined whether the preset convergence condition is met. If so, the trained VIT model is obtained; if not, the process returns to step S103 and continues to execute.

[0126] The target loss is determined based on the first target loss.

[0127] In S105, the first target loss is the target loss. The preset convergence condition may be that the number of training times of the original VIT model reaches a preset number of times, or that the target loss is less than a preset loss threshold, or other convergence conditions, etc. The embodiment of the present invention does not specifically limit the preset convergence condition of the original VIT model.

[0128] The above embodiment is selected. Since the positions of the first tokens contained in each first window in the first sample image are adjacent to each other and the first number is less than the total number of first tokens contained in a first window, for each first window of the first sample image, a first number of first vector tokens are removed from the first window, and the remaining first tokens are used as the first model input result, so that some of the first tokens in each first window of the first sample image can be eliminated.

[0129] The first model input result is input into the original VIT model to obtain a first classification result output by the VIT model, and a first target loss is calculated based on the first classification result and a first sample label corresponding to the first sample image. The model parameters of the original VIT model are adjusted according to the target loss determined by the first target loss, and the original VIT model is continuously trained until a preset convergence condition is met, thereby completing the training process of the VIT model.

[0130] Since some of the first tokens have been eliminated before the first model input results are input into the original VIT model, the VIT model does not need to screen and eliminate the first tokens, which reduces the time of eliminating the first tokens layer by layer according to self-attention in the VIT model, thereby improving the reasoning efficiency of the VIT model.

[0131] In one possible embodiment, see Figure 2 , is a flow chart of a second VIT model training method provided by an embodiment of the present invention, which is similar to the aforementioned Figure 1 Compared with the embodiment shown, the above method further includes S201-S208, and the above S105 can be implemented through S105A.

[0132] S201, acquiring a second sample image.

[0133] The second sample image has the same size as the first sample image. For example, assuming that the size of the first sample image is 224 pixels×224 pixels, the size of the second sample image is also 224 pixels×224 pixels.

[0134] The category to which the second sample image belongs may be the same as the category to which the first sample image belongs. For example, if the category to which the first sample image belongs is jeep, the category to which the second sample image belongs is also jeep.

[0135] The category to which the second sample image belongs may be different from the category to which the first sample image belongs. For example, the category to which the first sample image belongs is a jeep, and the category to which the second sample image belongs is a sedan. The category to which the second sample image belongs may also be a cat, a dog, etc., which are not related to cars.

[0136] S202: For each first window of the first sample image, remove a second number of first tokens from the first window.

[0137] The second number is less than the total number of first tokens included in a first window.

[0138] Exemplarily, assuming that the total number of first tokens contained in a first window is M, the second number is less than M. For example, M=4, then the second number is less than 4, that is, the second number can be 1, 2, or 3, which means that for each first window of the first sample image, 1 first token can be removed from the first window, 2 first tokens can be removed from the first window, or 3 first tokens can be removed from the first window.

[0139] It should be noted that the first quantity and the second quantity may be the same or different.

[0140] S203: For each second window of the second sample image, remove a third number of second tokens from the second window.

[0141] Among them, the second window of the second sample image corresponds one-to-one to the first window in the first sample image, and for the corresponding first window and second window, the position of the first token contained in the first window in the first sample image is the same as the position of the second token contained in the second window in the second sample image.

[0142] Exemplarily, it is assumed that the first windows in the first sample image are first window 1, first window 2, first window 3, and first window 4. First window 1 is located at the upper left corner of the first sample image, first window 2 is located at the upper right corner of the first sample image, first window 3 is located at the lower left corner of the first sample image, and first window 4 is located at the lower right corner of the first sample image.

[0143] The first window 1 includes the first token 1, the first token 2, the first token 3, and the first token 4. The first token 1 is located at the upper left corner of the first window 1, the first token 2 is located at the upper right corner of the first window 1, the first token 3 is located at the lower left corner of the first window 1, and the first token 4 is located at the lower right corner of the first window 1.

[0144] Assume that the second windows in the second sample image are second window 1, second window 2, second window 3, and second window 4. Second window 1 is located at the upper left corner of the second sample image, second window 2 is located at the upper right corner of the second sample image, second window 3 is located at the lower left corner of the second sample image, and second window 4 is located at the lower right corner of the second sample image.

[0145] The second window 1 includes the second token 1, the second token 2, the second token 3, and the second token 4. The second token 1 is located at the upper left corner of the second window 1, the second token 2 is located at the upper right corner of the second window 1, the second token 3 is located at the lower left corner of the second window 1, and the second token 4 is located at the lower right corner of the second window 1.

[0146] Among them, the first window 1 corresponds to the second window 1 , the first window 2 corresponds to the second window 2 , the first window 3 corresponds to the second window 3 , and the first window 4 corresponds to the second window 4 .

[0147] The position of the first token1 contained in the first window 1 in the first sample image is the same as the position of the second token1 contained in the second window 1 in the second sample image. The position of the first token2 contained in the first window 1 in the first sample image is the same as the position of the second token2 contained in the second window 1 in the second sample image. And so on.

[0148] In addition, the third number is less than the total number of second tokens included in a second window; the sum of the second number and the third number is equal to the total number of second tokens; and the total number of first tokens is equal to the total number of second tokens.

[0149] Exemplarily, assuming that the total number of first tokens contained in a second window is M, the third number is less than M. For example, M=4, then the third number is less than 4, that is, the third number can be 1, 2, or 3. Theoretically, for each second window of the second sample image, 1 first token can be removed from the second window, 2 first tokens can be removed from the first window, or 3 first tokens can be removed from the first window.

[0150] However, since the sum of the second number and the third number is equal to the total number of second tokens contained in a window, for the corresponding first window and second window, assuming that 1 first token is removed from the first window, 4-1=3 second tokens are removed from the second window. Assuming that 2 first tokens are removed from the first window, 4-2=2 second tokens are removed from the second window. Assuming that 3 second tokens are removed from the first window, 4-3=1 second token is removed from the second window.

[0151] S204: Superimpose the remaining first target tokens in the first sample image and the remaining second target tokens in the second sample image to obtain a second model input result.

[0152] Exemplarily, after removing the second number of first tokens from the first window and removing the third number of second tokens from the second window, the second model input result can be expressed by the following formula:

[0153]

[0154] in, represents the second model input result, Xi Indicates that the first sample image is the i-th image, X j It indicates that the second sample image is the jth image, r indicates the number of first tokens removed from a first window, and z indicates the total number of first tokens contained in a first window.

[0155] In a possible embodiment, the position of the first target token in the first sample image is the same as the position of the second token removed from the second sample image in the second sample image.

[0156] See also Figure 3 , which is a position comparison diagram provided by an embodiment of the present invention.

[0157] The gray square represents the position of the first target token in the first sample image, and the white square represents the position of the first token removed from the first sample image in the first sample image.

[0158] The black square indicates the position of the second target token in the second sample image, and the white square indicates the position of the second token removed from the second sample image in the second sample image.

[0159] from Figure 3 It can be seen that the position of the gray square in the first sample image is the same as the position of the white square in the second sample image. It can also be understood that the position of the first target token in the first sample image is complementary to the position of the second target token in the second sample image.

[0160] By selecting the above embodiment, the position of the first target token in the first sample image is the same as the position of the second token removed from the second sample image in the second sample image, so that the superimposed second model input result has no blurred area and is more beautiful.

[0161] S205: Fuse the first sample label with the second sample label corresponding to the second sample image to obtain a first fused sample label.

[0162] The second sample label indicates the category to which the second sample image belongs.

[0163] For example, assuming that 1 first token is removed from the first window and 4-1=3 second tokens are removed from the second window, the first sample image shows a cat and the second sample image shows a dog, then the probability that the first fusion sample label is a cat is 25%, and the probability that it is a dog is 75%.

[0164] Exemplarily, the first fusion sample label can be expressed by the following formula:

[0165]

[0166] in, represents the first fusion sample label, Y i Indicates that the first sample label is the label of the i-th image, Y j Indicates that the second sample label is the label of the jth image, r indicates the number of first tokens removed from a first window, and z indicates the total number of first tokens contained in a first window.

[0167] S206, inputting the second model input result into the original VIT model to obtain the second classification result output by the original VIT model.

[0168] S207, calculating a second target loss based on the second classification result and the first fusion sample label.

[0169] The second target loss is used to measure the difference between the second classification result and the first fusion sample label. The smaller the second target loss, the smaller the difference between the second classification result and the first fusion sample label.

[0170] For example, the second objective loss is a regularization loss and can be expressed as follows:

[0171]

[0172] in, represents the second objective loss, X i Indicates that the first sample image is the i-th image, X j Indicates that the second sample image is the jth image, Y i Indicates that the first sample label is the label of the i-th image, Y j Indicates that the second sample label is the label of the jth image, r indicates the number of first tokens removed from a first window, and z indicates the total number of first tokens contained in a first window.

[0173] S208: Calculate the target loss according to the first target loss and the second target loss.

[0174] In S208, the target loss is related to the first target loss, the second target loss, and the weight parameter of the second target loss.

[0175] Exemplarily, the relationship between the target loss and the first target loss and the second target loss is:

[0176] Loss=Loss ce +αLoss mixtoken

[0177] Among them, Loss is the target loss, Lossce is the first target loss, Loss mixtoken is the second objective loss, and α is the weight parameter of the second objective loss.

[0178] S105A, adjust the model parameters of the original VIT model according to the target loss.

[0179] After adjusting the model parameters of the original VIT model according to the target loss, it is determined whether the preset convergence condition is met. If so, the trained VIT model is obtained; if not, the process returns to step S103 and S206 to continue execution.

[0180] In S105A, the order of inputting the first model input result into the original VIT model and the order of inputting the second model input result into the original VIT model can be to input the first model input result into the original VIT model first and then input the second model input result into the original VIT model; or to input the second model input result into the original VIT model first and then input the first model input result into the original VIT model.

[0181] The embodiment of the present invention does not specifically limit the order in which the first model input result and the second model input result are input into the original VIT model.

[0182] By selecting the above embodiment, since the second target loss is calculated based on the second classification result and the first fused sample label, the second model input result is obtained by superimposing the remaining first target token in the first sample image and the remaining second target token in the second sample image, and the first fused sample label is obtained by fusing the first sample label with the second sample label corresponding to the second sample image, therefore, adjusting the model parameters of the original VIT model according to the target loss calculated by the first target loss and the second target loss can make the trained VIT model more accurate.

[0183] In one possible embodiment, see Figure 4 , is a flow chart of a third VIT model training method provided by an embodiment of the present invention, which is similar to the aforementioned Figure 1 Compared with the embodiment shown, the above method further includes S401-S406, and the above S105 can be implemented through S105B.

[0184] S401, obtaining a third sample image.

[0185] The third sample image has the same size as the first sample image. For example, assuming that the size of the first sample image is 224 pixels×224 pixels, the size of the third sample image is also 224 pixels×224 pixels.

[0186] The category to which the third sample image belongs may be the same as the category to which the first sample image belongs. For example, if the category to which the first sample image belongs is jeep, the category to which the third sample image belongs is also jeep.

[0187] The category to which the third sample image belongs may be different from the category to which the first sample image belongs. For example, the category to which the first sample image belongs is a jeep, and the category to which the third sample image belongs is a sedan. The category to which the third sample image belongs may also be a cat, a dog, etc., which are not related to cars.

[0188] S402: Fusing the first sample image with the third sample image to obtain a third model input result.

[0189] In S402, fusing the first sample image with the third sample image refers to fusing the first sample image with the third sample image according to a preset weight ratio.

[0190] Exemplarily, the formula for fusing the first sample image with the third sample image is as follows:

[0191]

[0192] in, represents the input result of the third model, x i Indicates that the first sample image is the i-th image, x j It indicates that the third sample image is the jth image, m indicates the fusion weight parameter, and the weight parameter conforms to the Beta distribution or other distribution methods.

[0193] S403: Fuse the first sample label with the third sample label corresponding to the third sample image to obtain a second fused sample label.

[0194] The third sample label indicates the category to which the third sample image belongs.

[0195] Exemplarily, the formula for fusing the first sample image with the third sample image is as follows:

[0196]

[0197] in, Represents the second fusion sample label, y i Indicates that the first sample label is the label of the i-th image, y j Indicates that the third sample label is the label of the jth image, m represents the fusion weight parameter, and the weight parameter conforms to the Beta distribution or other distribution methods.

[0198] Exemplarily, assuming that m=0.25, the first sample image shows a cat, and the third sample image shows a dog. Then, the probability that the second fused sample label is a cat is 25%, and the probability that it is a dog is 0.75%.

[0199] S404: Input the third model input result into the original VIT model to obtain a third classification result output by the original VIT model.

[0200] S405: Calculate a third target loss based on the third classification result and the second fusion sample label.

[0201] The third target loss is used to measure the difference between the second classification result and the second fusion sample label. The smaller the third target loss, the smaller the difference between the second classification result and the second fusion sample label.

[0202] For example, the third objective loss is a regularization loss and can be expressed as follows:

[0203]

[0204] in, represents the third target loss, x i Indicates that the first sample image is the i-th image, x j Indicates that the third sample image is the jth image, y i Indicates that the first sample label is the label of the i-th image, y j Indicates that the third sample label is the label of the jth image, m represents the fusion weight parameter, and the weight parameter conforms to the Beta distribution or other distribution methods.

[0205] S406: Calculate the target loss according to the first target loss and the third target loss.

[0206] The target loss is related to the first target loss, the third target loss, and the weight parameter of the third target loss.

[0207] Exemplarily, the relationship between the target loss and the first target loss and the third target loss is:

[0208] Loss=Loss ce +βLoss mixup

[0209] Among them, Loss is the target loss, Loss ce is the first target loss, Loss mixup is the third objective loss, and β is the weight parameter of the third objective loss.

[0210] S105B, adjust the model parameters of the original VIT model according to the target loss.

[0211] After adjusting the model parameters of the original VIT model according to the target loss, it is determined whether the preset convergence condition is met. If so, the trained VIT model is obtained; if not, the process returns to step S103 and S404 to continue execution.

[0212] In S105B, the order of inputting the first model input result into the original VIT model and the order of inputting the third model input result into the original VIT model can be to input the first model input result into the original VIT model first and then input the third model input result into the original VIT model; or to input the third model input result into the original VIT model first and then input the first model input result into the original VIT model.

[0213] By selecting the above embodiment, since the third target loss is calculated based on the third classification result and the second fused sample label, the third model input result is obtained by fusion of the first sample image and the third sample image, and the second fused sample label is obtained by fusion of the first sample label and the third sample label corresponding to the third sample image, therefore, the model parameters of the original VIT model are adjusted according to the target loss calculated by the first target loss and the third target loss, so as to improve the robustness of the VIT model.

[0214] In one possible embodiment, see Figure 5 , is a flow chart of a fourth VIT model training method provided by an embodiment of the present invention, which is similar to the aforementioned Figure 1 Compared with the embodiment shown, the above S102 can be implemented by S102A and S102B.

[0215] S102A, arrange the first tokens belonging to the same first window in the same row of the token matrix.

[0216] For example, assuming that the number of first tokens n=196, the first tokens are rearranged in a 14×14 manner:

[0217]

[0218] From the above formula, we can see that x1, x2, x 15 , x 16 Position adjacent, x1, x2, x 15 , x 16 Configured in the first row of the token matrix, we get:

[0219]

[0220] In addition, since x3, x4, x 17 , x 18Position adjacent, x3, x4, x 17 , x 18 Configured in the second row of the token matrix, x 181 , x 182 , x 195 , x 196 Position adjacent, x 181 , x 182 , x 195 , x 196 Configured in the last row of the token matrix.

[0221] If we randomly discard r first tokens from each row and rearrange the remaining first tokens, we get:

[0222]

[0223] Among them, X i Indicates that the first sample image is the i-th image, They are The first token. If r = 2, then

[0224] After rearranging The first token and 1 classification token are used as the first model input result.

[0225] Among them, the classification token is also called CLS (Classification) token. In the VIT model, the CLS token is located before the token obtained from the image and is used to predict the category of the entire image.

[0226] S102B: For each row of the token matrix, remove a first number of first tokens from the row, and use the remaining first tokens as the first model input result.

[0227] Still taking the token matrix in S102A as an example, assuming that the first number is 2, for the first row of the token matrix, the first row includes x1, x2, x 15 , x 16 , from x1, x2, x 15 , x 16 Remove the first two tokens, for example, remove x1 and x2, and the remaining x 15 , x 16 .

[0228] For the second row of the token matrix, the second row includes x3, x4, x 17 , x 18 , from x3, x4, x 17 , x18 Remove the first two tokens, for example, remove x 17 , x 18 , x3, x4 remain. And so on, for each row of the token matrix, remove the two first tokens from the row, and then use the remaining first tokens as the first model input result.

[0229] By selecting the above embodiment, the first tokens belonging to the same first window are arranged in the same row of the token matrix. For each row of the token matrix, a first number of first tokens are removed from the row. This means that for each first window of the first sample image, a first number of first vector tokens are removed from the first window. Therefore, some first tokens in each first window of the first sample image can be eliminated.

[0230] In one possible embodiment, see Figure 6 , is a flow chart of a fifth VIT model training method provided by an embodiment of the present invention, which is similar to the aforementioned Figure 1 Compared with the embodiment shown, the above method further includes S601-S613, and the above S105 can be implemented through S105C.

[0231] S601, acquiring a second sample image.

[0232] The second sample image has the same size as the first sample image.

[0233] S602: For each first window of the first sample image, remove a second number of first tokens from the first window.

[0234] The second number is less than the total number of first tokens included in a first window.

[0235] S603: For each second window of the second sample image, remove a third number of second tokens from the second window.

[0236] The second window of the second sample image corresponds one-to-one to the first window in the first sample image, and for the corresponding first window and second window, the position of the first token contained in the first window in the first sample image is the same as the position of the second token contained in the second window in the second sample image, and the third number is less than the total number of second tokens contained in a second window; the sum of the second number and the third number is equal to the total number of second tokens; and the total number of first tokens is equal to the total number of second tokens.

[0237] S604: Superimpose the remaining first target tokens in the first sample image and the remaining second target tokens in the second sample image to obtain a second model input result.

[0238] S605: Fuse the first sample label with the second sample label corresponding to the second sample image to obtain a first fused sample label.

[0239] The second sample label indicates the category to which the second sample image belongs.

[0240] S606, inputting the second model input result into the original VIT model to obtain the second classification result output by the original VIT model.

[0241] S607: Calculate a second target loss based on the second classification result and the first fusion sample label.

[0242] S608: Acquire a third sample image.

[0243] The third sample image has the same size as the first sample image.

[0244] S609: Fuse the first sample image with the third sample image to obtain a third model input result.

[0245] S610: Fuse the first sample label and the third sample label corresponding to the third sample image to obtain a second fused sample label.

[0246] The third sample label indicates the category to which the third sample image belongs.

[0247] S611, input the third model input result into the original VIT model to obtain the third classification result output by the original VIT model.

[0248] S612: Calculate a third target loss based on the third classification result and the second fusion sample label.

[0249] S613, calculating the target loss according to the first target loss, the second target loss and the third target loss.

[0250] In a possible embodiment, the relationship between the target loss and the first target loss, the second target loss, and the third target loss is:

[0251] Loss=Loss ce +αLoss mixtoken +βLoss mixup

[0252] Among them, Loss is the target loss, Loss ce is the first target loss, Loss mixtokenis the second target loss, Loss mixup is the third target loss, α is the weight parameter of the second target loss, and β is the weight parameter of the third target loss.

[0253] S105C, adjust the model parameters of the original VIT model according to the target loss.

[0254] After adjusting the model parameters of the original VIT model according to the target loss, it is determined whether the preset convergence condition is met. If so, the trained VIT model is obtained; if not, the process returns to continue execution from steps S103, S606, and S611.

[0255] In S105C, the order of inputting the first model input result into the original VIT model, inputting the second model input result into the original VIT model, and inputting the third model input result into the original VIT model can be that the first model input result is first input into the original VIT model, then the second model input result is input into the original VIT model, and then the third model input result is input into the original VIT model; or the order of inputting the second model input result into the original VIT model first, then the first model input result is input into the original VIT model, and then the third model input result is input into the original VIT model.

[0256] The embodiment of the present invention does not specifically limit the order of inputting the first model input result, the second model input result into the original VIT model, and the third model input result into the original VIT model.

[0257] Among them, the contents of S601-S607 are the same as those of S201-S207, and the contents of S608-S612 are the same as those of S401-S405, which will not be repeated here.

[0258] See Table 1, which is a model parameter comparison table provided in an embodiment of the present invention.

[0259] Table 1

[0260] Token processing method VIT model backbone r Gflops ACC Token not removed MAE-B 0 17.6 92.53 Remove some tokens MAE-B 1 13.1 92.14 Remove some tokens MAE-B 2 8.71 90.89 Remove some tokens MAE-B 4 4.41 82.27

[0261] It can be seen from Table 1 that when the number of tokens removed from a window r=0, the computational amount of the VIT model Gflops (Giga Floating-point Operations Per Second)=17.6, and the accuracy of the VIT model ACC (Accuracy)=92.53. When the number of tokens removed from a window r=1, the computational amount of the VIT model Gflops=13.1, and the accuracy of the VIT model ACC=92.14. This shows that compared with the data when r=0, the computational amount Gflops of the VIT model when r=1 is 0.74 times that of the VIT model when r=0, and the accuracy of the VIT model is only reduced by 0.39.

[0262] The present invention provides an image classification method. Figure 7 , is a flow chart of an image classification method provided in an embodiment of the present invention, the method includes S701-S703.

[0263] S701, obtaining an image to be classified.

[0264] In S701, the size of the image to be classified is the same as that of the first sample image. For example, assuming that the size of the first sample image is 224 pixels×224 pixels, the size of the image to be classified is also 224 pixels×224 pixels.

[0265] S702: For each window in the image to be classified, remove a first number of tokens from the window.

[0266] The specific process of removing the first number of vector tokens can be found in the aforementioned S102, which will not be described in detail here.

[0267] S703: Input the remaining tokens into the pre-trained VIT model to obtain the classification result of the image to be classified.

[0268] The pre-trained VIT model is obtained by training using any of the aforementioned VIT model training methods. The first number is equal to the number of first tokens taken from the first sample image when obtaining the first model input result during the process of training the VIT model.

[0269] By selecting the above embodiment, for each window in the image to be classified, a first number of tokens are removed from the window, some tokens in each window of the image to be classified can be eliminated, and the remaining tokens are input into the pre-trained VIT model to obtain the classification result of the image to be classified, and the category to which the image to be classified belongs can be obtained.

[0270] After some tokens in each window of the image to be classified are eliminated, the remaining tokens are input into the pre-trained VIT model. The VIT model does not need to screen and eliminate tokens again, which reduces the time of eliminating tokens layer by layer according to self-attention in the VIT model, thereby improving the efficiency of image classification.

[0271] Corresponding to the aforementioned VIT model training method, the embodiment of the present invention further provides a VIT model training device, see Figure 8 , is a schematic diagram of the structure of a VIT model training device provided by an embodiment of the present invention, the device comprising:

[0272] A first acquisition module 801 is used to acquire a first sample image;

[0273] A first removal module 802 is used for removing a first number of first tokens from each first window of the first sample image, and using the remaining first tokens as first model input results; wherein the first tokens included in each first window are adjacent to each other in the first sample image, and the first number is less than the total number of first tokens included in one first window;

[0274] A first input module 803 is used to input the first model input result into the original VIT model to obtain a first classification result output by the original VIT model;

[0275] A first calculation module 804 is used to calculate a first target loss based on the first classification result and a first sample label corresponding to the first sample image; the first sample label indicates the category to which the first sample image belongs;

[0276] The parameter adjustment module 805 is used to adjust the model parameters of the original VIT model according to the target loss, and return to the step of inputting the first model input result into the original VIT model to obtain the first classification result output by the original VIT model and continue to execute until the preset convergence condition is met to obtain the trained VIT model; the target loss is determined based on the first target loss.

[0277] The above embodiment is selected. Since the positions of the first tokens contained in each first window in the first sample image are adjacent to each other and the first number is less than the total number of first tokens contained in a first window, for each first window of the first sample image, a first number of first vector tokens are removed from the first window, and the remaining first tokens are used as the first model input result, so that some of the first tokens in each first window of the first sample image can be eliminated.

[0278] The first model input result is input into the original VIT model to obtain a first classification result output by the VIT model, and a first target loss is calculated based on the first classification result and a first sample label corresponding to the first sample image. The model parameters of the original VIT model are adjusted according to the target loss determined by the first target loss, and the original VIT model is continuously trained until a preset convergence condition is met, thereby completing the training process of the VIT model.

[0279] Since some of the first tokens have been eliminated before the first model input results are input into the original VIT model, the VIT model does not need to screen and eliminate the first tokens, which reduces the time of eliminating the first tokens layer by layer according to self-attention in the VIT model, thereby improving the reasoning efficiency of the VIT model.

[0280] In a possible embodiment, the above device further includes:

[0281] A second acquisition module, used for acquiring a second sample image, the second sample image having the same size as the first sample image;

[0282] A second removal module is used to remove a second number of first tokens from each first window of the first sample image; wherein the second number is less than the total number of first tokens included in one first window;

[0283] A third removal module is used to remove a third number of second tokens from each second window of the second sample image; wherein the second window of the second sample image corresponds to the first window in the first sample image one by one, and for the corresponding first window and second window, the position of the first token contained in the first window in the first sample image is the same as the position of the second token contained in the second window in the second sample image, and the third number is less than the total number of second tokens contained in one second window; the sum of the second number and the third number is equal to the total number of second tokens; and the total number of first tokens is equal to the total number of second tokens;

[0284] A vector superposition module, used to superimpose the remaining first target tokens in the first sample image with the remaining second target tokens in the second sample image to obtain a second model input result;

[0285] A first fusion module is used to fuse the first sample label with the second sample label corresponding to the second sample image to obtain a first fused sample label; wherein the second sample label indicates the category to which the second sample image belongs;

[0286] A second input module, used for inputting the second model input result into the original VIT model to obtain a second classification result output by the original VIT model;

[0287] A second calculation module, used to calculate a second target loss based on the second classification result and the first fusion sample label;

[0288] A third calculation module, used to calculate the target loss according to the first target loss and the second target loss;

[0289] The parameter adjustment module 805 is specifically used for:

[0290] Adjust the model parameters of the original VIT model according to the target loss, and return to the step of inputting the first model input result into the original VIT model to obtain the first classification result output by the original VIT model, and inputting the second model input result into the original VIT model to obtain the second classification result output by the original VIT model to continue execution.

[0291] By selecting the above embodiment, since the second target loss is calculated based on the second classification result and the first fused sample label, the second model input result is obtained by superimposing the remaining first target token in the first sample image and the remaining second target token in the second sample image, and the first fused sample label is obtained by fusing the first sample label with the second sample label corresponding to the second sample image, therefore, adjusting the model parameters of the original VIT model according to the target loss calculated by the first target loss and the second target loss can make the trained VIT model more accurate.

[0292] In a possible embodiment, the position of the first target token in the first sample image is the same as the position of the second token removed from the second sample image in the second sample image.

[0293] By selecting the above embodiment, the position of the first target token in the first sample image is the same as the position of the second token removed from the second sample image in the second sample image, so that the superimposed second model input result has no blurred area and is more beautiful.

[0294] In a possible embodiment, the above device further includes:

[0295] A third acquisition module, used for acquiring a third sample image, the third sample image having the same size as the first sample image;

[0296] A second fusion module, used for fusing the first sample image with the third sample image to obtain a third model input result;

[0297] A third fusion module is used to fuse the first sample label and the third sample label corresponding to the third sample image to obtain a second fused sample label, wherein the third sample label indicates the category to which the third sample image belongs;

[0298] A third input module, used for inputting the third model input result into the original VIT model to obtain a third classification result output by the original VIT model;

[0299] A fourth calculation module, used to calculate a third target loss based on the third classification result and the second fusion sample label;

[0300] a fifth calculation module, configured to calculate a target loss according to the first target loss and the third target loss;

[0301] The parameter adjustment module 805 is specifically used for:

[0302] Adjust the model parameters of the original VIT model according to the target loss, and return to the step of inputting the first model input result into the original VIT model to obtain the first classification result output by the original VIT model, and inputting the third model input result into the original VIT model to obtain the third classification result output by the original VIT model to continue execution.

[0303] By selecting the above embodiment, since the third target loss is calculated based on the third classification result and the second fused sample label, the third model input result is obtained by fusion of the first sample image and the third sample image, and the second fused sample label is obtained by fusion of the first sample label and the third sample label corresponding to the third sample image, therefore, the model parameters of the original VIT model are adjusted according to the target loss calculated by the first target loss and the third target loss, so as to improve the robustness of the VIT model.

[0304] In a possible embodiment, the first removal module 802 is specifically configured to:

[0305] Arrange the first tokens belonging to the same first window in the same row of the token matrix;

[0306] For each row of the token matrix, a first number of first tokens are removed from the row, and the remaining first tokens are used as the first model input result.

[0307] By selecting the above embodiment, the first tokens belonging to the same first window are arranged in the same row of the token matrix. For each row of the token matrix, a first number of first tokens are removed from the row. This means that for each first window of the first sample image, a first number of first vector tokens are removed from the first window. Therefore, some first tokens in each first window of the first sample image can be eliminated.

[0308] The present invention also provides an image classification device, see Fig. 9 , is a schematic diagram of the structure of an image classification device provided by an embodiment of the present invention, the device comprising:

[0309] An image acquisition module 901 is used to acquire an image to be classified;

[0310] A vector removal module 902, for removing a first number of tokens from each window in the image to be classified;

[0311] The vector input module 903 is used to input the remaining tokens into a pre-trained VIT model to obtain a classification result of the image to be classified, wherein the pre-trained VIT model is trained by any of the above-mentioned VIT model training methods;

[0312] The first number is equal to the number of first tokens taken from the first sample image when obtaining the first model input result during the process of training the VIT model.

[0313] By selecting the above embodiment, for each window in the image to be classified, a first number of tokens are removed from the window, some tokens in each window of the image to be classified can be eliminated, and the remaining tokens are input into the pre-trained VIT model to obtain the classification result of the image to be classified, and the category to which the image to be classified belongs can be obtained.

[0314] After some tokens in each window of the image to be classified are eliminated, the remaining tokens are input into the pre-trained VIT model. The VIT model does not need to screen and eliminate tokens again, which reduces the time of eliminating tokens layer by layer according to self-attention in the VIT model, thereby improving the efficiency of image classification.

[0315] The embodiment of the present invention further provides an electronic device, such as Fig.10 As shown, it includes a processor 1001, a communication interface 1002, a memory 1003 and a communication bus 1004, wherein the processor 1001, the communication interface 1002, and the memory 1003 communicate with each other through the communication bus 1004.

[0316] Memory 1003, used for storing computer programs;

[0317] The processor 1001 is used to execute the program stored in the memory 1003, and implements the following steps:

[0318] Acquire a first sample image;

[0319] For each first window of the first sample image, a first number of first vector tokens are removed from the first window, and the remaining first tokens are used as the first model input result; wherein the first tokens contained in each first window are adjacent to each other in the first sample image, and the first number is less than the total number of first tokens contained in one first window;

[0320] Inputting the first model input result into the original visual converter VIT model to obtain the first classification result output by the original VIT model;

[0321] Calculating a first target loss based on the first classification result and a first sample label corresponding to the first sample image; the first sample label indicates the category to which the first sample image belongs;

[0322] The model parameters of the original VIT model are adjusted according to the target loss, and the step of inputting the first model input result into the original VIT model to obtain the first classification result output by the original VIT model is returned to continue execution until a preset convergence condition is met to obtain a trained VIT model; the target loss is determined based on the first target loss.

[0323] The above embodiment is selected. Since the positions of the first tokens contained in each first window in the first sample image are adjacent to each other and the first number is less than the total number of first tokens contained in a first window, for each first window of the first sample image, a first number of first vector tokens are removed from the first window, and the remaining first tokens are used as the first model input result, so that some of the first tokens in each first window of the first sample image can be eliminated.

[0324] The first model input result is input into the original VIT model to obtain a first classification result output by the VIT model, and a first target loss is calculated based on the first classification result and a first sample label corresponding to the first sample image. The model parameters of the original VIT model are adjusted according to the target loss determined by the first target loss, and the original VIT model is continuously trained until a preset convergence condition is met, thereby completing the training process of the VIT model.

[0325] Since some of the first tokens have been eliminated before the first model input results are input into the original VIT model, the VIT model does not need to screen and eliminate the first tokens, which reduces the time of eliminating the first tokens layer by layer according to self-attention in the VIT model, thereby improving the reasoning efficiency of the VIT model.

[0326] The communication bus mentioned in the above electronic device can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0327] The communication interface is used for communication between the above electronic device and other devices.

[0328] The memory may include a random access memory (RAM) or a non-volatile memory (NVM), such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0329] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0330] The embodiment of the present invention further provides an electronic device, such as Fig.11 As shown, it includes a processor 1101, a communication interface 1102, a memory 1103 and a communication bus 1104, wherein the processor 1101, the communication interface 1102, and the memory 1103 communicate with each other through the communication bus 1104.

[0331] Memory 1103, used for storing computer programs;

[0332] The processor 1101 is used to implement the following steps when executing the program stored in the memory 1103:

[0333] Get the image to be classified;

[0334] For each window in the image to be classified, remove a first number of vector tokens from the window;

[0335] Input the remaining tokens into the pre-trained VIT model to obtain the classification result of the image to be classified, wherein the pre-trained VIT model is trained by any of the above VIT model training methods;

[0336] The first number is equal to the number of first tokens taken from the first sample image when obtaining the first model input result during the process of training the VIT model.

[0337] By selecting the above embodiment, for each window in the image to be classified, a first number of tokens are removed from the window, some tokens in each window of the image to be classified can be eliminated, and the remaining tokens are input into the pre-trained VIT model to obtain the classification result of the image to be classified, and the category to which the image to be classified belongs can be obtained.

[0338] After some tokens in each window of the image to be classified are eliminated, the remaining tokens are input into the pre-trained VIT model. The VIT model does not need to screen and eliminate tokens again, which reduces the time of eliminating tokens layer by layer according to self-attention in the VIT model, thereby improving the efficiency of image classification.

[0339] The communication bus mentioned in the above electronic device can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0340] The communication interface is used for communication between the above electronic device and other devices.

[0341] The memory may include a random access memory (RAM) or a non-volatile memory (NVM), such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0342] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0343] In another embodiment provided by the present invention, a computer-readable storage medium is also provided, in which a computer program is stored. When the computer program is executed by a processor, the steps of any of the above-mentioned VIT model training and image classification methods are implemented.

[0344] The above embodiment is selected. Since the positions of the first tokens contained in each first window in the first sample image are adjacent to each other and the first number is less than the total number of first tokens contained in a first window, for each first window of the first sample image, a first number of first vector tokens are removed from the first window, and the remaining first tokens are used as the first model input result, so that some of the first tokens in each first window of the first sample image can be eliminated.

[0345] The first model input result is input into the original VIT model to obtain a first classification result output by the VIT model, and a first target loss is calculated based on the first classification result and a first sample label corresponding to the first sample image. The model parameters of the original VIT model are adjusted according to the target loss determined by the first target loss, and the original VIT model is continuously trained until a preset convergence condition is met, thereby completing the training process of the VIT model.

[0346] Since some of the first tokens have been eliminated before the first model input results are input into the original VIT model, the VIT model does not need to screen and eliminate the first tokens, which reduces the time of eliminating the first tokens layer by layer according to self-attention in the VIT model, thereby improving the reasoning efficiency of the VIT model.

[0347] In another embodiment provided by the present invention, a computer program product including instructions is also provided, which, when executed on a computer, enables the computer to execute any VIT model training and image classification method in the above embodiments.

[0348] The above embodiment is selected. Since the positions of the first tokens contained in each first window in the first sample image are adjacent to each other and the first number is less than the total number of first tokens contained in a first window, for each first window of the first sample image, a first number of first vector tokens are removed from the first window, and the remaining first tokens are used as the first model input result, so that some of the first tokens in each first window of the first sample image can be eliminated.

[0349] The first model input result is input into the original VIT model to obtain a first classification result output by the VIT model, and a first target loss is calculated based on the first classification result and a first sample label corresponding to the first sample image. The model parameters of the original VIT model are adjusted according to the target loss determined by the first target loss, and the original VIT model is continuously trained until a preset convergence condition is met, thereby completing the training process of the VIT model.

[0350] Since some of the first tokens have been eliminated before the first model input results are input into the original VIT model, the VIT model does not need to screen and eliminate the first tokens, which reduces the time of eliminating the first tokens layer by layer according to self-attention in the VIT model, thereby improving the reasoning efficiency of the VIT model.

[0351] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented by software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present invention is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk Solid State Disk (SSD)), etc.

[0352] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0353] Each embodiment in this specification is described in a related manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device, electronic device, and storage medium embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0354] The above description is only a preferred embodiment of the present invention and is not intended to limit the protection scope of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention are included in the protection scope of the present invention.

Claims

1. A VIT model training method, characterized in that: The method comprises: Acquire a first sample image; For each first window of the first sample image, a first number of first vector tokens are removed from the first window, and the remaining first tokens are used as first model input results; wherein the first tokens included in each first window are adjacent to each other in the first sample image, and the first number is less than the total number of first tokens included in one first window; Inputting the first model input result into the original visual converter VIT model to obtain a first classification result output by the original VIT model; Calculating a first target loss based on the first classification result and a first sample label corresponding to the first sample image, wherein the first sample label indicates the category to which the first sample image belongs; The model parameters of the original VIT model are adjusted according to the target loss, and the step of inputting the first model input result into the original VIT model to obtain the first classification result output by the original VIT model is returned to continue execution until a preset convergence condition is met to obtain a trained VIT model; the target loss is determined based on the first target loss.

2. The method according to claim 1, characterized in that The method further comprises: Acquire a second sample image, where the second sample image has the same size as the first sample image; For each first window of the first sample image, remove a second number of first tokens from the first window; wherein the second number is less than the total number of first tokens included in one first window; For each second window of the second sample image, remove a third number of second tokens from the second window; wherein the second window of the second sample image corresponds one-to-one to the first window in the first sample image, and for the corresponding first window and second window, the position of the first token contained in the first window in the first sample image is the same as the position of the second token contained in the second window in the second sample image, and the third number is less than the total number of second tokens contained in one second window; the sum of the second number and the third number is equal to the total number of the second tokens; and the total number of the first tokens is equal to the total number of the second tokens; Superimposing the remaining first target tokens in the first sample image and the remaining second target tokens in the second sample image to obtain a second model input result; Fusing the first sample label with the second sample label corresponding to the second sample image to obtain a first fused sample label; wherein the second sample label indicates the category to which the second sample image belongs; Inputting the second model input result into the original VIT model to obtain a second classification result output by the original VIT model; Calculate a second target loss based on the second classification result and the first fusion sample label; Calculating a target loss according to the first target loss and the second target loss; The step of adjusting the model parameters of the original VIT model according to the target loss and returning to the step of inputting the first model input result into the original VIT model to obtain the first classification result output by the VIT model includes: Adjust the model parameters of the original VIT model according to the target loss, and return to the steps of inputting the first model input result into the original VIT model to obtain the first classification result output by the original VIT model, and inputting the second model input result into the original VIT model to obtain the second classification result output by the original VIT model to continue execution.

3. The method according to claim 2, characterized in that The position of the first target token in the first sample image is the same as the position of the second token removed from the second sample image in the second sample image.

4. The method according to claim 1, characterized in that: The method further comprises: Acquire a third sample image, where the third sample image has the same size as the first sample image; Fusing the first sample image with the third sample image to obtain a third model input result; Fusing the first sample label and the third sample label corresponding to the third sample image to obtain a second fused sample label, wherein the third sample label indicates the category to which the third sample image belongs; Inputting the third model input result into the original VIT model to obtain a third classification result output by the original VIT model; Calculating a third target loss based on the third classification result and the second fusion sample label; Calculating a target loss according to the first target loss and the third target loss; The step of adjusting the model parameters of the original VIT model according to the target loss and returning to the step of inputting the first model input result into the original VIT model to obtain the first classification result output by the VIT model includes: Adjust the model parameters of the original VIT model according to the target loss, and return to the steps of inputting the first model input result into the original VIT model to obtain the first classification result output by the original VIT model, and inputting the third model input result into the original VIT model to obtain the third classification result output by the original VIT model to continue execution.

5. The method according to any one of claims 1 to 4, characterized in that For each first window of the first sample image, removing a first number of first tokens from the first window, and using the remaining first tokens as first model input results, includes: Arrange the first tokens belonging to the same first window in the same row of the token matrix; For each row of the token matrix, a first number of first tokens are removed from the row, and the remaining first tokens are used as first model input results.

6. An image classification method, characterized in that: The method comprises: Get the image to be classified; For each window in the image to be classified, removing a first number of vector tokens from the window; Input the remaining tokens into a pre-trained visual converter VIT model to obtain a classification result of the image to be classified, wherein the pre-trained VIT model is trained by any of the method steps described in claims 1-5; The first number is equal to the number of first tokens taken from the first sample image when obtaining the first model input result during the process of training the VIT model.

7. A VIT model training device, characterized in that: The device comprises: A first acquisition module, used for acquiring a first sample image; A first removal module is used for removing a first number of first vector tokens from each first window of the first sample image, and using the remaining first tokens as first model input results; wherein the first tokens contained in each first window are adjacent to each other in the first sample image, and the first number is less than the total number of first tokens contained in one first window; A first input module, used for inputting the first model input result into the original visual converter VIT model to obtain a first classification result output by the original VIT model; A first calculation module, configured to calculate a first target loss based on the first classification result and a first sample label corresponding to the first sample image; the first sample label indicates the category to which the first sample image belongs; A parameter adjustment module is used to adjust the model parameters of the original VIT model according to the target loss, return to trigger the execution of the first input module until a preset convergence condition is met, and obtain the trained VIT model; the target loss is determined based on the first target loss.

8. An image classification device, characterized in that: The device comprises: An image acquisition module, used to acquire images to be classified; A vector removal module, configured to remove a first number of vector tokens from each window in the image to be classified; A vector input module, used for inputting the remaining tokens into a pre-trained visual converter VIT model to obtain the classification result of the image to be classified, wherein the pre-trained VIT model is trained by any method step of claims 1-5; The first number is equal to the number of first tokens taken from the first sample image when obtaining the first model input result during the process of training the VIT model.

9. An electronic device, characterized in that: It includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus; Memory, used to store computer programs; A processor, for implementing the method steps described in any one of claims 1 to 5 when executing a program stored in a memory.

10. An electronic device, characterized in that: It includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus; Memory, used to store computer programs; The processor is used to implement the method steps described in claim 6 when executing the program stored in the memory.

11. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method steps of any one of claims 1-5 or 6 are implemented.

Images