A visual Transformer model design method, device, storage medium and program product

Abstract

The present invention provides a visual Transformer model design method, device, storage medium and program product, wherein the method includes: setting the size of the minimum deformed image and the size of the minimum area block; obtaining the integer value of the ratio of the long side to the short side of the original image according to the width and height of the original image, and determining the side length of the deformed target image and the side length of the area block to be divided; calculating the width and height of the deformed target image and the width and height of the area block to be divided according to the integer value of the ratio of the long side to the short side of the original image; deforming and dividing the image according to the width and height of the deformed target image and the width and height of the area block to be divided, obtaining a plurality of area blocks, and sending them into the Transformer structure. Each area block is divided into a plurality of blocks, and the plurality of blocks are encoded to obtain the features of each area block; integrating all the area block features and sending them into the Transformer structure for training. The present invention can extract feature information that is more consistent with the original image under less deformation.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a visual Transformer model design method, device, storage medium and program product. Background Art

[0002] When processing visual tasks, the current mainstream method is to build the algorithm's network structure with the Transformer structure as the core. The classic approach is to compress or enlarge an image into a deformed image with equal width and height, and then divide it into multiple small area blocks. The area blocks are then sent to the Transformer structure. These area blocks are all squares, that is, their width and height are equal.

[0003] When the classic Transformer above processes images, images with large differences in width and height will be forcibly compressed into squares, resulting in deformation loss, which in turn affects the model's ability to learn important information about the original image data. In certain scenarios, the aspect ratios of the images to be processed vary greatly. In this case, it is inappropriate to still use a deformation method with an aspect ratio of 1.

[0004] In summary, there is an urgent need for a method that can train and infer the model while basically maintaining the image deformation, so as to solve the problem that existing methods cannot handle images with very large aspect ratio differences. Summary of the invention

[0005] In view of the shortcomings of the prior art, the present invention provides a visual Transformer model design method, device, storage medium and program product. According to the aspect ratio of the original image, the width and height to be compressed, as well as the width and height of the image blocks that need to be divided in the model are calculated. Model training and reasoning are performed based on the width and height data. Feature information that is more consistent with the original image can be extracted with less deformation. Model training and reasoning can be performed while basically maintaining the image without deformation, which solves the problem that the existing method cannot process images with very large difference in aspect ratio.

[0006] In a first aspect, the present invention provides a visual Transformer model design method, comprising the following steps:

[0007] Set the size of the minimum deformation image and the size of the minimum area block;

[0008] According to the width and height of the original image, the value of the ratio of the long side to the short side of the original image is obtained, and the side length of the deformed target image and the side length of the area block to be divided are determined;

[0009] According to the integer value of the ratio of the long side to the short side of the original image, the width and height of the deformed target image and the width and height of the area block to be divided are calculated;

[0010] According to the width and height of the deformed target image and the width and height of the area block to be divided, the image is deformed and divided to obtain several area blocks, which are sent to the Transformer structure;

[0011] Divide each region block into a number of blocks, encode the several blocks, and obtain features of each region block;

[0012] Integrate all regional block features and send them into the Transformer structure for training.

[0013] As a further improvement of the present invention, the step of setting the size of the minimum deformed image and the size of the minimum area block includes:

[0014] The size of the minimum deformed image is the minimum value of the long side and the short side of the deformed target image;

[0015] The size of the minimum area block is the minimum value of the long side and the short side of the area block;

[0016] The quotient of the size of the minimum deformed image divided by the size of the minimum area block is an integer, and the size of the minimum area block is ≤ the size of the minimum deformed image.

[0017] As a further improvement of the present invention, the integer value of the ratio of the long side to the short side of the original image is obtained by rounding off the value obtained by dividing the long side of the original image by the short side of the original image.

[0018] As a further improvement of the present invention, the step of determining the length of one side of the deformed target image and the length of one side of the region block to be divided includes:

[0019] Compare the width and height of the original image. If the height is greater than the width, the width of the deformed target image is the size of the minimum deformed image, and the width of the area block to be divided is the size of the minimum area block.

[0020] If height < width, the height of the deformed target image is the size of the minimum deformed image, and the height of the area block to be divided is the size of the minimum area block.

[0021] As a further improvement of the present invention, the width and height of the deformed target image and the width and height of the area block to be divided are calculated according to the integer value of the ratio of the long side to the short side of the original image, including:

[0022] Compare the width and height of the original image. If the height is greater than the width, the width of the deformed target image is the size of the smallest deformed image, the height of the deformed target image is the product of the integer value of the ratio of the long side to the short side of the original image and the size of the smallest deformed image, the width of the area block to be divided is the size of the smallest area block, and the height of the area block to be divided is the product of the integer value of the ratio of the long side to the short side of the original image and the size of the smallest area block;

[0023] If height < width, the height of the deformed target image is the size of the minimum deformed image, the width of the deformed target image is the product of the integer value of the ratio of the long side to the short side of the original image and the size of the minimum deformed image, the height of the area block to be divided is the size of the minimum area block, and the width of the area block to be divided is the product of the integer value of the ratio of the long side to the short side of the original image and the size of the minimum area block.

[0024] As a further improvement of the present invention, the number of the plurality of blocks is the integer value of the ratio of the long side to the short side of the original image, and the side length of the plurality of blocks is the size of the minimum area block.

[0025] As a further improvement of the present invention, the plurality of blocks are encoded using a multi-head attention mechanism.

[0026] In a second aspect, the present invention provides a computer device / equipment / system, comprising a memory, a processor and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the above method.

[0027] In a third aspect, the present invention provides a computer-readable storage medium having a computer program / instruction stored thereon, which implements the steps of the above method when the computer program / instruction is executed by a processor.

[0028] In a fourth aspect, the present invention provides a computer program product, comprising a computer program / instruction, which implements the steps of the above method when executed by a processor.

[0029] Compared with the prior art, the present invention has the following beneficial effects:

[0030] The present invention provides a method, device, storage medium and program product for designing a Transformer model based on aspect ratio. The method calculates the width and height to be compressed and the width and height of the image blocks to be divided in the model according to the aspect ratio of the original image. The model is trained and inferred based on the width and height data, so that feature information that is more consistent with the original image can be extracted with less deformation. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 This is a flowchart of a method for designing a visual Transformer model disclosed in the present invention. DETAILED DESCRIPTION

[0032] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the present invention will combine the accompanying drawings to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments, wherein steps S1, S2... in the embodiments described in the present invention do not limit the only execution steps of the present invention; the various models, simulation environments, and software described in the present invention are not the only limiting methods of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0033] In the present invention, computer devices / equipment / systems refer to related entities applied to computers, such as hardware, a combination of hardware and software, software or software in execution, etc. In detail, for example, software includes but is not limited to a process running on a processor, a processor, an object, executable software, an execution thread, a program and / or a computer. In addition, an application or script program running on a server, a server can also be software. One or more software can be in an execution process and / or thread, and the software can be localized on one computer and / or distributed between two or more computers, and can be run by various computer-readable media.

[0034] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application may be combined with each other.

[0035] In an embodiment of the present invention, a visual Transformer model design method is provided. Figure 1 The original image is shown, w represents the width of the original image, and h represents the height of the original image; Figure 1 shows the deformed target image, which refers to the image obtained by deforming the original image. resize Indicates the width of the deformed target image, h resize Indicates the height of the deformed target image; Figure 1 The region blocks are also shown. The region blocks refer to a number of images obtained by dividing the deformed target image. p Indicates the width of the area block, h p Indicates the height of the area block.

[0036] like Figure 1 As shown, the specific process can be as follows:

[0037] S1: Set the size of the minimum deformation image and the size of the minimum area block;

[0038] Preferably, the setting of the minimum deformation image size and the minimum area block size includes:

[0039] The size of the minimum deformed image is the minimum value of the long side and the short side of the deformed target image;

[0040] The size of the minimum area block is the minimum value of the long side and the short side of the area block;

[0041] The quotient of the size of the minimum deformed image divided by the size of the minimum area block is an integer, and the size of the minimum area block is ≤ the size of the minimum deformed image.

[0042] In one embodiment of the present invention, the size x of the minimum deformation image is set min and the minimum region block size p min Among them, the size of the minimum deformed image x min It represents the minimum value of the long side and the short side of the deformed target image; similarly, the minimum area block size p min Indicates the minimum value of the long side and short side of the region block. This step needs to satisfy is an integer, that is The remainder is 0, and p min ≤x min .

[0043] In one embodiment of the present invention, for example, the minimum region block size p can be set to min = 16, the size of the minimum deformed image x min =224.

[0044] S2: According to the width and height of the original image, the value of the ratio of the long side to the short side of the original image is obtained, and the side length of the deformed target image and the side length of the area block to be divided are determined;

[0045] Preferably, the integer value of the ratio of the long side to the short side of the original image is obtained by rounding off the value obtained by dividing the long side of the original image by the short side of the original image.

[0046] Preferably, the step of obtaining the integer value of the ratio of the long side to the short side of the original image according to the width and height of the original image, and determining the side length of the deformed target image and the side length of the area block to be divided includes:

[0047] Compare the width and height of the original image. If the height is greater than the width, the width of the deformed target image is the size of the minimum deformed image, and the width of the area block to be divided is the size of the minimum area block.

[0048] If the height is less than the width, the height of the deformed target image is the size of the minimum deformed image, and the height of the area block to be divided is the size of the minimum area block.

[0049] In one embodiment of the present invention, the size x of the minimum deformation image is set min and the minimum region block size p min. According to the width and height of the original image, the value of the ratio of the long side to the short side of the original image is rounded, recorded as r. Compare the width and height of the original image. If the height is greater than the width, the value of the ratio of the long side to the short side of the original image is rounded Among them, round means that the value is rounded to the nearest integer, w means the width of the original image, and h means the height of the original image. resize =x min , w p =P min , where w resize Indicates the width of the deformed target image, w p Indicates the width of the area block. Compare the width and height of the original image. If the height is less than the width, the ratio of the long side to the short side of the original image is rounded to an integer. Among them, round means that the value is rounded to the nearest integer, w means the width of the original image, and h means the height of the original image. resize =x min ,h p =p min , where h resize They represent the height of the deformed target image, h p Indicates the height of the area block.

[0050] In one embodiment of the present invention, the minimum region block size p min = 16, the size of the minimum deformed image x min = 224, the ratio of the long side to the short side of the original image is rounded to r = 2, in the original image, h>w, the width of the deformed target image is w resize =224, the width of the area block w p =16.

[0051] In another embodiment of the present invention, the minimum region block size p min = 16, the size of the minimum deformed image x min =224, the ratio of the long side to the short side of the original image is rounded to r=2, in the original image, w>h, the height h of the deformed target image resize =224, the height h of the region block p =16.

[0052] S3: Calculate the width and height of the deformed target image and the width and height of the area block to be divided according to the integer value of the ratio of the long side to the short side of the original image;

[0053] Preferably, the calculating the width and height of the deformed target image and the width and height of the area block to be divided according to the integer value of the ratio of the long side to the short side of the original image includes:

[0054] Compare the width and height of the original image. If the height is greater than the width, the width of the deformed target image is the size of the smallest deformed image, the height of the deformed target image is the product of the integer value of the ratio of the long side to the short side of the original image and the size of the smallest deformed image, the width of the area block to be divided is the size of the smallest area block, and the height of the area block to be divided is the product of the integer value of the ratio of the long side to the short side of the original image and the size of the smallest area block;

[0055] If the height is less than the width, the height of the deformed target image is the size of the minimum deformed image, the width of the deformed target image is the product of the integer value of the ratio of the long side to the short side of the original image and the size of the minimum deformed image, the height of the area block to be divided is the size of the minimum area block, and the width of the area block to be divided is the product of the integer value of the ratio of the long side to the short side of the original image and the size of the minimum area block.

[0056] In one embodiment of the present invention, the size x of the minimum deformation image is set min and the minimum region block size p min According to the width and height of the original image, we get the integer value of the ratio of the long side to the short side of the original image, recorded as r. Compare the width and height of the original image. When the height is greater than the width, W resize =x min ,h resize =r*x min , w p =p min ,h p =p min *r; when height < width, h resize =x min , w resize =r*x min ,h p = pmin , w p = pmin *r, where W resize Indicates the width of the deformed target image, h resize They represent the height of the deformed target image, w p Indicates the width of the area block, h p Indicates the height of the area block.

[0057] In one embodiment of the present invention, the minimum region block size p min = 16, the size of the minimum deformed image x min = 224, the ratio of the long side to the short side of the original image is rounded to r = 2, in the original image, h>w, the width of the deformed target image is w resize =224, high h resize =448, the width of the area block w p =16, high h p =32.

[0058] In another embodiment of the present invention, the minimum region block size pmin = 16, the size of the minimum deformed image x min =224, the ratio of the long side to the short side of the original image is rounded to r=2, in the original image, w>h, the height h of the deformed target image resize =224, width w resize =448, the height h of the area block p =16, width w p =32.

[0059] S4: According to the width and height of the deformed target image and the width and height of the area block to be divided, the image is deformed and divided to obtain several area blocks, which are sent to the Transformer structure.

[0060] In one embodiment of the present invention, the minimum region block size p is set min = 16, the size of the minimum deformed image x min = 224, the ratio of the long side to the short side of the original image is rounded to r = 2, in the original image, h>w, the width of the deformed target image is w resize =224, high h resize =448, the width of the area block w p =16, high h p =32, after the image is deformed, it can be divided into 14*14 rectangular blocks, a total of 196 rectangular blocks, and these 196 rectangular blocks are sent to the Transformer structure.

[0061] In another embodiment of the present invention, the minimum region block size p min = 16, the size of the minimum deformed image x min =224, the ratio of the long side to the short side of the original image is rounded to r=2, in the original image, w>h, the height h of the deformed target image resize =224, width w resize =448, the height h of the area block p =16, width w p =32, after the image is deformed, it can be divided into 14*14 rectangular blocks, a total of 196 rectangular blocks, and these 196 rectangular blocks are sent to the Transformer structure.

[0062] S5: Divide each region block into a number of blocks, encode the several blocks, and obtain features of each region block;

[0063] Preferably, the number of the plurality of blocks is the integer value of the ratio of the long side to the short side of the original image, and the side length of the plurality of blocks is the size of the minimum area block;

[0064] Preferably, the plurality of blocks are encoded using a multi-head attention mechanism.

[0065] In one embodiment of the present invention, the size x of the minimum deformation image is set min and the minimum region block size p min , according to the width and height of the original image, the value of the ratio of the long side to the short side of the original image is obtained, recorded as r, and each area block is divided into r p min *p min The r blocks are sent to the multi-head attention mechanism for encoding.

[0066] In one embodiment of the present invention, the minimum region block size p min = 16, the size of the minimum deformed image x min = 224, the ratio of the long side to the short side of the original image is rounded to r = 2, in the original image, h>w, the width of the deformed target image is w resize =224, high h resize =448, the width of the area block w p =16, high h p =32. Then the region block can be divided into 2 (16*16) blocks, and then the (2*16*16*3) image is sent to the multi-head attention mechanism, where 2 represents the integer value r of the ratio of the long side to the short side of the original image, 16 represents the size of the region block, and 3 represents the RGB value. A 1*d-dimensional region block vector representation is obtained, where d represents the feature dimension after Transformer encoding, and its value can be set in the Transformer model.

[0067] In another embodiment of the present invention, the minimum region block size p min = 16, the size of the minimum deformed image x min =224, the ratio of the long side to the short side of the original image is rounded to r=2, in the original image, w>h, the height h of the deformed target image resize =224, width w resize =448, the height h of the area block p =16, width w p =32. Then the region block can be divided into 2 (16*16) blocks, and then the (2*16*16*3) image is sent to the multi-head attention mechanism, where 2 represents the integer value r of the ratio of the long side to the short side of the original image, 16 represents the size of the region block, and 3 represents the RGB value. The 1*d-dimensional region block vector representation is obtained, where d represents the feature dimension after Transformer encoding, which can be used in Transformere r Set its value in the model.

[0068] S6: Integrate all regional block features and send them into the Transformer structure for training.

[0069] In one embodiment of the present invention, the minimum region block size p is set min = 16, the size of the minimum deformed image x min = 224, the ratio of the long side to the short side of the original image is rounded to r = 2, in the original image, h>w, the width of the deformed target image is w resize =224, high h resize =448, the width of the area block w p =16, high h p =32, which can be divided into 14*14 rectangular blocks, a total of 196. Through the above step S5, the features of each regional block are obtained, and after integration, a 196*d matrix is ​​obtained, which is sent to the Transformer structure for encoding. The same processing method is used during training and inference.

[0070] In another embodiment of the present invention, the minimum region block size p min = 16, the size of the minimum deformed image x min =224, the ratio of the long side to the short side of the original image is rounded to r=2, in the original image, w>h, the height h of the deformed target image resize =224, width w resize =448, the height h of the area block p =16, width w p =32, so it can be divided into 14*14 rectangular blocks, a total of 196. Through the above step S5, the features of each regional block are obtained, and after integration, a 196*d matrix is ​​obtained, which is sent to the Transformer structure for encoding. The same processing method is used during training and inference.

[0071] In a second aspect, the present invention provides a computer device / equipment / system, comprising a memory, a processor and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the above method.

[0072] In a third aspect, the present invention provides a computer-readable storage medium having a computer program / instruction stored thereon, which implements the steps of the above method when the computer program / instruction is executed by a processor.

[0073] In a fourth aspect, the present invention provides a computer program product, comprising a computer program / instruction, which implements the steps of the above method when executed by a processor.

[0074] The present invention proposes a visual Transformer model design method, device, storage medium and program product, which calculate the width and height to be compressed, as well as the width and height of the image area blocks to be divided in the model, based on the integer value of the ratio of the long side to the short side of the original image. Model training and reasoning are performed based on these two sets of width and height data, so that feature information that is more consistent with the original image can be extracted with less deformation.

Claims

1. A visual Transformer model design method, characterized in that: The following steps are involved: Set the size of the minimum deformation image and the size of the minimum area block; According to the width and height of the original image, the value of the ratio of the long side to the short side of the original image is rounded off; Compare the width and height of the original image. If the height is greater than the width, the width of the deformed target image is the size of the minimum deformed image, and the width of the area block to be divided is the size of the minimum area block. If height < width, the height of the deformed target image is the size of the minimum deformed image, and the height of the area block to be divided is the size of the minimum area block; According to the integer value of the ratio of the long side to the short side of the original image, the width and height of the deformed target image and the width and height of the area block to be divided are calculated; According to the width and height of the deformed target image and the width and height of the area block to be divided, the image is deformed and divided to obtain several area blocks, which are sent to the Transformer structure; Divide each region block into a number of blocks, encode the several blocks, and obtain features of each region block; Integrate all regional block features and send them into the Transformer structure for training.

2. The visual Transformer model design method according to claim 1, characterized in that: The step of setting the size of the minimum deformed image and the size of the minimum area block includes: The size of the minimum deformed image is the minimum value of the long side and the short side of the deformed target image; The size of the minimum area block is the minimum value of the long side and the short side of the area block; The quotient of the size of the minimum deformed image divided by the size of the minimum area block is an integer, and the size of the minimum area block is ≤ the size of the minimum deformed image.

3. The visual Transformer model design method according to claim 1, characterized in that: The integer value of the ratio of the long side to the short side of the original image is obtained by rounding off the value obtained by dividing the long side of the original image by the short side of the original image.

4. The visual Transformer model design method according to claim 1, characterized in that: The step of calculating the width and height of the deformed target image and the width and height of the area block to be divided according to the integer value of the ratio of the long side to the short side of the original image includes: Compare the width and height of the original image. If the height is greater than the width, the width of the deformed target image is the size of the smallest deformed image, the height of the deformed target image is the product of the integer value of the ratio of the long side to the short side of the original image and the size of the smallest deformed image, the width of the area block to be divided is the size of the smallest area block, and the height of the area block to be divided is the product of the integer value of the ratio of the long side to the short side of the original image and the size of the smallest area block; If height < width, the height of the deformed target image is the size of the minimum deformed image, the width of the deformed target image is the product of the integer value of the ratio of the long side to the short side of the original image and the size of the minimum deformed image, the height of the area block to be divided is the size of the minimum area block, and the width of the area block to be divided is the product of the integer value of the ratio of the long side to the short side of the original image and the size of the minimum area block.

5. The visual Transformer model design method according to claim 1, characterized in that: The number of the plurality of blocks is the integer value of the ratio of the long side to the short side of the original image, and the side length of the plurality of blocks is the size of the minimum area block.

6. The visual Transformer model design method according to claim 1, characterized in that: The blocks are encoded using a multi-head attention mechanism.

7. A computer device / equipment / system comprising a memory, a processor and a computer program stored in the memory, characterized in that: The processor executes the computer program to implement the steps of the method according to any one of claims 1 to 6.

8. A computer-readable storage medium having a computer program / instruction stored thereon, characterized in that: When the computer program / instructions are executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images