ViT differential reasoning method and device, computer equipment and storage medium

Abstract

The invention relates to a ViT differential reasoning method and device, computer equipment, a storage medium and a computer program product, and the method comprises the steps: dividing an image into semantic groups with different sizes in an image processing process, carrying out the preprocessing of each semantic group, obtaining difference data and centroid data, carrying out the linear calculation of the difference data and the centroid data, and obtaining a ViT differential reasoning result. The method comprises the steps of obtaining differential data, performing multi-head self-attention calculation on the differential data to obtain multi-head score data, performing asynchronous activation on the multi-head score data to obtain an activation value, and performing multi-head self-attention calculation based on the activation value; the problem that an existing VIT reasoning acceleration scheme cannot make full use of visual information and cannot adjust resources in a targeted mode according to different images, and consequently reasoning efficiency is low is solved. And the efficiency and the adaptability are high.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a ViT differential reasoning method, apparatus, computer equipment, storage medium and computer program product. Background Art

[0002] In the field of computer vision and inference acceleration, memory requirements, computational costs, and resource requirements are important challenges. Existing VIT inference acceleration solutions are mainly oriented towards sparse computing and labeled pruning.

[0003] However, these optimization techniques influenced by the NLP field cannot fully utilize visual information. For example, an image contains multiple levels of semantic information, including the sky, grass, people, etc. How to use the inherent semantic level information of the image itself to accelerate VIT reasoning is very meaningful. In addition, different images have different semantic information. How to adjust runtime resources for different images to calculate different semantic groups is the key to accelerating VIT reasoning. Summary of the invention

[0004] Based on this, it is necessary to provide a ViT differential reasoning method, apparatus, computer device, storage medium and computer program product to address the above technical problems.

[0005] In a first aspect, the present application provides a ViT differential reasoning method, the method comprising:

[0006] During image processing, the image is divided into semantic groups of different sizes;

[0007] Preprocess each semantic group to obtain difference data and centroid data;

[0008] Performing linear calculation on the differential data and the centroid data to obtain differential data;

[0009] Performing multi-head self-attention calculation on the differential data to obtain multi-head score data;

[0010] Asynchronously activating the long score data to obtain an activation value;

[0011] Multi-head self-attention calculation is performed based on the activation values.

[0012] In one embodiment, the preprocessing of each semantic group to obtain difference data and centroid data includes:

[0013] Preprocess each semantic group and calculate the centroid of each semantic group and the difference between the token in the group and the centroid;

[0014] The centroid data includes the centroid of each semantic group, and the difference data includes the difference between the token in each semantic group and the centroid.

[0015] In one embodiment, the centroid of each semantic group is expressed as:

[0016]

[0017] The difference between the token and the centroid in each semantic group is expressed as:

[0018]

[0019] Where i represents the semantic group number, subscript c represents the centroid, represents the centroid of the ith semantic group, L represents the length of the ith semantic group, Δx i Represents Token x i The difference from the centroid.

[0020] In one embodiment, the linear calculation is performed on the difference data and the centroid data to obtain the calculation formula of the difference data:

[0021] ΔQ i =Δx i ·W Q ,

[0022] ΔK i =Δx i ·W K ;

[0023] ΔV i =Δx i ·W V ;

[0024] in, Indicates the centroid value of the i-th semantic group Query, Key, Value; ΔQ i , ΔK i , ΔV i represents the differential data of the i-th semantic group Query, Key, Value; Q i represents the original data of the i-th semantic group Query; W Q , W K , W V Indicates the weight data of Query, Key, and Value.

[0025] In one embodiment, the multi-head self-attention calculation is performed on the differential data to obtain the calculation formula of the multi-head score data:

[0026]

[0027] in, is the centroid long fraction data, S i It is the Token long score data.

[0028] In one embodiment, the method further comprises:

[0029] A PE unit is established, and the PE unit can at least be used to perform the linear calculation and the multi-head self-attention calculation.

[0030] In a second aspect, the present application further provides a multi-ViT differential reasoning device, the device comprising:

[0031] A grouping module, used for dividing the image into semantic groups of different sizes during image processing;

[0032] A preprocessing module is used to preprocess each semantic group to obtain differential data and centroid data;

[0033] A first calculation module, used for performing linear calculation on the difference data and the centroid data to obtain difference data;

[0034] A second calculation module is used to perform multi-head self-attention calculation on the differential data to obtain multi-head score data;

[0035] An activation module, used for asynchronously activating the multi-head score data to obtain an activation value;

[0036] The third computing module is used to perform multi-head self-attention calculation based on the activation value.

[0037] In a third aspect, the present application further provides a computer device. The computer device includes a memory and a processor, the memory stores a computer program, and the processor implements the following steps when executing the computer program:

[0038] During image processing, the image is divided into semantic groups of different sizes;

[0039] Preprocess each semantic group to obtain difference data and centroid data;

[0040] Performing linear calculation on the difference data and the centroid data to obtain difference data;

[0041] Performing multi-head self-attention calculation on the differential data to obtain multi-head score data;

[0042] Asynchronously activating the long score data to obtain an activation value;

[0043] Multi-head self-attention calculation is performed based on the activation values.

[0044] In a fourth aspect, the present application further provides a computer-readable storage medium. The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the following steps are implemented:

[0045] During image processing, the image is divided into semantic groups of different sizes;

[0046] Preprocess each semantic group to obtain difference data and centroid data;

[0047] Performing linear calculation on the difference data and the centroid data to obtain difference data;

[0048] Performing multi-head self-attention calculation on the differential data to obtain multi-head score data;

[0049] Asynchronously activating the long score data to obtain an activation value;

[0050] Multi-head self-attention calculation is performed based on the activation values.

[0051] In a fifth aspect, the present application further provides a computer program product. The computer program product includes a computer program, and when the computer program is executed by a processor, the following steps are implemented:

[0052] During image processing, the image is divided into semantic groups of different sizes;

[0053] Preprocess each semantic group to obtain difference data and centroid data;

[0054] Performing linear calculation on the difference data and the centroid data to obtain difference data;

[0055] Performing multi-head self-attention calculation on the differential data to obtain multi-head score data;

[0056] Asynchronously activating the long score data to obtain an activation value;

[0057] Multi-head self-attention calculation is performed based on the activation values.

[0058] The above-mentioned ViT differential reasoning method, device, computer equipment, storage medium and computer program product, during the image processing process, divide the image into semantic groups of different sizes, pre-process each semantic group to obtain difference data and centroid data, perform linear calculation on the difference data and centroid data to obtain difference data, perform multi-head self-attention calculation on the difference data to obtain multi-head score data, perform asynchronous activation on the multi-head score data to obtain activation values, and perform multi-head self-attention calculation based on the activation values; solve the problem that the existing VIT reasoning acceleration scheme cannot make full use of visual information and cannot adjust resources according to different images in a targeted manner, resulting in low reasoning efficiency; realize the use of inherent semantic level information of the image to accelerate VIT reasoning, with high efficiency and adaptability. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] Figure 1 Schematic diagram of a flow chart of a ViT differential reasoning method in one embodiment;

[0060] Figure 2 A schematic diagram of a PE unit design in one embodiment;

[0061] Figure 3 is a structural block diagram of a ViT differential reasoning device in one embodiment;

[0062] Figure 4 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0063] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0064] In one embodiment of the present application, Figure 1 As shown, a ViT differential reasoning method is provided, the method comprising:

[0065] S100, during image processing, dividing the image into semantic groups of different sizes.

[0066] The image itself carries semantic information, such as the sky, grass, people, etc. Tokens with the same semantic information also have similar data. This application uses a hardware-friendly Locality-Sensetive Hashing (LSH) method to perform hash operations on each Token. Tokens with the same semantics have a high probability of having the same hash code. Therefore, different Tokens can be divided into different semantic groups by hash codes. Specifically, in an embodiment of the present application, the LSH technology is used to divide the image into semantic groups of different sizes. The formula is:

[0067]

[0068] In the formula, α represents a set of vectors with a range of values ​​in the normal distribution (0, 1), β represents a value with a range of values ​​in (0, γ), and γ represents the width of the hash bucket. i After the above formula, a hash code H(x i ). Tokens with similar semantic information have a high probability of having the same hash code. We divide tokens with the same hash code into a semantic group.

[0069] S200, preprocessing each semantic group to obtain difference data and centroid data.

[0070] The data in the same semantic group have similarity. In order to make full use of this similarity, we adjust the original Token data. In one embodiment, each semantic group needs to be preprocessed to calculate the centroid of each semantic group and the difference between the Token in the group and the centroid; the centroid data includes the centroid of each semantic group, and the difference data includes the difference between the Token in each semantic group and the centroid.

[0071] Specifically, the centroid of each semantic group is expressed as:

[0072]

[0073] The difference between the token and the centroid in each semantic group is expressed as:

[0074]

[0075] Where i represents the semantic group number and subscript c represents the centroid. represents the centroid of the ith semantic group, L represents the length of the ith semantic group, Δx i Represents Token x i The difference from the centroid.

[0076] Since the data in the same semantic group are similar, this difference will be much smaller than the original data. Therefore, we use a lower precision bit width to store the difference data. The benefit of this operation is that it can greatly reduce the amount of data storage without affecting the accuracy of the original data. It should be noted that the original data is the original data of the token in each semantic group.

[0077] S300, performing linear calculation on the difference data and centroid data to obtain difference data.

[0078] Specifically, a linear calculation is performed on the difference data and the centroid data to obtain the calculation formula of the difference data:

[0079] ΔQ i =Δx i ·W Q ,

[0080] ΔK i =Δx i ·W K ;

[0081] ΔV i =Δx i ·WV;

[0082] in, Indicates the centroid value of the i-th semantic group Query, Key, Value; ΔQ i , ΔK i , ΔV i represents the differential data of the i-th semantic group Query, Key, Value; Q i represents the original data of the i-th semantic group Query; W Q , W K , W V Indicates the weight data of Query, Key, and Value.

[0083] S400, performing multi-head self-attention calculation on the differential data to obtain multi-head score data.

[0084] Specifically, multi-head self-attention calculation is performed on the differential data to obtain the calculation formula of multi-head score data:

[0085]

[0086] in, is the centroid long fraction data, S i It is the Token long score data.

[0087] Take one of the semantic groups as an example. First, we calculate the Query centroid and Key centroid Then, we calculate the original data Q i Differential data ΔK from Key i Finally, we add the two data to get the final data S i .

[0088] S500, performing asynchronous activation on the long score data to obtain an activation value.

[0089] Specifically, based on the multi-head score data obtained by S400, we use an asynchronous activation method to calculate the SoftMax value. First, we calculate the local maximum of the local score and the sum of the local index scores, and maintain two global variables, the global maximum and the sum of the global index scores. Each time the current local score is calculated, the above two global variables need to be updated. When all local scores are calculated, the final activation value is updated through the above two global variables. and S i ′.

[0090] S600, performing multi-head self-attention calculation based on the activation value.

[0091] Specifically, based on the activation value obtained in S500 and S i ′, and perform the second half of the multi-head self-attention calculation. The calculation formula is as follows:

[0092]

[0093] Represents the output value of the i-th semantic group. and S i ′ respectively represent the centroid score value after activation and the original score value. Indicates the centroid value of Value. ΔV i Indicates the difference value of Value.

[0094] In addition, from the above steps, we can see that the calculation steps include two original data matrix multiplications, and also include differential data and original data matrix multiplications. Therefore, we designed a PE unit that can perform the above two calculations. This PE unit can be configured at runtime through the local control unit to flexibly switch between the two calculation modes. In addition, the PE unit has a local buffer that can cache weights and intermediate results without frequent communication with the global buffer.

[0095] Furthermore, different images have different semantic information. The size and number of semantic groups also vary significantly from image to image. To address the above challenges, please refer to Figure 2 We designed a configurable PE array. It consists of PE groups of the same size. Each PE group contains a fixed number of 4 PE units. We designed a global control unit to implement runtime scheduling. We use a polling method to continuously read the next computing task from the task pool and allocate multiple PE groups for computing according to the amount of computing. If there is no PE group combination that meets the requirements, the current task is skipped and the next task is polled.

[0096] The above-mentioned ViT differential reasoning method, device, computer equipment, storage medium and computer program product, during the image processing process, divide the image into semantic groups of different sizes, pre-process each semantic group to obtain difference data and centroid data, perform linear calculation on the difference data and centroid data to obtain difference data, perform multi-head self-attention calculation on the difference data to obtain multi-head score data, perform asynchronous activation on the multi-head score data to obtain activation values, and perform multi-head self-attention calculation based on the activation values; solve the problem that the existing VIT reasoning acceleration scheme cannot make full use of visual information and cannot adjust resources according to different images in a targeted manner, resulting in low reasoning efficiency; realize the use of inherent semantic level information of the image to accelerate VIT reasoning, with high efficiency and adaptability.

[0097] It should be understood that, although the steps in the flowcharts involved in the above embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps is not strictly limited in order, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above embodiments may include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0098] Based on the same inventive concept, the embodiment of the present application also provides a ViT differential reasoning device for implementing the above-mentioned. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above-mentioned method, so the specific limitations in one or more ViT differential reasoning device embodiments provided below can refer to the limitations of the ViT differential reasoning method above, and will not be repeated here.

[0099] In one embodiment, Figure 3As shown, a device for ViT differential reasoning is provided, including:

[0100] A grouping module 100, used for dividing the image into semantic groups of different sizes during image processing;

[0101] A preprocessing module 200 is used to preprocess each semantic group to obtain differential data and centroid data;

[0102] A first calculation module 300 is used to perform linear calculation on the difference data and the centroid data to obtain difference data;

[0103] A second calculation module 400 is used to perform multi-head self-attention calculation on the differential data to obtain multi-head score data;

[0104] An activation module 500 is used to perform asynchronous activation on the multi-head score data to obtain an activation value;

[0105] The third calculation module 600 is used to perform multi-head self-attention calculation based on the activation value.

[0106] In one embodiment, the preprocessing module 200 is further used to preprocess each semantic group, calculate the centroid of each semantic group and the difference between the token in the group and the centroid;

[0107] The centroid data includes the centroid of each semantic group, and the difference data includes the difference between the token in each semantic group and the centroid.

[0108] In one embodiment, the centroid of each semantic group is expressed as:

[0109]

[0110] The difference between the token and the centroid in each semantic group is expressed as:

[0111]

[0112] Where i represents the semantic group number and subscript c represents the centroid. represents the centroid of the ith semantic group, L represents the length of the ith semantic group, Δx i Represents Token x i The difference from the centroid.

[0113] In one embodiment, the linear calculation is performed on the difference data and the centroid data to obtain the calculation formula of the difference data:

[0114] ΔQ i =Δx i ·W Q ,

[0115] ΔK i =Δx i ·W K

[0116] ΔV i =Δx i ·W V

[0117] in, Indicates the centroid value of the i-th semantic group Query, Key, Value; ΔQ i , ΔK i , ΔV i represents the differential data of the i-th semantic group Query, Key, Value; Q i represents the original data of the i-th semantic group Query; W Q , W K , W V Indicates the weight data of Query, Key, and Value.

[0118] In one embodiment, the multi-head self-attention calculation is performed on the differential data to obtain the calculation formula of the multi-head score data:

[0119]

[0120] in, is the centroid long fraction data, S i It is the Token long score data.

[0121] In one of the embodiments, the device also includes an establishment module for: establishing a PE unit, wherein the PE unit can at least be used to perform the linear calculation and the multi-head self-attention calculation.

[0122] Each module in the above-mentioned ViT differential reasoning device can be implemented in whole or in part by software, hardware and a combination thereof. Each of the above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory in the computer device in the form of software, so that the processor can call and execute the operations corresponding to each of the above modules.

[0123] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 4As shown. The computer device includes a processor, a memory and a network interface connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store preset data. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a ViT differential reasoning method is implemented.

[0124] Those skilled in the art will understand that Figure 4 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0125] In one embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the above-mentioned ViT differential reasoning method when executing the computer program.

[0126] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the above-mentioned ViT differential reasoning method is implemented.

[0127] In one embodiment, a computer program product is provided, comprising a computer program, which implements the above-mentioned ViT differential reasoning method when executed by a processor.

[0128] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the above-mentioned computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited thereto. The processors involved in each embodiment provided in this application may be general-purpose processors, central processing units, graphics processors, digital signal processors, programmable logic devices, data processing logic devices based on quantum computing, etc., but are not limited thereto.

[0129] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0130] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A ViT differential reasoning method, characterized in that: The method comprises: During image processing, the image is divided into semantic groups of different sizes; Preprocess each semantic group to obtain difference data and centroid data; Performing linear calculation on the difference data and the centroid data to obtain difference data; Performing multi-head self-attention calculation on the differential data to obtain multi-head score data; Asynchronously activating the long score data to obtain an activation value; Multi-head self-attention calculation is performed based on the activation values.

2. The method according to claim 1, characterized in that The preprocessing of each semantic group to obtain difference data and centroid data includes: Preprocess each semantic group and calculate the centroid of each semantic group and the difference between the token in the group and the centroid; The centroid data includes the centroid of each semantic group, and the difference data includes the difference between the token in each semantic group and the centroid.

3. The method according to claim 2, characterized in that The centroid of each semantic group is expressed as: The difference between the token and the centroid in each semantic group is expressed as: Where i represents the semantic group number, subscript c represents the centroid, represents the centroid of the ith semantic group, L represents the length of the ith semantic group, Δx i Represents Tokenx i The difference from the centroid.

4. The method according to claim 1, characterized in that The linear calculation is performed on the difference data and the centroid data to obtain the calculation formula of the difference data: in, Indicates the centroid value of the i-th semantic group Query, Key, Value; ΔQ i , ΔK i , ΔV i represents the differential data of the i-th semantic group Query, Key, Value; Q i represents the original data of the i-th semantic group Query; W Q , W K , W V Indicates the weight data of Query, Key, and Value.

5. The method according to claim 4, characterized in that The multi-head self-attention calculation is performed on the differential data to obtain the calculation formula of the multi-head score data: in, is the centroid long fraction data, S i It is the Token long score data.

6. The method according to claim 1, characterized in that The method further comprises: A PE unit is established, and the PE unit can at least be used to perform the linear calculation and the multi-head self-attention calculation.

7. A ViT differential reasoning device, characterized in that: The device comprises: A grouping module, used for dividing the image into semantic groups of different sizes during image processing; A preprocessing module is used to preprocess each semantic group to obtain differential data and centroid data; A first calculation module, used for performing linear calculation on the difference data and the centroid data to obtain difference data; A second calculation module is used to perform multi-head self-attention calculation on the differential data to obtain multi-head score data; An activation module, used for asynchronously activating the multi-head score data to obtain an activation value; The third computing module is used to perform multi-head self-attention calculation based on the activation value.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images