ViT differential inference method and device, computer device, and storage medium

Abstract

The application relates to a ViT differential reasoning method and device, computer equipment, a storage medium and a computer program product. In an image processing process, the image is divided into semantic groups of different sizes, each semantic group is preprocessed to obtain difference data and centroid data, linear calculation is performed on the difference data and the centroid data to obtain differential data, multi-head self-attention calculation is performed on the differential data to obtain multi-head score data, asynchronous activation is performed on the multi-head score data to obtain activation values, and multi-head self-attention calculation is performed based on the activation values. The application solves the problem that the existing VIT reasoning acceleration scheme cannot fully utilize visual information and cannot adjust resources according to different images, resulting in low reasoning efficiency. The efficiency and adaptability are high.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a ViT differential inference method, apparatus, computer device, storage medium, and computer program product. Background Technology

[0002] In the fields of computer vision and inference acceleration, memory requirements, computational costs, and resource requirements are significant challenges. Existing VIT inference acceleration solutions primarily focus on sparse computation and label pruning.

[0003] However, these optimization techniques influenced by NLP cannot fully utilize visual information, such as the fact that an image contains multiple layers of semantic information, including the sky, grass, and people. It is highly significant to accelerate VIT inference by leveraging the inherent semantic information of the image itself. Furthermore, different images possess different semantic information; therefore, adjusting runtime resources for different images to compute different semantic groups is crucial for accelerating VIT inference. Summary of the Invention

[0004] Therefore, it is necessary to provide a ViT differential inference method, apparatus, computer device, storage medium, and computer program product to address the aforementioned technical problems.

[0005] Firstly, this application provides a ViT differential inference method, the method comprising:

[0006] During image processing, the image is divided into semantic groups of different sizes;

[0007] Preprocessing is performed on each semantic group to obtain difference data and centroid data;

[0008] Linear calculations are performed on the difference data and centroid data to obtain the difference data;

[0009] Multi-head self-attention calculation is performed on the differential data to obtain multi-head score data;

[0010] Asynchronous activation is performed on the multi-head score data to obtain activation values;

[0011] Multi-head self-attention calculation is performed based on the activation value.

[0012] In one embodiment, the preprocessing of each semantic group to obtain difference data and centroid data includes:

[0013] For each semantic group, preprocessing is performed to calculate the centroid of each semantic group and the difference between the tokens within the group and the centroid;

[0014] The centroid data includes the centroid of each semantic group, and the difference data includes the difference between the token and the centroid within each semantic group.

[0015] In one embodiment, the centroid of each semantic group is expressed as:

[0016]

[0017] The difference between the token and the centroid within each semantic group is expressed as:

[0018]

[0019] Where i represents the semantic group number, and c is the subscript representing the centroid. Let L represent the centroid of the i-th semantic group, and let Δx represent the length of the i-th semantic group. i Indicates Token x i The difference between the center of mass and the centroid.

[0020] In one embodiment, the formula for calculating the difference data by performing linear calculations on the difference data and centroid data is as follows:

[0021] ΔQ i =Δx i ·W Q ,

[0022] ΔK i =Δx i ·W K ;

[0023] ΔV i =Δx i ·W V ;

[0024] in, ΔQ represents the centroid value of the i-th semantic group (Query, Key, Value). i ΔK i ΔV i This represents the difference data of the i-th semantic group Query, Key, and Value; Q i W represents the original data of the i-th semantic group Query; Q W K W V This represents the weighted data for the Query, Key, and Value.

[0025] In one embodiment, the formula for calculating the multi-head self-attention data by performing multi-head self-attention calculation on the differential data is as follows:

[0026]

[0027] in, For the centroid multi-head fractional data, S i This is the score data for the Token multi-head position.

[0028] In one embodiment, the method further includes:

[0029] Establish a PE unit, which is at least capable of performing the linear computation and the multi-head self-attention computation.

[0030] Secondly, this application also provides a multi-ViT differential inference device, the device comprising:

[0031] A grouping module is used to divide the image into semantic groups of different sizes during image processing;

[0032] The preprocessing module is used to preprocess each semantic group to obtain differential data and centroid data;

[0033] The first calculation module is used to perform linear calculations on the difference data and centroid data to obtain difference data;

[0034] The second calculation module is used to perform multi-head self-attention calculation on the differential data to obtain multi-head score data;

[0035] The activation module is used to asynchronously activate the multi-head score data to obtain an activation value;

[0036] The third calculation module is used to perform multi-head self-attention calculation based on the activation value.

[0037] Thirdly, this application also provides a computer device. The computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to perform the following steps:

[0038] During image processing, the image is divided into semantic groups of different sizes;

[0039] Preprocessing is performed on each semantic group to obtain difference data and centroid data;

[0040] Linear calculations are performed on the difference data and centroid data to obtain the difference data;

[0041] Multi-head self-attention calculation is performed on the differential data to obtain multi-head score data;

[0042] Asynchronous activation is performed on the multi-head score data to obtain activation values;

[0043] Multi-head self-attention calculation is performed based on the activation value.

[0044] Fourthly, this application also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program thereon, which, when executed by a processor, performs the following steps:

[0045] During image processing, the image is divided into semantic groups of different sizes;

[0046] Preprocessing is performed on each semantic group to obtain difference data and centroid data;

[0047] Linear calculations are performed on the difference data and centroid data to obtain the difference data;

[0048] Multi-head self-attention calculation is performed on the differential data to obtain multi-head score data;

[0049] Asynchronous activation is performed on the multi-head score data to obtain activation values;

[0050] Multi-head self-attention calculation is performed based on the activation value.

[0051] Fifthly, this application also provides a computer program product. The computer program product includes a computer program that, when executed by a processor, performs the following steps:

[0052] During image processing, the image is divided into semantic groups of different sizes;

[0053] Preprocessing is performed on each semantic group to obtain difference data and centroid data;

[0054] Linear calculations are performed on the difference data and centroid data to obtain the difference data;

[0055] Multi-head self-attention calculation is performed on the differential data to obtain multi-head score data;

[0056] Asynchronous activation is performed on the multi-head score data to obtain activation values;

[0057] Multi-head self-attention calculation is performed based on the activation value.

[0058] The aforementioned ViT differential inference method, apparatus, computer device, storage medium, and computer program product, through image processing, divides the image into semantic groups of different sizes, preprocesses each semantic group to obtain difference data and centroid data, performs linear calculations on the difference data and centroid data to obtain difference data, performs multi-head self-attention calculations on the difference data to obtain multi-head score data, performs asynchronous activation on the multi-head score data to obtain activation values, and performs multi-head self-attention calculations based on the activation values; thus, it solves the problem that existing ViT inference acceleration schemes cannot fully utilize visual information and cannot adjust resources specifically for different images, resulting in low inference efficiency; it achieves acceleration of ViT inference by utilizing the inherent semantic level information of the image, with high efficiency and adaptability. Attached Figure Description

[0059] Figure 1 This is a flowchart illustrating the ViT differential inference method in one embodiment;

[0060] Figure 2 This is a schematic diagram of the PE unit design in one embodiment;

[0061] Figure 3 This is a structural block diagram of the ViT differential inference device in one embodiment;

[0062] Figure 4 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0063] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0064] In one embodiment of this application, such as Figure 1 As shown, a ViT differential inference method is provided, the method comprising:

[0065] S100, during image processing, the image is divided into semantic groups of different sizes.

[0066] Images inherently possess semantic information, such as sky, grass, and people. Tokens with the same semantic information also share similar data. This application employs a hardware-friendly Locality-Sense Hashing (LSH) method to hash each token. Tokens with the same semantics have a very high probability of sharing the same hash code. Therefore, different tokens can be divided into different semantic groups using their hash codes. Specifically, in this embodiment, LSH technology is used to divide the image into semantic groups of different sizes. The formula is:

[0067]

[0068] In the formula, α represents a set of vectors with values ​​ranging from (0, 1) according to a normal distribution, β represents a numerical value ranging from (0, γ), and γ represents the width of the hash bucket. Input x i Based on the above formula, a hash code H(x) is output. i Tokens with similar semantic information are highly likely to have the same hash code. We group tokens with the same hash code into a semantic group.

[0069] S200 preprocesses each semantic group to obtain difference data and centroid data.

[0070] Data within the same semantic group exhibits similarity. To fully leverage this similarity, we adjust the original token data. In one implementation, each semantic group needs to be preprocessed to calculate the centroid of each semantic group and the difference between the tokens within the group and the centroid; the centroid data includes the centroid of each semantic group, and the difference data includes the difference between the tokens within each semantic group and the centroid.

[0071] Specifically, the centroid of each semantic group is represented as follows:

[0072]

[0073] The difference between the token and the centroid within each semantic group is expressed as:

[0074]

[0075] Where i represents the semantic group number. The subscript c represents the centroid. Let L represent the centroid of the i-th semantic group, and let Δx represent the length of the i-th semantic group. i Indicates Token x i The difference between the center of mass and the centroid.

[0076] Because data within the same semantic group is similar, this difference will be much smaller than the original data. Therefore, we used a lower-precision bit width to store the difference data. The advantage of this operation is that it greatly reduces data storage without affecting the accuracy of the original data. It should be noted that the original data refers to the original data of the tokens within each semantic group.

[0077] S300, perform linear calculations on the difference data and centroid data to obtain the difference data.

[0078] Specifically, linear calculations are performed on the difference data and centroid data to obtain the calculation formula for the difference data:

[0079] ΔQ i =Δx i ·W Q ,

[0080] ΔK i =Δx i ·W K ;

[0081] ΔV i =Δx i WV;

[0082] in, ΔQ represents the centroid value of the i-th semantic group (Query, Key, Value). i ΔK i ΔV i This represents the difference data of the i-th semantic group Query, Key, and Value; Q i W represents the original data of the i-th semantic group Query; Q W K W V This represents the weighted data for the Query, Key, and Value.

[0083] S400, perform multi-head self-attention calculation on the differential data to obtain multi-head score data.

[0084] Specifically, multi-head self-attention calculation is performed on the differential data to obtain the multi-head score data calculation formula as follows:

[0085]

[0086] in, For the centroid multi-head fractional data, S i This is the score data for the Token multi-head position.

[0087] Let's take one of the semantic groups as an example. First, we calculate the query centroid. With Key centroid The linear product; then, we compute the original data Q in the query. i Difference data ΔK with Key i The product of the two data points; finally, we add the two data points together to get the final data S. i .

[0088] S500, asynchronous activation is performed on the multi-head score data to obtain the activation value.

[0089] Specifically, based on the multi-head score data obtained from S400, we use an asynchronous activation method to calculate the SoftMax value. First, we calculate the local maximum and the sum of the local exponential scores, and maintain two global variables: the global maximum and the global exponential score sum. After each calculation of the current local score, these two global variables need to be updated. Once all local scores have been calculated, the final activation value is updated using these two global variables. and S i ′.

[0090] S600, perform multi-head self-attention calculation based on the activation value.

[0091] Specifically, based on the activation values ​​obtained from S500 and S i This refers to the latter part of the multi-head self-attention calculation. The calculation formula is as follows:

[0092]

[0093] This represents the output value of the i-th semantic group. and S i ′ represents the centroid score after activation and the original score, respectively. This represents the centroid value of Value. ΔV i This represents the difference in the value.

[0094] Furthermore, as the above steps show, the calculation steps include multiplication of two original data matrices, as well as multiplication of the difference data with the original data matrix. Therefore, we designed a PE unit capable of performing both types of calculations. This PE unit can be configured at runtime through a local control unit to flexibly switch between the two calculation modes. In addition, the PE unit has a local buffer to cache weights and intermediate results, eliminating the need for frequent communication with the global buffer.

[0095] Furthermore, different images possess different semantic information. The size and number of semantic groups also vary significantly depending on the image. To address these challenges, please refer to... Figure 2 We designed a configurable PE array, consisting of PE groups of the same size. Each PE group contains a fixed number of 4 PE units. We designed a global control unit for runtime scheduling. We use a polling approach, continuously reading the next computation task from the task pool and allocating multiple PE groups for computation based on the computational load. If no suitable PE group combination is found, the current task is skipped, and the next task is polled.

[0096] The aforementioned ViT differential inference method, apparatus, computer device, storage medium, and computer program product, through image processing, divides the image into semantic groups of different sizes, preprocesses each semantic group to obtain difference data and centroid data, performs linear calculations on the difference data and centroid data to obtain difference data, performs multi-head self-attention calculations on the difference data to obtain multi-head score data, performs asynchronous activation on the multi-head score data to obtain activation values, and performs multi-head self-attention calculations based on the activation values; thus, it solves the problem that existing ViT inference acceleration schemes cannot fully utilize visual information and cannot adjust resources specifically for different images, resulting in low inference efficiency; it achieves acceleration of ViT inference by utilizing the inherent semantic level information of the image, with high efficiency and adaptability.

[0097] It should be understood that although the steps in the flowcharts of the above embodiments are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the above embodiments may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0098] Based on the same inventive concept, this application also provides a ViT differential inference device for implementing the above-described ViT differential inference method. The solution provided by this device is similar to the solution described in the above-described method; therefore, the specific limitations of one or more ViT differential inference device embodiments provided below can be found in the limitations of the ViT differential inference method described above, and will not be repeated here.

[0099] In one embodiment, such as Figure 3As shown, a ViT differential inference apparatus is provided, comprising:

[0100] The grouping module 100 is used to divide the image into semantic groups of different sizes during image processing;

[0101] The preprocessing module 200 is used to preprocess each semantic group to obtain differential data and centroid data;

[0102] The first calculation module 300 is used to perform linear calculations on the difference data and centroid data to obtain difference data;

[0103] The second calculation module 400 is used to perform multi-head self-attention calculation on the differential data to obtain multi-head score data;

[0104] Activation module 500 is used to asynchronously activate the multi-head score data to obtain an activation value;

[0105] The third calculation module 600 is used to perform multi-head self-attention calculation based on the activation value.

[0106] In one embodiment, the preprocessing module 200 is further configured to preprocess each semantic group, calculate the centroid of each semantic group and the difference between the tokens within the group and the centroid.

[0107] The centroid data includes the centroid of each semantic group, and the difference data includes the difference between the token and the centroid within each semantic group.

[0108] In one embodiment, the centroid of each semantic group is expressed as:

[0109]

[0110] The difference between the token and the centroid within each semantic group is expressed as:

[0111]

[0112] Where i represents the semantic group number. The subscript c represents the centroid. Let L represent the centroid of the i-th semantic group, and let Δx represent the length of the i-th semantic group. i Indicates Token x i The difference between the center of mass and the centroid.

[0113] In one embodiment, the formula for calculating the difference data by performing linear calculations on the difference data and centroid data is as follows:

[0114] ΔQ i =Δx i ·W Q ,

[0115] ΔK i =Δx i ·W K

[0116] ΔV i =Δx i ·W V

[0117] in, ΔQ represents the centroid value of the i-th semantic group (Query, Key, Value). i ΔK i ΔV i This represents the difference data of the i-th semantic group Query, Key, and Value; Q i W represents the original data of the i-th semantic group Query; Q W K W V This represents the weighted data for the Query, Key, and Value.

[0118] In one embodiment, the formula for calculating the multi-head self-attention data by performing multi-head self-attention calculation on the differential data is as follows:

[0119]

[0120] in, For the centroid multi-head fractional data, S i This is the score data for the Token multi-head position.

[0121] In one embodiment, the apparatus further includes a setup module for: setting up a PE unit, the PE unit being capable of performing at least the linear computation and the multi-head self-attention computation.

[0122] The modules in the aforementioned ViT differential inference device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the operations corresponding to each module.

[0123] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 4As shown, the computer device includes a processor, memory, and a network interface connected via a system bus. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The database stores preset data. The network interface communicates with external terminals via a network connection. When executed by the processor, the computer program implements a ViT differential inference method.

[0124] Those skilled in the art will understand that Figure 4 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0125] In one embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the ViT differential inference method described above.

[0126] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the ViT differential inference method described above.

[0127] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the ViT differential inference method described above.

[0128] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0129] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0130] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A ViT differential inference method, characterized in that, The method includes: During image processing, the image is divided into semantic groups of different sizes; Preprocessing is performed on each semantic group to obtain difference data and centroid data; Linear calculations are performed on the difference data and centroid data to obtain the difference data; Multi-head self-attention calculation is performed on the differential data to obtain multi-head score data; Asynchronous activation is performed on the multi-head score data to obtain activation values; Multi-head self-attention calculation is performed based on the activation value; The preprocessing of each semantic group to obtain difference data and centroid data includes: For each semantic group, preprocessing is performed to calculate the centroid of each semantic group and the difference between the tokens within the group and the centroid; The centroid data includes the centroid of each semantic group, and the difference data includes the difference between the token and the centroid within each semantic group; The formula for calculating the difference data by performing linear calculations on the difference data and centroid data is as follows: ； ； ； in, This represents the centroid value of the i-th semantic group Query, Key, and Value; This represents the difference data of the Query, Key, and Value of the i-th semantic group; This represents the original data of the i-th semantic group Query; This represents the weighted data for the Query, Key, and Value. This represents the value of the centroid of the i-th semantic group. Represents Token The difference from the center of mass; The formula for calculating the multi-head self-attention data by performing multi-head self-attention calculation on the difference data is as follows: ； in, For centroid multi-head fractional data, For token multi-head score data; The asynchronous activation of the multi-head score data to obtain the activation value includes: Based on the multi-head score data, an asynchronous activation method is used to calculate the SoftMax value. First, the local maximum and the sum of local exponential scores are calculated, and two global variables, the global maximum and the global exponential score sum, are maintained. After each calculation of the current local score, these two global variables need to be updated. Once all local scores have been calculated, the final activation value is updated using these two global variables. and ; The multi-head self-attention calculation based on the activation value includes: Based on the activation value and The latter part of the multi-head self-attention calculation is performed, and the calculation formula is as follows: ; This represents the output value of the i-th semantic group. and These represent the centroid long position score data and the token long position score data after activation, respectively. This represents the centroid value of Value. This represents the difference in the value.

2. The method according to claim 1, characterized in that, The centroid of each semantic group is represented as follows: ； The difference between the token and the centroid within each semantic group is expressed as: ； Where i represents the semantic group number, and c is the subscript representing the centroid. This represents the value of the centroid of the i-th semantic group. This represents the length of the i-th semantic group. Represents Token The difference between the center of mass and the centroid.

3. The method according to claim 1, characterized in that, The method further includes: Establish a PE unit, which is at least capable of performing the linear computation and the multi-head self-attention computation.

4. A ViT differential inference device, characterized in that, The device includes: A grouping module is used to divide the image into semantic groups of different sizes during image processing; The preprocessing module is used to preprocess each semantic group to obtain difference data and centroid data; The first calculation module is used to perform linear calculations on the difference data and centroid data to obtain difference data; The second calculation module is used to perform multi-head self-attention calculation on the differential data to obtain multi-head score data; The activation module is used to asynchronously activate the multi-head score data to obtain an activation value; The third calculation module is used to perform multi-head self-attention calculation based on the activation value; The preprocessing module is also used for: For each semantic group, preprocessing is performed to calculate the centroid of each semantic group and the difference between the tokens within the group and the centroid; The centroid data includes the centroid of each semantic group, and the difference data includes the difference between the token and the centroid within each semantic group; The formula for calculating the difference data by performing linear calculations on the difference data and centroid data is as follows: ； ； ； in, This represents the centroid value of the i-th semantic group Query, Key, and Value; This represents the difference data of the Query, Key, and Value of the i-th semantic group; This represents the original data of the i-th semantic group Query; This represents the weighted data for the Query, Key, and Value. This represents the value of the centroid of the i-th semantic group. Represents Token The difference from the center of mass; The formula for calculating the multi-head self-attention data by performing multi-head self-attention calculation on the difference data is as follows: ； in, For centroid multi-head fractional data, For token multi-head score data; The activation module is also used for: Based on the multi-head score data, an asynchronous activation method is used to calculate the SoftMax value. First, the local maximum value and the sum of local exponential scores are calculated, and two global variables, the global maximum value and the global exponential score sum, are maintained. After each calculation of the current local score, these two global variables need to be updated. Once all local scores have been calculated, the final activation value is updated using these two global variables. and ; The third calculation module is also used for: Based on the activation value and The latter part of the multi-head self-attention calculation is performed, and the calculation formula is as follows: ; This represents the output value of the i-th semantic group. and These represent the centroid long position score data and the token long position score data after activation, respectively. This represents the centroid value of Value. This represents the difference in the value.

5. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 3.

6. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 3.

7. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 3.

Images