Method, device, medium, equipment and product for implementing vertical pixel parallel depth calculation

Abstract

The present invention discloses a method, device, medium, equipment and product for implementing depth calculation in parallel with pixels in the vertical direction. The method comprises: combining each channel group with a channel group containing N T Map the thread bundles of the threads; obtain the jump step size for vertical data acquisition and the parameter configuration of the sliding window operator; allocate register resources for each thread to form a sliding window data buffer; determine the N number of the current scan round H When executing the current scanning round, in each column slice involved in the calculation, start from each row starting coordinate, take the jump step length as the span, and scan N T The pixel groups at the row and column positions are loaded into the register groups of each thread in the corresponding thread bundle in sequence, so that each thread can handle the computation task of the corresponding local sliding window field of view. The present invention implements independent computation task allocation at the thread level through the pixel parallelization strategy between threads, which can maximize the utilization of allocated hardware resources.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a method, device, computer-readable storage medium, electronic device, and computer program product for implementing vertical pixel-parallel depth operations. Background Art

[0002] When performing deep operations on artificial intelligence processors, such as depth-wise convolution, pooling, and grouped convolution, existing methods use a channel parallel strategy between threads. This reduces the computing power utilization of the artificial intelligence processor due to the mismatch between thread scheduling granularity and task allocation.

[0003] Taking depth-wise convolution as an example, the existing method assigns a single channel convolution calculation task to each thread. T threads, when the number of channels C of the input tensor cannot be N T When divided evenly, some threads in a thread bundle will be idle, that is, the enabled threads cannot be fully utilized, and the computing power utilization of the AI processor will be reduced to . Summary of the Invention

[0004] The purpose of the embodiments of the present invention is to provide a method, device, computer-readable storage medium, electronic device and computer program product for implementing vertical pixel-parallel depth calculations, which realize independent computing task allocation at the thread level through a pixel parallel strategy between threads, and assign corresponding local sliding window field of view computing tasks to each thread, thereby maximizing the utilization of allocated hardware resources and improving computing efficiency.

[0005] A first embodiment of the present invention provides a method for implementing pixel-parallel depth calculation in a vertical direction, comprising:

[0006] The input tensor is divided into m channel groups and assigned to m thread warps for independent processing; each thread warp contains N T threads; each channel group contains N C Pixels of channels; N T ≥1; N C ≥1;m≥1;

[0007] Get the jump step size for vertical data acquisition in the channel group, as well as the height, width, vertical step size and N of the sliding window operator H row base addresses; where N H is equal to said height;

[0008] Assign N to each of the threads Hregister clusters; each register cluster contains y register groups; each register group is used to store N registers at the same row and column position. C channel pixels form a pixel group; y is not less than the width and is equal to the number of column slices participating in the calculation in each channel group;

[0009] Determine the N of the current scanning round according to the row base address and the vertical step size H The starting coordinates of each row;

[0010] When executing the current scanning round, in each column slice participating in the calculation, starting from each row starting point coordinate, with the jump step length as the span, scan N T row and column positions, and scan the N T The pixel groups at the row and column positions are sequentially loaded into the corresponding register groups of each thread in the warp;

[0011] Each thread calculates the loaded pixel group to obtain an output result within the corresponding local sliding window field of view.

[0012] Optionally, the jump step size is the product of the height and the vertical step size.

[0013] Optionally, the method further includes:

[0014] By N H The round scan is executed so that each of the thread warps completes the coverage of the entire sliding window area of the sliding window operator in the corresponding channel group.

[0015] Optionally, when executing the current scanning round, in each column slice participating in the calculation, starting from each row starting point coordinate, scan N T row and column positions, and scan the N T The pixel groups at the row and column positions are sequentially loaded into the corresponding register groups of each thread in the warp, including:

[0016] When performing the i-th round of vertical data scanning, in the w-th column slice participating in the calculation in the j-th channel group, the h-th row starting point coordinate is used as the starting point of the h-th vertical scanning route, and the jump step length is used as the span to scan N T row and column positions, and then N T The pixel group at the row and column position is loaded into the wth register group of the hth register cluster of each thread in the jth thread warp; wherein, 1≤i≤N H ; 1≤j≤m; 1≤w≤y; 1≤h≤N H .

[0017] Optionally, the input tensor consists of a feature data block, or is obtained by splicing multiple feature data blocks in the vertical direction of the original space, or is composed of any sub-data block split from a feature data block.

[0018] Optionally, the column slices involved in the calculation are column slices traversed by the sliding window operator when sliding in the horizontal direction.

[0019] Optionally, the row base address and the vertical step size are used to determine the N number of the current scanning round. H The starting point coordinates of each row include:

[0020] The starting point coordinates of the row are determined by the following formula:

[0021] ;

[0022] in, For the Scanning round The starting coordinates of each row; For the row base address; is the vertical step length.

[0023] Optionally, the sliding window operator is a pooling operator;

[0024] The step of calculating the loaded pixel group by each thread to obtain an output result within the corresponding local sliding window field of view includes:

[0025] In any of the thread warps, each thread performs a channel-isolated pooling operation on the loaded pixel group to obtain N pixels within the corresponding local sliding window field of view. C The pooling result of channels.

[0026] Optionally, the sliding window operator is a single-channel convolution kernel;

[0027] The step of calculating the loaded pixel group by each thread to obtain an output result within the corresponding local sliding window field of view includes:

[0028] Broadcasting the convolution weight corresponding to each register group to all threads in the corresponding warp in scalar form;

[0029] In any of the thread bundles, each thread performs channel-isolated multiplication and addition operations on the loaded pixel groups and convolution weights to obtain the corresponding N local sliding window field of view. C The convolution result of channels.

[0030] Optionally, the sliding window operator is a multi-channel convolution kernel;

[0031] The step of calculating the loaded pixel group by each thread to obtain an output result within the corresponding local sliding window field of view includes:

[0032] Broadcasting the convolution weight corresponding to each register group to all threads in the corresponding warp in scalar form;

[0033] In any of the thread bundles, each thread performs a channel fusion multiplication and addition operation on the loaded pixel group and the convolution weight to obtain an output feature value within the corresponding local sliding window field of view; wherein each of the channel groups is used to represent an input group in the grouped convolution operation.

[0034] Optionally, the method further includes:

[0035] Before the thread performs the multiplication and addition operation, an out-of-bounds judgment is performed on the coordinates of the pixel values in the pixel group, and the pixel values corresponding to the out-of-bounds coordinates are masked to specified values to obtain new pixel values.

[0036] A second embodiment of the present invention provides a device for implementing vertical pixel-parallel depth calculation, including:

[0037] The thread warp configuration module is used to divide the input tensor into m channel groups and assign them to m thread warps for independent processing; each thread warp contains N T threads; each channel group contains N C Pixels of channels; N T ≥1; N C ≥1;m≥1;

[0038] The parameter acquisition module is used to obtain the jump step size for vertical data acquisition in the channel group, as well as the height, width, vertical step size and N of the sliding window operator. H row base addresses; where N H is equal to said height;

[0039] Thread configuration module, used to allocate N to each thread H register clusters; each register cluster contains y register groups; each register group is used to store N registers at the same row and column position. C channel pixels form a pixel group; y is not less than the width and is equal to the number of column slices participating in the calculation in each channel group;

[0040] A starting point determination module is used to determine the N number of the current scanning round according to the row base address and the vertical step size. H The starting coordinates of each row;

[0041] The pixel loading module is used to scan N pixels in each column slice participating in the calculation, starting from each row starting point coordinate and taking the jump step length as the span when executing the current scanning round. T row and column positions, and scan the N T The pixel groups at the row and column positions are sequentially loaded into the corresponding register groups of each thread in the warp;

[0042] The thread calculation module is used to calculate the loaded pixel group through each thread to obtain the output result within the corresponding local sliding window field of view.

[0043] An embodiment of the third aspect of the present invention provides a computer-readable storage medium, which includes a stored computer program; wherein, when the computer program is running, it controls the device where the computer-readable storage medium is located to execute the vertical pixel-parallel depth operation implementation method described in any one of the above-mentioned first aspects.

[0044] A fourth aspect of the present invention provides a computer program product, comprising a computer program, which, when executed by a processor, implements the method for implementing vertical pixel-parallel depth calculation as described in any one of the first aspects above.

[0045] An embodiment of the fifth aspect of the present invention provides an electronic device, comprising a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein when executing the computer program, the processor implements the vertical pixel parallel depth operation implementation method described in any one of the first aspects above.

[0046] Compared with the prior art, the embodiment of the present invention provides a method for implementing vertical pixel parallel depth operations, which has the following beneficial effects: the embodiment of the present invention first divides the input tensor into multiple channel groups and assigns them to different thread warps for independent processing; then, combined with the parameter configuration of the sliding window operator, register resources are allocated to each thread to form a sliding window data buffer; then, the N of the current scanning round is calculated. H The starting coordinates of each row are scanned in each column slice involved in the calculation, starting from each row starting coordinate and taking the jump step length as the span. T row and column positions, and scan the N T The pixel groups at the row and column positions are sequentially loaded into the register groups of each thread within the corresponding warp, allowing each thread to process the computational tasks for its corresponding local sliding window field of view. Therefore, the embodiments of the present invention implement independent computational task allocation at the thread level through a pixel parallelization strategy between threads. By assigning each thread the computational tasks for its corresponding local sliding window field of view, the allocated hardware resources can be maximized, thereby improving computational efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 This is a flow chart of an embodiment of implementing a vertical pixel-parallel depth calculation provided by the present invention;

[0048] Figure 2 is a schematic diagram of an embodiment of a row base address provided by the present invention;

[0049] Figure 3 is a schematic diagram of an embodiment of performing multi-channel vertical scanning in a channel group provided by the present invention;

[0050] Figure 4 is a schematic diagram of an embodiment of the relationship between a pixel group and a register group provided by the present invention;

[0051] Figure 5 This is a schematic diagram of an embodiment of the present invention in which a thread holds local sliding window view data;

[0052] Figure 6 is a schematic diagram of another embodiment of the relationship between the pixel group and the register group provided by the present invention;

[0053] Figure 7 This is a schematic diagram of another embodiment of the present invention in which a thread holds local sliding window view data;

[0054] Figure 8 This is an example diagram of channel parallelism when threads perform depth-wise convolution operations provided by the prior art;

[0055] Figure 9 1 is a schematic structural diagram of an embodiment of a device for implementing vertical pixel-parallel depth calculation provided by the present invention;

[0056] Figure 10 It is a structural diagram of an embodiment of an electronic device provided by the present invention. DETAILED DESCRIPTION

[0057] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in this technical field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0058] The artificial intelligence processor involved in the present invention can be any one of a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a TPU (Tensor Processing Unit), an NPU (Neural Network Processing Unit), a DPU (Deep Learning Processing Unit), an APU (Accelerated Processing Unit), and a GPGPU (General-Purpose computing on Graphics Processing Unit), and is determined when the embodiment of the present invention is applied to a specific product or technology.

[0059] Furthermore, in the embodiments of the present invention, the so-called "pixel" is not an image pixel value in the traditional sense, but an abstract unit used to represent any type of data. Specifically, a "pixel" corresponds to a data unit at a certain position in the input tensor.

[0060] The following uses a GPU as an example to illustrate the method, device, computer-readable storage medium, electronic device, and computer program product for implementing vertical pixel-parallel depth calculations provided by embodiments of the present invention.

[0061] See also Figure 1 , is a flow chart of an embodiment of the vertical pixel-parallel depth calculation implementation provided by the present invention.

[0062] The first embodiment of the present invention provides a method for implementing pixel-parallel depth calculation in the vertical direction, including steps S1 to S6, as follows:

[0063] Step S1: Divide the input tensor into m channel groups and assign them to m thread warps for independent processing; each thread warp contains N T threads; each channel group contains N C Pixels of channels; N T ≥1; N C ≥1;m≥1;

[0064] Step S2: Obtain the jump step size for vertical data acquisition in the channel group, as well as the height, width, vertical step size and N of the sliding window operator. H row base addresses; where N H is equal to said height;

[0065] Step S3: Allocate N to each of the threads H register clusters; each register cluster contains y register groups; each register group is used to store N registers at the same row and column position. C channel pixels form a pixel group; y is not less than the width and is equal to the number of column slices participating in the calculation in each channel group;

[0066] Step S4: Determine the N of the current scanning round according to the row base address and the vertical step size. H The starting coordinates of each row;

[0067] Step S5: When executing the current scanning round, in each column slice participating in the calculation, starting from each row starting point coordinate, with the jump step length as the span, scan N T row and column positions, and scan the N T The pixel groups at the row and column positions are sequentially loaded into the corresponding register groups of each thread in the warp;

[0068] Step S6: Calculate the loaded pixel group through each thread to obtain the output result within the corresponding local sliding window field of view.

[0069] It should be noted that the sliding window operator in the embodiment of the present invention includes: a pooling operator, a single-channel convolution kernel (which can be used for depth-wise convolution operations) or a multi-channel convolution kernel (which can be used for grouped convolution operations).

[0070] In step S1, the input tensor of dimension H×W×C is divided into m channel groups along the channel dimension, and each channel group is associated with a T The thread bundles of the threads form a binding relationship (N T Usually takes a value of 32); where H is the height of the input tensor, W is the width of the input tensor, C is the total number of channels of the input tensor, and N C The number of channels contained in each channel group; the dimension of each channel group is H×W×N C .

[0071] In step S2, the jump step length (i.e., the cross-row distance) of the vertical data access is obtained to implement vertical cross-row access in the channel group; for example, the jump step length S jump If it is 3, the current data to be read is in the 1st row, and the next data to be read will be in the 4th row.

[0072] At the same time, get the height K of the sliding window operator H , width K W , vertical step length , and the row base address set ;in, It is used to represent the row coordinate set corresponding to the starting position of each element in the sliding window operator. Obviously, The number of base addresses in the bank N H and the height K of the sliding window operator H Equal. Figure 2 FIG. 1 is a schematic diagram of an embodiment of a row base address provided by the present invention. Figure 2 middle, is the row and column position coordinates of the pixel data; is the row coordinate, The sliding window operator is a 3×3 single-channel convolution kernel, and the number of elements involved in the coverage operation in the two-dimensional plane is 9, and the elements are convolution weights. ;in, is the vertical offset between the convolution weight and the convolution center, is the horizontal offset between the convolution weight and the convolution center; the two-dimensional plane is determined by the vertical and horizontal directions of the input tensor and is independent of the depth / channel dimension. When the 3×3 single-channel convolution kernel performs the first coverage operation, it covers a total of 3 rows of pixel data in the two-dimensional plane. The row coordinates of these 3 rows are expressed as = In other words, the row base address refers to the row coordinates of each row element in the sliding window operator that are initially covered in the two-dimensional plane; for example, the first row of convolution weights in a 3×3 single-channel convolution kernel is The first row of the two-dimensional plane is covered with a coordinate of -1, so the first row base address is -1; similarly, the third row of convolution weights is In the two-dimensional plane, the first row to be covered has a coordinate of 1, and the base address of the third row is 1. Since the base address is independent of the depth (channel) dimension, the 3×3 single-channel convolution kernel is expanded to a multi-channel convolution kernel (such as 3×3×N C ) and then we get Still equal to .

[0073] In step S3, N is allocated to each thread in each warp. H register clusters, and each register cluster contains y register groups, that is, a single thread is allocated N H ×y dedicated register groups to form a sliding window data buffer; where y is the number of column slices involved in the calculation in each channel group, and the dimension of the column slice is H×1×N C In addition, each register group is used to store the N C Channel pixels form a pixel group, so the dimension of the pixel group is 1×1×N C In the case where the register can store 32-bit floating point numbers and each pixel value is a 16-bit floating point number, if NC =2, each register group consists of only one register; if N C >2, each register group consists of registers.

[0074] The number of column slices y is equal to the sliding window operator width K W In the case of K H ×K W register groups, each thread can store N in a local sliding window operator. C channel pixels (that is, the data dimension held by the thread is K H ×K W ×N C ). When the number of column slices y is greater than the sliding window operator width K W In the case of , it indicates that the thread has a sliding window operator horizontal sliding coverage data preloading mechanism to avoid repeated access to memory; for example, when y=K W +1, and horizontal step length = 1, the thread can process data from two horizontally adjacent local sliding window views. Through this preloading mechanism, when calculating the second local sliding window view, the data overlapping with the first local sliding window view can be directly reused, avoiding re-reading memory and effectively reducing unnecessary memory access overhead.

[0075] In step S4, according to the vertical step length Establish a scrolling mechanism for the row start coordinates. In the first round of vertical data scanning, N H The starting point coordinates of each row and N H The base addresses of the rows are equal. In each subsequent round of vertical data scanning, the row starting point coordinates of the previous round are incremented. Units, generate new row starting point coordinates. For example, when the row starting point coordinate set of the first round of scanning is {-1,0,1}, and the vertical step length is When is 1, the row starting point coordinate set of the second round of scanning will become {0, 1, 2}. This rolling mechanism ensures that after multiple rounds of scanning, each thread warp can cover the entire sliding window area in the corresponding channel group.

[0076] In step S5, when executing the current scanning round, each column slice starts N H Each vertical scanning route starts from the corresponding row starting point coordinate, takes the jump step length as the span, and continuously visits N T row and column positions, and sequentially place these N T The pixel groups at the row and column positions are loaded into the register group corresponding to each thread. For example, suppose the row starting point coordinate set of the first round of scanning is {-1,0,1}, and N T=32, the first vertical scanning route of the first column slice in the first channel group starts from the row starting point coordinate -1, takes the jump step size as the stride, continuously obtains 32 pixel groups, and sequentially loads these 32 pixel groups into the register group R0 in the register cluster 0 of the 32 threads in thread warp 0; similarly, the second vertical scanning route of the column slice starts from the row starting point coordinate 0, takes the jump step size as the stride, continuously obtains 32 pixel groups, and sequentially loads these 32 pixel groups into the register group R0 in the register cluster 1 of the 32 threads in thread warp 0.

[0077] In step S6, each thread processes K independently H ×y×N C pixel values (y≥K W ). As mentioned above, when y=K W When each thread holds K H ×K W ×N C pixel values; when y>K W When , the thread can preload the coverage data of the sliding window operator sliding in the horizontal direction. Therefore, each thread holds at least N of the local sliding window field of view. C It can store pixels in multiple channels and supports multiple operation modes, such as convolution and pooling.

[0078] It's worth noting that existing methods employ a channel-parallel strategy between threads. For example, for an input tensor with 90 channels, two thread warps (i.e., 128 threads) are required to complete the computation. However, these two warps can only process 90 local sliding window views (corresponding to the computation of 90 channels) during a single convolution calculation, failing to fully utilize the computing power of the hardware resources.

[0079] In contrast, the embodiment of the present invention adopts a pixel parallel strategy between threads, that is, a local sliding window field of view parallel strategy. Based on this strategy, when the above two thread warps perform a convolution calculation, they can process N pixels corresponding to at least 128 local sliding window fields of view. C Therefore, the embodiments of the present invention implement independent computational task allocation at the thread level through a pixel parallel strategy between threads, assigning each thread a computational task corresponding to a local sliding window field of view, thereby maximizing the utilization of allocated hardware resources and improving computational efficiency.

[0080] In an optional embodiment, the jump step length is the product of the height and the vertical step length.

[0081] Furthermore, through N H The round scan is executed so that each of the thread warps completes the coverage of the entire sliding window area of the sliding window operator in the corresponding channel group.

[0082] It should be noted that the embodiment of the present invention determines the jump step length using the following formula:

[0083] S jump = ×a; where S jump is the jumping stride length; is the vertical step size; a is the coverage scale factor, and a is a positive integer used to control the proportion of the local sliding window field of view covered by each thread warp after performing a round of scanning in the corresponding channel group to the total local sliding window field of view (i.e., 1 / a).

[0084] In order to more clearly describe the technical solutions provided by the embodiments of the present invention, some specific embodiments are provided below for reference:

[0085] Example 1: When a=1, S jump = , then when warp 0 performs a scan in channel group 0, the sliding window operator steps in the vertical direction according to the vertical step size Sliding (column-first order) is performed, so this scanning method is 1:1 global coverage and only one round of scanning is required.

[0086] Example 2: When a=K H When S jump = ×K H , then when warp 0 performs a round of scanning in channel group 0, the sliding window operator moves vertically according to ×K H This is equivalent to sliding from the original dense local sliding window view (corresponding to the step size ), the interval (K H -1) local sliding window field of view, sparse reading window. Therefore, this scanning method is 1:K H Sparse coverage, and need to perform K H Only by performing a round of staggered scanning can all sliding window areas in channel group 0 be vertically covered. In other words, each warp only covers 1 / K H =1 / N H The sliding window area of the proportion (sparse coverage), such a phased sparse coverage design splits the original dense sliding window task into K H (N H ) rounds, each round processes a different sparse subset.

[0087] It is worth noting that a=K HThis is a preferred solution in an embodiment of the present invention. In any scan round, each time a warp jumps the full height of the sliding window operator, the local sliding window field of view data it reads is completely staggered vertically (with no overlapping areas). This ensures that when executing multiple vertical scan routes within a column slice, the row and column positions of each scan are also staggered and non-overlapping, effectively avoiding memory access conflicts.

[0088] In an optional embodiment, the N number of the current scanning round is determined according to the row base address and the vertical step size. H The starting point coordinates of each row include:

[0089] The starting point coordinates of the row are determined by the following formula:

[0090] ;

[0091] in, For the Scanning round The starting coordinates of each row; For the row base address; is the vertical step length.

[0092] In combination with the above embodiment, in order to achieve complete sliding window coverage in the channel group, it is necessary to perform multiple rounds of staggered scanning in the vertical direction and follow the original sliding rule of the sliding window operator in the channel (i.e., according to the vertical step size Slide). Therefore, the coordinates of the starting point of the previous round are adjusted by the vertical step size Incrementally, the row starting point coordinates in the current scanning round are rolled up to ensure that the next scanning round covers the remaining sliding window area, thereby ensuring that after multiple scanning rounds are completed, each thread warp can cover the entire sliding window area in the corresponding channel group.

[0093] like Figure 3 FIG. 1 is a schematic diagram of an embodiment of performing multi-channel vertical scanning in a channel group provided by the present invention. Figure 3 In the example, channel group 0 contains pixels of 2 channels (N C =2); Represents a pixel group and consists of all pixels at the same row and column position in the channel group. The coordinates are ;in, is the row coordinate, is the column coordinate. The sliding window operator is a 3×3 single-channel convolution kernel with a horizontal step size of is 1, so the column slices 0, 1 and 2 involved in the calculation are adjacent and continuous. The height K of the sliding window operator H 3, which means that each column of slices needs to start 3 vertical scanning routes. is 1, covering the scaling factor a=K H Therefore, in the current scanning round, the thread warp jumps the sliding window operator for a complete height distance each time, and the local sliding window field of view data read is completely staggered in the vertical direction, as shown in Table 1.

[0094] Table 1. Example of data coverage based on vertical scanning

[0095]

[0096] exist Figure 3 In the figure, all column slices involved in the calculation have the same scanning path indicated by the gray arrow, black arrow and dotted arrow. In order to make the picture display clearer, the embodiment of the present invention only selectively shows part of the scanning path. As can be seen from Table 1, the first scanning path of the first round corresponds to the scanning path indicated by the gray arrow; the second scanning path of the first round corresponds to the scanning path indicated by the black arrow; the third scanning path of the first round corresponds to the scanning path indicated by the dotted arrow. In any round of scanning, the row positions scanned by the three vertical scanning routes of each column slice are completely staggered (no overlapping area), so no memory access conflict occurs. In addition, since each round of scanning only covers 1 / K H The sliding window area (sparse coverage) of the ratio is updated by rolling the starting coordinates of the rows, so that each round of scanning processes a different sparse subset. H =After 3 rounds of scanning, the sparse coverage is accumulated into full coverage, achieving complete access to the channel group.

[0097] In an optional embodiment, when executing the current scanning round, in each column slice participating in the calculation, starting from each row starting point coordinate, scan N T row and column positions, and scan the N T The pixel groups at the row and column positions are sequentially loaded into the corresponding register groups of each thread in the warp, including:

[0098] When performing the i-th round of vertical data scanning, in the w-th column slice participating in the calculation in the j-th channel group, the h-th row starting point coordinate is used as the starting point of the h-th vertical scanning route, and the jump step length is used as the span to scan N T row and column positions, and then N T The pixel group at the row and column position is loaded into the wth register group of the hth register cluster of each thread in the jth thread warp; wherein, 1≤i≤N H ; 1≤j≤m; 1≤w≤y; 1≤h≤N H .

[0099] like Figure 4FIG. 1 is a schematic diagram of an embodiment of the relationship between pixel groups and register groups provided by the present invention. Figure 4 In the example, Rx represents the index of the register group, and Tt represents the index of the thread in a warp; where 0≤x≤y-1, 0≤t≤N T -1. Combine Figure 3 and Figure 4 It can be seen that the specific mapping relationship between pixel groups and register groups in the embodiment of the present invention is: in the wth column slice participating in the calculation in the jth channel group, the kth pixel group on the hth vertical scanning path is stored in the wth register group of the hth register cluster of the kth thread in the jth thread warp; where 1≤k≤N T .

[0100] In an optional embodiment, the input tensor is composed of a feature data block, or is obtained by splicing multiple feature data blocks in the vertical direction of the original space, or is composed of any sub-data block split from a feature data block.

[0101] It should be noted that the input tensors in the embodiments of the present invention can be obtained through the following three main data organization methods, as long as the conditions of warp parallel processing of data and reasonable resource allocation are met:

[0102] (1) Single feature data block approach: If the feature data block has dimensions [H, W, C], all the sliding window areas it contains can meet the needs of the thread warp to perform multiple rounds of data scanning in the vertical direction, and the thread warp resources are sufficient to cover the total number of channels C of the entire feature data block, then the input tensor is directly constructed from the complete feature data block. This approach is suitable for scenarios where the feature data block size is small and computing resources are sufficient.

[0103] (2) Vertical splicing: When there are q feature data blocks, they can be spliced vertically along the height dimension to form a composite tensor of dimension [qH, W, C]. If the sliding window area of the composite tensor meets the multi-round scanning requirements of the thread warp in the vertical direction, and the thread warp resources can cover the total number of channels C of the composite tensor, then the composite tensor is used as the input tensor. This method is suitable for scenarios where multiple feature data blocks need to be integrated for unified processing.

[0104] (3) Dynamic splitting methods: ① Horizontal splitting: If the number of column slices involved in the calculation in any channel group in the feature data block exceeds the number of register groups that can be accommodated by a register cluster, the feature data block can be split into multiple sub-data blocks in the horizontal direction to ensure that the number of column slices in each sub-data block does not exceed the capacity of the register cluster; ② Vertical splitting: If the number of pixel groups contained in any column slice in the feature data block exceeds the data requirements that can be processed by the vertical scanning path, the data block can be split in the vertical direction to generate multiple sub-data blocks to ensure that the number of vertical pixel groups in each column slice adapts to the processing capacity of the scanning path; ③ Channel dimension splitting: If the thread warp resources cannot cover the total number of channels C of the feature data block, the feature data block can be split into multiple sub-data blocks according to the channel dimension based on the thread warp resource limitations. These sub-data blocks will be processed in batches, and in each batch, only one sub-data block will be used as the input tensor for calculation. This method is suitable for scenarios where the feature data block size is large or computing resources are limited, and the rational use of computing resources can be achieved through splitting.

[0105] It is worth noting that the method for obtaining input tensors is not limited to the use of the three methods described above alone, but can also be any combination of multiple methods, which is not limited in the embodiments of the present invention. For example, multiple feature data blocks can be first spliced into a large-sized feature data block along the vertical direction of the original space. Then, based on whether the number of pixel groups in any column slice just meets the data requirements of the vertical scanning route, the large-sized feature data block can be split vertically, ultimately obtaining multiple input tensors for batch processing.

[0106] In summary, the embodiments of the present invention can flexibly select the optimal construction method of the input tensor according to the actual hardware resource limitations and specific computing requirements, thereby maximizing computing efficiency while ensuring the rationality of resource allocation.

[0107] In an optional embodiment, the column slices involved in the calculation are column slices traversed by the sliding window operator when sliding in the horizontal direction.

[0108] It should be noted that, in the embodiment of the present invention, the dimension of the column slice is H×1×N C When selecting the column slices to be calculated, it is also necessary to follow the original sliding rules of the sliding window operator in the channel (i.e., according to the horizontal step size Sliding). Therefore, the column slices involved in the calculation are the column slices traversed when the sliding window operator slides in the horizontal direction.

[0109] In addition, the number of column slices y involved in the calculation in any channel group is greater than or equal to the width K of the sliding window operator W .

[0110] like Figure 3 In the channel group shown, the number of column slices involved in the calculation is y=K W =3, the corresponding warp executes N H =K H = After 3 rounds of scanning, each thread in the warp holds K H ×K W ×N C pixel values. Figure 5 The figure is a schematic diagram of an embodiment of the present invention in which a thread holds local sliding window view data. Figure 5 In the example, each thread holds a local sliding window view of N C Channel pixels (N C =2), C0 and C1 represent the first channel and the second channel respectively.

[0111] See also Figure 6 and Figure 7 , Figure 6 is a schematic diagram of another embodiment of the relationship between the pixel group and the register group provided by the present invention, Figure 7 Schematic diagram of another embodiment of the present invention that a thread holds local sliding window view data. Figure 6 In the channel group (N C =2) the number of column slices involved in the calculation is y=4, so each register cluster has 4 register groups, and the corresponding thread warp executes N H =K H =3 vertical scan routes, each thread in the warp holds K H ×y×N C Pixel values. In the embodiment of the present invention, when y>K W When , the coverage data of the sliding window operator sliding in the horizontal direction can be preloaded. Figure 7 In the horizontal step length = 1, each thread holds 2 local sliding window views N C Channel pixels. In short, each thread holds at least N local sliding window views. C channel pixels to support various operation modes, such as convolution and pooling operations.

[0112] In an optional embodiment, the sliding window operator is a pooling operator;

[0113] The step of calculating the loaded pixel group by each thread to obtain an output result within the corresponding local sliding window field of view includes:

[0114] In any of the thread warps, each thread performs a channel-isolated pooling operation on the loaded pixel group to obtain N pixels within the corresponding local sliding window field of view. C The pooling result of channels.

[0115] It should be noted that for each thread, the total dimension of the data composed of all pixel groups loaded is K H ×y×N C During the calculation process, the thread will perform independent pooling operations for each channel. Common pooling operations include maximum pooling, average pooling, etc. Taking maximum pooling as an example, any thread will operate on each channel separately within the local sliding window field of view (at least 1) it is responsible for, and select the maximum pixel value in the channel as the pooling result. In this way, each thread will eventually obtain N pixels within the corresponding local sliding window field of view. C The pooling result of channels.

[0116] In an optional embodiment, the sliding window operator is a single-channel convolution kernel;

[0117] The step of calculating the loaded pixel group by each thread to obtain an output result within the corresponding local sliding window field of view includes:

[0118] Broadcasting the convolution weight corresponding to each register group to all threads in the corresponding warp in scalar form;

[0119] In any of the thread bundles, each thread performs channel-isolated multiplication and addition operations on the loaded pixel groups and convolution weights to obtain the corresponding N local sliding window field of view. C The convolution result of channels.

[0120] It should be noted that if Figure 8 The figure shows an example of channel parallelism when threads perform depth-wise convolution operations provided by the prior art. Figure 8 The total number of channels of the input tensor is 12 (i.e. channels C0, C1, ..., C 11 ), so only 12 threads (T0~T11) participate in the convolution operation task, and the remaining threads (T12~T31) are idle. Each thread loads the pixel data of the local sliding window field of view in the corresponding channel, and also loads the convolution kernel corresponding to the channel. For example, thread T0 loads the 3×3 convolution kernel W0 corresponding to channel C0, and also loads the coverage pixels of the current convolution kernel W0 in channel C0; similarly, thread T10 loads the pixel data of channel C0. 10 The corresponding 3×3 convolution kernel W10 also loads the current convolution kernel W10 in channel C 10Obviously, the pixel data and convolution kernels loaded by different threads are different. Therefore, when performing a convolution operation, each thread needs to process three input vector operands: the first is the vector of input feature data, the second is the vector of convolution weights, and the third is the vector of accumulated input. However, the current GPU architecture is more suitable for processing instructions with no more than two vector operands, so the third operand usually needs to wait, resulting in a decrease in instruction execution efficiency. Based on this, the existing technology has high requirements for the read bandwidth of vector registers, which further limits the improvement of computing performance.

[0121] To address this issue, an embodiment of the present invention replaces the input convolution weights (the second vector operand) with scalar operands, reducing the number of input operands for the multiply-add instruction to two vector operands. This improvement not only better adapts to the characteristics of the GPU architecture, enabling the parallel execution of two instructions, but also significantly reduces the read bandwidth requirements of the vector registers. Furthermore, the scalar operands and the two vector operands can be executed synchronously and in parallel, reducing instruction latency and further improving computational efficiency.

[0122] Specifically, in this embodiment of the present invention, each thread within any warp holds data within the local sliding window view of its corresponding channel group. Because this sliding window data resides in the same channel group, all threads within the warp need access to the same convolution weights. Based on this characteristic, the present invention broadcasts the convolution weights to all threads within the warp in scalar form, allowing all threads within the same warp to share the same convolution weights. In other words, the convolution weights are shared within the warp in scalar form.

[0123] like Figure 4 As shown, each register group corresponds to a group ( , )’s convolution weight, i.e. ;in, is the vertical offset between the convolution weight and the convolution center, is the horizontal offset between the convolution weight and the convolution center; The offset position in the single-channel convolution kernel corresponding to channel C0 is ( , ) of the convolution weights. Specifically, in warp 0 corresponding to channel group 0, threads T0 to T31 share K H ×K W ×N C convolution weights; for example, when calculating the first local sliding window view, register group R0 in register cluster 0 of threads T0 to T31 needs to read the convolution weights , the register group R1 in the register cluster 0 of threads T0 to T31 needs to read the convolution weight , ..., the register group R2 in register cluster 2 of threads T0 to T31 needs to read the convolution weights Finally, each thread independently processes the pixels on each channel and the shared convolution weights, performs element-by-element multiplication, and accumulates the results to obtain the convolution result of the local sliding window field of view in the corresponding channel.

[0124] In an optional embodiment, the sliding window operator is a multi-channel convolution kernel;

[0125] The step of calculating the loaded pixel group by each thread to obtain an output result within the corresponding local sliding window field of view includes:

[0126] Broadcasting the convolution weight corresponding to each register group to all threads in the corresponding warp in scalar form;

[0127] In any of the thread bundles, each thread performs a channel fusion multiplication and addition operation on the loaded pixel group and the convolution weight to obtain an output feature value within the corresponding local sliding window field of view; wherein each of the channel groups is used to represent an input group in the grouped convolution operation.

[0128] It should be noted that in the grouped convolution operation, each channel group is used to represent an input group in the grouped convolution operation. For each input group, the corresponding thread warp will independently complete the calculation of one output channel in the group calculation through a shared multi-channel convolution kernel. For example, assuming that the first channel group is used as the first input group (with dimensions H×W×N C ), the number of output channels of a single group is 5, then the first thread warp needs to complete the calculation of these 5 output channels respectively and share 5 multi-channel convolution kernels (dimension is K H ×K W ×N C ), that is, when performing the calculation of an output channel, the 32 threads of the first thread bundle jointly access a multi-channel convolution kernel.

[0129] During the calculation process, each thread is responsible for processing an output feature value within the local sliding window field of view. H ×y×N C pixel values, the thread will extract a data block of the local sliding window field of view (dimension K H ×K W ×N C ), then the thread multiplies the data of the local sliding window field of view by the corresponding convolution weight element by element, and accumulates all the product values to obtain the convolution result of the local sliding window field of view, and uses it as an element in the corresponding output feature layer.

[0130] In an optional embodiment, the method further includes:

[0131] Before the thread performs the multiplication and addition operation, an out-of-bounds judgment is performed on the coordinates of the pixel values in the pixel group, and the pixel values corresponding to the out-of-bounds coordinates are masked to specified values to obtain new pixel values.

[0132] It should be noted that the operation of filling a specified value in the embodiment of the present invention (taking filling 0 as an example) includes but is not limited to the following two methods:

[0133] (1) After the thread register group obtains the pixel group, the coordinates of the center point in the local sliding window field of view and the offset of each convolution weight are used to determine whether the coordinates of the corresponding convolution weight at the current coverage position are out of bounds; if out of bounds, the pixel value corresponding to the out-of-bounds coordinate is masked to a specified value (such as a mask of 0).

[0134] (2) During the scanning operation, while loading the pixel group, it is actively determined whether the pixel coordinates are out of bounds; if so, the pixel value corresponding to the out-of-bounds coordinate is masked to a specified value, and the specified value is written into the corresponding register group; otherwise, the original pixel value is still written into the register group.

[0135] In addition to the above two methods, other suitable masking strategies may be adopted according to actual requirements and hardware architecture, which is not limited in the embodiment of the present invention.

[0136] See also Figure 9 , is a structural diagram of an embodiment of a device for implementing vertical pixel-parallel depth calculation provided by the present invention.

[0137] A second embodiment of the present invention provides a device for implementing vertical pixel-parallel depth calculation, including:

[0138] The thread warp configuration module 11 is used to divide the input tensor into m channel groups and assign them to m thread warps for independent processing; wherein each thread warp contains N T threads; each channel group contains N C Pixels of channels; N T ≥1; N C ≥1;m≥1;

[0139] The parameter acquisition module 12 is used to obtain the jump step size for vertical data acquisition in the channel group, as well as the height, width, vertical step size and N of the sliding window operator. H row base addresses; where N H is equal to said height;

[0140] The thread configuration module 13 is used to allocate N Hregister clusters; each register cluster contains y register groups; each register group is used to store N registers at the same row and column position. C channel pixels form a pixel group; y is not less than the width and is equal to the number of column slices participating in the calculation in each channel group;

[0141] The starting point determination module 14 is used to determine the N number of the current scanning round according to the row base address and the vertical step size. H The starting coordinates of each row;

[0142] The pixel loading module 15 is used to scan N pixels in each column slice participating in the calculation, starting from each row starting point coordinate and taking the jump step length as the span when executing the current scanning round. T row and column positions, and scan the N T The pixel groups at the row and column positions are sequentially loaded into the corresponding register groups of each thread in the warp;

[0143] The thread calculation module 16 is configured to calculate the loaded pixel group through each thread to obtain an output result within a corresponding local sliding window field of view.

[0144] It should be noted that the device for implementing depth operations with parallel depth calculations of vertical pixels provided in the embodiment of the second aspect of the present invention can implement all the processes of the method for implementing depth operations with parallel depth calculations of vertical pixels described in any embodiment of the first aspect above. The functions of each module and unit in the device and the technical effects achieved are respectively the same as the functions and technical effects achieved by the method for implementing depth operations with parallel depth calculations of vertical pixels described in any embodiment of the first aspect above, and will not be repeated here.

[0145] An embodiment of the third aspect of the present invention provides a computer-readable storage medium, which includes a stored computer program; wherein, when the computer program is running, it controls the device where the computer-readable storage medium is located to execute the vertical pixel-parallel depth operation implementation method described in any embodiment of the first aspect above.

[0146] An embodiment of a fourth aspect of the present invention provides a computer program product, comprising a computer program, which, when executed by a processor, implements the method for implementing vertical pixel-parallel depth operations as described in any embodiment of the first aspect.

[0147] See also Figure 10 , is a structural diagram of an embodiment of an electronic device provided by the present invention.

[0148] An embodiment of the fifth aspect of the present invention provides an electronic device, comprising a processor 21, a memory 22, and a computer program stored in the memory 22 and configured to be executed by the processor 21, wherein when executing the computer program, the processor implements the vertical pixel parallel depth operation implementation method described in any embodiment of the first aspect above.

[0149] Preferably, the computer program can be divided into one or more modules / units (e.g., computer program 1, computer program 2, ...). These one or more modules / units are stored in the memory 22 and executed by the processor 21 to implement the present invention. The one or more modules / units can be a series of computer program instruction segments capable of implementing specific functions, and these instruction segments are used to describe the execution process of the computer program in the electronic device.

[0150] The processor 21 can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor, or the processor 21 can be any conventional processor. The processor 21 is the control center of the electronic device, and uses various interfaces and lines to connect various parts of the electronic device.

[0151] The memory 22 primarily includes a program storage area and a data storage area. The program storage area can store an operating system, at least one application required for a function, and the data storage area can store related data. Furthermore, the memory 22 can be a high-speed random access memory or a non-volatile memory, such as a plug-in hard drive, a Smart Media Card (SMC), a Secure Digital (SD) card, or a flash card. Alternatively, the memory 22 can be another volatile solid-state memory device.

[0152] It should be noted that the above electronic device may include, but is not limited to, a processor and a memory. Those skilled in the art will understand that Figure 10The structural block diagram shown is only an example of the structure of the above-mentioned electronic device and does not constitute a structural limitation of the above-mentioned electronic device. The above-mentioned electronic device may include more or fewer components than shown in the figure, or combine certain components, or different components.

[0153] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the technical principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A method for implementing vertical pixel-parallel depth calculation, characterized in that: include: The input tensor is divided into m channel groups and assigned to m thread warps for independent processing; each thread warp contains N T threads; each channel group contains N C Pixels of channels; N T ≥1; N C ≥1;m≥1; Get the jump step size for vertical data acquisition in the channel group, as well as the height, width, vertical step size and N of the sliding window operator H row base addresses; where N H is equal to said height; Assign N to each of the threads H register clusters; each register cluster contains y register groups; each register group is used to store N registers at the same row and column position. C channel pixels form a pixel group; y is not less than the width and is equal to the number of column slices participating in the calculation in each channel group; Determine the N of the current scanning round according to the row base address and the vertical step size H The starting coordinates of each row; When executing the current scanning round, in each column slice participating in the calculation, starting from each row starting point coordinate, with the jump step length as the span, scan N T row and column positions, and scan the N T The pixel groups at the row and column positions are sequentially loaded into the corresponding register groups of each thread in the warp; Each thread calculates the loaded pixel group to obtain an output result within the corresponding local sliding window field of view.

2. The method for implementing vertical pixel-parallel depth calculation according to claim 1, wherein: The jump step length is the product of the height and the vertical step length.

3. The method for implementing vertical pixel-parallel depth calculation according to claim 2, wherein: The method further comprises: By N H The round scan is executed so that each of the thread warps completes the coverage of the entire sliding window area of the sliding window operator in the corresponding channel group.

4. The method for implementing vertical pixel-parallel depth calculation according to claim 2, wherein: When executing the current scanning round, in each column slice participating in the calculation, starting from each row starting point coordinate, scan N according to the jump step size. T row and column positions, and scan the N T The pixel groups at the row and column positions are sequentially loaded into the corresponding register groups of each thread in the warp, including: When performing the i-th round of vertical data scanning, in the w-th column slice participating in the calculation in the j-th channel group, the h-th row starting point coordinate is used as the starting point of the h-th vertical scanning route, and the jump step length is used as the span to scan N T row and column positions, and then N T The pixel group at the row and column position is loaded into the wth register group of the hth register cluster of each thread in the jth thread warp; wherein, 1≤i≤N H ; 1≤j≤m; 1≤w≤y; 1≤h≤N H .

5. The method for implementing vertical pixel-parallel depth calculation according to claim 1, wherein: The input tensor is composed of a feature data block, or is obtained by splicing multiple feature data blocks in the vertical direction of the original space, or is composed of any sub-data block split from a feature data block.

6. The method for implementing vertical pixel-parallel depth calculation according to claim 1, wherein: The column slices involved in the calculation are column slices traversed when the sliding window operator slides in the horizontal direction.

7. The method for implementing vertical pixel-parallel depth calculation according to claim 2, wherein: The N of the current scanning round is determined according to the row base address and the vertical step size. H The starting point coordinates of each row include: The starting point coordinates of the row are determined by the following formula: ； in, For the Scanning round The starting coordinates of each row; For the row base address; is the vertical step length.

8. The method for implementing vertical pixel-parallel depth calculation according to claim 1, wherein: The sliding window operator is a pooling operator; The step of calculating the loaded pixel group by each thread to obtain an output result within the corresponding local sliding window field of view includes: In any of the thread warps, each thread performs a channel-isolated pooling operation on the loaded pixel group to obtain N pixels within the corresponding local sliding window field of view. C The pooling result of channels.

9. The method for implementing vertical pixel-parallel depth calculation according to claim 1, wherein: The sliding window operator is a single-channel convolution kernel; The step of calculating the loaded pixel group by each thread to obtain an output result within the corresponding local sliding window field of view includes: Broadcasting the convolution weight corresponding to each register group to all threads in the corresponding warp in scalar form; In any of the thread bundles, each thread performs channel-isolated multiplication and addition operations on the loaded pixel groups and convolution weights to obtain the corresponding N local sliding window field of view. C The convolution result of channels.

10. The method for implementing vertical pixel-parallel depth calculation according to claim 1, wherein: The sliding window operator is a multi-channel convolution kernel; The step of calculating the loaded pixel group by each thread to obtain an output result within the corresponding local sliding window field of view includes: Broadcasting the convolution weight corresponding to each register group to all threads in the corresponding warp in scalar form; In any of the thread bundles, each thread performs a channel fusion multiplication and addition operation on the loaded pixel group and the convolution weight to obtain an output feature value within the corresponding local sliding window field of view; wherein each of the channel groups is used to represent an input group in the grouped convolution operation.

11. The method for implementing vertical pixel-parallel depth calculation according to claim 9 or 10, wherein: The method further comprises: Before the thread performs the multiplication and addition operation, an out-of-bounds judgment is performed on the coordinates of the pixel values in the pixel group, and the pixel values corresponding to the out-of-bounds coordinates are masked to specified values to obtain new pixel values.

12. A device for implementing vertical pixel-parallel depth calculation, characterized in that: include: The thread warp configuration module is used to divide the input tensor into m channel groups and assign them to m thread warps for independent processing; each thread warp contains N T threads; each channel group contains N C Pixels of channels; N T ≥1; N C ≥1;m≥1; The parameter acquisition module is used to obtain the jump step size for vertical data acquisition in the channel group, as well as the height, width, vertical step size and N of the sliding window operator. H row base addresses; where N H is equal to said height; Thread configuration module, used to allocate N to each thread H register clusters; each register cluster contains y register groups; each register group is used to store N registers at the same row and column position. C channel pixels form a pixel group; y is not less than the width and is equal to the number of column slices participating in the calculation in each channel group; A starting point determination module is used to determine the N number of the current scanning round according to the row base address and the vertical step size. H The starting coordinates of each row; The pixel loading module is used to scan N pixels in each column slice participating in the calculation, starting from each row starting point coordinate and taking the jump step length as the span when executing the current scanning round. T row and column positions, and scan the N T The pixel groups at the row and column positions are sequentially loaded into the corresponding register groups of each thread in the warp; The thread calculation module is used to calculate the loaded pixel group through each thread to obtain the output result within the corresponding local sliding window field of view.

13. A computer-readable storage medium, characterized in that The computer-readable storage medium includes a stored computer program; wherein, when the computer program is run, it controls the device where the computer-readable storage medium is located to execute the vertical pixel-parallel depth calculation implementation method according to any one of claims 1 to 11.

14. A computer program product, characterized in that The method comprises a computer program, which, when executed by a processor, implements the vertical pixel-parallel depth calculation implementation method according to any one of claims 1 to 11.

15. An electronic device, characterized in that: The method comprises a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein the processor implements the vertical pixel parallel depth operation implementation method according to any one of claims 1 to 11 when executing the computer program.

Images