Resource allocation method and apparatus

Abstract

The present application relates to the field of computer science. Provided are a resource allocation method and apparatus, which aim to solve the problem of low computational efficiency caused by an unbalanced computing resource load in a convolution computation process. The method is applied to a device using a single-instruction multi-data architecture, and comprises: acquiring first information of valid data in data to be processed, and a first convolution kernel used for processing the data to be processed, wherein the first information is related to position information of the valid data in the data to be processed; on the basis of the first information, acquiring a first quantity, wherein the first quantity is the total quantity of valid data corresponding to the first convolution kernel; and on the basis of the first quantity and a second quantity, acquiring a third quantity, wherein the second quantity is associated with one or more processing units comprised in a device, and the third quantity is an average processing quantity planned for the processing units.

Description

Resource allocation methods and devices Technical Field

[0001] This application relates to the field of computer science, and more particularly to a resource allocation method and apparatus. Background Technology

[0002] When processing three-dimensional (3D) point cloud data, voxelization is typically performed to analyze and process it. Voxelization can be understood as first dividing the space into a voxel grid, with each voxel representing a cubic unit. Then, the voxel containing the point cloud data is determined based on its location and filled, thus filling the voxel grid for subsequent feature extraction. However, 3D point cloud data becomes highly sparse after voxelization, with only a small number of voxels containing valid data. For example, the point cloud data in the KITTI dataset, commonly used for evaluating computer vision algorithms in autonomous driving scenarios, may have a sparsity exceeding 95% after voxelization.

[0003] The sparsity of data leads to the inefficiency of traditional convolution operations, resulting in a significant waste of computational resources. For example, when using the sliding window method to extract features, the sliding window calculation is performed across the entire voxel grid. However, due to the sparsity of data, there are a large number of blank areas (voxel grids without valid data) in the voxel grid. This causes a large amount of computational resources to be used to process these blank areas, wasting computational resources and reducing computational efficiency.

[0004] Summary of the Invention

[0005] This application provides a resource allocation method and apparatus that can balance the load during convolution calculation, improve computational efficiency, optimize resource management, and enhance system performance.

[0006] To achieve the above objectives, this application provides the following technical solution:

[0007] In a first aspect, this application provides a resource allocation method applied to a device using a single instruction multiple data architecture. The method includes: acquiring first information about valid data in data to be processed, and a first convolutional kernel for processing the data to be processed; the first information is related to the position information of the valid data in the data to be processed; acquiring a first quantity based on the first information; the first quantity is the total number of valid data corresponding to the first convolutional kernel; acquiring a third quantity based on the first quantity and a second quantity; the second quantity is associated with one or more processing units included in the device, and the third quantity is the average processing volume planned for the processing unit.

[0008] In this application, the average processing volume of valid data to be processed is planned for each processing unit based on the total amount of valid data to be computed and the number of processing units. This allows for the rational allocation of computing resources based on the average processing volume, ensuring that the actual amount of valid data processed by different processing units is not significantly different, typically fluctuating around the average processing volume. Since each processing unit processes approximately the same amount of data, computing resources can be fully utilized, effectively optimizing the resource allocation of computing cores and achieving load balancing. Furthermore, load balancing and efficient resource utilization help improve overall computing efficiency and speed, enabling computing tasks to run in a more efficient and balanced manner, optimizing resource management, and enhancing system performance.

[0009] According to the first aspect, the first information can be the location information of valid data in the data to be processed. The data to be processed can be voxel data after voxelization of image data. The data to be processed includes valid voxels and invalid voxels. Valid voxels are voxels containing valid data. The first convolution kernel can be the convolution kernel in the model. The second quantity can be the number of processing units when processing the data to be processed.

[0010] In some examples, valid data is determined based on a preset method. This preset method could be whether the value is zero; if it is not zero, it is considered valid data; otherwise, it is considered invalid data. Another preset method could be whether the value is greater than a preset threshold; if it is, it is considered valid data; otherwise, it is considered invalid data.

[0011] In some examples, the second quantity can be the total number of currently idle computing resources in the device. The second quantity can also be less than the total number of currently idle computing resources in the device.

[0012] According to the first aspect, or any implementation of the first aspect above, obtaining the first quantity based on the first information includes: obtaining the number of valid data corresponding to each convolutional kernel element in the first convolutional kernel based on the first information; and obtaining the first quantity based on the number of valid data corresponding to each convolutional kernel element in the first convolutional kernel.

[0013] In some examples, based on the first information of each valid data point and the first convolutional kernel, a first hash table and a second hash table corresponding to the first convolutional kernel are obtained by traversing each valid data point. The number of valid data points corresponding to each convolutional kernel element in the first convolutional kernel is obtained based on either the first or second hash table. A first count is calculated based on the number of valid data points corresponding to each convolutional kernel element in the first convolutional kernel.

[0014] The first hash table records information about the valid voxels used in multiplication operations with each element of the convolution kernel. The second hash table records information about the valid voxels to which the calculation result obtained after multiplying each element of the convolution kernel should be written.

[0015] In other examples, multiple second convolutional kernels are obtained based on the first convolutional kernel, and the number of kernel elements contained in the second convolutional kernel is less than the number of kernel elements contained in the first convolutional kernel.

[0016] In other examples, multiple second convolutional kernels are obtained based on the first convolutional kernel. Based on the first information of each valid data point and the multiple second convolutional kernels, a first hash table and a second hash table corresponding to each second convolutional kernel are obtained. The number of valid data points corresponding to each convolutional kernel element in each second convolutional kernel is obtained based on the first or second hash table corresponding to each second convolutional kernel. The total number of valid data points corresponding to each second convolutional kernel is calculated based on the data of the valid data points corresponding to each convolutional kernel element in each second convolutional kernel. A sixth quantity is determined based on the total number of valid data points corresponding to each second convolutional kernel. The sixth quantity is the total number of valid data points corresponding to the multiple second convolutional kernels.

[0017] In this application, the processing of effective data to be processed by each convolution kernel unit and the total amount of effective data to be processed by the convolution kernel are determined, so as to reasonably allocate processing units to the convolution kernel, balance resource allocation, and improve processing efficiency.

[0018] According to the first aspect, or any implementation of the first aspect above, the size of the memory space corresponding to each convolutional kernel element in the first convolutional kernel is the first threshold.

[0019] According to the first aspect, or any implementation of the first aspect above, the first threshold is a preset value, or the first threshold is the maximum value of the memory space used by a single convolutional kernel element in the first convolutional kernel when processing data.

[0020] According to the first aspect, or any of the above implementations of the first aspect, the first threshold is related to the processing scenario of the data to be processed, and the first threshold is different for different processing scenarios.

[0021] In this application, a memory space of size equal to a first threshold is pre-allocated for each convolution kernel element, eliminating the need for dynamic memory allocation and effectively avoiding the additional overhead caused by dynamic memory allocation. Furthermore, allocating sufficient memory space for each convolution kernel element before computation improves computational efficiency and system performance.

[0022] According to the first aspect, or any implementation of the first aspect above, the method further includes: allocating a corresponding number of processing units to each convolution kernel element based on the number of valid data corresponding to each convolution kernel element in the first convolution kernel and the third number.

[0023] According to the first aspect, or any implementation of the first aspect above, the first convolutional kernel includes a first convolutional kernel element and a second convolutional kernel element. Then, according to the number of valid data corresponding to each convolutional kernel element in the first convolutional kernel and a third number, a corresponding number of processing units are allocated to each convolutional kernel element, including: allocating a fourth number of first processing units to the first convolutional kernel element according to the number of first valid data corresponding to the first convolutional kernel element and the third number; allocating a fifth number of second processing units to the second convolutional kernel element according to the number of second valid data corresponding to the second convolutional kernel element and the third number, wherein the sum of the fourth number and the fifth number is less than or equal to the second number.

[0024] According to the first aspect, or any implementation of the first aspect above, the method further includes: processing the first valid data corresponding to the first convolution kernel element through a fourth number of first processing units; and processing the second valid data corresponding to the second convolution kernel element through a fifth number of second processing units.

[0025] In other examples, when multiple second convolutional kernels are obtained based on the first convolutional kernel, the number of processing units for processing convolutional computation is obtained based on the multiple second convolutional kernels; this number is a second number. A third number is obtained by dividing a sixth number by the second number, and the third number represents the average amount of effective data that the processing units are planned to process. Based on the amount of effective data corresponding to each convolutional kernel element in each second convolutional kernel and the third number, a corresponding number of processing units are allocated to each convolutional kernel element in each second convolutional kernel.

[0026] In this application, processing units are reasonably allocated to each convolution kernel element based on a third quantity. Each processing unit processes approximately the same amount of data, and the processing time for each unit is approximately the same, eliminating the need for waiting. This method can fully utilize computing resources, effectively optimize the resource allocation of computing kernels, and achieve load balancing.

[0027] According to the first aspect, or any implementation of the first aspect above, the processing unit is the currently idle computing resource in the device.

[0028] In a second aspect, this application provides a resource allocation apparatus comprising: a processor and a memory, the memory being coupled to the processor and used to store computer-readable instructions. When the processor reads the computer-readable instructions from the memory, the resource allocation apparatus causes the resource allocation apparatus to perform the method as described in the first aspect and any embodiment thereof.

[0029] Thirdly, this application provides a domain controller comprising: a processor and a memory, the memory being coupled to the processor, the memory being used to store computer-readable instructions, and when the processor reads the computer-readable instructions from the memory, causing the domain controller to perform the method as described in the first aspect and any embodiment of the first aspect.

[0030] Fourthly, this application provides a vehicle that includes a resource allocation device as described in the second aspect or a domain controller as described in the third aspect.

[0031] Fifthly, this application provides a chip system including at least one processor and at least one interface circuit, wherein the at least one interface circuit is used to perform transceiver functions, and the at least one processor is used to perform the method as described in the first aspect and any embodiment of the first aspect.

[0032] In a sixth aspect, this application provides a computer-readable storage medium including a computer program that, when executed by a processor, describes the method as described in the first aspect and any embodiment of the first aspect.

[0033] In a seventh aspect, this application provides a computer program product comprising a computer program that, when executed by a processor, is described as described in the first aspect and any embodiment thereof.

[0034] The technical effects corresponding to any implementation method of aspects two through seven, and all other aspects, can be found in the first aspect and the technical effects corresponding to any implementation method of the first aspect mentioned above, and will not be repeated here. Attached Figure Description

[0035] Figure 1 is a schematic diagram of a voxel grid provided in an embodiment of this application;

[0036] Figure 2 is a schematic diagram of a vehicle structure provided in an embodiment of this application;

[0037] Figure 3 is a schematic diagram of a computer structure provided in an embodiment of this application;

[0038] Figure 4 is a flowchart illustrating a resource allocation method provided in an embodiment of this application;

[0039] Figure 5 is a schematic flowchart of a resource allocation method provided in an embodiment of this application;

[0040] Figure 6 is a schematic diagram of a resource allocation device provided in an embodiment of this application;

[0041] Figure 7 is a schematic diagram of the structure of a chip system provided in an embodiment of this application. Detailed Implementation

[0042] The technical solutions of the embodiments of this application are described below with reference to the accompanying drawings. In the description of the embodiments of this application, the terminology used in the following embodiments is for the purpose of describing specific embodiments only and is not intended to be a limitation of this application. As used in the specification and appended claims of this application, the singular expressions “a,” “an,” “the,” “the,” “the,” and “this” are intended to include expressions such as “one or more,” unless the context clearly indicates otherwise. It should also be understood that in the following embodiments of this application, “at least one” and “one or more” refer to one or more (including two).

[0043] References to "one embodiment" or "some embodiments" in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized. The term "connection" includes direct connections and indirect connections, unless otherwise stated. "First" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated.

[0044] In the embodiments of this application, the words "exemplarily" or "for example" are used to indicate examples, illustrations, or explanations. Any embodiment or design described as "exemplarily" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design solutions. Specifically, the use of the words "exemplarily" or "for example" is intended to present the relevant concepts in a specific manner.

[0045] To avoid wasting computational resources, in some examples, sparse convolution algorithms are used only for valid voxels after voxelization, thus avoiding the processing of invalid voxels and saving computational resources. The criterion for determining a valid voxel in sparse convolution is that if the voxel corresponding to the center position of the convolution kernel (which can also be described as the center element of the convolution kernel) contains valid data, then the voxel is determined to be a valid voxel. Valid data refers to data with non-empty values, or data with values ​​greater than a preset threshold.

[0046] For example, taking a two-dimensional (2D) space as an example, Figure 1 shows a voxelized 2D voxel mesh. This 2D voxel mesh is 12*12 in size, with 12 voxels in both length and width, totaling 144 voxels. The first voxel in the upper left corner is used as the origin, and the coordinates of each voxel are determined sequentially according to the X and Y axes. Six voxels in this 2D voxel mesh contain valid data (shown as shaded data), and their coordinates are (1,1), (2,2), (2,7), (3,8), (6,2), and (7,3). The remaining voxels are invalid voxels without data. The convolution kernel size for the convolution operation on this 2D voxel mesh is 3*3, and the kernel consists of nine elements: the 1st, 2nd, 3rd, 4th, 5th, 6th, 7th, 8th, and 9th elements. The 5th element is the center element of this convolution kernel. Each convolution kernel element corresponds to a specific weight.

[0047] It is understood that the kernel elements in the convolution kernel are ordered according to a preset method based on the kernel size. This preset method can be to determine the position or label of each kernel element sequentially from top to bottom and from left to right. This application does not limit the specific implementation method of the sorting order of the kernel elements in the convolution kernel.

[0048] For example, taking a 3x3 convolution kernel size as an example, first, obtain the top row and sort the three convolution kernel elements from left to right to get the 1st, 2nd, and 3rd elements. Then, obtain the middle row and, based on the sorting result of the first row, sort the three convolution kernel elements from left to right to get the 4th, 5th, and 6th elements. Finally, obtain the last row and, based on the sorting result of the second row, sort the three convolution kernel elements from left to right to get the 7th, 8th, and 9th elements.

[0049] For example, if the sliding window method is used to extract voxel features, then the convolution kernel is used to perform convolution calculation on each voxel (effective voxel and invalid voxel) in the 2D voxel grid.

[0050] Understandably, in convolution computation, the number of kernel elements in each kernel directly determines the number of multiplication operations, addition operations, and read operations required. For each voxel, the convolution computation first reads data, and the number of data read operations is equal to the number of kernel element data. Then, convolution computation is performed based on the read data (i.e., multiplication and addition operations to obtain a convolution result). The number of multiplication and addition operations in the convolution computation should be the sum of the number of multiplication operations and the number of addition operations. After the convolution computation, the result is written to the voxel.

[0051] For example, as shown in Figure 1, the number of voxels in the X-axis direction is 12, the number of voxels in the Y-axis direction is 12, and the number of elements in the convolution kernel is 9. The number of data read operations performed in the convolution kernel during convolution computation should be 12 * 12 * 9 = 1296, the number of multiply-accumulate operations should be 12 * 12 * 9 * 2 = 2592, and the number of data write operations should be 12 * 12 = 144.

[0052] For example, if sparse convolution is used to extract voxel features, then the convolution kernel is used to perform convolution calculations on the effective voxels in the 2D voxel grid.

[0053] Specifically, the effective voxels are first determined based on the convolution kernel. For example, for a voxel with coordinates (1, 1), if the voxel corresponding to the center element of the convolution kernel (i.e., the 5th element in the convolution kernel) contains effective data, then this voxel is determined to be an effective voxel. In this way, the 2D voxel grid is determined to contain 6 effective voxels, and the coordinates of the 6 effective voxels are (1, 1), (2, 2), (2, 7), (3, 8), (6, 2), and (7, 3).

[0054] Then, convolution calculations are performed sequentially on the effective voxels. When performing convolution calculations on each effective voxel, only the effective voxels in that convolution kernel are calculated.

[0055] Specifically, depending on the current position of the convolution kernel, if there are other valid voxels in the region surrounding the valid voxel (for ease of distinction, the valid voxel corresponding to the center element of the convolution kernel is denoted as the first voxel, and the other valid voxels in the region surrounding the first voxel are denoted as the second voxels; these other valid voxels can also be understood as valid voxels corresponding to elements other than the center element of the convolution kernel), then convolution is performed based on the first and second voxels. The number of first voxels is one, and the number of second voxels is one or more.

[0056] It is understandable that when using a convolution kernel to process the first voxel, a local region matching the size of the convolution kernel is extracted from the voxel grid. The first voxel is located at the center of this local region, and it also corresponds to the center element of the convolution kernel. This local region can also be understood as the current position of the convolution kernel when processing the first voxel.

[0057] It should be understood that the positions of each kernel element within the convolution kernel do not change. However, the coordinates of each first voxel in the voxel grid are different, and the current position (i.e., local region) of the convolution kernel corresponding to the first voxel is also different. The effective voxels (such as the first voxel and the second voxel) contained in the current position of the convolution kernel corresponding to the first voxel are also different.

[0058] For example, based on Figure 1 above, the coordinates of the first voxel are (1,1). The size of the current position (i.e., local region) of the convolution kernel corresponding to the first voxel is 3*3. The current position of the convolution kernel is composed of a grid of 9 voxels with coordinates (0,0), (0,1), (0,2), (1,0), (1,1), (1,2), (2,2), (2,1), and (2,2). These 9 voxel grids correspond to the 1st, 2nd, 3rd, 4th, 5th, 6th, 7th, 8th, and 9th elements of the convolution kernel, respectively. If the coordinates of the first voxel are (3,8), then the current position of the convolution kernel is composed of a grid of 9 voxels with coordinates (2,7), (2,8), (2,9), (3,7), (3,8), (3,9), (4,7), (4,8), and (4,9). The nine voxel grids correspond to the first, second, third, fourth, fifth, sixth, seventh, eighth, and ninth elements of the convolution kernel, respectively.

[0059] For example, based on Figure 1 above, the coordinates of the first voxel are (1, 1). The center element of the convolution kernel (i.e., the 5th element) corresponds to the first voxel. The voxels in the region surrounding the first voxel are the voxels corresponding to the other convolution kernel elements (i.e., the 1st, 2nd, 3rd, 4th, 6th, 7th, 8th, and 9th elements) besides the center element of the convolution kernel (i.e., the 5th element). Only one valid voxel is found in the region surrounding the first voxel, and the coordinates of this valid voxel are (2, 2). This valid voxel is determined as the second voxel. Thus, the first voxel with coordinates (1, 1) and the second voxel with coordinates (2, 2) are convolved.

[0060] For example, based on Figure 1 above, taking the coordinates of the first voxel as (3, 8), the center element of the convolution kernel (i.e., the 5th element) corresponds to the first voxel. There is only one valid voxel in the region surrounding the first voxel, and the coordinates of this valid voxel are (2, 7). This valid voxel is determined as the second voxel. Thus, the first voxel with coordinates (3, 8) and the second voxel with coordinates (2, 7) are convolved.

[0061] The specific steps of convolution calculation are as follows: Based on the current position of the convolution kernel, the first voxel and the second voxel are multiplied by their corresponding convolution kernel elements to obtain multiple multiplication results. These multiple multiplication results are then summed to obtain the final calculation result, which is written to the first voxel. For example, the first voxel is multiplied by its corresponding convolution kernel element, and the second voxel is multiplied by its corresponding convolution kernel element, resulting in multiple multiplication results. These multiple multiplication results are then summed to obtain the convolution calculation result for the first voxel, which is then written to the first voxel.

[0062] For example, based on Figure 1 above, with the first voxel having coordinates (1, 1) and the second voxel having coordinates (2, 2), the convolution kernel element corresponding to the first voxel is the center element of the convolution kernel (i.e., the 5th element), and the convolution kernel element corresponding to the second voxel is the 9th voxel. Convolution is performed on the first voxel at coordinates (1, 1). Multiplying the first voxel at coordinates (1, 1) with the 5th element of the convolution kernel yields multiplication result 1. Multiplying the second voxel at coordinates (2, 2) with the 9th element of the convolution kernel yields multiplication result 2. Multiplication result 1 and multiplication result 2 are added together to obtain the convolution result for the first voxel at coordinates (1, 1).

[0063] For example, based on Figure 1 above, with the first voxel having coordinates (3, 8) and the second voxel having coordinates (2, 7), the convolution kernel element corresponding to the first voxel is the center element of the convolution kernel (i.e., the 5th element), and the convolution kernel element corresponding to the second voxel is the 1st voxel. Convolution is performed on the first voxel with coordinates (3, 8). Multiplying the first voxel with coordinates (3, 8) by the 5th element of the convolution kernel yields multiplication result 3. Multiplying the second voxel with coordinates (2, 7) by the 1st element of the convolution kernel yields multiplication result 4. Multiplication result 3 and multiplication result 4 are added together to obtain the convolution result for the first voxel with coordinates (3, 8).

[0064] To facilitate the description of different voxel coordinates and convolution kernel elements, kernel_x is used below to represent the convolution kernel element, where x is a positive integer. For example, kernel_1 represents the first element in the convolution kernel, kernel_2 represents the second element, ..., kernel_9 represents the ninth element. map[a][b] represents the coordinates of a voxel in the 2D voxel grid, where a and b are both natural numbers. Any voxel in the 2D voxel grid can be represented by map[a][b]. For example, the voxel with coordinates (1, 1) can be represented as map[1][1], and the voxel with coordinates (1, 2) can be represented as map[1][2].

[0065] Depending on the current position of the convolution kernel, if there are no other valid voxels (i.e., second voxels) in the surrounding area of ​​the valid voxel (i.e., the first voxel), then convolution is performed only on the first voxel. The first voxel contains only one data point. The specific steps of the convolution calculation are as follows: based on the current position of the convolution kernel, the first voxel and its corresponding convolution kernel center element are multiplied, and the result of the multiplication is written into the first voxel as the calculation result.

[0066] Understandably, when performing convolution calculations on each first voxel, the current position of the convolution kernel must be determined based on the specific coordinates of the first voxel. This then determines whether the current position of the convolution kernel contains a second voxel. If it does, convolution is performed based on the first voxel and its corresponding second voxel. If it does not contain a second voxel, convolution is performed based on the first voxel. If a first voxel has a corresponding second voxel, the number and position of the second voxels are determined based on the current position of the convolution kernel. Different first voxels may correspond to different second voxels.

[0067] Thus, the calculation result corresponding to each effective voxel (i.e., the first voxel) is determined in the above manner.

[0068] As shown in Figure 1, when the center element of the convolution kernel is located at (1, 1), the voxel corresponding to the 9th element in the convolution kernel (with coordinates (2, 2)) contains valid data, and this voxel is also a valid voxel. That is, the first voxel is the voxel with coordinates (1, 1), and the second voxel is the voxel with coordinates (2, 2). The formula for convolution calculation of the valid voxel with coordinates (1, 1) is kernel_5*map[1][1]+kernel_9*map[2][2]. Wherein, kernel_5 is the 5th element in the convolution kernel, map[1][1] is the valid voxel with coordinates (1, 1) in the 2D voxel grid, kernel_9 is the 9th element in the convolution kernel, and map[2][2] is the valid voxel with coordinates (2, 2) in the 2D voxel grid.

[0069] It is understandable that the above calculation formula is only to indicate which convolution kernel elements and effective voxels are used to calculate the result of each effective voxel. For example, for an effective voxel with coordinates (1, 1), its convolution calculation result is calculated by each data in kernel_5*map[1][1]+kernel_9*map[2][2], and the convolution calculation result should be written into the effective voxel with coordinates (1, 1).

[0070] Accordingly, the calculation formula for each effective voxel is determined according to the above method, so as to determine the convolution calculation result according to the calculation formula and write the convolution calculation result into the effective voxel.

[0071] When the center element of the convolution kernel is located at (2, 2), the first voxel is the voxel with coordinates (2, 2), and the second voxel is the voxel with coordinates (1, 1). The formula for convolution calculation of the effective voxel with coordinates (2, 2) is kernel_5*map[2][2]+kernel_1*map[1][1]. Wherein, kernel_5 is the 5th element in the convolution kernel, map[2][2] is the effective voxel with coordinates (2, 2) in the 2D voxel grid, kernel_1 is the 1st element in the convolution kernel, and map[1][1] is the effective voxel with coordinates (1, 1) in the 2D voxel grid.

[0072] When the center element of the convolution kernel is located at (2, 7), the first voxel is the voxel with coordinates (2, 7), and the second voxel is the voxel with coordinates (3, 8). The formula for convolution calculation of the effective voxel with coordinates (2, 7) is kernel_5*map[2][7]+kernel_9*map[3][8]. Wherein, kernel_5 is the 5th element in the convolution kernel, map[2][7] is the effective voxel with coordinates (2, 7) in the 2D voxel grid, kernel_9 is the 9th element in the convolution kernel, and map[3][8] is the effective voxel with coordinates (3, 8) in the 2D voxel grid.

[0073] When the center element of the convolution kernel is located at (3, 8), the first voxel is the voxel with coordinates (3, 8), and the second voxel is the voxel with coordinates (2, 7). The formula for convolution calculation of the effective voxel with coordinates (3, 8) is kernel_5*map[3][8]+kernel_1*map[2][7]. Wherein, kernel_5 is the 5th element in the convolution kernel, map[3][8] is the effective voxel with coordinates (3, 8) in the 2D voxel grid, kernel_1 is the 1st element in the convolution kernel, and map[2][7] is the effective voxel with coordinates (2, 7) in the 2D voxel grid.

[0074] When the center element of the convolution kernel is located at (6, 2), the first voxel is the voxel with coordinates (6, 2), and the second voxel is the voxel with coordinates (7, 3). The formula for convolution calculation of the effective voxel with coordinates (6, 2) is kernel_5*map[6][2]+kernel_9*map[7][3]. Wherein, kernel_5 is the 5th element in the convolution kernel, map[6][2] is the effective voxel with coordinates (6, 2) in the 2D voxel grid, kernel_9 is the 9th element in the convolution kernel, and map[7][3] is the effective voxel with coordinates (7, 3) in the 2D voxel grid.

[0075] When the center element of the convolution kernel is located at (7, 3), the first voxel is the voxel with coordinates (7, 3), and the second voxel is the voxel with coordinates (6, 2). The formula for convolution calculation of the effective voxel with coordinates (7, 3) is kernel_5*map[7][3]+kernel_1*map[6][2]. Wherein, kernel_5 is the 5th element in the convolution kernel, map[7][3] is the effective voxel with coordinates (7, 3) in the 2D voxel grid, kernel_1 is the 1st element in the convolution kernel, and map[6][2] is the effective voxel with coordinates (6, 2) in the 2D voxel grid.

[0076] Based on the above, it can be determined that sparse convolution requires 12 multiplication operations and 6 addition operations, totaling 18 multiplication-addition operations. It also requires 12 data read operations and 6 data write operations.

[0077] It should be understood that the number of multiply-accumulate operations and data read / write operations in sparse convolution is far less than that in sliding window convolution, and sparse convolution can effectively reduce the waste of computing resources.

[0078] After determining the calculation formula for the convolution operation of each valid voxel, traverse each element in the convolution kernel and establish two mapping tables, such as the first mapping table and the second mapping table. The first mapping table records the mapping relationship between each convolution kernel element and the valid voxel in the multiplication operation. The second mapping table records the mapping relationship between the multiplication result of each convolution kernel element and the valid voxel to be written. For example, the mapping tables can be hash tables, such as the first hash table and the second hash table. The first hash table records the information of the valid voxels in the multiplication operation with each convolution kernel element. The second hash table records the information of the valid voxels to be written to after the calculation result of the multiplication operation of each convolution kernel element.

[0079] It is understood that the embodiments of this application do not limit the specific implementation of the mapping relationship table.

[0080] For example, taking the mapping relationship table as a hash table, as shown in Table 1, it contains information about the effective voxels that are multiplied with each element in the convolution kernel, determined according to the convolution formula of each effective voxel in the above 2D voxel grid.

[0081] Table 1

[0082] As shown in Table 2, this table contains the information of the effective voxels to be written after performing a multiplication operation on each convolution kernel determined according to the convolution formula of each effective voxel in the above 2D voxel grid.

[0083] Table 2

[0084] The relationship between Table 1 and Table 2 is as follows: The convolution kernel element in Table 1 is multiplied by its nth effective voxel to obtain the multiplication result. This multiplication result is then written into the nth effective voxel of the convolution kernel element in Table 2.

[0085] For example, for kernel_1 in Table 1 and Table 2, kernel_1 is multiplied by the first effective voxel map[1][1] of kernel_1 in Table 1, and the result of the multiplication is written into the first effective voxel map[2][2] of kernel_1 in Table 2 (effective voxel with coordinates (2,2); kernel_1 is multiplied by the second effective voxel map[2][7] of kernel_1 in Table 1, and the result of the multiplication is written into the second effective voxel map[3][8] of kernel_1 in Table 2 (effective voxel with coordinates (3,8); kernel_1 is multiplied by map[6][2] and the third effective voxel of kernel_1 in Table 1, and the result of the multiplication is written into the third effective voxel map[7][3] of kernel_1 in Table 2 (effective voxel with coordinates (7,3)).

[0086] For example, for kernel_5 in Table 1 and Table 2, kernel_5 is multiplied by the first effective voxel map[1][1] of kernel_5 in Table 1, and the result of the multiplication is written into the first effective voxel map[1][1] of kernel_5 in Table 2 (the effective voxel with coordinates (1,1); kernel_5 is multiplied by the second effective voxel map[2][2] of kernel_5 in Table 1, and the result of the multiplication is written into the second effective voxel map[2][2] of kernel_5 in Table 2 (the effective voxel with coordinates (2,2)).

[0087] According to Table 2, map[2][2] (the effective voxel with coordinates (2, 2)) is written with data twice (the result of multiplying kernel_1 and map[1][1] and the result of multiplying kernel_5 and map[2][2], respectively). The sum of the two data is the convolution calculation result of map[2][2]. That is, based on the information in Tables 1 and 2, the calculation formula for the effective voxel with coordinates (2, 2) can be derived as kernel_5*map[2][2]+kernel_1*map[1][1]. Correspondingly, the calculation formulas for other effective voxels can also be determined based on the information in Tables 1 and 2.

[0088] Understandably, when handling complex calculations, a hash table is typically used to establish the computational relationship between voxel data and convolution kernel data. Data is read from Table 1, multiplication is performed to obtain the multiplication result, and the result is written from Table 2, with an accumulation operation performed during the writing process. Based on the information in Tables 1 and 2, the device accurately and quickly reads the data related to the convolution calculation of each effective voxel, as well as the writing position of the calculation result for each effective voxel. The device can accurately and quickly read, calculate, and output (i.e., write) based on the information in the hash table, thereby optimizing computation, utilizing computing resources more efficiently, reducing unnecessary redundant calculations and data access, and improving overall computing performance and response speed.

[0089] After establishing the first hash table and the second hash table, the values ​​of the valid data and the convolution kernel elements in each valid voxel in the 2D voxel grid are read according to the information in the first hash table and the second hash table. Multiplication is performed based on the correspondence between the convolution kernel elements and valid voxels in the first hash table to obtain the multiplication result. The multiplication result is then written into the corresponding valid voxel based on the information in the second hash table to complete the convolution calculation.

[0090] It should be understood that building a hash table is an initial operation for performing sparse convolution, so that subsequent sparse convolution operations can clearly identify which valid voxels need to be multiplied and which multiplication results need to be added. However, sparse convolution is typically used on computing platforms that use a single instruction multiple threads (SIMT) architecture.

[0091] It's understandable that the sparse convolution processing described above refers to the specific processing procedure when processing a single frame of an image. However, devices typically acquire multiple frames when processing images, processing each frame according to the sparse convolution method described above. When a device can acquire multiple frames, the data contained in each frame may differ. Moreover, the point cloud data in each frame is highly sparse after voxelization, and the amount and location of effective data contained in different point cloud data after voxelization may vary. Therefore, when processing different data through sparse convolution, the amount of data processed by each element of the convolution kernel also differs. That is, when the convolution kernel processes multiple frames of images through sparse convolution, the amount of data processed during the sparse convolution process is irregular and dynamically changing. Therefore, sparse convolution requires flexible adjustment of computation strategies and flexible allocation of resources to cope with the dynamically changing amount of data during the sparse convolution computation process.

[0092] The SIMT architecture has a large number of computational cores (also described as computational resources, cores, or simply cores), but each core has relatively low computational power (e.g., each core processes one data point at a time). Therefore, the SIMT architecture is suitable for handling a large number of small, fragmented, and flexible computational tasks. Furthermore, the SIMT architecture supports dynamic shape inference, enabling it to flexibly handle input data of varying sizes. When performing sparse convolution, the SIMT architecture can flexibly and dynamically allocate computational resources based on the computational load, avoiding ineffective computation and resource waste. Therefore, the aforementioned sparse convolution processing methods are typically applied to the SIMT architecture.

[0093] For example, Compute Unified Device Architecture (CUDA) is a general-purpose parallel computing platform and programming model that uses the SIMT architecture. Chips that implement high-performance parallel computing based on the CUDA architecture can use CUDA to achieve high-performance parallel computing, etc. The number of computing cores in a chip implementing high-performance parallel computing based on the CUDA architecture can be 4096. Specifically, chips implementing high-performance parallel computing based on the CUDA architecture are system-on-chips (SoCs) designed for autonomous driving and edge computing. Therefore, when this chip processes sparse convolution based on the SIMT architecture, it allocates a reasonable number of computing cores (which can also be understood as the number of threads) to each convolutional kernel element according to the amount of data to be computed by each kernel element. Based on the information in Table 1 or Table 2 above, the effective data volume corresponding to kernel_1 is determined to be 3, the effective data volume corresponding to kernel_5 is 6, and the effective data volume corresponding to kernel_9 is 3; other convolutional kernel elements are not computed. Therefore, 3 computing cores are allocated to kernel_1, 6 computing cores to kernel_5, and 3 computing cores to kernel_9, with each computing core processing only one data point.

[0094] Understandably, the SIMT architecture allocates a corresponding number of computing cores based on the actual amount of data that each convolutional kernel element needs to process, thus balancing the data processing across a large number of cores. However, if the amount of data exceeds the number of currently available computing cores, each core, due to its limited computing power, can only process one data point at a time. Therefore, it first processes the corresponding number of data points based on the available computing cores, and then processes the remaining data. Completing the computation in multiple batches increases the computation time. Moreover, each time a core performs a computation or accesses data, it incurs some overhead, thus affecting the overall efficiency of the platform.

[0095] SIMD architecture has a small number of computational cores, but each core has high computational power (e.g., each core processes multiple data points at once). Therefore, SIMD architecture is suitable for processing dense data. However, the data in sparse convolution is sparsely distributed and discontinuous. Furthermore, the amount of data to be computed by each core element can only be determined after traversing all core elements. Due to the irregularity and uncertainty of data volume in sparse convolution, SIMD architecture struggles to effectively partition and parallelize the data, leading to an imbalance in the amount of data processed by each core and wasted computational resources. This uneven load on cores can cause waiting during inter-core synchronization. That is, cores performing only a small amount of processing or no processing must wait for the core with the largest amount of data to finish processing before executing other computations, impacting computational efficiency and platform performance.

[0096] For example, based on the 2D voxel mesh example above, if an intelligent driving computing platform implemented based on a SIMD architecture has 9 computing cores (e.g., computing core 1, computing core 2... computing core 9), each computing core is empirically assigned to each convolutional kernel element. Computing core 1 handles the computation in kernel_1, computing core 2 handles the computation in kernel_2, ..., and computing core 9 handles the computation in kernel_9. In actual computation, only the convolutional kernels handling kernel_1, kernel_5, and kernel_9 perform computations; the remaining computing cores do not process data. Furthermore, the load on the computing cores is unbalanced: computing core 1 processes 3 data points, computing core 5 processes 6 data points, and computing core 9 processes 3 data points. Computing cores 2, 3, 4, 6, 7, and 8 wait for computing cores 1 and 9 to complete their computations, and then collectively wait for computing core 5 to complete its computation before performing other computations, severely impacting computational efficiency. For example, an intelligent driving computing platform implemented based on SIMD architecture can be a mobile data center (MDC), etc. This application does not limit the devices or intelligent driving computing platforms that apply SIMD architecture.

[0097] To address the issues of unbalanced load and low computational efficiency during convolution calculations on SIMD-based devices, this application provides a resource allocation method. This method rationally allocates computing resources based on the amount of data to be computed and the number of available computing cores, fully utilizing computing resources and effectively optimizing the resource allocation of computing cores to achieve load balancing. Furthermore, load balancing and efficient resource utilization contribute to improving overall computational efficiency and speed, enabling computing tasks to run in a more efficient and balanced manner, optimizing resource management, and enhancing system performance.

[0098] The technical solutions of the embodiments of this application will be described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments.

[0099] The resource allocation method in this application can be applied to any convolutional computation scenario. For example, features can be extracted from image data through convolutional computation. In autonomous driving scenarios, target recognition, target segmentation, and target localization can be performed based on image data. Target recognition can include identifying obstacles, roads, pedestrians, etc. Target segmentation can include dividing different areas such as roads and sidewalks. Target localization can include detecting and locating pedestrians, signs, etc. In industrial applications, equipment and workpiece detection and analysis can be performed based on image data. In medical imaging scenarios, lesion detection, organ segmentation, and surgical simulation can be performed based on image data. 3D environments can be reconstructed based on image data. For example, 3D images of cultural relics can be reconstructed to assist in their restoration; geological structures can be reconstructed for geological exploration and environmental monitoring; or the environment surrounding buildings or vehicles can be reconstructed for planning purposes. This application does not limit the specific application scenarios of convolutional computation.

[0100] The technical solutions provided in this application can be applied to devices using a SIMD architecture or chips of devices using a SIMD architecture, wherein the device possesses artificial intelligence (AI) computing capabilities. For example, the device can perform convolution calculations. Exemplarily, the device can be a mobile intelligent device. Exemplarily, the mobile intelligent device can include, but is not limited to, vehicles, artificial intelligence (AI) devices (such as robots), etc. Alternatively, the device can be an electronic device. Exemplarily, the electronic device can include, but is not limited to, computers, ultra-mobile personal computers (UMPCs), handheld computers, netbooks, personal digital assistants (PDAs), in-vehicle devices, virtual reality devices, etc. Alternatively, the device can be an Internet of Things (IoT) device, a high-performance computing device, and an embedded device, etc. This application does not impose any limitations on the specific type of device.

[0101] Taking a vehicle as an example and autonomous driving as an application scenario, Figure 2 is a structural schematic diagram of a vehicle 200 provided in an embodiment of this application.

[0102] Figure 2 is a schematic diagram of the structure of a vehicle 200 provided in an embodiment of this application. Referring to Figure 2, the vehicle 200 may include various subsystems, such as a sensor system 210, a control system 220, one or more peripheral devices 230, and a computer system 240.

[0103] Optionally, vehicle 200 may include more or fewer subsystems, and each subsystem may include multiple components. Furthermore, each subsystem and component of vehicle 200 may be interconnected via wired or wireless means.

[0104] Sensor system 210 may include several sensors for sensing information about the environment surrounding vehicle 200. For example, sensor system 210 may include lidar 211 and camera 212.

[0105] The lidar 211 senses image data of the environment surrounding the vehicle 200 by sending laser pulses and measuring their reflection times. In some embodiments, the lidar 211 emits laser beams in multiple directions to scan the environment surrounding the vehicle 200 and generates point cloud data based on the reflection time of each laser beam. In this embodiment, the lidar 211 senses the environment surrounding the vehicle 200 and generates point cloud data.

[0106] The control system 220 controls the operation of the vehicle 200 and its components. The control system 220 may include various elements, including a steering system 221, an accelerator 222, a braking unit 223, a computer vision system 224, a driving trajectory control system 225, and an obstacle avoidance system 226.

[0107] The computer vision system 224 can process and analyze image data detected by the sensor system 210 to identify information about the surrounding environment of the vehicle 200 and the physical and facial features of the driver in the vehicle's cockpit. The surrounding environment information may include obstacles, traffic signals, road conditions, etc.; the driver's physical and facial features may include the driver's behavior, gaze, and facial expressions. For example, the computer vision system 224 analyzes and processes point cloud data generated by the lidar 211. Alternatively, it analyzes and processes images captured by the camera 212. In some embodiments, the computer vision system 224 can also be used for environmental mapping, object detection, predicting object trajectories, estimating object speeds, determining driver behavior, and authentication. In this embodiment, the computer vision system 224 is used to process image data (such as point cloud data) for target detection, obstacle recognition, map building, trajectory planning, navigation, and other functions.

[0108] Vehicle 200 interacts with external sensors, other vehicles, other computer systems, or users via peripheral devices 230. Peripheral devices 230 may include a wireless communication system 231, an onboard computer 232, a microphone 233, and / or a speaker 234.

[0109] In this embodiment, the vehicle acquires image data from other devices via peripheral device 230, and the peripheral device 230 sends the image data to computer system 240. Alternatively, the vehicle sends the processed result from computer system 240 to the user via peripheral device 230 so that the user can perform corresponding processing.

[0110] Some or all of the functions of vehicle 200 are controlled by computer system 240. Computer system 240 may include at least one processor 241, which executes instructions 2421 stored in a non-transitory computer-readable medium such as memory 242. Computer system 240 may also be multiple computing devices that control individual components or subsystems of vehicle 200 in a distributed manner.

[0111] In some implementations, computer system 240 includes at least one processor integrated as a system-on-chip (SOC), which is commonly referred to as an SOC by those skilled in the art. The SOC may include at least one processor, and when the SOC includes multiple processors, the types of processors may differ, such as a central processing unit (CPU) and a microcontroller unit (MCU).

[0112] Processor 241 can be any conventional processor, such as a commercially available central processing unit (CPU). Alternatively, processor 241 can also be a dedicated device such as an application-specific integrated circuit (ASIC) or other hardware-based processor. In this embodiment, the processor 241 is designed with a SIMD architecture.

[0113] In some embodiments, memory 242 may contain instructions 2421 (e.g., program logic) that can be executed by processor 241 to perform various functions of vehicle 200. Memory 242 may also contain additional instructions, including instructions for sending data to, receiving data from, interacting with, and / or controlling one or more of the mobility system 210, sensor system 210, control system 220, and peripheral devices 230.

[0114] In addition to instructions 2421, memory 242 may also store data such as road maps, trajectory information, vehicle position, direction, speed, and other vehicle data, as well as other information. This information can be used by vehicle 200 and computer system 240 during operation of vehicle 200 in autonomous, semi-autonomous, and / or manual modes.

[0115] In this embodiment, the processor 241 interacts with the LiDAR 211 to acquire image data (such as point cloud data). Alternatively, the processor 241 interacts with other devices via the peripheral device 230 to acquire image data. The processor 241 preprocesses the image data (e.g., noise removal, calibration, etc.). The computer vision system 224 performs processing tasks on the image data through the processor 241. These processing tasks may include object detection, obstacle recognition, map building, trajectory planning, and navigation. The processor 241 is responsible for coordinating and executing the processing tasks to ensure that the image data acquired from the LiDAR 211 is effectively utilized and processed to support the vehicle's intelligent driving functions. When performing convolution calculations in the processing tasks, the processor 241 executes the resource allocation method of this application to balance resources and improve computational efficiency.

[0116] For example, processor 241 is also configured to transmit the processing result to the user via peripheral device 230, so that the user can control the vehicle according to the processing result. Alternatively, processor 241 is also configured to send the processing result to control system 220, so that control system can control the vehicle according to the processing result. For example, if the processing result detects an obstacle in front of the vehicle, processor 241 can send the processing result to peripheral device 230, and the on-board computer 232 in peripheral device 230 displays the obstacle information so that the user can brake the vehicle according to the obstacle information. The processor can also send the processing result to control system 220, etc., and the obstacle avoidance system 226 in control system 220 actively controls the vehicle to avoid obstacles according to the obstacle information.

[0117] In this embodiment of the application, the architecture of the processor 241 may be a SIMD architecture.

[0118] In some embodiments, the computer system 240 may also be described as a vehicle controller, which may also be described as a powertrain controller, intelligent driving computing platform, or domain controller, and is the core control component of the entire vehicle. It collects input information from various systems and components, makes corresponding judgments based on the input information, and controls the actions of various components in the vehicle 200 to drive the vehicle 200.

[0119] For example, a vehicle controller can be a domain controller (DC) within the vehicle. A domain controller divides the vehicle's system into different domains according to function, such as powertrain domain, chassis domain, body domain, cockpit domain, and autonomous driving domain. Each domain is centrally controlled by a high-performance computing platform. A domain controller mainly includes, but is not limited to, one or more processors, memory, communication interfaces, a power management module, and sensor interfaces. The processor is responsible for running various software algorithms within the domain, processing sensor data, making decisions, and controlling actuators. Memory is used to store the operating system, applications, and data, or for temporarily storing running programs and data. The communication interface is used to communicate with other electronic devices in the vehicle, enabling data transmission and interaction. The power management module is responsible for managing and distributing power to the domain controller, ensuring a stable power supply to the system under different operating conditions. The sensor interfaces connect to various types of sensors, such as cameras, LiDAR, and accelerometers, depending on the domain, allowing the processor to receive, process, and analyze sensor data from the corresponding domain.

[0120] In this embodiment, the domain controller acquires image data collected by radar, laser, etc. through the sensor interface. When the processor performs convolution calculation on the image data, it uses the resource allocation method of this application to allocate resources, so that the computing task can run in a more efficient and balanced way, and optimize resource allocation.

[0121] For example, the computing device can also be the intelligent driving domain controller in the vehicle, such as a mobile data center (MDC). The intelligent driving domain controller can be positioned as the computing platform for intelligent driving and is a key hardware architecture for realizing software-defined vehicles. The hardware architecture of the intelligent driving domain controller includes, but is not limited to, a main control chip, sensor interfaces, communication modules, a power management module, and a cooling system. The main control chip provides powerful artificial intelligence computing power to handle a large number of perception and computing tasks in intelligent driving, as well as system control functions. The sensor interfaces can have rich interfaces to connect to various types of sensors. These interfaces ensure that the intelligent driving domain controller can quickly and accurately receive information from different sensors, providing data support for intelligent driving decisions. The communication module has high-speed communication capabilities, supporting communication methods such as in-vehicle Ethernet, ensuring efficient communication with other electronic control units (ECUs) in the vehicle and the cloud, enabling real-time data transmission and interaction. The power management module is responsible for managing and distributing the power of the entire system, ensuring that each component receives a stable and reliable power supply under different operating conditions, while also considering energy consumption optimization to reduce the overall power consumption of the system. A cooling system can ensure that the system temperature remains within a reasonable range during long-term high-load operation. An efficient cooling system is the key to ensuring the stable operation of the intelligent driving computing platform.

[0122] Alternatively, one or more of these components may be installed separately from or associated with vehicle 200. For example, memory 242 may exist partially or completely separately from vehicle 200. The components may be communicatively coupled together in a wired and / or wireless manner.

[0123] Optionally, the above components are just an example. In actual applications, the components in the above modules may be added or deleted according to actual needs. Figure 2 should not be construed as a limitation on the embodiments of this application.

[0124] The aforementioned vehicle 200 can be a new energy vehicle, electric vehicle, sedan, truck, motorcycle, bus, ship, airplane, helicopter, lawnmower, recreational vehicle, amusement park vehicle, construction vehicle, tram, and train, etc., and this application embodiment does not impose any particular limitation. The power of the aforementioned vehicle can be provided by gasoline, diesel, electricity, solar energy, hydrogen energy, etc.

[0125] In other embodiments of this application, the vehicle 200 may further include hardware structures and / or software modules to implement the above functions in the form of hardware structures, software modules, or a combination of hardware structures and software modules. Whether a particular function is implemented in the form of hardware structures, software modules, or a combination of hardware structures and software modules depends on the specific application and design constraints of the technical solution.

[0126] Taking a computer as the device and a medical imaging scenario as an example, Figure 3 is a schematic diagram of the structure of a computer provided in an embodiment of this application. The computer includes at least one processor 301, a communication line 302, a memory 303, at least one communication interface 304, and a sensing module 305.

[0127] The processor 301 can be a general-purpose central processing unit (CPU), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits used to control the execution of the program in this application. In this embodiment, the processor 301 has a SIMD architecture. The processor 301 interacts with the sensing module 305 or the communication interface 304 to acquire image data. The processor 301 performs processing tasks on the image data. These processing tasks may include lesion detection, organ segmentation, surgical simulation, etc. When performing convolution calculations in the processing tasks, the processor 301 executes the resource allocation method of this application to balance resources and improve computational efficiency.

[0128] Communication line 302 may include a path or bus for transmitting information between the aforementioned components.

[0129] The memory 303 may be a read-only memory (ROM), a random access memory (RAM), or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer. The memory 303 may exist independently and be connected to the processor 301 via communication line 302. Alternatively, the memory 303 may be integrated with the processor 301.

[0130] The memory 303 stores computer execution instructions for implementing the solutions of this application. The processor 301 executes the computer execution instructions stored in the memory 303 to implement the methods provided in the following embodiments of this application.

[0131] The communication interface 304 is used for communicating with other devices. In this embodiment, the communication interface 304 may be a module, circuit, bus, interface, transceiver, or other device capable of implementing communication functions. In this embodiment, the communication interface 304 may also receive image data transmitted by other devices.

[0132] The sensing module 305 is used to acquire image data. For example, the sensing module 305 includes a laser scanner. The laser scanner scans the patient's body parts to obtain three-dimensional point cloud data of those body parts.

[0133] In other embodiments of this application, the device may include more or fewer components than those shown in Figures 2 and 3, or combine some components, or split some components, or replace some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0134] The following section uses a mobile intelligent device, specifically a vehicle, as an example to illustrate the resource allocation method of this application.

[0135] For example, Figure 4 shows a flowchart of a resource allocation method provided in an embodiment of this application. The executing entity of this method can be a mobile intelligent device as described above, or it can be a processor in a mobile intelligent device. This embodiment of the application takes a processor in a vehicle as an example. As shown in Figure 4, the method includes the following steps:

[0136] S410. Obtain the number of valid data corresponding to each element of the first convolution kernel.

[0137] In this embodiment, the vehicle includes sensors for acquiring image data. The image data may be point cloud data. For example, the sensors may be radar or laser.

[0138] For example, during vehicle operation, the radar and laser in the vehicle sense the surrounding environment by sending laser pulses and measuring their reflection time, generating point cloud data of the surrounding environment.

[0139] In this embodiment of the application, the processor in the vehicle acquires image data by interacting with sensors.

[0140] In this embodiment, a model for processing sparse convolution calculations is configured in the vehicle. For example, the model for processing sparse convolution calculations can be an AI model, which can be equipped on an onboard computer system or vehicle processor, and its processing tasks can be executed by the system. Optionally, the processor can input image data into the trained AI model so that the AI ​​model can perform the processing tasks. Furthermore, the model for processing sparse convolution calculations can also be a specially designed algorithm model, a sparse matrix operation model (such as a sparse matrix operation model in a graphics processing unit (GPU), or a dedicated computing model, etc. This embodiment does not limit the model used for processing sparse convolution calculations.

[0141] The following explanation uses the model used in a vehicle to handle sparse convolution calculations as an example of an AI model.

[0142] In some examples, the AI ​​model can be a model used to implement perception functions, recognizing and analyzing image data to perceive the vehicle's surrounding environment and assist driving. For instance, in autonomous driving and driver assistance systems, vehicles acquire image data through sensors such as LiDAR, and the AI ​​model processes this image data to extract features to identify vehicles, pedestrians, obstacles, traffic lights, etc., within the image data.

[0143] In other examples, the AI ​​model can also be used to implement decision-making and control functions, perceiving environmental data based on image data and making intelligent decisions and real-time vehicle control based on the perceived environmental data. For example, based on the information identified in the above examples, such as vehicles, pedestrians, obstacles, and traffic lights, the vehicle speed can be intelligently adjusted to ensure a safe distance from surrounding vehicles and pedestrians.

[0144] The processing tasks can include object detection and recognition, image segmentation, map building, behavior prediction, trajectory planning, or navigation. For example, object detection and recognition can include identifying objects in an image (including but not limited to obstacles, pedestrians, roads, etc.) and labeling object categories. Image segmentation can include semantic segmentation and instance segmentation. Semantic segmentation involves dividing an image into multiple speech regions, each representing a different category of object, such as vehicles and buildings. Instance segmentation, based on semantic segmentation, further distinguishes different instances within the same category; for example, the category of "vehicles" includes multiple vehicles. Map building can involve identifying the environment and constructing a map based on the collected image data for subsequent localization, trajectory planning, and navigation. Behavior prediction can involve analyzing the behavior of objects in an image based on multiple frames of collected image data to inform subsequent decisions regarding vehicle trajectory planning and navigation. Trajectory planning can plan the vehicle's driving trajectory based on image data and trajectory planning algorithms to ensure vehicle safety, optimize the driving trajectory, improve driving efficiency, and enhance vehicle intelligence. Navigation can control or guide vehicle driving based on the analysis results of image data, flexibly responding to changes in road conditions and emergencies to improve the travel experience.

[0145] It is understood that the AI ​​model in this application embodiment needs to perform convolution calculations when performing processing tasks.

[0146] In this embodiment, the first convolutional kernel is the convolutional kernel used by the AI ​​model when performing inference (i.e., executing the aforementioned processing task). That is, the first convolutional kernel is the convolutional kernel in the AI ​​model.

[0147] As is understandable, convolutional kernels are the core element in AI models performing convolutional calculations. AI models use convolutional kernels to perform convolution operations on the data input to the AI ​​model to extract features from the input data. At the start of training, initial convolutional kernels are randomly set, but these kernels are optimized and adjusted during training to suit the needs of the processing task. After training, the configured convolutional kernels are obtained and applied to the AI ​​model to perform the corresponding processing task.

[0148] For example, the size of the first convolutional kernel can be 3x3. Or, the size of the first convolutional kernel can also be 5x5.

[0149] For example, the number of convolutional kernels in an AI model can be one or more. For instance, if the AI ​​model includes one 3x3 convolutional kernel 1, then when performing sparse convolution, convolutional kernel 1 is used as the first convolutional kernel for convolution calculation based on the resource allocation method provided in this application embodiment. Alternatively, if the AI ​​model includes three 3x3 convolutional kernels, namely convolutional kernel 2, convolutional kernel 3, and convolutional kernel 4, then when performing sparse convolution, different convolutional kernels are used for convolution calculation based on the resource allocation method provided in this application embodiment. For example, the three convolutional kernels in the AI ​​model, convolutional kernel 2, convolutional kernel 3, and convolutional kernel 4, are performed sequentially, with three processing steps based on the number of convolutional kernels in the AI ​​model. In the first processing step, convolutional kernel 2 is determined as the first convolutional kernel, and convolution calculation is performed based on convolutional kernel 2. In the second processing step, convolutional kernel 3 is determined as the first convolutional kernel, and convolution calculation is performed based on convolutional kernel 3. In the third processing step, convolutional kernel 4 is determined as the first convolutional kernel, and convolution calculation is performed based on convolutional kernel 4.

[0150] It is understandable that even if the AI ​​model includes multiple convolution kernels, each convolution kernel performs convolution calculations using the resource allocation method of this application in order to balance the load and improve efficiency and performance.

[0151] In this embodiment, a convolution kernel element (or convolution kernel unit) is the basic unit of the first convolution kernel. A convolution kernel element represents its position within the convolution kernel. The value of each convolution kernel element is the weight value of its corresponding position within the first convolution kernel. For example, if the first convolution kernel is 3*3 in size, it contains 9 convolution kernel elements, ordered from left to right and top to bottom as the first element (kernel_1), the second element (kernel_2), ..., the ninth element (kernel_9).

[0152] In this embodiment, when processing image data, the AI ​​model first performs voxelization on the image data, and then performs convolution calculations on the voxelized data (as an example of data to be processed). The data processed by the first convolution kernel is, for example, the data to be processed. For example, the voxel mesh obtained after voxelizing point cloud data is used as the data to be processed.

[0153] A voxel mesh is a mesh structure used to represent objects in three-dimensional space. It divides the three-dimensional space into uniform cubic units called voxels. Each voxel in the mesh has a specific attribute value, such as density, color, or material type. Voxel meshes are commonly used in 3D modeling, medical imaging, and computer vision to provide a detailed, discretized representation of three-dimensional objects, facilitating processing and analysis.

[0154] In this embodiment, the data to be processed includes valid voxels and invalid voxels. Valid voxels are voxels containing valid data, and invalid voxels are voxels that do not contain valid data.

[0155] In some examples, valid data can be determined by whether the voxel data is zero. If it is not zero, it is valid data, and the voxel containing valid data is a valid voxel. If the value is zero, it is invalid data, and the voxel containing invalid data is an invalid voxel.

[0156] In other examples, valid and invalid data can be distinguished based on a preset threshold. If the value is greater than the preset threshold, it is considered valid data; otherwise, it is considered invalid data. It is understood that the embodiments of this application do not limit the method for determining valid data.

[0157] The following explanation assumes that valid data is not zero and invalid data is zero.

[0158] In some examples, the vehicle processor determines the valid voxels in the data to be processed based on the first convolution kernel. If the specific value of the voxel corresponding to the center element of the first convolution kernel is not zero, the voxel is determined to be a valid voxel; otherwise, the voxel is determined to be an invalid voxel.

[0159] For example, based on the example in Figure 1 above, the first convolution kernel is a 3*3 convolution kernel, and the voxel grid obtained after voxelizing the image data is shown in Figure 1. The coordinates of the effective voxels are (1,1), (2,2), (2,7), (3,8), (6,2) and (7,3).

[0160] It is understandable that the image data collected by the same sensor at different times will differ for a vehicle. For example, for a moving vehicle, the first frame of point cloud data collected by the LiDAR at time T1 will be different from the second frame of point cloud data collected by the LiDAR at time T2 (the next time after T1). Therefore, the number of valid voxels (such as the number, position, and value of valid voxels) contained in the voxel grid corresponding to each frame of point cloud data will also be different. Correspondingly, for each frame of point cloud data, the number of valid data to be processed in each element of the first convolution kernel will also be different. Therefore, when processing each frame of point cloud data, the processor in the vehicle needs to allocate a corresponding amount of memory for each frame of point cloud data in order to process valid data containing different amounts of data. For example, when processing the first frame of point cloud data at time T1, a corresponding amount of memory is allocated based on the amount of data contained in the first frame of point cloud data; when processing the second frame of point cloud data at time T2, a corresponding amount of memory is allocated based on the amount of data contained in the second frame of point cloud data.

[0161] During convolution calculations, different sizes of memory need to be dynamically allocated or requested for each element of the first convolution kernel to ensure that each kernel element can complete the convolution calculation. It's understandable that the purpose of allocating or requesting memory for each kernel element is to support data storage and computation during the convolution calculation. This memory stores convolution kernel parameters, input data, output data, etc., and performs the convolution calculation within that memory. After the convolution calculation is complete, the occupied memory is released to allow other tasks to be executed. In other words, the vehicle processor needs to request memory in real-time based on the current data volume each time a convolution calculation is performed, and release the memory after the calculation is complete. The amount of memory requested for each convolution calculation is different, and there are overheads associated with allocating and releasing memory, affecting computational efficiency and system performance.

[0162] In this embodiment, the size of the memory space corresponding to each convolutional kernel element in the first convolutional kernel can be a first threshold.

[0163] In some examples, this first threshold is a preset value. For example, the preset value is the maximum amount of memory space used by a single convolutional kernel element when processing data, as determined experimentally or empirically.

[0164] Specifically, for example, the maximum amount of data each convolution kernel element may need to process during convolution calculations can be determined experimentally. This maximum amount can be understood as a first value. For example, the first value might be 20480 or 20480 bits. In some examples, based on this first value, the memory space required by each convolution kernel element when processing data of the first value is determined as a first threshold. For example, if the first value is 20480, then the first threshold can be 20480. In other examples, a second value is determined based on the first value, and this second value is greater than the first value. Based on this second value, the memory space required by each convolution kernel element when processing data of the second value is determined as a first threshold. For example, if the first value is 20480 or 20480 bits, then the first threshold can be a value larger than the first value, such as 21000 or 21000 bits. It is understood that, in this example, although there is an experimentally determined maximum value required by the convolution kernel element for convolution calculations, experimental data may have errors. To avoid problems such as insufficient memory allocation due to errors, preventing convolution calculations from being performed. When determining the memory size, select an array larger than the experimental maximum value and set an even larger value to prevent the amount of data to be processed for the convolution calculation from exceeding the experimental maximum value. This better addresses different computational needs and different data processing requirements.

[0165] It is understandable that the first threshold is a pre-set value for the elements of the convolution kernel, and is independent of the size of the first convolution kernel.

[0166] In other examples, the first threshold is a user-defined preset value.

[0167] Understandably, the device can be configured with different first thresholds for different processing scenarios, and these first thresholds vary from scenario to scenario. Each processing scenario corresponds to a processing task. Different processing scenarios have different data processing requirements and task complexity. For tasks with higher requirements and greater complexity, the first threshold is larger.

[0168] For example, in autonomous driving scenarios, image data acquired by LiDAR can be used for target detection and trajectory planning. The first threshold for target detection is higher than the first threshold for trajectory planning. Target detection mainly involves feature extraction and target recognition, requiring relatively little memory. Trajectory planning, however, requires acquiring environmental information about the vehicle's surroundings, identifying objects based on this information, and then performing complex path calculations and dynamic adjustments, demanding significantly more memory to process large amounts of data.

[0169] Understandably, the amount of data in an image and its resolution will affect the size of the first threshold. The higher the image resolution or the larger the amount of data in the image, the larger the first threshold will be.

[0170] For example, the first threshold for a medical imaging scenario is lower than the first threshold for an autonomous driving scenario. Autonomous driving scenarios require acquiring point cloud data of the vehicle's surrounding environment. However, medical imaging scenarios only require acquiring point cloud data of a specific organ. The amount of point cloud data acquired in medical imaging scenarios is typically smaller than that in autonomous driving scenarios. Therefore, the first threshold for medical imaging scenarios is usually lower than the first threshold for autonomous driving scenarios.

[0171] Understandably, to avoid dynamic memory allocation, this application pre-allocates the maximum memory space for each convolution kernel element, eliminating the need for dynamic memory partitioning and effectively avoiding the additional overhead caused by dynamic memory partitioning. Furthermore, allocating sufficient memory space for each convolution kernel element before computation improves computational efficiency and system performance.

[0172] In this embodiment, the data corresponding to each convolution kernel element in the first convolution kernel is written into a memory space of size 1 threshold, and convolution calculation is performed in this memory space. There is no need to dynamically allocate memory, which avoids the overhead of dynamically allocating and releasing memory during the convolution calculation process, and improves the computational efficiency and overall performance of the convolution operation.

[0173] It is understood that in this embodiment, before executing step S410, the vehicle processor acquires image data and performs voxelization processing on the image data to convert it into a voxel grid composed of voxels. Based on the first convolution kernel, the effective voxels in the voxel grid are determined, and the calculation formula for convolution calculation of each effective voxel is determined sequentially to establish a first hash table and a second hash table corresponding to the first convolution kernel. The first hash table records information about the effective voxels that are multiplied by each convolution kernel element. The second hash table records information about the effective voxels to which the calculation result obtained after multiplying each convolution kernel element should be written after multiplication.

[0174] It should be understood that the technical solution of this application is applied to the process of the processor reading data (i.e., the specific values ​​of the effective voxels and the specific values ​​of the first convolution kernel) based on the first hash table and the second hash table after obtaining the first hash table and the second hash table, and then performing convolution calculation.

[0175] In this embodiment of the application, first information of valid data in the data to be processed and a first convolution kernel for processing the data to be processed are obtained; based on the first information, the number of valid data corresponding to each convolution kernel element in the first convolution kernel is obtained.

[0176] The first piece of information is the location information of the valid data within the data to be processed. For example, the coordinates of the valid data mentioned above. The number of valid data corresponding to each convolution kernel element is based on the number of valid voxels that each convolution kernel element in the first convolution kernel needs to process when processing the data to be processed.

[0177] Specifically, the processor obtains the coordinates of valid data in the data to be processed, as well as the first convolutional kernel in the AI ​​model that processes that data. Based on the coordinates of each valid data point and the first convolutional kernel, it iterates through each valid data point to obtain a first hash table and a second hash table corresponding to the first convolutional kernel. The processor then uses either the first hash table or the second hash table to obtain the number of valid data points corresponding to each element of the first convolutional kernel.

[0178] For example, based on the examples in Table 1 or Table 2 above, the number of valid data corresponding to kernel_1 (the meaning of kernel_1 is as described above) is 3. kernel_2, kernel_3, kernel_4, kernel_6, kernel_7, and kernel_8 have no valid data, the number of valid data corresponding to kernel_5 is 6, and the number of valid data corresponding to kernel_9 is 3.

[0179] It is understandable that the number of valid data points corresponding to each convolution kernel element is the same in both the first and second hash tables. The number of valid data points corresponding to each convolution kernel element is the amount of data to be processed by that convolution kernel element.

[0180] Specifically, if the size of the first convolutional kernel is a*a, where a is a positive integer, usually an odd number, and the first threshold is m, the vehicle processor pre-allocates multiple memory spaces. The number of these memory spaces is equal to the number of elements in the convolutional kernel, and the size of each memory space is equal to the first threshold.

[0181] For example, if the size of the first convolutional kernel is 3*3, the first threshold is 20480 or 20480 bits. The vehicle processor pre-allocates a memory space of 20480 bits for each convolutional kernel element of the first convolutional kernel. That is, there are a total of 9 memory spaces of 20480 bits each. Each convolutional kernel element of the 3*3 first convolutional kernel uses one memory space.

[0182] For example, by reading data from the first hash table or the second hash table, the vehicle processor determines, based on the first hash table or the second hash table, that the number of valid data points corresponding to kernel_1 is 9216, the number of valid data points corresponding to kernel_2 is 9216, the number of valid data points corresponding to kernel_3 is 9216, the number of valid data points corresponding to kernel_4 is 9216, the number of valid data points corresponding to kernel_5 is 18432, the number of valid data points corresponding to kernel_6 is 9216, the number of valid data points corresponding to kernel_7 is 9216, the number of valid data points corresponding to kernel_8 is 9216, and the number of valid data points corresponding to kernel_9 is 9216. The vehicle processor writes the valid data corresponding to each convolution kernel element into the memory space corresponding to that convolution kernel element for subsequent convolution calculations.

[0183] For example, based on the example in Figure 1 above, if the size of the first convolutional kernel is 3*3 and the first threshold is 70, the vehicle processor pre-allocates a memory space of 70 bits for each element of the first convolutional kernel. The vehicle processor determines, based on the first hash table and the second hash table, that the number of valid data corresponding to kernel_1 is 3, the number of valid data corresponding to kernel_2 is 0, the number of valid data corresponding to kernel_3 is 0, the number of valid data corresponding to kernel_4 is 0, the number of valid data corresponding to kernel_5 is 6, the number of valid data corresponding to kernel_6 is 0, the number of valid data corresponding to kernel_7 is 0, the number of valid data corresponding to kernel_8 is 0, and the number of valid data corresponding to kernel_9 is 3.

[0184] S420: Obtain the number of computing cores.

[0185] In this embodiment, the computing core is a processing unit in the vehicle, and multiple computing cores can execute instructions or process data in parallel. Optionally, the computing core can also be a core or logic processor in a processor; the computing core can also be a graphics processing unit (GPU); the computing core can also be other hardware units in the vehicle that can be used for computing.

[0186] The computing core can be a currently idle computing core in the device.

[0187] In some examples, the number of idle computing cores in the current device can be queried using command-line tools.

[0188] For example, based on the example of a first threshold of 20480 in step S410, the vehicle processor determines the number of idle computing cores to be 10 using a command-line tool.

[0189] For example, based on the example where the first threshold is 70 in step S410, the vehicle processor determines the number of idle computing cores to be 4 using a command-line tool.

[0190] It is understandable that this application obtains the number of currently available computing cores in order to perform convolution calculations later.

[0191] S430. Determine the number of computational cores corresponding to each convolutional kernel element based on the number of valid data corresponding to each convolutional kernel element and the number of idle computational cores.

[0192] In this embodiment, the vehicle processor can calculate the total number of valid data to be processed based on the number of valid data corresponding to each convolution kernel element. Based on the total number of valid data to be processed and the number of currently idle computing cores, an average amount of valid data to be processed is planned for each computing core. The number of computing cores corresponding to each convolution kernel element is determined based on the number of valid data to be processed for each convolution kernel element and the average amount of valid data to be processed by each computing core.

[0193] Specifically, in some examples, the total number of valid data points to be processed for each convolutional kernel element is summed to obtain the total number of valid data points to be processed. This total number of valid data points is then divided by the number of currently idle computational kernels to obtain the average number of valid data points that each computational kernel should process. For each convolutional kernel element, the number of computational kernels corresponding to that convolutional kernel is determined based on the number of valid data points to be processed for that convolutional kernel element and the average number of valid data points that each computational kernel should process.

[0194] For example, based on the example where the first threshold is 20480 in steps S410 and S420, the total number of valid data to be processed is (9216+9216+9216+9216+18432+9216+9216+9216+9216) = 92160. The average number of valid data to be processed by each calculation core is (92160 / 10) = 9216. The number of computing cores corresponding to kernel_1 is (9216 / 9216) = 1, kernel_2 is (9216 / 9216) = 1, kernel_3 is (9216 / 9216) = 1, kernel_4 is (9216 / 9216) = 1, kernel_5 is (18432 / 9216) = 2, kernel_6 is (9216 / 9216) = 1, kernel_7 is (9216 / 9216) = 1, kernel_8 is (9216 / 9216) = 1, and kernel_9 is (9216 / 9216) = 1. In this way, of the 10 computational cores, 2 are allocated to kernel_5, and each of the other convolutional kernel elements in the first convolutional kernel is allocated one computational core. Each computational core can compute the same number of operations, and the workload is reasonably distributed among the cores to achieve load balancing. Furthermore, each core can complete computations synchronously without waiting, which improves computational efficiency, platform performance, and platform stability.

[0195] For example, based on the first threshold of 70 in steps S410 and S420, and the example in Figure 1, the total number of valid data to be processed is (3+0+0+0+6+0+0+0+3) = 12. The average number of valid data to be processed by each computing core is (12 / 4) = 3. The number of computing cores corresponding to kernel_1 is (3 / 3) = 1, the number of computing cores corresponding to kernel_2 is (0 / 1) = 0, the number of computing cores corresponding to kernel_3 is (0 / 1) = 0, the number of computing cores corresponding to kernel_4 is (6 / 3) = 2, the number of computing cores corresponding to kernel_5 is (0 / 1) = 0, the number of computing cores corresponding to kernel_6 is (0 / 1) = 0, the number of computing cores corresponding to kernel_7 is (0 / 1) = 0, the number of computing cores corresponding to kernel_8 is (0 / 1) = 0, and the number of computing cores corresponding to kernel_9 is (3 / 3) = 1. Thus, of the four computational kernels, two are allocated to kernel_5, one to kernel_1 and one to kernel_9, and no kernels are allocated to the other convolutional kernel elements.

[0196] Understandably, the purpose of this application is to balance the amount of data processed by each computational core, and the actual amount of data processed by each computational core can fluctuate around an average amount. Moreover, point cloud data contains a large amount of data, usually tens of thousands or even hundreds of millions of data points. Therefore, when using the scheme of this application to calculate the number of computational cores corresponding to each convolutional core, the average amount of data that each computational core should process is usually hundreds, thousands, or tens of thousands of data points or more. Even if a certain computational core processes more data, the amount of data processed by most computational cores is balanced, which can improve computational efficiency and avoid resource waste.

[0197] Understandably, in the above example, the number of valid data elements to be processed by each convolution kernel is generally an integer multiple of the average number of valid data elements that each computation kernel should process. However, in actual computation, other situations may exist (such as the number of valid data elements to be processed by each convolution kernel not being divisible by the average number of valid data elements that each computation kernel should process). In this case, the number of computation kernels corresponding to each convolution kernel can be determined based on the number of valid data elements to be processed by each convolution kernel and the average number of valid data elements that each computation kernel should process.

[0198] Optionally, if the number of valid data to be processed by each convolution kernel element is not divisible by the average number of valid data to be processed by each computation kernel, the number of computation kernels corresponding to each convolution kernel element can be determined based on the number of valid data to be processed by each convolution kernel element, the average number of valid data to be processed by each computation kernel, the number of currently idle computation kernels, and a preset strategy.

[0199] The preset strategy can be methods such as rounding or integer rounding. It fully utilizes each idle computing core, rationally allocates the amount of effective data actually processed by each core, balances the load on the cores, and improves computational efficiency.

[0200] In this embodiment, after determining the number of computational cores corresponding to each convolutional kernel element, a corresponding number of computational cores are allocated to each convolutional kernel element based on the number of computational cores corresponding to each convolutional kernel element. Each computational core processes the convolution calculation of its corresponding convolutional kernel element.

[0201] For example, if the number of computational cores corresponding to kernel_x is determined to be n, where n is a positive integer, then n computational cores are allocated to kernel_x. These n computational cores handle the convolution computations of kernel_x.

[0202] For example, based on the example of the first threshold being 20480 in steps S410 and S420 above, the vehicle processor allocates two computational kernels to kernel_5 in the first convolutional kernel. These two computational kernels are used to process the convolution calculation in kernel_5. The vehicle processor allocates one computational kernel to each of kernel_1, kernel_2, kernel_3, kernel_4, kernel_6, kernel_7, kernel_8, and kernel_9 in the first convolutional kernel. Each computational kernel is used to process the convolution calculation of one of its convolutional kernel elements. This achieves convolution calculation of image data.

[0203] It is understandable that the vehicle processor can perform the above steps S410-S430 for each acquired image data, and perform convolution calculations on each image data based on the currently available computing kernels to extract features from the image data.

[0204] In this application, by employing the aforementioned method, computing resources are rationally allocated based on the amount of data to be computed and the number of available computing cores, making full use of computing resources and effectively optimizing the resource allocation of computing cores to achieve load balancing. Furthermore, load balancing and efficient resource utilization help improve overall computing efficiency and speed, enabling computing tasks to run in a more efficient and balanced manner, optimizing resource management, and enhancing system performance.

[0205] It is understandable that the above examples perform convolution calculations based on the first convolution kernel. In some examples, the first convolution kernel may be decomposed into or include multiple second convolution kernels. The number of kernel elements contained in the second convolution kernel is less than the number of kernel elements contained in the first convolution kernel.

[0206] Optionally, the first convolutional kernel can be decomposed into multiple second convolutional kernels using decomposition methods such as depthwise separable convolution, short-circuit convolution, and matrix factorization.

[0207] For example, a convolutional kernel of size M*M can be decomposed into L convolutional kernels of size K*K using any of the decomposition methods described above. Here, M is a positive integer, L is a positive integer, K is a positive integer, M is greater than K, and L is greater than 1.

[0208] It is understandable that the specific implementation of decomposing the first convolutional kernel into multiple second convolutional kernels can be found in existing technologies, and will not be elaborated here.

[0209] For example, the first convolution kernel is a 5x5 convolution kernel, which can be decomposed into two 3x3 convolution kernels. The second convolution kernel is a 3x3 convolution kernel, and there are two second convolution kernels.

[0210] The specific steps to decompose a 5x5 convolutional kernel into two 3x3 convolutional kernels are as follows: First, the first 3x3 convolutional kernel is used to convolve the image data to obtain an intermediate feature map. Then, the second 3x3 convolutional kernel is used to convolve the intermediate feature map.

[0211] It's understandable that the receptive field of a 5x5 convolutional kernel processing image data is the same as that of two 3x3 convolutional kernels processing image data. The receptive field refers to the size of the area that a convolutional kernel can see in the input image data. Thus, two 3x3 convolutional kernels can simulate the effect of a 5x5 convolutional kernel.

[0212] When the first convolutional kernel is decomposed into multiple second convolutional kernels, one implementation of the above resource allocation method may include: obtaining multiple second convolutional kernels after the first convolutional kernel is decomposed, and determining the number of valid data corresponding to each convolutional kernel element in each second convolutional kernel. Obtaining the number of computational kernels. Based on the number of valid data corresponding to each convolutional kernel element of each second convolutional kernel and the number of idle computational kernels, determining the number of computational kernels corresponding to each convolutional kernel element of each second convolutional kernel. Allocating computational kernels to each convolutional kernel element of each second convolutional kernel based on the number of computational kernels corresponding to each convolutional kernel element of each second convolutional kernel.

[0213] It is understandable that when the first convolution kernel is decomposed into multiple second convolution kernels, the hash table corresponding to the second convolution kernel can be determined by processing the first hash table and the second hash table corresponding to each second convolution kernel according to the processing order of the multiple second convolution kernels and the image data.

[0214] For example, consider decomposing a 5x5 convolutional kernel into two 3x3 convolutional kernels, kernel a and kernel b, both 3x3 in size. The processing order is as follows: kernel a is used to perform convolution calculations on the image data, and then kernel b is used to perform convolution calculations on the result of kernel a. Based on this order, the hash table corresponding to kernel a should be determined first, followed by the hash table corresponding to kernel b. Specifically, the convolution calculation formula for each effective voxel in the image data is determined based on kernel a. Each element of kernel a is then traversed to obtain the first and second hash tables corresponding to kernel a. Kernel a then performs convolution processing on the image data to obtain an intermediate feature map. Based on the convolution kernel b, determine the operation formula for convolution calculation of each effective voxel in the intermediate feature map, traverse each convolution kernel element in the convolution kernel b, and obtain the first hash table and the second hash table corresponding to the convolution kernel b.

[0215] It is understandable that the AI ​​model described above can also be executed when the first convolution kernel is decomposed into multiple second convolution kernels. In the process of processing image data based on multiple second convolution kernels, the resource allocation method of this application can be used to allocate computing kernels to multiple second convolution kernels.

[0216] Understandably, performing convolution calculations based on convolution kernels with smaller kernel sizes can significantly reduce the computational load. In other words, the smaller the number of elements in a convolutional kernel, the less computation it requires. For example, a 5x5 convolutional kernel requires 25 multiplication and addition operations, while two 3x3 convolutional kernels require only 18. Furthermore, decomposing an MxM convolutional kernel into L KxK kernels is equivalent to layering the convolutional computation. That is, a single-layer MxM convolutional kernel is divided into a multi-layer structure, consisting of L layers, each with a KxK convolutional kernel. For example, a 5x5 convolutional kernel can be decomposed into two 3x3 convolutional kernels. Convolutional recording based on multi-layered kernels allows for the introduction of more non-linear features and depth, capturing more complex features and giving the model (i.e., the AI ​​model mentioned above) stronger expressive power. Smaller kernel sizes make convolutional computation easier and more efficient, improving computational efficiency.

[0217] As shown in Figure 5, another resource allocation method provided in this application embodiment is applied to a device using a single instruction multiple data architecture. The method includes:

[0218] S510. Obtain first information about the valid data in the data to be processed, and a first convolution kernel for processing the data to be processed; the first information is related to the position information of the valid data in the data to be processed.

[0219] The first piece of information can be the location information of the valid data within the data to be processed. For example, the first piece of information could be the coordinates of the valid data within the data to be processed. Alternatively, the first piece of information could be information that allows the location information of the valid data within the data to be processed to be derived.

[0220] The data to be processed can be voxel data derived from image data after voxelization. Voxel data includes valid voxels and invalid voxels. Valid voxels are voxels containing valid data, and invalid voxels are voxels containing invalid data.

[0221] Optionally, valid data can be determined according to a preset method. This preset method could be whether the value is zero; if it is not zero, it is valid data; otherwise, it is invalid data. Alternatively, the preset method could be whether the value is greater than a preset threshold; if it is greater than the threshold, it is valid data; otherwise, it is invalid data.

[0222] The first convolutional kernel can be a convolutional kernel in the AI ​​model. The AI ​​model uses this first convolutional kernel to process the data.

[0223] It is understood that the above examples are based on 2D voxel meshes to more intuitively illustrate the convolution calculation process. It should be understood that the schemes in the embodiments of this application can also be applied to the process of convolution processing images in three-dimensional space.

[0224] S520. Based on the first information, obtain the first quantity; the first quantity is the total number of valid data corresponding to the first convolution kernel.

[0225] In some examples, obtaining a first quantity based on first information includes: obtaining the number of valid data corresponding to each convolutional kernel element in the first convolutional kernel based on the first information; and obtaining the first quantity based on the number of valid data corresponding to each convolutional kernel element in the first convolutional kernel.

[0226] Specifically, based on the first information of each valid data point and the first convolutional kernel, the first hash table and the second hash table corresponding to the first convolutional kernel are obtained by traversing each valid data point. The number of valid data points corresponding to each convolutional kernel element in the first convolutional kernel is obtained based on either the first hash table or the second hash table. The first count is calculated based on the number of valid data points corresponding to each convolutional kernel element in the first convolutional kernel.

[0227] The first hash table records information about the valid voxels used in multiplication operations with each element of the convolution kernel. The second hash table records information about the valid voxels to which the calculation result obtained after multiplying each element of the convolution kernel should be written.

[0228] For example, based on the first convolution kernel and the valid data, the valid voxels in the data to be processed are determined, and the operation formula for convolution calculation of each valid voxel is determined to establish a first hash table and a second hash table. The number of valid data corresponding to each convolution kernel element in the first convolution kernel is obtained according to the first hash table or the second hash table. The number of valid data corresponding to each convolution kernel element in the first convolution kernel is summed to obtain the total number of valid data corresponding to the first convolution kernel (i.e., the first number).

[0229] Optionally, multiple second convolutional kernels can be obtained based on the first convolutional kernel, wherein the number of kernel elements in each second convolutional kernel is less than the number of kernel elements in the first convolutional kernel. For example, the first convolutional kernel can be decomposed into multiple second convolutional kernels.

[0230] Specifically, multiple second convolutional kernels are obtained based on the first convolutional kernel. Based on the first information of each valid data point and the multiple second convolutional kernels, a first hash table and a second hash table corresponding to each second convolutional kernel are obtained. The number of valid data points corresponding to each convolutional kernel element in each second convolutional kernel is obtained based on the first or second hash table corresponding to each second convolutional kernel. The total number of valid data points corresponding to each second convolutional kernel is calculated based on the data of the valid data points corresponding to each convolutional kernel element in each second convolutional kernel. Based on the total number of valid data points corresponding to each second convolutional kernel, a sixth quantity is determined. The sixth quantity is the total number of valid data points corresponding to the multiple second convolutional kernels.

[0231] For example, based on the first information of each valid data and multiple second convolutional kernels, the first hash table and the second hash table corresponding to each second convolutional kernel are obtained, including: according to the processing order of the multiple second convolutional kernels, the multiple second convolutional kernels process the first information of each valid data, and obtain the first hash table and the second hash table corresponding to each second convolutional kernel.

[0232] For example, the first convolutional kernel is decomposed to obtain multiple second convolutional kernels. These multiple second convolutional kernels include a first sub-convolutional kernel and a second sub-convolutional kernel. The processing order is to process the first sub-convolutional kernel first, then the second sub-convolutional kernel. The first sub-convolutional kernel processes the first information of each valid data point in the data to be processed, obtaining a first hash table and a second hash table corresponding to the first sub-convolutional kernel. The first sub-convolutional kernel processes the data to be processed to obtain an intermediate feature map. The second sub-convolutional kernel determines the first information of each valid data point in the intermediate feature map, and processes this first information to obtain a first hash table and a second hash table corresponding to the second sub-convolutional kernel. That is, for multiple second convolutional kernels, processing is performed according to the processing order of the multiple second convolutional kernels, with the x-th processing based on the result of the (x-1)-th processing. Here, x is a positive integer.

[0233] In this context, the size of the memory space corresponding to each convolutional kernel element in the first convolutional kernel is defined as the first threshold. The first threshold is a preset value, or it can be the maximum memory space used by a single convolutional kernel element when processing data. The first threshold is related to the processing scenario of the data to be processed; different processing scenarios correspond to different first thresholds.

[0234] The specific value of the first threshold is independent of the size of the first convolution kernel.

[0235] S530. Obtain a third quantity based on the first quantity and the second quantity; the second quantity is associated with one or more processing units included in the device, and the third quantity is the average processing capacity planned for the processing units.

[0236] The processing unit refers to the currently idle computing resources in the device.

[0237] The second quantity can be the number of processing units when processing data to be processed. For example, the second quantity can be the total number of currently idle computing resources in the device. The second quantity can also be less than the total number of currently idle computing resources in the device, that is, the processing units when processing data to be processed can be one or more of the currently idle computing resources in the device.

[0238] In some examples, the number of processing units used to perform the convolution computation is obtained as a second number. A third number is then obtained by dividing the first number by the second number.

[0239] In some examples, the method further includes: allocating a corresponding number of processing units to each convolution kernel element based on the number of valid data corresponding to each convolution kernel element in the first convolution kernel and the third number.

[0240] Specifically, the first convolution kernel includes a first convolution kernel element and a second convolution kernel element. Then, based on the number of valid data corresponding to each convolution kernel element in the first convolution kernel and a third quantity, a corresponding number of processing units are allocated to each convolution kernel element, including: allocating a fourth number of first processing units to the first convolution kernel element based on the number of first valid data corresponding to the first convolution kernel element and the third quantity; and allocating a fifth number of second processing units to the second convolution kernel element based on the number of second valid data corresponding to the second convolution kernel element and the third quantity, wherein the sum of the fourth and fifth quantities is less than or equal to the second quantity.

[0241] In some examples, the method further includes: processing first valid data corresponding to the first convolution kernel element through a fourth number of first processing units; and processing second valid data corresponding to the second convolution kernel element through a fifth number of second processing units.

[0242] Optionally, when multiple second convolutional kernels are obtained based on the first convolutional kernel, the number of processing units for processing convolutional computation is obtained based on the multiple second convolutional kernels. This number is the second number. A third number is obtained by dividing the sixth number by the second number. The third number is the average amount of effective data that the processing units are planned to process. Based on the amount of effective data corresponding to each convolutional kernel element in each second convolutional kernel and the third number, a corresponding number of processing units are allocated to each convolutional kernel element in each second convolutional kernel.

[0243] In this application, the above method plans the average processing volume of valid data to be processed for each processing unit based on the total amount of valid data to be calculated and the number of processing units. Computational resources are then rationally allocated according to this average processing volume, ensuring that the actual amount of valid data processed by different processing units is not significantly different, typically fluctuating around the average processing volume. Thus, each processing unit processes approximately the same amount of data, fully utilizing computational resources, effectively optimizing resource allocation of computational cores, and achieving load balancing. Furthermore, load balancing and efficient resource utilization help improve overall computational efficiency and speed, enabling computational tasks to run in a more efficient and balanced manner, optimizing resource management, and enhancing system performance.

[0244] The foregoing mainly describes the solutions provided by the embodiments of this application from a methodological perspective. To achieve the above functions, it includes corresponding hardware structures and / or software modules for executing each function. Those skilled in the art should readily recognize that, based on the units and algorithm steps of the examples described in conjunction with the embodiments disclosed herein, this application can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed in hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0245] Based on the same inventive concept, this application provides a resource allocation device. Figure 6 shows a schematic diagram of the structure of a resource allocation device provided in this application. Exemplarily, the resource allocation device 600 may specifically include: a processing module 601 and an acquisition module 602. This resource allocation device 600 is used to execute the resource allocation method described in the method embodiment of Figure 4 or Figure 5.

[0246] The processing module 601 is used by the resource allocation device 600 to perform the processing function of any one of Figures 4 or 5. The acquisition module 602 is used to support the resource allocation device 600 in performing the acquisition function of any one of Figures 4 or 5.

[0247] The resource allocation device 600 may further include a storage module (not shown in FIG. 6) storing programs or instructions. When the processing module 601 and the acquisition module 602 execute the program or instructions, the resource allocation device 600 shown in FIG. 6 can perform the resource allocation method described in the above method embodiments. Of course, the resource allocation device 600 may also include other modules, or the resource allocation device 600 may include fewer modules. This application embodiment does not limit this.

[0248] The operation and / or function of each unit in the resource allocation device 600 are respectively to implement the corresponding process of the resource allocation method described in the above method embodiments. All relevant contents of each step involved in the above method embodiments can be referred to the functional description of the corresponding functional unit. The technical effect of the resource allocation device can be referred to the technical effect of the method described in the above method embodiments. For the sake of brevity, it will not be repeated here.

[0249] This application provides another resource allocation device, which includes a processor and a memory. The memory is coupled to the processor and is used to store computer-readable instructions. When the processor reads the computer-readable instructions from the memory, the resource allocation device executes the resource allocation method described in the method embodiment of FIG4 or FIG5.

[0250] This application also provides a domain controller, which includes a processor and a memory. The memory is coupled to the processor and is used to store computer-readable instructions. When the processor reads the computer-readable instructions from the memory, the domain controller executes the resource allocation method described in the embodiments of the method described in FIG4 or FIG5.

[0251] This application also provides a chip system, as shown in FIG7. The chip system 700 includes at least one processor 701 and at least one interface circuit 702. As an example, when the chip system 700 includes one processor and one interface circuit, the processor can be the processor 701 shown in the solid box in FIG7 (or the processor 701 shown in the dashed box), and the interface circuit can be the interface circuit 702 shown in the solid box in FIG7 (or the interface circuit 702 shown in the dashed box). When the chip system 700 includes two processors and two interface circuits, the two processors include the processor 701 shown in the solid box and the processor 701 shown in the dashed box in FIG7, and the two interface circuits include the interface circuit 702 shown in the solid box and the interface circuit 702 shown in the dashed box in FIG7. This is not a limitation.

[0252] The processor 701 and the interface circuit 702 can be interconnected via a line. For example, the interface circuit 702 can be used to receive signals. As another example, the interface circuit 702 can be used to send signals to other devices (such as the processor 701). Exemplarily, the interface circuit 702 can read instructions stored in memory and send the instructions to the processor 701. When the instructions are executed by the processor 701, the various steps in the above embodiments can be performed. Of course, the chip system may also include other discrete devices, and this application embodiment does not specifically limit this.

[0253] Optionally, there can be one or more processors in the chip system. The processor can be implemented in hardware or software. When implemented in hardware, the processor can be a logic circuit, integrated circuit, etc. When implemented in software, the processor can be a general-purpose processor that reads software code stored in memory.

[0254] Optionally, the chip system may also include a memory (not shown in Figure 7). There may be one or more memories. The memory may be integrated with the processor or disposed separately from the processor; this application does not limit this. For example, the memory may be a non-transient processor, such as read-only memory (ROM), which may be integrated with the processor on the same chip or disposed separately on different chips. This application does not specifically limit the type of memory or the arrangement of the memory and processor.

[0255] For example, the chip system may be a field programmable gate array (FPGA), an application-specific integrated circuit (ASIC), a system on a chip (SoC), a central processing unit (CPU), a network processor (NP), a digital signal processor (DSP), a microcontroller unit (MCU), a programmable logic device (PLD), or other integrated chips.

[0256] It should be understood that each step in the above method embodiments can be completed by integrated logic circuits in the processor hardware or by instructions in software form. The method steps disclosed in the embodiments of this application can be directly manifested as being executed by a hardware processor, or being executed by a combination of hardware and software modules in the processor.

[0257] This application also provides a computer-readable storage medium for storing one or more computer programs, the one or more computer programs including instructions that, when executed by a computer, cause the computer to perform the corresponding flow of the method described in the above embodiments.

[0258] Computer-readable storage media include, but are not limited to, any of the following: USB flash drive, portable hard drive, read-only memory (ROM), random access memory (RAM), magnetic disk or optical disk, and other media capable of storing program code.

[0259] In some embodiments, the disclosed method may be implemented as computer program instructions encoded in a machine-readable format on a computer-readable storage medium or on other non-transitory media or articles of art.

[0260] This application also provides a computer program product, which includes a computer program or instructions that, when run on a computer, cause the computer to execute the corresponding process of the method described in the above embodiments.

[0261] In addition, this application also provides an apparatus, which may specifically be a chip, component or module. The apparatus may include a connected processor and a memory. The memory is used to store computer execution instructions. When the apparatus is running, the processor can execute the computer execution instructions stored in the memory to cause the apparatus to perform the corresponding process of the method described in the above embodiments.

[0262] In addition, this application also provides a system, which may specifically be a chip, component or module. The system may include a connected processor and a memory. The memory is used to store computer execution instructions. When the system is running, the processor can execute the computer execution instructions stored in the memory to make the system execute the corresponding process of the method described in the above embodiments.

[0263] The apparatus, computer-readable storage medium, computer program product, or chip provided in the embodiments of this application are all used to execute the corresponding methods provided above. Therefore, the beneficial effects they can achieve can be referred to the beneficial effects of the corresponding methods provided above, and will not be repeated here.

[0264] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A resource allocation method, applied to a device using a single instruction multiple data architecture, characterized in that, The method includes: First information about valid data in the data to be processed is obtained, as well as a first convolutional kernel for processing the data to be processed; the first information is related to the position information of the valid data in the data to be processed. Based on the first information, a first quantity is obtained; the first quantity is the total number of valid data corresponding to the first convolutional kernel. A third quantity is obtained based on the first and second quantities; the second quantity is associated with one or more processing units included in the device, and the third quantity is the average processing volume planned for the processing units.

2. The method according to claim 1, characterized in that, The step of obtaining the first quantity based on the first information includes: Based on the first information, obtain the number of valid data corresponding to each convolutional kernel element in the first convolutional kernel; The first quantity is obtained based on the number of valid data corresponding to each element of the first convolution kernel.

3. The method according to claim 1, characterized in that, The size of the memory space corresponding to each element of the first convolutional kernel is the first threshold.

4. The method according to claim 3, characterized in that, The first threshold is a preset value, or the first threshold is the maximum value of the memory space used by a single convolutional kernel element in the first convolutional kernel when processing data.

5. The method according to claim 3 or 4, characterized in that, The first threshold is related to the processing scenario of the data to be processed, and the first threshold is different for different processing scenarios.

6. The method according to any one of claims 1 to 5, characterized in that, The method further includes: Based on the number of valid data corresponding to each convolution kernel element in the first convolution kernel and the third quantity, a corresponding number of processing units are allocated to each convolution kernel element.

7. The method according to claim 6, characterized in that, The first convolutional kernel includes a first convolutional kernel element and a second convolutional kernel element. The step of allocating a corresponding number of processing units to each convolutional kernel element based on the number of valid data corresponding to each convolutional kernel element in the first convolutional kernel and the third quantity includes: Based on the number of first valid data corresponding to the first convolution kernel element and the third number, allocate a fourth number of first processing units to the first convolution kernel element; Based on the number of second valid data corresponding to the second convolution kernel element and the third number, a fifth number of second processing units are allocated to the second convolution kernel element, wherein the sum of the fourth number and the fifth number is less than or equal to the second number.

8. The method according to claim 7, characterized in that, The method further includes: The first valid data corresponding to the first convolution kernel element is processed by the fourth number of the first processing units. And the second valid data corresponding to the second convolution kernel element is processed by the fifth number of the second processing units.

9. The method according to any one of claims 1 to 8, characterized in that, The processing unit is the currently idle computing resource in the device.

10. A resource allocation device, characterized in that, include: A processor and a memory, the memory being coupled to the processor, the memory being used to store computer-readable instructions, which, when read from the memory by the processor, cause the resource allocation device to perform the method as described in any one of claims 1-9.

11. A domain controller, characterized in that, include: A processor and a memory, the memory being coupled to the processor, the memory being used to store computer-readable instructions, which, when read from the memory by the processor, cause the domain controller to perform the method as described in any one of claims 1-9.

12. A vehicle, characterized in that, The vehicle includes the resource allocation device as claimed in claim 10 or the domain controller as claimed in claim 11.

13. A chip system, characterized in that, It includes at least one processor and at least one interface circuit, the at least one interface circuit being used to perform transceiver functions, and the at least one processor being used to perform the method as described in any one of claims 1-9.

14. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a computer program that, when executed by a processor, implements the method as described in any one of claims 1-9.

15. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the method as described in any one of claims 1-9.

Images