Model reasoning method and device, medium and program product

Abstract

The invention relates to the technical field of artificial intelligence, and discloses a model reasoning method and device, a medium and a program product. The method comprises the following steps: acquiring an initial input tensor and a height dimension parameter corresponding to a current operator; according to the initial input tensor and the height dimension parameter, calculating to obtain a target output height corresponding to each calculation core, and according to the target output height, obtaining a target input height range corresponding to each calculation core; and according to the target input height range corresponding to each calculation core, carrying out height dimension value filling on the initial input tensor to obtain a target input tensor, and through each calculation core, carrying out calculation of a current operator according to the target input tensor to obtain a calculation result. By dividing the calculation task of the single operator and distributing the calculation task to the plurality of calculation cores, full and efficient utilization of the calculation cores can be realized, the idle probability of the calculation cores can be reduced, and the utilization rate of calculation resources can be improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, and in particular to a model inference method, device, medium and program product. BACKGROUND

[0002] With the growth of real-time service demand, inference tasks are usually submitted in single batch or very small batch. In the model inference process, single batch optimization has become a key step to improve system performance.

[0003] At present, the existing model inference method needs to perform the calculation of each operator in turn for a single batch inference task composed of a series of serial execution operators. However, the existing computing device usually has hundreds or thousands of computing cores, and the calculation load of each operator in the prior art cannot utilize all computing cores. In most execution time, only a small part of computing cores are in active state, resulting in serious idling of computing cores and waste of computing resources. SUMMARY

[0004] The present application provides a model inference method, device, medium and program product, which can realize full and efficient utilization of computing cores, reduce the idle probability of computing cores, and improve the utilization rate of computing resources.

[0005] According to an aspect of the present application, a model inference method is provided, comprising:

[0006] obtaining an initial input tensor and a height dimension parameter corresponding to a current operator;

[0007] According to the initial input tensor and the height dimension parameter, a target output height corresponding to each computing core is calculated, and according to the target output height, a target input height range corresponding to each computing core is obtained;

[0008] According to the target input height range corresponding to each computing core, the initial input tensor is filled with numerical values in the height dimension to obtain a target input tensor, and the current operator is calculated according to the target input tensor through each computing core to obtain a calculation result.

[0009] According to another aspect of the present application, a model inference device is provided, comprising:

[0010] An initial input tensor obtaining module is configured to obtain an initial input tensor and a height dimension parameter corresponding to a current operator;

[0011] The target input height range acquisition module is configured to calculate a target output height corresponding to each calculation core according to the initial input tensor and the height dimension parameter, and acquire a target input height range corresponding to each calculation core according to the target output height.

[0012] The calculation result acquisition module is configured to perform numerical padding of the initial input tensor in the height dimension according to the target input height range corresponding to each calculation core, acquire a target input tensor, and perform calculation of the current operator according to the target input tensor through each calculation core to acquire a calculation result.

[0013] According to another aspect of the present application, an electronic device is provided, which comprises:

[0014] at least one processor; and

[0015] a memory connected to the at least one processor in communication; wherein

[0016] The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor to enable the at least one processor to execute the model inference method according to any one of the embodiments of the present application.

[0017] According to another aspect of the present application, a computer readable storage medium is provided, which stores a computer program for enabling a processor to implement the model inference method according to any one of the embodiments of the present application when executed by the processor.

[0018] According to another aspect of the present application, a computer program product is provided, which comprises a computer program for implementing the model inference method according to any one of the embodiments of the present application when executed by a processor.

[0019] The technical solution of the embodiments of the present application acquires an initial input tensor and a height dimension parameter corresponding to a current operator, calculates a target output height corresponding to each calculation core according to the initial input tensor and the height dimension parameter, and acquires a target input height range corresponding to each calculation core according to the target output height; numerical padding of the initial input tensor in the height dimension is performed according to the target input height range corresponding to each calculation core to acquire a target input tensor, and calculation of the current operator is performed according to the target input tensor through each calculation core to acquire a calculation result; by dividing and distributing the calculation task of a single operator to multiple calculation cores, sufficient and efficient utilization of the calculation cores can be achieved, the idle probability of the calculation cores can be reduced, and the utilization rate of the calculation resources can be improved.

[0020] It is to be understood that the details set forth in the description contained herein do not limit the scope of the application. Other embodiments of the application will be readily apparent to those skilled in the art from the description herein. With reference to the drawings, embodiments of the application are herein described. BRIEF DESCRIPTION OF DRAWINGS

[0021] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed to be used in the embodiments will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative effort based on these drawings.

[0022] Figure 1 is a flow chart of a model reasoning method according to an embodiment of the present application;

[0023] Figure 2 is a flow chart of a model reasoning method according to an embodiment of the present application;

[0024] Figure 3 is a flow chart of an operator splitting method according to an embodiment of the present application;

[0025] Figure 4 is a structural schematic diagram of a model reasoning device according to an embodiment of the present application;

[0026] Figure 5 is a structural schematic diagram of an electronic device implementing a model reasoning method according to an embodiment of the present application. DETAILED DESCRIPTION

[0027] In order to make the technical personnel in the art better understand the present application, the technical solutions in the embodiments of the present application will be described clearly and completely with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some embodiments of the present application, not all. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative effort should be within the scope of protection of the present application.

[0028] It should be noted that the terms "first", "second", "alteration", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and do not necessarily describe a specific order or sequence. It should be understood that the terms used in this way can be interchanged under appropriate circumstances, so that the embodiments of the application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device including a series of steps or units does not necessarily limit to those steps or units clearly listed, but can include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0029] Embodiment one

[0030] Figure 1 A flowchart of a model inference method provided for embodiment one of the present application, the present embodiment can be applicable to the case of operator calculation for single batch or small batch model inference task, the method can be executed by a model inference device, which can be realized in the form of hardware and / or software. Typically, the model inference device can be configured in an electronic device, for example, a computer device, a server, etc. As shown in the figure, the method comprises: Figure 1

[0031] S110, obtaining an initial input tensor and a height dimension parameter corresponding to a current operator.

[0032] The current operator can be a basic calculation unit of the model inference process, for example, it can be a two-dimensional convolution operator. The initial input tensor can be a multidimensional array input into the current operator, for example, the initial input tensor can be an array with data dimensions of [N, H, W, C]. Wherein, N represents the batch dimension, H represents the height dimension, W represents the width dimension, and C represents the channel dimension. The height dimension parameter can include the parameter value of the height direction in the operator operation, for example, it can correspond to the first element of the tuple parameter, i.e. the element corresponding to index 0.

[0033] S120, calculating the target output height corresponding to each calculation core according to the initial input tensor and the height dimension parameter, and obtaining the target input height range corresponding to each calculation core according to the target output height.

[0034] ​In the embodiment, a mapping relationship between an input height, a height dimension parameter and an output height can be preset for a current operator. Thus, when calculating a target output height, an initial input tensor corresponding input height can be acquired first, and the current output height can be acquired by looking up the preset mapping relationship according to the current input height and the height dimension parameter. Then, the output height can be distributed among the calculation cores in a loop according to the identifier from small to large, so as to determine the target output height corresponding to each calculation core.

[0035] After the target output height is acquired, the output height range corresponding to each calculation core can be determined according to the target output height. For example, if the target output height of the calculation core 0 is 4, the output height range corresponding thereto is [0, 3] (0-3 rows); if the target output height of the calculation core 1 is 4, the output height range corresponding thereto is [4, 7] (4-7 rows). Finally, the minimum output height value and the maximum output height value corresponding to each calculation core can be acquired according to the output height range, and the minimum input height value corresponding to the current minimum output height value and the maximum input height value corresponding to the current maximum output height value can be acquired according to the preset mapping relationship between the minimum output height value and the minimum input height value and the mapping relationship between the maximum output height value and the maximum input height value, so as to determine the target input height range based on the minimum input height value and the maximum input height value.

[0036] Optionally, calculating the target output height corresponding to each calculation core according to the initial input tensor and the height dimension parameter can include:

[0037] calculating an initial output height according to the initial input tensor and the height dimension parameter;

[0038] acquiring the number of calculation cores, and calculating the target output height corresponding to each calculation core according to the initial output height and the number.

[0039] In an optional example, the initial input height corresponding to the initial input tensor can be obtained, and the initial output height corresponding to the initial input height can be found based on the preset mapping relationship between input height, height dimension parameters, and output height. Then, the initial output height can be divided by the number of computation cores to obtain the quotient and remainder. If the remainder is not 0, then when the computation core identifier i is less than the remainder, the target output height corresponding to computation core i is determined to be equal to the quotient + 1; and when the identifier i is greater than or equal to the remainder, the corresponding target output height is determined to be equal to the quotient. For example, if the initial output height is 14, the number of computation cores is 4, and the identifiers are 0-3, then the quotient is 3 and the remainder is 2. For computation cores 0 and 1, since their identifiers are less than the remainder, the corresponding target output height is 3 + 1 = 4; while for computation cores 2 and 3, since their identifiers are equal to or greater than the remainder, the corresponding target output height is 3.

[0040] Optionally, calculating the initial output height based on the initial input tensor and the height dimension parameter may include:

[0041] Based on the initial input tensor, obtain the initial input height and initial height dimension padding value, and based on the height dimension parameters, obtain the convolution kernel height, height dimension stride, and height dimension dilation.

[0042] The initial output height is calculated based on the initial input height, the initial height dimension padding value, the convolution kernel height, the height dimension stride, and the height dimension dilation.

[0043] When the current operator is a two-dimensional convolution operator (Conv2D), the height dimension parameter may include the convolution kernel height (kernel_size[0]), the height dimension stride (stride[0]), and the height dimension dilation (dilation[0]).

[0044] In this embodiment, the H value can be extracted based on the data dimensions [N,H,W,C] of the initial input tensor to serve as the initial input height. Simultaneously, numerical detection can be performed on the initial input tensor to obtain the number of rows where all values ​​are 0, which is used as the initial height dimension padding value (padding[0]). Then, based on the following formula, the initial input height can be... The initial output height is calculated using padding[0], kernel_size[0], stride[0], and dilation[0]. .For example, Given 28, padding[0]=1, kernel_size[0]=3, stride[0]=2, dilation[0]=1, then =14.

[0045] .

[0046] Optionally, obtaining the target input height range corresponding to each computing core based on the target output height may include:

[0047] Based on the target output height, obtain the output height range corresponding to each computing core;

[0048] The target input height range corresponding to each computing core is calculated based on the output height range, the initial height dimension padding value, the convolution kernel height, the height dimension stride, and the height dimension dilation.

[0049] In this embodiment, the output height range corresponding to each computing core can be determined based on the target output height corresponding to each computing core. For example, if the initial output height is 14 and the target output height corresponding to computing cores 0-3 is {4,4,3,3}, then the corresponding output height range is {[0,3],[4,7],[8,10],[11,13]}. Then, the output height range can be derived in reverse based on the initial height dimension padding value, convolution kernel height, height dimension stride, and height dimension dilation rate to obtain the corresponding target input height range. In a specific example, the minimum output height value of the output height range can be used as the basis for the following formula. and maximum output height value The input starting height value is calculated. and range size and according to and Generate target input height range [ , + For example, if the output height range is {[0,3],[4,7],[8,10], [11,13]}, then the corresponding target input height range is {[-1,7],[7,15],[15,21], [21,27]}.

[0050] ;

[0051] .

[0052] S130. Based on the target input height range corresponding to each computing core, the initial input tensor is numerically filled with height dimension values ​​to obtain the target input tensor. Then, through each computing core, the current operator is calculated based on the target input tensor to obtain the calculation result.

[0053] In this embodiment, the initial input height of the initial input tensor can be evenly divided among the computing cores to obtain the initial input height range corresponding to each computing core. Then, the initial input height range can be compared with the target input height range to obtain padding information, which is then merged with the initial input tensor to obtain the target input tensor. Finally, based on the target input height range and padding information corresponding to each computing core, the input height range of the computing core within the target input tensor can be determined. The target input tensor is then divided based on this input height range to obtain tensor fragments corresponding to each computing core. These tensor fragments are then sent to the corresponding computing cores, allowing them to perform operator calculations based on these fragments to obtain the corresponding calculation results. Alternatively, the target input tensor and the input height range corresponding to the computing core can be sent completely to the computing core. In this case, the computing core needs to first extract the corresponding tensor fragments from the target input tensor based on the input height range, and then perform operator calculations based on these tensor fragments.

[0054] The technical solution of this invention involves obtaining the initial input tensor and height dimension parameter corresponding to the current operator; calculating the target output height corresponding to each computing core based on the initial input tensor and height dimension parameter, and obtaining the target input height range corresponding to each computing core based on the target output height; filling the initial input tensor with height dimension values ​​based on the target input height range corresponding to each computing core to obtain the target input tensor; and performing the calculation of the current operator based on the target input tensor through each computing core to obtain the calculation result. By dividing and allocating the calculation task of a single operator to multiple computing cores, the full and efficient utilization of computing cores can be achieved, the idle probability of computing cores can be reduced, and the utilization rate of computing resources can be improved.

[0055] Example 2

[0056] Figure 2 This is a flowchart of a model reasoning method provided in Embodiment 2 of the present invention. This embodiment is a further refinement of the above technical solution, and the technical solution in this embodiment can be combined with one or more of the above implementation methods. Figure 2 As shown, the method includes:

[0057] S210. Obtain the initial input tensor and height dimension parameters corresponding to the current operator.

[0058] S220. Based on the initial input tensor and the height dimension parameter, calculate the target output height corresponding to each computing core, and based on the target output height, obtain the target input height range corresponding to each computing core.

[0059] S230. Based on the initial input tensor, obtain the initial input height, and based on the initial input height and the number of computing cores, obtain the initial input height range corresponding to each computing core.

[0060] In this embodiment, the H value in the data dimension of the initial input tensor can be extracted to obtain the initial input height. Then, the initial input height can be divided by the number of computing cores to obtain the quotient value as the input height corresponding to each computing core. Based on this input height and the identifier of each computing core, the range of initial input height corresponding to each computing core is determined. For example, if the initial input height is 28 and the number is 4, then the input height corresponding to computing cores 0-3 is {7,7,7,7}, and the range of initial input height corresponding to computing cores 0-3 is {[0,6],[7,13],[14,20],[21,27]}.

[0061] S240. Based on the initial input height range and the target input height range, obtain the filling position and filling row number corresponding to each computing core, and fill the initial input tensor with numerical values ​​in the height dimension based on the filling position and filling row number corresponding to each computing core to obtain the target input tensor.

[0062] Specifically, the initial input height range and the target input height range are compared. If the initial input height range does not completely encompass the target input height range, the initial input height range is padded to completely encompass the target input height range, and the padded position and the number of padded rows are obtained. For example, if the initial input height range is {[0,6],[7,13],[14,20],[21,27]} and the target input height range is {[-1,7],[7,15],[15,21],[21,27]}, then the padded information is {[1,1],[0,2],[0,1], [0,0]}. Here, the first digit indicates the beginning of the data block, the second digit indicates the end of the data block, and the value indicates the number of padded rows. For example, [1,1] means padded one row before row 0 and one row after row 6, with both rows containing 0. Finally, based on the determined padded position and the number of padded rows, the initial input tensor can be padded with numerical values ​​for the height dimension to obtain the target input tensor.

[0063] S250. Based on the target input height range corresponding to each computing core, obtain the current input height range corresponding to each computing core.

[0064] In this embodiment, after the numerical filling is completed, the target input height range can be shifted to the right to eliminate negative values, thereby obtaining the current input height range. For example, if the target input height range is {[-1,7],[7,15],[15,21],[21,27]}, then the adjusted current input height range is {[0,8],[8,16],[16,22],[22,28]}.

[0065] S260. Through each of the aforementioned calculation cores, the current operator is calculated based on the corresponding current input height range and the target input tensor, and the calculation result is obtained.

[0066] Specifically, based on the current input height range, the tensor fragment corresponding to each computational core can be determined in the target input tensor. Then, the computational core can perform tensor computation based on the corresponding tensor fragment to obtain the corresponding computation result.

[0067] Optionally, by using each of the aforementioned computational cores to perform calculations on the current operator based on the corresponding current input height range and the target input tensor, and obtaining the calculation results, the following may be included:

[0068] Based on the current input height range corresponding to the current computing core, the target input tensor is segmented to obtain the segmented tensor corresponding to the current computing core, and the segmented tensor is stored in the memory region corresponding to the current computing core according to the memory address corresponding to the current computing core.

[0069] The current computing core performs calculations on the current operator based on the segmented tensor stored in the corresponding memory region, and obtains the calculation results.

[0070] In this embodiment, operators can be divided into partitioning operators and non-partitioning operators. To make the transfer of tensor data between these two types more reasonable and efficient, a new memory management method is designed, including a contiguous memory strategy for non-partitioning operators and a cross-storage and access strategy for partitioning operators. Specifically, for the contiguous memory strategy for non-partitioning operators, the target input tensor is not pre-partitioned; the system allocates a large contiguous block of memory in the device address space. During execution, multiple computational cores do not copy data but directly and in parallel access the specified data range in this contiguous memory block through an address mapping mechanism. This strategy ensures the globality and continuity of the data layout, greatly simplifies the synchronization logic between computational cores, avoids unnecessary data transfer overhead, and is particularly suitable for operators with high computational density and strong data dependencies.

[0071] For the cross-storage and access strategy of the partitioning operator, the system manages a dedicated cross-segment address space. When an operator is logically partitioned, its data blocks are no longer stored contiguously, but are distributed and stored in multiple memory regions associated with different computing cores. All partitioning operators uniformly obtain their required input data slices from this cross-segment address space. Each computing core is pre-allocated an address range of a specific length within the cross-segment, which can obtain the corresponding tensor data as input.

[0072] In an optional example, the target input tensor can be pre-segmented based on the current input height range to obtain the segmented tensor corresponding to the current computing core. Then, according to the preset mapping relationship between computing cores and memory addresses, the segmented tensor can be stored in the corresponding memory region. Finally, when it is the current computing core's turn to perform tensor computation, the current computing core reads the corresponding segmented tensor from the memory region and performs the specific tensor computation to obtain the corresponding computation result.

[0073] For example, the operator segmentation process can be as follows: Figure 3 As shown, each segmented tensor is defined as an independent data region, labeled as Region[d0] to Region[d3]. Region[d0] serves as the operator input sent to computation core 0, and similarly, Region[d1] corresponds to computation core 1, and so on. In this way, each computation core obtains a subset of the original global input tensor and independently executes the same operator operations, but processes different spatial parts of the input data.

[0074] In this embodiment, when data exchange is required, for contiguously stored data, different blocks can directly access the padding data via offsets. Only the beginning of computing core 0 and the end of computing core 3 need additional processing, with each row of padding data being 0. For separated storage, the entire row of data needs to be copied from adjacent computing cores to the padding location, with 0s added at the beginning of computing core 0 and the end of computing core 3.

[0075] It should be noted that during data flow, existing technologies insert a four-core group synchronization operation before each operator execution to ensure that all required input data is ready, and data exchange operations typically occur immediately after operator computation and synchronization. However, in this embodiment, the synchronization operations in two specific scenarios can be completely eliminated. Specifically, for continuous data and unsegmented operators, when the data is stored contiguously and neither adjacent operator uses a segmentation mode, consistency is guaranteed without synchronization due to the simple and clear data dependencies. For separated data and segmented operators, when the data itself is stored separately and both adjacent operators use a segmentation mode, the computation cores access data through cross-segment addresses, and their dependencies are naturally decoupled through memory address mapping, requiring no additional synchronization operations.

[0076] The solution of this invention provides a more rational memory allocation mechanism, using different memory allocation modes in different scenarios, thus making more efficient use of the memory resources in the computing card. Secondly, it reduces redundant synchronization operations, significantly lowering the synchronization overhead in parallel inference and accelerating the entire model's inference process.

[0077] The technical solution of this invention involves obtaining an initial input height based on an initial input tensor, and then obtaining the initial input height range corresponding to each computing core based on the initial input height and the number of computing cores. Based on the initial input height range and the target input height range, the filling position and number of filling rows corresponding to each computing core are obtained. The initial input tensor is then numerically filled in the height dimension based on the filling position and number of filling rows corresponding to each computing core to obtain the target input tensor. By comparing the initial input height range and the target input height range to determine the filling position and number of filling rows, and then filling the data based on these positions and rows, data dimension alignment can be achieved, improving the accuracy of operator calculation. Based on the target input height range corresponding to each computing core, the current input height range corresponding to each computing core is obtained. Through each computing core, the current operator is calculated based on the corresponding current input height range and the target input tensor to obtain the calculation result. By correcting the target input height range and performing operator calculations based on the corrected input height range, the accuracy of operator calculation can be further improved, thereby improving the accuracy of model inference.

[0078] In this embodiment, the input tensor of the operator is segmented within the framework according to the segmentation formula and operator properties. The segmented tensor data is then distributed to the corresponding computing cores. This process does not require modification of the operator calculation logic; tensor segmentation is performed only on the framework side. After the calculation is completed, the data from the computing cores is merged. By using different types of memory allocation methods for different scenarios during this process, memory allocation and release are performed more rationally, reducing memory fragmentation and improving the utilization of video memory bandwidth. Secondly, by adopting a synchronization strategy optimization, synchronization and data exchange are integrated into one operation, reducing the number of kernels and eliminating some engineering-related synchronization operations.

[0079] By deeply integrating two innovative optimization techniques—tensor partitioning and synchronization point elimination—this invention achieves a breakthrough in the inference performance of target networks (including ResNet50 and YOLO series). Tensor partitioning intelligently divides small batches of input data into finer-grained subtasks, enabling highly parallel load balancing across computing cores and fully exploiting hardware potential. Simultaneously, synchronization point elimination accurately identifies and removes unnecessary computational synchronization in traditional pipelines, reducing inter-processor waiting time and communication overhead.

[0080] These two technologies work together to optimize performance bottlenecks in the inference process, forming a smooth and efficient computational pipeline. Ultimately, without sacrificing any model accuracy, the end-to-end overall inference time for such complex networks was reduced by more than half, achieving a significant performance improvement of more than 100%, laying a solid technical foundation for deployment and application in high-concurrency, low-latency real-time scenarios.

[0081] Example 3

[0082] Figure 4 This is a schematic diagram of the structure of a model inference device provided in Embodiment 3 of the present invention. Figure 4 As shown, the device includes: an initial input tensor acquisition module 310, a target input height range acquisition module 320, and a calculation result acquisition module 330; wherein,

[0083] The initial input tensor acquisition module 310 is used to acquire the initial input tensor and height dimension parameter corresponding to the current operator;

[0084] The target input height range acquisition module 320 is used to calculate the target output height corresponding to each computing core based on the initial input tensor and the height dimension parameter, and to acquire the target input height range corresponding to each computing core based on the target output height.

[0085] The calculation result acquisition module 330 is used to fill the initial input tensor with the height dimension according to the target input height range corresponding to each of the calculation cores, obtain the target input tensor, and perform the calculation of the current operator according to the target input tensor through each of the calculation cores to obtain the calculation result.

[0086] The technical solution of this invention involves obtaining the initial input tensor and height dimension parameter corresponding to the current operator; calculating the target output height corresponding to each computing core based on the initial input tensor and height dimension parameter, and obtaining the target input height range corresponding to each computing core based on the target output height; filling the initial input tensor with height dimension values ​​based on the target input height range corresponding to each computing core to obtain the target input tensor; and performing the calculation of the current operator based on the target input tensor through each computing core to obtain the calculation result. By dividing and allocating the calculation task of a single operator to multiple computing cores, the full and efficient utilization of computing cores can be achieved, the idle probability of computing cores can be reduced, and the utilization rate of computing resources can be improved.

[0087] Optionally, the target input height range acquisition module 320 includes:

[0088] An initial output height calculation unit is used to calculate the initial output height based on the initial input tensor and the height dimension parameter;

[0089] The target output height calculation unit is used to obtain the number of computing cores and calculate the target output height corresponding to each computing core based on the initial output height and the number.

[0090] Optionally, the initial output height calculation unit is specifically used to obtain the initial input height and the initial height dimension padding value based on the initial input tensor, and to obtain the convolution kernel height, height dimension stride and height dimension hole rate based on the height dimension parameters.

[0091] The initial output height is calculated based on the initial input height, the initial height dimension padding value, the convolution kernel height, the height dimension stride, and the height dimension dilation.

[0092] Optionally, the target input height range acquisition module 320 includes:

[0093] The output height range acquisition unit is used to acquire the output height range corresponding to each of the computing cores based on the target output height.

[0094] The target input height range calculation unit is used to calculate the target input height range corresponding to each calculation core based on the output height range, the initial height dimension padding value, the convolution kernel height, the height dimension stride, and the height dimension dilation rate.

[0095] Optionally, the calculation result acquisition module 330 includes:

[0096] An initial input height range acquisition unit is used to acquire an initial input height based on the initial input tensor, and to acquire the initial input height range corresponding to each computing core based on the initial input height and the number of computing cores;

[0097] The target input tensor acquisition unit is used to obtain the filling position and filling row number corresponding to each computing core according to the initial input height range and the target input height range, and to fill the initial input tensor with numerical values ​​of the height dimension according to the filling position and filling row number corresponding to each computing core to obtain the target input tensor.

[0098] Optionally, the calculation result acquisition module 330 includes:

[0099] The current input height range acquisition unit is used to acquire the current input height range corresponding to each computing core based on the target input height range corresponding to each computing core.

[0100] The calculation result acquisition unit is used to perform calculations on the current operator based on the corresponding current input height range and the target input tensor through each of the calculation cores, and to obtain the calculation result.

[0101] Optionally, the calculation result acquisition unit is specifically used to perform tensor segmentation on the target input tensor according to the current input height range corresponding to the current computing core, obtain the segmented tensor corresponding to the current computing core, and store the segmented tensor in the memory area corresponding to the current computing core according to the memory address corresponding to the current computing core.

[0102] The current computing core performs calculations on the current operator based on the segmented tensor stored in the corresponding memory region, and obtains the calculation results.

[0103] The model reasoning apparatus provided in the embodiments of the present invention can execute the model reasoning method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0104] The collection, storage, use, processing, transmission, provision, and disclosure of user personal information involved in the technical solution disclosed herein comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0105] Example 4

[0106] Figure 5 A schematic diagram of an electronic device 40 that can be used to implement embodiments of the present invention is shown. The electronic device 40 is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device 40 can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices (such as helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0107] like Figure 5 As shown, the electronic device 40 includes at least one processor 41 and a memory, such as a read-only memory (ROM) 42 or a random access memory (RAM) 43, communicatively connected to the at least one processor 41. The memory stores computer programs executable by the at least one processor. The processor 41 can perform various appropriate actions and processes based on the computer program stored in the read-only memory 42 or loaded from the storage unit 48 into the random access memory 43. The RAM 43 can also store various programs and data required for the operation of the electronic device 40. The processor 41, ROM 42, and RAM 43 are interconnected via a bus 44. An input / output (I / O) interface 45 is also connected to the bus 44.

[0108] Multiple components in electronic device 40 are connected to I / O interface 45, including: input unit 46, such as keyboard, mouse, etc.; output unit 47, such as various types of monitors, speakers, etc.; storage unit 48, such as disk, optical disk, etc.; and communication unit 49, such as network card, modem, wireless transceiver, etc. Communication unit 49 allows electronic device 40 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0109] Processor 41 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 41 include, but are not limited to, central processing units, graphics processing units, various special-purpose artificial intelligence computing chips, various processors running machine learning model algorithms, digital signal processors, and any suitable processor, controller, microcontroller, etc. Processor 41 performs the various methods and processes described above, such as model inference methods.

[0110] In some embodiments, the model inference method may be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 48. In some embodiments, part or all of the computer program may be loaded and / or mounted on electronic device 40 via ROM 42 and / or communication unit 49. When the computer program is loaded into RAM 43 and executed by processor 41, one or more steps of the model inference method described above may be performed. Alternatively, in other embodiments, processor 41 may be configured to execute the model inference method by any other suitable means (e.g., by means of firmware).

[0111] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays, application-specific integrated circuits (ASICs), application-specific standard products (ASICs), system-on-a-chip (SoCs), complex programmable logic devices, computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0112] Computer programs used to implement the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be performed. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0113] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory, read-only memory, erasable programmable read-only memory, optical fibers, portable compact disk read-only memory, optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0114] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device 40, which includes: a display device (e.g., a cathode ray tube or liquid crystal display) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device 40. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0115] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or computing systems that include middleware components (e.g., application servers), or computing systems that include frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0116] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact via a communication network. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server.

[0117] This embodiment may also include a computer program product, which includes a computer program that, when executed by a processor, implements the model reasoning method provided in any embodiment of the present invention.

[0118] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.

[0119] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A model inference method, comprising: The method comprises the following steps: obtaining an initial input tensor and a height dimension parameter corresponding to a current operator; calculating a target output height corresponding to each calculation core according to the initial input tensor and the height dimension parameter, and obtaining a target input height range corresponding to each calculation core according to the target output height; performing numerical padding of the initial input tensor in the height dimension according to the target input height range corresponding to each calculation core, obtaining a target input tensor, and performing calculation of the current operator according to the target input tensor through each calculation core to obtain a calculation result.

2. The method of claim 1, wherein, The method comprises the following steps: calculating a target output height corresponding to each calculation core according to the initial input tensor and the height dimension parameter, and obtaining a target input height range corresponding to each calculation core according to the target output height; calculating a target output height corresponding to each calculation core according to the initial input tensor and the height dimension parameter, and obtaining a target input height range corresponding to each calculation core according to the target output height.

3. The method of claim 2, wherein, The method comprises the following steps: obtaining an initial input height and an initial height dimension padding value according to the initial input tensor, and obtaining a convolution kernel height, a height dimension step and a height dimension hole rate according to the height dimension parameter; calculating an initial output height according to the initial input height, the initial height dimension padding value, the convolution kernel height, the height dimension step and the height dimension hole rate.

4. The method of claim 3, wherein, The method comprises the following steps: obtaining an output height range corresponding to each calculation core according to the target output height; calculating a target input height range corresponding to each calculation core according to the output height range, the initial height dimension padding value, the convolution kernel height, the height dimension step and the height dimension hole rate.

5. The method of claim 1, wherein, The method comprises the following steps: obtaining an initial input height according to the initial input tensor, and obtaining an initial input height range corresponding to each calculation core according to the initial input height and the number of calculation cores; obtaining a padding position and a padding row number corresponding to each calculation core according to the initial input height range and the target input height range, and performing numerical padding of the initial input tensor in the height dimension according to the padding position and the padding row number corresponding to each calculation core to obtain a target input tensor.

6. The method of claim 5, wherein, The method comprises the following steps: obtaining a current input height range corresponding to each calculation core according to the target input height range corresponding to each calculation core; performing calculation of the current operator according to the current input height range corresponding to each calculation core and the target input tensor through each calculation core to obtain a calculation result.

7. The method of claim 6, wherein, The calculation result is obtained by performing calculation of the current operator on the corresponding current input height range and the target input tensor through each calculation core, and includes: According to the current input height range corresponding to the current calculation core, the target input tensor is tensor-sliced to obtain a sliced tensor corresponding to the current calculation core, and the sliced tensor is stored in a memory area corresponding to the current calculation core according to a memory address corresponding to the current calculation core; The calculation result is obtained by performing calculation of the current operator on the sliced tensor stored in the memory area corresponding to the current calculation core through the current calculation core.

8. An electronic device, comprising: The electronic device includes: at least one processor, and a memory connected in communication with the at least one processor; wherein The memory stores a computer program that can be executed by the at least one processor, and the computer program is executed by the at least one processor to enable the at least one processor to execute the model inference method of any one of claims 1-7.

9. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a computer program, and the computer program is used to enable the processor to implement the model inference method of any one of claims 1-7 when executed.

10. A computer program product, characterised in that, The computer program is used to implement the model inference method of any one of claims 1-7 when executed by the processor.

Images