Shape reasoning method and apparatus, data processing method and apparatus, and storage medium

Abstract

This disclosure provides a shape reasoning method, a data processing method, a shape reasoning apparatus, a data processing apparatus, and a non-transitory computer-readable storage medium. The shape reasoning method includes: acquiring input shape information; determining multiple reasoning shapes based on the input shape information; performing shape reasoning on the multiple reasoning shapes based on a network to obtain multiple reasoning information groups corresponding to each of the multiple reasoning shapes, wherein the network includes multiple operators, and each reasoning information group corresponding to a reasoning shape includes multiple kernel functions of the multiple operators corresponding to the reasoning shape, multiple tensor information of the multiple operators corresponding to the reasoning shape, and memory configuration information corresponding to the reasoning shape; and storing the multiple reasoning information groups.

Description

Technical Field

[0001] Embodiments of this disclosure relate to a shape reasoning method, a data processing method, a shape reasoning apparatus, a data processing apparatus, and a non-transitory computer-readable storage medium. Background Technology

[0002] Generally, deep learning inference engines infer inputs with fixed (immutable) shapes. In this case, the inference engine performs specific optimizations for each input shape to maximize model inference performance. However, in deep learning, input shapes are often variable. For example, in speech recognition and natural language processing, sentence lengths are typically variable; additionally, in computer vision (CV), image sizes frequently change. When the input shape changes, the shape of each tensor in the network needs to be re-inferred, significantly impacting model performance. Summary of the Invention

[0003] At least one embodiment of this disclosure provides a shape inference method, comprising: acquiring input shape information; determining multiple inference shapes based on the input shape information; performing shape inference on the multiple inference shapes based on a network to obtain multiple inference information groups corresponding to the multiple inference shapes respectively, wherein the network includes multiple operators, and the inference information group corresponding to each inference shape includes multiple kernel functions of the multiple operators corresponding to the inference shape, multiple tensor information of the multiple operators corresponding to the inference shape, and memory configuration information corresponding to the inference shape; and storing the multiple inference information groups.

[0004] For example, in a shape reasoning method provided in at least one embodiment of this disclosure, the shape reasoning includes: for each reasoning shape: reasoning the reasoning shape based on the network to obtain a reasoning information group corresponding to the reasoning shape; establishing a mapping relationship between the reasoning shape and the reasoning information group; wherein, reasoning the reasoning shape based on the network to obtain a reasoning information group corresponding to the reasoning shape includes: reasoning the reasoning shape based on the network to generate multiple tensor information of the multiple operators corresponding to the reasoning shape; determining memory configuration information corresponding to the reasoning shape based on the multiple tensor information; and compiling multiple kernel functions of the multiple operators corresponding to the reasoning shape based on the multiple tensor information and the operator information corresponding to the multiple operators.

[0005] For example, in the shape reasoning method provided in at least one embodiment of this disclosure, the plurality of reasoned shapes are determined based on hardware information and the network.

[0006] For example, in the shape reasoning method provided in at least one embodiment of this disclosure, the input shape information includes an input shape range and a number of shapes to be reasoned, or the input shape information includes multiple input shapes.

[0007] For example, in the shape reasoning method provided in at least one embodiment of this disclosure, the shape reasoning method is executed by a reasoning engine during the establishment phase.

[0008] At least one embodiment of this disclosure also provides a data processing method, comprising: obtaining a current input to a first network; obtaining an inference shape matching the shape of the current input from an inference cache information set as a current inference shape; obtaining an inference information group corresponding to the current inference shape from the inference cache information set as a current inference information group, wherein the inference cache information set includes multiple inference information groups and multiple inference shapes corresponding one-to-one with the multiple inference information groups, the multiple inference information groups being obtained according to the shape inference method described in any embodiment of this disclosure; preprocessing the current input based on the current inference shape to obtain a processed input; processing the processed input using the first network based on the current inference information group to obtain an intermediate output; and postprocessing the intermediate output based on the current input to obtain a final output.

[0009] For example, in a data processing method provided in at least one embodiment of this disclosure, preprocessing the current input based on the current inference shape to obtain a processed input includes: comparing the shape of the current input with the current inference shape; directly using the current input as the processed input in response to the current input having the same shape as the current inference shape; and processing the current input based on the current inference shape in response to the current input having a different shape than the current inference shape to obtain the processed input, wherein the shape of the processed input has the same shape as the current inference shape.

[0010] For example, in a data processing method provided in at least one embodiment of this disclosure, post-processing the intermediate output based on the current input to obtain a final output includes: determining a target output corresponding to the current input based on the current input; comparing the shape of the target output with the shape of the intermediate output; directly using the intermediate output as the final output in response to the target output having the same shape as the intermediate output; and processing the intermediate output based on the target output in response to the target output having a different shape than the intermediate output to obtain the final output, wherein the shape of the final output is the same as the shape of the target output.

[0011] For example, in a data processing method provided in at least one embodiment of this disclosure, processing the current input based on the current inference shape to obtain the processed input includes: performing fill processing or image scaling processing on the current input based on the current inference shape to obtain the processed input.

[0012] For example, in the data processing method provided in at least one embodiment of this disclosure, the current input includes image, audio data, or text data.

[0013] For example, in a data processing method provided in at least one embodiment of this disclosure, the inference cache information set corresponds to the current input.

[0014] For example, in a data processing method provided in at least one embodiment of this disclosure, the current inference shape is greater than or equal to the shape of the current input.

[0015] At least one embodiment of this disclosure also provides a shape reasoning apparatus, comprising: one or more memories storing computer-executable instructions non-transitory; and one or more processors configured to execute the computer-executable instructions, wherein the computer-executable instructions, when executed by the one or more processors, implement the shape reasoning method according to any embodiment of this disclosure.

[0016] At least one embodiment of this disclosure also provides a data processing apparatus, comprising: one or more memories storing computer-executable instructions non-transitoryly; and one or more processors configured to execute the computer-executable instructions, wherein the computer-executable instructions are executed by the one or more processors to implement the data processing method according to any embodiment of this disclosure.

[0017] At least one embodiment of this disclosure also provides a non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, implement the shape reasoning method or the data processing method according to any embodiment of this disclosure. Attached Figure Description

[0018] To more clearly illustrate the technical solutions of the embodiments of this disclosure, the accompanying drawings of the embodiments will be briefly described below. Obviously, the drawings described below only relate to some embodiments of this disclosure and are not intended to limit this disclosure.

[0019] Figure 1 A schematic flowchart illustrating a shape reasoning method provided for at least one embodiment of this disclosure;

[0020] Figure 2A schematic diagram illustrating a shape reasoning method provided in at least one embodiment of this disclosure;

[0021] Figure 3 A schematic flowchart illustrating a data processing method provided for at least one embodiment of this disclosure;

[0022] Figure 4 A schematic diagram illustrating a data processing method provided in at least one embodiment of this disclosure;

[0023] Figure 5 A schematic block diagram of a shape reasoning device provided for at least one embodiment of this disclosure;

[0024] Figure 6 A schematic block diagram of a shape reasoning device provided for at least one embodiment of this disclosure; and

[0025] Figure 7 A schematic diagram of a non-transitory computer-readable storage medium provided for at least one embodiment of the present disclosure. Detailed Implementation

[0026] To make the objectives, technical solutions, and advantages of the embodiments of this disclosure clearer, the technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this disclosure. All other embodiments obtained by those skilled in the art based on the described embodiments of this disclosure without creative effort are within the scope of protection of this disclosure.

[0027] Unless otherwise defined, the technical or scientific terms used in this disclosure shall have the ordinary meaning understood by one of ordinary skill in the art to which this disclosure pertains. The terms “first,” “second,” and similar terms used in this disclosure do not indicate any order, quantity, or importance, but are merely used to distinguish different components. Terms such as “comprising” or “including” mean that the element or object preceding the word encompasses the elements or objects listed following the word and their equivalents, without excluding other elements or objects. Terms such as “connected” or “linked” are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. Terms such as “upper,” “lower,” “left,” and “right” are used only to indicate relative positional relationships, and these relative positional relationships may change accordingly when the absolute position of the described objects changes.

[0028] To keep the following description of the embodiments of this disclosure clear and concise, detailed descriptions of some known functions and known components have been omitted.

[0029] For deep learning inference engines, current technology adopts a solution of generating a set of kernel functions that support different shapes. When the input shape changes, the shape of each tensor in the network needs to be re-inferred, which has a significant impact on performance. In addition, the operators in the network also need to support dynamic input shapes (i.e., inputs of different shapes), which places high demands on the operators. It is difficult to achieve optimal performance for operators that support dynamic input shapes, which restricts the performance improvement of the entire network and also restricts the model coverage of dynamic shapes. In addition to the operators themselves, the allocation mechanism of each level of cache is also difficult to achieve optimal performance. Therefore, existing solutions that support dynamic input shapes have a significant performance penalty and place high demands on operators.

[0030] At least one embodiment of this disclosure provides a shape reasoning method, which includes: acquiring input shape information; determining multiple reasoning shapes based on the input shape information; performing shape reasoning on the multiple reasoning shapes based on a network to obtain multiple reasoning information groups corresponding to the multiple reasoning shapes respectively, wherein the network includes multiple operators, and the reasoning information group corresponding to each reasoning shape includes multiple kernel functions of the multiple operators corresponding to the reasoning shape, multiple tensor information of the multiple operators corresponding to the reasoning shape, and memory configuration information corresponding to the reasoning shape; and storing the multiple reasoning information groups.

[0031] The shape reasoning method provided in the embodiments of this disclosure avoids the time consumption of re-performing shape reasoning after each change in the input shape by pre-reasoning multiple reasoning information groups corresponding to various shapes. It can optimize memory allocation to the greatest extent, especially the use of cache. It can also optimize the calculation speed of operators to the greatest extent, and achieve better reasoning performance in networks with more operators. It is highly practical and has a very wide range of application prospects.

[0032] At least one embodiment of this disclosure also provides a data processing method, which includes: obtaining the current input of a first network; obtaining an inference shape matching the shape of the current input from an inference cache information set as the current inference shape; obtaining an inference information group corresponding to the current inference shape from the inference cache information set as the current inference information group, wherein the inference cache information set includes multiple inference information groups and multiple inference shapes corresponding one-to-one with the multiple inference information groups, and the multiple inference information groups are obtained according to the shape inference method described in any embodiment of this disclosure; preprocessing the current input based on the current inference shape to obtain a processed input; processing the processed input using the first network based on the current inference information group to obtain an intermediate output; and postprocessing the intermediate output based on the current input to obtain a final output.

[0033] The data processing method provided by the embodiments of this disclosure can efficiently process dynamic input shapes based on multiple inference information groups corresponding to various shapes obtained through pre-inference. This avoids the time consumption of re-performing shape inference after each change in input shape, thereby optimizing the network model, accelerating the processing speed of the network model, and improving network performance. In addition, it provides powerful pre- and post-processing functions for user convenience. Moreover, it does not contradict conventional dynamic input shape technical solutions, allowing users to choose according to specific needs. It is highly practical and has a very broad application prospect.

[0034] Embodiments of this disclosure also provide a shape reasoning apparatus, a data processing apparatus, and a computer-readable storage medium. The shape reasoning method described above can be applied to the shape reasoning apparatus provided in the embodiments of this disclosure, and the data processing method described above can be applied to the data processing apparatus provided in the embodiments of this disclosure. The shape reasoning apparatus and the data processing apparatus can be configured on an electronic device. The electronic device can be a personal computer, a mobile terminal, etc., and the mobile terminal can be a mobile phone, headphones, a tablet computer, an in-vehicle device, etc.

[0035] It should be noted that, in the embodiments of this disclosure, dynamic input shape means that the input shape of the deep learning network model may change dynamically between multiple inferences.

[0036] It's important to note that the inference engine can be divided into two phases: the build phase and the runtime phase. The build phase primarily handles configuration tasks required for deep learning model inference, such as shape inference, pre-allocation of cache, and generation and loading of necessary kernels. This build phase is time-consuming but does not affect the time spent in the subsequent runtime phase. The runtime phase corresponds to the actual production environment. Here, real input (images, language, text, etc.) is propagated and loaded into the memory space pre-allocated in the build phase. The kernels generated in the build phase are then invoked for inference.

[0037] The embodiments of this disclosure will now be described in detail with reference to the accompanying drawings, but this disclosure is not limited to these specific embodiments.

[0038] Figure 1 A schematic flowchart illustrating a shape reasoning method provided in at least one embodiment of this disclosure. Figure 2 This is a schematic diagram of a shape reasoning method provided for at least one embodiment of the present disclosure.

[0039] The shape reasoning method provided in the embodiments of this disclosure can be implemented by a reasoning engine, and the shape reasoning method is executed during the build phase. For example, as Figure 1 As shown, an embodiment of the present disclosure provides a shape reasoning method including steps S100 to S130.

[0040] In step S100, the input shape information is obtained.

[0041] In step S110, multiple inference shapes are determined based on the input shape information.

[0042] In step S120, shape reasoning is performed on multiple inference shapes based on the network to obtain multiple inference information groups corresponding to each inference shape. For example, the network includes multiple operators, and the inference information group corresponding to each inference shape includes multiple kernel functions of multiple operators corresponding to the inference shape, multiple tensor information of multiple operators corresponding to the inference shape, and memory configuration information corresponding to the inference shape.

[0043] In step S130, multiple groups of inference information are stored.

[0044] For example, in step S100, the input shape information can be provided by the user. The user can input several commonly used specific input shapes or a range of input shapes and the number of shapes to be inferred according to actual application needs. Then, the inference engine can provide suggested inference shapes. For example, the inference engine will provide recommended inference shapes based on the automatically monitored underlying hardware information and feed the recommended inference shapes back to the user. The recommended inference shapes have a high degree of hardware matching, which will maximize hardware utilization and improve computing performance.

[0045] In some embodiments, in step S100, the input shape information includes an input shape range and a number of shapes to be inferred. In this case, in step S110, the inference engine determines multiple inference shapes based on the input shape range, the number of shapes to be inferred, and the underlying hardware information. These multiple inference shapes are within the input shape range, and the number of inference shapes is equal to the number of shapes to be inferred. In other embodiments, in step S100, the input shape information includes multiple input shapes. In this case, in step S110, the multiple inference shapes can be multiple input shapes. It should be noted that, depending on the actual hardware information and considering various performance aspects, the multiple inference shapes may also differ from the multiple input shapes.

[0046] For example, multiple inference shapes are determined based on factors such as the hardware information of the device that specifically applies the shape inference method and the network performing the inference (e.g., the network structure, etc.) so that the inference shape can be aligned with the hardware and better adapted to the network.

[0047] For example, in some embodiments, in step S120, shape reasoning includes: for each reasoning shape: reasoning about the reasoning shape based on the network to obtain a reasoning information group corresponding to the reasoning shape; and establishing a mapping relationship between the reasoning shape and the reasoning information group.

[0048] For example, in some embodiments, reasoning is performed on the inference shape based on the network to obtain a set of inference information corresponding to the inference shape, including: reasoning on the inference shape based on the network to generate multiple tensor information corresponding to the inference shape for multiple operators; determining memory configuration information corresponding to the inference shape based on the multiple tensor information; and compiling multiple kernel functions corresponding to the inference shape for multiple operators based on the multiple tensor information and the operator information corresponding to the multiple operators.

[0049] The shape inference method provided in the embodiments of this disclosure is an efficient implementation of dynamic input shape. When performing shape inference, multiple tensor information corresponding to the inference shape of multiple operators are generated in advance, and multiple kernel functions corresponding to each inference shape of multiple operators are compiled, thereby avoiding the time consumption of re-performing shape inference every time the shape changes. Since the operators do not need to support dynamic input shape, the performance of the operators can be optimized and the coverage efficiency of the model can be improved. In addition, the memory configuration information corresponding to each inference shape can be determined, thereby maximizing the optimization of memory allocation, especially the use of high-speed cache.

[0050] For example, a network can be of various types, such as a network that performs image recognition, a network that performs speech recognition, a network that performs classification (e.g., the ResNet50 network model), a network that performs prediction (e.g., the BERT network model), etc.

[0051] For example, in step S130, multiple groups of inference information can be stored in the inference engine that implements the shape inference method, such as in a cache, or they can be stored in the cloud.

[0052] The following is combined with Figure 2 The process of providing a shape reasoning method according to an embodiment of this disclosure is described.

[0053] like Figure 2As shown, based on the input shape information, n inference shapes shape1 to shapen can be determined. Then, shape inference is performed on the n inference shapes shape1 to shapen based on the network to obtain n inference information groups IG1 to IGn, which correspond one-to-one with the n inference shapes. The network includes c operators op1 to opc. During the build phase, the inference engine infers all the tensor information required in the corresponding network in advance for each inference shape, such as the shape of the tensor, as well as the memory configuration information (u# information) of each inference shape. Then, the graphics processing unit (GPU) compiles the kernel function of each operator corresponding to each inference shape. Finally, the inference engine can cache the tensor information, memory configuration information, and kernel functions of all operators for launch. c and n are both positive integers.

[0054] For a given inference shape, the corresponding inference information set includes multiple kernel functions corresponding to the inference shape for c operators, multiple tensor information for c operators corresponding to the inference shape, and memory configuration information corresponding to the inference shape. For example, Figure 2 As shown, the inference information group IG1 corresponding to inference shape 1 includes m tensor information T11~T1m, memory configuration information Ma1, and v1 kernel functions K11~K1v1; the inference information group IG2 corresponding to inference shape 2 includes m tensor information T21~T2m, memory configuration information Ma2, and v2 kernel functions K21~K2v2; and so on, the inference information group IGn corresponding to inference shape n includes m tensor information Tn1~Tnm, memory configuration information Ma1, and v1 kernel functions K11~K1v1. n Kn1~Knv kernel functions n m, v1, v2 and v n All are positive integers. It should be noted that the number of tensor information and the number of kernel functions in each inference information group can be different, i.e., m and v1, v2, and v... n They can be different. Furthermore, v1, v2, and v... n They can be different from each other, or partially the same. The information and memory configuration information of tensors in each layer of the network are independent, but if the type and shape of the operators corresponding to the kernels are the same, the kernels can be reused. Therefore, the number of kernel functions may be less than the number of tensor information.

[0055] In the embodiments of this disclosure, the inference engine stores the network's structural information and the shape information of multiple sets of tensors obtained during the build phase. The weight file contains the weight parameters trained on the model. During model execution, the weight file shares the same set of information, meaning it remains fixed each time the model is run. The weight file is analogous to the parameters in a function; it changes during the training phase but remains unchanged during the inference phase after training. Based on the shape information of the multiple sets of tensors and the information of the operators in the network, the inference engine generates multiple corresponding kernels during the build phase for subsequent execution phase calls. Simultaneously, a mapping mechanism between inference shapes and corresponding kernels can be established so that the corresponding kernel can be quickly determined directly based on the inference shape during the execution phase.

[0056] In the shape inference method provided in the embodiments of this disclosure, during the build phase, the inference engine provides users with multiple shape setting interfaces. For each shape, the shape information of all required tensors in the network is inferred in advance. Simultaneously, the GPU compiles the kernel function corresponding to each operator for each shape and caches it for launch. Furthermore, this shape inference method can be combined with existing dynamic input shape implementation schemes, thereby providing users with two dynamic input shape implementation schemes. Users can choose the appropriate implementation scheme according to their actual situation, making the implementation of dynamic input shapes more flexible and improving the user experience.

[0057] At least one embodiment of this disclosure also provides a data processing method. Figure 3 This is a schematic flowchart illustrating a data processing method provided in at least one embodiment of the present disclosure. Figure 4 This is a schematic diagram of a data processing method provided in at least one embodiment of the present disclosure.

[0058] The data processing method provided in the embodiments of this disclosure can be executed during the runtime phase, and this data processing method can be executed by a processor. For example... Figure 3 As shown, an embodiment of this disclosure provides a data processing method including steps S200 to S230.

[0059] In step S200, the current input of the first network is obtained.

[0060] In step S210, the inference shape that matches the shape of the current input is obtained from the inference cache information set as the current inference shape.

[0061] In step S220, a reasoning information group corresponding to the current reasoning shape is obtained from the reasoning cache information set as the current reasoning information group. For example, the reasoning cache information set includes multiple reasoning information groups and multiple reasoning shapes that correspond one-to-one with the multiple reasoning information groups. The multiple reasoning information groups are obtained by reasoning for the first network according to the shape reasoning method described in any embodiment of this disclosure. That is, the first network in step S200 is the network in the above-described shape reasoning method.

[0062] In step S230, the current input is preprocessed based on the current inference shape to obtain the processed input.

[0063] In step S240, based on the current inference information group, the processed input is processed using the first network to obtain the intermediate output.

[0064] In step S250, the intermediate outputs are post-processed based on the current input to obtain the final output.

[0065] For example, in step S200, the current input includes images, audio data, or text data. Images can be images taken with a digital camera or mobile phone, and can be grayscale or color images. Text data can include text in various languages ​​(handwritten text and / or machine-printed text), symbols, graphics, etc. Audio data can include human or animal sounds, musical instrument sounds, machine sounds, etc.

[0066] For example, the current input can be pre-processed data. For instance, to avoid the impact of data quality issues, data imbalances, or other factors on the recognition of the current input, the data processing method provided in this disclosure may further include pre-processing the current input before processing it. Pre-processing can eliminate irrelevant or noisy information in the current input, facilitating better processing of the current input. Pre-processing may include scaling, cropping, gamma correction, data augmentation, or noise reduction filtering of the current input.

[0067] For example, the inference cache information set corresponds to the current input. It should be noted that "the inference cache information set corresponds to the current input" means that the shape of the current input is within the shape range determined by the input shape information, and the inference information group in the inference cache information set is determined based on an input shape information.

[0068] For example, the inference cache information set also corresponds to the first network, that is, the inference cache information set is the information set obtained by performing shape inference on the first network.

[0069] For example, in step S210, the current inference shape can be obtained from the multiple inference shapes included in the inference cache information set that is greater than or equal to the shape of the current input and is closest to the shape of the current input. That is, the current inference shape is greater than or equal to the shape of the current input.

[0070] For example, in step S220, after the current inference shape is determined, the inference information group corresponding to the current inference shape can be determined as the current inference information group based on the mapping relationship between the inference shape and the inference information group.

[0071] For example, in step S230, during the runtime phase, the actual current input needs to be preprocessed to map it to the current inference shape. Step S230 may include: comparing the shape of the current input with the current inference shape; if the shape of the current input is the same as the current inference shape, directly using the current input as the processed input; if the shape of the current input is different from the current inference shape, processing the current input based on the current inference shape to obtain the processed input. For example, the shape of the processed input may be the same as the current inference shape.

[0072] For example, in some embodiments, preprocessing includes processing that can adjust the shape, such as padding or resizing. Processing the current input based on the current inference shape to obtain the processed input includes: performing padding or resizing on the current input based on the current inference shape to obtain the processed input.

[0073] For example, in step S250, post-processing includes extraction processing, image resizing processing, and other processes to extract useful information.

[0074] For example, step S250 includes: determining a target output corresponding to the current input based on the current input; comparing the shape of the target output with the shape of an intermediate output; directly using the intermediate output as the final output if the shape of the target output is the same as the shape of the intermediate output; and processing the intermediate output based on the target output to obtain the final output if the shape of the target output is different from the shape of the intermediate output. For example, the shape of the final output is the same as the shape of the target output.

[0075] If the shape of the intermediate output obtained from the first network processing is inconsistent with the shape of the target output corresponding to the current input, post-processing is required to adjust the shape of the intermediate output to match the shape of the target output corresponding to the current input. For some network models, the shape of the target output can be directly obtained from the shape of the input; for others, shape reasoning is required to obtain the target output shape; and for still others, the shape of the output is fixed, in which case the network model's output is the target output, thus eliminating the need for post-processing.

[0076] The following is combined with Figure 4 The process of describing a data processing method provided by an embodiment of this disclosure is described.

[0077] like Figure 4 As shown, after obtaining the current input CI, a shape matching process can be performed in conjunction with the corresponding inference cache information set to obtain the current inference shape. Then, the current input is preprocessed based on the current inference shape to obtain a processed input, the shape of which is the same as the current inference shape. Next, memory allocation strategy matching can be performed to achieve memory copying, that is, to obtain the memory configuration information in the inference information group corresponding to the current inference shape and to perform memory configuration based on the memory configuration information. Then, kernel function matching can be performed, that is, to obtain multiple kernel functions in the inference information group corresponding to the current inference shape. Then, multiple operators in the first network can perform calculations based on these multiple kernel functions to obtain intermediate outputs. Finally, the intermediate outputs are post-processed based on the shape information of the current input to obtain the final output.

[0078] The shape reasoning method and data processing method provided by the embodiments of this disclosure are illustrated below with two examples.

[0079] In one example, taking the ResNet50 network model, during the build phase, the shape inference method is executed. Users can determine multiple inference shapes corresponding to multiple inference inputs based on the input shape information. For example, multiple inference inputs include three inference inputs: (2,3,192,192), (2,3,224,224), and (2,3,256,256). Then, the inference engine can infer three sets of inference information based on the shapes of these three inference inputs and cache these three sets for later use. For example, for each inference input, the inference engine can first infer the specific shapes of all intermediate tensors (tensors in each layer of the deep learning model) based on the shape of the inference input. Then, based on the specific shapes of all the inferred intermediate tensors, it determines the memory configuration information corresponding to the inference input. Finally, based on the specific shapes of all the inferred intermediate tensors and the information of the network itself, it obtains the kernel function of each operator in the network. During the runtime phase, data processing methods are executed. If an input (2,3,250,250) is received, shape matching can be performed based on this input. For example, this input (2,3,250,250) can match the inference input (2,3,256,256). Since the shape of this input (2,3,250,250) is different from the inference shape corresponding to the inference input (2,3,256,256), the input (2,3,250,250) can then be resized to obtain the processed input. The shape of this processed input is similar to the inference input (2,3,256,256). The input (2,3,256,256) has the same shape, meaning the processed input can also be represented as (2,3,256,256). Then, based on the inference information group (including tensor information, memory configuration information, and kernel) corresponding to the inference input (2,3,256,256), launch and computation are performed, and an output (i.e., intermediate output) is obtained. Finally, since the ResNet50 network model performs a classification task, the shape of the output result of the ResNet50 network model is fixed as (2*100), that is, the shape of the output is (2*100), so the output (2*100) can be directly used as the final classification result (i.e., the final output) and output to the user.

[0080] In one example, taking the BERT network model, the BERT network model has two inputs. During the build phase, the shape inference method is executed. The user can determine multiple inference shapes corresponding to multiple inference input groups based on the input shape information. Each inference input group includes two inference inputs, so each inference shape includes two sub-inference shapes corresponding to the two inference inputs. For example, multiple inference input groups include three inference input groups: inference input group {(1*200),(1*200)}, inference input group {(1*300),(1*300)}, and inference input group {(1*400),(1*400)}. Then, the inference engine can infer three inference information groups based on the three inference input groups mentioned above, and cache these three inference information groups for later use. For example, each inference input group corresponds to one inference information group. For each inference input group, the inference engine can perform shape inference based on the inference input group to obtain the specific shape of all intermediate tensors in the BERT network model. Then, based on the specific shapes of all the inferred intermediate tensors, the memory configuration information corresponding to the inference input group is determined. Finally, based on the specific shapes of all the inferred intermediate tensors and the information of the network itself, the kernel function of each operator in the network is obtained. During the runtime phase, data processing methods are executed. If the received input group is {(input1:1*220,input2:1*220)}, shape matching can be performed based on this input group. For example, this input group {(input1:1*220,input2:1*220)} can match the inference input group {(1*300),(1*300)}. Since the shapes corresponding to the two inputs in this input group {(input1:1*220,input2:1*220)} are different from the two sub-inference shapes corresponding to the inference input group {(1*300),(1*300)}, the input group {(input1:1*220,input2:1*220)} can then be used for shape matching. The input group {(input1:1*300, input2:1*300)} is padded to the inference input group {(input1:1*300, input2:1*300)}. Then, based on the inference information group (including tensor information, memory configuration information, and kernel) corresponding to the inference input group {(input1:1*300, input2:1*300)}, launch and calculation are performed to obtain an output (i.e., intermediate output), which is represented for example as (1*300*30522). Finally, the output is post-processed. Since the input group is {(input1:1*220, input2:1*220)}, the output can be truncated to obtain the final output, which can be represented as (1*220*30522).

[0081] At least one embodiment of this disclosure also provides a shape reasoning device. Figure 5 A schematic block diagram of a shape reasoning device provided for at least one embodiment of the present disclosure.

[0082] For example, such as Figure 5 As shown, the shape reasoning apparatus 500 includes one or more memories 501 and one or more processors 502. The one or more memories 501 are configured to non-transitory store computer-executable instructions; the one or more processors 502 are configured to execute the computer-executable instructions. The computer-executable instructions are executed by the one or more processors 502 to implement the shape reasoning method according to any of the above embodiments. For specific implementation details and explanations of the various steps of this shape reasoning method, please refer to the description of the embodiments of the shape reasoning method described above, and will not be repeated here.

[0083] For example, in some embodiments, the shape reasoning device 500 may further include a communication interface and a communication bus. The memory 501, processor 502, and communication interface can communicate with each other via the communication bus, and components such as the memory 501, processor 502, and communication interface can also communicate via a network connection. This disclosure does not limit the type and function of the network.

[0084] For example, a communication bus can be a Peripheral Component Interconnect Standard (PCI) bus or an Extended Industry Standard Architecture (EISA) bus. This communication bus can be divided into address bus, data bus, control bus, etc.

[0085] For example, a communication interface is used to enable communication between the shape inference device 500 and other devices. The communication interface can be a Universal Serial Bus (USB) interface, etc.

[0086] For example, memory 501 and processor 502 can be located on the server side (or in the cloud).

[0087] For example, processor 502 can control other components in shape reasoning device 500 to perform desired functions. Processor 502 can be a central processing unit (CPU), network processor (NP), etc.; it can also be a digital signal processor (DSP), application-specific integrated circuit (ASIC), field-programmable gate array (FPGA), or other programmable logic device, discrete gate circuit or transistor logic device, discrete hardware component. The central processing unit (CPU) can be an x86 or ARM architecture, etc.

[0088] For example, memory 501 may include any combination of one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) and / or cache memory. Non-volatile memory may include, for example, read-only memory (ROM), hard disk, erasable programmable read-only memory (EPROM), portable compact disc read-only memory (CD-ROM), USB memory, flash memory, etc. One or more computer-executable instructions may be stored on the computer-readable storage medium, and processor 502 may execute these computer-executable instructions to implement various functions of the shape reasoning device 500. Various application programs and various data may also be stored in the storage medium.

[0089] For example, a detailed description of the shape reasoning process performed by the shape reasoning device 500 can be found in the relevant descriptions in the embodiments of the shape reasoning method, and will not be repeated here. Furthermore, the shape reasoning device 500 can achieve the technical effects achieved by the shape reasoning method described above.

[0090] At least one embodiment of this disclosure also provides a data processing apparatus. Figure 6 A schematic block diagram of a shape reasoning device provided for at least one embodiment of the present disclosure.

[0091] For example, such as Figure 6 As shown, the data processing apparatus 600 includes one or more memories 601 and one or more processors 602. The one or more memories 601 are configured to non-transitory store computer-executable instructions; the one or more processors 602 are configured to execute the computer-executable instructions. The computer-executable instructions are executed by the one or more processors 602 to implement the data processing method according to any of the above embodiments. For specific implementations and explanations of the various steps of this data processing method, please refer to the description of the embodiments of the data processing method described above, and will not be repeated here.

[0092] For example, in some embodiments, the data processing apparatus 600 may further include a communication interface and a communication bus. The memory 601, processor 602, and communication interface can communicate with each other via the communication bus, and components such as the memory 601, processor 602, and communication interface can also communicate with each other via a network connection. This disclosure does not limit the type and function of the network.

[0093] For example, a communication bus can be a Peripheral Component Interconnect Standard (PCI) bus or an Extended Industry Standard Architecture (EISA) bus. This communication bus can be divided into address bus, data bus, control bus, etc.

[0094] For example, the communication interface is used to enable communication between the data processing device 600 and other devices. The communication interface can be a Universal Serial Bus (USB) interface, etc.

[0095] For example, memory 601 and processor 602 can be located on the server side (or in the cloud).

[0096] For example, processor 602 can control other components in data processing device 600 to perform desired functions. Processor 602 can be a central processing unit (CPU), network processor (NP), etc.; it can also be a digital signal processor (DSP), application-specific integrated circuit (ASIC), field-programmable gate array (FPGA), or other programmable logic device, discrete gate circuit or transistor logic device, discrete hardware component. The central processing unit (CPU) can be an x86 or ARM architecture, etc.

[0097] For example, memory 601 may include any combination of one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) and / or cache memory. Non-volatile memory may include, for example, read-only memory (ROM), hard disk, erasable programmable read-only memory (EPROM), portable compact disc read-only memory (CD-ROM), USB memory, flash memory, etc. One or more computer-executable instructions may be stored on the computer-readable storage medium, and processor 602 may execute these computer-executable instructions to implement various functions of data processing device 600. Various application programs and various types of data may also be stored in the storage medium.

[0098] For example, a detailed description of the data processing process performed by the data processing device 600 can be found in the relevant descriptions in the embodiments of the data processing method, and will not be repeated here. Furthermore, the data processing device 600 can achieve the technical effects achieved by the shape reasoning method described above.

[0099] Figure 7 This is a schematic diagram of a non-transitory computer-readable storage medium provided for at least one embodiment of the present disclosure. For example, such as... Figure 7 As shown, one or more computer-executable instructions 701 may be stored non-transitory on the non-transitory computer-readable storage medium 700. For example, when the computer-executable instructions 701 are executed by a processor, one or more steps of the shape reasoning method and / or data processing method described above may be performed.

[0100] For example, the non-transitory computer-readable storage medium 700 can be applied to the shape reasoning device 500 and / or the data processing device 600 described above. For example, it may include the memory 501 in the shape reasoning device 500 and / or the memory 601 in the data processing device 600.

[0101] For details regarding the non-transitory computer-readable storage medium 700, please refer to... Figure 5 Description and / or reference to memory 501 in the embodiment of the shape reasoning device 500 shown Figure 6 The description of the memory 601 in the embodiment of the data processing device 600 shown will not be repeated here.

[0102] The following points should be noted regarding this disclosure:

[0103] (1) The accompanying drawings of the embodiments of this disclosure only involve the structures involved in the embodiments of this disclosure. Other structures can be referred to the general design.

[0104] (2) For clarity, the thickness and dimensions of layers or structures are enlarged in the accompanying drawings used to describe embodiments of the invention. It will be understood that when an element such as a layer, film, region, or substrate is referred to as being “above” or “below” another element, the element may be “directly” located “above” or “below” the other element, or there may be intermediate elements present.

[0105] (3) Where there is no conflict, the embodiments of this disclosure and the features in the embodiments can be combined with each other to obtain new embodiments.

[0106] The above description is only a specific embodiment of this disclosure, but the protection scope of this disclosure is not limited thereto. The protection scope of this disclosure should be determined by the protection scope of the claims.

Claims

1. A shape reasoning method, comprising: Obtain the input shape information; Based on the input shape information, multiple different inference shapes are determined; Shape reasoning is performed on the multiple different inference shapes based on the first network to obtain multiple different inference information groups corresponding to the multiple different inference shapes respectively. The first network includes multiple operators, and the inference information group corresponding to each inference shape includes multiple kernel functions of the multiple operators corresponding to the inference shape, multiple tensor information of the multiple operators corresponding to the inference shape, and memory configuration information corresponding to the inference shape. The system stores multiple different inference information groups, each corresponding one-to-one with a multiple different inference shape. These multiple different inference information groups and shapes belong to an inference cache information set. The inference cache information set is used to determine a current inference shape that matches the shape of the current input of the first network. This allows multiple operators to perform calculations on the shape of the current input based on multiple kernel functions in the current inference information group corresponding to the current inference shape. Each of the multiple different inference shapes corresponds to multiple kernel functions in different groups. The multiple different inference information groups include a first inference information group and a second inference information group. The number of kernel functions in the first inference information group is different from the number of kernel functions in the second inference information group.

2. The shape reasoning method according to claim 1, wherein, The shape reasoning includes: For each reasoning shape: Based on the first network, reasoning is performed on the reasoning shape to obtain a group of reasoning information corresponding to the reasoning shape; Establish a mapping relationship between the reasoning shape and the reasoning information group; Specifically, the reasoning based on the first network to obtain a group of reasoning information corresponding to the reasoning shape includes: Based on the first network, reasoning is performed on the reasoning shape to generate multiple tensor information of the multiple operators corresponding to the reasoning shape; Based on the multiple tensor information, determine the memory configuration information corresponding to the inference shape; Based on the tensor information and the operator information corresponding to the multiple operators, compile multiple kernel functions of the multiple operators that correspond to the inference shape.

3. The shape reasoning method according to claim 1 or 2, wherein, The multiple different inference shapes are determined based on hardware information and the first network.

4. The shape reasoning method according to claim 1 or 2, wherein, The input shape information includes the range of input shapes and the number of shapes to be inferred, or the input shape information includes multiple input shapes.

5. The shape reasoning method according to claim 1 or 2, wherein, The shape reasoning method is executed by the reasoning engine during the establishment phase.

6. A data processing method, comprising: Get the current input of the first network; Obtain the inference shape that matches the shape of the current input from the inference cache information set as the current inference shape; The inference information set is used to obtain the inference information group corresponding to the current inference shape from the inference cache information set as the current inference information group. The inference cache information set includes multiple different inference information groups and multiple different inference shapes that correspond one-to-one with the multiple different inference information groups. The multiple different inference information groups are obtained by the shape inference method according to any one of claims 1 to 5. The current input is preprocessed based on the current inference shape to obtain the processed input; Based on multiple kernel functions in the current inference information group, multiple operators in the first network are used to process the processed input to obtain an intermediate output. The multiple different inference shapes correspond to multiple kernel functions in different groups. The multiple different inference information groups include a first inference information group and a second inference information group. The number of kernel functions in the multiple kernel functions in the first inference information group is different from the number of kernel functions in the multiple kernel functions in the second inference information group. Based on the current input, the intermediate output is post-processed to obtain the final output.

7. The data processing method according to claim 6, wherein, The current input is preprocessed based on the current inference shape to obtain the processed input, including: Compare the shape of the current input with the shape of the current inference; In response to the fact that the shape of the current input is the same as the current inference shape, the current input is directly used as the processed input; In response to the fact that the shape of the current input is different from the current inference shape, the current input is processed based on the current inference shape to obtain the processed input, wherein the shape of the processed input is the same as the current inference shape.

8. The data processing method according to claim 6 or 7, wherein, Based on the current input, the intermediate output is post-processed to obtain the final output, including: Determine the target output corresponding to the current input based on the current input; Compare the shape of the target output with the shape of the intermediate output; Since the shape of the target output is the same as the shape of the intermediate output, the intermediate output is directly used as the final output. In response to the fact that the shape of the target output is different from the shape of the intermediate output, the intermediate output is processed based on the target output to obtain the final output, wherein the shape of the final output is the same as the shape of the target output.

9. The data processing method according to claim 6 or 7, wherein, The current input is processed based on the current inference shape to obtain the processed input, including: The current input is filled or scaled based on the current inference shape to obtain the processed input.

10. The data processing method according to claim 6 or 7, wherein, The current input includes image, audio data, or text data.

11. The data processing method according to claim 6 or 7, wherein, The inference cache information set corresponds to the current input.

12. The data processing method according to claim 6 or 7, wherein, The current inference shape is greater than or equal to the current input shape.

13. A shape reasoning device, comprising: One or more memories that store computer-executable instructions non-transitory; One or more processors, configured to run the computer-executable instructions, The computer-executable instructions are executed by the one or more processors to implement the shape reasoning method according to any one of claims 1 to 5.

14. A data processing apparatus, comprising: One or more memories that store computer-executable instructions non-transitory; One or more processors, configured to run the computer-executable instructions, The computer-executable instructions are executed by the one or more processors to implement the data processing method according to any one of claims 6 to 12.

15. A non-transitory computer-readable storage medium, wherein, The non-transitory computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, implement the shape reasoning method according to any one of claims 1 to 5 or the data processing method according to any one of claims 6 to 12.

Images