A model mixed precision reasoning method, device, equipment and storage medium

Abstract

The present invention discloses a model mixed precision reasoning method, device, equipment and storage medium, including: inputting input samples into a deep learning model in a chip, calculating the input samples through the computing nodes in the chip, and obtaining a target result of type float32; obtaining a segmentation list of the model, and adjusting the precision selection parameters of each segment according to the mixed precision results and target results of the model under preset precision selection parameters for each segment; inputting the target precision selection parameters of each computing node in each segment into the control node as a control signal, selecting a matching precision calculation branch through the control node in the chip, and completing mixed precision reasoning through the computing node according to the precision calculation branch. The technical solution of the embodiment of the present invention can effectively obtain a mixed precision reasoning scheme that meets the model precision requirements and improve the mixed precision reasoning efficiency of the model.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a model mixed precision reasoning method, device, equipment and storage medium. Background Art

[0002] The mixed-precision inference method for deep learning models refers to accelerating the process of deep neural network inference by mixing float16 and float32 data types and reducing memory usage and access, thereby enabling the inference of larger neural networks.

[0003] There are usually two ways to perform mixed-precision inference in existing chip computing frameworks (such as TensorFlow or Pytorch). The first is to customize the inference precision (such as float32 or float16) used by each computing node in the model; the second is to select based on the blacklist and whitelist defined by the framework.

[0004] However, the first method requires users to have strong theoretical knowledge of model calculation, and the second method may not necessarily find a mixed-precision inference solution that meets the accuracy requirements; and existing inference frameworks (such as TensorRT, etc.) need to use an iterative method to find an effective mixed-precision solution when building a mixed-precision inference solution. Each iteration requires the chip to compile the model. As the number of iterations increases, the compilation time increases exponentially. Summary of the Invention

[0005] The present invention provides a model mixed precision inference method, device, equipment and storage medium, which can effectively obtain a mixed precision inference scheme that meets the model accuracy requirements, improve the mixed precision inference efficiency of the model, and save the computing resources of the chip during the model mixed precision inference process.

[0006] According to one aspect of the present invention, a model mixed precision reasoning method is provided, the method comprising:

[0007] Input samples are fed into the deep learning model within the chip. Multiple computing nodes within the chip corresponding to the deep learning model compute the input samples and obtain the target result of type float32. The initial precision selection parameter of each computing node is set to true.

[0008] Obtain a list of segments corresponding to the model, and adjust the precision selection parameters corresponding to each segment according to the mixed precision results of the model for each segment under preset precision selection parameters and the target result; wherein each segment includes at least one computing node;

[0009] Input the target precision selection parameter corresponding to each computing node in each segment as a control signal to the control node corresponding to each computing node, select the precision calculation branch that matches the computing node according to the control signal through the control node in the chip, and complete the mixed precision inference according to the precision calculation branch through the computing node;

[0010] Among them, each computing node is pre-corresponded to a float32 precision computing branch and a float16 precision computing branch.

[0011] Optionally, before inputting the input sample into the deep learning model in the chip, the following steps are also included:

[0012] sorting a plurality of topological structures included in the model;

[0013] According to the topological sorting results, a float16 precision calculation branch is added to the calculation nodes corresponding to each topological structure.

[0014] Optionally, after obtaining the segment list corresponding to the model, the following is further included:

[0015] Presetting the precision selection parameters corresponding to each segment according to the segment type corresponding to each segment;

[0016] The input sample is input into a deep learning model, and the model selects parameters according to the preset accuracy corresponding to each segment, and processes the input sample to obtain a mixed precision result.

[0017] Optionally, according to the segment type corresponding to each segment, the accuracy selection parameter corresponding to each segment is preset, including:

[0018] Obtain the longest segment, segments of known data types, and segments of unknown data types from the segment list;

[0019] Set the precision selection parameter corresponding to the longest segment to false;

[0020] According to the target data type corresponding to the known data type segment, the precision selection parameter corresponding to the known data type segment is set to true or false;

[0021] Set the precision selection parameter corresponding to the unknown data type segment to true.

[0022] Optionally, according to the mixed precision result of the model for each segment under the preset precision selection parameter and the target result, the precision selection parameter corresponding to each segment is adjusted, including:

[0023] Constructing an evaluation standard based on the blending result and the target result;

[0024] According to the evaluation criteria, judging whether the mixed sperm results are qualified;

[0025] If so, set the precision selection parameter corresponding to each computing node in the longest segment to false, remove the longest segment from the segment list, and then return to execute the operation of obtaining the longest segment in the segment list.

[0026] Optionally, after determining whether the blended sperm results are qualified according to the evaluation criteria, the method further includes:

[0027] If not, determining whether the longest segment meets the fission conditions;

[0028] If so, the longest segment is split into a first segment and a second segment, and the first segment and the second segment are added to the segment list, and then the operation of obtaining the longest segment in the segment list is returned.

[0029] Optionally, before inputting the target accuracy selection parameter corresponding to each computing node in each segment as a control signal to the control node corresponding to each computing node, the method further includes:

[0030] Determine whether the segment list is empty;

[0031] If so, the current precision selection parameters corresponding to each computing node in each segment are obtained, and the current precision selection parameters are used as the target precision selection parameters.

[0032] According to another aspect of the present invention, a model mixed precision reasoning device is provided, the device comprising:

[0033] The target result generation module is used to input the input sample into the deep learning model in the chip, calculate the input sample through multiple computing nodes in the chip corresponding to the deep learning model, and obtain the target result of float32 type; the initial precision selection parameter of each computing node is true;

[0034] a parameter adjustment module, configured to obtain a list of segments corresponding to the model, and adjust the precision selection parameters corresponding to each segment based on the mixed precision results of the model for each segment under preset precision selection parameters and the target result; wherein each segment includes at least one computing node;

[0035] A branch selection module is used to input the target precision selection parameters corresponding to each computing node in each segment as control signals into the control nodes corresponding to each computing node, select the precision calculation branch that matches the computing node according to the control signals through the control nodes in the chip, and complete mixed precision inference according to the precision calculation branches through the computing nodes;

[0036] Among them, each computing node is pre-corresponded to a float32 precision computing branch and a float16 precision computing branch.

[0037] According to another aspect of the present invention, there is provided an electronic device, the device comprising:

[0038] at least one processor; and

[0039] a memory communicatively connected to the at least one processor; wherein,

[0040] The memory stores a computer program that can be executed by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can perform the model mixed precision reasoning method described in any embodiment of the present invention.

[0041] According to another aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the model mixed precision reasoning method described in any embodiment of the present invention when executed.

[0042] The technical solution provided by the embodiment of the present invention inputs input samples into the deep learning model in the chip, calculates the input samples through multiple computing nodes corresponding to the deep learning model in the chip, obtains the target result of float32 type, obtains the segmentation list corresponding to the model, and adjusts the precision selection parameters corresponding to each segment according to the model under the preset precision selection parameters, and the target result. The target precision selection parameters corresponding to each computing node in each segment are input as control signals to the control nodes corresponding to each computing node. The control node in the chip selects the precision calculation branch matching the computing node according to the control signal, and the computing node completes the mixed precision reasoning according to the precision calculation branch. This technical means can effectively obtain a mixed precision reasoning scheme that meets the model precision requirements, improve the mixed precision reasoning efficiency of the model, and save the computing resources of the chip during the mixed precision reasoning process of the model.

[0043] The technical solution provided by this embodiment of the present invention can be applied to text detection and image recognition. When a deep learning model experiences overflow or underflow during inference with FP16 precision, leading to inference errors, the technical solution of this embodiment can automatically correct the overflow and use FP32 precision for inference on the overflowed nodes.

[0044] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present invention, nor is it intended to limit the scope of the present invention. Other features of the present invention will become readily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0046] Figure 1 This is a flow chart of a model mixed precision reasoning method provided according to an embodiment of the present invention;

[0047] Figure 2 is a flowchart of another model mixed precision reasoning method provided according to an embodiment of the present invention;

[0048] Figure 3 is a flowchart of another model mixed precision reasoning method provided according to an embodiment of the present invention;

[0049] Figure 4 2. It is a structural diagram of a model mixed precision reasoning device provided according to an embodiment of the present invention;

[0050] Figure 5 It is a schematic diagram of the structure of an electronic device for implementing the model mixed precision reasoning method of an embodiment of the present invention. DETAILED DESCRIPTION

[0051] In order to enable those skilled in the art to better understand the solutions of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of the present invention.

[0052] It should be noted that the terms "first", "second", etc. in the description and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that the numbers used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0053] Figure 1This is a flow chart of a model mixed precision reasoning method provided in the first embodiment of the present invention. This embodiment is applicable to the case of mixed precision reasoning of deep learning models. The method can be executed by a model mixed precision reasoning device. The model mixed precision reasoning device can be implemented in the form of hardware and / or software. The model mixed precision reasoning device can be configured in an electronic device. Figure 1 As shown, the method includes:

[0054] Step 110: Input the input sample into the deep learning model in the chip, calculate the input sample through multiple computing nodes in the chip corresponding to the deep learning model, and obtain the target result of float32 type.

[0055] In this embodiment, the input sample can be a user-selected learning sample corresponding to the deep learning model. After obtaining the input sample, it can be input into the model. Each computing node within the chip calculates the input sample according to the initial precision selection parameter, and obtains the target result of type float32. Specifically, the precision selection parameter is used to indicate the calculation precision used by the computing node when processing data.

[0056] In a specific embodiment, each computing node is pre-assigned a float32 precision computing branch and a float16 precision computing branch. The initial precision selection parameter of each computing node can be true, that is, in this step, each computing node can use the float32 precision computing branch to calculate the input sample.

[0057] Step 120: Obtain a segment list corresponding to the model, and adjust the precision selection parameters corresponding to each segment according to the mixed precision results of the model for each segment under the preset precision selection parameters and the target result.

[0058] In this embodiment, the plurality of computing nodes in the model may constitute a plurality of segments, and the plurality of segments form the segment list, wherein each segment may include at least one computing node.

[0059] In this step, after obtaining the segment list corresponding to the model, you can preset the precision selection parameters of the calculation nodes in each segment (for example, set them to true or false). Then, under the preset precision selection parameters, each calculation node uses a mixture of float16 and float32 precision calculation branches to process the input samples and obtain a mixed precision result.

[0060] In a specific embodiment, optionally, after obtaining the above-mentioned blending result, it can be determined whether the blending result meets the preset standard based on the target result. If not, the accuracy selection parameters corresponding to each segment are adjusted, and a new blending result is obtained again until the blending result meets the preset standard.

[0061] Step 130: Input the target precision selection parameter corresponding to each computing node in each segment as a control signal into the control node corresponding to each computing node. The control node in the chip selects the precision calculation branch that matches the computing node according to the control signal, and completes the mixed precision inference through the precision calculation branch.

[0062] In this embodiment, the adjusted precision selection parameters of each segment can be used as the target precision selection parameters corresponding to the computing nodes in each segment. After obtaining the target precision selection parameters corresponding to the computing nodes, the target precision selection parameters can be used as control signals and input into the control node corresponding to the computing nodes.

[0063] In a specific embodiment, if the target precision selection parameter is "true", the control node can select the float32 precision calculation branch that matches the calculation node and process the data through the calculation branch; conversely, if the target precision selection parameter is "false", the control node can select the float16 precision calculation branch that matches the calculation node and process the data through the calculation branch, thereby completing mixed-precision reasoning.

[0064] In this embodiment, by pre-determining the precision calculation branches that match each computing node in the model before the model mixed precision reasoning, the model can be compiled once during the model mixed precision reasoning process, thereby avoiding the time consumption caused by the chip compiling the model multiple times; secondly, by adjusting the precision selection parameters of each segment according to the target result and the mixed precision result, a mixed precision reasoning solution that meets the model accuracy requirements can be effectively obtained without the user having strong theoretical knowledge of model calculation.

[0065] By inputting input samples into the deep learning model in the chip, calculating the input samples through multiple computing nodes corresponding to the deep learning model in the chip, obtaining the target result of float32 type, obtaining the segmentation list corresponding to the model, and adjusting the precision selection parameters corresponding to each segment according to the model under the preset precision selection parameters, and the target result, the target precision selection parameters corresponding to each computing node in each segment are input as control signals to the control nodes corresponding to each computing node. The control nodes in the chip select the precision calculation branches that match the computing nodes according to the control signals, and the computing nodes complete the mixed precision reasoning according to the precision calculation branches. This technical means can effectively obtain a mixed precision reasoning scheme that meets the model precision requirements, improve the mixed precision reasoning efficiency of the model, and save the computing resources of the chip during the mixed precision reasoning process of the model.

[0066] Figure 2 This is a flow chart of a model mixed precision reasoning method provided in the second embodiment of the present invention. This embodiment is a further refinement of the above embodiment. Figure 2 As shown, the method includes:

[0067] Step 210: Sort the multiple topological structures included in the model, and add float16 precision calculation branches to the calculation nodes corresponding to each topological structure according to the topological sorting result.

[0068] In this step, multiple topological structures included in the model can be first obtained, then the multiple topological structures are sorted, and based on the topological sorting results, multiple computation nodes corresponding to each topological structure are sequentially obtained. Each computation node is pre-assigned to a float32 precision node, and the computation node and the float32 precision node constitute a float32 precision computation branch.

[0069] After obtaining multiple computing nodes, a precision conversion node (ie, f32 / f16 data type conversion node) may be added to each computing node, thereby forming a float16 precision computing branch of the computing node.

[0070] In this embodiment, each computing node in the chip can correspond to a control node. The control node can be combined with the high-precision and low-precision computing branches corresponding to the computing node into one operator for execution, thereby avoiding waste of the chip's computing power and storage resources.

[0071] Step 220: Input the input sample into the deep learning model in the chip, calculate the input sample through multiple computing nodes in the chip corresponding to the deep learning model, and obtain the target result of float32 type; the initial precision selection parameter of each computing node is true.

[0072] Step 230: Obtain a segment list corresponding to the model, and preset the accuracy selection parameter corresponding to each segment according to the segment type corresponding to each segment.

[0073] In this step, optionally, different precision selection parameters can be preset for each segment according to different segment types, so that each computing node uses a mixture of float16 and float32 precision calculation branches under the preset precision selection parameters to process the input samples and obtain mixed precision results.

[0074] Step 240: Input the input sample into a deep learning model, and use the model to select parameters according to the preset accuracy corresponding to each segment to process the input sample and obtain a mixed precision result.

[0075] Step 250: Adjust the precision selection parameters corresponding to each segment according to the mixed precision results of the model for each segment under the preset precision selection parameters and the target result.

[0076] Step 260: Input the target precision selection parameters corresponding to each computing node in each segment as control signals into the control nodes corresponding to each computing node. The control nodes in the chip select the precision calculation branches that match the computing nodes according to the control signals, and complete the mixed precision inference according to the precision calculation branches through the computing nodes.

[0077] The technical solution provided by the embodiment of the present invention sorts multiple topological structures included in the model, adds a float16 precision calculation branch to the computing node corresponding to each topological structure according to the topological sorting result, inputs the preset input sample into the model, calculates the input sample through multiple computing nodes corresponding to the deep learning model in the chip, obtains a target result of float32 type, obtains a segmentation list of the model, presets the precision selection parameter of each segment according to the segmentation type of each segment, inputs the input sample into the deep learning model, processes the input sample according to the preset precision selection parameter of each segment through the model to obtain a mixed precision result, adjusts the precision selection parameter corresponding to each segment according to the mixed precision result and the target result, inputs the target precision selection parameter of each computing node as a control signal into the control node, selects the precision calculation branch through the control node in the chip, and completes the mixed precision reasoning through the precision calculation branch. The technical means can effectively obtain a mixed precision reasoning scheme that meets the model precision requirements, improve the mixed precision reasoning efficiency of the model, and save the computing resources of the chip during the mixed precision reasoning process of the model.

[0078] Figure 3 This is a flow chart of another model mixed precision reasoning method provided in the third embodiment of the present invention. This embodiment is a further refinement of the above embodiment. Figure 3 As shown, the method includes:

[0079] Step 310: Input the input sample into the deep learning model in the chip, calculate the input sample through multiple computing nodes in the chip corresponding to the deep learning model, and obtain the target result of float32 type; the initial precision selection parameter of each computing node is true.

[0080] Step 320: Obtain a segment list corresponding to the model, and obtain the longest segment, segments of known data types, and segments of unknown data types in the segment list.

[0081] In this embodiment, the known data type segment may be a segment that has been determined to use a specific calculation precision, and the unknown data type segment may be a segment whose calculation precision is unknown and waiting for precision selection parameter adjustment.

[0082] Step 330: Set the precision selection parameter corresponding to the longest segment to false; set the precision selection parameter corresponding to the known data type segment to true or false according to the target data type corresponding to the known data type segment; set the precision selection parameter corresponding to the unknown data type segment to true.

[0083] In this step, the precision selection parameter of the longest segment can be set to false, that is, each computing node in the longest segment uses the float16 precision computing branch for calculation.

[0084] If float16 precision is determined to be used in the known data type segment, the corresponding precision selection parameter can be set to false. Conversely, if float32 precision is determined to be used, the corresponding precision selection parameter can be set to true.

[0085] Step 340: Input the input sample into the deep learning model, and use the model to select parameters according to the preset accuracy corresponding to each segment to process the input sample to obtain a mixed precision result.

[0086] Step 350: Construct an evaluation standard based on the blending result and the target result, and determine whether the blending result is qualified based on the evaluation standard. If so, execute step 360; if not, execute step 370.

[0087] Step 360: Set the precision selection parameter corresponding to each computing node in the longest segment to false, remove the longest segment from the segment list, and then return to execute the operation of obtaining the longest segment from the segment list in step 320.

[0088] In this step, if the above-mentioned blending result is qualified, the longest segment can be removed, and then the longest segment is re-obtained from the segment list to obtain a new blending result.

[0089] Step 370: If the longest segment meets the fission condition, the longest segment is split into a first segment and a second segment, and the first segment and the second segment are added to the segment list, and then the operation of obtaining the longest segment in the segment list in step 320 is returned to be executed.

[0090] In one implementation of this embodiment, if the segment length of the longest segment (i.e., the number of computing nodes in the segment) is greater than 1, the longest segment is considered to meet the fission conditions. In this case, the longest segment can be split using a binary split method to obtain a first segment and a second segment. The first segment and the second segment are then added to the segment list, and the longest segment is retrieved from the segment list again to obtain a new blending result, until the blending result is qualified.

[0091] In one implementation of this embodiment, if the segment length of the longest segment is equal to 1, it can be considered that the longest segment does not meet the fission condition. In this case, the accuracy selection parameter of the longest segment can be set to true, and the longest segment can be removed from the segment list.

[0092] Step 380: If the segment list is empty, the target precision selection parameter corresponding to each computing node in each segment is input as a control signal to the control node corresponding to each computing node. The control node selects the precision calculation branch that matches the computing node according to the control signal, and completes the mixed precision reasoning through the precision calculation branch.

[0093] In this embodiment, after processing each segment, it is possible to determine whether the segment list is empty. If so, the current precision selection parameters corresponding to each computing node in each segment can be obtained and used as the target precision selection parameters.

[0094] The technical solution provided by the embodiment of the present invention obtains a target result of type float32 by inputting an input sample into the model, sets the precision selection parameter of the longest segment to false, sets the precision selection parameter of the known data type segment to true or false, and sets the precision selection parameter of the unknown data type segment to true. The input sample is processed by the model according to the preset precision selection parameter to obtain a mixed precision result. If the mixed precision result is qualified, the precision selection parameter of each computing node in the longest segment is set to false, and the longest segment is removed and then the operation of obtaining the longest segment is returned. If it is unqualified, it is determined whether the longest segment meets the fission condition. If so, the longest segment is fissioned into the first segment and the second segment, and the first segment and the second segment are added to the segment list, and then the operation of obtaining the longest segment is returned. If the segment list is empty, the target precision selection parameter of each computing node is input into the control node as a control signal, and the precision calculation branch is selected by the control node. The mixed precision reasoning technical means is completed through the precision calculation branch. The mixed precision reasoning solution that meets the model precision requirements can be effectively obtained, thereby improving the mixed precision reasoning efficiency of the model.

[0095] Figure 4 This is a schematic diagram of the structure of a model mixed precision inference device provided by the fourth embodiment of the present invention, which is applied to electronic equipment. Figure 4 As shown, the apparatus includes: a target result generating module 410 , a parameter adjusting module 420 and a branch selecting module 430 .

[0096] The target result generation module 410 is used to input the input sample into the deep learning model in the chip, calculate the input sample through multiple computing nodes in the chip corresponding to the deep learning model, and obtain the target result of float32 type; the initial precision selection parameter of each computing node is true;

[0097] A parameter adjustment module 420 is configured to obtain a list of segments corresponding to the model and adjust the precision selection parameters corresponding to each segment based on the mixed precision results of the model for each segment under preset precision selection parameters and the target result; wherein each segment includes at least one computing node;

[0098] The branch selection module 430 is used to input the target precision selection parameter corresponding to each computing node in each segment as a control signal to the control node corresponding to each computing node, and the control node in the chip selects the precision calculation branch that matches the computing node according to the control signal, and the computing node performs mixed precision inference according to the precision calculation branch;

[0099] Among them, each computing node is pre-corresponded to a float32 precision computing branch and a float16 precision computing branch.

[0100] The technical solution provided by the embodiment of the present invention inputs input samples into the deep learning model in the chip, calculates the input samples through multiple computing nodes corresponding to the deep learning model in the chip, obtains the target result of float32 type, obtains the segmentation list corresponding to the model, and adjusts the precision selection parameters corresponding to each segment according to the model under the preset precision selection parameters, and the target result. The target precision selection parameters corresponding to each computing node in each segment are input as control signals to the control nodes corresponding to each computing node. The control node in the chip selects the precision calculation branch matching the computing node according to the control signal, and the computing node completes the mixed precision reasoning according to the precision calculation branch. This technical means can effectively obtain a mixed precision reasoning scheme that meets the model precision requirements, improve the mixed precision reasoning efficiency of the model, and save the computing resources of the chip during the mixed precision reasoning process of the model.

[0101] Based on the above embodiment, the device further includes:

[0102] A topological sorting module, configured to sort the multiple topological structures included in the model;

[0103] The branch adding module is used to add float16 precision calculation branches to the calculation nodes in each topological structure according to the topological sorting results.

[0104] The parameter adjustment module 420 includes:

[0105] A parameter preset unit, configured to preset the accuracy selection parameter corresponding to each segment according to the segment type corresponding to each segment;

[0106] A sample input unit, configured to input the input sample into a deep learning model, and process the input sample to obtain a mixed precision result through the model according to a preset precision selection parameter corresponding to each segment;

[0107] A segment acquisition unit, configured to acquire the longest segment, segments of known data types, and segments of unknown data types from the segment list;

[0108] A segmentation parameter setting unit is configured to set the precision selection parameter corresponding to the longest segment to false; set the precision selection parameter corresponding to the known data type segment to true or false according to the target data type corresponding to the known data type segment; and set the precision selection parameter corresponding to the unknown data type segment to true;

[0109] An evaluation standard construction unit, configured to construct an evaluation standard based on the blending result and the target result;

[0110] A blending result judging unit, configured to judge whether the blending result is qualified according to the evaluation criteria;

[0111] A segment removal unit is configured to, when the precision mixing result is qualified, set the precision selection parameter corresponding to each computing node in the longest segment to false, remove the longest segment from the segment list, and then return to execute the operation of obtaining the longest segment in the segment list;

[0112] A segment determination unit, configured to determine whether the longest segment meets the fission condition;

[0113] a fission unit, configured to, when the longest segment meets the fission condition, split the longest segment into a first segment and a second segment, add the first segment and the second segment to a segment list, and then return to execute an operation of obtaining the longest segment in the segment list;

[0114] The segment list judgment unit is used to judge whether the segment list is empty; if so, obtain the current precision selection parameters corresponding to each computing node in each segment, and use the current precision selection parameters as target precision selection parameters.

[0115] The above device can execute the methods provided by all the above embodiments of the present invention, and has the corresponding functional modules and beneficial effects of executing the above methods. For technical details not fully described in the embodiments of the present invention, please refer to the methods provided by all the above embodiments of the present invention.

[0116] Figure 5 A schematic diagram of the structure of an electronic device 10 that can be used to implement an embodiment of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or claimed herein.

[0117] like Figure 5As shown, the electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., which is communicatively connected to the at least one processor 11. The memory stores a computer program that can be executed by the at least one processor. The processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 into the random access memory (RAM) 13. Various programs and data required for the operation of the electronic device 10 can also be stored in the RAM 13. The processor 11, ROM 12, and RAM 13 are connected to each other via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0118] Multiple components in the electronic device 10 are connected to the I / O interface 15, including an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a magnetic disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0119] The processor 11 can be any general-purpose and / or specialized processing component with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various specialized artificial intelligence (AI) computing chips, various processors that run machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 11 executes the various methods and processes described above, such as the model mixed precision inference method.

[0120] In some embodiments, the model mixed precision reasoning method can be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as the storage unit 18. In some embodiments, part or all of the computer program can be loaded and / or installed on the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of the model mixed precision reasoning method described above can be performed. Alternatively, in other embodiments, the processor 11 can be configured to execute the model mixed precision reasoning method in any other appropriate manner (for example, by means of firmware).

[0121] Various embodiments of the systems and techniques described herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), system-on-chip systems (SOCs), programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include being implemented in one or more computer programs that are executable and / or interpreted on a programmable system that includes at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0122] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when the computer program is executed by the processor, the functions / operations specified in the flowcharts and / or block diagrams are implemented. The computer program may be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0123] In the context of the present invention, computer-readable storage media can be tangible media that can contain or store a computer program for use with an instruction execution system, device or equipment or used in combination with an instruction execution system, device or equipment. Computer-readable storage media can include but are not limited to electronic, magnetic, optical, electromagnetic, infrared or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, computer-readable storage media can be machine-readable signal media. More specific examples of machine-readable storage media can include electrical connections based on one or more lines, portable computer disks, hard disks, random access memories (RAM), read-only memories (ROM), erasable programmable read-only memories (EPROM or flash memory), optical fibers, portable compact disk read-only memories (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0124] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0125] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0126] A computing system may include clients and servers. The clients and servers are typically remote from each other and typically interact via a communication network. This client-server relationship arises through computer programs running on the respective computers, creating a client-server relationship. The server may be a cloud server, also known as a cloud computing server or cloud host. This server is a hosting product within the cloud computing service ecosystem that addresses the management difficulties and limited scalability of traditional physical hosting and VPS services.

[0127] It should be understood that the various forms of the processes shown above can be used to reorder, add, or delete steps. For example, the steps described in the present invention can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solution of the present invention can be achieved. This is not limited herein.

[0128] The above specific embodiments do not limit the scope of protection of the present invention. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions may be made based on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention are intended to be included within the scope of protection of the present invention.

Claims

1. A model mixed precision inference method, characterized in that: The method comprises: Input samples are fed into the deep learning model within the chip. Multiple computing nodes within the chip corresponding to the deep learning model compute the input samples and obtain the target result of type float32. The initial precision selection parameter of each computing node is set to true. Obtain a list of segments corresponding to the model, and adjust the precision selection parameters corresponding to each segment according to the mixed precision results of the model for each segment under preset precision selection parameters and the target result; wherein each segment includes at least one computing node; Input the target precision selection parameter corresponding to each computing node in each segment as a control signal to the control node corresponding to each computing node, select the precision calculation branch that matches the computing node according to the control signal through the control node in the chip, and complete the mixed precision inference according to the precision calculation branch through the computing node; Among them, each computing node is pre-corresponded to a float32 precision computing branch and a float16 precision computing branch.

2. The method according to claim 1, characterized in that Before the input sample is fed into the deep learning model in the chip, it also includes: sorting a plurality of topological structures included in the model; According to the topological sorting results, a float16 precision calculation branch is added to the calculation nodes corresponding to each topological structure.

3. The method according to claim 1, characterized in that After obtaining the segment list corresponding to the model, it also includes: Presetting the precision selection parameters corresponding to each segment according to the segment type corresponding to each segment; The input sample is input into a deep learning model, and the model selects parameters according to the preset accuracy corresponding to each segment, and processes the input sample to obtain a mixed precision result.

4. The method according to claim 3, characterized in that According to the segment type corresponding to each segment, the accuracy selection parameters corresponding to each segment are preset, including: Obtain the longest segment, segments of known data types, and segments of unknown data types from the segment list; Set the precision selection parameter corresponding to the longest segment to false; According to the target data type corresponding to the known data type segment, the precision selection parameter corresponding to the known data type segment is set to true or false; Set the precision selection parameter corresponding to the unknown data type segment to true.

5. The method according to claim 4, characterized in that According to the mixed precision results of the model for each segment under the preset precision selection parameters and the target result, the precision selection parameters corresponding to each segment are adjusted, including: Constructing an evaluation standard based on the blending result and the target result; According to the evaluation criteria, judging whether the mixed sperm results are qualified; If so, set the precision selection parameter corresponding to each computing node in the longest segment to false, remove the longest segment from the segment list, and then return to execute the operation of obtaining the longest segment in the segment list.

6. The method according to claim 5, characterized in that After determining whether the blended sperm results are qualified according to the evaluation criteria, the method further includes: If not, determining whether the longest segment meets the fission conditions; If so, the longest segment is split into a first segment and a second segment, and the first segment and the second segment are added to the segment list, and then the operation of obtaining the longest segment in the segment list is returned.

7. The method according to claim 1, characterized in that Before inputting the target accuracy selection parameters corresponding to each computing node in each segment as control signals to the control nodes corresponding to each computing node, the method further includes: Determine whether the segment list is empty; If so, the current precision selection parameters corresponding to each computing node in each segment are obtained, and the current precision selection parameters are used as the target precision selection parameters.

8. A model mixed precision inference device, characterized in that: The device comprises: The target result generation module is used to input the input sample into the deep learning model in the chip, calculate the input sample through multiple computing nodes in the chip corresponding to the deep learning model, and obtain the target result of float32 type; the initial precision selection parameter of each computing node is true; a parameter adjustment module, configured to obtain a list of segments corresponding to the model, and adjust the precision selection parameters corresponding to each segment based on the mixed precision results of the model for each segment under preset precision selection parameters and the target result; wherein each segment includes at least one computing node; A branch selection module is used to input the target precision selection parameters corresponding to each computing node in each segment as control signals into the control nodes corresponding to each computing node, select the precision calculation branch that matches the computing node according to the control signals through the control nodes in the chip, and complete mixed precision inference according to the precision calculation branches through the computing nodes; Among them, each computing node is pre-corresponded to a float32 precision computing branch and a float16 precision computing branch.

9. An electronic device, characterized in that: The device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor. The computer program is executed by the at least one processor to enable the at least one processor to perform the model mixed precision inference method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the model mixed precision reasoning method according to any one of claims 1 to 7 when executed.

Images