Method, device, equipment and medium for reducing peak utilization of neural network CPU

Abstract

The application discloses a method, device and equipment for reducing the CPU peak utilization of a neural network and a medium, and relates to the technical field of computers.The method comprises the following steps: judging the type of a feature vector sequence; if the feature vector sequence is a key frame, inputting the key frame into a neural network, distributing the operation process of forward calculation of the neural network to the key frame according to a first preset proportion, and obtaining the output of an intermediate hidden layer of the neural network; if the feature vector sequence is a transition frame, taking the output of the first intermediate hidden layer as the input of a subsequent intermediate hidden layer, distributing the operation process of forward calculation of the neural network to the transition frame according to a second preset proportion, and obtaining the output of the subsequent intermediate hidden layer; repeating the above process until the final output result of the neural network is obtained; and repeating the above steps until all the feature vector sequences are processed. The application reduces the CPU peak calculation amount in the neural network calculation, thereby reducing the CPU peak utilization and peak power consumption of the neural network.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and in particular to a method, apparatus, device, and medium for reducing the peak CPU utilization of neural networks. Background Technology

[0002] In recent years, with the continuous improvement of computer hardware and algorithms, neural networks have achieved significant breakthroughs in pattern recognition fields, such as speech recognition, voice wake-up, face recognition, and machine translation. Generally speaking, the performance of neural networks increases with the amount of training data and the size of parameters. However, for embedded systems, such as smart hardware, due to limited memory and computing power, the parameter size of neural networks is often limited to a certain range to ensure that the CPU utilization of the neural network is controllable during operation, thereby ensuring the stability of the entire system.

[0003] CPU utilization can be divided into two types: average CPU utilization, which is the CPU usage over a certain time period divided by the time length, and peak CPU utilization, which is the maximum CPU utilization over a certain time period. In existing technologies, reducing the power consumption of neural networks generally falls into two categories: one is related to the neural network itself, such as reducing the number of neural network parameters or optimizing computation using platform acceleration instructions; the other utilizes the continuous nature of neural network input, allowing only keyframes to be computed, while transition frames can directly use the neural network output of the keyframes. This way, transition frames do not undergo neural network computation, thus reducing the average CPU utilization. For example, the frame skipping mechanism disclosed in earlier application CN108764469A reduces the average CPU utilization.

[0004] However, the aforementioned prior applications did not reduce CPU peak utilization, which is crucial to system stability. If not properly controlled, it may cause other modules of the system to lag or even crash. Summary of the Invention

[0005] In view of the aforementioned defects or deficiencies in the prior art, the present invention provides a method, apparatus, device, and medium for reducing the peak CPU utilization of a neural network. By allocating the forward computation process of the neural network in key frames to transition frames in a certain proportion, the computation of the entire neural network is no longer concentrated on key frames, but is distributed among key frames and transition frames in a certain proportion. This reduces the peak CPU computation load in the neural network computation, thereby reducing the peak CPU utilization and peak power consumption of the neural network and improving the stability of the entire system. At the same time, under the same system computing power, the number of parameters in the neural network is increased, thereby improving system performance.

[0006] One aspect of the present invention provides a method for reducing the peak CPU utilization of a neural network, comprising the following steps:

[0007] The judgment step involves determining the type of the feature vector sequence based on its input order into the neural network. The type of the feature vector sequence includes keyframes and transition frames.

[0008] The keyframe processing step is as follows: if the feature vector sequence to be input into the neural network is a keyframe, then the keyframe is input into the neural network, and the forward computation process of the neural network is allocated to the keyframe according to a first preset ratio, thereby obtaining the output of the first intermediate hidden layer of the neural network.

[0009] The transition frame processing step involves the following steps: if the feature vector sequence to be input into the neural network is a transition frame following the keyframe, then the output of the first intermediate hidden layer is used as the input of the subsequent intermediate hidden layer, and the forward computation process of the neural network is allocated to the transition frame according to a second preset ratio to obtain the output of the subsequent intermediate hidden layer; the above process is repeated until the computation process of the transition frame includes the output layer of the neural network, and the final output result of the neural network is obtained.

[0010] Repeat the keyframe processing steps and transition frame processing steps until all feature vector sequences have been processed.

[0011] Furthermore, the first preset ratio and the second preset ratio are evenly distributed based on the computational load of the neural network forward calculation.

[0012] Furthermore, the operations from the input layer to the hidden layer are assigned to the keyframes, and the operations from the hidden layer to the output layer are assigned to the transition layer.

[0013] Furthermore, a predetermined number of transition frames are spaced between keyframes in the feature vector sequence to be input into the neural network.

[0014] In another aspect, the present invention provides an apparatus for reducing the peak CPU utilization of a neural network, comprising:

[0015] The judgment module is configured to determine the type of the feature vector sequence based on the input order of the feature vector sequence to be input into the neural network, wherein the type of the feature vector sequence includes keyframes and transition frames;

[0016] The keyframe processing module is configured to input the keyframe into the neural network if the feature vector sequence to be input into the neural network is a keyframe, and to allocate the forward computation process of the neural network to the keyframe according to a first preset ratio, thereby obtaining the output of the first intermediate hidden layer of the neural network.

[0017] The transition frame processing module is configured to, if the feature vector sequence to be input into the neural network is a transition frame, use the output of the first intermediate hidden layer as the input of the subsequent intermediate hidden layer, allocate the forward computation process of the neural network to the transition frame according to a second preset ratio, so as to obtain the output of the subsequent intermediate hidden layer; repeat the above process until the computation process of the transition frame includes the output layer of the neural network, and obtain the final output result of the neural network.

[0018] Repeat the execution process of the keyframe processing module and the transition frame processing module until all feature vector sequences have been processed.

[0019] Furthermore, the first preset ratio and the second preset ratio are evenly distributed based on the computational load of the neural network forward calculation.

[0020] Furthermore, the operations from the input layer to the hidden layer are assigned to the keyframes, and the operations from the hidden layer to the output layer are assigned to the transition layer.

[0021] Furthermore, a predetermined number of transition frames are spaced between keyframes in the feature vector sequence to be input into the neural network.

[0022] In another aspect, the present invention provides an electronic device comprising:

[0023] One or more processors;

[0024] Storage device for storing one or more programs;

[0025] When the one or more programs are executed by the one or more processors, the one or more processors perform any of the methods described above.

[0026] In another aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method described in any of the preceding claims.

[0027] The present invention provides a method, apparatus, electronic device, and medium for reducing the peak CPU utilization of neural networks, which has the following beneficial effects:

[0028] (1) Under the same neural network model, the peak CPU utilization and peak power consumption required for neural network computation are reduced, thus ensuring the stability of the system.

[0029] (2) Under the same system computing power, the number of neural network parameters was increased, thereby improving system performance. Attached Figure Description

[0030] Other features, objects, and advantages of this application will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings:

[0031] Figure 1 This is a schematic diagram of the DNN neural network calculating each layer according to an embodiment of this application;

[0032] Figure 2 This is a schematic diagram of the skip frame distribution of keyframes and transition frames of the feature vector sequence of the input neural network provided in one embodiment of this application;

[0033] Figure 3 This is a first flowchart of a method for reducing the peak CPU utilization of a neural network according to an embodiment of this application;

[0034] Figure 4 This is a second flowchart of a method for reducing the peak CPU utilization of a neural network according to an embodiment of this application;

[0035] Figure 5 This is a schematic diagram of the structure of a device for reducing the peak CPU utilization of a neural network according to an embodiment of this application;

[0036] Figure 6 This is a schematic diagram of the structure of an electronic device provided in one embodiment of this application. Detailed Implementation

[0037] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0038] The terminology used in the embodiments of this invention is for the purpose of describing particular embodiments only and is not intended to limit the invention. The singular forms “a,” “the,” and “the” as used in the embodiments of this invention and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise.

[0039] It should be understood that although the terms first, second, third, etc., may be used to describe the acquisition modules in the embodiments of the present invention, these acquisition modules should not be limited to these terms. These terms are only used to distinguish the acquisition modules from each other.

[0040] Depending on the context, the word "if" as used here can be interpreted as "when," "when," "in response to determination," or "in response to detection." Similarly, depending on the context, the phrase "if determination" or "if detection (of the stated condition or event)" can be interpreted as "when determination," "in response to determination," "when detection (of the stated condition or event)," or "in response to detection (of the stated condition or event)."

[0041] It should be noted that the directional terms such as "upper," "lower," "left," and "right" used in the embodiments of the present invention are used to describe the angles shown in the accompanying drawings and should not be construed as limiting the embodiments of the present invention. Furthermore, in the context, it should be understood that when it is mentioned that an element is formed "upper" or "lower" of another element, it can not only be formed directly "upper" or "lower" of the other element, but also indirectly "upper" or "lower" of the other element through an intermediate element.

[0042] See Figure 1 Taking a DNN neural network as an example, a neural network includes an input layer, multiple hidden layers, and an output layer. Its forward computation process is as follows:

[0043] The first layer needs to be calculated.

[0044] The second layer requires calculation.

[0045] The third layer needs to be calculated.

[0046] The fourth layer needs to be calculated.

[0047] Where f represents the activation function of the neural network (e.g., ReLU, Sigmoid, etc.), X represents the input feature vector, and W and b1, b2, b3, b o The symbols represent the weights and biases of a neural network, H represents the output of a hidden layer, O represents the output of an output layer, ih indicates the relationship between the input and hidden layers, hh indicates the relationship between hidden layers, and ho indicates the relationship between the hidden and output layers. It is evident that the computational process of a neural network mainly consists of matrix multiplication and activation functions. The more parameters a neural network has, the greater the computational load required for matrix multiplication and activation functions, resulting in higher CPU resources and a higher peak CPU power consumption. Therefore, the peak CPU power consumption of a device is closely related to the computational load of the forward pass of the neural network; the greater the computational load of the forward pass, the higher the peak CPU power consumption of the device.

[0048] The frame skipping scheme disclosed in the earlier application CN108764469A can significantly improve the efficiency of computing neural networks and significantly reduce the average CPU utilization of neural networks. However, since its computing task is still the complete forward computation of the neural network, this method does not improve the peak CPU utilization of the neural network. Excessive peak CPU utilization can easily cause system instability, such as causing system lag or crashes.

[0049] This invention distributes the computation of the keyframe neural network to the transition frames in a certain proportion. In this way, the computation of the entire neural network is no longer concentrated on the keyframe, but is distributed in a certain proportion between the keyframe and the transition frames. Thus, under the same neural network model, the peak computation amount in the neural network computation process is reduced, thereby reducing the peak CPU utilization and peak power consumption of the device.

[0050] For details, see Figure 3 , 4 One embodiment of the present invention discloses a method for reducing the peak CPU utilization of a neural network, comprising:

[0051] Step S101, the judgment step, determines the type of the feature vector sequence based on the input order of the feature vector sequence to be input into the neural network. The type of the feature vector sequence includes keyframes and transition frames.

[0052] Specifically, the feature vector sequence is input into the neural network according to a certain period and time sequence. The feature vector sequence input into the neural network is divided into keyframes and transition frames. Keyframes are the feature vectors that need to be input into the neural network calculation, while transition frames are the feature vectors that the neural network can skip in the calculation. For example Figure 2 As shown, each keyframe can be spaced several transition frames, which are periodically and sequentially input into the neural network. Therefore, based on the input order of the feature vector sequence to the neural network, it is possible to determine whether the feature vector sequence is a keyframe or a transition frame.

[0053] Step S102, key frame processing step: If the feature vector sequence to be input into the neural network is a key frame, then the key frame is input into the neural network, and the forward computation process of the neural network is allocated to the key frame according to a first preset ratio, thereby obtaining the output of the first intermediate hidden layer of the neural network.

[0054] Specifically, the structure of the neural network is analyzed in advance, and the forward computation process of the neural network is allocated to keyframes and transition frames in a certain proportion. Preferably, the proportion should be as balanced as possible; for example, it can be evenly distributed based on the computational load of the forward computation of the neural network. In this step, for a feature vector sequence to be input into the neural network, if the frame is determined to be a keyframe, then the frame is input into the neural network, and the computational load of the forward computation is allocated to the computation process of the keyframe according to the predetermined allocation proportion, thereby obtaining the output of the first intermediate hidden layer in the neural network.

[0055] Step S103, transition frame processing step: If the feature vector sequence to be input into the neural network is a transition frame after the key frame, then the output of the first intermediate hidden layer is used as the input of the subsequent intermediate hidden layer, and the forward computation process of the neural network is allocated to the transition frame according to the second preset ratio to obtain the output of the subsequent intermediate hidden layer. The above process is repeated until the computation process of the transition frame includes the output layer of the neural network, and the final output result of the neural network is obtained.

[0056] Specifically, for a feature vector sequence to be input into the neural network, if the frame is determined to be a transition frame, the output of the first intermediate hidden layer of the neural network in step S102 is used as the input of the next hidden layer of the neural network, and is allocated to the calculation process of the transition frame according to a predetermined allocation ratio to obtain the output of the next hidden layer of the neural network; if the calculation process of the transition frame includes a part of the output layer of the neural network, the final output result of the neural network is obtained.

[0057] It should be noted that the forward computation of a neural network has a hierarchical structure. First, operations from the input layer to the hidden layers are performed; then, operations occur between the hidden layers; and finally, operations from the hidden layers to the output layer are performed. The output of the output layer is the final output of the neural network. In this embodiment, a complete neural network operation is distributed across keyframes and transition frames according to a certain ratio. Keyframes generally involve operations from the input layer to the hidden layers and therefore do not include the output layer. Whether a transition frame includes the output layer depends on whether its operations include operations from the hidden layers to the output layer. Generally speaking, the last transition frame is the final part of the forward computation and will definitely include the output layer, while other transition frames will not.

[0058] Step S104: Repeat the keyframe processing steps and transition frame processing steps until all feature vector sequences have been processed.

[0059] This embodiment distributes the forward computation of the neural network in keyframes to transition frames in a certain proportion. This means the computation of the entire neural network is no longer concentrated on the keyframes, but rather distributed proportionally between them. This reduces the peak CPU computation load in the neural network, thereby lowering the peak CPU utilization and peak power consumption, and improving the stability of the entire system. Furthermore, under the same system computing power, this invention also increases the number of neural network parameters, further improving system performance.

[0060] See Figure 5 Another embodiment of the present invention provides an apparatus 200 for reducing the peak CPU utilization of a neural network, including a judgment module 201, a key frame processing module 202, and a transition frame processing module 203. The apparatus 200 for reducing the peak CPU utilization of a neural network is capable of performing the various steps in the method embodiment.

[0061] A device 200 for reducing the peak CPU utilization of neural networks, comprising:

[0062] The judgment module 201 is configured to determine the type of the feature vector sequence based on the input timing of the feature vector sequence to be input into the neural network, wherein the type of the feature vector sequence includes keyframes and transition frames.

[0063] The keyframe processing module 202 is configured to input the keyframe into the neural network if the feature vector sequence to be input into the neural network is a keyframe, and to allocate the forward computation process of the neural network to the keyframe according to a first preset ratio, thereby obtaining the output of the first intermediate hidden layer of the neural network.

[0064] The transition frame processing module 203 is configured to, if the feature vector sequence to be input into the neural network is a transition frame, use the output of the first intermediate hidden layer as the input of the subsequent intermediate hidden layer, allocate the forward computation process of the neural network to the transition frame according to a second preset ratio, so as to obtain the output of the subsequent intermediate hidden layer; repeat the above process until the computation process of the transition frame includes the output layer of the neural network, and obtain the final output result of the neural network.

[0065] The execution process of the keyframe processing module 202 and the transition frame processing module 203 is repeated until all feature vector sequences have been processed.

[0066] It should be noted that the device 200 for reducing the peak CPU utilization of the neural network provided in this embodiment corresponds to the technical solution that can be used to execute the various method embodiments. Its implementation principle and technical effect are similar to the method, and will not be repeated here.

[0067] See Figure 6Another embodiment of the present invention also provides an electronic device. The electronic device 400 in the embodiments of the present invention may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), in-vehicle terminals (e.g., in-vehicle navigation terminals), wearable electronic devices, etc., as well as fixed terminals such as digital TVs, desktop computers, smart home devices, etc. Figure 6 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of use of the embodiments of the present invention.

[0068] like Figure 6 As shown, electronic device 400 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 401, which can perform various appropriate actions and processes to implement the methods of embodiments of the present invention according to a program stored in read-only memory (ROM) 402 or a program loaded from storage device 408 into random access memory (RAM) 403. Various programs and data required for the operation of electronic device 400 are also stored in RAM 403. The processing device 401, ROM 402, and RAM 403 are interconnected via bus 404. Input / output (I / O) interface 405 is also connected to bus 404.

[0069] Typically, the following devices can be connected to I / O interface 405: input devices 406 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 407 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 408 including, for example, magnetic tapes, hard disks, etc.; and communication devices 409. Communication device 409 allows electronic device 400 to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 6 An electronic device 400 with various devices is shown; however, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively.

[0070] Embodiments of the present invention also provide a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowchart, thereby implementing the methods as described above. In such an embodiment, the computer program can be downloaded and installed from a network via a communication device 409, or installed from a storage device 408, or installed from a ROM 402. When the computer program is executed by a processing device 401, it performs the functions defined in the methods of the embodiments of the present invention.

[0071] It should be noted that the computer-readable medium described above in this invention can be a computer-readable signal medium, a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this invention, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this invention, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0072] The above description is merely a preferred embodiment of the present invention. Those skilled in the art should understand that the scope of disclosure involved in the present invention is not limited to the technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described concept. For example, technical solutions formed by substituting the above-described features with (but not limited to) technical features with similar functions disclosed in the present invention.

Claims

1. A method for reducing the peak CPU utilization of a neural network, characterized in that, Includes the following steps: The judgment step involves determining the type of the feature vector sequence based on the input timing of the feature vector sequence to be input into the neural network. The type of the feature vector sequence includes keyframes and transition frames. The keyframe processing step is as follows: if the feature vector sequence to be input into the neural network is a keyframe, then the keyframe is input into the neural network, and the forward computation process of the neural network is allocated to the keyframe according to a first preset ratio, thereby obtaining the output of the first intermediate hidden layer of the neural network. The transition frame processing step involves the following steps: if the feature vector sequence to be input into the neural network is a transition frame following the keyframe, then the output of the first intermediate hidden layer is used as the input of the subsequent intermediate hidden layer, and the forward computation process of the neural network is allocated to the transition frame according to a second preset ratio to obtain the output of the subsequent intermediate hidden layer; the above process is repeated until the computation process of the transition frame includes the output layer of the neural network, and the final output result of the neural network is obtained. Repeat the keyframe processing steps and transition frame processing steps until all feature vector sequences have been processed.

2. The method for reducing the peak CPU utilization of a neural network according to claim 1, characterized in that, The first preset ratio and the second preset ratio are evenly distributed based on the computational load of the neural network forward calculation.

3. The method for reducing the peak CPU utilization of a neural network according to claim 1, characterized in that, The operations from the input layer to the hidden layer are assigned to the keyframes, and the operations from the hidden layer to the output layer are assigned to the transition frames.

4. The method for reducing the peak CPU utilization of a neural network according to claim 1, characterized in that, A predetermined number of transition frames are spaced between keyframes in the feature vector sequence to be input into the neural network.

5. An apparatus for reducing the peak CPU utilization of a neural network, characterized in that, include: The judgment module is configured to determine the type of the feature vector sequence to be input into the neural network based on the input timing of the feature vector sequence, wherein the type of the feature vector sequence includes keyframes and transition frames; The keyframe processing module is configured to input the keyframe into the neural network if the feature vector sequence to be input into the neural network is a keyframe, and to allocate the forward computation process of the neural network to the keyframe according to a first preset ratio, thereby obtaining the output of the first intermediate hidden layer of the neural network. The transition frame processing module is configured to, if the feature vector sequence to be input into the neural network is a transition frame, use the output of the first intermediate hidden layer as the input of the subsequent intermediate hidden layer, allocate the forward computation process of the neural network to the transition frame according to a second preset ratio, so as to obtain the output of the subsequent intermediate hidden layer; repeat the above process until the computation process of the transition frame includes the output layer of the neural network, and obtain the final output result of the neural network. Repeat the execution process of the keyframe processing module and the transition frame processing module until all feature vector sequences have been processed.

6. The apparatus for reducing the peak CPU utilization of a neural network according to claim 5, characterized in that, The first preset ratio and the second preset ratio are evenly distributed based on the computational load of the neural network forward calculation.

7. The apparatus for reducing the peak CPU utilization of a neural network according to claim 5, characterized in that, The operations from the input layer to the hidden layer are assigned to the keyframes, and the operations from the hidden layer to the output layer are assigned to the transition frames.

8. The apparatus for reducing the peak CPU utilization of a neural network according to claim 5, characterized in that, A predetermined number of transition frames are spaced between keyframes in the feature vector sequence to be input into the neural network.

9. An electronic device, characterized in that, include: One or more processors; Storage device for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1-4.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the method as described in any one of claims 1-4.

Images